base-layer/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

Require generically renderable inspection results

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-06 13:58:28 +00:00
parent a0f696181f
commit 951739eda0
1 changed files with 106 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
docs/roadmap/inspect-backend.md
View File

@@ -9,8 +9,9 @@ presentation; Promptkit exposes the selected resource's resolved configuration.
The direction is accepted for planning; implementation has not begun. Selected The direction is accepted for planning; implementation has not begun. Selected
work includes fuller prompt inspection, profile inspection of the resolved work includes fuller prompt inspection, profile inspection of the resolved
profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt
construction through `PromptConfig`/`PromptSource`, and simplification of construction through `PromptConfig`/`PromptSource`, generically renderable
Promptkit's post-handoff null handling. Execution-policy reporting inspection results, and simplification of Promptkit's post-handoff null handling.
Execution-policy reporting
and prepared-request enhancements have been removed from this roadmap. The and prepared-request enhancements have been removed from this roadmap. The
decisions below record the accepted scope and representation approach. decisions below record the accepted scope and representation approach.
@@ -76,9 +77,8 @@ values, including zero. Absence is not displayed as zero or `unspecified`.
Treat null like omission for nullable resource fields after ConfigKit resolution, Treat null like omission for nullable resource fields after ConfigKit resolution,
under the selected post-handoff semantics below. Literal nulls inside provider under the selected post-handoff semantics below. Literal nulls inside provider
data retain their meaning; they are not optional resource-field presence states. data retain their meaning; they are not optional resource-field presence states.
The previous draft's rule to omit suppressed outbound controls belonged to an Accepted decision 4 selects typed results without a third state for nullable
execution report and does not define resource inspection. Accepted decision 4 resource fields.
selects typed results without a third state for nullable resource fields.
Inspection reports effective resource values rather than original YAML layout, Inspection reports effective resource values rather than original YAML layout,
comments, or the history of each inherited field. `InspectConfiguration` remains comments, or the history of each inherited field. `InspectConfiguration` remains
@@ -90,10 +90,52 @@ Retain existing captured-state, namespace, ownership, and credential-handling
contracts unless a change is deliberately selected. Mutable results should not contracts unless a change is deliberately selected. Mutable results should not
let a consumer change retained engine state; that does not require repeated let a consumer change retained engine state; that does not require repeated
copies between private helpers or copying immutable strings. Ordinary resource copies between private helpers or copying immutable strings. Ordinary resource
inspection needs no rendering, provider calls, or credential availability checks. inspection needs no template rendering, provider calls, or credential
availability checks.
Templates and arbitrary provider settings remain application data whose Templates and arbitrary provider settings remain application data whose
presentation and disclosure are controlled by the consuming application. presentation and disclosure are controlled by the consuming application.
## Generic Rendering And Presence
**Requirement:** Consumers must be able to render `InspectPrompt`,
`InspectProfile`, and `InspectBackend` results generically, without maintaining
parallel resource schemas or field-by-field adapters. Keep typed Go results for
programmatic use and make them directly encodable as JSON and YAML using
ordinary encoders. A consumer calls inspection, handles the error, and passes
the result to its selected encoder; it need not interpret individual settings
or register each nested Promptkit type.
Promptkit owns serialization field names, nesting, and omission semantics for
these results. Use consistent names aligned with configuration fields where
applicable. CommandKit owns command selection, output format, presentation, and
delivery. Adding an inspection field should not require another CommandKit
field mapping after updating its Promptkit dependency. Consumers may still
choose custom layouts, but generic rendering must be sufficient for a complete
resource report.
Capture presence before converting configured values to execution values that
collapse omission into zero, false, or an empty string. This applies across all
three resource kinds, including prompt input `required`, output
`repair_attempts`, and backend capacity settings. Use small inspection-specific
values where existing types such as `PromptInputDefinition` or `OutputContract`
cannot preserve that distinction. Do not change shared execution types merely
to make them suitable for inspection. Existing configuration validation rules,
including rejection of null for non-nullable fields, remain intact.
Omit absent optional settings and nullable fields resolved to null. Retain
explicit zero, false, empty strings, and explicit empty collections where
represented in the resolved resource, along with literal nulls inside provider
data. Generic encoding must preserve these distinctions; a blanket `omitempty`
on scalar or collection fields is not sufficient. Prefer ordinary optional
fields and serialization tags, with focused encoding support only where needed
for the selected semantics. Consumers must not reconstruct presence themselves.
This is a modest serialization contract for inspection results, not a new
reporting framework, configuration round-trip API, or promise of byte-for-byte
output stability. Formatting choices such as indentation and mapping-key order
belong to the encoder or consumer; semantic sequence order remains meaningful.
Promptkit need not provide CLI layouts or a generic reflection framework.
## Expanded Prompt Inspection ## Expanded Prompt Inspection
Expose description, ordered message templates with roles and cache-control Expose description, ordered message templates with roles and cache-control
@@ -158,9 +200,12 @@ Remove the existing special treatment of null in these profile fields:
Retain explicit values, including zero, false, and the existing empty-string Retain explicit values, including zero, false, and the existing empty-string
clearing behavior for `endpoint` and `api_key_env`. Retain the ordinary map clearing behavior for `endpoint` and `api_key_env`. Retain the ordinary map
overlay behavior: `extra_params: {}` does not clear backend defaults, and overlay behavior: `extra_params: {}` does not clear backend defaults, and
`extra_params: ""` is not a valid mapping. Whole-map suppression of backend `extra_params: ""` is not a valid mapping. Profile-level suppression of the
parameters is deliberately removed; do not replace it with another clearing whole backend provider-parameter map via null is deliberately removed; do not
option without a concrete requirement. replace it with another clearing option without a concrete requirement.
Operators remain free to remove or replace parameters in backend definitions
through ordinary ConfigKit composition. This does not prohibit those workflows
or change ConfigKit's null semantics.
Null entries within arbitrary provider maps and arrays remain literal provider Null entries within arbitrary provider maps and arrays remain literal provider
data. This is not recursive null removal. Required-field checks and fields that data. This is not recursive null removal. Required-field checks and fields that
@@ -212,17 +257,22 @@ without combining their settings into the inspected resource. Stored references
retain their exact meaning; caller namespace defaults do not rewrite them. retain their exact meaning; caller namespace defaults do not rewrite them.
A particular connected layout is a consumer choice, not a new Promptkit API. A particular connected layout is a consumer choice, not a new Promptkit API.
Coordinate CommandKit adoption of the changed profile result and richer prompt Coordinate CommandKit adoption of generically rendered resource results,
and backend views. In particular, migrate any formatter that currently expects replacing parallel report structures and per-field conversions where they are
`EffectiveModelParams`. The profile semantic change must be documented as such, no longer needed. In particular, migrate consumers of `EffectiveModelParams`.
not presented as merely adding more fields. Document the Go/serialization Also account for prompt inspection fields whose types need to change to retain
migration for the typed result selected in decision 4 and follow the existing presence; compatibility work is not limited to `ProfileInspection`. Document
release policy for compatibility changes. the Go and serialized-result migrations with the final types, and follow the
existing release policy. The profile semantic change must be documented as such,
not presented as merely adding more fields.
Document the post-handoff null change as a configuration compatibility change Document the post-handoff null change as a configuration compatibility change
as well. Existing null endpoint/credential-source clearing can use the existing as well. Existing null endpoint/credential-source clearing can use the existing
explicit empty-string form when that behavior is desired. There is no new explicit empty-string form when that behavior is desired. There is no new
replacement for whole-map suppression of backend provider parameters. Update replacement for profile-level whole-map suppression of backend parameters.
Include a migration example showing that a profile's null now permits backend
endpoint, credential-source, or provider-parameter fallback that it previously
suppressed; this changes execution as well as inspection. Update
the [format reference](../formats.md#defaults-and-overrides) when implementation the [format reference](../formats.md#defaults-and-overrides) when implementation
lands; it describes the current behavior until then. lands; it describes the current behavior until then.
@@ -294,9 +344,9 @@ source round-tripping are not selected deliverables.
## Decisions ## Decisions
All five decisions below are resolved, including the post-handoff null All five decisions below are resolved, including generic rendering and the
simplification in decision 4. Numbering is retained for continuity with the post-handoff null simplification in decision 4. Numbering is retained for
discussion; implementation is pending. continuity with the discussion; implementation is pending.
### 1. How Should An Unknown Backend Be Classified? ### 1. How Should An Unknown Backend Be Classified?
@@ -367,6 +417,8 @@ is unnecessary; raw mappings continue to serve those cases.
suitable public nested values. Change `ProfileInspection` deliberately to report suitable public nested values. Change `ProfileInspection` deliberately to report
the resolved profile rather than retaining a second execution-settings view for the resolved profile rather than retaining a second execution-settings view for
compatibility. Apply the same resource semantics to prompt/backend additions. compatibility. Apply the same resource semantics to prompt/backend additions.
Require generic rendering under the contract above; typed access must not force
consumers to maintain parallel report types or interpret every setting.
Represent absent settings and explicit values, including zero, false, and empty Represent absent settings and explicit values, including zero, false, and empty
strings where valid. Treat nullable resource fields set to null as absent in strings where valid. Treat nullable resource fields set to null as absent in
@@ -381,7 +433,7 @@ represent the uncompleted profile and must not substitute for it.
consumers interpret a generic configuration mapping. Endpoint and credential consumers interpret a generic configuration mapping. Endpoint and credential
overrides already take precedence without first clearing a value, and explicit overrides already take precedence without first clearing a value, and explicit
empty strings already support clearing those settings. No concrete requirement empty strings already support clearing those settings. No concrete requirement
justifies retaining whole-map suppression of backend provider parameters. justifies retaining profile-level whole-map suppression of backend parameters.
Removing these exceptions simplifies interpretation and avoids carrying source Removing these exceptions simplifies interpretation and avoids carrying source
syntax and inheritance history into every field. syntax and inheritance history into every field.
@@ -391,10 +443,10 @@ typed discoverability and introduces a broader mapping-value contract. It is not
selected as the primary resource-inspection result; `InspectConfiguration` selected as the primary resource-inspection result; `InspectConfiguration`
remains available for explicit configuration/provenance inspection. remains available for explicit configuration/provenance inspection.
Document migration for existing `EffectiveModelParams` consumers and any Document migration for existing profile and prompt inspection consumers with
serialization changes with the final Go types. CommandKit owns report formatting; the final Go types and their serialization semantics. Generic JSON/YAML encoding
new JSON contracts or custom serializers are not automatically required. Do not is selected; bespoke consumer reports and custom serializers are not inherently
leak internal package types or mutable captured state. required. Do not leak internal package types or mutable captured state.
### 5. Backend Diagnostics Show Configured Values, Not Derived Destinations ### 5. Backend Diagnostics Show Configured Values, Not Derived Destinations
@@ -409,25 +461,34 @@ in resource inspection.
## Implementation And Verification ## Implementation And Verification
1. Define the typed results and migration selected in decision 4. Source-input 1. Define typed results, serialization names/nesting, and consumer migration
helper types and inspection results have distinct purposes; select their selected in decision 4. Account for presence across prompt, profile, and
shapes accordingly, preserving explicit values without a null presence state. backend results. Source-input helpers, inspection results, and execution
2. Simplify Promptkit's post-handoff handling of nullable fields and remove values have distinct purposes; reuse types only where their semantics fit.
presence machinery made unnecessary by that change. Retain presence needed 2. Implement and verify the post-handoff null change as a focused unit, separate
to distinguish omission from explicit zero, false, or empty values. Expose from inspection projection. Remove presence machinery made unnecessary by
expanded prompt content and resolved profile fields through existing that change while retaining omission versus explicit-value information.
capture/lookup paths, before execution completion, without another Verify the changed execution behavior and document its migration example.
composition pass or changes to ConfigKit's retained result. 3. Expose expanded prompt content and resolved profile fields through existing
3. Add backend inspection and the accepted public missing-backend error, capture/lookup paths, before execution completion, without another composition
pass or changes to ConfigKit's retained result. Support generic encoding with
the selected omission rules and preserve explicit values.
4. Add backend inspection and the accepted public missing-backend error,
returning configured diagnostic metadata without endpoint derivation. returning configured diagnostic metadata without endpoint derivation.
4. Add typed backend inheritance and the prompt source/helper types, delegating 5. Add typed backend inheritance and the prompt source/helper types, delegating
both to the ordinary ConfigKit mapping and loading path. both to the ordinary ConfigKit mapping and loading path.
5. Update GoDoc, formats, consumer/internal guidance, and release notes for the 6. Update GoDoc, formats, consumer/internal guidance, and release notes for the
selected behavior, profile-result migration, and null compatibility change. selected behavior, inspection-result migrations, and null compatibility
Coordinate CommandKit reporting. change. Coordinate CommandKit's generic reporting adoption.
Use focused offline tests for actual boundary risks: Use focused offline tests for actual boundary risks:
- Directly encode each inspection result with ordinary JSON and YAML encoders,
without consumer-specific report structures. Verify semantic field names,
nesting, omitted settings, explicit zero/false/empty values, and nested
provider data. Include prompt input/output and backend capacity presence,
rather than testing only profile fields. Assert semantic output, not incidental
whitespace or mapping-key order.
- Child resources show the resolved inherited fields and overrides, not an - Child resources show the resolved inherited fields and overrides, not an
unresolved `extends` directive; explicit zero survives. Omitted/null nullable unresolved `extends` directive; explicit zero survives. Omitted/null nullable
settings use the same absent representation; provider-data nulls retain their settings use the same absent representation; provider-data nulls retain their
@@ -452,13 +513,15 @@ Use focused offline tests for actual boundary risks:
relative content-file case using ordinary source context. Reuse generic loading, relative content-file case using ordinary source context. Reuse generic loading,
validation, and inheritance coverage rather than duplicating it for each helper. validation, and inheritance coverage rather than duplicating it for each helper.
CommandKit owns presentation tests for the migrated resource view. Follow the Promptkit owns the inspection serialization checks; CommandKit owns presentation
and delivery tests for the migrated resource view. Follow the
[testing policy](../policy/testing.md), combine cases where useful, and reuse [testing policy](../policy/testing.md), combine cases where useful, and reuse
existing coverage. No new benchmark threshold or release gate is selected. existing coverage. No new benchmark threshold or release gate is selected.
Completion means operators can inspect the resolved selected prompt, profile, Completion means operators can inspect the resolved selected prompt, profile,
or backend accurately without assembling execution settings or navigating raw or backend accurately through generic rendering, without per-field consumer
configuration paths. All three resource kinds have typed source helpers, adapters, assembling execution settings, or navigating raw configuration paths.
including inheritance support, and consumers have migration guidance for changed All three resource kinds have typed source helpers, including inheritance
profile inspection and post-handoff null semantics. support, and consumers have migration guidance for changed inspection results
and post-handoff null semantics.
No engine-policy or prepared-request additions are part of completion. No engine-policy or prepared-request additions are part of completion.