promptkit/docs/roadmap/inspect-backend.md

Operator-Facing Resource Inspection And Configuration API Roadmap

Purpose And Status

Help operators using CommandKit-based applications inspect the configured prompt, profile, or backend selected by ID. CommandKit owns commands and report presentation; Promptkit exposes the selected resource's resolved configuration.

The direction is accepted for planning; implementation has not begun. Selected work includes fuller prompt inspection, profile inspection of the resolved profile itself, a new InspectBackend, BackendConfig.Extends, typed prompt construction through PromptConfig/PromptSource, and simplification of Promptkit's post-handoff null handling. Execution-policy reporting and prepared-request enhancements have been removed from this roadmap. The decisions below record the accepted scope and representation approach.

Resource-Inspection Boundary

“Fully resolved” means that ConfigKit has applied source composition and inheritance to the selected resource. A child profile should show its resulting keys and values, not an unresolved extends directive. It does not mean that inspection assembles everything affecting execution.

API	Resource shown
InspectPrompt	Resolved prompt definition, including captured imported message content.
InspectProfile	Resolved profile definition, before backend-to-profile execution completion.
Proposed InspectBackend	Resolved backend definition, including configured diagnostic metadata.

For example, given:

profiles:
  - id: base
    backend: openrouter
    model: example-model
    temperature: 0.5
  - id: precise
    extends: {id: base}
    temperature: 0

The resource settings shown for precise are:

id: precise
backend: openrouter
model: example-model
temperature: 0

ConfigKit inheritance contributes the parent's model and backend reference. Promptkit's execution-time completion would obtain settings from that backend, but those settings do not become fields of the inspected profile. The operator can inspect the referenced backend separately. Likewise, prompt inspection reports a default profile reference without incorporating that profile's values.

Use the captured, resolved configuration, including normalizations/defaults already recorded in that configuration. Do not add framework execution defaults, request overrides, credential values, or provider-derived defaults to fill absent resource fields. For example, an absent profile timeout is not populated from the framework generation timeout, while a queue default already recorded in the resolved backend remains visible. See the format reference.

Design Balance And Reporting

Provide useful resource information already captured or available with modest projection. Reuse ConfigKit's resolved handoff and existing resource capture; avoid another composition pass, execution resolver, parallel provenance system, or duplicate retained graphs solely for inspection. These are proportionality guidelines, not prohibitions on a small helper or type when it serves the API.

Selected presentation: Omit absent optional settings. Preserve explicit values, including zero. Absence is not displayed as zero or unspecified. Treat null like omission for nullable resource fields after ConfigKit resolution, under the selected post-handoff semantics below. Literal nulls inside provider data retain their meaning; they are not optional resource-field presence states. The previous draft's rule to omit suppressed outbound controls belonged to an execution report and does not define resource inspection. Accepted decision 4 selects typed results without a third state for nullable resource fields.

Inspection reports effective resource values rather than original YAML layout, comments, or the history of each inherited field. InspectConfiguration remains available for provenance. Existing opaque prompt identity/hash information can remain alongside the resource view; it does not turn inspection into a source round-trip API.

Retain existing captured-state, namespace, ownership, and credential-handling contracts unless a change is deliberately selected. Mutable results should not let a consumer change retained engine state; that does not require repeated copies between private helpers or copying immutable strings. Ordinary resource inspection needs no rendering, provider calls, or credential availability checks. Templates and arbitrary provider settings remain application data whose presentation and disclosure are controlled by the consuming application.

Expanded Prompt Inspection

Expose description, ordered message templates with roles and cache-control metadata, and the session-ID template, alongside the existing identity, hash, default profile reference, input declarations, and output contract. Include schema references and repair settings as part of the prompt definition.

Captured prompt definitions already retain the effective content of inline and imported text/Markdown messages. Return that text without rendering or reopening files. Original source layout and parsed-template objects are unnecessary. The schema remains a reference; inspecting the prompt does not load it.

Keep prompt inspection independent of request readiness and default-profile lookup. A captured prompt remains inspectable without a runnable default profile.

Resolved Profile Inspection

This is a deliberate change to existing InspectProfile behavior. It currently returns backend-completed EffectiveModelParams; the new resource boundary requires the resolved profile's own fields instead.

Report fields supplied by the profile or inherited through ConfigKit, including backend reference, endpoint/model where present, generation controls, credential metadata, and provider parameters. Do not merge in backend provider defaults or resolve an environment variable. Profiles may be partial, and their references remain references. Distinguish omitted settings from explicit values, including zero, using the resolved resource's information rather than inferring presence from an execution target. Null uses the same absent representation as omission.

Execution target presence and prepared metadata describe a later resolution stage and are not substitutes for profile configuration presence. Use the existing configuration capture path to expose the needed information without introducing a second configuration interpreter. Follow accepted decision 4 for typed results and migration from EffectiveModelParams.

This roadmap adds no preparation API surface. Execution continues to apply backend completion, framework defaults, and request overrides through its existing path, with the deliberate configuration behavior change below.

Post-Handoff Null Semantics

Selected behavior: For Promptkit's interpretation of nullable resource fields, null and omission mean the same thing. Apply this consistently to resource inspection and execution, rather than hiding a runtime distinction only in inspection results.

ConfigKit owns everything before the handoff of its resolved configuration, including source composition, inheritance, and null's effects during those operations. This work makes no upstream change or request, removes no nulls from input sources, and does not reinterpret or repeat ConfigKit's composition. The retained ConfigKit result and its provenance remain available through InspectConfiguration; resource projection does not rewrite that tree.

Remove the existing special treatment of null in these profile fields:

Field	Selected meaning of either omission or null after handoff
endpoint	Permit the selected backend's endpoint fallback.
api_key_env	Permit the ordinary backend credential-source fallback, subject to the existing api_key_required rule.
Whole-map extra_params	Permit the backend's configured provider parameters.

Retain explicit values, including zero, false, and the existing empty-string clearing behavior for endpoint and api_key_env. Retain the ordinary map overlay behavior: extra_params: {} does not clear backend defaults, and extra_params: "" is not a valid mapping. Whole-map suppression of backend parameters is deliberately removed; do not replace it with another clearing option without a concrete requirement.

Null entries within arbitrary provider maps and arrays remain literal provider data. This is not recursive null removal. Required-field checks and fields that currently reject null remain governed by their existing validation rules; this decision does not broaden accepted configuration shapes.

For example, ConfigKit may resolve a child's null over a parent's endpoint. Promptkit then treats the resulting unset profile endpoint like omission and uses the selected backend's endpoint, if available. The parent's endpoint is not restored or recomputed by Promptkit.

Backend Inspection

Proposed entry point:

func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)

Expose the full backend identity and its resolved fields: base endpoint, credential environment-variable name, provider defaults, concurrency/queue settings, and configured smoke_tests metadata. Reuse suitable public nested values where their semantics match; exact names and nesting follow decision 4.

Diagnostic metadata includes the configured authentication kind, profile reference, and endpoint overrides after ConfigKit composition/inheritance. An inherited endpoint override is a resolved configured value. An endpoint computed later by a smoke-test adapter is not. Do not synthesize /key URLs, reachability fallbacks, or a smoke_tests block that is absent from the resolved backend. No shared diagnostic URL derivation work is required for this feature.

Inspecting a backend does not resolve its smoke-test default profile or contact an endpoint. Report the configured reference even if that profile cannot be selected. Capacity values describe configured policy, not current occupancy.

Follow the existing namespace-view selection rules: qualify slashless caller IDs with the backend scope, pass qualified IDs through, and return the selected full ID. Bundled and caller-defined resources use the same lookup rules. Follow existing inspection conventions for invalid engines, invalid selections, and cancellation. Missing-backend classification follows accepted decision 1.

Consumer Integration And Compatibility

CommandKit can offer navigation between prompt, profile, and backend references without combining their settings into the inspected resource. Stored references retain their exact meaning; caller namespace defaults do not rewrite them. A particular connected layout is a consumer choice, not a new Promptkit API.

Coordinate CommandKit adoption of the changed profile result and richer prompt and backend views. In particular, migrate any formatter that currently expects EffectiveModelParams. The profile semantic change must be documented as such, not presented as merely adding more fields. Document the Go/serialization migration for the typed result selected in decision 4 and follow the existing release policy for compatibility changes.

Document the post-handoff null change as a configuration compatibility change as well. Existing null endpoint/credential-source clearing can use the existing explicit empty-string form when that behavior is desired. There is no new replacement for whole-map suppression of backend provider parameters. Update the format reference when implementation lands; it describes the current behavior until then.

Existing preparation APIs remain available for separate concrete-request uses; this roadmap adds neither preview commands nor execution-policy report fields. Transport timeouts, engine retry settings, caller deadlines, and injected-client policy are outside the inspected resource unless they are actual fields of its resolved definition. This is the scope of these APIs, not a pending engine-policy feature decision.

Typed Backend Inheritance

Add Extends *string to BackendConfig, matching ProfileConfig.Extends. BackendSource encodes a non-nil value as extends: {id: value} and omits it when nil. ConfigKit owns sibling inheritance and its diagnostics. The source helper does not look up parents or populate inherited fields itself.

Keep existing root/collection identity and ConfigKit composition semantics, including its handling of null. Raw mappings remain available for generic controls outside the convenience helper. See resource inheritance and the existing profile helper.

Typed Prompt Sources

Add PromptConfig and PromptSource to complete the typed source-helper set alongside BackendConfig/BackendSource and ProfileConfig/ProfileSource. The proposed helper follows their calling convention:

func PromptSource(label string, prompts []PromptConfig) configkit.Source

Cover prompt identity, sibling inheritance through Extends, default profile, description, session template, inputs, messages with inline or file content and cache controls, and output settings. Choose nested input types for configuration semantics; reuse existing types where suitable without forcing execution or inspection values into the source-input role.

Keep the helper a straightforward mapping constructor. Encode supplied fields under a prompts list, then delegate to configkit.Mapping. Optional nil fields mean omission; explicit values, including zero, false, and empty strings where supported, must remain representable. Preserve nil-versus-empty collections where that distinction matters to composition. Raw ConfigKit mappings remain the escape hatch for ConfigKit's explicit-null composition operations and other advanced controls outside the typed convenience surface. The post-handoff interpretation above does not change how source helpers supply values to ConfigKit.

The helper does not resolve inheritance, apply execution defaults, render messages, read content files, or independently validate a prompt. Existing ConfigKit composition and engine loading retain those responsibilities. Relative content references use the ordinary input/source resource-context mechanism; creating a typed source does not select a new filesystem root or file-loading workflow. Reuse ConfigKit's source snapshot behavior rather than adding another copier or recursive compatibility converter.

Keep all three source-helper families consistent in their scope and omission conventions. They need not reproduce every generic configuration operation or share a reflection-based encoding framework. Maintain the field mappings and focused behavioral tests as the configuration model evolves.

Related Work

Listing return-type consistency is a possible later cleanup, not required here. Collections, schemas, and engine options do not need identical APIs merely for symmetry. The deferred fingerprint proposal remains separate. Live capacity monitoring, provider health guarantees, and source round-tripping are not selected deliverables.

Decisions

All five decisions below are resolved, including the post-handoff null simplification in decision 4. Numbering is retained for continuity with the discussion; implementation is pending.

1. How Should An Unknown Backend Be Classified?

Status: Accepted; implementation pending.

Decision: Add public ErrBackendNotFound for an absent backend selected by InspectBackend, following the existing prompt/profile sentinel pattern. An ordinary missing-backend lookup should match ErrBackendNotFound through errors.Is and should not also match ErrInvalidRequest. Reuse the registry's existing internal missing-backend classification at the public boundary.

Invalid selections follow existing ID rules and match ErrInvalidRequest; invalid engines match ErrInvalidConfig. Preserve context cancellation through the existing inspection conventions. Invalid backend definitions remain an engine-construction concern. Exact error wording is not a new contract.

Initially apply the new public classification to InspectBackend; do not implicitly reclassify construction or smoke-test failures. CommandKit owns operator wording and exit-status selection.

Rationale: This gives consumers a useful distinction with a small addition consistent with prompt and profile inspection, without a new error hierarchy.

Alternatives considered: Using ErrInvalidRequest saves one public sentinel but combines malformed selections and missing identities. A typed generic resource-not-found error could carry resource kind and ID, but adds a broader contract and integration questions for existing sentinels. Neither is selected; the caller already knows the operation and requested ID.

2. Resource Inspection Does Not Include Execution-Policy Reporting

Status: Resolved; the former engine-policy proposal is removed from scope.

Decision: Inspect the selected resolved resource, not all settings that may affect its execution. ConfigKit composition and inheritance are included; backend-to-profile execution completion, framework execution defaults, request overrides, and engine transport/retry reporting are not.

Remove the proposed engine-settings API, prepared-result policy fields, and prepared-result presence enhancements from this roadmap. No decision about where to place those additions remains necessary for resource inspection.

3. Should Typed Prompt Construction Be Added In This Round?

Status: Accepted; implementation pending.

Decision: Add PromptConfig and PromptSource, including inheritance, following the thin mapping-constructor pattern described above. Retain the backend/profile helpers and raw ConfigKit mapping support.

Rationale: API symmetry and convenient programmatic construction have value as Promptkit approaches 1.0. The existing helpers are small explicit encoders; a prompt helper adds nested input values and mappings, not a new authoring, composition, or validation subsystem. That bounded maintenance cost is justified by consistent discoverability and compiler assistance across all three resources.

Alternatives considered: Deferring prompt helpers would leave asymmetric Go conveniences. Removing the existing typed helpers would simplify Promptkit's surface but move string-keyed mapping construction to consumers and require migration. Neither is selected. Full typed coverage of all ConfigKit operations is unnecessary; raw mappings continue to serve those cases.

4. How Should Resolved Resource Results And Presence Be Represented?

Status: Accepted; implementation pending.

Decision (Option A): Use resource-oriented typed inspection results, reusing suitable public nested values. Change ProfileInspection deliberately to report the resolved profile rather than retaining a second execution-settings view for compatibility. Apply the same resource semantics to prompt/backend additions.

Represent absent settings and explicit values, including zero, false, and empty strings where valid. Treat nullable resource fields set to null as absent in the result and adopt the post-handoff behavior described above. This supersedes the earlier proposal to preserve three profile-specific null distinctions. ConfigKit composition and literal nulls within provider data retain their existing ownership and meaning. No third resource-field presence state or generic resource framework is needed. Execution-target presence does not represent the uncompleted profile and must not substitute for it.

Rationale: Typed results provide discoverable Go fields without making consumers interpret a generic configuration mapping. Endpoint and credential overrides already take precedence without first clearing a value, and explicit empty strings already support clearing those settings. No concrete requirement justifies retaining whole-map suppression of backend provider parameters. Removing these exceptions simplifies interpretation and avoids carrying source syntax and inheritance history into every field.

Alternative considered: A caller-owned resource mapping naturally represents missing keys and null and can reduce per-field projection work, but sacrifices typed discoverability and introduces a broader mapping-value contract. It is not selected as the primary resource-inspection result; InspectConfiguration remains available for explicit configuration/provenance inspection.

Document migration for existing EffectiveModelParams consumers and any serialization changes with the final Go types. CommandKit owns report formatting; new JSON contracts or custom serializers are not automatically required. Do not leak internal package types or mutable captured state.

5. Backend Diagnostics Show Configured Values, Not Derived Destinations

Status: Resolved by the resource-inspection boundary.

Decision: Return diagnostic settings present in the resolved backend, including inherited endpoint overrides. Leave omitted endpoint settings absent; deriving a default destination belongs to diagnostic execution. The earlier proposal to expose both declared and computed destinations is removed. This avoids an additional representation and provider-specific derivation machinery in resource inspection.

Implementation And Verification

1. Define the typed results and migration selected in decision 4. Source-input helper types and inspection results have distinct purposes; select their shapes accordingly, preserving explicit values without a null presence state.
2. Simplify Promptkit's post-handoff handling of nullable fields and remove presence machinery made unnecessary by that change. Retain presence needed to distinguish omission from explicit zero, false, or empty values. Expose expanded prompt content and resolved profile fields through existing capture/lookup paths, before execution completion, without another composition pass or changes to ConfigKit's retained result.
3. Add backend inspection and the accepted public missing-backend error, returning configured diagnostic metadata without endpoint derivation.
4. Add typed backend inheritance and the prompt source/helper types, delegating both to the ordinary ConfigKit mapping and loading path.
5. Update GoDoc, formats, consumer/internal guidance, and release notes for the selected behavior, profile-result migration, and null compatibility change. Coordinate CommandKit reporting.

Use focused offline tests for actual boundary risks:

• Child resources show the resolved inherited fields and overrides, not an unresolved extends directive; explicit zero survives. Omitted/null nullable settings use the same absent representation; provider-data nulls retain their literal meaning.
• Null and omitted profile endpoint, credential-source, and provider-map fields permit the same backend fallback. Explicit empty-string clearing and empty-map overlay behavior remain intact. Include a ConfigKit inheritance case where null clears a parent value before Promptkit applies backend defaults, and verify that resource inspection does not rewrite the retained ConfigKit tree.
• Profile inspection excludes settings supplied only by a referenced backend, framework execution defaults, or a request. Existing execution still applies those settings through its ordinary resolution path.
• Prompt inspection exposes captured inline/imported content, metadata, and order without rendering or adding related-resource lookup dependencies.
• Backend inspection reports configured/inherited diagnostic overrides while leaving omitted overrides absent. It does not require diagnostic readiness.
• Resource selection, error classification, and mutable-result ownership follow the selected contract; reuse existing namespace and ownership coverage.
• Typed backend and prompt inheritance work through real ConfigKit composition. Prompt sources preserve nested message/input/output settings, omission and explicit values, ordering, and source snapshot behavior. Include a representative relative content-file case using ordinary source context. Reuse generic loading, validation, and inheritance coverage rather than duplicating it for each helper.

CommandKit owns presentation tests for the migrated resource view. Follow the testing policy, combine cases where useful, and reuse existing coverage. No new benchmark threshold or release gate is selected.

Completion means operators can inspect the resolved selected prompt, profile, or backend accurately without assembling execution settings or navigating raw configuration paths. All three resource kinds have typed source helpers, including inheritance support, and consumers have migration guidance for changed profile inspection and post-handoff null semantics. No engine-policy or prepared-request additions are part of completion.