promptkit/docs/roadmap/inspect-backend.md

Operator-Facing Resource Inspection And Configuration API Roadmap

Purpose And Status

Help operators using CommandKit-based applications understand Promptkit's captured prompt, profile, and backend configuration and the settings applicable to a concrete request. CommandKit owns operator-facing commands and presentation; Promptkit supplies facts from its existing configuration and execution paths.

The direction is accepted for planning; implementation has not begun. Selected work includes expanded prompt inspection, presence-aware profile/prepared inspection, backend inspection with effective diagnostic destinations, and BackendConfig.Extends. Open decisions below cover unresolved scope and public representation choices. They are not implicit implementation requirements.

Design Balance

Provide useful information already captured or cheaply derived. Prefer exposing existing values and small shared helpers over introducing new resolution paths, parallel report models, provenance tracking, or retained snapshots solely for inspection. This is a design preference, not a prohibition on a small adapter or additional field when it serves a demonstrated consumer need.

“Fully resolved” describes the applicable resolution stage. It is not a promise that inspection can reconstruct the entire authored source, report every future configuration field automatically, predict provider defaults, or reproduce a future invocation independently of its request and external state.

View	Information available
Resource inspection	Captured values after composition, inheritance, normalization, and applicable Promptkit defaults.
Concrete request preparation	Selected profile, request overrides, rendered messages, input metadata, session ID, and loaded schema for supplied request data.
Execution	Provider responses, actual usage, and other observed runtime facts.

The planned fields below define the initial additions. Additional cheap, useful fields can be considered during implementation without building a uniform resource framework or promising exhaustive future coverage. If a field requires substantial new machinery, record the tradeoff before expanding scope.

Review Findings And Existing Boundaries

The current implementation provides a relatively small path to the core work:

• Captured prompt definitions already retain description, session template, message text, roles, and cache-control metadata alongside compiled templates. Inspection can expose that data without reconstructing source files or ASTs.
• Internal prepared results already retain TargetPresence; the root API has an ExecutionTargetPresence type. Presence-aware output need not introduce a second interpretation of configuration or provider-value conversion.
• Smoke selection already has endpoint derivation, including the OpenRouter key URL helper. Reusing or factoring that logic need not create an adapter capability registry or provider metadata service.
• Transport timeout and retry policy live in different components and are not all carried by prepared results. Their reporting scope and cost need a decision; they are no longer an unconditional completion requirement.

These observations are implementation evidence, not promises to retain those private types or helper arrangements. See the current source/capture, runner, and model-client descriptions.

Retain established public ownership, credential handling, captured-state, and namespace behavior unless a change is deliberately selected. Independent mutable results prevent callers from changing engine state; they do not require fresh copies of immutable strings or repeated deep copies between private layers. Existing Inspect* APIs avoid provider work and request preparation. Extending their returned data should not silently change those behaviors. These boundaries follow the existing API and the selected operator use case, rather than a new requirement that all future inspection APIs have identical behavior.

Reporting Semantics

Selected presentation: Omit unset optional settings from operator reports. Preserve explicit values, including zero, and applicable Promptkit defaults. Suppressed provider controls are also absent from an effective-settings report. Neither zero nor unspecified should stand in for an omitted control.

This concerns the effective setting, not an explanation of every reason it was omitted. ConfigKit inspection remains useful for authored values and history. Promptkit supplies sufficient presence information; CommandKit chooses report layout. Representation and serialization choices remain open in decision 4.

Do not infer provider defaults from absent controls. Likewise, a configured model alias is the alias, not a prediction of the model it will select later. Credential source names and requirements are useful inspection facts; resolved credential values remain excluded under existing contracts. Templates and arbitrary provider parameters are application data, not automatically safe for disclosure. No new general-purpose sanitization guarantee is selected.

Expanded Prompt Inspection

Expand PromptInspection to include description, ordered message templates with roles and cache-control metadata, and the session-ID template. Retain the existing identity, opaque hash, default profile reference, input declarations, and output contract, including schema reference and repair settings.

Expose effective content captured from imported text or Markdown files, as well as inline content. Return the template text available in the captured definition; source-layout reconstruction, original YAML formatting, and parsed-template objects are unnecessary for this use case. There is no new byte-for-byte source round-trip requirement. Message order and text should reflect what rendering consumes. Inspection continues to use captured state without rendering or reopening files; the schema stays a reference until ordinary preparation loads it.

Prompt inspection remains useful without a supplied request or a resolvable default profile. The existing independent lookup permits CommandKit to show a prompt even when related execution settings need attention.

Presence-Aware Profile And Prepared Inspection

ProfileInspection exposes effective settings without the presence information that execution uses for explicit-zero controls. PreparedRun has the same public reporting gap, including values returned by PreparedExecution.Details. Expose enough existing resolved presence information to distinguish omission from explicit settings in each of those results.

For example, an unset temperature produces no temperature entry in the report; an explicit zero produces temperature: 0. A concrete request's override appears in its prepared view. Backend completion, inheritance, and request precedence remain the execution path's responsibility. Inspectors need not reconstruct that history or distinguish all historical reasons for an absent setting.

Reuse the captured target and presence semantics, including existing handling of nonzero values. Decision 4 weighs exposing the current presence type against another public representation; it does not reopen the omission-reporting decision. See defaults and overrides.

Backend Inspection

Proposed entry point:

func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)

The result should expose the full backend identity, base endpoint, credential environment-variable name, provider defaults, effective concurrency/queue settings, and diagnostic settings and destinations. These are backend defaults, not settings after a particular profile or request overrides them. Capacity values describe policy, not current occupancy.

A dedicated top-level inspection result is recommended because BackendConfig is a presence-aware source input. This does not require a new type for every nested value: reuse suitable public values where their semantics match. Exact field names and nesting can be selected during implementation within the public representation decisions below; internal types remain behind the facade.

Follow existing namespace-view selection: qualify slashless caller IDs with the backend scope, pass qualified IDs through, and report the selected full ID. Use the same identity rules for bundled and caller-supplied backends. The new operation should follow existing inspection conventions for invalid engines, invalid selections, and context cancellation, without adding cross-error precedence guarantees. Missing-backend classification is open decision 1.

Effective Diagnostic Destinations

Include effective reachability and applicable authentication destinations, using explicit overrides or existing backend-relative defaults. Share the existing derivation logic where practical; neither CommandKit nor a new inspection-specific resolver should maintain a competing set of provider rules. A small helper extraction is sufficient if that is all the current adapters need.

Default reachability applies even without authored smoke_tests metadata. Report the configured authentication kind and optional inference-default profile reference without fabricating authored settings. An authentication kind of none has no active authentication destination. Decision 5 covers whether the result also needs separate declared endpoint fields.

Destinations describe captured backend settings, not provider reachability or HTTP diagnostic support in an injected client. Retain the inference-default profile as a reference; backend inspection need not resolve it or establish inference readiness. Ordinary inspection is expected to return a result or an error, following existing methods. If derivation reveals a meaningful case where useful captured settings would be lost, bring that case back for a decision rather than automatically adding partial-result and warning machinery.

Connected Views And Concrete Requests

CommandKit can follow prompt default-profile and selected-backend references through the existing inspections to build a connected view. This is a suggested consumer workflow, not a required report format or a restriction on future convenience APIs. Use stored references exactly; namespace defaults apply to caller selections, not to reinterpreting captured references. Endpoint-only profiles need no backend node. A consumer can retain successful lookups while showing a failure for a related resource.

Use existing Prepare for concrete variables, inputs, profile selection, and request overrides. It performs artifact/schema reads, rendering, and applicable credential checks without inference. PrepareExecution and its details serve callers that want later execution tied to the same preparation, subject to that API's existing contract. This roadmap adds no stronger replay guarantee and selects no second preparation workflow.

Model parameters are not all execution policy. Per-generation timeout, transport cap, retry settings, backend capacity, and caller deadlines are different facts. When reported, keep their meanings distinct. Decision 2 determines whether and how this round should expose additional engine-policy information. Report known settings without claiming knowledge of an injected client's internal behavior.

CommandKit owns CLI commands, formatting, and disclosure. Coordinate downstream adoption when the Promptkit additions land; this roadmap does not prescribe CommandKit's internal adapter design or require a new stable CLI serialization contract in Promptkit.

Typed Backend Inheritance

Add Extends *string to BackendConfig, matching ProfileConfig.Extends. BackendSource should encode a non-nil value as extends: {id: value} and omit the field when nil. ConfigKit owns inheritance resolution and its diagnostics. The helper exposes existing sibling inheritance, rather than performing parent lookup or filling inherited fields itself.

Keep existing root/collection identity, replacement, and null semantics. Raw mappings remain available for explicit null and other generic controls outside the helper's convenience surface. See the inheritance reference and profile helper.

Related Work And Scope

Typed prompt construction is open decision 3. Listing return types remain a possible later cleanup: ListPrompts returns an ID-only PromptIdentity wrapper, while backend/profile listings return strings. Standardizing on strings may be useful at a breaking-change window, but is not required for these inspections.

Collections, schemas, and engine options have different roles; symmetry does not require identical APIs for them. The deferred fingerprint proposal remains separate. Live capacity monitoring, provider health guarantees, and source round-tripping are not selected deliverables, rather than permanent prohibitions on future work.

Open Decisions

1. How Should An Unknown Backend Be Classified?

Recommendation: Add public ErrBackendNotFound, analogous to the existing prompt/profile sentinels. The registry already identifies missing backends; public mapping lets consumers distinguish a missing ID without parsing text. Keep existing operations' classifications unless a change is explicitly chosen.

Alternative: Use ErrInvalidRequest, avoiding another exported sentinel but combining malformed selection and a well-formed missing identity. Consumers that need the distinction would require inventory lookup or other error detail.

2. How Much Engine-Policy Reporting Belongs In This Round?

The previous draft made a comprehensive execution-policy section a completion requirement before assessing the cost of carrying transport and retry facts through preparation. That commitment is now open, including the option to defer.

Recommendation: Start with facts already held by the relevant owner and add small fields to an existing prepared view where they are inexpensive to carry accurately. Evaluate transport cap and effective retry settings individually. Avoid a normalized policy model covering every client, invocation, and deadline. If a useful fact requires substantial new plumbing, explicitly defer that fact and document the report's boundary instead of inventing a default or guarantee. The field set and location need agreement before implementation.

Alternative: Add a dedicated engine-settings view for captured defaults, paired with prepared results for request-specific settings. This is useful to operators without a concrete request, but adds an API and requires distinguishing defaults from overrides. A third option is to defer engine-policy reporting entirely for this round, delivering the resource/presence improvements first; that has the lowest cost but leaves transport/retry reporting to later work.

3. Should Typed Prompt Construction Be Added In This Round?

Recommendation: Keep PromptConfig/PromptSource separate until a programmatic authoring need justifies the public input model. Files and ConfigKit mappings already support prompts; this does not block operator inspection.

Alternative: Add the typed helper for discoverability and compiler assistance. It would need useful message/input/output and omission semantics, inheritance, and source context for relative content files. Its maintenance cost is larger than adding backend Extends; composition can still delegate to ConfigKit.

4. How Should Presence And Public Result Values Be Represented?

The operator requirement is accurate omission, not a particular Go structure, new parallel report schema, or fixed JSON policy.

Recommendation: Reuse existing public nested values and expose the existing ExecutionTargetPresence semantics alongside effective profile/prepared targets where practical. This takes advantage of data already used by execution and limits new translation. CommandKit can omit absent fields in its own output. Document how values and presence interact rather than introducing an independent presence/default/provenance system. Prefer additive changes when straightforward.

Alternative: Use optional-valued fields in an inspection-specific settings result, making absence directly expressible to consumers. That can simplify reporting, but duplicates an execution-settings surface and needs conversion and maintenance as fields evolve. Choose it only if the consumer simplification outweighs that cost. Ordinary JSON tags or a deliberate serialization change can also be considered; neither mandatory JSON stability for new results nor a ban on useful serialization is selected. Existing documented JSON contracts still need an explicit compatibility decision before changing.

5. Are Both Declared And Effective Diagnostic Endpoints Needed?

Effective destinations are selected scope. The previous draft also suggested keeping declared overrides separately, which could create two representations and unnecessary origin metadata without a demonstrated reporting need.

Recommendation: Report the effective destination, authentication kind, and default profile reference. Use InspectConfiguration for authored values and provenance. Reuse existing optional values where helpful; no duplicate declared endpoint or derived-status flag is required merely for symmetry.

Alternative: Include declared overrides alongside effective destinations when operators need that distinction in one backend view. Existing metadata may make this cheap, but the result then needs clear field meanings and consumer handling for both. Prefer directly available fields over a new provenance subsystem.

Implementation And Verification

1. Resolve open public-contract choices relevant to selected work. Agree on any engine-policy additions separately so an unresolved optional expansion does not silently become a prerequisite for the resource APIs.
2. Expand prompt inspection from captured data and expose existing presence in profile/prepared results. Reuse or modestly extend current lookup/projection paths; choose private interface layouts during implementation.
3. Add namespace-aware backend inspection and effective destination derivation, following the selected public error and result representation decisions.
4. Add typed backend inheritance. Implement any separately selected engine-policy reporting using the owning components' effective values.
5. Update GoDoc, consumer/internal guidance, and release notes with implemented behavior and any deliberate compatibility changes. Coordinate CommandKit's richer reports and omission handling without prescribing a new CLI framework.

Select focused offline tests for actual behavior and boundary risks:

• Newly exposed prompt fields reflect captured inline/imported content and order, without introducing rendering or related-resource dependencies.
• Profile and prepared output preserve explicit zero and distinguish controls that execution omits, including relevant inheritance and override cases.
• Backend results cover effective defaults/destinations, namespace selection, and applicable error behavior. Use cases with missing optional credentials or unresolved diagnostic profile references protect independent inspection.
• Returned mutable values cannot alter retained state. Reuse existing ownership tests and copy helpers rather than prescribing extra snapshots or allocations.
• Typed backend inheritance works through real ConfigKit composition; its generic composition rules do not need another full test matrix.
• Any selected policy-reporting fields reflect their owners' effective settings.

CommandKit can test its own output omission and explicit-zero presentation. Follow the testing policy; combine cases where useful, reuse existing coverage, and do not turn this list into a test-per-field mandate. No new benchmark threshold or release gate follows from this roadmap.

Completion means the selected fields give operators a materially fuller and accurate resource view, omission-aware settings are available to report builders, and existing preparation remains usable for a concrete request. Backend source inheritance is exposed. The resolved decisions define additional policy fields, public representations, and compatibility changes; exhaustive execution-policy coverage, typed prompt construction, listing changes, and speculative future adapters are not implicit completion conditions.