promptkit/docs/roadmap/inspect-backend.md

Operator-Facing Resource Inspection And Configuration API Roadmap

Purpose And Status

Give operators using CommandKit-based applications a complete, accurate view of Promptkit's effective prompt, profile, and backend configuration. CommandKit is the primary consumer of these inspection APIs and owns operator-facing commands and formatting. Promptkit owns resolved typed facts and their semantics.

The selected direction is accepted for planning; implementation has not begun. Expand InspectPrompt, expose setting presence in profile and prepared results, and add InspectBackend with effective diagnostic destinations. Retain the planned BackendConfig.Extends convenience addition. Remaining decisions are listed below; typed prompt construction and listing return-type changes are lower-priority considerations, not prerequisites for operator inspection.

This use case supersedes the earlier recommendation to keep prompt inspection limited to interface metadata and backend diagnostics limited to declared settings. Operators need to see captured message content and the destinations that default diagnostic rules select. The implementation should reuse existing capture, resolution, and adapter logic rather than build a second resolver.

What Fully Resolved Means

View	What can be resolved
Resource inspection	Captured values after composition, inheritance, normalization, and applicable Promptkit defaults.
Concrete request preparation	Selected profile, request overrides, rendered messages, inputs, session ID, and loaded schema for supplied request data.
Execution	Provider responses, actual usage, and other observed runtime facts.

A prompt ID alone cannot determine future request overrides or template values. Resource inspection therefore shows effective templates and references, while preparation shows the concrete request. Neither should invent provider defaults, resolve a moving model alias to a hypothetical future model, or claim execution success from configuration alone.

Reporting decision: Omit unset optional settings from operator reports. Retain explicitly configured values, including zero, and effective Promptkit configuration defaults. Do not print omitted controls as zero or as unspecified. Suppressed provider controls should likewise be absent from an effective-settings report. This is a presentation of execution behavior; InspectConfiguration remains available when an operator wants the authored null, omission, or inheritance history.

Promptkit must expose sufficient typed presence information for CommandKit to make those choices correctly. A zero-valued Go field alone is not sufficient. This does not remove fields from existing public types or silently change their existing JSON representation; CommandKit owns its report projection.

Relationship To Existing APIs

API	Responsibility
ListBackends	Discover captured backend identities.
Proposed InspectBackend	Describe one backend's effective settings by ID.
Expanded InspectPrompt	Describe the captured prompt, including effective template content.
Enhanced InspectProfile	Report backend-completed settings with the presence information needed for accurate output.
Prepare / PreparedExecution.Details	Describe a concrete prepared request, including setting presence.
InspectConfiguration	Inspect effective configuration and source provenance.
SmokeTest	Execute an explicitly requested diagnostic and report evidence.

See the consumer guide for the implemented inspection and discovery boundaries, and the architecture policy for library responsibilities.

Complete Prompt Inspection

Expand PromptInspection to include description, ordered message templates with roles and cache-control metadata, and the session-ID template. Retain identity, opaque prompt hash, default profile reference, declared inputs, and the complete output contract, including schema reference and repair settings.

For messages imported from text or Markdown files, return the effective content captured at engine construction. Operators should not have to locate or reopen the original file to see what execution will render. Return templates as text, not internal parsed-template objects. Inspection neither renders templates nor rereads content files; edits require a new engine. The schema remains a reference here; concrete preparation loads the selected schema.

Keep prompt inspection independent of profile selection and request readiness. An absent or incomplete default profile must not hide an otherwise captured prompt. A prompt without a default profile remains inspectable. Return owned mutable results, preserving message and input order and request isolation.

Presence-Aware Profile And Prepared Inspection

ProfileInspection currently exposes effective model settings but not the ExecutionTargetPresence information that distinguishes omitted controls from explicit zero values. PreparedRun has the same reporting gap. Expose the existing presence semantics alongside the effective target in both results, including the PreparedRun returned through PreparedExecution.Details.

Use the resolved target and presence captured by the execution path; do not infer presence from numeric values or re-run ConfigKit composition. Preserve backend completion, profile inheritance, explicit suppression, and request precedence. Keep credential requirement metadata distinct from the credential source name; no credential value is returned or checked by resource inspection.

For example, the report for a profile that leaves temperature unset should have no temperature entry. A profile explicitly setting temperature to zero should show temperature: 0. A concrete request that overrides that profile should show its effective setting instead. Presence metadata is for accurate report construction; CommandKit need not expose an implementation-oriented bit map to operators.

Keep the meaning of effective defaults explicit. A configured zero generation timeout disables that particular deadline; it does not disable the transport cap or caller cancellation. An omitted provider control supplies no assertion about the provider's own default. See the defaults reference.

Proposed Backend Inspection Contract

func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)

Use a dedicated public result rather than returning BackendConfig. BackendConfig is an input type whose pointers express omitted and explicit values; inspection describes the normalized effective definition after composition and defaulting. Do not expose internal domain types.

The proposed result includes:

Information	Meaning
BackendID	Full selected runtime identity.
Endpoint	Captured backend base endpoint.
APIKeyEnv	Configured environment-variable name, never its resolved value.
ExtraParams	Backend provider defaults, before profile or request overrides.
ConcurrencyLimit	Configured generation concurrency limit.
QueueCapacity	Effective queue capacity, including construction-time defaults.
SmokeTests	Diagnostic settings and effective destinations, including the authentication kind and optional inference default profile reference.

Scalar fields describe effective values rather than reproducing source-field presence. Diagnostic optionality must remain expressible, including no configured authentication adapter or inference default. Exact nested type names are implementation choices. Follow existing typed inspection conventions without adding a stable JSON serialization contract for this feature. Applications can define their own output formats.

Selection And Errors

Follow the existing namespace selection rules: a scoped engine view qualifies slashless IDs with its backend namespace; qualified IDs pass through unchanged. An unscoped view looks up the supplied ID directly. Return the full selected ID. Do not add fallback search across namespaces, implicit backend selection, or special handling for bundled backends. Use the existing ID rules, including case sensitivity, rather than introducing new normalization.

A nil or zero-value engine should match ErrInvalidConfig; a blank or otherwise invalid selection should match ErrInvalidRequest. Cancellation should retain the context error, and failures should return no partial result. The public classification for a missing backend is open decision 1.

Effective Diagnostic Destinations

Include the effective reachability destination and, when an authentication adapter applies, its effective destination. Apply explicit endpoint overrides and existing backend-relative defaults through the same derivation logic used by diagnostic execution. CommandKit must not reconstruct provider-specific URL rules. Include the configured authentication kind and inference default profile reference; an inapplicable authentication kind has no active probe destination.

Default reachability can apply even without a smoke_tests block. The result must be able to report that effective destination without fabricating an authored block or an authentication configuration. Where distinguishing an explicit endpoint override from a derived destination is useful, keep those facts separate rather than overloading a field's meaning. Source history remains owned by InspectConfiguration.

These are destinations under the backend's captured settings, not guarantees about runtime overrides, the selected client's HTTP diagnostic support, or reachability. Report a default inference profile reference without resolving it; a missing or incomplete referenced profile must not make backend inspection fail.

Captured State And Ownership

Inspect the engine's captured backend definition without reloading sources, reading credential values, preparing a prompt, acquiring capacity, or making provider calls. Engine construction already validates effective definitions; inspection should not repeat that work or create an additional acceptance path.

Capacity fields describe configuration, not current occupancy or queue length. Diagnostic settings describe configuration, not provider availability, key acceptance, or inference readiness. In particular, report a smoke-test default profile reference without resolving or validating that profile merely to inspect the backend. Its selection-time behavior remains owned by SmokeTest.

Return independently owned mutable values, including nested provider maps and diagnostic settings. Mutating a result must not affect the engine or another inspection. Reuse existing ownership operations and avoid consecutive copies whose only purpose is to cross private helper boundaries.

Provider defaults should retain their values, consistent with typed target inspection. They are application data and may contain sensitive content; the result is not automatically safe for logging or disclosure. Do not resolve credentials or silently substitute ConfigKit's redacted reporting values for the typed settings. Operator provenance and redacted configuration reporting remain available through InspectConfiguration.

Connected Reports And Concrete Request Preview

CommandKit can assemble a connected report by following prompt default-profile and selected-backend identities through the three inspection APIs. Promptkit supplies the references and effective values; CommandKit chooses nested sections or links. Use captured references exactly, without requalifying them through a view's caller-selection namespace. Endpoint-only profiles need no backend node. A missing related resource can be reported while preserving successfully inspected resources; do not make each resource inspection recursively dependent on all related objects.

Use Prepare for an operator preview with concrete variables, inputs, profile selection, and request overrides. It performs artifact/schema reads, rendering, and applicable credential checks without inference. Do not describe it as a pure configuration lookup. Use PrepareExecution and its existing details when later execution must use the same captured preparation; a separate preview followed by an independent run is not a promise that mutable inputs stay equal. No second request-resolution API is selected by this roadmap.

An execution report must distinguish per-generation timeout, built-in transport cap, retry policy, and backend capacity. Effective model parameters alone are not the entire execution policy. Caller deadlines and injected-client behavior can add constraints that Promptkit cannot predict for a future invocation. How to expose the missing engine-policy facts is open decision 2. Do not infer them from model fields or ask CommandKit to duplicate Promptkit defaults.

CommandKit owns text/YAML presentation and report disclosure. Prompt templates, input content, and arbitrary provider parameters may be sensitive; preserve existing credential exclusions without claiming that every inspected value is safe to publish. No CommandKit implementation is included in this repository's roadmap; coordinate its consumer adaptation when the Promptkit APIs land.

Typed Backend Inheritance

Add Extends *string to BackendConfig, matching ProfileConfig.Extends. When non-nil, BackendSource should encode it as extends: {id: value} in the backend record. Nil omits the field. Preserve the supplied value and let ConfigKit resolve and validate inheritance through the existing source-loading path, just as it does for profiles.

The purpose is to let programmatic callers derive a backend variant without switching from BackendSource to a raw mapping. The helper must not resolve a parent, fill inherited fields, or become a second inheritance implementation. It remains a source constructor rather than an engine registry mutation API.

This exposes existing sibling inheritance, not a new configuration feature. Parent IDs select siblings in the same root or collection resource list; namespace views do not qualify these configuration references. Existing same-ID replacement, omitted-field, and explicit-null behavior remain owned by ConfigKit composition. Raw mappings remain available when callers need explicit null or other generic configuration controls that the typed helper cannot express. See the inheritance reference and the existing profile source helper.

Scope Of API Consistency

Backends, profiles, and prompts already share source layering, collection membership, namespace selection, sibling inheritance, and construction-time definition validation. The gaps addressed here are in public convenience and inspection APIs, not in those underlying configuration capabilities.

Inspection remains purpose-specific while becoming complete for operator use: prompt templates are captured but unrendered, profile settings include backend completion, and backend settings include effective diagnostic destinations. Provenance and authored representation stay with InspectConfiguration.

ListPrompts currently returns []PromptIdentity, whose only field is ID; ListProfiles and ListBackends return []string. At a suitable breaking-change window, prefer considering []string for all three over adding thin identity wrappers for backends and profiles. That would simplify consumer code, but requires a deliberate compatibility decision and migration instructions. Keep the existing return types in this work; listing consistency does not block the backend additions.

Collections, schemas, and engine configuration remain distinct concepts rather than additional peers requiring identical resource APIs. Prompt requirement inspection and backend capacity/diagnostic settings retain their specialized roles. Profile/backend hashes are not added for symmetry; the existing deferred fingerprint proposal owns that separate question.

Open Decisions

1. How Should An Unknown Backend Be Classified?

Recommendation: Add public ErrBackendNotFound for this explicit lookup, matching the role of ErrProfileNotFound and ErrPromptNotFound. The registry already has an internal backend-not-found sentinel. Map it at the facade so applications can distinguish a missing backend from malformed input without inspecting text or internal packages.

Limit this addition to the new inspection operation unless existing callers justify a broader change; do not automatically reclassify smoke-test or engine construction errors.

Alternative: Classify an absent ID as ErrInvalidRequest. This avoids another exported sentinel but combines a well-formed missing identity with an invalid selection. Consumers needing different presentation would then require an inventory lookup or additional error detail. A sentinel is the smaller consumer-facing solution when that distinction matters.

2. Where Should Effective Engine Execution Policy Be Exposed?

Resource inspection and PreparedRun do not currently provide a complete typed view of the built-in transport cap and retry policy. These can originate in Go engine options, not just the composed configuration inspected by ConfigKit.

Recommendation: Extend the existing prepared-result view with a small, separately named execution-policy section for facts applicable to that request. Report the effective built-in retry setting and transport cap, distinguishing injected-client ownership where Promptkit cannot supply those facts. Keep model settings, backend capacity, and caller context constraints distinct. Reuse the same policy selection as execution, including request overrides; do not add a second preview resolver or claim to know an injected client's internal policy. This makes the concrete preview useful without another top-level inspection API.

Alternative: Add a separate engine-settings inspection API for captured engine defaults, then let CommandKit combine that with resource inspection and request-specific preparation. This also serves operators who have no concrete request, but adds a public surface and requires a clear distinction between engine defaults and request overrides. Select the surface before implementation; neither option authorizes inspection of arbitrary client internals.

3. Should Typed Prompt Construction Be Added In This Round?

Recommendation: Keep PromptConfig and PromptSource as a separate discussion item until a programmatic prompt-authoring use case establishes their value. File sources and ConfigKit mappings already support prompt definitions. The absence of a helper is a Go convenience gap, not a missing configuration capability, and does not block the backend additions.

Alternative: Add a typed prompt source alongside the existing backend and profile helpers. It would improve discoverability, compiler assistance, and consistency for applications constructing prompts in Go. Its cost is a larger public input model covering messages, inline/file content, inputs, output contracts, session templates, and omission semantics. If selected, define that scope separately, including sibling inheritance and source context for relative content files, while retaining ConfigKit's ownership of composition and raw mapping support for explicit null. Do not implement a second prompt parser or validation path merely to provide the helper.

Implementation Scope And Verification

1. Resolve the error and engine-policy decisions and define public result fields and GoDoc. Typed prompt construction is a separate scope decision; it does not block the selected inspection improvements.
2. Expand prompt inspection from captured metadata and message content. Add presence information to profile and prepared results using the execution path's existing resolved state. Preserve existing values and error behavior.
3. Connect namespace-aware backend inspection to captured lookup and add shared diagnostic destination derivation. Keep the public facade responsible for owned results; avoid a new backend cache, loader, or recursive inspector.
4. Implement the selected engine-policy reporting surface without repeating policy resolution. Keep the concrete-request preview on existing preparation.
5. Add BackendConfig.Extends and encode it through BackendSource, leaving composition with ConfigKit.
6. Update public GoDoc, consumer guidance, relevant internal documentation, and release notes when implementation lands. Coordinate CommandKit changes for the fuller reports and omission-aware formatting, without modifying its CLI contract from this roadmap.

Use focused public tests for these consequential behaviors:

• Captured imported message content, description, session template, roles, cache controls, and output contract appear without rendering, rereading files, or requiring a resolvable default profile.
• Profile and prepared results identify omitted/suppressed controls as absent and preserve explicit-zero and nonzero controls across inheritance and request overrides. Details retain the same frozen settings and presence as their prepared execution; no new distinction between reasons for omission is needed.
• Backend inspection handles root/scoped/qualified identities, bundled and caller-defined records, capacity defaults, optional metadata, explicit and derived diagnostic destinations, errors, and cancellation. Missing credential environment values and missing smoke default profiles do not prevent lookup.
• Mutating returned templates, maps, diagnostic settings, or prepared details cannot change captured state or another result. Inspection performs no provider calls; preparation retains its established resource-I/O boundary.
• Engine-policy facts match the selected built-in or injected-client behavior and applicable request overrides under the chosen reporting surface.
• Typed backend inheritance through real ConfigKit composition retains a parent field and applies a child override. Reuse generic inheritance coverage rather than retesting all of ConfigKit's composition rules.

Use controlled offline fixtures and reuse existing namespace, configuration, ownership, and preparation coverage. CommandKit should own report-format tests showing absent settings omitted and explicit zero preserved. Follow the testing policy; avoid tests coupled to private projection helpers or new recurring release gates.

Completion means an operator can inspect captured prompt content, effective profile settings with accurate omission semantics, and backend settings with effective diagnostic destinations, then use ordinary preparation to preview a concrete request. The chosen engine-policy surface supplies the additional facts needed to describe execution policy without inventing provider or client defaults. Typed backend inheritance is available through the source helper. Existing composition and execution semantics remain intact. Typed prompt construction, listing return-type changes, provider health guarantees, and live capacity monitoring are not completion requirements.