promptkit/docs/roadmap/inspect-backend.md

Backend Inspection And Configuration API Roadmap

Purpose And Status

Add Engine.InspectBackend so applications can retrieve one backend's effective definition by its runtime identity, and expose existing backend inheritance through BackendConfig and BackendSource. These additions are selected for planning; implementation has not begun. The open decisions below remain to be resolved. Typed prompt construction is a separate discussion item, and listing return-type consistency is a possible later cleanup.

The consumer benefit is backend-detail reporting without selecting an arbitrary profile or reconstructing domain settings from ConfigKit paths and list indexes. For example, Scriptorium could show a backend's endpoint, credential source, capacity settings, provider defaults, and diagnostic configuration. Promptkit owns this reusable interpretation; downstream applications own presentation, disclosure, CLI commands, and exit codes.

API symmetry alone does not justify additions. Prioritize backend inspection with the recommended missing-backend error, followed by typed backend inheritance. Both expose behavior or information Promptkit already owns, with no new source composition, validation framework, or execution lifecycle.

Relationship To Existing APIs

API	Responsibility
ListBackends	Discover captured backend identities.
Proposed InspectBackend	Describe one backend's effective settings by ID.
InspectProfile	Resolve a profile's backend-completed execution settings.
InspectConfiguration	Inspect effective configuration and source provenance.
SmokeTest	Execute an explicitly requested diagnostic and report evidence.

See the consumer guide for the implemented inspection and discovery boundaries, and the architecture policy for library responsibilities.

Proposed Backend Inspection Contract

func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)

Use a dedicated public result rather than returning BackendConfig. BackendConfig is an input type whose pointers express omitted and explicit values; inspection describes the normalized effective definition after composition and defaulting. Do not expose internal domain types.

The proposed result includes:

Information	Meaning
BackendID	Full selected runtime identity.
Endpoint	Captured backend base endpoint.
APIKeyEnv	Configured environment-variable name, never its resolved value.
ExtraParams	Backend provider defaults, before profile or request overrides.
ConcurrencyLimit	Configured generation concurrency limit.
QueueCapacity	Effective queue capacity, including construction-time defaults.
SmokeTests	Optional diagnostic settings, including the inference default profile reference.

Scalar fields describe effective values rather than reproducing source-field presence. Diagnostic optionality must remain expressible; its representation depends on open decision 1. Exact nested type names are implementation choices. Follow existing typed inspection conventions without adding a stable JSON serialization contract for this feature. Applications can define their own output formats.

Selection And Errors

Follow the existing namespace selection rules: a scoped engine view qualifies slashless IDs with its backend namespace; qualified IDs pass through unchanged. An unscoped view looks up the supplied ID directly. Return the full selected ID. Do not add fallback search across namespaces, implicit backend selection, or special handling for bundled backends. Use the existing ID rules, including case sensitivity, rather than introducing new normalization.

A nil or zero-value engine should match ErrInvalidConfig; a blank or otherwise invalid selection should match ErrInvalidRequest. Cancellation should retain the context error, and failures should return no partial result. The public classification for a missing backend is open decision 2.

Captured State And Ownership

Inspect the engine's captured backend definition without reloading sources, reading credential values, preparing a prompt, acquiring capacity, or making provider calls. Engine construction already validates effective definitions; inspection should not repeat that work or create an additional acceptance path.

Capacity fields describe configuration, not current occupancy or queue length. Diagnostic settings describe configuration, not provider availability, key acceptance, or inference readiness. In particular, report a smoke-test default profile reference without resolving or validating that profile merely to inspect the backend. Its selection-time behavior remains owned by SmokeTest.

Return independently owned mutable values, including nested provider maps and diagnostic settings. Mutating a result must not affect the engine or another inspection. Reuse existing ownership operations and avoid consecutive copies whose only purpose is to cross private helper boundaries.

Provider defaults should retain their values, consistent with typed target inspection. They are application data and may contain sensitive content; the result is not automatically safe for logging or disclosure. Do not resolve credentials or silently substitute ConfigKit's redacted reporting values for the typed settings. Operator provenance and redacted configuration reporting remain available through InspectConfiguration.

Typed Backend Inheritance

Add Extends *string to BackendConfig, matching ProfileConfig.Extends. When non-nil, BackendSource should encode it as extends: {id: value} in the backend record. Nil omits the field. Preserve the supplied value and let ConfigKit resolve and validate inheritance through the existing source-loading path, just as it does for profiles.

The purpose is to let programmatic callers derive a backend variant without switching from BackendSource to a raw mapping. The helper must not resolve a parent, fill inherited fields, or become a second inheritance implementation. It remains a source constructor rather than an engine registry mutation API.

This exposes existing sibling inheritance, not a new configuration feature. Parent IDs select siblings in the same root or collection resource list; namespace views do not qualify these configuration references. Existing same-ID replacement, omitted-field, and explicit-null behavior remain owned by ConfigKit composition. Raw mappings remain available when callers need explicit null or other generic configuration controls that the typed helper cannot express. See the inheritance reference and the existing profile source helper.

Scope Of API Consistency

Backends, profiles, and prompts already share source layering, collection membership, namespace selection, sibling inheritance, and construction-time definition validation. The gaps addressed here are in public convenience and inspection APIs, not in those underlying configuration capabilities.

Inspection remains purpose-specific: InspectPrompt reports prompt interface metadata, InspectProfile reports backend-completed execution settings, and InspectBackend will describe an effective backend. This work does not turn them into uniform complete-definition retrieval APIs. Expanding prompt inspection to return descriptions, message templates, or session templates should follow a concrete consumer need rather than the new backend result's shape.

ListPrompts currently returns []PromptIdentity, whose only field is ID; ListProfiles and ListBackends return []string. At a suitable breaking-change window, prefer considering []string for all three over adding thin identity wrappers for backends and profiles. That would simplify consumer code, but requires a deliberate compatibility decision and migration instructions. Keep the existing return types in this work; listing consistency does not block the backend additions.

Collections, schemas, and engine configuration remain distinct concepts rather than additional peers requiring identical resource APIs. Prompt requirement inspection and backend capacity/diagnostic settings retain their specialized roles. Profile/backend hashes are not added for symmetry; the existing deferred fingerprint proposal owns that separate question.

Open Decisions

1. Declared Diagnostic Settings Or Derived Destinations?

Recommendation: Return normalized captured diagnostic settings, preserving omitted endpoint overrides and the default profile reference. An absent override continues to mean that the diagnostic applies its existing default when run. Use dedicated result types where needed to express this without treating the whole inspection result as configuration input.

This keeps inspection independent of provider-specific destination derivation and execution selection. It also avoids a second place that must reproduce authentication-adapter rules. Construction-time defaults already captured in the backend, such as queue capacity, still appear as effective values.

Alternative: Also expose derived reachability and authentication URLs. This would help applications display the destination before invoking a diagnostic, but would require sharing the adapter's derivation logic and clearly separating declared settings from derived results. It must not imply that the selected client supports HTTP diagnostics or that the destination is reachable. Add this only if consumers need that extra information.

2. How Should An Unknown Backend Be Classified?

Recommendation: Add public ErrBackendNotFound for this explicit lookup, matching the role of ErrProfileNotFound and ErrPromptNotFound. The registry already has an internal backend-not-found sentinel. Map it at the facade so applications can distinguish a missing backend from malformed input without inspecting text or internal packages.

Limit this addition to the new inspection operation unless existing callers justify a broader change; do not automatically reclassify smoke-test or engine construction errors.

Alternative: Classify an absent ID as ErrInvalidRequest. This avoids another exported sentinel but combines a well-formed missing identity with an invalid selection. Consumers needing different presentation would then require an inventory lookup or additional error detail. A sentinel is the smaller consumer-facing solution when that distinction matters.

3. Should Typed Prompt Construction Be Added In This Round?

Recommendation: Keep PromptConfig and PromptSource as a separate discussion item until a programmatic prompt-authoring use case establishes their value. File sources and ConfigKit mappings already support prompt definitions. The absence of a helper is a Go convenience gap, not a missing configuration capability, and does not block the backend additions.

Alternative: Add a typed prompt source alongside the existing backend and profile helpers. It would improve discoverability, compiler assistance, and consistency for applications constructing prompts in Go. Its cost is a larger public input model covering messages, inline/file content, inputs, output contracts, session templates, and omission semantics. If selected, define that scope separately, including sibling inheritance and source context for relative content files, while retaining ConfigKit's ownership of composition and raw mapping support for explicit null. Do not implement a second prompt parser or validation path merely to provide the helper.

Implementation Scope And Verification

1. Resolve backend inspection decisions 1–2 and define the result and GoDoc. Decide separately whether to expand this round for typed prompt construction; the recommended backend scope does not depend on that expansion.
2. Connect namespace-aware engine selection to the existing captured backend lookup and project the result at the public facade. Introduce only the private collaborators needed for this path; no new backend cache or loader.
3. Add focused public tests for root and namespaced identities, bundled and caller-defined backends, effective capacity defaults, provider maps, optional diagnostic settings, errors, cancellation, and independent mutable results. Cover a missing smoke default profile and absent credential environment variable to establish that inspection does not perform execution readiness checks. Use controlled local fixtures; no live provider calls are needed.
4. Add BackendConfig.Extends and encode it through BackendSource. Verify an inherited backend through real ConfigKit composition, including an overridden field and a retained parent field. Reuse existing inheritance coverage for generic missing-parent, cycle, and replacement semantics rather than retesting ConfigKit's entire feature set.
5. Update the consumer guide and relevant internal documentation when the API lands, with a short ListBackends followed by InspectBackend example. Include typed backend inheritance guidance. Record the additive APIs and selected error behavior in release notes.

Reuse meaningful existing namespace, configuration, and ownership coverage; avoid tests that merely mirror private projection helpers. Run checks appropriate to the implementation under the testing policy.

Completion means consumers can inspect a backend without selecting a profile or navigating raw configuration, and derive a backend through the typed source helper without constructing a raw mapping. Existing execution, inspection, diagnostic, and configuration-composition behavior remains intact. No separate provider capability registry, live capacity monitor, or backend health API is included. Typed prompt construction and listing return-type changes are not completion requirements for the selected backend scope.