promptkit/docs/roadmap/inspect-backend.md

# Operator-Facing Resource Inspection And Configuration API Roadmap

## Purpose And Status

Help operators using CommandKit-based applications inspect the configured
prompt, profile, or backend selected by ID. CommandKit owns commands and report
presentation; Promptkit exposes the selected resource's resolved configuration.

The direction is accepted for planning; implementation has not begun. Selected
work includes fuller prompt inspection, profile inspection of the resolved
profile itself, a new `InspectBackend`, and `BackendConfig.Extends`. Execution-
policy reporting and prepared-request enhancements have been removed from this
roadmap. Decisions below record the accepted boundary and remaining choices.

## Resource-Inspection Boundary

“Fully resolved” means that ConfigKit has applied source composition and
inheritance to the selected resource. A child profile should show its resulting
keys and values, not an unresolved `extends` directive. It does not mean that
inspection assembles everything affecting execution.

| API | Resource shown |
| --- | --- |
| `InspectPrompt` | Resolved prompt definition, including captured imported message content. |
| `InspectProfile` | Resolved profile definition, before backend-to-profile execution completion. |
| Proposed `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. |

For example, given:

```yaml
profiles:
  - id: base
    backend: openrouter
    model: example-model
    temperature: 0.5
  - id: precise
    extends: {id: base}
    temperature: 0
```

The resource settings shown for `precise` are:

```yaml
id: precise
backend: openrouter
model: example-model
temperature: 0
```

ConfigKit inheritance contributes the parent's model and backend reference.
Promptkit's execution-time completion would obtain settings from that backend,
but those settings do not become fields of the inspected profile. The operator
can inspect the referenced backend separately. Likewise, prompt inspection
reports a default profile reference without incorporating that profile's values.

Use the captured, resolved configuration, including normalizations/defaults
already recorded in that configuration. Do not add framework execution defaults,
request overrides, credential values, or provider-derived defaults to fill
absent resource fields. For example, an absent profile timeout is not populated
from the framework generation timeout, while a queue default already recorded
in the resolved backend remains visible. See the
[format reference](../formats.md#source-headers-collections-and-resource-ids).

## Design Balance And Reporting

Provide useful resource information already captured or available with modest
projection. Reuse ConfigKit's resolved handoff and existing resource capture;
avoid another composition pass, execution resolver, parallel provenance system,
or duplicate retained graphs solely for inspection. These are proportionality
guidelines, not prohibitions on a small helper or type when it serves the API.

**Selected presentation:** Omit absent optional settings. Preserve explicit
values, including zero. Absence is not displayed as zero or `unspecified`.
An explicit null retained in the resolved resource is a configured value, not
an absent key; preserve that distinction where it affects resource semantics.
The previous draft's rule to omit suppressed outbound controls belonged to an
execution report and does not define resource inspection. Representation of
presence/null in public results is open decision 4.

Inspection reports effective resource values rather than original YAML layout,
comments, or the history of each inherited field. `InspectConfiguration` remains
available for provenance. Existing opaque prompt identity/hash information can
remain alongside the resource view; it does not turn inspection into a source
round-trip API.

Retain existing captured-state, namespace, ownership, and credential-handling
contracts unless a change is deliberately selected. Mutable results should not
let a consumer change retained engine state; that does not require repeated
copies between private helpers or copying immutable strings. Ordinary resource
inspection needs no rendering, provider calls, or credential availability checks.
Templates and arbitrary provider settings remain application data whose
presentation and disclosure are controlled by the consuming application.

## Expanded Prompt Inspection

Expose description, ordered message templates with roles and cache-control
metadata, and the session-ID template, alongside the existing identity, hash,
default profile reference, input declarations, and output contract. Include
schema references and repair settings as part of the prompt definition.

Captured prompt definitions already retain the effective content of inline and
imported text/Markdown messages. Return that text without rendering or reopening
files. Original source layout and parsed-template objects are unnecessary.
The schema remains a reference; inspecting the prompt does not load it.

Keep prompt inspection independent of request readiness and default-profile
lookup. A captured prompt remains inspectable without a runnable default profile.

## Resolved Profile Inspection

This is a deliberate change to existing `InspectProfile` behavior. It currently
returns backend-completed `EffectiveModelParams`; the new resource boundary
requires the resolved profile's own fields instead.

Report fields supplied by the profile or inherited through ConfigKit, including
backend reference, endpoint/model where present, generation controls, credential
metadata, and provider parameters. Do not merge in backend provider defaults or
resolve an environment variable. Profiles may be partial, and their references
remain references. Distinguish omitted values, explicit zero, and meaningful
explicit null using the resolved resource's information, rather than inferring
presence from an execution target.

Execution target presence and prepared metadata describe a later resolution
stage and are not substitutes for profile configuration presence. Use the
existing configuration capture path to expose the needed information without
introducing a second configuration interpreter. The precise result shape and
migration from `EffectiveModelParams` are open decision 4.

This roadmap changes no `Prepare`, `PreparedRun`, `PrepareExecution`, or
`PreparedExecution.Details` contract. Execution continues to apply backend
completion, framework defaults, and request overrides through its existing path.

## Backend Inspection

Proposed entry point:

```go
func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)
```

Expose the full backend identity and its resolved fields: base endpoint,
credential environment-variable name, provider defaults, concurrency/queue
settings, and configured `smoke_tests` metadata. Reuse suitable public nested
values where their semantics match; exact names and nesting follow decision 4.

Diagnostic metadata includes the configured authentication kind, profile
reference, and endpoint overrides after ConfigKit composition/inheritance.
An inherited endpoint override is a resolved configured value. An endpoint
computed later by a smoke-test adapter is not. Do not synthesize `/key` URLs,
reachability fallbacks, or a `smoke_tests` block that is absent from the resolved
backend. No shared diagnostic URL derivation work is required for this feature.

Inspecting a backend does not resolve its smoke-test default profile or contact
an endpoint. Report the configured reference even if that profile cannot be
selected. Capacity values describe configured policy, not current occupancy.

Follow the existing namespace-view selection rules: qualify slashless caller
IDs with the backend scope, pass qualified IDs through, and return the selected
full ID. Bundled and caller-defined resources use the same lookup rules.
Follow existing inspection conventions for invalid engines, invalid selections,
and cancellation. Missing-backend classification follows accepted decision 1.

## Consumer Integration And Compatibility

CommandKit can offer navigation between prompt, profile, and backend references
without combining their settings into the inspected resource. Stored references
retain their exact meaning; caller namespace defaults do not rewrite them.
A particular connected layout is a consumer choice, not a new Promptkit API.

Coordinate CommandKit adoption of the changed profile result and richer prompt
and backend views. In particular, migrate any formatter that currently expects
`EffectiveModelParams`. The profile semantic change must be documented as such,
not presented as merely adding more fields. Select the Go/serialization migration
in decision 4 and follow the existing release policy for compatibility changes.

Existing preparation APIs remain available for separate concrete-request uses;
this roadmap adds neither preview commands nor execution-policy report fields.
Transport timeouts, engine retry settings, caller deadlines, and injected-client
policy are outside the inspected resource unless they are actual fields of its
resolved definition. This is the scope of these APIs, not a pending engine-policy
feature decision.

## Typed Backend Inheritance

Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
`BackendSource` encodes a non-nil value as `extends: {id: value}` and omits it
when nil. ConfigKit owns sibling inheritance and its diagnostics. The source
helper does not look up parents or populate inherited fields itself.

Keep existing root/collection identity, replacement, and null semantics. Raw
mappings remain available for generic controls outside the convenience helper.
See [resource inheritance](../formats.md#resource-inheritance) and the existing
[profile helper](../../profiles.go).

## Related Work

Typed prompt construction remains open decision 3. Listing return-type
consistency is a possible later cleanup, not required here. Collections,
schemas, and engine options do not need identical APIs merely for symmetry.
The [deferred fingerprint proposal](deferred.md#semantic-execution-target-fingerprints)
remains separate. Live capacity monitoring, provider health guarantees, and
source round-tripping are not selected deliverables.

## Decisions

Decisions 1, 2, and 5 are accepted. Decisions 3 and 4 remain open. Numbering is
retained for continuity with the discussion; implementation is pending.

### 1. How Should An Unknown Backend Be Classified?

**Status:** Accepted; implementation pending.

**Decision:** Add public `ErrBackendNotFound` for an absent backend selected by
`InspectBackend`, following the existing prompt/profile sentinel pattern.
An ordinary missing-backend lookup should match `ErrBackendNotFound` through
`errors.Is` and should not also match `ErrInvalidRequest`. Reuse the registry's
existing internal missing-backend classification at the public boundary.

Invalid selections follow existing ID rules and match `ErrInvalidRequest`;
invalid engines match `ErrInvalidConfig`. Preserve context cancellation through
the existing inspection conventions. Invalid backend definitions remain an
engine-construction concern. Exact error wording is not a new contract.

Initially apply the new public classification to `InspectBackend`; do not
implicitly reclassify construction or smoke-test failures. CommandKit owns
operator wording and exit-status selection.

**Rationale:** This gives consumers a useful distinction with a small addition
consistent with prompt and profile inspection, without a new error hierarchy.

**Alternatives considered:** Using `ErrInvalidRequest` saves one public sentinel
but combines malformed selections and missing identities. A typed generic
resource-not-found error could carry resource kind and ID, but adds a broader
contract and integration questions for existing sentinels. Neither is selected;
the caller already knows the operation and requested ID.

### 2. Resource Inspection Does Not Include Execution-Policy Reporting

**Status:** Resolved; the former engine-policy proposal is removed from scope.

**Decision:** Inspect the selected resolved resource, not all settings that may
affect its execution. ConfigKit composition and inheritance are included;
backend-to-profile execution completion, framework execution defaults, request
overrides, and engine transport/retry reporting are not.

Remove the proposed engine-settings API, prepared-result policy fields, and
prepared-result presence enhancements from this roadmap. No decision about
where to place those additions remains necessary for resource inspection.

### 3. Should Typed Prompt Construction Be Added In This Round?

**Recommendation:** Keep `PromptConfig`/`PromptSource` separate until a
programmatic authoring need justifies the public input model. Files and ConfigKit
mappings already support prompts; this does not block operator inspection.

**Alternative:** Add the typed helper for discoverability and compiler assistance.
It would need useful message/input/output and omission semantics, inheritance,
and source context for relative content files. Its maintenance cost is larger
than adding backend `Extends`; composition can still delegate to ConfigKit.

### 4. How Should Resolved Resource Results And Presence Be Represented?

The chosen meaning of inspection is settled; the public Go shape and migration
from the existing backend-completed profile result are not.

**Recommendation:** Use a resource-oriented typed profile result, reusing suitable
public nested values and representing absent/explicit-zero/explicit-null values
where needed. Prefer a deliberate, documented change to `ProfileInspection` over
retaining a second execution-settings view just for compatibility. Choose the
smallest presence representation supported by captured configuration; do not
reuse `ExecutionTargetPresence` as if it represented the uncompleted profile.
Apply the same resource semantics to prompt/backend result additions without
requiring a generic resource framework.

**Alternative:** Expose resolved resource fields as a caller-owned mapping,
possibly alongside typed identity and prompt metadata. That naturally represents
missing keys and null and can reduce per-field projection work, but gives up Go
field discoverability and requires careful definition of the mapping's values
and compatibility. Compare that cost with the typed representation rather than
assuming another complete family of public configuration types is necessary.

Decide the migration for existing `EffectiveModelParams` consumers and any
serialization changes explicitly. CommandKit can own its report formatting;
new JSON contracts or custom serializers are not automatically required. Neither
option authorizes leaking internal package types or mutable captured state.

### 5. Backend Diagnostics Show Configured Values, Not Derived Destinations

**Status:** Resolved by the resource-inspection boundary.

**Decision:** Return diagnostic settings present in the resolved backend,
including inherited endpoint overrides. Leave omitted endpoint settings absent;
deriving a default destination belongs to diagnostic execution. The earlier
proposal to expose both declared and computed destinations is removed. This
avoids an additional representation and provider-specific derivation machinery
in resource inspection.

## Implementation And Verification

1. Resolve result representation/migration in decision 4. Decide typed prompt
   construction separately; it is not a prerequisite for the selected work.
2. Expose expanded prompt content and resolved profile fields through existing
   capture/lookup paths, before execution completion. Preserve configured
   presence/null distinctions without another composition pass.
3. Add backend inspection and the accepted public missing-backend error,
   returning configured diagnostic metadata without endpoint derivation.
4. Add typed backend inheritance through the existing ConfigKit source helper.
5. Update GoDoc, consumer/internal guidance, and release notes for the selected
   behavior and profile compatibility change. Coordinate CommandKit reporting.

Use focused offline tests for actual boundary risks:

- Child resources show the resolved inherited fields and overrides, not an
  unresolved `extends` directive; explicit zero survives and absent fields
  remain distinguishable from configured null where meaningful.
- Profile inspection excludes settings supplied only by a referenced backend,
  framework execution defaults, or a request. Existing execution still applies
  those settings through its ordinary resolution path.
- Prompt inspection exposes captured inline/imported content, metadata, and
  order without rendering or adding related-resource lookup dependencies.
- Backend inspection reports configured/inherited diagnostic overrides while
  leaving omitted overrides absent. It does not require diagnostic readiness.
- Resource selection, error classification, and mutable-result ownership follow
  the selected contract; reuse existing namespace and ownership coverage.
- Typed backend inheritance works through real ConfigKit composition without
  retesting ConfigKit's entire generic inheritance feature set.

CommandKit owns presentation tests for the migrated resource view. Follow the
[testing policy](../policy/testing.md), combine cases where useful, and reuse
existing coverage. No new benchmark threshold or release gate is selected.

Completion means operators can inspect the resolved selected prompt, profile,
or backend accurately without assembling execution settings or navigating raw
configuration paths. Backend inheritance is available through the typed source
helper, and consumers have migration guidance for changed profile inspection.
No engine-policy or prepared-request additions are part of completion.