promptkit/docs/roadmap/inspect-backend.md

# Operator-Facing Resource Inspection And Configuration API Roadmap

## Purpose And Status

Help operators using CommandKit-based applications understand Promptkit's
captured prompt, profile, and backend configuration and the settings applicable
to a concrete request. CommandKit owns operator-facing commands and presentation;
Promptkit supplies facts from its existing configuration and execution paths.

The direction is accepted for planning; implementation has not begun. Selected
work includes expanded prompt inspection, presence-aware profile/prepared
inspection, backend inspection with effective diagnostic destinations, and
`BackendConfig.Extends`. Open decisions below cover unresolved scope and public
representation choices. They are not implicit implementation requirements.

## Design Balance

Provide useful information already captured or cheaply derived. Prefer exposing
existing values and small shared helpers over introducing new resolution paths,
parallel report models, provenance tracking, or retained snapshots solely for
inspection. This is a design preference, not a prohibition on a small adapter
or additional field when it serves a demonstrated consumer need.

“Fully resolved” describes the applicable resolution stage. It is not a promise
that inspection can reconstruct the entire authored source, report every future
configuration field automatically, predict provider defaults, or reproduce a
future invocation independently of its request and external state.

| View | Information available |
| --- | --- |
| Resource inspection | Captured values after composition, inheritance, normalization, and applicable Promptkit defaults. |
| Concrete request preparation | Selected profile, request overrides, rendered messages, input metadata, session ID, and loaded schema for supplied request data. |
| Execution | Provider responses, actual usage, and other observed runtime facts. |

The planned fields below define the initial additions. Additional cheap,
useful fields can be considered during implementation without building a
uniform resource framework or promising exhaustive future coverage. If a field
requires substantial new machinery, record the tradeoff before expanding scope.

### Review Findings And Existing Boundaries

The current implementation provides a relatively small path to the core work:

- Captured prompt definitions already retain description, session template,
  message text, roles, and cache-control metadata alongside compiled templates.
  Inspection can expose that data without reconstructing source files or ASTs.
- Internal prepared results already retain `TargetPresence`; the root API has
  an `ExecutionTargetPresence` type. Presence-aware output need not introduce
  a second interpretation of configuration or provider-value conversion.
- Smoke selection already has endpoint derivation, including the OpenRouter
  key URL helper. Reusing or factoring that logic need not create an adapter
  capability registry or provider metadata service.
- Transport timeout and retry policy live in different components and are not
  all carried by prepared results. Their reporting scope and cost need a
  decision; they are no longer an unconditional completion requirement.

These observations are implementation evidence, not promises to retain those
private types or helper arrangements. See the current
[source/capture](../internal/sources.md), [runner](../internal/runner.md), and
[model-client](../internal/llm.md) descriptions.

Retain established public ownership, credential handling, captured-state, and
namespace behavior unless a change is deliberately selected. Independent
mutable results prevent callers from changing engine state; they do not require
fresh copies of immutable strings or repeated deep copies between private layers.
Existing `Inspect*` APIs avoid provider work and request preparation. Extending
their returned data should not silently change those behaviors. These boundaries
follow the existing API and the selected operator use case, rather than a new
requirement that all future inspection APIs have identical behavior.

## Reporting Semantics

**Selected presentation:** Omit unset optional settings from operator reports.
Preserve explicit values, including zero, and applicable Promptkit defaults.
Suppressed provider controls are also absent from an effective-settings report.
Neither zero nor `unspecified` should stand in for an omitted control.

This concerns the effective setting, not an explanation of every reason it was
omitted. ConfigKit inspection remains useful for authored values and history.
Promptkit supplies sufficient presence information; CommandKit chooses report
layout. Representation and serialization choices remain open in decision 4.

Do not infer provider defaults from absent controls. Likewise, a configured
model alias is the alias, not a prediction of the model it will select later.
Credential source names and requirements are useful inspection facts; resolved
credential values remain excluded under existing contracts. Templates and
arbitrary provider parameters are application data, not automatically safe
for disclosure. No new general-purpose sanitization guarantee is selected.

## Expanded Prompt Inspection

Expand `PromptInspection` to include description, ordered message templates
with roles and cache-control metadata, and the session-ID template. Retain the
existing identity, opaque hash, default profile reference, input declarations,
and output contract, including schema reference and repair settings.

Expose effective content captured from imported text or Markdown files, as well
as inline content. Return the template text available in the captured definition;
source-layout reconstruction, original YAML formatting, and parsed-template
objects are unnecessary for this use case. There is no new byte-for-byte source
round-trip requirement. Message order and text should reflect what rendering
consumes. Inspection continues to use captured state without rendering or
reopening files; the schema stays a reference until ordinary preparation loads it.

Prompt inspection remains useful without a supplied request or a resolvable
default profile. The existing independent lookup permits CommandKit to show a
prompt even when related execution settings need attention.

## Presence-Aware Profile And Prepared Inspection

`ProfileInspection` exposes effective settings without the presence information
that execution uses for explicit-zero controls. `PreparedRun` has the same
public reporting gap, including values returned by `PreparedExecution.Details`.
Expose enough existing resolved presence information to distinguish omission
from explicit settings in each of those results.

For example, an unset temperature produces no temperature entry in the report;
an explicit zero produces `temperature: 0`. A concrete request's override appears
in its prepared view. Backend completion, inheritance, and request precedence
remain the execution path's responsibility. Inspectors need not reconstruct
that history or distinguish all historical reasons for an absent setting.

Reuse the captured target and presence semantics, including existing handling
of nonzero values. Decision 4 weighs exposing the current presence type against
another public representation; it does not reopen the omission-reporting decision.
See [defaults and overrides](../formats.md#defaults-and-overrides).

## Backend Inspection

Proposed entry point:

```go
func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)
```

The result should expose the full backend identity, base endpoint, credential
environment-variable name, provider defaults, effective concurrency/queue
settings, and diagnostic settings and destinations. These are backend defaults,
not settings after a particular profile or request overrides them. Capacity
values describe policy, not current occupancy.

A dedicated top-level inspection result is recommended because `BackendConfig`
is a presence-aware source input. This does not require a new type for every
nested value: reuse suitable public values where their semantics match. Exact
field names and nesting can be selected during implementation within the public
representation decisions below; internal types remain behind the facade.

Follow existing namespace-view selection: qualify slashless caller IDs with the
backend scope, pass qualified IDs through, and report the selected full ID.
Use the same identity rules for bundled and caller-supplied backends. The new
operation should follow existing inspection conventions for invalid engines,
invalid selections, and context cancellation, without adding cross-error
precedence guarantees. Missing-backend classification follows accepted decision 1
below.

### Effective Diagnostic Destinations

Include effective reachability and applicable authentication destinations,
using explicit overrides or existing backend-relative defaults. Share the
existing derivation logic where practical; neither CommandKit nor a new
inspection-specific resolver should maintain a competing set of provider rules.
A small helper extraction is sufficient if that is all the current adapters need.

Default reachability applies even without authored `smoke_tests` metadata.
Report the configured authentication kind and optional inference-default profile
reference without fabricating authored settings. An authentication kind of
`none` has no active authentication destination. Decision 5 covers whether the
result also needs separate declared endpoint fields.

Destinations describe captured backend settings, not provider reachability or
HTTP diagnostic support in an injected client. Retain the inference-default
profile as a reference; backend inspection need not resolve it or establish
inference readiness. Ordinary inspection is expected to return a result or an
error, following existing methods. If derivation reveals a meaningful case where
useful captured settings would be lost, bring that case back for a decision
rather than automatically adding partial-result and warning machinery.

## Connected Views And Concrete Requests

CommandKit can follow prompt default-profile and selected-backend references
through the existing inspections to build a connected view. This is a suggested
consumer workflow, not a required report format or a restriction on future
convenience APIs. Use stored references exactly; namespace defaults apply to
caller selections, not to reinterpreting captured references. Endpoint-only
profiles need no backend node. A consumer can retain successful lookups while
showing a failure for a related resource.

Use existing `Prepare` for concrete variables, inputs, profile selection, and
request overrides. It performs artifact/schema reads, rendering, and applicable
credential checks without inference. `PrepareExecution` and its details serve
callers that want later execution tied to the same preparation, subject to that
API's existing contract. This roadmap adds no stronger replay guarantee and
selects no second preparation workflow.

Model parameters are not all execution policy. Per-generation timeout, transport
cap, retry settings, backend capacity, and caller deadlines are different facts.
When reported, keep their meanings distinct. Decision 2 determines whether and
how this round should expose additional engine-policy information. Report known
settings without claiming knowledge of an injected client's internal behavior.

CommandKit owns CLI commands, formatting, and disclosure. Coordinate downstream
adoption when the Promptkit additions land; this roadmap does not prescribe
CommandKit's internal adapter design or require a new stable CLI serialization
contract in Promptkit.

## Typed Backend Inheritance

Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
`BackendSource` should encode a non-nil value as `extends: {id: value}` and omit
the field when nil. ConfigKit owns inheritance resolution and its diagnostics.
The helper exposes existing sibling inheritance, rather than performing parent
lookup or filling inherited fields itself.

Keep existing root/collection identity, replacement, and null semantics. Raw
mappings remain available for explicit null and other generic controls outside
the helper's convenience surface. See the
[inheritance reference](../formats.md#resource-inheritance) and
[profile helper](../../profiles.go).

## Related Work And Scope

Typed prompt construction is open decision 3. Listing return types remain a
possible later cleanup: `ListPrompts` returns an ID-only `PromptIdentity` wrapper,
while backend/profile listings return strings. Standardizing on strings may be
useful at a breaking-change window, but is not required for these inspections.

Collections, schemas, and engine options have different roles; symmetry does
not require identical APIs for them. The
[deferred fingerprint proposal](deferred.md#semantic-execution-target-fingerprints)
remains separate. Live capacity monitoring, provider health guarantees, and
source round-tripping are not selected deliverables, rather than permanent
prohibitions on future work.

## Decisions

Decision 1 is accepted; decisions 2–5 remain open. Implementation is pending.

### 1. How Should An Unknown Backend Be Classified?

**Status:** Accepted; implementation pending.

**Decision:** Add public `ErrBackendNotFound` for an absent backend selected by
`InspectBackend`, following the existing prompt/profile sentinel pattern.
An ordinary missing-backend lookup should match `ErrBackendNotFound` through
`errors.Is` and should not also match `ErrInvalidRequest`. Reuse the registry's
existing internal missing-backend classification at the public boundary.

Invalid selections follow existing ID rules and match `ErrInvalidRequest`;
invalid engines match `ErrInvalidConfig`. Preserve context cancellation through
the existing inspection conventions. Invalid backend definitions remain an
engine-construction concern. Exact error wording is not a new contract.

Initially apply the new public classification to `InspectBackend`; do not
implicitly reclassify construction or smoke-test failures. CommandKit owns
operator wording and exit-status selection.

**Rationale:** This gives consumers a useful distinction with a small addition
consistent with prompt and profile inspection, without a new error hierarchy.

**Alternatives considered:** Using `ErrInvalidRequest` saves one public sentinel
but combines malformed selections and missing identities. A typed generic
resource-not-found error could carry resource kind and ID, but adds a broader
contract and integration questions for existing sentinels. Neither is selected;
the caller already knows the operation and requested ID.

### 2. How Much Engine-Policy Reporting Belongs In This Round?

The previous draft made a comprehensive execution-policy section a completion
requirement before assessing the cost of carrying transport and retry facts
through preparation. That commitment is now open, including the option to defer.

**Recommendation:** Start with facts already held by the relevant owner and add
small fields to an existing prepared view where they are inexpensive to carry
accurately. Evaluate transport cap and effective retry settings individually.
Avoid a normalized policy model covering every client, invocation, and deadline.
If a useful fact requires substantial new plumbing, explicitly defer that fact
and document the report's boundary instead of inventing a default or guarantee.
The field set and location need agreement before implementation.

**Alternative:** Add a dedicated engine-settings view for captured defaults,
paired with prepared results for request-specific settings. This is useful to
operators without a concrete request, but adds an API and requires distinguishing
defaults from overrides. A third option is to defer engine-policy reporting
entirely for this round, delivering the resource/presence improvements first;
that has the lowest cost but leaves transport/retry reporting to later work.

### 3. Should Typed Prompt Construction Be Added In This Round?

**Recommendation:** Keep `PromptConfig`/`PromptSource` separate until a
programmatic authoring need justifies the public input model. Files and ConfigKit
mappings already support prompts; this does not block operator inspection.

**Alternative:** Add the typed helper for discoverability and compiler assistance.
It would need useful message/input/output and omission semantics, inheritance,
and source context for relative content files. Its maintenance cost is larger
than adding backend `Extends`; composition can still delegate to ConfigKit.

### 4. How Should Presence And Public Result Values Be Represented?

The operator requirement is accurate omission, not a particular Go structure,
new parallel report schema, or fixed JSON policy.

**Recommendation:** Reuse existing public nested values and expose the existing
`ExecutionTargetPresence` semantics alongside effective profile/prepared targets
where practical. This takes advantage of data already used by execution and
limits new translation. CommandKit can omit absent fields in its own output.
Document how values and presence interact rather than introducing an independent
presence/default/provenance system. Prefer additive changes when straightforward.

**Alternative:** Use optional-valued fields in an inspection-specific settings
result, making absence directly expressible to consumers. That can simplify
reporting, but duplicates an execution-settings surface and needs conversion
and maintenance as fields evolve. Choose it only if the consumer simplification
outweighs that cost. Ordinary JSON tags or a deliberate serialization change can
also be considered; neither mandatory JSON stability for new results nor a ban
on useful serialization is selected. Existing documented JSON contracts still
need an explicit compatibility decision before changing.

### 5. Are Both Declared And Effective Diagnostic Endpoints Needed?

Effective destinations are selected scope. The previous draft also suggested
keeping declared overrides separately, which could create two representations
and unnecessary origin metadata without a demonstrated reporting need.

**Recommendation:** Report the effective destination, authentication kind, and
default profile reference. Use `InspectConfiguration` for authored values and
provenance. Reuse existing optional values where helpful; no duplicate declared
endpoint or derived-status flag is required merely for symmetry.

**Alternative:** Include declared overrides alongside effective destinations when
operators need that distinction in one backend view. Existing metadata may make
this cheap, but the result then needs clear field meanings and consumer handling
for both. Prefer directly available fields over a new provenance subsystem.

## Implementation And Verification

1. Resolve open public-contract choices relevant to selected work. Agree on any
   engine-policy additions separately so an unresolved optional expansion does
   not silently become a prerequisite for the resource APIs.
2. Expand prompt inspection from captured data and expose existing presence in
   profile/prepared results. Reuse or modestly extend current lookup/projection
   paths; choose private interface layouts during implementation.
3. Add namespace-aware backend inspection and effective destination derivation,
   following the selected public error and result representation decisions.
4. Add typed backend inheritance. Implement any separately selected engine-policy
   reporting using the owning components' effective values.
5. Update GoDoc, consumer/internal guidance, and release notes with implemented
   behavior and any deliberate compatibility changes. Coordinate CommandKit's
   richer reports and omission handling without prescribing a new CLI framework.

Select focused offline tests for actual behavior and boundary risks:

- Newly exposed prompt fields reflect captured inline/imported content and
  order, without introducing rendering or related-resource dependencies.
- Profile and prepared output preserve explicit zero and distinguish controls
  that execution omits, including relevant inheritance and override cases.
- Backend results cover effective defaults/destinations, namespace selection,
  and applicable error behavior. Use cases with missing optional credentials or
  unresolved diagnostic profile references protect independent inspection.
- Returned mutable values cannot alter retained state. Reuse existing ownership
  tests and copy helpers rather than prescribing extra snapshots or allocations.
- Typed backend inheritance works through real ConfigKit composition; its
  generic composition rules do not need another full test matrix.
- Any selected policy-reporting fields reflect their owners' effective settings.

CommandKit can test its own output omission and explicit-zero presentation.
Follow the [testing policy](../policy/testing.md); combine cases where useful,
reuse existing coverage, and do not turn this list into a test-per-field mandate.
No new benchmark threshold or release gate follows from this roadmap.

Completion means the selected fields give operators a materially fuller and
accurate resource view, omission-aware settings are available to report builders,
and existing preparation remains usable for a concrete request. Backend source
inheritance is exposed. The resolved decisions define additional policy fields,
public representations, and compatibility changes; exhaustive execution-policy
coverage, typed prompt construction, listing changes, and speculative future
adapters are not implicit completion conditions.