base-layer/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Align inspection roadmap with operator reporting needs

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-06 02:49:13 +00:00
parent ffaaf552c8
commit a43e0f17ee
1 changed files with 230 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
docs/roadmap/inspect-backend.md
View File

@@ -1,25 +1,51 @@
# Backend Inspection And Configuration API Roadmap
# Operator-Facing Resource Inspection And Configuration API Roadmap
## Purpose And Status
Add `Engine.InspectBackend` so applications can retrieve one backend's effective
definition by its runtime identity, and expose existing backend inheritance
through `BackendConfig` and `BackendSource`. These additions are selected for
planning; implementation has not begun. The open decisions below remain to be
resolved. Typed prompt construction is a separate discussion item, and listing
return-type consistency is a possible later cleanup.
Give operators using CommandKit-based applications a complete, accurate view of
Promptkit's effective prompt, profile, and backend configuration. CommandKit is
the primary consumer of these inspection APIs and owns operator-facing commands
and formatting. Promptkit owns resolved typed facts and their semantics.
The consumer benefit is backend-detail reporting without selecting an arbitrary
profile or reconstructing domain settings from ConfigKit paths and list indexes.
For example, Scriptorium could show a backend's endpoint, credential source,
capacity settings, provider defaults, and diagnostic configuration. Promptkit
owns this reusable interpretation; downstream applications own presentation,
disclosure, CLI commands, and exit codes.
The selected direction is accepted for planning; implementation has not begun.
Expand `InspectPrompt`, expose setting presence in profile and prepared results,
and add `InspectBackend` with effective diagnostic destinations. Retain the
planned `BackendConfig.Extends` convenience addition. Remaining decisions are
listed below; typed prompt construction and listing return-type changes are
lower-priority considerations, not prerequisites for operator inspection.
API symmetry alone does not justify additions. Prioritize backend inspection
with the recommended missing-backend error, followed by typed backend
inheritance. Both expose behavior or information Promptkit already owns, with
no new source composition, validation framework, or execution lifecycle.
This use case supersedes the earlier recommendation to keep prompt inspection
limited to interface metadata and backend diagnostics limited to declared
settings. Operators need to see captured message content and the destinations
that default diagnostic rules select. The implementation should reuse existing
capture, resolution, and adapter logic rather than build a second resolver.
## What Fully Resolved Means
| View | What can be resolved |
| --- | --- |
| Resource inspection | Captured values after composition, inheritance, normalization, and applicable Promptkit defaults. |
| Concrete request preparation | Selected profile, request overrides, rendered messages, inputs, session ID, and loaded schema for supplied request data. |
| Execution | Provider responses, actual usage, and other observed runtime facts. |
A prompt ID alone cannot determine future request overrides or template values.
Resource inspection therefore shows effective templates and references, while
preparation shows the concrete request. Neither should invent provider defaults,
resolve a moving model alias to a hypothetical future model, or claim execution
success from configuration alone.
**Reporting decision:** Omit unset optional settings from operator reports.
Retain explicitly configured values, including zero, and effective Promptkit
configuration defaults. Do not print omitted controls as zero or as
`unspecified`. Suppressed provider controls should likewise be absent from an
effective-settings report. This is a presentation of execution behavior;
`InspectConfiguration` remains available when an operator wants the authored
null, omission, or inheritance history.
Promptkit must expose sufficient typed presence information for CommandKit to
make those choices correctly. A zero-valued Go field alone is not sufficient.
This does not remove fields from existing public types or silently change their
existing JSON representation; CommandKit owns its report projection.
## Relationship To Existing APIs
@@ -27,7 +53,9 @@ no new source composition, validation framework, or execution lifecycle.
| --- | --- |
| `ListBackends` | Discover captured backend identities. |
| Proposed `InspectBackend` | Describe one backend's effective settings by ID. |
| `InspectProfile` | Resolve a profile's backend-completed execution settings. |
| Expanded `InspectPrompt` | Describe the captured prompt, including effective template content. |
| Enhanced `InspectProfile` | Report backend-completed settings with the presence information needed for accurate output. |
| `Prepare` / `PreparedExecution.Details` | Describe a concrete prepared request, including setting presence. |
| `InspectConfiguration` | Inspect effective configuration and source provenance. |
| `SmokeTest` | Execute an explicitly requested diagnostic and report evidence. |
@@ -35,6 +63,52 @@ See the [consumer guide](../consumers/pkg-promptkit.md#discover-available-resour
for the implemented inspection and discovery boundaries, and the
[architecture policy](../policy/architecture.md) for library responsibilities.
## Complete Prompt Inspection
Expand `PromptInspection` to include description, ordered message templates
with roles and cache-control metadata, and the session-ID template. Retain
identity, opaque prompt hash, default profile reference, declared inputs, and
the complete output contract, including schema reference and repair settings.
For messages imported from text or Markdown files, return the effective content
captured at engine construction. Operators should not have to locate or reopen
the original file to see what execution will render. Return templates as text,
not internal parsed-template objects. Inspection neither renders templates nor
rereads content files; edits require a new engine. The schema remains a
reference here; concrete preparation loads the selected schema.
Keep prompt inspection independent of profile selection and request readiness.
An absent or incomplete default profile must not hide an otherwise captured
prompt. A prompt without a default profile remains inspectable. Return owned
mutable results, preserving message and input order and request isolation.
## Presence-Aware Profile And Prepared Inspection
`ProfileInspection` currently exposes effective model settings but not the
`ExecutionTargetPresence` information that distinguishes omitted controls from
explicit zero values. `PreparedRun` has the same reporting gap. Expose the
existing presence semantics alongside the effective target in both results,
including the `PreparedRun` returned through `PreparedExecution.Details`.
Use the resolved target and presence captured by the execution path; do not
infer presence from numeric values or re-run ConfigKit composition. Preserve
backend completion, profile inheritance, explicit suppression, and request
precedence. Keep credential requirement metadata distinct from the credential
source name; no credential value is returned or checked by resource inspection.
For example, the report for a profile that leaves temperature unset should have
no temperature entry. A profile explicitly setting temperature to zero should
show `temperature: 0`. A concrete request that overrides that profile should
show its effective setting instead. Presence metadata is for accurate report
construction; CommandKit need not expose an implementation-oriented bit map to
operators.
Keep the meaning of effective defaults explicit. A configured zero generation
timeout disables that particular deadline; it does not disable the transport
cap or caller cancellation. An omitted provider control supplies no assertion
about the provider's own default. See the
[defaults reference](../formats.md#defaults-and-overrides).
## Proposed Backend Inspection Contract
```go
@@ -59,11 +133,12 @@ The proposed result includes:
| `ExtraParams` | Backend provider defaults, before profile or request overrides. |
| `ConcurrencyLimit` | Configured generation concurrency limit. |
| `QueueCapacity` | Effective queue capacity, including construction-time defaults. |
| `SmokeTests` | Optional diagnostic settings, including the inference default profile reference. |
| `SmokeTests` | Diagnostic settings and effective destinations, including the authentication kind and optional inference default profile reference. |
Scalar fields describe effective values rather than reproducing source-field
presence. Diagnostic optionality must remain expressible; its representation
depends on open decision 1. Exact nested type names are implementation choices.
presence. Diagnostic optionality must remain expressible, including no configured
authentication adapter or inference default. Exact nested type names are
implementation choices.
Follow existing typed inspection conventions without adding a stable JSON
serialization contract for this feature. Applications can define their own
output formats.
@@ -80,7 +155,28 @@ case sensitivity, rather than introducing new normalization.
A nil or zero-value engine should match `ErrInvalidConfig`; a blank or otherwise
invalid selection should match `ErrInvalidRequest`. Cancellation should retain
the context error, and failures should return no partial result. The public
classification for a missing backend is open decision 2.
classification for a missing backend is open decision 1.
### Effective Diagnostic Destinations
Include the effective reachability destination and, when an authentication
adapter applies, its effective destination. Apply explicit endpoint overrides
and existing backend-relative defaults through the same derivation logic used
by diagnostic execution. CommandKit must not reconstruct provider-specific URL
rules. Include the configured authentication kind and inference default profile
reference; an inapplicable authentication kind has no active probe destination.
Default reachability can apply even without a `smoke_tests` block. The result
must be able to report that effective destination without fabricating an authored
block or an authentication configuration. Where distinguishing an explicit
endpoint override from a derived destination is useful, keep those facts
separate rather than overloading a field's meaning. Source history remains
owned by `InspectConfiguration`.
These are destinations under the backend's captured settings, not guarantees
about runtime overrides, the selected client's HTTP diagnostic support, or
reachability. Report a default inference profile reference without resolving it;
a missing or incomplete referenced profile must not make backend inspection fail.
### Captured State And Ownership
@@ -107,6 +203,38 @@ credentials or silently substitute ConfigKit's redacted reporting values for
the typed settings. Operator provenance and redacted configuration reporting
remain available through `InspectConfiguration`.
## Connected Reports And Concrete Request Preview
CommandKit can assemble a connected report by following prompt default-profile
and selected-backend identities through the three inspection APIs. Promptkit
supplies the references and effective values; CommandKit chooses nested sections
or links. Use captured references exactly, without requalifying them through a
view's caller-selection namespace. Endpoint-only profiles need no backend node.
A missing related resource can be reported while preserving successfully
inspected resources; do not make each resource inspection recursively dependent
on all related objects.
Use `Prepare` for an operator preview with concrete variables, inputs, profile
selection, and request overrides. It performs artifact/schema reads, rendering,
and applicable credential checks without inference. Do not describe it as a
pure configuration lookup. Use `PrepareExecution` and its existing details when
later execution must use the same captured preparation; a separate preview
followed by an independent run is not a promise that mutable inputs stay equal.
No second request-resolution API is selected by this roadmap.
An execution report must distinguish per-generation timeout, built-in transport
cap, retry policy, and backend capacity. Effective model parameters alone are
not the entire execution policy. Caller deadlines and injected-client behavior
can add constraints that Promptkit cannot predict for a future invocation.
How to expose the missing engine-policy facts is open decision 2. Do not infer
them from model fields or ask CommandKit to duplicate Promptkit defaults.
CommandKit owns text/YAML presentation and report disclosure. Prompt templates,
input content, and arbitrary provider parameters may be sensitive; preserve
existing credential exclusions without claiming that every inspected value is
safe to publish. No CommandKit implementation is included in this repository's
roadmap; coordinate its consumer adaptation when the Promptkit APIs land.
## Typed Backend Inheritance
Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
@@ -136,12 +264,10 @@ membership, namespace selection, sibling inheritance, and construction-time
definition validation. The gaps addressed here are in public convenience and
inspection APIs, not in those underlying configuration capabilities.
Inspection remains purpose-specific: `InspectPrompt` reports prompt interface
metadata, `InspectProfile` reports backend-completed execution settings, and
`InspectBackend` will describe an effective backend. This work does not turn
them into uniform complete-definition retrieval APIs. Expanding prompt inspection
to return descriptions, message templates, or session templates should follow a
concrete consumer need rather than the new backend result's shape.
Inspection remains purpose-specific while becoming complete for operator use:
prompt templates are captured but unrendered, profile settings include backend
completion, and backend settings include effective diagnostic destinations.
Provenance and authored representation stay with `InspectConfiguration`.
`ListPrompts` currently returns `[]PromptIdentity`, whose only field is `ID`;
`ListProfiles` and `ListBackends` return `[]string`. At a suitable breaking-change
@@ -160,27 +286,7 @@ owns that separate question.
## Open Decisions
### 1. Declared Diagnostic Settings Or Derived Destinations?
**Recommendation:** Return normalized captured diagnostic settings, preserving
omitted endpoint overrides and the default profile reference. An absent override
continues to mean that the diagnostic applies its existing default when run.
Use dedicated result types where needed to express this without treating the
whole inspection result as configuration input.
This keeps inspection independent of provider-specific destination derivation
and execution selection. It also avoids a second place that must reproduce
authentication-adapter rules. Construction-time defaults already captured in
the backend, such as queue capacity, still appear as effective values.
**Alternative:** Also expose derived reachability and authentication URLs. This
would help applications display the destination before invoking a diagnostic,
but would require sharing the adapter's derivation logic and clearly separating
declared settings from derived results. It must not imply that the selected
client supports HTTP diagnostics or that the destination is reachable. Add this
only if consumers need that extra information.
### 2. How Should An Unknown Backend Be Classified?
### 1. How Should An Unknown Backend Be Classified?
**Recommendation:** Add public `ErrBackendNotFound` for this explicit lookup,
matching the role of `ErrProfileNotFound` and `ErrPromptNotFound`. The registry
@@ -198,6 +304,28 @@ invalid selection. Consumers needing different presentation would then require
an inventory lookup or additional error detail. A sentinel is the smaller
consumer-facing solution when that distinction matters.
### 2. Where Should Effective Engine Execution Policy Be Exposed?
Resource inspection and `PreparedRun` do not currently provide a complete typed
view of the built-in transport cap and retry policy. These can originate in
Go engine options, not just the composed configuration inspected by ConfigKit.
**Recommendation:** Extend the existing prepared-result view with a small,
separately named execution-policy section for facts applicable to that request.
Report the effective built-in retry setting and transport cap, distinguishing
injected-client ownership where Promptkit cannot supply those facts. Keep model
settings, backend capacity, and caller context constraints distinct. Reuse the
same policy selection as execution, including request overrides; do not add a
second preview resolver or claim to know an injected client's internal policy.
This makes the concrete preview useful without another top-level inspection API.
**Alternative:** Add a separate engine-settings inspection API for captured
engine defaults, then let CommandKit combine that with resource inspection and
request-specific preparation. This also serves operators who have no concrete
request, but adds a public surface and requires a clear distinction between
engine defaults and request overrides. Select the surface before implementation;
neither option authorizes inspection of arbitrary client internals.
### 3. Should Typed Prompt Construction Be Added In This Round?
**Recommendation:** Keep `PromptConfig` and `PromptSource` as a separate
@@ -218,36 +346,58 @@ validation path merely to provide the helper.
## Implementation Scope And Verification
1. Resolve backend inspection decisions 1–2 and define the result and GoDoc.
Decide separately whether to expand this round for typed prompt construction;
the recommended backend scope does not depend on that expansion.
2. Connect namespace-aware engine selection to the existing captured backend
lookup and project the result at the public facade. Introduce only the
private collaborators needed for this path; no new backend cache or loader.
3. Add focused public tests for root and namespaced identities, bundled and
caller-defined backends, effective capacity defaults, provider maps, optional
diagnostic settings, errors, cancellation, and independent mutable results.
Cover a missing smoke default profile and absent credential environment
variable to establish that inspection does not perform execution readiness
checks. Use controlled local fixtures; no live provider calls are needed.
4. Add `BackendConfig.Extends` and encode it through `BackendSource`. Verify
an inherited backend through real ConfigKit composition, including an
overridden field and a retained parent field. Reuse existing inheritance
coverage for generic missing-parent, cycle, and replacement semantics rather
than retesting ConfigKit's entire feature set.
5. Update the consumer guide and relevant internal documentation when the API
lands, with a short `ListBackends` followed by `InspectBackend` example.
Include typed backend inheritance guidance. Record the additive APIs and
selected error behavior in release notes.
1. Resolve the error and engine-policy decisions and define public result fields
and GoDoc. Typed prompt construction is a separate scope decision; it does
not block the selected inspection improvements.
2. Expand prompt inspection from captured metadata and message content. Add
presence information to profile and prepared results using the execution
path's existing resolved state. Preserve existing values and error behavior.
3. Connect namespace-aware backend inspection to captured lookup and add shared
diagnostic destination derivation. Keep the public facade responsible for
owned results; avoid a new backend cache, loader, or recursive inspector.
4. Implement the selected engine-policy reporting surface without repeating
policy resolution. Keep the concrete-request preview on existing preparation.
5. Add `BackendConfig.Extends` and encode it through `BackendSource`, leaving
composition with ConfigKit.
6. Update public GoDoc, consumer guidance, relevant internal documentation, and
release notes when implementation lands. Coordinate CommandKit changes for
the fuller reports and omission-aware formatting, without modifying its CLI
contract from this roadmap.
Reuse meaningful existing namespace, configuration, and ownership coverage;
avoid tests that merely mirror private projection helpers. Run checks appropriate
to the implementation under the [testing policy](../policy/testing.md).
Use focused public tests for these consequential behaviors:
Completion means consumers can inspect a backend without selecting a profile or
navigating raw configuration, and derive a backend through the typed source
helper without constructing a raw mapping. Existing execution, inspection,
diagnostic, and configuration-composition behavior remains intact. No separate
provider capability registry, live capacity monitor, or backend health API is
included. Typed prompt construction and listing return-type changes are not
completion requirements for the selected backend scope.
- Captured imported message content, description, session template, roles,
cache controls, and output contract appear without rendering, rereading files,
or requiring a resolvable default profile.
- Profile and prepared results identify omitted/suppressed controls as absent
and preserve explicit-zero and nonzero controls across inheritance and request
overrides. Details retain the same frozen settings and presence as their
prepared execution; no new distinction between reasons for omission is needed.
- Backend inspection handles root/scoped/qualified identities, bundled and
caller-defined records, capacity defaults, optional metadata, explicit and
derived diagnostic destinations, errors, and cancellation. Missing credential
environment values and missing smoke default profiles do not prevent lookup.
- Mutating returned templates, maps, diagnostic settings, or prepared details
cannot change captured state or another result. Inspection performs no
provider calls; preparation retains its established resource-I/O boundary.
- Engine-policy facts match the selected built-in or injected-client behavior
and applicable request overrides under the chosen reporting surface.
- Typed backend inheritance through real ConfigKit composition retains a parent
field and applies a child override. Reuse generic inheritance coverage rather
than retesting all of ConfigKit's composition rules.
Use controlled offline fixtures and reuse existing namespace, configuration,
ownership, and preparation coverage. CommandKit should own report-format tests
showing absent settings omitted and explicit zero preserved. Follow the
[testing policy](../policy/testing.md); avoid tests coupled to private projection
helpers or new recurring release gates.
Completion means an operator can inspect captured prompt content, effective
profile settings with accurate omission semantics, and backend settings with
effective diagnostic destinations, then use ordinary preparation to preview a
concrete request. The chosen engine-policy surface supplies the additional facts
needed to describe execution policy without inventing provider or client defaults.
Typed backend inheritance is available through the source helper. Existing
composition and execution semantics remain intact. Typed prompt construction,
listing return-type changes, provider health guarantees, and live capacity
monitoring are not completion requirements.