From a43e0f17eea5df1727942a380241458ca48f8a1b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 6 Nov 2026 02:49:13 +0000 Subject: [PATCH] Align inspection roadmap with operator reporting needs --- docs/roadmap/inspect-backend.md | 310 +++++++++++++++++++++++--------- 1 file changed, 230 insertions(+), 80 deletions(-) diff --git a/docs/roadmap/inspect-backend.md b/docs/roadmap/inspect-backend.md index 53fc336..89ba8eb 100644 --- a/docs/roadmap/inspect-backend.md +++ b/docs/roadmap/inspect-backend.md @@ -1,25 +1,51 @@ -# Backend Inspection And Configuration API Roadmap +# Operator-Facing Resource Inspection And Configuration API Roadmap ## Purpose And Status -Add `Engine.InspectBackend` so applications can retrieve one backend's effective -definition by its runtime identity, and expose existing backend inheritance -through `BackendConfig` and `BackendSource`. These additions are selected for -planning; implementation has not begun. The open decisions below remain to be -resolved. Typed prompt construction is a separate discussion item, and listing -return-type consistency is a possible later cleanup. +Give operators using CommandKit-based applications a complete, accurate view of +Promptkit's effective prompt, profile, and backend configuration. CommandKit is +the primary consumer of these inspection APIs and owns operator-facing commands +and formatting. Promptkit owns resolved typed facts and their semantics. -The consumer benefit is backend-detail reporting without selecting an arbitrary -profile or reconstructing domain settings from ConfigKit paths and list indexes. -For example, Scriptorium could show a backend's endpoint, credential source, -capacity settings, provider defaults, and diagnostic configuration. Promptkit -owns this reusable interpretation; downstream applications own presentation, -disclosure, CLI commands, and exit codes. +The selected direction is accepted for planning; implementation has not begun. +Expand `InspectPrompt`, expose setting presence in profile and prepared results, +and add `InspectBackend` with effective diagnostic destinations. Retain the +planned `BackendConfig.Extends` convenience addition. Remaining decisions are +listed below; typed prompt construction and listing return-type changes are +lower-priority considerations, not prerequisites for operator inspection. -API symmetry alone does not justify additions. Prioritize backend inspection -with the recommended missing-backend error, followed by typed backend -inheritance. Both expose behavior or information Promptkit already owns, with -no new source composition, validation framework, or execution lifecycle. +This use case supersedes the earlier recommendation to keep prompt inspection +limited to interface metadata and backend diagnostics limited to declared +settings. Operators need to see captured message content and the destinations +that default diagnostic rules select. The implementation should reuse existing +capture, resolution, and adapter logic rather than build a second resolver. + +## What Fully Resolved Means + +| View | What can be resolved | +| --- | --- | +| Resource inspection | Captured values after composition, inheritance, normalization, and applicable Promptkit defaults. | +| Concrete request preparation | Selected profile, request overrides, rendered messages, inputs, session ID, and loaded schema for supplied request data. | +| Execution | Provider responses, actual usage, and other observed runtime facts. | + +A prompt ID alone cannot determine future request overrides or template values. +Resource inspection therefore shows effective templates and references, while +preparation shows the concrete request. Neither should invent provider defaults, +resolve a moving model alias to a hypothetical future model, or claim execution +success from configuration alone. + +**Reporting decision:** Omit unset optional settings from operator reports. +Retain explicitly configured values, including zero, and effective Promptkit +configuration defaults. Do not print omitted controls as zero or as +`unspecified`. Suppressed provider controls should likewise be absent from an +effective-settings report. This is a presentation of execution behavior; +`InspectConfiguration` remains available when an operator wants the authored +null, omission, or inheritance history. + +Promptkit must expose sufficient typed presence information for CommandKit to +make those choices correctly. A zero-valued Go field alone is not sufficient. +This does not remove fields from existing public types or silently change their +existing JSON representation; CommandKit owns its report projection. ## Relationship To Existing APIs @@ -27,7 +53,9 @@ no new source composition, validation framework, or execution lifecycle. | --- | --- | | `ListBackends` | Discover captured backend identities. | | Proposed `InspectBackend` | Describe one backend's effective settings by ID. | -| `InspectProfile` | Resolve a profile's backend-completed execution settings. | +| Expanded `InspectPrompt` | Describe the captured prompt, including effective template content. | +| Enhanced `InspectProfile` | Report backend-completed settings with the presence information needed for accurate output. | +| `Prepare` / `PreparedExecution.Details` | Describe a concrete prepared request, including setting presence. | | `InspectConfiguration` | Inspect effective configuration and source provenance. | | `SmokeTest` | Execute an explicitly requested diagnostic and report evidence. | @@ -35,6 +63,52 @@ See the [consumer guide](../consumers/pkg-promptkit.md#discover-available-resour for the implemented inspection and discovery boundaries, and the [architecture policy](../policy/architecture.md) for library responsibilities. +## Complete Prompt Inspection + +Expand `PromptInspection` to include description, ordered message templates +with roles and cache-control metadata, and the session-ID template. Retain +identity, opaque prompt hash, default profile reference, declared inputs, and +the complete output contract, including schema reference and repair settings. + +For messages imported from text or Markdown files, return the effective content +captured at engine construction. Operators should not have to locate or reopen +the original file to see what execution will render. Return templates as text, +not internal parsed-template objects. Inspection neither renders templates nor +rereads content files; edits require a new engine. The schema remains a +reference here; concrete preparation loads the selected schema. + +Keep prompt inspection independent of profile selection and request readiness. +An absent or incomplete default profile must not hide an otherwise captured +prompt. A prompt without a default profile remains inspectable. Return owned +mutable results, preserving message and input order and request isolation. + +## Presence-Aware Profile And Prepared Inspection + +`ProfileInspection` currently exposes effective model settings but not the +`ExecutionTargetPresence` information that distinguishes omitted controls from +explicit zero values. `PreparedRun` has the same reporting gap. Expose the +existing presence semantics alongside the effective target in both results, +including the `PreparedRun` returned through `PreparedExecution.Details`. + +Use the resolved target and presence captured by the execution path; do not +infer presence from numeric values or re-run ConfigKit composition. Preserve +backend completion, profile inheritance, explicit suppression, and request +precedence. Keep credential requirement metadata distinct from the credential +source name; no credential value is returned or checked by resource inspection. + +For example, the report for a profile that leaves temperature unset should have +no temperature entry. A profile explicitly setting temperature to zero should +show `temperature: 0`. A concrete request that overrides that profile should +show its effective setting instead. Presence metadata is for accurate report +construction; CommandKit need not expose an implementation-oriented bit map to +operators. + +Keep the meaning of effective defaults explicit. A configured zero generation +timeout disables that particular deadline; it does not disable the transport +cap or caller cancellation. An omitted provider control supplies no assertion +about the provider's own default. See the +[defaults reference](../formats.md#defaults-and-overrides). + ## Proposed Backend Inspection Contract ```go @@ -59,11 +133,12 @@ The proposed result includes: | `ExtraParams` | Backend provider defaults, before profile or request overrides. | | `ConcurrencyLimit` | Configured generation concurrency limit. | | `QueueCapacity` | Effective queue capacity, including construction-time defaults. | -| `SmokeTests` | Optional diagnostic settings, including the inference default profile reference. | +| `SmokeTests` | Diagnostic settings and effective destinations, including the authentication kind and optional inference default profile reference. | Scalar fields describe effective values rather than reproducing source-field -presence. Diagnostic optionality must remain expressible; its representation -depends on open decision 1. Exact nested type names are implementation choices. +presence. Diagnostic optionality must remain expressible, including no configured +authentication adapter or inference default. Exact nested type names are +implementation choices. Follow existing typed inspection conventions without adding a stable JSON serialization contract for this feature. Applications can define their own output formats. @@ -80,7 +155,28 @@ case sensitivity, rather than introducing new normalization. A nil or zero-value engine should match `ErrInvalidConfig`; a blank or otherwise invalid selection should match `ErrInvalidRequest`. Cancellation should retain the context error, and failures should return no partial result. The public -classification for a missing backend is open decision 2. +classification for a missing backend is open decision 1. + +### Effective Diagnostic Destinations + +Include the effective reachability destination and, when an authentication +adapter applies, its effective destination. Apply explicit endpoint overrides +and existing backend-relative defaults through the same derivation logic used +by diagnostic execution. CommandKit must not reconstruct provider-specific URL +rules. Include the configured authentication kind and inference default profile +reference; an inapplicable authentication kind has no active probe destination. + +Default reachability can apply even without a `smoke_tests` block. The result +must be able to report that effective destination without fabricating an authored +block or an authentication configuration. Where distinguishing an explicit +endpoint override from a derived destination is useful, keep those facts +separate rather than overloading a field's meaning. Source history remains +owned by `InspectConfiguration`. + +These are destinations under the backend's captured settings, not guarantees +about runtime overrides, the selected client's HTTP diagnostic support, or +reachability. Report a default inference profile reference without resolving it; +a missing or incomplete referenced profile must not make backend inspection fail. ### Captured State And Ownership @@ -107,6 +203,38 @@ credentials or silently substitute ConfigKit's redacted reporting values for the typed settings. Operator provenance and redacted configuration reporting remain available through `InspectConfiguration`. +## Connected Reports And Concrete Request Preview + +CommandKit can assemble a connected report by following prompt default-profile +and selected-backend identities through the three inspection APIs. Promptkit +supplies the references and effective values; CommandKit chooses nested sections +or links. Use captured references exactly, without requalifying them through a +view's caller-selection namespace. Endpoint-only profiles need no backend node. +A missing related resource can be reported while preserving successfully +inspected resources; do not make each resource inspection recursively dependent +on all related objects. + +Use `Prepare` for an operator preview with concrete variables, inputs, profile +selection, and request overrides. It performs artifact/schema reads, rendering, +and applicable credential checks without inference. Do not describe it as a +pure configuration lookup. Use `PrepareExecution` and its existing details when +later execution must use the same captured preparation; a separate preview +followed by an independent run is not a promise that mutable inputs stay equal. +No second request-resolution API is selected by this roadmap. + +An execution report must distinguish per-generation timeout, built-in transport +cap, retry policy, and backend capacity. Effective model parameters alone are +not the entire execution policy. Caller deadlines and injected-client behavior +can add constraints that Promptkit cannot predict for a future invocation. +How to expose the missing engine-policy facts is open decision 2. Do not infer +them from model fields or ask CommandKit to duplicate Promptkit defaults. + +CommandKit owns text/YAML presentation and report disclosure. Prompt templates, +input content, and arbitrary provider parameters may be sensitive; preserve +existing credential exclusions without claiming that every inspected value is +safe to publish. No CommandKit implementation is included in this repository's +roadmap; coordinate its consumer adaptation when the Promptkit APIs land. + ## Typed Backend Inheritance Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`. @@ -136,12 +264,10 @@ membership, namespace selection, sibling inheritance, and construction-time definition validation. The gaps addressed here are in public convenience and inspection APIs, not in those underlying configuration capabilities. -Inspection remains purpose-specific: `InspectPrompt` reports prompt interface -metadata, `InspectProfile` reports backend-completed execution settings, and -`InspectBackend` will describe an effective backend. This work does not turn -them into uniform complete-definition retrieval APIs. Expanding prompt inspection -to return descriptions, message templates, or session templates should follow a -concrete consumer need rather than the new backend result's shape. +Inspection remains purpose-specific while becoming complete for operator use: +prompt templates are captured but unrendered, profile settings include backend +completion, and backend settings include effective diagnostic destinations. +Provenance and authored representation stay with `InspectConfiguration`. `ListPrompts` currently returns `[]PromptIdentity`, whose only field is `ID`; `ListProfiles` and `ListBackends` return `[]string`. At a suitable breaking-change @@ -160,27 +286,7 @@ owns that separate question. ## Open Decisions -### 1. Declared Diagnostic Settings Or Derived Destinations? - -**Recommendation:** Return normalized captured diagnostic settings, preserving -omitted endpoint overrides and the default profile reference. An absent override -continues to mean that the diagnostic applies its existing default when run. -Use dedicated result types where needed to express this without treating the -whole inspection result as configuration input. - -This keeps inspection independent of provider-specific destination derivation -and execution selection. It also avoids a second place that must reproduce -authentication-adapter rules. Construction-time defaults already captured in -the backend, such as queue capacity, still appear as effective values. - -**Alternative:** Also expose derived reachability and authentication URLs. This -would help applications display the destination before invoking a diagnostic, -but would require sharing the adapter's derivation logic and clearly separating -declared settings from derived results. It must not imply that the selected -client supports HTTP diagnostics or that the destination is reachable. Add this -only if consumers need that extra information. - -### 2. How Should An Unknown Backend Be Classified? +### 1. How Should An Unknown Backend Be Classified? **Recommendation:** Add public `ErrBackendNotFound` for this explicit lookup, matching the role of `ErrProfileNotFound` and `ErrPromptNotFound`. The registry @@ -198,6 +304,28 @@ invalid selection. Consumers needing different presentation would then require an inventory lookup or additional error detail. A sentinel is the smaller consumer-facing solution when that distinction matters. +### 2. Where Should Effective Engine Execution Policy Be Exposed? + +Resource inspection and `PreparedRun` do not currently provide a complete typed +view of the built-in transport cap and retry policy. These can originate in +Go engine options, not just the composed configuration inspected by ConfigKit. + +**Recommendation:** Extend the existing prepared-result view with a small, +separately named execution-policy section for facts applicable to that request. +Report the effective built-in retry setting and transport cap, distinguishing +injected-client ownership where Promptkit cannot supply those facts. Keep model +settings, backend capacity, and caller context constraints distinct. Reuse the +same policy selection as execution, including request overrides; do not add a +second preview resolver or claim to know an injected client's internal policy. +This makes the concrete preview useful without another top-level inspection API. + +**Alternative:** Add a separate engine-settings inspection API for captured +engine defaults, then let CommandKit combine that with resource inspection and +request-specific preparation. This also serves operators who have no concrete +request, but adds a public surface and requires a clear distinction between +engine defaults and request overrides. Select the surface before implementation; +neither option authorizes inspection of arbitrary client internals. + ### 3. Should Typed Prompt Construction Be Added In This Round? **Recommendation:** Keep `PromptConfig` and `PromptSource` as a separate @@ -218,36 +346,58 @@ validation path merely to provide the helper. ## Implementation Scope And Verification -1. Resolve backend inspection decisions 1–2 and define the result and GoDoc. - Decide separately whether to expand this round for typed prompt construction; - the recommended backend scope does not depend on that expansion. -2. Connect namespace-aware engine selection to the existing captured backend - lookup and project the result at the public facade. Introduce only the - private collaborators needed for this path; no new backend cache or loader. -3. Add focused public tests for root and namespaced identities, bundled and - caller-defined backends, effective capacity defaults, provider maps, optional - diagnostic settings, errors, cancellation, and independent mutable results. - Cover a missing smoke default profile and absent credential environment - variable to establish that inspection does not perform execution readiness - checks. Use controlled local fixtures; no live provider calls are needed. -4. Add `BackendConfig.Extends` and encode it through `BackendSource`. Verify - an inherited backend through real ConfigKit composition, including an - overridden field and a retained parent field. Reuse existing inheritance - coverage for generic missing-parent, cycle, and replacement semantics rather - than retesting ConfigKit's entire feature set. -5. Update the consumer guide and relevant internal documentation when the API - lands, with a short `ListBackends` followed by `InspectBackend` example. - Include typed backend inheritance guidance. Record the additive APIs and - selected error behavior in release notes. +1. Resolve the error and engine-policy decisions and define public result fields + and GoDoc. Typed prompt construction is a separate scope decision; it does + not block the selected inspection improvements. +2. Expand prompt inspection from captured metadata and message content. Add + presence information to profile and prepared results using the execution + path's existing resolved state. Preserve existing values and error behavior. +3. Connect namespace-aware backend inspection to captured lookup and add shared + diagnostic destination derivation. Keep the public facade responsible for + owned results; avoid a new backend cache, loader, or recursive inspector. +4. Implement the selected engine-policy reporting surface without repeating + policy resolution. Keep the concrete-request preview on existing preparation. +5. Add `BackendConfig.Extends` and encode it through `BackendSource`, leaving + composition with ConfigKit. +6. Update public GoDoc, consumer guidance, relevant internal documentation, and + release notes when implementation lands. Coordinate CommandKit changes for + the fuller reports and omission-aware formatting, without modifying its CLI + contract from this roadmap. -Reuse meaningful existing namespace, configuration, and ownership coverage; -avoid tests that merely mirror private projection helpers. Run checks appropriate -to the implementation under the [testing policy](../policy/testing.md). +Use focused public tests for these consequential behaviors: -Completion means consumers can inspect a backend without selecting a profile or -navigating raw configuration, and derive a backend through the typed source -helper without constructing a raw mapping. Existing execution, inspection, -diagnostic, and configuration-composition behavior remains intact. No separate -provider capability registry, live capacity monitor, or backend health API is -included. Typed prompt construction and listing return-type changes are not -completion requirements for the selected backend scope. +- Captured imported message content, description, session template, roles, + cache controls, and output contract appear without rendering, rereading files, + or requiring a resolvable default profile. +- Profile and prepared results identify omitted/suppressed controls as absent + and preserve explicit-zero and nonzero controls across inheritance and request + overrides. Details retain the same frozen settings and presence as their + prepared execution; no new distinction between reasons for omission is needed. +- Backend inspection handles root/scoped/qualified identities, bundled and + caller-defined records, capacity defaults, optional metadata, explicit and + derived diagnostic destinations, errors, and cancellation. Missing credential + environment values and missing smoke default profiles do not prevent lookup. +- Mutating returned templates, maps, diagnostic settings, or prepared details + cannot change captured state or another result. Inspection performs no + provider calls; preparation retains its established resource-I/O boundary. +- Engine-policy facts match the selected built-in or injected-client behavior + and applicable request overrides under the chosen reporting surface. +- Typed backend inheritance through real ConfigKit composition retains a parent + field and applies a child override. Reuse generic inheritance coverage rather + than retesting all of ConfigKit's composition rules. + +Use controlled offline fixtures and reuse existing namespace, configuration, +ownership, and preparation coverage. CommandKit should own report-format tests +showing absent settings omitted and explicit zero preserved. Follow the +[testing policy](../policy/testing.md); avoid tests coupled to private projection +helpers or new recurring release gates. + +Completion means an operator can inspect captured prompt content, effective +profile settings with accurate omission semantics, and backend settings with +effective diagnostic destinations, then use ordinary preparation to preview a +concrete request. The chosen engine-policy surface supplies the additional facts +needed to describe execution policy without inventing provider or client defaults. +Typed backend inheritance is available through the source helper. Existing +composition and execution semantics remain intact. Typed prompt construction, +listing return-type changes, provider health guarantees, and live capacity +monitoring are not completion requirements.