From d7007eb072b0fe4dd1e06811c449fe39c9e07d1b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 6 Nov 2026 02:34:38 +0000 Subject: [PATCH] Plan backend inspection API --- docs/roadmap/inspect-backend.md | 169 ++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 docs/roadmap/inspect-backend.md diff --git a/docs/roadmap/inspect-backend.md b/docs/roadmap/inspect-backend.md new file mode 100644 index 0000000..0637e63 --- /dev/null +++ b/docs/roadmap/inspect-backend.md @@ -0,0 +1,169 @@ +# Backend Inspection Roadmap + +## Purpose And Status + +Add `Engine.InspectBackend` so applications can retrieve one backend's effective +definition by its runtime identity. The direction is accepted for planning; +implementation has not begun. The open decisions below remain to be resolved. + +The consumer benefit is backend-detail reporting without selecting an arbitrary +profile or reconstructing domain settings from ConfigKit paths and list indexes. +For example, Scriptorium could show a backend's endpoint, credential source, +capacity settings, provider defaults, and diagnostic configuration. Promptkit +owns this reusable interpretation; downstream applications own presentation, +disclosure, CLI commands, and exit codes. + +API symmetry alone does not justify the addition. The scope is a small typed +view over captured information that Promptkit already owns, with no new source +composition, validation framework, or execution lifecycle. + +## Relationship To Existing APIs + +| API | Responsibility | +| --- | --- | +| `ListBackends` | Discover captured backend identities. | +| Proposed `InspectBackend` | Describe one backend's effective settings by ID. | +| `InspectProfile` | Resolve a profile's backend-completed execution settings. | +| `InspectConfiguration` | Inspect effective configuration and source provenance. | +| `SmokeTest` | Execute an explicitly requested diagnostic and report evidence. | + +See the [consumer guide](../consumers/pkg-promptkit.md#discover-available-resources-and-collections) +for the implemented inspection and discovery boundaries, and the +[architecture policy](../policy/architecture.md) for library responsibilities. + +## Proposed Public Contract + +```go +func (e *Engine) InspectBackend( + ctx context.Context, + backendID string, +) (*BackendInspection, error) +``` + +Use a dedicated public result rather than returning `BackendConfig`. +`BackendConfig` is an input type whose pointers express omitted and explicit +values; inspection describes the normalized effective definition after +composition and defaulting. Do not expose internal domain types. + +The proposed result includes: + +| Information | Meaning | +| --- | --- | +| `BackendID` | Full selected runtime identity. | +| `Endpoint` | Captured backend base endpoint. | +| `APIKeyEnv` | Configured environment-variable name, never its resolved value. | +| `ExtraParams` | Backend provider defaults, before profile or request overrides. | +| `ConcurrencyLimit` | Configured generation concurrency limit. | +| `QueueCapacity` | Effective queue capacity, including construction-time defaults. | +| `SmokeTests` | Optional diagnostic settings, including the inference default profile reference. | + +Scalar fields describe effective values rather than reproducing source-field +presence. Diagnostic optionality must remain expressible; its representation +depends on open decision 1. Exact nested type names are implementation choices. +Follow existing typed inspection conventions without adding a stable JSON +serialization contract for this feature. Applications can define their own +output formats. + +### Selection And Errors + +Follow the existing namespace selection rules: a scoped engine view qualifies +slashless IDs with its backend namespace; qualified IDs pass through unchanged. +An unscoped view looks up the supplied ID directly. Return the full selected ID. +Do not add fallback search across namespaces, implicit backend selection, or +special handling for bundled backends. Use the existing ID rules, including +case sensitivity, rather than introducing new normalization. + +A nil or zero-value engine should match `ErrInvalidConfig`; a blank or otherwise +invalid selection should match `ErrInvalidRequest`. Cancellation should retain +the context error, and failures should return no partial result. The public +classification for a missing backend is open decision 2. + +### Captured State And Ownership + +Inspect the engine's captured backend definition without reloading sources, +reading credential values, preparing a prompt, acquiring capacity, or making +provider calls. Engine construction already validates effective definitions; +inspection should not repeat that work or create an additional acceptance path. + +Capacity fields describe configuration, not current occupancy or queue length. +Diagnostic settings describe configuration, not provider availability, key +acceptance, or inference readiness. In particular, report a smoke-test default +profile reference without resolving or validating that profile merely to inspect +the backend. Its selection-time behavior remains owned by `SmokeTest`. + +Return independently owned mutable values, including nested provider maps and +diagnostic settings. Mutating a result must not affect the engine or another +inspection. Reuse existing ownership operations and avoid consecutive copies +whose only purpose is to cross private helper boundaries. + +Provider defaults should retain their values, consistent with typed target +inspection. They are application data and may contain sensitive content; +the result is not automatically safe for logging or disclosure. Do not resolve +credentials or silently substitute ConfigKit's redacted reporting values for +the typed settings. Operator provenance and redacted configuration reporting +remain available through `InspectConfiguration`. + +## Open Decisions + +### 1. Declared Diagnostic Settings Or Derived Destinations? + +**Recommendation:** Return normalized captured diagnostic settings, preserving +omitted endpoint overrides and the default profile reference. An absent override +continues to mean that the diagnostic applies its existing default when run. +Use dedicated result types where needed to express this without treating the +whole inspection result as configuration input. + +This keeps inspection independent of provider-specific destination derivation +and execution selection. It also avoids a second place that must reproduce +authentication-adapter rules. Construction-time defaults already captured in +the backend, such as queue capacity, still appear as effective values. + +**Alternative:** Also expose derived reachability and authentication URLs. This +would help applications display the destination before invoking a diagnostic, +but would require sharing the adapter's derivation logic and clearly separating +declared settings from derived results. It must not imply that the selected +client supports HTTP diagnostics or that the destination is reachable. Add this +only if consumers need that extra information. + +### 2. How Should An Unknown Backend Be Classified? + +**Recommendation:** Add public `ErrBackendNotFound` for this explicit lookup, +matching the role of `ErrProfileNotFound` and `ErrPromptNotFound`. The registry +already has an internal backend-not-found sentinel. Map it at the facade so +applications can distinguish a missing backend from malformed input without +inspecting text or internal packages. + +Limit this addition to the new inspection operation unless existing callers +justify a broader change; do not automatically reclassify smoke-test or engine +construction errors. + +**Alternative:** Classify an absent ID as `ErrInvalidRequest`. This avoids +another exported sentinel but combines a well-formed missing identity with an +invalid selection. Consumers needing different presentation would then require +an inventory lookup or additional error detail. A sentinel is the smaller +consumer-facing solution when that distinction matters. + +## Implementation Scope And Verification + +1. Resolve the two public-contract decisions and define the result and GoDoc. +2. Connect namespace-aware engine selection to the existing captured backend + lookup and project the result at the public facade. Introduce only the + private collaborators needed for this path; no new backend cache or loader. +3. Add focused public tests for root and namespaced identities, bundled and + caller-defined backends, effective capacity defaults, provider maps, optional + diagnostic settings, errors, cancellation, and independent mutable results. + Cover a missing smoke default profile and absent credential environment + variable to establish that inspection does not perform execution readiness + checks. Use controlled local fixtures; no live provider calls are needed. +4. Update the consumer guide and relevant internal documentation when the API + lands, with a short `ListBackends` followed by `InspectBackend` example. + Record the additive API and selected error behavior in release notes. + +Reuse meaningful existing namespace, configuration, and ownership coverage; +avoid tests that merely mirror private projection helpers. Run checks appropriate +to the implementation under the [testing policy](../policy/testing.md). + +Completion means consumers can inspect a backend without selecting a profile or +navigating raw configuration, while existing execution, inspection, diagnostic, +and configuration-composition behavior remains intact. No separate provider +capability registry, live capacity monitor, or backend health API is included.