Keep inspection roadmap contracts proportionate
This commit is contained in:
@@ -2,114 +2,132 @@
|
||||
|
||||
## Purpose And Status
|
||||
|
||||
Give operators using CommandKit-based applications a complete, accurate view of
|
||||
Promptkit's effective prompt, profile, and backend configuration. CommandKit is
|
||||
the primary consumer of these inspection APIs and owns operator-facing commands
|
||||
and formatting. Promptkit owns resolved typed facts and their semantics.
|
||||
Help operators using CommandKit-based applications understand Promptkit's
|
||||
captured prompt, profile, and backend configuration and the settings applicable
|
||||
to a concrete request. CommandKit owns operator-facing commands and presentation;
|
||||
Promptkit supplies facts from its existing configuration and execution paths.
|
||||
|
||||
The selected direction is accepted for planning; implementation has not begun.
|
||||
Expand `InspectPrompt`, expose setting presence in profile and prepared results,
|
||||
and add `InspectBackend` with effective diagnostic destinations. Retain the
|
||||
planned `BackendConfig.Extends` convenience addition. Remaining decisions are
|
||||
listed below; typed prompt construction and listing return-type changes are
|
||||
lower-priority considerations, not prerequisites for operator inspection.
|
||||
The direction is accepted for planning; implementation has not begun. Selected
|
||||
work includes expanded prompt inspection, presence-aware profile/prepared
|
||||
inspection, backend inspection with effective diagnostic destinations, and
|
||||
`BackendConfig.Extends`. Open decisions below cover unresolved scope and public
|
||||
representation choices. They are not implicit implementation requirements.
|
||||
|
||||
This use case supersedes the earlier recommendation to keep prompt inspection
|
||||
limited to interface metadata and backend diagnostics limited to declared
|
||||
settings. Operators need to see captured message content and the destinations
|
||||
that default diagnostic rules select. The implementation should reuse existing
|
||||
capture, resolution, and adapter logic rather than build a second resolver.
|
||||
## Design Balance
|
||||
|
||||
## What Fully Resolved Means
|
||||
Provide useful information already captured or cheaply derived. Prefer exposing
|
||||
existing values and small shared helpers over introducing new resolution paths,
|
||||
parallel report models, provenance tracking, or retained snapshots solely for
|
||||
inspection. This is a design preference, not a prohibition on a small adapter
|
||||
or additional field when it serves a demonstrated consumer need.
|
||||
|
||||
| View | What can be resolved |
|
||||
“Fully resolved” describes the applicable resolution stage. It is not a promise
|
||||
that inspection can reconstruct the entire authored source, report every future
|
||||
configuration field automatically, predict provider defaults, or reproduce a
|
||||
future invocation independently of its request and external state.
|
||||
|
||||
| View | Information available |
|
||||
| --- | --- |
|
||||
| Resource inspection | Captured values after composition, inheritance, normalization, and applicable Promptkit defaults. |
|
||||
| Concrete request preparation | Selected profile, request overrides, rendered messages, inputs, session ID, and loaded schema for supplied request data. |
|
||||
| Concrete request preparation | Selected profile, request overrides, rendered messages, input metadata, session ID, and loaded schema for supplied request data. |
|
||||
| Execution | Provider responses, actual usage, and other observed runtime facts. |
|
||||
|
||||
A prompt ID alone cannot determine future request overrides or template values.
|
||||
Resource inspection therefore shows effective templates and references, while
|
||||
preparation shows the concrete request. Neither should invent provider defaults,
|
||||
resolve a moving model alias to a hypothetical future model, or claim execution
|
||||
success from configuration alone.
|
||||
The planned fields below define the initial additions. Additional cheap,
|
||||
useful fields can be considered during implementation without building a
|
||||
uniform resource framework or promising exhaustive future coverage. If a field
|
||||
requires substantial new machinery, record the tradeoff before expanding scope.
|
||||
|
||||
**Reporting decision:** Omit unset optional settings from operator reports.
|
||||
Retain explicitly configured values, including zero, and effective Promptkit
|
||||
configuration defaults. Do not print omitted controls as zero or as
|
||||
`unspecified`. Suppressed provider controls should likewise be absent from an
|
||||
effective-settings report. This is a presentation of execution behavior;
|
||||
`InspectConfiguration` remains available when an operator wants the authored
|
||||
null, omission, or inheritance history.
|
||||
### Review Findings And Existing Boundaries
|
||||
|
||||
Promptkit must expose sufficient typed presence information for CommandKit to
|
||||
make those choices correctly. A zero-valued Go field alone is not sufficient.
|
||||
This does not remove fields from existing public types or silently change their
|
||||
existing JSON representation; CommandKit owns its report projection.
|
||||
The current implementation provides a relatively small path to the core work:
|
||||
|
||||
## Relationship To Existing APIs
|
||||
- Captured prompt definitions already retain description, session template,
|
||||
message text, roles, and cache-control metadata alongside compiled templates.
|
||||
Inspection can expose that data without reconstructing source files or ASTs.
|
||||
- Internal prepared results already retain `TargetPresence`; the root API has
|
||||
an `ExecutionTargetPresence` type. Presence-aware output need not introduce
|
||||
a second interpretation of configuration or provider-value conversion.
|
||||
- Smoke selection already has endpoint derivation, including the OpenRouter
|
||||
key URL helper. Reusing or factoring that logic need not create an adapter
|
||||
capability registry or provider metadata service.
|
||||
- Transport timeout and retry policy live in different components and are not
|
||||
all carried by prepared results. Their reporting scope and cost need a
|
||||
decision; they are no longer an unconditional completion requirement.
|
||||
|
||||
| API | Responsibility |
|
||||
| --- | --- |
|
||||
| `ListBackends` | Discover captured backend identities. |
|
||||
| Proposed `InspectBackend` | Describe one backend's effective settings by ID. |
|
||||
| Expanded `InspectPrompt` | Describe the captured prompt, including effective template content. |
|
||||
| Enhanced `InspectProfile` | Report backend-completed settings with the presence information needed for accurate output. |
|
||||
| `Prepare` / `PreparedExecution.Details` | Describe a concrete prepared request, including setting presence. |
|
||||
| `InspectConfiguration` | Inspect effective configuration and source provenance. |
|
||||
| `SmokeTest` | Execute an explicitly requested diagnostic and report evidence. |
|
||||
These observations are implementation evidence, not promises to retain those
|
||||
private types or helper arrangements. See the current
|
||||
[source/capture](../internal/sources.md), [runner](../internal/runner.md), and
|
||||
[model-client](../internal/llm.md) descriptions.
|
||||
|
||||
See the [consumer guide](../consumers/pkg-promptkit.md#discover-available-resources-and-collections)
|
||||
for the implemented inspection and discovery boundaries, and the
|
||||
[architecture policy](../policy/architecture.md) for library responsibilities.
|
||||
Retain established public ownership, credential handling, captured-state, and
|
||||
namespace behavior unless a change is deliberately selected. Independent
|
||||
mutable results prevent callers from changing engine state; they do not require
|
||||
fresh copies of immutable strings or repeated deep copies between private layers.
|
||||
Existing `Inspect*` APIs avoid provider work and request preparation. Extending
|
||||
their returned data should not silently change those behaviors. These boundaries
|
||||
follow the existing API and the selected operator use case, rather than a new
|
||||
requirement that all future inspection APIs have identical behavior.
|
||||
|
||||
## Complete Prompt Inspection
|
||||
## Reporting Semantics
|
||||
|
||||
**Selected presentation:** Omit unset optional settings from operator reports.
|
||||
Preserve explicit values, including zero, and applicable Promptkit defaults.
|
||||
Suppressed provider controls are also absent from an effective-settings report.
|
||||
Neither zero nor `unspecified` should stand in for an omitted control.
|
||||
|
||||
This concerns the effective setting, not an explanation of every reason it was
|
||||
omitted. ConfigKit inspection remains useful for authored values and history.
|
||||
Promptkit supplies sufficient presence information; CommandKit chooses report
|
||||
layout. Representation and serialization choices remain open in decision 4.
|
||||
|
||||
Do not infer provider defaults from absent controls. Likewise, a configured
|
||||
model alias is the alias, not a prediction of the model it will select later.
|
||||
Credential source names and requirements are useful inspection facts; resolved
|
||||
credential values remain excluded under existing contracts. Templates and
|
||||
arbitrary provider parameters are application data, not automatically safe
|
||||
for disclosure. No new general-purpose sanitization guarantee is selected.
|
||||
|
||||
## Expanded Prompt Inspection
|
||||
|
||||
Expand `PromptInspection` to include description, ordered message templates
|
||||
with roles and cache-control metadata, and the session-ID template. Retain
|
||||
identity, opaque prompt hash, default profile reference, declared inputs, and
|
||||
the complete output contract, including schema reference and repair settings.
|
||||
with roles and cache-control metadata, and the session-ID template. Retain the
|
||||
existing identity, opaque hash, default profile reference, input declarations,
|
||||
and output contract, including schema reference and repair settings.
|
||||
|
||||
For messages imported from text or Markdown files, return the effective content
|
||||
captured at engine construction. Operators should not have to locate or reopen
|
||||
the original file to see what execution will render. Return templates as text,
|
||||
not internal parsed-template objects. Inspection neither renders templates nor
|
||||
rereads content files; edits require a new engine. The schema remains a
|
||||
reference here; concrete preparation loads the selected schema.
|
||||
Expose effective content captured from imported text or Markdown files, as well
|
||||
as inline content. Return the template text available in the captured definition;
|
||||
source-layout reconstruction, original YAML formatting, and parsed-template
|
||||
objects are unnecessary for this use case. There is no new byte-for-byte source
|
||||
round-trip requirement. Message order and text should reflect what rendering
|
||||
consumes. Inspection continues to use captured state without rendering or
|
||||
reopening files; the schema stays a reference until ordinary preparation loads it.
|
||||
|
||||
Keep prompt inspection independent of profile selection and request readiness.
|
||||
An absent or incomplete default profile must not hide an otherwise captured
|
||||
prompt. A prompt without a default profile remains inspectable. Return owned
|
||||
mutable results, preserving message and input order and request isolation.
|
||||
Prompt inspection remains useful without a supplied request or a resolvable
|
||||
default profile. The existing independent lookup permits CommandKit to show a
|
||||
prompt even when related execution settings need attention.
|
||||
|
||||
## Presence-Aware Profile And Prepared Inspection
|
||||
|
||||
`ProfileInspection` currently exposes effective model settings but not the
|
||||
`ExecutionTargetPresence` information that distinguishes omitted controls from
|
||||
explicit zero values. `PreparedRun` has the same reporting gap. Expose the
|
||||
existing presence semantics alongside the effective target in both results,
|
||||
including the `PreparedRun` returned through `PreparedExecution.Details`.
|
||||
`ProfileInspection` exposes effective settings without the presence information
|
||||
that execution uses for explicit-zero controls. `PreparedRun` has the same
|
||||
public reporting gap, including values returned by `PreparedExecution.Details`.
|
||||
Expose enough existing resolved presence information to distinguish omission
|
||||
from explicit settings in each of those results.
|
||||
|
||||
Use the resolved target and presence captured by the execution path; do not
|
||||
infer presence from numeric values or re-run ConfigKit composition. Preserve
|
||||
backend completion, profile inheritance, explicit suppression, and request
|
||||
precedence. Keep credential requirement metadata distinct from the credential
|
||||
source name; no credential value is returned or checked by resource inspection.
|
||||
For example, an unset temperature produces no temperature entry in the report;
|
||||
an explicit zero produces `temperature: 0`. A concrete request's override appears
|
||||
in its prepared view. Backend completion, inheritance, and request precedence
|
||||
remain the execution path's responsibility. Inspectors need not reconstruct
|
||||
that history or distinguish all historical reasons for an absent setting.
|
||||
|
||||
For example, the report for a profile that leaves temperature unset should have
|
||||
no temperature entry. A profile explicitly setting temperature to zero should
|
||||
show `temperature: 0`. A concrete request that overrides that profile should
|
||||
show its effective setting instead. Presence metadata is for accurate report
|
||||
construction; CommandKit need not expose an implementation-oriented bit map to
|
||||
operators.
|
||||
Reuse the captured target and presence semantics, including existing handling
|
||||
of nonzero values. Decision 4 weighs exposing the current presence type against
|
||||
another public representation; it does not reopen the omission-reporting decision.
|
||||
See [defaults and overrides](../formats.md#defaults-and-overrides).
|
||||
|
||||
Keep the meaning of effective defaults explicit. A configured zero generation
|
||||
timeout disables that particular deadline; it does not disable the transport
|
||||
cap or caller cancellation. An omitted provider control supplies no assertion
|
||||
about the provider's own default. See the
|
||||
[defaults reference](../formats.md#defaults-and-overrides).
|
||||
## Backend Inspection
|
||||
|
||||
## Proposed Backend Inspection Contract
|
||||
Proposed entry point:
|
||||
|
||||
```go
|
||||
func (e *Engine) InspectBackend(
|
||||
@@ -118,286 +136,225 @@ func (e *Engine) InspectBackend(
|
||||
) (*BackendInspection, error)
|
||||
```
|
||||
|
||||
Use a dedicated public result rather than returning `BackendConfig`.
|
||||
`BackendConfig` is an input type whose pointers express omitted and explicit
|
||||
values; inspection describes the normalized effective definition after
|
||||
composition and defaulting. Do not expose internal domain types.
|
||||
The result should expose the full backend identity, base endpoint, credential
|
||||
environment-variable name, provider defaults, effective concurrency/queue
|
||||
settings, and diagnostic settings and destinations. These are backend defaults,
|
||||
not settings after a particular profile or request overrides them. Capacity
|
||||
values describe policy, not current occupancy.
|
||||
|
||||
The proposed result includes:
|
||||
A dedicated top-level inspection result is recommended because `BackendConfig`
|
||||
is a presence-aware source input. This does not require a new type for every
|
||||
nested value: reuse suitable public values where their semantics match. Exact
|
||||
field names and nesting can be selected during implementation within the public
|
||||
representation decisions below; internal types remain behind the facade.
|
||||
|
||||
| Information | Meaning |
|
||||
| --- | --- |
|
||||
| `BackendID` | Full selected runtime identity. |
|
||||
| `Endpoint` | Captured backend base endpoint. |
|
||||
| `APIKeyEnv` | Configured environment-variable name, never its resolved value. |
|
||||
| `ExtraParams` | Backend provider defaults, before profile or request overrides. |
|
||||
| `ConcurrencyLimit` | Configured generation concurrency limit. |
|
||||
| `QueueCapacity` | Effective queue capacity, including construction-time defaults. |
|
||||
| `SmokeTests` | Diagnostic settings and effective destinations, including the authentication kind and optional inference default profile reference. |
|
||||
|
||||
Scalar fields describe effective values rather than reproducing source-field
|
||||
presence. Diagnostic optionality must remain expressible, including no configured
|
||||
authentication adapter or inference default. Exact nested type names are
|
||||
implementation choices.
|
||||
Follow existing typed inspection conventions without adding a stable JSON
|
||||
serialization contract for this feature. Applications can define their own
|
||||
output formats.
|
||||
|
||||
### Selection And Errors
|
||||
|
||||
Follow the existing namespace selection rules: a scoped engine view qualifies
|
||||
slashless IDs with its backend namespace; qualified IDs pass through unchanged.
|
||||
An unscoped view looks up the supplied ID directly. Return the full selected ID.
|
||||
Do not add fallback search across namespaces, implicit backend selection, or
|
||||
special handling for bundled backends. Use the existing ID rules, including
|
||||
case sensitivity, rather than introducing new normalization.
|
||||
|
||||
A nil or zero-value engine should match `ErrInvalidConfig`; a blank or otherwise
|
||||
invalid selection should match `ErrInvalidRequest`. Cancellation should retain
|
||||
the context error, and failures should return no partial result. The public
|
||||
classification for a missing backend is open decision 1.
|
||||
Follow existing namespace-view selection: qualify slashless caller IDs with the
|
||||
backend scope, pass qualified IDs through, and report the selected full ID.
|
||||
Use the same identity rules for bundled and caller-supplied backends. The new
|
||||
operation should follow existing inspection conventions for invalid engines,
|
||||
invalid selections, and context cancellation, without adding cross-error
|
||||
precedence guarantees. Missing-backend classification is open decision 1.
|
||||
|
||||
### Effective Diagnostic Destinations
|
||||
|
||||
Include the effective reachability destination and, when an authentication
|
||||
adapter applies, its effective destination. Apply explicit endpoint overrides
|
||||
and existing backend-relative defaults through the same derivation logic used
|
||||
by diagnostic execution. CommandKit must not reconstruct provider-specific URL
|
||||
rules. Include the configured authentication kind and inference default profile
|
||||
reference; an inapplicable authentication kind has no active probe destination.
|
||||
Include effective reachability and applicable authentication destinations,
|
||||
using explicit overrides or existing backend-relative defaults. Share the
|
||||
existing derivation logic where practical; neither CommandKit nor a new
|
||||
inspection-specific resolver should maintain a competing set of provider rules.
|
||||
A small helper extraction is sufficient if that is all the current adapters need.
|
||||
|
||||
Default reachability can apply even without a `smoke_tests` block. The result
|
||||
must be able to report that effective destination without fabricating an authored
|
||||
block or an authentication configuration. Where distinguishing an explicit
|
||||
endpoint override from a derived destination is useful, keep those facts
|
||||
separate rather than overloading a field's meaning. Source history remains
|
||||
owned by `InspectConfiguration`.
|
||||
Default reachability applies even without authored `smoke_tests` metadata.
|
||||
Report the configured authentication kind and optional inference-default profile
|
||||
reference without fabricating authored settings. An authentication kind of
|
||||
`none` has no active authentication destination. Decision 5 covers whether the
|
||||
result also needs separate declared endpoint fields.
|
||||
|
||||
These are destinations under the backend's captured settings, not guarantees
|
||||
about runtime overrides, the selected client's HTTP diagnostic support, or
|
||||
reachability. Report a default inference profile reference without resolving it;
|
||||
a missing or incomplete referenced profile must not make backend inspection fail.
|
||||
Destinations describe captured backend settings, not provider reachability or
|
||||
HTTP diagnostic support in an injected client. Retain the inference-default
|
||||
profile as a reference; backend inspection need not resolve it or establish
|
||||
inference readiness. Ordinary inspection is expected to return a result or an
|
||||
error, following existing methods. If derivation reveals a meaningful case where
|
||||
useful captured settings would be lost, bring that case back for a decision
|
||||
rather than automatically adding partial-result and warning machinery.
|
||||
|
||||
### Captured State And Ownership
|
||||
## Connected Views And Concrete Requests
|
||||
|
||||
Inspect the engine's captured backend definition without reloading sources,
|
||||
reading credential values, preparing a prompt, acquiring capacity, or making
|
||||
provider calls. Engine construction already validates effective definitions;
|
||||
inspection should not repeat that work or create an additional acceptance path.
|
||||
CommandKit can follow prompt default-profile and selected-backend references
|
||||
through the existing inspections to build a connected view. This is a suggested
|
||||
consumer workflow, not a required report format or a restriction on future
|
||||
convenience APIs. Use stored references exactly; namespace defaults apply to
|
||||
caller selections, not to reinterpreting captured references. Endpoint-only
|
||||
profiles need no backend node. A consumer can retain successful lookups while
|
||||
showing a failure for a related resource.
|
||||
|
||||
Capacity fields describe configuration, not current occupancy or queue length.
|
||||
Diagnostic settings describe configuration, not provider availability, key
|
||||
acceptance, or inference readiness. In particular, report a smoke-test default
|
||||
profile reference without resolving or validating that profile merely to inspect
|
||||
the backend. Its selection-time behavior remains owned by `SmokeTest`.
|
||||
Use existing `Prepare` for concrete variables, inputs, profile selection, and
|
||||
request overrides. It performs artifact/schema reads, rendering, and applicable
|
||||
credential checks without inference. `PrepareExecution` and its details serve
|
||||
callers that want later execution tied to the same preparation, subject to that
|
||||
API's existing contract. This roadmap adds no stronger replay guarantee and
|
||||
selects no second preparation workflow.
|
||||
|
||||
Return independently owned mutable values, including nested provider maps and
|
||||
diagnostic settings. Mutating a result must not affect the engine or another
|
||||
inspection. Reuse existing ownership operations and avoid consecutive copies
|
||||
whose only purpose is to cross private helper boundaries.
|
||||
Model parameters are not all execution policy. Per-generation timeout, transport
|
||||
cap, retry settings, backend capacity, and caller deadlines are different facts.
|
||||
When reported, keep their meanings distinct. Decision 2 determines whether and
|
||||
how this round should expose additional engine-policy information. Report known
|
||||
settings without claiming knowledge of an injected client's internal behavior.
|
||||
|
||||
Provider defaults should retain their values, consistent with typed target
|
||||
inspection. They are application data and may contain sensitive content;
|
||||
the result is not automatically safe for logging or disclosure. Do not resolve
|
||||
credentials or silently substitute ConfigKit's redacted reporting values for
|
||||
the typed settings. Operator provenance and redacted configuration reporting
|
||||
remain available through `InspectConfiguration`.
|
||||
|
||||
## Connected Reports And Concrete Request Preview
|
||||
|
||||
CommandKit can assemble a connected report by following prompt default-profile
|
||||
and selected-backend identities through the three inspection APIs. Promptkit
|
||||
supplies the references and effective values; CommandKit chooses nested sections
|
||||
or links. Use captured references exactly, without requalifying them through a
|
||||
view's caller-selection namespace. Endpoint-only profiles need no backend node.
|
||||
A missing related resource can be reported while preserving successfully
|
||||
inspected resources; do not make each resource inspection recursively dependent
|
||||
on all related objects.
|
||||
|
||||
Use `Prepare` for an operator preview with concrete variables, inputs, profile
|
||||
selection, and request overrides. It performs artifact/schema reads, rendering,
|
||||
and applicable credential checks without inference. Do not describe it as a
|
||||
pure configuration lookup. Use `PrepareExecution` and its existing details when
|
||||
later execution must use the same captured preparation; a separate preview
|
||||
followed by an independent run is not a promise that mutable inputs stay equal.
|
||||
No second request-resolution API is selected by this roadmap.
|
||||
|
||||
An execution report must distinguish per-generation timeout, built-in transport
|
||||
cap, retry policy, and backend capacity. Effective model parameters alone are
|
||||
not the entire execution policy. Caller deadlines and injected-client behavior
|
||||
can add constraints that Promptkit cannot predict for a future invocation.
|
||||
How to expose the missing engine-policy facts is open decision 2. Do not infer
|
||||
them from model fields or ask CommandKit to duplicate Promptkit defaults.
|
||||
|
||||
CommandKit owns text/YAML presentation and report disclosure. Prompt templates,
|
||||
input content, and arbitrary provider parameters may be sensitive; preserve
|
||||
existing credential exclusions without claiming that every inspected value is
|
||||
safe to publish. No CommandKit implementation is included in this repository's
|
||||
roadmap; coordinate its consumer adaptation when the Promptkit APIs land.
|
||||
CommandKit owns CLI commands, formatting, and disclosure. Coordinate downstream
|
||||
adoption when the Promptkit additions land; this roadmap does not prescribe
|
||||
CommandKit's internal adapter design or require a new stable CLI serialization
|
||||
contract in Promptkit.
|
||||
|
||||
## Typed Backend Inheritance
|
||||
|
||||
Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
|
||||
When non-nil, `BackendSource` should encode it as `extends: {id: value}` in the
|
||||
backend record. Nil omits the field. Preserve the supplied value and let
|
||||
ConfigKit resolve and validate inheritance through the existing source-loading
|
||||
path, just as it does for profiles.
|
||||
`BackendSource` should encode a non-nil value as `extends: {id: value}` and omit
|
||||
the field when nil. ConfigKit owns inheritance resolution and its diagnostics.
|
||||
The helper exposes existing sibling inheritance, rather than performing parent
|
||||
lookup or filling inherited fields itself.
|
||||
|
||||
The purpose is to let programmatic callers derive a backend variant without
|
||||
switching from `BackendSource` to a raw mapping. The helper must not resolve a
|
||||
parent, fill inherited fields, or become a second inheritance implementation.
|
||||
It remains a source constructor rather than an engine registry mutation API.
|
||||
Keep existing root/collection identity, replacement, and null semantics. Raw
|
||||
mappings remain available for explicit null and other generic controls outside
|
||||
the helper's convenience surface. See the
|
||||
[inheritance reference](../formats.md#resource-inheritance) and
|
||||
[profile helper](../../profiles.go).
|
||||
|
||||
This exposes existing sibling inheritance, not a new configuration feature.
|
||||
Parent IDs select siblings in the same root or collection resource list;
|
||||
namespace views do not qualify these configuration references. Existing
|
||||
same-ID replacement, omitted-field, and explicit-null behavior remain owned by
|
||||
ConfigKit composition. Raw mappings remain available when callers need explicit
|
||||
null or other generic configuration controls that the typed helper cannot express.
|
||||
See the [inheritance reference](../formats.md#resource-inheritance) and the
|
||||
existing [profile source helper](../../profiles.go).
|
||||
## Related Work And Scope
|
||||
|
||||
## Scope Of API Consistency
|
||||
Typed prompt construction is open decision 3. Listing return types remain a
|
||||
possible later cleanup: `ListPrompts` returns an ID-only `PromptIdentity` wrapper,
|
||||
while backend/profile listings return strings. Standardizing on strings may be
|
||||
useful at a breaking-change window, but is not required for these inspections.
|
||||
|
||||
Backends, profiles, and prompts already share source layering, collection
|
||||
membership, namespace selection, sibling inheritance, and construction-time
|
||||
definition validation. The gaps addressed here are in public convenience and
|
||||
inspection APIs, not in those underlying configuration capabilities.
|
||||
|
||||
Inspection remains purpose-specific while becoming complete for operator use:
|
||||
prompt templates are captured but unrendered, profile settings include backend
|
||||
completion, and backend settings include effective diagnostic destinations.
|
||||
Provenance and authored representation stay with `InspectConfiguration`.
|
||||
|
||||
`ListPrompts` currently returns `[]PromptIdentity`, whose only field is `ID`;
|
||||
`ListProfiles` and `ListBackends` return `[]string`. At a suitable breaking-change
|
||||
window, prefer considering `[]string` for all three over adding thin identity
|
||||
wrappers for backends and profiles. That would simplify consumer code, but
|
||||
requires a deliberate compatibility decision and migration instructions. Keep
|
||||
the existing return types in this work; listing consistency does not block the
|
||||
backend additions.
|
||||
|
||||
Collections, schemas, and engine configuration remain distinct concepts rather
|
||||
than additional peers requiring identical resource APIs. Prompt requirement
|
||||
inspection and backend capacity/diagnostic settings retain their specialized
|
||||
roles. Profile/backend hashes are not added for symmetry; the existing
|
||||
Collections, schemas, and engine options have different roles; symmetry does
|
||||
not require identical APIs for them. The
|
||||
[deferred fingerprint proposal](deferred.md#semantic-execution-target-fingerprints)
|
||||
owns that separate question.
|
||||
remains separate. Live capacity monitoring, provider health guarantees, and
|
||||
source round-tripping are not selected deliverables, rather than permanent
|
||||
prohibitions on future work.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
### 1. How Should An Unknown Backend Be Classified?
|
||||
|
||||
**Recommendation:** Add public `ErrBackendNotFound` for this explicit lookup,
|
||||
matching the role of `ErrProfileNotFound` and `ErrPromptNotFound`. The registry
|
||||
already has an internal backend-not-found sentinel. Map it at the facade so
|
||||
applications can distinguish a missing backend from malformed input without
|
||||
inspecting text or internal packages.
|
||||
**Recommendation:** Add public `ErrBackendNotFound`, analogous to the existing
|
||||
prompt/profile sentinels. The registry already identifies missing backends;
|
||||
public mapping lets consumers distinguish a missing ID without parsing text.
|
||||
Keep existing operations' classifications unless a change is explicitly chosen.
|
||||
|
||||
Limit this addition to the new inspection operation unless existing callers
|
||||
justify a broader change; do not automatically reclassify smoke-test or engine
|
||||
construction errors.
|
||||
**Alternative:** Use `ErrInvalidRequest`, avoiding another exported sentinel but
|
||||
combining malformed selection and a well-formed missing identity. Consumers that
|
||||
need the distinction would require inventory lookup or other error detail.
|
||||
|
||||
**Alternative:** Classify an absent ID as `ErrInvalidRequest`. This avoids
|
||||
another exported sentinel but combines a well-formed missing identity with an
|
||||
invalid selection. Consumers needing different presentation would then require
|
||||
an inventory lookup or additional error detail. A sentinel is the smaller
|
||||
consumer-facing solution when that distinction matters.
|
||||
### 2. How Much Engine-Policy Reporting Belongs In This Round?
|
||||
|
||||
### 2. Where Should Effective Engine Execution Policy Be Exposed?
|
||||
The previous draft made a comprehensive execution-policy section a completion
|
||||
requirement before assessing the cost of carrying transport and retry facts
|
||||
through preparation. That commitment is now open, including the option to defer.
|
||||
|
||||
Resource inspection and `PreparedRun` do not currently provide a complete typed
|
||||
view of the built-in transport cap and retry policy. These can originate in
|
||||
Go engine options, not just the composed configuration inspected by ConfigKit.
|
||||
**Recommendation:** Start with facts already held by the relevant owner and add
|
||||
small fields to an existing prepared view where they are inexpensive to carry
|
||||
accurately. Evaluate transport cap and effective retry settings individually.
|
||||
Avoid a normalized policy model covering every client, invocation, and deadline.
|
||||
If a useful fact requires substantial new plumbing, explicitly defer that fact
|
||||
and document the report's boundary instead of inventing a default or guarantee.
|
||||
The field set and location need agreement before implementation.
|
||||
|
||||
**Recommendation:** Extend the existing prepared-result view with a small,
|
||||
separately named execution-policy section for facts applicable to that request.
|
||||
Report the effective built-in retry setting and transport cap, distinguishing
|
||||
injected-client ownership where Promptkit cannot supply those facts. Keep model
|
||||
settings, backend capacity, and caller context constraints distinct. Reuse the
|
||||
same policy selection as execution, including request overrides; do not add a
|
||||
second preview resolver or claim to know an injected client's internal policy.
|
||||
This makes the concrete preview useful without another top-level inspection API.
|
||||
|
||||
**Alternative:** Add a separate engine-settings inspection API for captured
|
||||
engine defaults, then let CommandKit combine that with resource inspection and
|
||||
request-specific preparation. This also serves operators who have no concrete
|
||||
request, but adds a public surface and requires a clear distinction between
|
||||
engine defaults and request overrides. Select the surface before implementation;
|
||||
neither option authorizes inspection of arbitrary client internals.
|
||||
**Alternative:** Add a dedicated engine-settings view for captured defaults,
|
||||
paired with prepared results for request-specific settings. This is useful to
|
||||
operators without a concrete request, but adds an API and requires distinguishing
|
||||
defaults from overrides. A third option is to defer engine-policy reporting
|
||||
entirely for this round, delivering the resource/presence improvements first;
|
||||
that has the lowest cost but leaves transport/retry reporting to later work.
|
||||
|
||||
### 3. Should Typed Prompt Construction Be Added In This Round?
|
||||
|
||||
**Recommendation:** Keep `PromptConfig` and `PromptSource` as a separate
|
||||
discussion item until a programmatic prompt-authoring use case establishes
|
||||
their value. File sources and ConfigKit mappings already support prompt
|
||||
definitions. The absence of a helper is a Go convenience gap, not a missing
|
||||
configuration capability, and does not block the backend additions.
|
||||
**Recommendation:** Keep `PromptConfig`/`PromptSource` separate until a
|
||||
programmatic authoring need justifies the public input model. Files and ConfigKit
|
||||
mappings already support prompts; this does not block operator inspection.
|
||||
|
||||
**Alternative:** Add a typed prompt source alongside the existing backend and
|
||||
profile helpers. It would improve discoverability, compiler assistance, and
|
||||
consistency for applications constructing prompts in Go. Its cost is a larger
|
||||
public input model covering messages, inline/file content, inputs, output
|
||||
contracts, session templates, and omission semantics. If selected, define that
|
||||
scope separately, including sibling inheritance and source context for relative
|
||||
content files, while retaining ConfigKit's ownership of composition and raw
|
||||
mapping support for explicit null. Do not implement a second prompt parser or
|
||||
validation path merely to provide the helper.
|
||||
**Alternative:** Add the typed helper for discoverability and compiler assistance.
|
||||
It would need useful message/input/output and omission semantics, inheritance,
|
||||
and source context for relative content files. Its maintenance cost is larger
|
||||
than adding backend `Extends`; composition can still delegate to ConfigKit.
|
||||
|
||||
## Implementation Scope And Verification
|
||||
### 4. How Should Presence And Public Result Values Be Represented?
|
||||
|
||||
1. Resolve the error and engine-policy decisions and define public result fields
|
||||
and GoDoc. Typed prompt construction is a separate scope decision; it does
|
||||
not block the selected inspection improvements.
|
||||
2. Expand prompt inspection from captured metadata and message content. Add
|
||||
presence information to profile and prepared results using the execution
|
||||
path's existing resolved state. Preserve existing values and error behavior.
|
||||
3. Connect namespace-aware backend inspection to captured lookup and add shared
|
||||
diagnostic destination derivation. Keep the public facade responsible for
|
||||
owned results; avoid a new backend cache, loader, or recursive inspector.
|
||||
4. Implement the selected engine-policy reporting surface without repeating
|
||||
policy resolution. Keep the concrete-request preview on existing preparation.
|
||||
5. Add `BackendConfig.Extends` and encode it through `BackendSource`, leaving
|
||||
composition with ConfigKit.
|
||||
6. Update public GoDoc, consumer guidance, relevant internal documentation, and
|
||||
release notes when implementation lands. Coordinate CommandKit changes for
|
||||
the fuller reports and omission-aware formatting, without modifying its CLI
|
||||
contract from this roadmap.
|
||||
The operator requirement is accurate omission, not a particular Go structure,
|
||||
new parallel report schema, or fixed JSON policy.
|
||||
|
||||
Use focused public tests for these consequential behaviors:
|
||||
**Recommendation:** Reuse existing public nested values and expose the existing
|
||||
`ExecutionTargetPresence` semantics alongside effective profile/prepared targets
|
||||
where practical. This takes advantage of data already used by execution and
|
||||
limits new translation. CommandKit can omit absent fields in its own output.
|
||||
Document how values and presence interact rather than introducing an independent
|
||||
presence/default/provenance system. Prefer additive changes when straightforward.
|
||||
|
||||
- Captured imported message content, description, session template, roles,
|
||||
cache controls, and output contract appear without rendering, rereading files,
|
||||
or requiring a resolvable default profile.
|
||||
- Profile and prepared results identify omitted/suppressed controls as absent
|
||||
and preserve explicit-zero and nonzero controls across inheritance and request
|
||||
overrides. Details retain the same frozen settings and presence as their
|
||||
prepared execution; no new distinction between reasons for omission is needed.
|
||||
- Backend inspection handles root/scoped/qualified identities, bundled and
|
||||
caller-defined records, capacity defaults, optional metadata, explicit and
|
||||
derived diagnostic destinations, errors, and cancellation. Missing credential
|
||||
environment values and missing smoke default profiles do not prevent lookup.
|
||||
- Mutating returned templates, maps, diagnostic settings, or prepared details
|
||||
cannot change captured state or another result. Inspection performs no
|
||||
provider calls; preparation retains its established resource-I/O boundary.
|
||||
- Engine-policy facts match the selected built-in or injected-client behavior
|
||||
and applicable request overrides under the chosen reporting surface.
|
||||
- Typed backend inheritance through real ConfigKit composition retains a parent
|
||||
field and applies a child override. Reuse generic inheritance coverage rather
|
||||
than retesting all of ConfigKit's composition rules.
|
||||
**Alternative:** Use optional-valued fields in an inspection-specific settings
|
||||
result, making absence directly expressible to consumers. That can simplify
|
||||
reporting, but duplicates an execution-settings surface and needs conversion
|
||||
and maintenance as fields evolve. Choose it only if the consumer simplification
|
||||
outweighs that cost. Ordinary JSON tags or a deliberate serialization change can
|
||||
also be considered; neither mandatory JSON stability for new results nor a ban
|
||||
on useful serialization is selected. Existing documented JSON contracts still
|
||||
need an explicit compatibility decision before changing.
|
||||
|
||||
Use controlled offline fixtures and reuse existing namespace, configuration,
|
||||
ownership, and preparation coverage. CommandKit should own report-format tests
|
||||
showing absent settings omitted and explicit zero preserved. Follow the
|
||||
[testing policy](../policy/testing.md); avoid tests coupled to private projection
|
||||
helpers or new recurring release gates.
|
||||
### 5. Are Both Declared And Effective Diagnostic Endpoints Needed?
|
||||
|
||||
Completion means an operator can inspect captured prompt content, effective
|
||||
profile settings with accurate omission semantics, and backend settings with
|
||||
effective diagnostic destinations, then use ordinary preparation to preview a
|
||||
concrete request. The chosen engine-policy surface supplies the additional facts
|
||||
needed to describe execution policy without inventing provider or client defaults.
|
||||
Typed backend inheritance is available through the source helper. Existing
|
||||
composition and execution semantics remain intact. Typed prompt construction,
|
||||
listing return-type changes, provider health guarantees, and live capacity
|
||||
monitoring are not completion requirements.
|
||||
Effective destinations are selected scope. The previous draft also suggested
|
||||
keeping declared overrides separately, which could create two representations
|
||||
and unnecessary origin metadata without a demonstrated reporting need.
|
||||
|
||||
**Recommendation:** Report the effective destination, authentication kind, and
|
||||
default profile reference. Use `InspectConfiguration` for authored values and
|
||||
provenance. Reuse existing optional values where helpful; no duplicate declared
|
||||
endpoint or derived-status flag is required merely for symmetry.
|
||||
|
||||
**Alternative:** Include declared overrides alongside effective destinations when
|
||||
operators need that distinction in one backend view. Existing metadata may make
|
||||
this cheap, but the result then needs clear field meanings and consumer handling
|
||||
for both. Prefer directly available fields over a new provenance subsystem.
|
||||
|
||||
## Implementation And Verification
|
||||
|
||||
1. Resolve open public-contract choices relevant to selected work. Agree on any
|
||||
engine-policy additions separately so an unresolved optional expansion does
|
||||
not silently become a prerequisite for the resource APIs.
|
||||
2. Expand prompt inspection from captured data and expose existing presence in
|
||||
profile/prepared results. Reuse or modestly extend current lookup/projection
|
||||
paths; choose private interface layouts during implementation.
|
||||
3. Add namespace-aware backend inspection and effective destination derivation,
|
||||
following the selected public error and result representation decisions.
|
||||
4. Add typed backend inheritance. Implement any separately selected engine-policy
|
||||
reporting using the owning components' effective values.
|
||||
5. Update GoDoc, consumer/internal guidance, and release notes with implemented
|
||||
behavior and any deliberate compatibility changes. Coordinate CommandKit's
|
||||
richer reports and omission handling without prescribing a new CLI framework.
|
||||
|
||||
Select focused offline tests for actual behavior and boundary risks:
|
||||
|
||||
- Newly exposed prompt fields reflect captured inline/imported content and
|
||||
order, without introducing rendering or related-resource dependencies.
|
||||
- Profile and prepared output preserve explicit zero and distinguish controls
|
||||
that execution omits, including relevant inheritance and override cases.
|
||||
- Backend results cover effective defaults/destinations, namespace selection,
|
||||
and applicable error behavior. Use cases with missing optional credentials or
|
||||
unresolved diagnostic profile references protect independent inspection.
|
||||
- Returned mutable values cannot alter retained state. Reuse existing ownership
|
||||
tests and copy helpers rather than prescribing extra snapshots or allocations.
|
||||
- Typed backend inheritance works through real ConfigKit composition; its
|
||||
generic composition rules do not need another full test matrix.
|
||||
- Any selected policy-reporting fields reflect their owners' effective settings.
|
||||
|
||||
CommandKit can test its own output omission and explicit-zero presentation.
|
||||
Follow the [testing policy](../policy/testing.md); combine cases where useful,
|
||||
reuse existing coverage, and do not turn this list into a test-per-field mandate.
|
||||
No new benchmark threshold or release gate follows from this roadmap.
|
||||
|
||||
Completion means the selected fields give operators a materially fuller and
|
||||
accurate resource view, omission-aware settings are available to report builders,
|
||||
and existing preparation remains usable for a concrete request. Backend source
|
||||
inheritance is exposed. The resolved decisions define additional policy fields,
|
||||
public representations, and compatibility changes; exhaustive execution-policy
|
||||
coverage, typed prompt construction, listing changes, and speculative future
|
||||
adapters are not implicit completion conditions.
|
||||
|
||||
Reference in New Issue
Block a user