Clarify provider selection and diagnostic defaults in the roadmap

This commit is contained in:
2026-11-22 15:41:20 +00:00
parent a10d1a2245
commit 0254eb1b5b

View File

@@ -1,6 +1,7 @@
# Invocation-Scoped Execution Observation # Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending. Status: agreed direction; default-provider and vLLM scope under discussion;
implementation pending.
## Purpose ## Purpose
@@ -143,11 +144,18 @@ promptkit:
api_key_env: OPENROUTER_API_KEY api_key_env: OPENROUTER_API_KEY
``` ```
An omitted or null provider retains ordinary OpenAI-compatible behavior without Normalize the final effective provider value by trimming whitespace and
provider-specific defaults or metadata interpretation. Initially, `openrouter` lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
selects the OpenRouter conventions described here. A vLLM-specific selection can Reject an unsupported nonempty normalized value as an ordinary configuration
follow when its relevant response contract and representative fixtures have been error rather than silently using generic behavior. `openrouter` selects the
reviewed; this work does not claim vLLM-specific support. OpenRouter conventions described here.
The current generic fallback retains ordinary OpenAI-compatible behavior without
provider-specific defaults or metadata interpretation. Before finalizing this
roadmap, discuss making this fallback an explicit lowest-common-denominator
provider and adding a `vllm` selection. Their names, supported conventions, and
final scope remain to be settled; this document does not yet select a vLLM
contract.
Resolve the field through ordinary ConfigKit composition and inheritance. Carry Resolve the field through ordinary ConfigKit composition and inheritance. Carry
the selected value through configuration capture and prepared execution, expose the selected value through configuration capture and prepared execution, expose
@@ -182,6 +190,21 @@ existing behavior.
These are conventions for explicitly requested diagnostics. Loading a backend, These are conventions for explicitly requested diagnostics. Loading a backend,
constructing an engine, and ordinary generation do not initiate probes. constructing an engine, and ordinary generation do not initiate probes.
Resolve provider-supplied diagnostic defaults at the provider boundary without
writing them into the loaded configuration or recording them as explicit source
values. Configuration inspection preserves the distinction between configured
overrides and provider-supplied fallbacks, including their existing source
provenance.
### Diagnostic Migration
This changes the current authentication metadata contract: an authentication
object currently requires `kind`, and clearing the object removes its configured
adapter. Under this roadmap, endpoint-only overrides become valid, and clearing
the object restores any selected provider's default. `kind: none` suppresses
that default. Explain these changes in the framework format documentation and
release notes, alongside the new provider field.
### Provider Behavior Ownership ### Provider Behavior Ownership
Centralize provider-specific behavior within `internal/llm`, which already owns Centralize provider-specific behavior within `internal/llm`, which already owns
@@ -200,6 +223,8 @@ method to a generic operation as part of this consolidation.
Provider-specific files and straightforward dispatch are sufficient initially. Provider-specific files and straightforward dispatch are sufficient initially.
No new `internal/providers` package, registration framework, or broad interface No new `internal/providers` package, registration framework, or broad interface
requiring every provider to implement every capability is needed. requiring every provider to implement every capability is needed.
This is configurable selection among PromptKit-supported provider implementations,
not a public API for downstream registration of additional implementations.
## Optional Provider-Reported Cost ## Optional Provider-Reported Cost
@@ -321,9 +346,14 @@ logging infrastructure are outside this scope.
provider-default probes, explicit kind and endpoint overrides, `kind: none`, provider-default probes, explicit kind and endpoint overrides, `kind: none`,
and existing explicit probes without a provider selection. and existing explicit probes without a provider selection.
- Verify provider selection independently of backend names and namespaces, - Verify provider selection independently of backend names and namespaces,
including renamed and inherited backends, generic behavior when omitted/null, including renamed and inherited backends, equivalent omitted/null/blank values,
trimming and case normalization, rejection of unsupported nonempty values,
and retention through endpoint overrides. Confirm provider metadata is not and retention through endpoint overrides. Confirm provider metadata is not
serialized into the completion request body. serialized into the completion request body. Use the generic fallback selected
when the remaining provider discussion is complete.
- Verify that provider diagnostic defaults do not become explicit values in the
loaded configuration or replace source provenance in inspection. Cover the
changed clearing behavior and endpoint-only authentication overrides.
- Carry optional OpenRouter cost through response decoding, internal/public - Carry optional OpenRouter cost through response decoding, internal/public
conversions, successful run results, attempt and terminal events, and existing conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer. group summaries. Keep result cost available without registering an observer.
@@ -353,11 +383,12 @@ logging infrastructure are outside this scope.
and update internal observation and model-client documentation for changed flows. and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the Document `provider`, its smoke-test defaults and override semantics in the
framework format reference and relevant public GoDoc; update inspection and framework format reference and relevant public GoDoc; update inspection and
diagnostic documentation with the implemented behavior. diagnostic documentation with the implemented behavior. Include the diagnostic
migration in release notes.
## Implementation Planning Guidance ## Implementation Planning Guidance
Plan two separately reviewable stages: Organize the eventual implementation plan around two workstreams:
1. Contextual observer selection, including ordinary, prepared, coordinated, 1. Contextual observer selection, including ordinary, prepared, coordinated,
and nested execution boundaries and the existing group collector. and nested execution boundaries and the existing group collector.
@@ -366,9 +397,11 @@ Plan two separately reviewable stages:
through decoding, accounting, public results, events, and group summaries. through decoding, accounting, public results, events, and group summaries.
Include the corresponding configuration and API documentation. Include the corresponding configuration and API documentation.
The second stage is more extensive than the context helper and may be divided The second area is more extensive than the context helper. Keep backend provider
into focused implementation units. Backend provider selection is decided above; configuration, diagnostic consolidation, and cost accounting in separately
avoid coupling delivery of contextual observation to provider cost support. reviewable implementation units rather than one large implementation prompt.
Complete the remaining provider discussion before preparing that plan; avoid
coupling delivery of contextual observation to provider cost support.
Today, successful results accumulate generation responses in the runner, while Today, successful results accumulate generation responses in the runner, while
optional observation recorders account for dispatched attempts. Adding cost to optional observation recorders account for dispatched attempts. Adding cost to
@@ -385,5 +418,19 @@ field separately so an unusable cost becomes unknown without changing existing
framing or content validation. This needs no general numeric-conversion layer. framing or content validation. This needs no general numeric-conversion layer.
Use the completion criteria above to select focused offline checks for each Use the completion criteria above to select focused offline checks for each
stage. This roadmap owns target behavior; a separate implementation plan owns unit. This roadmap owns target behavior; a separate implementation plan owns
the detailed work sequence. the detailed work sequence.
## Topics For Further Discussion
Before finalizing the roadmap, settle:
1. A default, lowest-common-denominator OpenAI API-compatible provider used when
the backend does not specify a provider, including whether it has an explicit
selectable identifier and which common conventions it owns.
2. A `vllm` provider with vLLM-specific conventions, identifying the response and
diagnostic behavior to support from reviewed upstream contracts and fixtures.
The normalization, unsupported-value handling, override precedence, and
configuration-inspection boundaries above remain agreed decisions. These topics
do not yet add further provider behavior or an implementation commitment.