Clarify provider selection and diagnostic defaults in the roadmap
This commit is contained in:
@@ -1,6 +1,7 @@
|
|||||||
# Invocation-Scoped Execution Observation
|
# Invocation-Scoped Execution Observation
|
||||||
|
|
||||||
Status: accepted scope; implementation pending.
|
Status: agreed direction; default-provider and vLLM scope under discussion;
|
||||||
|
implementation pending.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
@@ -143,11 +144,18 @@ promptkit:
|
|||||||
api_key_env: OPENROUTER_API_KEY
|
api_key_env: OPENROUTER_API_KEY
|
||||||
```
|
```
|
||||||
|
|
||||||
An omitted or null provider retains ordinary OpenAI-compatible behavior without
|
Normalize the final effective provider value by trimming whitespace and
|
||||||
provider-specific defaults or metadata interpretation. Initially, `openrouter`
|
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
|
||||||
selects the OpenRouter conventions described here. A vLLM-specific selection can
|
Reject an unsupported nonempty normalized value as an ordinary configuration
|
||||||
follow when its relevant response contract and representative fixtures have been
|
error rather than silently using generic behavior. `openrouter` selects the
|
||||||
reviewed; this work does not claim vLLM-specific support.
|
OpenRouter conventions described here.
|
||||||
|
|
||||||
|
The current generic fallback retains ordinary OpenAI-compatible behavior without
|
||||||
|
provider-specific defaults or metadata interpretation. Before finalizing this
|
||||||
|
roadmap, discuss making this fallback an explicit lowest-common-denominator
|
||||||
|
provider and adding a `vllm` selection. Their names, supported conventions, and
|
||||||
|
final scope remain to be settled; this document does not yet select a vLLM
|
||||||
|
contract.
|
||||||
|
|
||||||
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
||||||
the selected value through configuration capture and prepared execution, expose
|
the selected value through configuration capture and prepared execution, expose
|
||||||
@@ -182,6 +190,21 @@ existing behavior.
|
|||||||
These are conventions for explicitly requested diagnostics. Loading a backend,
|
These are conventions for explicitly requested diagnostics. Loading a backend,
|
||||||
constructing an engine, and ordinary generation do not initiate probes.
|
constructing an engine, and ordinary generation do not initiate probes.
|
||||||
|
|
||||||
|
Resolve provider-supplied diagnostic defaults at the provider boundary without
|
||||||
|
writing them into the loaded configuration or recording them as explicit source
|
||||||
|
values. Configuration inspection preserves the distinction between configured
|
||||||
|
overrides and provider-supplied fallbacks, including their existing source
|
||||||
|
provenance.
|
||||||
|
|
||||||
|
### Diagnostic Migration
|
||||||
|
|
||||||
|
This changes the current authentication metadata contract: an authentication
|
||||||
|
object currently requires `kind`, and clearing the object removes its configured
|
||||||
|
adapter. Under this roadmap, endpoint-only overrides become valid, and clearing
|
||||||
|
the object restores any selected provider's default. `kind: none` suppresses
|
||||||
|
that default. Explain these changes in the framework format documentation and
|
||||||
|
release notes, alongside the new provider field.
|
||||||
|
|
||||||
### Provider Behavior Ownership
|
### Provider Behavior Ownership
|
||||||
|
|
||||||
Centralize provider-specific behavior within `internal/llm`, which already owns
|
Centralize provider-specific behavior within `internal/llm`, which already owns
|
||||||
@@ -200,6 +223,8 @@ method to a generic operation as part of this consolidation.
|
|||||||
Provider-specific files and straightforward dispatch are sufficient initially.
|
Provider-specific files and straightforward dispatch are sufficient initially.
|
||||||
No new `internal/providers` package, registration framework, or broad interface
|
No new `internal/providers` package, registration framework, or broad interface
|
||||||
requiring every provider to implement every capability is needed.
|
requiring every provider to implement every capability is needed.
|
||||||
|
This is configurable selection among PromptKit-supported provider implementations,
|
||||||
|
not a public API for downstream registration of additional implementations.
|
||||||
|
|
||||||
## Optional Provider-Reported Cost
|
## Optional Provider-Reported Cost
|
||||||
|
|
||||||
@@ -321,9 +346,14 @@ logging infrastructure are outside this scope.
|
|||||||
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
||||||
and existing explicit probes without a provider selection.
|
and existing explicit probes without a provider selection.
|
||||||
- Verify provider selection independently of backend names and namespaces,
|
- Verify provider selection independently of backend names and namespaces,
|
||||||
including renamed and inherited backends, generic behavior when omitted/null,
|
including renamed and inherited backends, equivalent omitted/null/blank values,
|
||||||
|
trimming and case normalization, rejection of unsupported nonempty values,
|
||||||
and retention through endpoint overrides. Confirm provider metadata is not
|
and retention through endpoint overrides. Confirm provider metadata is not
|
||||||
serialized into the completion request body.
|
serialized into the completion request body. Use the generic fallback selected
|
||||||
|
when the remaining provider discussion is complete.
|
||||||
|
- Verify that provider diagnostic defaults do not become explicit values in the
|
||||||
|
loaded configuration or replace source provenance in inspection. Cover the
|
||||||
|
changed clearing behavior and endpoint-only authentication overrides.
|
||||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||||
conversions, successful run results, attempt and terminal events, and existing
|
conversions, successful run results, attempt and terminal events, and existing
|
||||||
group summaries. Keep result cost available without registering an observer.
|
group summaries. Keep result cost available without registering an observer.
|
||||||
@@ -353,11 +383,12 @@ logging infrastructure are outside this scope.
|
|||||||
and update internal observation and model-client documentation for changed flows.
|
and update internal observation and model-client documentation for changed flows.
|
||||||
Document `provider`, its smoke-test defaults and override semantics in the
|
Document `provider`, its smoke-test defaults and override semantics in the
|
||||||
framework format reference and relevant public GoDoc; update inspection and
|
framework format reference and relevant public GoDoc; update inspection and
|
||||||
diagnostic documentation with the implemented behavior.
|
diagnostic documentation with the implemented behavior. Include the diagnostic
|
||||||
|
migration in release notes.
|
||||||
|
|
||||||
## Implementation Planning Guidance
|
## Implementation Planning Guidance
|
||||||
|
|
||||||
Plan two separately reviewable stages:
|
Organize the eventual implementation plan around two workstreams:
|
||||||
|
|
||||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||||
and nested execution boundaries and the existing group collector.
|
and nested execution boundaries and the existing group collector.
|
||||||
@@ -366,9 +397,11 @@ Plan two separately reviewable stages:
|
|||||||
through decoding, accounting, public results, events, and group summaries.
|
through decoding, accounting, public results, events, and group summaries.
|
||||||
Include the corresponding configuration and API documentation.
|
Include the corresponding configuration and API documentation.
|
||||||
|
|
||||||
The second stage is more extensive than the context helper and may be divided
|
The second area is more extensive than the context helper. Keep backend provider
|
||||||
into focused implementation units. Backend provider selection is decided above;
|
configuration, diagnostic consolidation, and cost accounting in separately
|
||||||
avoid coupling delivery of contextual observation to provider cost support.
|
reviewable implementation units rather than one large implementation prompt.
|
||||||
|
Complete the remaining provider discussion before preparing that plan; avoid
|
||||||
|
coupling delivery of contextual observation to provider cost support.
|
||||||
|
|
||||||
Today, successful results accumulate generation responses in the runner, while
|
Today, successful results accumulate generation responses in the runner, while
|
||||||
optional observation recorders account for dispatched attempts. Adding cost to
|
optional observation recorders account for dispatched attempts. Adding cost to
|
||||||
@@ -385,5 +418,19 @@ field separately so an unusable cost becomes unknown without changing existing
|
|||||||
framing or content validation. This needs no general numeric-conversion layer.
|
framing or content validation. This needs no general numeric-conversion layer.
|
||||||
|
|
||||||
Use the completion criteria above to select focused offline checks for each
|
Use the completion criteria above to select focused offline checks for each
|
||||||
stage. This roadmap owns target behavior; a separate implementation plan owns
|
unit. This roadmap owns target behavior; a separate implementation plan owns
|
||||||
the detailed work sequence.
|
the detailed work sequence.
|
||||||
|
|
||||||
|
## Topics For Further Discussion
|
||||||
|
|
||||||
|
Before finalizing the roadmap, settle:
|
||||||
|
|
||||||
|
1. A default, lowest-common-denominator OpenAI API-compatible provider used when
|
||||||
|
the backend does not specify a provider, including whether it has an explicit
|
||||||
|
selectable identifier and which common conventions it owns.
|
||||||
|
2. A `vllm` provider with vLLM-specific conventions, identifying the response and
|
||||||
|
diagnostic behavior to support from reviewed upstream contracts and fixtures.
|
||||||
|
|
||||||
|
The normalization, unsupported-value handling, override precedence, and
|
||||||
|
configuration-inspection boundaries above remain agreed decisions. These topics
|
||||||
|
do not yet add further provider behavior or an implementation commitment.
|
||||||
|
|||||||
Reference in New Issue
Block a user