Select provider conventions through backend configuration
This commit is contained in:
@@ -126,6 +126,81 @@ must not inflate its totals. Terminal usage is an aggregate, not an additional
|
||||
contribution to be added to attempt usage. These rules apply to cost as well as
|
||||
token counts.
|
||||
|
||||
## Backend Provider Selection
|
||||
|
||||
Add an optional backend `provider` field identifying the API implementation
|
||||
whose conventions PromptKit should use. It identifies the service contacted,
|
||||
not the organization that trained the selected model. Backend IDs remain resource
|
||||
identities; names, namespaces, and endpoint hostnames do not select provider
|
||||
behavior.
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
backends:
|
||||
- id: research-router
|
||||
provider: openrouter
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
```
|
||||
|
||||
An omitted or null provider retains ordinary OpenAI-compatible behavior without
|
||||
provider-specific defaults or metadata interpretation. Initially, `openrouter`
|
||||
selects the OpenRouter conventions described here. A vLLM-specific selection can
|
||||
follow when its relevant response contract and representative fixtures have been
|
||||
reviewed; this work does not claim vLLM-specific support.
|
||||
|
||||
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
||||
the selected value through configuration capture and prepared execution, expose
|
||||
it in backend inspection, and declare it in the bundled OpenRouter backend.
|
||||
It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a
|
||||
provider does not choose a model or alter the configured endpoint. An endpoint
|
||||
override retains the provider selection.
|
||||
|
||||
### Smoke-Test Defaults And Explicit Overrides
|
||||
|
||||
When an authentication smoke test is explicitly requested, the selected provider
|
||||
supplies its default probe convention. `provider: openrouter` supplies the
|
||||
existing `openrouter_key` convention. Preserve explicit
|
||||
`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint`
|
||||
settings as overrides:
|
||||
|
||||
- A supplied kind selects the diagnostic adapter instead of the provider default.
|
||||
`kind: none` continues to declare authentication testing inapplicable.
|
||||
- An omitted/null kind uses the provider default, if available. This permits an
|
||||
endpoint override without repeating the provider's default kind.
|
||||
- A supplied endpoint overrides the selected adapter's default destination.
|
||||
An omitted/null endpoint uses that adapter's existing endpoint fallback.
|
||||
|
||||
An absent/null authentication block likewise leaves selection to the provider
|
||||
default. Clearing the block removes its explicit overrides; `kind: none` is the
|
||||
explicit way to declare authentication inapplicable despite a provider default.
|
||||
Without a provider default or explicit adapter, retain existing unsupported
|
||||
diagnostic behavior. Existing explicit adapters continue to work without a
|
||||
provider declaration. Reachability and inference-default settings keep their
|
||||
existing behavior.
|
||||
|
||||
These are conventions for explicitly requested diagnostics. Loading a backend,
|
||||
constructing an engine, and ordinary generation do not initiate probes.
|
||||
|
||||
### Provider Behavior Ownership
|
||||
|
||||
Centralize provider-specific behavior within `internal/llm`, which already owns
|
||||
OpenAI-compatible transport and OpenRouter authentication-response recognition.
|
||||
Keep common request construction, credentials, HTTP execution, framing, and
|
||||
common response fields shared. Provider-specific code owns additional response
|
||||
metadata interpretation and authentication-probe conventions.
|
||||
|
||||
Use one provider-selection mechanism inside that package. Generation,
|
||||
observation, and smoke-test coordination consume provider-neutral facts or invoke
|
||||
generic diagnostic operations rather than repeating provider dispatch and
|
||||
response interpretation. Explicit diagnostic adapter selection remains supported
|
||||
at this boundary. Update the current OpenRouter-specific diagnostic collaborator
|
||||
method to a generic operation as part of this consolidation.
|
||||
|
||||
Provider-specific files and straightforward dispatch are sufficient initially.
|
||||
No new `internal/providers` package, registration framework, or broad interface
|
||||
requiring every provider to implement every capability is needed.
|
||||
|
||||
## Optional Provider-Reported Cost
|
||||
|
||||
### Initial Provider Support
|
||||
@@ -137,18 +212,11 @@ and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-t
|
||||
Do not add `cost_details` breakdowns or combine upstream inference cost with the
|
||||
top-level charge; those describe different facts and could double-count cost.
|
||||
|
||||
For the built-in client, select this interpretation for the resolved
|
||||
`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's
|
||||
field named `cost`, or add endpoint discovery or verification. An endpoint
|
||||
override on that backend continues to select its OpenRouter response contract.
|
||||
Other built-in backend selections report unknown cost initially. Additional
|
||||
provider adapters or cost-capability configuration are not needed for this scope.
|
||||
|
||||
This baseline selects by exact backend identity: a separately named or
|
||||
collection-qualified backend pointing to OpenRouter would report unknown cost.
|
||||
Backend configuration as an alternative to identity-based selection is now an
|
||||
open discussion; settle that selection before implementing provider cost support.
|
||||
The contextual-observation stage does not depend on that decision.
|
||||
For the built-in client, select this interpretation through `provider: openrouter`,
|
||||
regardless of backend identity or namespace. Do not infer USD semantics from an
|
||||
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
||||
Without an OpenRouter provider selection, built-in cost is unknown initially.
|
||||
No separate cost-reporting selector is needed.
|
||||
|
||||
Use the cost included in the normal completion response. No additional usage
|
||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||
@@ -245,6 +313,17 @@ logging infrastructure are outside this scope.
|
||||
- Preserve existing explicit selection and coordinator behavior while applying
|
||||
the composition and lifetime rules above. Preserve existing observer selection
|
||||
when no contextual observer is selected.
|
||||
- Add `provider` to the backend configuration model, typed source helpers,
|
||||
capture, inspection, and execution plumbing. Update the bundled OpenRouter
|
||||
backend to select it explicitly. Retain the selection through prepared runs.
|
||||
- Consolidate provider-specific response and authentication-probe behavior in
|
||||
`internal/llm`; keep coordination and accounting provider-neutral. Cover
|
||||
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
||||
and existing explicit probes without a provider selection.
|
||||
- Verify provider selection independently of backend names and namespaces,
|
||||
including renamed and inherited backends, generic behavior when omitted/null,
|
||||
and retention through endpoint overrides. Confirm provider metadata is not
|
||||
serialized into the completion request body.
|
||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
@@ -263,7 +342,7 @@ logging infrastructure are outside this scope.
|
||||
failures or cancellation, and known zero versus unknown or incomplete counts.
|
||||
Use offline fixtures and injected clients; live inference is unnecessary.
|
||||
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
|
||||
non-OpenRouter built-in selection, optional injected-client cost, mixed known
|
||||
generic built-in selection, optional injected-client cost, mixed known
|
||||
and unknown totals, repair/retry aggregation, and duplicate group registration.
|
||||
Use tolerant comparisons where floating-point addition requires them; do not
|
||||
turn telemetry into an exact-decimal guarantee.
|
||||
@@ -272,6 +351,9 @@ logging infrastructure are outside this scope.
|
||||
availability, units, and completeness. Document OpenRouter cost decoding in
|
||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
Document `provider`, its smoke-test defaults and override semantics in the
|
||||
framework format reference and relevant public GoDoc; update inspection and
|
||||
diagnostic documentation with the implemented behavior.
|
||||
|
||||
## Implementation Planning Guidance
|
||||
|
||||
@@ -279,12 +361,14 @@ Plan two separately reviewable stages:
|
||||
|
||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||
and nested execution boundaries and the existing group collector.
|
||||
2. Provider-reported cost, carried through decoding, accounting, public results,
|
||||
events, and group summaries, with the corresponding documentation.
|
||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||
smoke-test defaults and explicit overrides, followed by provider-reported cost
|
||||
through decoding, accounting, public results, events, and group summaries.
|
||||
Include the corresponding configuration and API documentation.
|
||||
|
||||
The second stage is more extensive than the context helper. Resolve the backend
|
||||
selection discussion before finalizing that stage; avoid coupling delivery of
|
||||
contextual observation to provider cost support.
|
||||
The second stage is more extensive than the context helper and may be divided
|
||||
into focused implementation units. Backend provider selection is decided above;
|
||||
avoid coupling delivery of contextual observation to provider cost support.
|
||||
|
||||
Today, successful results accumulate generation responses in the runner, while
|
||||
optional observation recorders account for dispatched attempts. Adding cost to
|
||||
|
||||
Reference in New Issue
Block a user