Add vLLM provider conventions to the observation roadmap
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Invocation-Scoped Execution Observation
|
||||
|
||||
Status: agreed direction; vLLM scope under discussion; implementation pending.
|
||||
Status: accepted scope; implementation pending.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -9,8 +9,8 @@ operation, rather than attach an observer to every PromptKit request. Nested,
|
||||
concurrent, and multi-engine calls using that operation's context should
|
||||
participate automatically. Include optional provider-reported cost alongside
|
||||
token usage, with built-in cost support initially limited to OpenRouter. Extend
|
||||
OpenRouter token reporting with reasoning counts, documented cache reads and
|
||||
writes, and a derived token cache-hit ratio.
|
||||
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads
|
||||
and writes, and a shared derived token cache-hit ratio.
|
||||
|
||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||
could use this facility to report token usage for an entire command or pipeline.
|
||||
@@ -151,19 +151,16 @@ promptkit:
|
||||
Normalize the final effective provider value by trimming whitespace and
|
||||
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
|
||||
Reject an unsupported nonempty normalized value as an ordinary configuration
|
||||
error rather than silently using generic behavior. The agreed provider selections
|
||||
are `openai-compatible` and `openrouter`; the possible `vllm` addition remains
|
||||
under discussion.
|
||||
error rather than silently using generic behavior. The supported selections for
|
||||
this feature are `openai-compatible`, `openrouter`, and `vllm`.
|
||||
|
||||
Omitted/null/blank provider values select the same effective behavior as explicit
|
||||
`provider: openai-compatible`. This is a runtime fallback, not an explicitly
|
||||
sourced configuration value. Backend inspection retains the distinction between
|
||||
an explicit provider declaration and an omitted/null/blank declaration.
|
||||
|
||||
The `openrouter` provider reuses the shared OpenAI-compatible foundation and adds
|
||||
the OpenRouter response-metadata and diagnostic conventions below. Before
|
||||
finalizing this roadmap, discuss the conventions and scope of a `vllm` selection;
|
||||
this document does not yet select a vLLM contract.
|
||||
The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation
|
||||
and add their response-metadata and diagnostic conventions below.
|
||||
|
||||
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
||||
the selected value through configuration capture and prepared execution, expose
|
||||
@@ -201,11 +198,29 @@ Continue sending caller-selected supported settings and return backend errors
|
||||
normally, without capability negotiation or fallback requests. This is the
|
||||
existing text Chat Completions scope, not a claim to support the full OpenAI API.
|
||||
|
||||
### vLLM Provider
|
||||
|
||||
`provider: vllm` selects the shared generation path plus vLLM-specific usage
|
||||
normalization and a default authentication diagnostic. It supplies no default
|
||||
endpoint, model, API-key environment variable, or sampling settings, and does not
|
||||
create a backend definition automatically.
|
||||
|
||||
Preserve caller-selected reasoning settings, structured-output requests, and
|
||||
extra parameters. Do not automatically change chat-template arguments, enable
|
||||
thinking, or translate model-specific controls based on the provider label.
|
||||
No separate generation protocol is needed; vLLM documents an
|
||||
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
||||
|
||||
The token mappings are specified below. Monetary cost remains unknown, not zero,
|
||||
including when vLLM is locally hosted. This provider adds no price estimator,
|
||||
server administration, metrics collection, or automatic capability/version probe.
|
||||
|
||||
### Smoke-Test Defaults And Explicit Overrides
|
||||
|
||||
When an authentication smoke test is explicitly requested, the selected provider
|
||||
supplies its default probe convention. `provider: openrouter` supplies the
|
||||
existing `openrouter_key` convention. Preserve explicit
|
||||
existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models`
|
||||
convention described below. Preserve explicit
|
||||
`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint`
|
||||
settings as overrides:
|
||||
|
||||
@@ -233,6 +248,30 @@ values. Configuration inspection preserves the distinction between configured
|
||||
overrides and provider-supplied fallbacks, including their existing source
|
||||
provenance.
|
||||
|
||||
### vLLM Authentication Diagnostic
|
||||
|
||||
The `vllm_models` adapter makes one GET request to `/models` relative to the
|
||||
configured API base, supplying an available credential through the existing
|
||||
credential rules. A base ending in `/v1` therefore produces `/v1/models`.
|
||||
Existing explicit diagnostic endpoint overrides remain available. The adapter
|
||||
can dispatch without a credential; selecting vLLM does not declare authentication
|
||||
required or inapplicable.
|
||||
|
||||
Recognize a successful models-list response as acceptance of the probe request
|
||||
and a 401 as rejection. Unrecognized responses leave acceptance unknown under
|
||||
the existing diagnostic reporting model. Report observed credential presence and
|
||||
response facts without claiming that authentication is enforced or that models
|
||||
listing establishes inference authorization. Preserve the shared override rules,
|
||||
including `kind: none`; explicit `vllm_models` selection also works without a
|
||||
`vllm` provider declaration, just as explicit OpenRouter adapters do.
|
||||
|
||||
vLLM documents `/v1/models` as protected when its API-key authentication is
|
||||
configured, while `/health` does not establish authentication acceptance. See its
|
||||
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
|
||||
Retain existing reachability and inference diagnostic behavior rather than
|
||||
adding health monitoring. These requests occur only during explicitly requested
|
||||
diagnostics, never during ordinary generation or engine construction.
|
||||
|
||||
### Diagnostic Migration
|
||||
|
||||
This changes the current authentication metadata contract: an authentication
|
||||
@@ -263,9 +302,9 @@ requiring every provider to implement every capability is needed.
|
||||
This is configurable selection among PromptKit-supported provider implementations,
|
||||
not a public API for downstream registration of additional implementations.
|
||||
|
||||
## OpenRouter Token And Cache Reporting
|
||||
## Token And Cache Reporting
|
||||
|
||||
### Response Mapping
|
||||
### OpenRouter Response Mapping
|
||||
|
||||
Use OpenRouter's documented Chat Completions usage fields. Its
|
||||
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
||||
@@ -290,15 +329,47 @@ nested value; an omitted nested value remains unknown.
|
||||
|
||||
Add `ReasoningTokens` to the public/domain token values, presence indicators,
|
||||
and observed token usage, including conversions, result serialization, attempt
|
||||
and terminal events, and group summaries. Initially, OpenRouter supplies this
|
||||
and terminal events, and group summaries. OpenRouter and vLLM supply this
|
||||
additional built-in response interpretation; the generic provider preserves its
|
||||
existing behavior. Injected clients may supply the canonical count and its
|
||||
presence through the existing model-client boundary.
|
||||
|
||||
### vLLM Response Mapping And Availability
|
||||
|
||||
Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens`
|
||||
counts. For `provider: vllm`, map these additional fields relative to `usage`:
|
||||
|
||||
| Response field | PromptKit field |
|
||||
| --- | --- |
|
||||
| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` |
|
||||
| `prompt_tokens_details.cached_tokens` | `CachedTokens` |
|
||||
| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` |
|
||||
|
||||
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
|
||||
uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its
|
||||
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
|
||||
describe the creation count as prompt tokens counted as local prefix-cache
|
||||
writes. Normalize that token activity to `CacheWriteTokens` without implying a
|
||||
billable write charge. Use the selected provider's documented field, not a sum
|
||||
of alternative cache-write fields; an absent vLLM creation count stays unknown.
|
||||
|
||||
Detail availability depends on server version and configuration. Current vLLM
|
||||
gates prompt details on `--enable-prompt-tokens-details` and reasoning details on
|
||||
a configured reasoning parser. Document these reporting settings, but do not
|
||||
configure or discover them automatically. Older or differently configured
|
||||
servers may omit the fields; accept those responses with the affected counts
|
||||
unknown. Do not infer reasoning counts from generated text or cache writes from
|
||||
input tokens minus cache reads.
|
||||
|
||||
Once normalized, these counts use the shared presence, aggregation, and
|
||||
cache-hit rules below. No vLLM-specific result type, ratio, or group accounting
|
||||
is needed.
|
||||
|
||||
### Count Relationships And Availability
|
||||
|
||||
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
|
||||
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
|
||||
vLLM likewise reports its reasoning detail within the generated-token usage.
|
||||
Cache reads and writes describe input-token breakdowns. Present these as subsets
|
||||
or details rather than additional consumption to add to input/output totals.
|
||||
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
|
||||
@@ -363,8 +434,8 @@ For the built-in client, select this interpretation through `provider: openroute
|
||||
regardless of backend identity or namespace. Do not infer USD semantics from an
|
||||
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
||||
Built-in cost interpretation is an extension unique to `openrouter` initially;
|
||||
`openai-compatible` leaves cost unknown even if a response contains `usage.cost`.
|
||||
No separate cost-reporting selector is needed. Aggregation remains
|
||||
`openai-compatible` and `vllm` leave cost unknown even if a response contains
|
||||
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains
|
||||
provider-neutral after the model-client boundary supplies interpreted USD facts.
|
||||
|
||||
Use the cost included in the normal completion response. No additional usage
|
||||
@@ -485,6 +556,17 @@ outside this scope.
|
||||
decoding, including omitted/null/zero values and the nested value taking
|
||||
precedence when a top-level value is also present. Preserve generic-provider
|
||||
token mappings and existing field meanings.
|
||||
- Add the `vllm` provider selection, reasoning and cache-detail mappings, and
|
||||
`vllm_models` authentication adapter. Cover supplied/missing credentials,
|
||||
recognized acceptance, rejection, unrecognized responses, provider defaults,
|
||||
and explicit kind/endpoint overrides, including `none` and adapter selection
|
||||
without a provider declaration. Verify that ordinary generation performs no
|
||||
diagnostic requests and leaves monetary cost unknown.
|
||||
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
|
||||
details, including responses representative of servers without detail
|
||||
reporting enabled. Verify `created_cache_tokens` mapping independently of
|
||||
OpenRouter's cache-write field and carry normalized counts through results,
|
||||
events, and groups. Do not require a live server or version discovery.
|
||||
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
|
||||
incomplete counts, zero input, inconsistent counts, independence from
|
||||
cache-write availability, and token-weighted aggregate ratios. Verify subset
|
||||
@@ -515,13 +597,16 @@ outside this scope.
|
||||
aggregation example to the consumer observation guide, including cost
|
||||
availability, units, and completeness. Document the reasoning/cache fields and
|
||||
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
|
||||
without double-counting. Document OpenRouter token and cost decoding in
|
||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter
|
||||
cost decoding in the
|
||||
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
Document `provider`, its smoke-test defaults and override semantics in the
|
||||
framework format reference and relevant public GoDoc; update inspection and
|
||||
diagnostic documentation with the implemented behavior. Include the diagnostic
|
||||
migration in release notes.
|
||||
migration in release notes. Include vLLM usage availability, server reporting
|
||||
settings, the meaning of cache creation, and the limits of its authentication
|
||||
diagnostic in the relevant consumer and integration documentation.
|
||||
|
||||
## Implementation Planning Guidance
|
||||
|
||||
@@ -530,7 +615,7 @@ Organize the eventual implementation plan around two workstreams:
|
||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||
and nested execution boundaries and the existing group collector.
|
||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||
smoke-test defaults and explicit overrides, followed by OpenRouter token
|
||||
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
|
||||
extensions and provider-reported cost through decoding, accounting, public
|
||||
results, events, and group summaries, plus the derived cache-hit helper.
|
||||
Include the corresponding configuration and API documentation.
|
||||
@@ -539,8 +624,7 @@ The second area is more extensive than the context helper. Keep backend provider
|
||||
configuration, diagnostic consolidation, token/cache reporting, and cost
|
||||
accounting in separately reviewable implementation units rather than one large
|
||||
implementation prompt.
|
||||
Complete the remaining provider discussion before preparing that plan; avoid
|
||||
coupling delivery of contextual observation to provider cost support.
|
||||
Avoid coupling delivery of contextual observation to provider cost support.
|
||||
|
||||
Today, successful results accumulate generation responses in the runner, while
|
||||
optional observation recorders account for dispatched attempts. Adding cost to
|
||||
@@ -559,14 +643,3 @@ framing or content validation. This needs no general numeric-conversion layer.
|
||||
Use the completion criteria above to select focused offline checks for each
|
||||
unit. This roadmap owns target behavior; a separate implementation plan owns
|
||||
the detailed work sequence.
|
||||
|
||||
## Topics For Further Discussion
|
||||
|
||||
Before finalizing the roadmap, discuss a `vllm` provider with vLLM-specific
|
||||
conventions, identifying the response and diagnostic behavior to support from
|
||||
reviewed upstream contracts and fixtures.
|
||||
|
||||
The `openai-compatible` default, normalization, unsupported-value handling,
|
||||
override precedence, and configuration-inspection boundaries above are agreed
|
||||
decisions. The vLLM discussion does not yet add provider behavior or an
|
||||
implementation commitment.
|
||||
|
||||
Reference in New Issue
Block a user