Add vLLM provider conventions to the observation roadmap

This commit is contained in:
2026-11-22 15:57:37 +00:00
parent 23ad9e2a26
commit 440d9a2c88

View File

@@ -1,6 +1,6 @@
# Invocation-Scoped Execution Observation
Status: agreed direction; vLLM scope under discussion; implementation pending.
Status: accepted scope; implementation pending.
## Purpose
@@ -9,8 +9,8 @@ operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should
participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost support initially limited to OpenRouter. Extend
OpenRouter token reporting with reasoning counts, documented cache reads and
writes, and a derived token cache-hit ratio.
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads
and writes, and a shared derived token cache-hit ratio.
The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline.
@@ -151,19 +151,16 @@ promptkit:
Normalize the final effective provider value by trimming whitespace and
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
Reject an unsupported nonempty normalized value as an ordinary configuration
error rather than silently using generic behavior. The agreed provider selections
are `openai-compatible` and `openrouter`; the possible `vllm` addition remains
under discussion.
error rather than silently using generic behavior. The supported selections for
this feature are `openai-compatible`, `openrouter`, and `vllm`.
Omitted/null/blank provider values select the same effective behavior as explicit
`provider: openai-compatible`. This is a runtime fallback, not an explicitly
sourced configuration value. Backend inspection retains the distinction between
an explicit provider declaration and an omitted/null/blank declaration.
The `openrouter` provider reuses the shared OpenAI-compatible foundation and adds
the OpenRouter response-metadata and diagnostic conventions below. Before
finalizing this roadmap, discuss the conventions and scope of a `vllm` selection;
this document does not yet select a vLLM contract.
The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation
and add their response-metadata and diagnostic conventions below.
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
the selected value through configuration capture and prepared execution, expose
@@ -201,11 +198,29 @@ Continue sending caller-selected supported settings and return backend errors
normally, without capability negotiation or fallback requests. This is the
existing text Chat Completions scope, not a claim to support the full OpenAI API.
### vLLM Provider
`provider: vllm` selects the shared generation path plus vLLM-specific usage
normalization and a default authentication diagnostic. It supplies no default
endpoint, model, API-key environment variable, or sampling settings, and does not
create a backend definition automatically.
Preserve caller-selected reasoning settings, structured-output requests, and
extra parameters. Do not automatically change chat-template arguments, enable
thinking, or translate model-specific controls based on the provider label.
No separate generation protocol is needed; vLLM documents an
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
The token mappings are specified below. Monetary cost remains unknown, not zero,
including when vLLM is locally hosted. This provider adds no price estimator,
server administration, metrics collection, or automatic capability/version probe.
### Smoke-Test Defaults And Explicit Overrides
When an authentication smoke test is explicitly requested, the selected provider
supplies its default probe convention. `provider: openrouter` supplies the
existing `openrouter_key` convention. Preserve explicit
existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models`
convention described below. Preserve explicit
`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint`
settings as overrides:
@@ -233,6 +248,30 @@ values. Configuration inspection preserves the distinction between configured
overrides and provider-supplied fallbacks, including their existing source
provenance.
### vLLM Authentication Diagnostic
The `vllm_models` adapter makes one GET request to `/models` relative to the
configured API base, supplying an available credential through the existing
credential rules. A base ending in `/v1` therefore produces `/v1/models`.
Existing explicit diagnostic endpoint overrides remain available. The adapter
can dispatch without a credential; selecting vLLM does not declare authentication
required or inapplicable.
Recognize a successful models-list response as acceptance of the probe request
and a 401 as rejection. Unrecognized responses leave acceptance unknown under
the existing diagnostic reporting model. Report observed credential presence and
response facts without claiming that authentication is enforced or that models
listing establishes inference authorization. Preserve the shared override rules,
including `kind: none`; explicit `vllm_models` selection also works without a
`vllm` provider declaration, just as explicit OpenRouter adapters do.
vLLM documents `/v1/models` as protected when its API-key authentication is
configured, while `/health` does not establish authentication acceptance. See its
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
Retain existing reachability and inference diagnostic behavior rather than
adding health monitoring. These requests occur only during explicitly requested
diagnostics, never during ordinary generation or engine construction.
### Diagnostic Migration
This changes the current authentication metadata contract: an authentication
@@ -263,9 +302,9 @@ requiring every provider to implement every capability is needed.
This is configurable selection among PromptKit-supported provider implementations,
not a public API for downstream registration of additional implementations.
## OpenRouter Token And Cache Reporting
## Token And Cache Reporting
### Response Mapping
### OpenRouter Response Mapping
Use OpenRouter's documented Chat Completions usage fields. Its
[response schema](https://openrouter.ai/docs/api_reference/overview),
@@ -290,15 +329,47 @@ nested value; an omitted nested value remains unknown.
Add `ReasoningTokens` to the public/domain token values, presence indicators,
and observed token usage, including conversions, result serialization, attempt
and terminal events, and group summaries. Initially, OpenRouter supplies this
and terminal events, and group summaries. OpenRouter and vLLM supply this
additional built-in response interpretation; the generic provider preserves its
existing behavior. Injected clients may supply the canonical count and its
presence through the existing model-client boundary.
### vLLM Response Mapping And Availability
Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens`
counts. For `provider: vllm`, map these additional fields relative to `usage`:
| Response field | PromptKit field |
| --- | --- |
| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` |
| `prompt_tokens_details.cached_tokens` | `CachedTokens` |
| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` |
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
describe the creation count as prompt tokens counted as local prefix-cache
writes. Normalize that token activity to `CacheWriteTokens` without implying a
billable write charge. Use the selected provider's documented field, not a sum
of alternative cache-write fields; an absent vLLM creation count stays unknown.
Detail availability depends on server version and configuration. Current vLLM
gates prompt details on `--enable-prompt-tokens-details` and reasoning details on
a configured reasoning parser. Document these reporting settings, but do not
configure or discover them automatically. Older or differently configured
servers may omit the fields; accept those responses with the affected counts
unknown. Do not infer reasoning counts from generated text or cache writes from
input tokens minus cache reads.
Once normalized, these counts use the shared presence, aggregation, and
cache-hit rules below. No vLLM-specific result type, ratio, or group accounting
is needed.
### Count Relationships And Availability
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
vLLM likewise reports its reasoning detail within the generated-token usage.
Cache reads and writes describe input-token breakdowns. Present these as subsets
or details rather than additional consumption to add to input/output totals.
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
@@ -363,8 +434,8 @@ For the built-in client, select this interpretation through `provider: openroute
regardless of backend identity or namespace. Do not infer USD semantics from an
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
Built-in cost interpretation is an extension unique to `openrouter` initially;
`openai-compatible` leaves cost unknown even if a response contains `usage.cost`.
No separate cost-reporting selector is needed. Aggregation remains
`openai-compatible` and `vllm` leave cost unknown even if a response contains
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains
provider-neutral after the model-client boundary supplies interpreted USD facts.
Use the cost included in the normal completion response. No additional usage
@@ -485,6 +556,17 @@ outside this scope.
decoding, including omitted/null/zero values and the nested value taking
precedence when a top-level value is also present. Preserve generic-provider
token mappings and existing field meanings.
- Add the `vllm` provider selection, reasoning and cache-detail mappings, and
`vllm_models` authentication adapter. Cover supplied/missing credentials,
recognized acceptance, rejection, unrecognized responses, provider defaults,
and explicit kind/endpoint overrides, including `none` and adapter selection
without a provider declaration. Verify that ordinary generation performs no
diagnostic requests and leaves monetary cost unknown.
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
details, including responses representative of servers without detail
reporting enabled. Verify `created_cache_tokens` mapping independently of
OpenRouter's cache-write field and carry normalized counts through results,
events, and groups. Do not require a live server or version discovery.
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
incomplete counts, zero input, inconsistent counts, independence from
cache-write availability, and token-weighted aggregate ratios. Verify subset
@@ -515,13 +597,16 @@ outside this scope.
aggregation example to the consumer observation guide, including cost
availability, units, and completeness. Document the reasoning/cache fields and
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
without double-counting. Document OpenRouter token and cost decoding in
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter
cost decoding in the
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the
framework format reference and relevant public GoDoc; update inspection and
diagnostic documentation with the implemented behavior. Include the diagnostic
migration in release notes.
migration in release notes. Include vLLM usage availability, server reporting
settings, the meaning of cache creation, and the limits of its authentication
diagnostic in the relevant consumer and integration documentation.
## Implementation Planning Guidance
@@ -530,7 +615,7 @@ Organize the eventual implementation plan around two workstreams:
1. Contextual observer selection, including ordinary, prepared, coordinated,
and nested execution boundaries and the existing group collector.
2. Backend provider selection and consolidation within `internal/llm`, including
smoke-test defaults and explicit overrides, followed by OpenRouter token
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
extensions and provider-reported cost through decoding, accounting, public
results, events, and group summaries, plus the derived cache-hit helper.
Include the corresponding configuration and API documentation.
@@ -539,8 +624,7 @@ The second area is more extensive than the context helper. Keep backend provider
configuration, diagnostic consolidation, token/cache reporting, and cost
accounting in separately reviewable implementation units rather than one large
implementation prompt.
Complete the remaining provider discussion before preparing that plan; avoid
coupling delivery of contextual observation to provider cost support.
Avoid coupling delivery of contextual observation to provider cost support.
Today, successful results accumulate generation responses in the runner, while
optional observation recorders account for dispatched attempts. Adding cost to
@@ -559,14 +643,3 @@ framing or content validation. This needs no general numeric-conversion layer.
Use the completion criteria above to select focused offline checks for each
unit. This roadmap owns target behavior; a separate implementation plan owns
the detailed work sequence.
## Topics For Further Discussion
Before finalizing the roadmap, discuss a `vllm` provider with vLLM-specific
conventions, identifying the response and diagnostic behavior to support from
reviewed upstream contracts and fixtures.
The `openai-compatible` default, normalization, unsupported-value handling,
override precedence, and configuration-inspection boundaries above are agreed
decisions. The vLLM discussion does not yet add provider behavior or an
implementation commitment.