Files
promptkit/docs/roadmap/invocation-scoped-token-usage.md

41 KiB

Invocation-Scoped Execution Observation

Status: accepted scope; implementation pending.

Purpose

Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically. Include optional provider-reported cost alongside token usage, with built-in cost decoding initially limited to OpenRouter and an explicit backend zero-cost fallback. Distinguish reported and configured cost sources in individual and aggregate metadata. Extend OpenRouter and vLLM token reporting with reasoning counts, documented cache reads and writes, and a shared derived token cache-hit ratio.

The motivating consumer is CommandKit's optional ckprompt integration, which could use this facility to report token usage for an entire command or pipeline. PromptKit supplies application-neutral observation and aggregation; consumers choose how to present or use the results.

Current Baseline

PromptKit v0.24.0 already provides:

  • ExecutionObserver and ComposeObservers for synchronous metadata callbacks and ordered composition.
  • ExecutionGroup for concurrent collection, detached snapshots, and aggregation of the first terminal event for each observed run.
  • Presence-aware prompt, completion, total, cached, and cache-write token counts, including aggregate accounting across retries and repairs.
  • Explicit selection through RunRequest.Observer, retained for prepared execution, and CoordinateRequest.Observer for coordinator-selected continuation runs.

The missing capabilities are contextual observer selection, cost accounting and source metadata, an explicit zero-cost declaration, reasoning-token counts, and a presence-aware cache-hit helper. OpenRouter cache writes also need their documented nested response mapping; the current client reads a top-level usage.cache_write_tokens field instead. Applications currently have to enroll each request explicitly, and PromptKit does not expose provider charges. The existing consumer observation guide describes that workflow.

Target Public API

Add a public context helper:

func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context

The returned context associates the observer with executions using that context or derived contexts that retain its values. It composes with any observer already associated with the context; it does not replace the inherited observer. A nil observer leaves the inherited selection unchanged.

No engine option, process-wide default, or replacement group type is needed. Extend the existing token types and result, event, and group surfaces with the token and cost metadata described below, plus a derived cache-hit helper.

The intended consumer workflow is:

var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)

err := runApplicationPipeline(ctx)
// After all application-owned work has finished, including on failure:
summary := group.Snapshot()

The consumer owns context propagation, waiting for its work, and using the snapshot and error. Calls using unrelated contexts are not automatically enrolled. Group coverage continues to describe observed members, not future work or overall application progress.

Observer Selection And Delivery

Composition

Preserve existing explicit observers. Deliver each event to the explicit request observer first, then any coordinator observer already composed with it, then contextual observers in registration order, from inherited to newly added.

Preserve the existing synchronous callback and panic behavior. Concurrent executions may invoke a shared observer concurrently; consumers remain responsible for synchronizing their callback state. ExecutionGroup already supports concurrent use.

Ordinary And Nested Execution

Run selects contextual observation from its invocation context. Each execution has its own observation identity, recorder, and accounting, including a nested execution using a context passed to an application callback or injected client.

Inherited observer selection is distinct from private execution state. Existing clearing of an enclosing execution's private state at a public execution boundary must preserve the contextual observer selection. Inheriting observation does not reuse an enclosing run's recorder, totals, or lifecycle.

Prepared Execution

RunPrepared selects contextual observation from the context supplied when the handle is executed. A handle prepared in one scope and executed in another participates in the execution scope, not the preparation scope.

The explicit RunRequest.Observer remains retained by a successfully prepared handle as today and composes with the execution-time contextual observers. Preparation itself emits no execution events. Discarded handles and rejected execution claims retain their existing observation behavior.

Coordinated Execution

Continuation runs receive contextual observation through their ordinary Run boundary. CoordinateRequest.Observer continues to select continuation runs only. Additional calls made by application callbacks participate when they use the scoped context, without being automatically selected by the coordinator's explicit observer.

Contextual observers are incorporated once per execution, rather than being re-added by coordination or another forwarding layer.

Duplicate Registration And Accounting

Registering the same callback through multiple explicit or contextual paths may deliver the same event more than once. Do not attempt to identify or deduplicate arbitrary Go function values.

Retain one execution lifecycle per run and reuse ExecutionGroup's existing terminal-event deduplication. Selecting the same group through multiple paths must not inflate its totals. Terminal usage is an aggregate, not an additional contribution to be added to attempt usage. These rules apply to cost as well as token counts.

Backend Provider Selection

Add an optional backend provider field identifying the API implementation whose conventions PromptKit should use. It identifies the service contacted, not the organization that trained the selected model. Backend IDs remain resource identities; names, namespaces, and endpoint hostnames do not select provider behavior.

promptkit:
  backends:
    - id: research-router
      provider: openrouter
      endpoint: https://openrouter.ai/api/v1
      api_key_env: OPENROUTER_API_KEY

Normalize the final effective provider value by trimming whitespace and lowercasing. Omitted, null, empty, and whitespace-only values are equivalent. Reject an unsupported nonempty normalized value as an ordinary configuration error rather than silently using generic behavior. The supported selections for this feature are openai-compatible, openrouter, and vllm.

Omitted/null/blank provider values select the same effective behavior as explicit provider: openai-compatible. This is a runtime fallback, not an explicitly sourced configuration value. Backend inspection retains the distinction between an explicit provider declaration and an omitted/null/blank declaration.

The openrouter and vllm providers reuse the shared OpenAI-compatible foundation and add their response-metadata and diagnostic conventions below.

Resolve the field through ordinary ConfigKit composition and inheritance. Carry the selected value through configuration capture and prepared execution, expose it in backend inspection, and declare it in the bundled OpenRouter backend. It is PromptKit metadata, not an outbound extra_params entry. Selecting a provider does not choose a model or alter the configured endpoint. An endpoint override retains the provider selection.

Default OpenAI-Compatible Provider

openai-compatible names PromptKit's current supported text Chat Completions conventions. Preserve the existing shared behavior documented in the integration contract, including:

  • JSON POST requests to /chat/completions relative to the configured base;
  • existing optional bearer-credential handling, including explicitly required credential settings when selected;
  • text messages, explicitly selected generation controls and structured-output requests, and configured extra parameters;
  • first-choice text extraction, existing supported token counts and their presence-aware observation; and
  • existing response validation, error handling, timeouts, cancellation, and transient retries.

This selection supplies no default endpoint, model, API-key environment variable, or sampling settings. It does not decode provider-reported monetary cost or provide a default authentication-probe adapter. The backend zero-cost declaration below remains available independently of provider selection. Without an explicit authentication adapter, report the existing unsupported diagnostic outcome rather than assuming authentication is inapplicable. Existing reachability and inference diagnostics and explicit authentication adapters remain available.

"Lowest common denominator" identifies the shared protocol foundation; it does not remove existing optional features or promise every backend supports them. Continue sending caller-selected supported settings and return backend errors normally, without capability negotiation or fallback requests. This is the existing text Chat Completions scope, not a claim to support the full OpenAI API.

vLLM Provider

provider: vllm selects the shared generation path plus vLLM-specific usage normalization and a default authentication diagnostic. It supplies no default endpoint, model, API-key environment variable, or sampling settings, and does not create a backend definition automatically.

Preserve caller-selected reasoning settings, structured-output requests, and extra parameters. Do not automatically change chat-template arguments, enable thinking, or translate model-specific controls based on the provider label. No separate generation protocol is needed; vLLM documents an OpenAI-compatible Chat Completions API.

The token mappings are specified below. vLLM supplies no provider-reported cost; cost remains unknown unless the backend explicitly declares the zero-cost fallback below. Local hosting alone does not select that fallback. This provider adds no price estimator, server administration, metrics collection, or automatic capability/version probe.

Smoke-Test Defaults And Explicit Overrides

When an authentication smoke test is explicitly requested, the selected provider supplies its default probe convention. provider: openrouter supplies the existing openrouter_key convention; provider: vllm supplies the vllm_models convention described below. Preserve explicit smoke_tests.authentication.kind and smoke_tests.authentication.endpoint settings as overrides:

  • A supplied kind selects the diagnostic adapter instead of the provider default. kind: none continues to declare authentication testing inapplicable.
  • An omitted/null kind uses the provider default, if available. This permits an endpoint override without repeating the provider's default kind.
  • A supplied endpoint overrides the selected adapter's default destination. An omitted/null endpoint uses that adapter's existing endpoint fallback.

An absent/null authentication block likewise leaves selection to the provider default. Clearing the block removes its explicit overrides; kind: none is the explicit way to declare authentication inapplicable despite a provider default. Without a provider default or explicit adapter, retain existing unsupported diagnostic behavior. Existing explicit adapters continue to work without a provider declaration. Reachability and inference-default settings keep their existing behavior.

These are conventions for explicitly requested diagnostics. Loading a backend, constructing an engine, and ordinary generation do not initiate probes.

Resolve provider-supplied diagnostic defaults at the provider boundary without writing them into the loaded configuration or recording them as explicit source values. Configuration inspection preserves the distinction between configured overrides and provider-supplied fallbacks, including their existing source provenance.

vLLM Authentication Diagnostic

The vllm_models adapter makes one GET request to /models relative to the configured API base, supplying an available credential through the existing credential rules. A base ending in /v1 therefore produces /v1/models. Existing explicit diagnostic endpoint overrides remain available. The adapter can dispatch without a credential; selecting vLLM does not declare authentication required or inapplicable.

Recognize a successful models-list response as acceptance of the probe request and a 401 as rejection. Unrecognized responses leave acceptance unknown under the existing diagnostic reporting model. Report observed credential presence and response facts without claiming that authentication is enforced or that models listing establishes inference authorization. Preserve the shared override rules, including kind: none; explicit vllm_models selection also works without a vllm provider declaration, just as explicit OpenRouter adapters do.

vLLM documents /v1/models as protected when its API-key authentication is configured, while /health does not establish authentication acceptance. See its authentication documentation. Retain existing reachability and inference diagnostic behavior rather than adding health monitoring. These requests occur only during explicitly requested diagnostics, never during ordinary generation or engine construction.

Diagnostic Migration

This changes the current authentication metadata contract: an authentication object currently requires kind, and clearing the object removes its configured adapter. Under this roadmap, endpoint-only overrides become valid, and clearing the object restores any selected provider's default. kind: none suppresses that default. Explain these changes in the framework format documentation and release notes, alongside the new provider field.

Provider Behavior Ownership

Centralize provider-specific behavior within internal/llm, which already owns OpenAI-compatible transport and OpenRouter authentication-response recognition. Keep common request construction, credentials, HTTP execution, framing, and common response fields shared. Provider-specific code owns additional response metadata interpretation and authentication-probe conventions.

Use one provider-selection mechanism inside that package. Generation, observation, and smoke-test coordination consume provider-neutral facts or invoke generic diagnostic operations rather than repeating provider dispatch and response interpretation. Explicit diagnostic adapter selection remains supported at this boundary. Update the current OpenRouter-specific diagnostic collaborator method to a generic operation as part of this consolidation.

Provider-specific files and straightforward dispatch are sufficient initially. No new internal/providers package, registration framework, or broad interface requiring every provider to implement every capability is needed. This is configurable selection among PromptKit-supported provider implementations, not a public API for downstream registration of additional implementations.

Token And Cache Reporting

OpenRouter Response Mapping

Use OpenRouter's documented Chat Completions usage fields. Its response schema, usage-accounting guide, and caching guide define the following mapping, relative to usage:

Quantity Response field PromptKit field
Input tokens prompt_tokens Existing PromptTokens
Output tokens, including reasoning completion_tokens Existing CompletionTokens
Reasoning tokens completion_tokens_details.reasoning_tokens New ReasoningTokens
Cache reads prompt_tokens_details.cached_tokens Existing CachedTokens
Cache writes prompt_tokens_details.cache_write_tokens Existing CacheWriteTokens
Total tokens total_tokens Existing TotalTokens

Preserve CachedTokens as the public field name and clarify its meaning as cache-read tokens. Correct the OpenRouter cache-write mapping to the nested field above; retain the generic provider's existing top-level handling. Do not add the nested and top-level values together. OpenRouter uses its documented nested value; an omitted nested value remains unknown.

Add ReasoningTokens to the public/domain token values, presence indicators, and observed token usage, including conversions, result serialization, attempt and terminal events, and group summaries. OpenRouter and vLLM supply this additional built-in response interpretation; the generic provider preserves its existing behavior. Injected clients may supply the canonical count and its presence through the existing model-client boundary.

vLLM Response Mapping And Availability

Reuse the ordinary prompt_tokens, completion_tokens, and total_tokens counts. For provider: vllm, map these additional fields relative to usage:

Response field PromptKit field
completion_tokens_details.reasoning_tokens ReasoningTokens
prompt_tokens_details.cached_tokens CachedTokens
prompt_tokens_details.created_cache_tokens CacheWriteTokens

vLLM's response construction uses created_cache_tokens, unlike OpenRouter's cache_write_tokens. Its output definitions describe the creation count as prompt tokens counted as local prefix-cache writes. Normalize that token activity to CacheWriteTokens without implying a billable write charge. Use the selected provider's documented field, not a sum of alternative cache-write fields; an absent vLLM creation count stays unknown.

Detail availability depends on server version and configuration. Current vLLM gates prompt details on --enable-prompt-tokens-details and reasoning details on a configured reasoning parser. Document these reporting settings, but do not configure or discover them automatically. Older or differently configured servers may omit the fields; accept those responses with the affected counts unknown. Do not infer reasoning counts from generated text or cache writes from input tokens minus cache reads.

Once normalized, these counts use the shared presence, aggregation, and cache-hit rules below. No vLLM-specific result type, ratio, or group accounting is needed.

Count Relationships And Availability

Reasoning tokens are included in completion tokens, as explained in OpenRouter's reasoning documentation. vLLM likewise reports its reasoning detail within the generated-token usage. Cache reads and writes describe input-token breakdowns. Present these as subsets or details rather than additional consumption to add to input/output totals. Preserve the provider's reported TotalTokens; do not recompute or reconcile it.

Consumers can derive non-reasoning output as completion tokens minus reasoning tokens when both counts are available and consistent. No additional stored non-reasoning count is needed in this scope. Capturing reasoning-token counts does not capture reasoning text or change reasoning request settings.

Reuse existing per-field presence and aggregation rules: omitted/null means unknown, explicit zero means known zero, and aggregate Known and Complete flags describe the reported contributions across dispatched attempts. Include initial generations, retries, and repairs through the existing observation accounting, retaining terminal-only group aggregation and deduplication. Preserve the existing distinction between integer-only result usage and presence-aware observations; zero in TokenUsage alone does not establish provider reporting.

Token Cache-Hit Ratio

Add this derived helper on the existing presence-aware usage type:

func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)

The ratio is CachedTokens / PromptTokens, represented as a fraction from zero to one. It measures the proportion of input tokens served from cache, not the proportion of requests with a cache hit or monetary savings. The Boolean reports whether a ratio is available; an unavailable result returns zero and false.

Return an available ratio only when both counts are known and complete, input tokens are positive, and cache reads are between zero and input tokens inclusive. Zero cache reads with known positive input yields an available zero ratio. Missing or incomplete counts, zero input, or inconsistent counts leave it unavailable; preserve the raw reported counts rather than clamping them or failing generation. Cache-write availability does not affect this calculation.

For a run or group, divide the aggregated cache-read count by the aggregated input count; do not average per-attempt or per-run percentages. Do not use reads / (reads + writes), because cache writes need not account for all uncached input. The same helper works on attempts, terminal events, and group summaries without storing another independently maintained field.

The ratio describes the included observations. A group ratio does not supersede TerminalCoverageComplete or imply that all application work has finished. For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts are known and complete, even if cache writes are unknown. Output 840 with reasoning 600 still represents 840 output tokens, not 1,440.

Optional Cost Reporting

Initial Provider Support

Capture OpenRouter Chat Completions' top-level usage.cost, which OpenRouter defines as the amount charged to the account, denominated in USD. See its usage accounting reference and USD cost explanation. Do not add cost_details breakdowns or combine upstream inference cost with the top-level charge; those describe different facts and could double-count cost.

For the built-in client, select this interpretation through provider: openrouter, regardless of backend identity or namespace. Do not infer USD semantics from an arbitrary backend's field named cost, or add endpoint discovery or verification. Built-in provider-reported cost interpretation is an extension unique to openrouter initially; openai-compatible and vllm do not interpret usage.cost. Without an applicable zero-cost declaration, their built-in cost remains unknown. No separate provider cost-decoding selector is needed. Aggregation remains provider-neutral after the model-client boundary supplies interpreted USD facts.

Use the cost included in the normal completion response. No additional usage lookup, pricing request, or opt-in provider request parameter is needed.

Explicit Backend Zero-Cost Declaration

Add optional Boolean backend metadata zero_cost, independent of provider:

promptkit:
  backends:
    - id: local-inference
      provider: vllm
      endpoint: http://localhost:8000/v1
      zero_cost: true

For each dispatched attempt, use a usable provider/model-client-reported cost first, including an explicitly reported zero. If no usable reported cost is available and the effective backend has zero_cost: true, supply USD zero with a configured-zero source. Omitted, null, or false leaves the fallback disabled; it does not suppress a reported cost. Apply this policy to built-in and injected client execution using the selected backend, regardless of its provider type.

The fallback also applies to dispatched attempts that fail, are canceled, or produce no decodable response. It is an operator declaration, not a provider observation. Non-dispatched work contributes no cost. Do not infer this setting from a local address, provider name, token counts, or missing credentials, and do not install it automatically on any backend. Without a selected backend declaration, no configured-zero fallback applies.

Resolve the field through ordinary final ConfigKit composition and inheritance; include it in typed backend helpers, configuration capture, and backend inspection. Retain the effective policy through prepared execution without reloading configuration. It is local accounting metadata, not an outbound request parameter. Preserve explicit/null/omitted source provenance in inspection rather than manufacturing an explicit false value for the disabled fallback.

This supports an explicit zero declaration, not configurable input/output or cache-read/write token rates, fixed nonzero request charges, pricing tables, or cost estimation. It does not alter credentials, generation, or token reporting.

Monetary Representation

Use float64 amounts in USD. Make the unit explicit in public field names such as CostUSD, rather than introducing a generic amount with an implicit currency. At the individual model-client response boundary, use an optional amount (*float64): nil means unknown, and a pointer to zero means reported zero. Injected clients may supply the same optional USD value without needing an OpenRouter backend; no provider-specific interface is needed.

Expose cost separately from TokenUsage and TokenUsagePresence, whose fields remain token counts. For run results, execution events, and group summaries, use a small presence-aware cost value with Value float64, Known bool, and Complete bool, following the existing observed-count conventions, plus the source metadata below. Successful run results should expose cost even when no observer was registered; observers also provide per-attempt and terminal cost, including available facts on failure.

Binary floating point is sufficient for operational telemetry and keeps decoding, addition, and the public API lightweight. Do not round to cents or another fixed scale during capture or aggregation; consumers choose display precision. Decimal strings or a decimal arithmetic dependency would be appropriate for exact financial ledgers, while fixed-scale integers would require a selected precision and rounding policy. Neither is warranted for this reporting feature. The API does not promise exact decimal arithmetic or invoice reconciliation.

Cost-Source Metadata

Give the presence-aware cost value a Source discriminator with these meanings:

  • provider: a cost supplied by the provider or injected model-client response. This records the reporting path, not independent verification of a charge.
  • configured_zero: zero supplied by the explicit backend fallback.
  • mixed: an aggregate includes both provider and configured-zero contributions.
  • Empty: no known contribution has a source. This covers unknown cost and the known, complete zero aggregate when no attempts were dispatched.

Derive aggregate source from the union of known contribution sources, including zero-valued contributions. Unknown contributions supply no source but affect completeness. A provider-only subtotal can therefore be incomplete without being mixed. Carry this metadata consistently through attempt events, terminal events, successful results, and group summaries. Reuse existing aggregation and terminal deduplication rather than adding a parallel collector or cost history.

Consumers can distinguish reported charges, configured zeros, and mixed totals without treating configured values as provider reports. Source metadata is separate from configuration inspection and does not require provenance traversal during execution.

Presence, Aggregation, And Failures

  • An absent or null provider cost is unknown unless the backend's explicit zero-cost fallback supplies it. A reported numeric zero remains known zero with a provider source, even when the fallback is enabled.
  • On a dispatched attempt, a usable reported cost or configured-zero fallback is known and complete for that attempt. Cost completeness is independent of token-field completeness and does not assert provider reporting.
  • Sum known costs across dispatched attempts, including initial generations, retries, and repairs. Known means at least one attempt has an available cost from either source; Complete means every dispatched attempt does under this policy. Unknown values contribute no amount, but make the total incomplete. With no dispatched attempts, follow the existing known, complete zero convention, with an empty source.
  • Group summaries reuse existing terminal-only aggregation and deduplication: sum known terminal values, OR their Known flags, and AND their Complete flags. Retain existing empty-group and terminal-coverage semantics. A subtotal from OpenRouter mixed with unreported local-backend cost is incomplete unless those local attempts have an explicit configured-zero fallback. Combine cost-source metadata as described above without changing token accounting.
  • Keep valid cost observed from a decoded successful envelope when a later choice/content check fails, as with token usage. Invalid framing and non-2xx error bodies continue to supply no provider usage, but the configured-zero fallback still applies to dispatched attempts. Without that fallback, cancellation or failure may leave incomplete cost reporting even when billable upstream work occurred.
  • Treat an unusable optional cost value, including a nonnumeric value or one not representable as a finite float64, as unknown rather than making an otherwise valid generation fail; the zero-cost fallback can then apply. This does not relax existing response framing or content validation. An injected client likewise cannot supply a nonfinite usable cost.

Completeness describes coverage by available reported or explicitly configured cost, not verification of the provider's billing or proof that every provider reported a charge. Preserve usable reported amounts without pricing inference or reconciliation.

For a successful run, result cost and terminal-event cost describe the same dispatched attempts and must agree on value, presence, completeness, and source. Result cost must also be the same with observation enabled or disabled. For example, an unreported failed attempt followed by a successful attempt reporting USD 0.002 yields Value: 0.002, Known: true, Complete: false, and a provider source when no zero-cost fallback applies. Apply the same accounting through repairs and ordinary or prepared execution. An invocation with fully reported OpenRouter charges and configured-zero local calls can have a complete subtotal with a mixed source, without claiming all providers reported costs.

Group cost completeness concerns the terminal observations included in its subtotal. If two completed runs reported all their costs but a third observed run has no terminal event, the subtotal may have Complete: true while TerminalCoverageComplete is false. Preserve this distinction using the existing coverage field; no additional completion flag or group lifecycle is needed.

Accounting And Responsibility Boundaries

Reuse existing events, token presence/completeness, and group summaries, extending them with the cost facts above. Available usage remains observable after an error or cancellation under the existing execution-observation behavior; this does not promise that upstream providers report usage for every attempt.

PromptKit owns observer selection, event generation, and aggregation. CommandKit and other consumers own invocation lifecycle, CLI flags, presentation, output delivery, and interpretation. This feature adds no CommandKit dependency or application-specific concepts to PromptKit.

Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation by this change. Reasoning-text capture, changes to reasoning or caching request behavior, additional built-in cost providers, configurable token-rate estimation, fixed nonzero request pricing, multiple currencies, currency conversion, pricing estimates, billing reconciliation, persistent history, and logging infrastructure are outside this scope.

Work Needed And Completion Criteria

  • Add the context helper and connect observer selection to ordinary and prepared execution, keeping contextual selection separate from private run state.
  • Preserve existing explicit selection and coordinator behavior while applying the composition and lifetime rules above. Preserve existing observer selection when no contextual observer is selected.
  • Add provider to the backend configuration model, typed source helpers, capture, inspection, and execution plumbing. Update the bundled OpenRouter backend to select it explicitly. Retain the selection through prepared runs.
  • Add zero_cost to backend settings, typed helpers, capture, inspection, and prepared-execution plumbing. Apply the fallback outside provider-specific decoding so it works with built-in and injected clients. Verify composition and inheritance, omitted/null/false behavior, explicit overrides, provider independence, retained inspection provenance, and absence from provider bodies.
  • Consolidate provider-specific response and authentication-probe behavior in internal/llm; keep coordination and accounting provider-neutral. Cover provider-default probes, explicit kind and endpoint overrides, kind: none, and existing explicit probes without a provider selection.
  • Verify provider selection independently of backend names and namespaces, including renamed and inherited backends, equivalent omitted/null/blank values, trimming and case normalization, rejection of unsupported nonempty values, and retention through endpoint overrides. Confirm provider metadata is not serialized into the completion request body. Verify that explicit openai-compatible and omitted/null/blank selection preserve the same existing shared behavior without manufacturing explicit configuration provenance.
  • Verify that provider diagnostic defaults do not become explicit values in the loaded configuration or replace source provenance in inspection. Cover the changed clearing behavior and endpoint-only authentication overrides.
  • Carry OpenRouter reasoning-token counts through token values, presence, results, events, and group summaries. Verify nested OpenRouter cache-write decoding, including omitted/null/zero values and the nested value taking precedence when a top-level value is also present. Preserve generic-provider token mappings and existing field meanings.
  • Add the vllm provider selection, reasoning and cache-detail mappings, and vllm_models authentication adapter. Cover supplied/missing credentials, recognized acceptance, rejection, unrecognized responses, provider defaults, and explicit kind/endpoint overrides, including none and adapter selection without a provider declaration. Verify that ordinary generation performs no diagnostic requests and leaves monetary cost unknown unless the backend explicitly declares the zero-cost fallback.
  • Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache details, including responses representative of servers without detail reporting enabled. Verify created_cache_tokens mapping independently of OpenRouter's cache-write field and carry normalized counts through results, events, and groups. Do not require a live server or version discovery.
  • Add ObservedTokenUsage.CacheHitRatio. Cover known zero reads, unknown and incomplete counts, zero input, inconsistent counts, independence from cache-write availability, and token-weighted aggregate ratios. Verify subset accounting without adding reasoning or cached tokens to the reported totals.
  • Carry optional OpenRouter cost through response decoding, internal/public conversions, successful run results, attempt and terminal events, and existing group summaries. Keep result cost available without registering an observer. Preserve existing token fields and accounting.
  • Carry cost-source metadata through results, events, conversions, and group snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs, including zero-valued reported contributions and empty/non-dispatched cases. Verify source union and terminal deduplication independently of cost value.
  • Verify that usable reported cost wins over zero_cost: true, and missing, null, or unusable reported cost permits fallback. Cover dispatched failures, cancellation, undecodable responses, retries, repairs, ordinary/prepared execution, and observer-enabled/disabled results. Verify that mixed OpenRouter and declared zero-cost local calls can produce complete cost totals while token completeness remains independent.
  • Verify that successful-result cost agrees with terminal-event cost and is unchanged when observation is disabled, including the unreported-retry example above. Verify group snapshots with fully reported completed runs alongside a member without a terminal observation.
  • Cover ordinary and prepared execution, coordinated continuations, nested calls using derived contexts, concurrent calls, multiple engines sharing a scope, and independent scopes on a reused engine.
  • Cover callback ordering, nested contextual registration, nil registration, duplicate group enrollment, prepared-handle scope selection, and existing preparation/discard/rejected-claim behavior.
  • Verify aggregate accounting through retries and repairs, available usage after failures or cancellation, and known zero versus unknown or incomplete counts. Use offline fixtures and injected clients; live inference is unnecessary.
  • Cover fractional USD values, explicit zero, omitted/null and unusable cost, generic built-in selection, optional injected-client cost, mixed known and unknown totals, repair/retry aggregation, and duplicate group registration. Use tolerant comparisons where floating-point addition requires them; do not turn telemetry into an exact-decimal guarantee.
  • Document the exact helper contract in GoDoc and add a focused invocation-level aggregation example to the consumer observation guide, including cost availability, units, completeness, and source. Explain that configured zeros can make cost complete without provider reporting. Document the reasoning/cache fields and cache-hit helper in their owning GoDoc and illustrate input/output breakdowns without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter cost decoding in the integration contract, and update internal observation and model-client documentation for changed flows. Document provider, its smoke-test defaults and override semantics in the framework format reference and relevant public GoDoc; update inspection and diagnostic documentation with the implemented behavior. Document zero_cost and its fallback precedence, inspection behavior, and prepared lifetime in the relevant format, GoDoc, and consumer owners. Include the diagnostic migration in release notes. Include vLLM usage availability, server reporting settings, the meaning of cache creation, and the limits of its authentication diagnostic in the relevant consumer and integration documentation.

Implementation Planning Guidance

Organize the eventual implementation plan around two workstreams:

  1. Contextual observer selection, including ordinary, prepared, coordinated, and nested execution boundaries and the existing group collector.
  2. Backend provider selection and consolidation within internal/llm, including smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token extensions, provider-reported cost, and configured-zero fallback through decoding, accounting, public results, events, and group summaries, plus the derived cache-hit helper. Include the corresponding configuration and API documentation.

The second area is more extensive than the context helper. Keep backend provider configuration, diagnostic consolidation, token/cache reporting, and cost/source accounting in separately reviewable implementation units rather than one large implementation prompt. Avoid coupling delivery of contextual observation to provider cost support.

Today, successful results accumulate generation responses in the runner, while optional observation recorders account for dispatched attempts. Adding cost to response objects alone does not preserve completeness across unreported retries. Reuse a small accounting component for result and observation paths so they apply the same rules, including when no observer is selected. Preserve existing token accounting rather than expanding this work into a redesign of its public contract.

Keep tolerant optional-cost decoding local to the model-client boundary. Adding a plain *float64 JSON field to the typed envelope would make a nonnumeric or overflowing value fail decoding of the entire response. Decode the optional field separately so an unusable provider cost becomes unknown before applying any configured-zero fallback, without changing existing framing or content validation. This needs no general numeric-conversion layer.

Use the completion criteria above to select focused offline checks for each unit. This roadmap owns target behavior; a separate implementation plan owns the detailed work sequence.