Files
promptkit/docs/roadmap/invocation-scoped-token-usage.md

766 lines
43 KiB
Markdown
Raw Normal View History

# Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending.
## Purpose
Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should
participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost decoding initially limited to OpenRouter and an
explicit backend zero-cost fallback. Distinguish reported and configured cost
sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
reporting with reasoning counts, documented cache reads and writes, and a shared
derived token cache-hit ratio.
The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline.
PromptKit supplies application-neutral observation and aggregation; consumers
choose how to present or use the results.
## Current Baseline
PromptKit v0.24.0 already provides:
- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous
metadata callbacks and ordered composition.
- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached
snapshots, and aggregation of the first terminal event for each observed run.
- Presence-aware prompt, completion, total, cached, and cache-write token counts,
including aggregate accounting across retries and repairs.
- Explicit selection through `RunRequest.Observer`, retained for prepared
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs.
The missing capabilities are contextual observer selection, cost accounting and
source metadata, an explicit zero-cost declaration, reasoning-token counts, and
a presence-aware cache-hit helper.
OpenRouter cache writes also need their documented nested response mapping;
the current client reads a top-level `usage.cache_write_tokens` field instead.
Applications currently have to enroll each request explicitly, and PromptKit
does not expose provider charges. The existing
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
describes that workflow.
## Target Public API
Add a public context helper:
```go
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
```
The returned context associates the observer with executions using that context
or derived contexts that retain its values. It composes with any observer already
associated with the context; it does not replace the inherited observer. A nil
observer leaves the inherited selection unchanged.
No engine option, process-wide default, or replacement group type is needed.
Extend the existing token types and result, event, and group surfaces with the
token and cost metadata described below, plus a derived cache-hit helper.
The intended consumer workflow is:
```go
var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
err := runApplicationPipeline(ctx)
// After all application-owned work has finished, including on failure:
summary := group.Snapshot()
```
The consumer owns context propagation, waiting for its work, and using the
snapshot and error. Calls using unrelated contexts are not automatically
enrolled. Group coverage continues to describe observed members, not future
work or overall application progress.
## Observer Selection And Delivery
### Composition
Preserve existing explicit observers. Deliver each event to the explicit request
observer first, then any coordinator observer already composed with it, then
contextual observers in registration order, from inherited to newly added.
Preserve the existing synchronous callback and panic behavior. Concurrent
executions may invoke a shared observer concurrently; consumers remain
responsible for synchronizing their callback state. `ExecutionGroup` already
supports concurrent use.
### Ordinary And Nested Execution
`Run` selects contextual observation from its invocation context. Each execution
has its own observation identity, recorder, and accounting, including a nested
execution using a context passed to an application callback or injected client.
Inherited observer selection is distinct from private execution state. Existing
clearing of an enclosing execution's private state at a public execution boundary
must preserve the contextual observer selection. Inheriting observation does not
reuse an enclosing run's recorder, totals, or lifecycle.
### Prepared Execution
`RunPrepared` selects contextual observation from the context supplied when the
handle is executed. A handle prepared in one scope and executed in another
participates in the execution scope, not the preparation scope.
The explicit `RunRequest.Observer` remains retained by a successfully prepared
handle as today and composes with the execution-time contextual observers.
Preparation itself emits no execution events. Discarded handles and rejected
execution claims retain their existing observation behavior.
### Coordinated Execution
Continuation runs receive contextual observation through their ordinary `Run`
boundary. `CoordinateRequest.Observer` continues to select continuation runs
only. Additional calls made by application callbacks participate when they use
the scoped context, without being automatically selected by the coordinator's
explicit observer.
Contextual observers are incorporated once per execution, rather than being
re-added by coordination or another forwarding layer.
### Duplicate Registration And Accounting
Registering the same callback through multiple explicit or contextual paths may
deliver the same event more than once. Do not attempt to identify or deduplicate
arbitrary Go function values.
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
terminal-event deduplication. Selecting the same group through multiple paths
must not inflate its totals. Terminal usage is an aggregate, not an additional
contribution to be added to attempt usage. These rules apply to cost as well as
token counts.
## Backend Provider Selection
Add an optional backend `provider` field identifying the API implementation
whose conventions PromptKit should use. It identifies the service contacted,
not the organization that trained the selected model. Backend IDs remain resource
identities; names, namespaces, and endpoint hostnames do not select provider
behavior.
```yaml
promptkit:
backends:
- id: research-router
provider: openrouter
endpoint: https://openrouter.ai/api/v1
api_key_env: OPENROUTER_API_KEY
```
Normalize the final effective provider value by trimming whitespace and
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
Reject an unsupported nonempty normalized value as an ordinary configuration
error rather than silently using generic behavior. The supported selections for
this feature are `openai-compatible`, `openrouter`, and `vllm`.
Omitted/null/blank provider values select the same effective behavior as explicit
`provider: openai-compatible`. This is a runtime fallback, not an explicitly
sourced configuration value. Backend inspection retains the distinction between
an explicit provider declaration and an omitted/null/blank declaration.
The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation
and add their response-metadata and diagnostic conventions below.
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
the selected value through configuration capture and prepared execution, expose
it in backend inspection, and declare it in the bundled OpenRouter backend.
It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a
provider does not choose a model or alter the configured endpoint. An endpoint
override retains the provider selection.
### Default OpenAI-Compatible Provider
`openai-compatible` names PromptKit's current supported text Chat Completions
conventions. Preserve the existing shared behavior documented in the
[integration contract](../integrations/openai-compatible-chat.md), including:
- JSON POST requests to `/chat/completions` relative to the configured base;
- existing optional bearer-credential handling, including explicitly required
credential settings when selected;
- text messages, explicitly selected generation controls and structured-output
requests, and configured extra parameters;
- first-choice text extraction, existing supported token counts and their
presence-aware observation; and
- existing response validation, error handling, timeouts, cancellation, and
transient retries.
This selection supplies no default endpoint, model, API-key environment variable,
or sampling settings. It does not decode provider-reported monetary cost or
provide a default authentication-probe adapter. The backend zero-cost declaration
below remains available independently of provider selection. Without an explicit
authentication adapter, report the existing unsupported diagnostic outcome rather
than assuming authentication is inapplicable. Existing reachability and inference
diagnostics and explicit authentication adapters remain available.
"Lowest common denominator" identifies the shared protocol foundation; it does
not remove existing optional features or promise every backend supports them.
Continue sending caller-selected supported settings and return backend errors
normally, without capability negotiation or fallback requests. This is the
existing text Chat Completions scope, not a claim to support the full OpenAI API.
### vLLM Provider
`provider: vllm` selects the shared generation path plus vLLM-specific usage
normalization and a default authentication diagnostic. It supplies no default
endpoint, model, API-key environment variable, or sampling settings, and does not
create a backend definition automatically.
Preserve caller-selected reasoning settings, structured-output requests, and
extra parameters. Do not automatically change chat-template arguments, enable
thinking, or translate model-specific controls based on the provider label.
No separate generation protocol is needed; vLLM documents an
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
The token mappings are specified below. vLLM supplies no provider-reported cost;
cost remains unknown unless the backend explicitly declares the zero-cost
fallback below. Local hosting alone does not select that fallback. This provider
adds no price estimator, server administration, metrics collection, or automatic
capability/version probe.
### Smoke-Test Defaults And Explicit Overrides
When an authentication smoke test is explicitly requested, the selected provider
supplies its default probe convention. `provider: openrouter` supplies the
existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models`
convention described below. Preserve explicit
`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint`
settings as overrides:
- A supplied kind selects the diagnostic adapter instead of the provider default.
`kind: none` continues to declare authentication testing inapplicable.
- An omitted/null kind uses the provider default, if available. This permits an
endpoint override without repeating the provider's default kind.
- A supplied endpoint overrides the selected adapter's default destination.
An omitted/null endpoint uses that adapter's existing endpoint fallback.
An absent/null authentication block likewise leaves selection to the provider
default. Clearing the block removes its explicit overrides; `kind: none` is the
explicit way to declare authentication inapplicable despite a provider default.
Without a provider default or explicit adapter, retain existing unsupported
diagnostic behavior. Existing explicit adapters continue to work without a
provider declaration. Reachability and inference-default settings keep their
existing behavior.
These are conventions for explicitly requested diagnostics. Loading a backend,
constructing an engine, and ordinary generation do not initiate probes.
Resolve provider-supplied diagnostic defaults at the provider boundary without
writing them into the loaded configuration or recording them as explicit source
values. Configuration inspection preserves the distinction between configured
overrides and provider-supplied fallbacks, including their existing source
provenance.
### vLLM Authentication Diagnostic
The `vllm_models` adapter makes one GET request to `/models` relative to the
configured API base, supplying an available credential through the existing
credential rules. A base ending in `/v1` therefore produces `/v1/models`.
Existing explicit diagnostic endpoint overrides remain available. The adapter
can dispatch without a credential; selecting vLLM does not declare authentication
required or inapplicable.
Recognize HTTP 200 with a valid JSON object containing `object: "list"` and an
array-valued `data` as acceptance of the probe request, and a 401 as rejection.
An empty `data` array counts as acceptance; recognition does not require a
particular model or inspect model capabilities. Reuse the existing bounded
diagnostic decoding and framing rules. Unrecognized responses leave acceptance
unknown under the existing diagnostic reporting model. Report observed credential presence and
response facts without claiming that authentication is enforced or that models
listing establishes inference authorization. Preserve the shared override rules,
including `kind: none`; explicit `vllm_models` selection also works without a
`vllm` provider declaration, just as explicit OpenRouter adapters do.
vLLM documents `/v1/models` as protected when its API-key authentication is
configured, while `/health` does not establish authentication acceptance. See its
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
Retain existing reachability and inference diagnostic behavior rather than
adding health monitoring. These requests occur only during explicitly requested
diagnostics, never during ordinary generation or engine construction.
### Diagnostic Migration
This changes the current authentication metadata contract: an authentication
object currently requires `kind`, and clearing the object removes its configured
adapter. Under this roadmap, endpoint-only overrides become valid, and clearing
the object restores any selected provider's default. `kind: none` suppresses
that default. Explain these changes in the framework format documentation and
release notes, alongside the new provider field.
### Provider Behavior Ownership
Centralize provider-specific behavior within `internal/llm`, which already owns
OpenAI-compatible transport and OpenRouter authentication-response recognition.
Keep common request construction, credentials, HTTP execution, framing, and
common response fields shared. Provider-specific code owns additional response
metadata interpretation and authentication-probe conventions.
Use one provider-selection mechanism inside that package. Generation,
observation, and smoke-test coordination consume provider-neutral facts or invoke
generic diagnostic operations rather than repeating provider dispatch and
response interpretation. Explicit diagnostic adapter selection remains supported
at this boundary. Update the current OpenRouter-specific diagnostic collaborator
method to a generic operation as part of this consolidation.
Provider-specific files and straightforward dispatch are sufficient initially.
No new `internal/providers` package, registration framework, or broad interface
requiring every provider to implement every capability is needed.
This is configurable selection among PromptKit-supported provider implementations,
not a public API for downstream registration of additional implementations.
## Token And Cache Reporting
### OpenRouter Response Mapping
Use OpenRouter's documented Chat Completions usage fields. Its
[response schema](https://openrouter.ai/docs/api_reference/overview),
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching)
define the following mapping, relative to `usage`:
| Quantity | Response field | PromptKit field |
| --- | --- | --- |
| Input tokens | `prompt_tokens` | Existing `PromptTokens` |
| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` |
| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` |
| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` |
| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` |
| Total tokens | `total_tokens` | Existing `TotalTokens` |
Preserve `CachedTokens` as the public field name and clarify its meaning as
cache-read tokens. Correct the OpenRouter cache-write mapping to the nested
field above; retain the generic provider's existing top-level handling. Do not
add the nested and top-level values together. OpenRouter uses its documented
nested value; an omitted nested value remains unknown.
Add `ReasoningTokens` to the public/domain token values, presence indicators,
and observed token usage, including conversions, result serialization, attempt
and terminal events, and group summaries. OpenRouter and vLLM supply this
additional built-in response interpretation; the generic provider preserves its
existing behavior. Injected clients may supply the canonical count and its
presence through the existing model-client boundary.
### vLLM Response Mapping And Availability
Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens`
counts. For `provider: vllm`, map these additional fields relative to `usage`:
| Response field | PromptKit field |
| --- | --- |
| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` |
| `prompt_tokens_details.cached_tokens` | `CachedTokens` |
| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` |
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
describe the creation count as prompt tokens counted as local prefix-cache
writes. Normalize that token activity to `CacheWriteTokens` without implying a
billable write charge. Use the selected provider's documented field, not a sum
of alternative cache-write fields; an absent vLLM creation count stays unknown.
Detail availability depends on server version and configuration. Current vLLM
gates prompt details on `--enable-prompt-tokens-details` and reasoning details on
a configured reasoning parser. Document these reporting settings, but do not
configure or discover them automatically. Older or differently configured
servers may omit the fields; accept those responses with the affected counts
unknown. Do not infer reasoning counts from generated text or cache writes from
input tokens minus cache reads.
Once normalized, these counts use the shared presence, aggregation, and
cache-hit rules below. No vLLM-specific result type, ratio, or group accounting
is needed.
### Count Relationships And Availability
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
vLLM likewise reports its reasoning detail within the generated-token usage.
Cache reads and writes describe input-token breakdowns. Present these as subsets
or details rather than additional consumption to add to input/output totals.
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
Consumers can derive non-reasoning output as completion tokens minus reasoning
tokens when both counts are available and consistent. No additional stored
non-reasoning count is needed in this scope. Capturing reasoning-token counts
does not capture reasoning text or change reasoning request settings.
Reuse existing per-field presence and aggregation rules: omitted/null means
unknown, explicit zero means known zero, and aggregate `Known` and `Complete`
flags describe the reported contributions across dispatched attempts. Include
initial generations, retries, and repairs through the existing observation
accounting, retaining terminal-only group aggregation and deduplication. Preserve
the existing distinction between integer-only result usage and presence-aware
observations; zero in `TokenUsage` alone does not establish provider reporting.
### Token Cache-Hit Ratio
Add this derived helper on the existing presence-aware usage type:
```go
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
```
The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero
to one. It measures the proportion of input tokens served from cache, not the
proportion of requests with a cache hit or monetary savings. The Boolean reports
whether a ratio is available; an unavailable result returns zero and false.
Return an available ratio only when both counts are known and complete, input
tokens are positive, and cache reads are between zero and input tokens inclusive.
Zero cache reads with known positive input yields an available zero ratio. Missing
or incomplete counts, zero input, or inconsistent counts leave it unavailable;
preserve the raw reported counts rather than clamping them or failing generation.
Cache-write availability does not affect this calculation.
For a run or group, divide the aggregated cache-read count by the aggregated
input count; do not average per-attempt or per-run percentages. Do not use
`reads / (reads + writes)`, because cache writes need not account for all
uncached input. The same helper works on attempts, terminal events, and group
summaries without storing another independently maintained field.
The ratio describes the included observations. A group ratio does not supersede
`TerminalCoverageComplete` or imply that all application work has finished.
For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
are known and complete, even if cache writes are unknown. Output 840 with
reasoning 600 still represents 840 output tokens, not 1,440.
## Optional Cost Reporting
### Initial Provider Support
Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter
defines as the amount charged to the account, denominated in USD. See its
[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
Do not add `cost_details` breakdowns or combine upstream inference cost with the
top-level charge; those describe different facts and could double-count cost.
For the built-in client, select this interpretation through `provider: openrouter`,
regardless of backend identity or namespace. Do not infer USD semantics from an
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
Built-in provider-reported cost interpretation is an extension unique to
`openrouter` initially; `openai-compatible` and `vllm` do not interpret
`usage.cost`. Without an applicable zero-cost declaration, their built-in cost
remains unknown. No separate provider cost-decoding selector is needed.
Aggregation remains provider-neutral after the model-client boundary supplies
interpreted USD facts.
Use the cost included in the normal completion response. No additional usage
lookup, pricing request, or opt-in provider request parameter is needed.
### Explicit Backend Zero-Cost Declaration
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
```yaml
promptkit:
backends:
- id: local-inference
provider: vllm
endpoint: http://localhost:8000/v1
zero_cost: true
```
For each dispatched attempt, use a usable provider/model-client-reported cost
first, including an explicitly reported zero. If no usable reported cost is
available and the effective backend has `zero_cost: true`, supply USD zero with
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
it does not suppress a reported cost. Apply this policy to built-in and injected
client execution using the selected backend, regardless of its provider type.
The fallback also applies to dispatched attempts that fail, are canceled, or
produce no decodable response. It is an operator declaration, not a provider
observation. Non-dispatched work contributes no cost. Do not infer this setting
from a local address, provider name, token counts, or missing credentials, and
do not install it automatically on any backend. Without a selected backend
declaration, no configured-zero fallback applies.
Resolve the field through ordinary final ConfigKit composition and inheritance;
include it in typed backend helpers, configuration capture, and backend
inspection. Retain the effective policy through prepared execution without
reloading configuration. It is local accounting metadata, not an outbound
request parameter. `InspectBackend` shows the resolved declared setting,
preserving explicit `false`; omitted and null values can remain absent. Do not
manufacture an explicit false value for the runtime fallback or add null-tracking
machinery to resource inspection. `InspectConfiguration` retains ConfigKit's
source history and explicit/null/omitted distinctions. Source provenance belongs
to configuration inspection, not resource inspection.
This supports an explicit zero declaration, not configurable input/output or
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
cost estimation. It does not alter credentials, generation, or token reporting.
### Monetary Representation
Use `float64` amounts in USD. Make the unit explicit in public field names such
as `CostUSD`, rather than introducing a generic amount with an implicit currency.
At the individual model-client response boundary, use an optional amount
(`*float64`): nil means unknown, and a pointer to zero means reported zero.
Injected clients may supply the same optional USD value without needing an
OpenRouter backend; no provider-specific interface is needed.
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
remain token counts. For run results, execution events, and group summaries,
use a small presence-aware cost value with `Value float64`, `Known bool`, and
`Complete bool`, following the existing observed-count conventions, plus the
source metadata below. Successful run results should expose cost even when no
observer was registered; observers also provide per-attempt and terminal cost,
including available facts on failure.
Binary floating point is sufficient for operational telemetry and keeps decoding,
addition, and the public API lightweight. Do not round to cents or another fixed
scale during capture or aggregation; consumers choose display precision.
Decimal strings or a decimal arithmetic dependency would be appropriate for
exact financial ledgers, while fixed-scale integers would require a selected
precision and rounding policy. Neither is warranted for this reporting feature.
The API does not promise exact decimal arithmetic or invoice reconciliation.
### Cost-Source Metadata
Give the presence-aware cost value a `Source` discriminator with these meanings:
- `provider`: a cost supplied by the provider or injected model-client response.
This records the reporting path, not independent verification of a charge.
- `configured_zero`: zero supplied by the explicit backend fallback.
- `mixed`: an aggregate includes both provider and configured-zero contributions.
- Empty: no known contribution has a source. This covers unknown cost and the
known, complete zero aggregate when no attempts were dispatched.
Derive aggregate source from the union of known contribution sources, including
zero-valued contributions. Unknown contributions supply no source but affect
completeness. A provider-only subtotal can therefore be incomplete without being
`mixed`. Carry this metadata consistently through attempt events, terminal
events, successful results, and group summaries. Reuse existing aggregation and
terminal deduplication rather than adding a parallel collector or cost history.
Consumers can distinguish reported charges, configured zeros, and mixed totals
without treating configured values as provider reports. Source metadata is
separate from configuration inspection and does not require provenance traversal
during execution.
### Presence, Aggregation, And Failures
- An absent or null provider cost is unknown unless the backend's explicit
zero-cost fallback supplies it. A reported numeric zero remains known zero
with a provider source, even when the fallback is enabled.
- On a dispatched attempt, a usable reported cost or configured-zero fallback
is known and complete for that attempt. Cost completeness is independent of
token-field completeness and does not assert provider reporting.
- Sum known costs across dispatched attempts, including initial generations,
retries, and repairs. `Known` means at least one attempt has an available cost
from either source; `Complete` means every dispatched attempt does under this
policy. Unknown values contribute no amount, but make the total incomplete.
With no dispatched attempts, follow the existing known, complete zero
convention, with an empty source.
- Group summaries reuse existing terminal-only aggregation and deduplication:
sum known terminal values, OR their `Known` flags, and AND their `Complete`
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
from OpenRouter mixed with unreported local-backend cost is incomplete unless
those local attempts have an explicit configured-zero fallback. Combine
cost-source metadata as described above without changing token accounting.
- Keep valid cost observed from a decoded successful envelope when a later
choice/content check fails, as with token usage. Invalid framing and non-2xx
error bodies continue to supply no provider usage, but the configured-zero
fallback still applies to dispatched attempts. Without that fallback,
cancellation or failure may leave incomplete cost reporting even when billable
upstream work occurred.
- Treat an unusable optional cost value, including a nonnumeric value or one
not representable as a finite `float64`, as unknown rather than making an
otherwise valid generation fail; the zero-cost fallback can then apply. This
does not relax existing response framing or content validation. An injected
client likewise cannot supply a nonfinite usable cost.
Completeness describes coverage by available reported or explicitly configured
cost, not verification of the provider's billing or proof that every provider
reported a charge. Preserve usable reported amounts without pricing inference or
reconciliation.
For a successful run, result cost and terminal-event cost describe the same
dispatched attempts and must agree on value, presence, completeness, and source.
Result cost must also be the same with observation enabled or disabled. For
example, an unreported failed attempt followed by a successful attempt reporting USD
0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
source when no zero-cost fallback applies. Apply the same accounting through
repairs and ordinary or prepared execution. An invocation with fully reported
OpenRouter charges and configured-zero local calls can have a complete subtotal
with a mixed source, without claiming all providers reported costs.
Group cost completeness concerns the terminal observations included in its
subtotal. If two completed runs reported all their costs but a third observed
run has no terminal event, the subtotal may have `Complete: true` while
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
coverage field; no additional completion flag or group lifecycle is needed.
## Accounting And Responsibility Boundaries
Reuse existing events, token presence/completeness, and group summaries, extending
them with the cost facts above. Available usage remains observable after an error
or cancellation under the existing execution-observation behavior; this does not
promise that upstream providers report usage for every attempt.
PromptKit owns observer selection, event generation, and aggregation. CommandKit
and other consumers own invocation lifecycle, CLI flags, presentation, output
delivery, and interpretation. This feature adds no CommandKit dependency or
application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Reasoning-text
capture, changes to reasoning or caching request behavior, additional built-in
cost providers, configurable token-rate estimation, fixed nonzero request
pricing, multiple currencies, currency conversion, pricing estimates,
billing reconciliation, persistent history, and logging infrastructure are
outside this scope.
## Work Needed And Completion Criteria
- Add the context helper and connect observer selection to ordinary and prepared
execution, keeping contextual selection separate from private run state.
- Preserve existing explicit selection and coordinator behavior while applying
the composition and lifetime rules above. Preserve existing observer selection
when no contextual observer is selected.
- Add `provider` to the backend configuration model, typed source helpers,
capture, inspection, and execution plumbing. Update the bundled OpenRouter
backend to select it explicitly. Retain the selection through prepared runs.
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
prepared-execution plumbing. Apply the fallback outside provider-specific
decoding so it works with built-in and injected clients. Verify composition
and inheritance, omitted/null/false behavior, explicit overrides, provider
independence, configuration-inspection provenance, and absence from provider
bodies. Verify that resource inspection preserves explicit false without
manufacturing values or source-history fields for omitted/null settings.
- Consolidate provider-specific response and authentication-probe behavior in
`internal/llm`; keep coordination and accounting provider-neutral. Cover
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
and existing explicit probes without a provider selection.
- Verify provider selection independently of backend names and namespaces,
including renamed and inherited backends, equivalent omitted/null/blank values,
trimming and case normalization, rejection of unsupported nonempty values,
and retention through endpoint overrides. Confirm provider metadata is not
serialized into the completion request body. Verify that explicit
`openai-compatible` and omitted/null/blank selection preserve the same existing
shared behavior without manufacturing explicit configuration provenance.
- Verify that provider diagnostic defaults do not become explicit values in the
loaded configuration or replace source provenance in inspection. Cover the
changed clearing behavior and endpoint-only authentication overrides.
- Carry OpenRouter reasoning-token counts through token values, presence,
results, events, and group summaries. Verify nested OpenRouter cache-write
decoding, including omitted/null/zero values and the nested value taking
precedence when a top-level value is also present. Preserve generic-provider
token mappings and existing field meanings.
- Add the `vllm` provider selection, reasoning and cache-detail mappings, and
`vllm_models` authentication adapter. Cover supplied/missing credentials,
recognized acceptance (including an empty models list), rejection,
unrecognized responses, provider defaults, and explicit kind/endpoint overrides,
including `none` and adapter selection
without a provider declaration. Verify that ordinary generation performs no
diagnostic requests and leaves monetary cost unknown unless the backend
explicitly declares the zero-cost fallback.
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
details, including responses representative of servers without detail
reporting enabled. Verify `created_cache_tokens` mapping independently of
OpenRouter's cache-write field and carry normalized counts through results,
events, and groups. Do not require a live server or version discovery.
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
incomplete counts, zero input, inconsistent counts, independence from
cache-write availability, and token-weighted aggregate ratios. Verify subset
accounting without adding reasoning or cached tokens to the reported totals.
- Carry optional OpenRouter cost through response decoding, internal/public
conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer.
Preserve existing token fields and accounting.
- Carry cost-source metadata through results, events, conversions, and group
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
including zero-valued reported contributions and empty/non-dispatched cases.
Verify source union and terminal deduplication independently of cost value.
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
null, or unusable reported cost permits fallback. Cover dispatched failures,
cancellation, undecodable responses, retries, repairs, ordinary/prepared
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
and declared zero-cost local calls can produce complete cost totals while
token completeness remains independent.
- Verify that successful-result cost agrees with terminal-event cost and is
unchanged when observation is disabled, including the unreported-retry example
above. Verify group snapshots with fully reported completed runs alongside a
member without a terminal observation.
- Cover ordinary and prepared execution, coordinated continuations, nested calls
using derived contexts, concurrent calls, multiple engines sharing a scope,
and independent scopes on a reused engine.
- Cover callback ordering, nested contextual registration, nil registration,
duplicate group enrollment, prepared-handle scope selection, and existing
preparation/discard/rejected-claim behavior.
- Verify aggregate accounting through retries and repairs, available usage after
failures or cancellation, and known zero versus unknown or incomplete counts.
Use offline fixtures and injected clients; live inference is unnecessary.
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
generic built-in selection, optional injected-client cost, mixed known
and unknown totals, repair/retry aggregation, and duplicate group registration.
Use tolerant comparisons where floating-point addition requires them; do not
turn telemetry into an exact-decimal guarantee.
- Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide, including cost
availability, units, completeness, and source. Explain that configured zeros
can make cost complete without provider reporting. Document the reasoning/cache
fields and cache-hit helper in their owning GoDoc and illustrate input/output
breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
OpenRouter cost decoding in the
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the
framework format reference and relevant public GoDoc; update inspection and
diagnostic documentation with the implemented behavior. Document `zero_cost`
and its fallback precedence, inspection behavior, and prepared lifetime in the
relevant format, GoDoc, and consumer owners. Include the diagnostic
migration in release notes. Include vLLM usage availability, server reporting
settings, the meaning of cache creation, and the limits of its authentication
diagnostic in the relevant consumer and integration documentation.
## Implementation Planning Guidance
Organize the eventual implementation plan around two workstreams:
1. Contextual observer selection, including ordinary, prepared, coordinated,
and nested execution boundaries and the existing group collector.
2. Backend provider selection and consolidation within `internal/llm`, including
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
extensions, provider-reported cost, and configured-zero fallback through
decoding, accounting, public results, events, and group summaries, plus the
derived cache-hit helper. Include the corresponding configuration and API
documentation.
The second area is more extensive than the context helper. Keep backend provider
configuration, diagnostic consolidation, token/cache reporting, and cost/source
accounting in separately reviewable implementation units rather than one large
implementation prompt.
Avoid coupling delivery of contextual observation to provider cost support.
Today, successful results accumulate generation responses in the runner, while
optional observation recorders account for dispatched attempts. Adding cost to
response objects alone does not preserve completeness across unreported retries.
Reuse a small accounting component for result and observation paths so they
apply the same rules, including when no observer is selected. Preserve existing
token accounting rather than expanding this work into a redesign of its public
contract.
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
overflowing value fail decoding of the entire response. Decode the optional
field separately so an unusable provider cost becomes unknown before applying
any configured-zero fallback, without changing existing framing or content
validation. This needs no general numeric-conversion layer.
Apply configured-zero fallback after decoding, outside provider-specific code,
so built-in and injected clients share the same policy. Aggregate cost sources
from known contributions even when their amounts are zero; a reported zero and
a configured zero together have a mixed source. Keep subtotal completeness
separate from terminal coverage, using the existing coverage field and the
examples above. These rules belong in shared accounting rather than separate
provider, result, and group implementations.
Use the completion criteria above to select focused offline checks for each
unit. This roadmap owns target behavior; a separate implementation plan owns
the detailed work sequence.