2026-11-22 09:39:27 +00:00
|
|
|
# Invocation-Scoped Execution Observation
|
|
|
|
|
|
2026-11-22 18:38:11 +00:00
|
|
|
Status: accepted scope; implementation complete. Current behavior is documented
|
|
|
|
|
in the linked canonical contracts; roadmap retirement is separate housekeeping.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
## Purpose And Scope
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Let consumers select observation once for an application-owned operation and
|
|
|
|
|
obtain usage across its nested, concurrent, coordinated, and multi-engine
|
|
|
|
|
PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a
|
|
|
|
|
token cache-hit ratio, and optional USD cost with its source and completeness.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
CommandKit's optional `ckprompt` integration motivates this work, but the API
|
|
|
|
|
remains application-neutral. PromptKit owns observation, provider response
|
|
|
|
|
interpretation, and aggregation. Consumers own context propagation, waiting for
|
|
|
|
|
their work, invocation lifecycle, presentation, and interpretation.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
This document defines the target state and the changes required to reach it.
|
|
|
|
|
A separate implementation plan will own sequencing and prompt-sized stages.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
## Current Baseline
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
PromptKit v0.24.0 already provides
|
|
|
|
|
[execution observers and ordered composition](../../observations.go),
|
|
|
|
|
a concurrency-safe [execution group](../../execution_group.go), and
|
|
|
|
|
[request-level observation](../consumers/pkg-promptkit.md#observe-an-ordinary-run).
|
|
|
|
|
Groups aggregate the first terminal event for each observed run. Existing
|
|
|
|
|
observation distinguishes known, unknown, and incomplete prompt, completion,
|
|
|
|
|
total, cached, and cache-write token counts across retries and repairs.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Today, consumers explicitly enroll requests. Prepared handles retain the request
|
|
|
|
|
observer; [coordinator observers](../../coordinate.go) select continuation runs.
|
|
|
|
|
There is no contextual selection, reasoning-token count, cache-hit helper, or
|
|
|
|
|
monetary cost reporting. OpenRouter cache writes also need their documented
|
|
|
|
|
nested mapping: the current client reads `usage.cache_write_tokens` instead.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
## Contextual Observer Selection
|
2026-11-22 09:48:38 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Add the public helper:
|
2026-11-22 09:39:27 +00:00
|
|
|
|
|
|
|
|
```go
|
2026-11-22 17:52:58 +00:00
|
|
|
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
2026-11-22 09:39:27 +00:00
|
|
|
```
|
|
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
The returned context composes the supplied observer with any inherited contextual
|
|
|
|
|
observers. A nil observer leaves that selection unchanged. Executions using the
|
|
|
|
|
context, or derived contexts retaining its values, participate automatically;
|
|
|
|
|
unrelated contexts do not. No engine option or process-wide default is needed.
|
|
|
|
|
|
|
|
|
|
A consumer can create an invocation-owned group, associate `group.Observe` with
|
|
|
|
|
the context, run its pipeline, and snapshot the group after its owned work
|
|
|
|
|
finishes, including when the pipeline returns an error. Group coverage describes
|
|
|
|
|
observed members, not future work or overall application progress.
|
|
|
|
|
|
|
|
|
|
### Delivery And Execution Boundaries
|
|
|
|
|
|
|
|
|
|
- `Run` selects contextual observers from its invocation context.
|
|
|
|
|
- `RunPrepared` selects them from its execution context, not the context used
|
|
|
|
|
to prepare the handle. The explicitly supplied request observer remains
|
|
|
|
|
retained by the handle and composes with those execution-time observers.
|
|
|
|
|
- Preparation emits no execution events. Discarded handles and rejected
|
|
|
|
|
execution claims retain their existing observation behavior.
|
|
|
|
|
- Coordinated continuations acquire contextual observation through their
|
|
|
|
|
ordinary `Run` boundary. `CoordinateRequest.Observer` still selects
|
|
|
|
|
continuation runs only; additional calls made by application callbacks
|
|
|
|
|
participate when they use the scoped context.
|
|
|
|
|
- Every execution, including a nested one using a callback or injected-client
|
|
|
|
|
context, has its own identity, recorder, and totals. Clearing an enclosing
|
|
|
|
|
execution's private state at a public boundary preserves contextual observer
|
|
|
|
|
selection, not the enclosing recorder.
|
|
|
|
|
|
|
|
|
|
Deliver events to explicit request observers first, then any coordinator observer
|
|
|
|
|
already composed with them, then contextual observers from inherited to newly
|
|
|
|
|
registered. Incorporate contextual selection once per execution, not again in
|
|
|
|
|
forwarding or coordination layers.
|
|
|
|
|
|
|
|
|
|
Preserve synchronous callback delivery and panic behavior. Concurrent runs may
|
|
|
|
|
invoke a shared observer concurrently; consumers synchronize their callback state.
|
|
|
|
|
The existing group remains concurrency-safe.
|
|
|
|
|
|
|
|
|
|
Registering the same callback through multiple paths may deliver duplicate
|
|
|
|
|
events; arbitrary Go functions are not deduplicated. Retain one lifecycle per
|
|
|
|
|
execution and reuse the group's terminal-event deduplication. Terminal totals
|
|
|
|
|
are aggregates, not additional contributions to add to attempt totals.
|
|
|
|
|
|
|
|
|
|
## Backend Provider And Accounting Metadata
|
|
|
|
|
|
|
|
|
|
Add two optional backend fields:
|
|
|
|
|
|
|
|
|
|
| Field | Meaning |
|
|
|
|
|
| --- | --- |
|
|
|
|
|
| `provider` | Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. |
|
|
|
|
|
| `zero_cost` | Enables a configured-zero cost fallback when no usable reported cost is available. |
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
For example:
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
```yaml
|
|
|
|
|
promptkit:
|
|
|
|
|
backends:
|
|
|
|
|
- id: research-router
|
|
|
|
|
provider: openrouter
|
|
|
|
|
endpoint: https://openrouter.ai/api/v1
|
|
|
|
|
api_key_env: OPENROUTER_API_KEY
|
|
|
|
|
- id: local-inference
|
|
|
|
|
provider: vllm
|
|
|
|
|
endpoint: http://localhost:8000/v1
|
|
|
|
|
zero_cost: true
|
|
|
|
|
```
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Trim and lowercase the final effective provider value. Omitted, null, empty, and
|
|
|
|
|
whitespace-only values select `openai-compatible`. Reject unsupported nonempty
|
|
|
|
|
normalized values as ordinary configuration errors. The supported selections are
|
|
|
|
|
`openai-compatible`, `openrouter`, and `vllm`.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
`zero_cost` is Boolean. Omitted, null, or false disables its fallback; true
|
|
|
|
|
enables it. It is independent of provider selection and does not suppress a
|
|
|
|
|
usable reported cost.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Resolve both fields through ordinary final ConfigKit composition and inheritance.
|
|
|
|
|
Carry their effective selections through capture and prepared execution without
|
|
|
|
|
reloading configuration. Declare `provider: openrouter` in the bundled
|
|
|
|
|
OpenRouter backend. Do not automatically declare any backend zero-cost.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
These fields are local metadata, not outbound request parameters or entries
|
|
|
|
|
added to `extra_params`. Provider selection does not choose an endpoint or model,
|
|
|
|
|
and endpoint overrides retain the selected provider and accounting policy.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
### Inspection Boundary
|
2026-11-22 09:39:27 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
`InspectBackend` shows the normalized/resolved declared provider and zero-cost
|
|
|
|
|
setting, preserving explicit false for `zero_cost`. Omitted/null declarations
|
|
|
|
|
may remain absent, and blank provider values need no extra history representation.
|
|
|
|
|
Do not manufacture an explicit `openai-compatible` declaration or false
|
|
|
|
|
`zero_cost` value from runtime defaults.
|
2026-11-22 09:48:38 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
`InspectConfiguration` owns ConfigKit source history and explicit/null/omitted
|
|
|
|
|
distinctions. Resource inspection does not acquire source-history fields or
|
|
|
|
|
null-tracking machinery. Provider diagnostic defaults likewise remain runtime
|
|
|
|
|
fallbacks, not values written into loaded configuration or recorded as explicit
|
|
|
|
|
source declarations.
|
2026-11-22 15:27:12 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
## Provider Conventions
|
2026-11-22 15:27:12 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
All selections reuse the existing
|
|
|
|
|
[OpenAI-compatible text Chat Completions foundation](../integrations/openai-compatible-chat.md):
|
|
|
|
|
base-relative `/chat/completions` JSON POST requests, existing credential rules,
|
|
|
|
|
text messages, selected generation controls and structured output, extra
|
|
|
|
|
parameters, first-choice content extraction, common token fields, and existing
|
|
|
|
|
response validation, errors, cancellation, timeouts, and transient retries.
|
2026-11-22 15:27:12 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
| Provider | Additional response interpretation | Default authentication adapter |
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
| `openai-compatible` | Existing token mappings; no provider-reported monetary cost decoding. | None. |
|
|
|
|
|
| `openrouter` | OpenRouter reasoning/cache details and `usage.cost` in USD. | `openrouter_key`. |
|
|
|
|
|
| `vllm` | vLLM reasoning/cache details; no provider-reported monetary cost decoding. | `vllm_models`. |
|
|
|
|
|
|
|
|
|
|
The default provider names PromptKit's current supported conventions, not the
|
|
|
|
|
entire OpenAI API. "Lowest common denominator" does not strip existing optional
|
|
|
|
|
features or promise every backend supports them. Continue sending selected
|
|
|
|
|
supported settings and return backend errors normally.
|
|
|
|
|
|
|
|
|
|
Provider selection supplies no default endpoint, model, credential environment
|
|
|
|
|
variable, or sampling settings, and does not create backend definitions. Keep
|
|
|
|
|
vLLM reasoning, chat-template, and other model-specific controls caller-selected;
|
|
|
|
|
do not automatically enable thinking or translate those controls. No new
|
|
|
|
|
generation protocol, capability negotiation, or version discovery is needed.
|
|
|
|
|
vLLM documents its
|
|
|
|
|
[OpenAI-compatible interface](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
|
|
|
|
|
|
|
|
|
### Authentication Diagnostics
|
|
|
|
|
|
|
|
|
|
Provider defaults apply only to explicitly requested authentication diagnostics.
|
|
|
|
|
Loading configuration, constructing an engine, and ordinary generation do not
|
|
|
|
|
initiate probes. Existing reachability and inference-default behavior remains.
|
|
|
|
|
|
|
|
|
|
Explicit `smoke_tests.authentication` metadata overrides provider defaults:
|
|
|
|
|
|
|
|
|
|
- A supplied `kind` selects the adapter; `none` declares authentication testing
|
|
|
|
|
inapplicable.
|
|
|
|
|
- Omitted/null `kind` uses the provider default, allowing endpoint-only overrides.
|
|
|
|
|
- A supplied endpoint overrides the selected adapter's destination; omitted/null
|
|
|
|
|
uses that adapter's existing fallback.
|
|
|
|
|
- An absent/null authentication block removes its overrides and leaves provider
|
|
|
|
|
defaults applicable. `kind: none`, not clearing the block, suppresses them.
|
|
|
|
|
|
|
|
|
|
Without a provider default or explicit adapter, preserve the existing unsupported
|
|
|
|
|
diagnostic outcome rather than assuming authentication is inapplicable. Explicit
|
|
|
|
|
adapters remain selectable without a matching provider declaration.
|
|
|
|
|
|
|
|
|
|
Retain `openrouter_key` behavior. The new `vllm_models` adapter makes one GET to
|
|
|
|
|
`/models` relative to the configured API base, supplying an available credential
|
|
|
|
|
under existing rules and allowing dispatch without one. A base ending in `/v1`
|
|
|
|
|
therefore produces `/v1/models`; explicit diagnostic endpoint overrides apply.
|
|
|
|
|
|
|
|
|
|
For `vllm_models`, HTTP 200 with a valid JSON object containing `object: "list"`
|
|
|
|
|
and array-valued `data` means acceptance of the probe request. An empty array is
|
|
|
|
|
accepted; recognition does not inspect model capabilities or require a particular
|
|
|
|
|
model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown.
|
|
|
|
|
Reuse existing bounded diagnostic decoding and framing. Report observed
|
|
|
|
|
credential presence and response facts without claiming authentication is
|
|
|
|
|
enforced or models-list acceptance establishes inference authorization.
|
|
|
|
|
|
|
|
|
|
vLLM documents `/v1/models` as protected when API-key authentication is configured;
|
|
|
|
|
`/health` is not an authentication check. See its
|
2026-11-22 15:57:37 +00:00
|
|
|
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
|
2026-11-22 17:52:58 +00:00
|
|
|
No health-monitoring facility is added here.
|
2026-11-22 15:27:12 +00:00
|
|
|
|
2026-11-22 15:57:37 +00:00
|
|
|
## Token And Cache Reporting
|
2026-11-22 15:54:02 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
### Provider Response Mapping
|
2026-11-22 15:54:02 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Use common `usage.prompt_tokens`, `usage.completion_tokens`, and
|
|
|
|
|
`usage.total_tokens` for `PromptTokens`, `CompletionTokens`, and `TotalTokens`.
|
|
|
|
|
Normalize these additional provider details, relative to `usage`:
|
2026-11-22 15:54:02 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
| Public field | OpenRouter | vLLM |
|
2026-11-22 15:54:02 +00:00
|
|
|
| --- | --- | --- |
|
2026-11-22 17:52:58 +00:00
|
|
|
| New `ReasoningTokens` | `completion_tokens_details.reasoning_tokens` | `completion_tokens_details.reasoning_tokens` |
|
|
|
|
|
| Existing `CachedTokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cached_tokens` |
|
|
|
|
|
| Existing `CacheWriteTokens` | `prompt_tokens_details.cache_write_tokens` | `prompt_tokens_details.created_cache_tokens` |
|
2026-11-22 15:57:37 +00:00
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
OpenRouter documents these in its
|
|
|
|
|
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
|
|
|
|
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
|
|
|
|
|
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching).
|
2026-11-22 15:57:37 +00:00
|
|
|
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
|
2026-11-22 17:52:58 +00:00
|
|
|
uses `created_cache_tokens`; its
|
2026-11-22 15:57:37 +00:00
|
|
|
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
|
2026-11-22 17:52:58 +00:00
|
|
|
describe local prefix-cache writes, not a billable write charge.
|
|
|
|
|
|
|
|
|
|
Preserve `CachedTokens` as the public name and clarify its meaning as cache reads.
|
|
|
|
|
Use each provider's documented cache-write field, not a sum or guessed alias;
|
|
|
|
|
a missing nested value is unknown even if an alternative field is present.
|
|
|
|
|
Retain the generic provider's existing mappings, including its top-level
|
|
|
|
|
`usage.cache_write_tokens` handling.
|
|
|
|
|
|
|
|
|
|
Add `ReasoningTokens` to public/domain token values, presence indicators, observed
|
|
|
|
|
usage, conversions, and result serialization, and carry it through attempt events,
|
|
|
|
|
terminal events, and group summaries. Injected clients may supply canonical
|
|
|
|
|
counts and presence through the existing model-client boundary, retaining its
|
|
|
|
|
legacy `UsagePresence` fallback; explicit presence distinguishes measured zero.
|
|
|
|
|
|
|
|
|
|
### Availability And Optional Metadata
|
|
|
|
|
|
|
|
|
|
Omitted/null details are unknown, not zero. Explicit integer zero is known zero.
|
|
|
|
|
For the newly interpreted provider-specific reasoning and cache-write details,
|
|
|
|
|
wrong-type, nonintegral, or unrepresentable integer values are also unknown.
|
|
|
|
|
An unusable new detail does not invalidate otherwise usable content or erase
|
|
|
|
|
independently available usage. Preserve existing common-field decoding and
|
|
|
|
|
response framing/content validation; this is a targeted optional-metadata rule,
|
|
|
|
|
not a general relaxation of the response contract. Keep this interpretation local
|
|
|
|
|
to the model-client boundary without a general numeric-conversion framework.
|
|
|
|
|
|
|
|
|
|
vLLM detail availability depends on version and configuration: prompt details
|
|
|
|
|
currently require `--enable-prompt-tokens-details`, and reasoning details depend
|
|
|
|
|
on a configured reasoning parser. Document those reporting settings without
|
|
|
|
|
configuring or discovering them automatically. Older or differently configured
|
|
|
|
|
servers may omit details and still produce usable responses. Do not estimate
|
|
|
|
|
reasoning counts from text or cache writes from input minus cache reads.
|
|
|
|
|
|
|
|
|
|
### Count Relationships And Aggregation
|
|
|
|
|
|
|
|
|
|
Reasoning is included in completion tokens, not additional output to add to them.
|
|
|
|
|
OpenRouter explains this in its
|
|
|
|
|
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens);
|
|
|
|
|
vLLM similarly includes reasoning detail within generated-token usage. Cache reads
|
|
|
|
|
and writes are input-token details, not extra input consumption. Preserve the
|
|
|
|
|
reported total without recomputation or reconciliation.
|
|
|
|
|
|
|
|
|
|
Consumers can derive non-reasoning output from consistent, known completion and
|
|
|
|
|
reasoning counts; no additional stored count is needed. Capturing counts does
|
|
|
|
|
not capture reasoning text or change generation/caching settings.
|
|
|
|
|
|
|
|
|
|
Reuse existing per-field token accounting: aggregates sum known contributions,
|
|
|
|
|
`Known` identifies available reporting, and `Complete` describes its coverage
|
|
|
|
|
across dispatched attempts. Preserve initial-generation, retry, and repair
|
|
|
|
|
observation accounting and terminal-only group deduplication. Existing integer
|
|
|
|
|
result usage remains distinct from presence-aware observations; a zero result
|
|
|
|
|
field alone does not establish that the provider reported it.
|
2026-11-22 15:54:02 +00:00
|
|
|
|
|
|
|
|
### Token Cache-Hit Ratio
|
|
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Add a derived helper, not another stored total:
|
2026-11-22 15:54:02 +00:00
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
|
|
|
|
|
```
|
|
|
|
|
|
2026-11-22 17:52:58 +00:00
|
|
|
Compute `CachedTokens / PromptTokens` as a fraction from zero to one. This is the
|
|
|
|
|
proportion of input tokens served from cache, not request hit frequency or cost
|
|
|
|
|
savings. For aggregates, divide summed reads by summed input; do not average
|
|
|
|
|
individual percentages or use `reads / (reads + writes)`.
|
|
|
|
|
|
|
|
|
|
Return an available ratio only when input and cache-read counts are known and
|
|
|
|
|
complete, input is positive, and reads lie between zero and input inclusive.
|
|
|
|
|
Known zero reads with positive input yields an available zero. Missing,
|
|
|
|
|
incomplete, zero-denominator, or inconsistent data yields zero and false while
|
|
|
|
|
preserving raw counts; it does not fail generation or clamp reported values.
|
|
|
|
|
Cache-write availability is irrelevant.
|
|
|
|
|
|
|
|
|
|
The same helper serves attempts, terminals, and group summaries. A group ratio
|
|
|
|
|
describes its included observations and does not supersede terminal coverage or
|
|
|
|
|
imply all application work has finished.
|
|
|
|
|
|
|
|
|
|
## Cost Reporting
|
|
|
|
|
|
|
|
|
|
### Reported Cost And Configured-Zero Precedence
|
|
|
|
|
|
|
|
|
|
Initially, only the OpenRouter built-in provider interprets `usage.cost` as the
|
|
|
|
|
account charge in USD. See its
|
|
|
|
|
[usage-accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
|
|
|
|
|
and [USD explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
|
|
|
|
|
Do not add `cost_details` amounts to that charge or interpret arbitrary providers'
|
|
|
|
|
fields named `cost`. The generic and vLLM providers do not decode monetary cost.
|
|
|
|
|
|
|
|
|
|
Use the cost included in the ordinary response; no extra request or usage lookup
|
|
|
|
|
is needed. At the model-client boundary, represent an optional USD amount as
|
|
|
|
|
`*float64`: nil means unknown; a pointer to zero means reported zero. Injected
|
|
|
|
|
clients may supply the same value regardless of provider selection.
|
|
|
|
|
|
|
|
|
|
Resolve cost at completion of each dispatched attempt, outside provider-specific
|
|
|
|
|
decoding and whether or not decoding occurred or succeeded:
|
|
|
|
|
|
|
|
|
|
- A usable provider/model-client-reported cost takes precedence, including zero.
|
|
|
|
|
- Otherwise, `zero_cost: true` on the effective backend supplies configured USD zero.
|
|
|
|
|
- Otherwise, cost remains unknown.
|
|
|
|
|
|
|
|
|
|
This applies to built-in and injected clients, including failed or canceled
|
|
|
|
|
dispatched attempts and undecodable responses. Without a selected backend
|
|
|
|
|
declaration, no configured-zero fallback applies. Non-dispatched work contributes
|
|
|
|
|
no cost. Do not infer zero cost from local hosting, provider name, credentials,
|
|
|
|
|
or token counts, or install the declaration automatically on bundled backends.
|
|
|
|
|
|
|
|
|
|
An unusable reported amount, including a nonnumeric value or one not representable
|
|
|
|
|
as a finite `float64`, is unknown before fallback; it does not make an otherwise
|
|
|
|
|
usable generation fail. Keep available cost from a decoded successful envelope
|
|
|
|
|
when a later choice/content check fails. Invalid framing and non-2xx bodies supply
|
|
|
|
|
no provider usage, although the configured-zero fallback still applies to a
|
|
|
|
|
dispatched attempt. Existing framing, content, and error handling remain intact.
|
|
|
|
|
|
|
|
|
|
### Monetary Representation And Sources
|
|
|
|
|
|
|
|
|
|
Expose cost separately from token counts and their presence indicators. Use
|
|
|
|
|
explicit USD field naming such as `CostUSD` and a small public cost value for
|
|
|
|
|
results, events, and group summaries with:
|
|
|
|
|
|
|
|
|
|
- `Value float64`: the available USD amount or subtotal;
|
|
|
|
|
- `Known bool`: availability;
|
|
|
|
|
- `Complete bool`: coverage under the selected cost policy; and
|
|
|
|
|
- `Source`: the contribution-source discriminator below.
|
|
|
|
|
|
|
|
|
|
| Source | Meaning |
|
|
|
|
|
| --- | --- |
|
|
|
|
|
| `provider` | Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. |
|
|
|
|
|
| `configured_zero` | Supplied by the explicit backend fallback. |
|
|
|
|
|
| `mixed` | An aggregate includes both reported and configured-zero contributions. |
|
|
|
|
|
| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. |
|
|
|
|
|
|
|
|
|
|
Use binary floating point for operational telemetry, without rounding to cents
|
|
|
|
|
or another fixed scale during capture or aggregation. Consumers choose display
|
|
|
|
|
precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency
|
|
|
|
|
rounding, pricing inference, and invoice reconciliation are not part of this work.
|
|
|
|
|
|
|
|
|
|
### Attempt, Run, And Group Accounting
|
|
|
|
|
|
|
|
|
|
A dispatched attempt with usable reported cost or configured zero is known and
|
|
|
|
|
complete for cost, independently of token completeness. Sum known amounts across
|
|
|
|
|
all dispatched attempts, including retries and repairs. Aggregate `Known` means
|
|
|
|
|
at least one known contribution; `Complete` means every dispatched attempt has a
|
|
|
|
|
cost from either source. Unknown contributions add no amount and make coverage
|
|
|
|
|
incomplete. Completeness does not assert that every provider reported a charge.
|
|
|
|
|
|
|
|
|
|
Aggregate sources by union of known contributions, including zero amounts.
|
|
|
|
|
Unknown contributions add no source: a provider-only subtotal can be incomplete
|
|
|
|
|
without becoming `mixed`. A reported zero plus a configured zero is `mixed`.
|
|
|
|
|
|
|
|
|
|
Successful results expose cost even without an observer. Result and terminal
|
|
|
|
|
cost agree on amount, availability, completeness, and source, and enabling
|
|
|
|
|
observation does not change result cost. For example, an unreported failed
|
|
|
|
|
attempt followed by a reported USD 0.002 yields an incomplete provider-source
|
|
|
|
|
subtotal when no zero-cost fallback applies.
|
|
|
|
|
|
|
|
|
|
Groups aggregate terminal costs only, counting each run's first terminal once.
|
|
|
|
|
Combine known amounts, OR `Known`, AND `Complete`, and union sources. Preserve
|
|
|
|
|
existing empty/no-terminal group semantics: a completed execution with no
|
|
|
|
|
dispatched attempts yields known, complete zero with empty source; an empty group
|
|
|
|
|
does likewise, while a group with members but no terminals has unknown,
|
|
|
|
|
incomplete usage.
|
|
|
|
|
|
|
|
|
|
Cost completeness concerns included terminal observations, independently of
|
|
|
|
|
`TerminalCoverageComplete`. Two completed runs with full cost coverage can yield
|
|
|
|
|
a complete subtotal while a third observed run lacks a terminal. Conversely,
|
|
|
|
|
fully reported OpenRouter runs and explicitly zero-cost local runs can yield a
|
|
|
|
|
complete, mixed-source total. Neither case verifies a provider invoice or future
|
|
|
|
|
application progress.
|
|
|
|
|
|
|
|
|
|
## Internal Responsibilities And Scope Limits
|
|
|
|
|
|
|
|
|
|
Provider-specific metadata interpretation and authentication conventions belong
|
|
|
|
|
in `internal/llm`, sharing common transport, request construction, credentials,
|
|
|
|
|
framing, and response fields. Use one provider-selection mechanism there.
|
|
|
|
|
Generation/observation coordination consumes normalized facts or generic
|
|
|
|
|
diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator
|
|
|
|
|
method with a generic operation while retaining explicit adapter selection.
|
|
|
|
|
|
|
|
|
|
Provider-specific files and straightforward dispatch suffice; no public provider
|
|
|
|
|
registration API, new `internal/providers` package, or broad mandatory-capability
|
|
|
|
|
interface is needed. This is configuration-driven selection among supported
|
|
|
|
|
implementations, not downstream registration of implementations.
|
|
|
|
|
|
|
|
|
|
Reuse a small shared cost-accounting component for result and observation paths
|
|
|
|
|
so dispatched attempts, fallback, and source union follow the same rules even
|
|
|
|
|
when observation is disabled. Provider code interprets wire metadata; common
|
|
|
|
|
accounting owns fallback and aggregation. No ConfigKit provenance traversal
|
|
|
|
|
belongs in ordinary execution.
|
2026-11-22 09:39:27 +00:00
|
|
|
|
|
|
|
|
Smoke diagnostics retain their separate report/evidence model and are not
|
2026-11-22 17:52:58 +00:00
|
|
|
enrolled in ordinary execution observation. CLI integration, output delivery,
|
|
|
|
|
persistent history, logging infrastructure, reasoning-text capture, changes to
|
|
|
|
|
reasoning/caching request behavior, server administration/metrics, additional
|
|
|
|
|
built-in cost decoders, multiple currencies, configurable token rates, fixed
|
|
|
|
|
nonzero request prices, and cost estimation remain outside scope.
|
|
|
|
|
|
|
|
|
|
## Work Needed And Acceptance
|
|
|
|
|
|
|
|
|
|
The following areas describe required changes, not an execution order.
|
|
|
|
|
|
|
|
|
|
- Extend the public context API and ordinary/prepared execution wiring while
|
|
|
|
|
preserving independent nested lifecycles, explicit observers, coordinator
|
|
|
|
|
selection, and existing preparation/discard/rejected-claim behavior.
|
|
|
|
|
- Extend backend settings, typed helpers, final-value normalization/validation,
|
|
|
|
|
capture, prepared metadata, and inspection for `provider` and `zero_cost`.
|
|
|
|
|
Update the bundled OpenRouter declaration. Preserve configuration provenance
|
|
|
|
|
and resource-inspection simplicity, and keep metadata out of request bodies.
|
|
|
|
|
- Consolidate provider behavior in `internal/llm`, adding vLLM usage normalization
|
|
|
|
|
and the models-list diagnostic alongside OpenRouter interpretation and the
|
|
|
|
|
shared default path. Preserve common decoding while tolerating unusable new
|
|
|
|
|
optional details and cost locally to the model-client boundary.
|
|
|
|
|
- Extend public/domain token types, presence, serialization, conversions, events,
|
|
|
|
|
and groups with reasoning counts and correct provider-specific cache writes.
|
|
|
|
|
Add the presence-aware cache-hit helper without changing existing token
|
|
|
|
|
meanings or introducing a separate ratio accumulator.
|
|
|
|
|
- Carry optional reported cost and configured-zero policy through attempt
|
|
|
|
|
completion and shared accounting into results, events, and groups, including
|
|
|
|
|
source metadata and observer-independent result accounting.
|
|
|
|
|
|
|
|
|
|
Select focused offline checks under the [testing policy](../policy/testing.md).
|
|
|
|
|
Use real internal collaborators and controlled HTTP/injected-client boundaries,
|
|
|
|
|
with proportionate public contract coverage. Important behaviors include:
|
|
|
|
|
|
|
|
|
|
- Context inheritance, registration order, nil/duplicate enrollment, concurrent
|
|
|
|
|
calls, multiple engines, independent scopes, nested calls, and execution-time
|
|
|
|
|
selection for a handle prepared elsewhere.
|
|
|
|
|
- Provider selection independent of backend identity; normalization and unsupported
|
|
|
|
|
value handling; omitted/null/blank defaults; inheritance, endpoint overrides,
|
|
|
|
|
explicit false, and inspection without manufactured values or source history.
|
|
|
|
|
- Provider-default and explicit diagnostic selection, endpoint-only overrides,
|
|
|
|
|
clearing metadata versus `kind: none`, vLLM acceptance including an empty list,
|
|
|
|
|
rejection, unrecognized/framing-limited responses, and optional credentials.
|
|
|
|
|
- Provider token mappings, absent/null/zero details, malformed new optional counts
|
|
|
|
|
without generation failure, unchanged common-field handling, and responses
|
|
|
|
|
representative of vLLM servers with detail reporting disabled.
|
|
|
|
|
- Cache-hit availability, inconsistent counts without generation failure,
|
|
|
|
|
token-weighted aggregates, independence from cache writes, and no double-counting
|
|
|
|
|
of reasoning or cache details.
|
|
|
|
|
- Reported cost precedence, configured-zero fallback on dispatched failure and
|
|
|
|
|
cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources,
|
|
|
|
|
zero-valued contributions, and independently incomplete token reporting.
|
|
|
|
|
- Result/terminal cost agreement with observation enabled or disabled, duplicate
|
|
|
|
|
terminal deduplication, empty/no-terminal groups, and subtotal completeness
|
|
|
|
|
distinct from terminal coverage.
|
|
|
|
|
|
|
|
|
|
Synthetic fixtures and offline examples suffice; live providers are not required.
|
|
|
|
|
Keep ownership localized and avoid repeating the same assertions across layers.
|
|
|
|
|
|
|
|
|
|
## Documentation And Migration
|
|
|
|
|
|
|
|
|
|
When implemented, exact exported contracts belong in GoDoc; the
|
|
|
|
|
[consumer guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) should show
|
|
|
|
|
invocation-scoped aggregation, reasoning/cache breakdowns without double-counting,
|
|
|
|
|
and cost availability, source, and completeness. Document provider wire mappings,
|
|
|
|
|
optional-metadata handling, and diagnostic response recognition in the
|
|
|
|
|
[integration contract](../integrations/openai-compatible-chat.md#response-handling).
|
|
|
|
|
The [framework format reference](../formats.md#backend-definitions) owns the new
|
|
|
|
|
backend fields, default selection, inheritance, and diagnostic overrides.
|
|
|
|
|
Update internal documentation for changed ownership and flows.
|
|
|
|
|
|
|
|
|
|
Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache
|
|
|
|
|
creation meaning without promising availability on every deployment. Consumers
|
|
|
|
|
own presentation. Models-list acceptance does not establish inference
|
|
|
|
|
authorization, and complete cost metadata does not independently verify billing.
|
|
|
|
|
|
|
|
|
|
Release notes should call out that authentication objects no longer require
|
|
|
|
|
`kind` when a provider default supplies it, endpoint-only overrides are supported,
|
|
|
|
|
and clearing an authentication block restores provider defaults. `kind: none`
|
|
|
|
|
suppresses that default. Preserve existing explicit adapters without requiring a
|
|
|
|
|
provider declaration.
|