766 lines
43 KiB
Markdown
766 lines
43 KiB
Markdown
# Invocation-Scoped Execution Observation
|
|
|
|
Status: accepted scope; implementation pending.
|
|
|
|
## Purpose
|
|
|
|
Allow consumers to select execution observation once for an application-owned
|
|
operation, rather than attach an observer to every PromptKit request. Nested,
|
|
concurrent, and multi-engine calls using that operation's context should
|
|
participate automatically. Include optional provider-reported cost alongside
|
|
token usage, with built-in cost decoding initially limited to OpenRouter and an
|
|
explicit backend zero-cost fallback. Distinguish reported and configured cost
|
|
sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
|
|
reporting with reasoning counts, documented cache reads and writes, and a shared
|
|
derived token cache-hit ratio.
|
|
|
|
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
|
could use this facility to report token usage for an entire command or pipeline.
|
|
PromptKit supplies application-neutral observation and aggregation; consumers
|
|
choose how to present or use the results.
|
|
|
|
## Current Baseline
|
|
|
|
PromptKit v0.24.0 already provides:
|
|
|
|
- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous
|
|
metadata callbacks and ordered composition.
|
|
- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached
|
|
snapshots, and aggregation of the first terminal event for each observed run.
|
|
- Presence-aware prompt, completion, total, cached, and cache-write token counts,
|
|
including aggregate accounting across retries and repairs.
|
|
- Explicit selection through `RunRequest.Observer`, retained for prepared
|
|
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
|
coordinator-selected continuation runs.
|
|
|
|
The missing capabilities are contextual observer selection, cost accounting and
|
|
source metadata, an explicit zero-cost declaration, reasoning-token counts, and
|
|
a presence-aware cache-hit helper.
|
|
OpenRouter cache writes also need their documented nested response mapping;
|
|
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
|
Applications currently have to enroll each request explicitly, and PromptKit
|
|
does not expose provider charges. The existing
|
|
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
|
describes that workflow.
|
|
|
|
## Target Public API
|
|
|
|
Add a public context helper:
|
|
|
|
```go
|
|
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
|
```
|
|
|
|
The returned context associates the observer with executions using that context
|
|
or derived contexts that retain its values. It composes with any observer already
|
|
associated with the context; it does not replace the inherited observer. A nil
|
|
observer leaves the inherited selection unchanged.
|
|
|
|
No engine option, process-wide default, or replacement group type is needed.
|
|
Extend the existing token types and result, event, and group surfaces with the
|
|
token and cost metadata described below, plus a derived cache-hit helper.
|
|
|
|
The intended consumer workflow is:
|
|
|
|
```go
|
|
var group promptkit.ExecutionGroup
|
|
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
|
|
|
|
err := runApplicationPipeline(ctx)
|
|
// After all application-owned work has finished, including on failure:
|
|
summary := group.Snapshot()
|
|
```
|
|
|
|
The consumer owns context propagation, waiting for its work, and using the
|
|
snapshot and error. Calls using unrelated contexts are not automatically
|
|
enrolled. Group coverage continues to describe observed members, not future
|
|
work or overall application progress.
|
|
|
|
## Observer Selection And Delivery
|
|
|
|
### Composition
|
|
|
|
Preserve existing explicit observers. Deliver each event to the explicit request
|
|
observer first, then any coordinator observer already composed with it, then
|
|
contextual observers in registration order, from inherited to newly added.
|
|
|
|
Preserve the existing synchronous callback and panic behavior. Concurrent
|
|
executions may invoke a shared observer concurrently; consumers remain
|
|
responsible for synchronizing their callback state. `ExecutionGroup` already
|
|
supports concurrent use.
|
|
|
|
### Ordinary And Nested Execution
|
|
|
|
`Run` selects contextual observation from its invocation context. Each execution
|
|
has its own observation identity, recorder, and accounting, including a nested
|
|
execution using a context passed to an application callback or injected client.
|
|
|
|
Inherited observer selection is distinct from private execution state. Existing
|
|
clearing of an enclosing execution's private state at a public execution boundary
|
|
must preserve the contextual observer selection. Inheriting observation does not
|
|
reuse an enclosing run's recorder, totals, or lifecycle.
|
|
|
|
### Prepared Execution
|
|
|
|
`RunPrepared` selects contextual observation from the context supplied when the
|
|
handle is executed. A handle prepared in one scope and executed in another
|
|
participates in the execution scope, not the preparation scope.
|
|
|
|
The explicit `RunRequest.Observer` remains retained by a successfully prepared
|
|
handle as today and composes with the execution-time contextual observers.
|
|
Preparation itself emits no execution events. Discarded handles and rejected
|
|
execution claims retain their existing observation behavior.
|
|
|
|
### Coordinated Execution
|
|
|
|
Continuation runs receive contextual observation through their ordinary `Run`
|
|
boundary. `CoordinateRequest.Observer` continues to select continuation runs
|
|
only. Additional calls made by application callbacks participate when they use
|
|
the scoped context, without being automatically selected by the coordinator's
|
|
explicit observer.
|
|
|
|
Contextual observers are incorporated once per execution, rather than being
|
|
re-added by coordination or another forwarding layer.
|
|
|
|
### Duplicate Registration And Accounting
|
|
|
|
Registering the same callback through multiple explicit or contextual paths may
|
|
deliver the same event more than once. Do not attempt to identify or deduplicate
|
|
arbitrary Go function values.
|
|
|
|
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
|
|
terminal-event deduplication. Selecting the same group through multiple paths
|
|
must not inflate its totals. Terminal usage is an aggregate, not an additional
|
|
contribution to be added to attempt usage. These rules apply to cost as well as
|
|
token counts.
|
|
|
|
## Backend Provider Selection
|
|
|
|
Add an optional backend `provider` field identifying the API implementation
|
|
whose conventions PromptKit should use. It identifies the service contacted,
|
|
not the organization that trained the selected model. Backend IDs remain resource
|
|
identities; names, namespaces, and endpoint hostnames do not select provider
|
|
behavior.
|
|
|
|
```yaml
|
|
promptkit:
|
|
backends:
|
|
- id: research-router
|
|
provider: openrouter
|
|
endpoint: https://openrouter.ai/api/v1
|
|
api_key_env: OPENROUTER_API_KEY
|
|
```
|
|
|
|
Normalize the final effective provider value by trimming whitespace and
|
|
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
|
|
Reject an unsupported nonempty normalized value as an ordinary configuration
|
|
error rather than silently using generic behavior. The supported selections for
|
|
this feature are `openai-compatible`, `openrouter`, and `vllm`.
|
|
|
|
Omitted/null/blank provider values select the same effective behavior as explicit
|
|
`provider: openai-compatible`. This is a runtime fallback, not an explicitly
|
|
sourced configuration value. Backend inspection retains the distinction between
|
|
an explicit provider declaration and an omitted/null/blank declaration.
|
|
|
|
The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation
|
|
and add their response-metadata and diagnostic conventions below.
|
|
|
|
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
|
the selected value through configuration capture and prepared execution, expose
|
|
it in backend inspection, and declare it in the bundled OpenRouter backend.
|
|
It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a
|
|
provider does not choose a model or alter the configured endpoint. An endpoint
|
|
override retains the provider selection.
|
|
|
|
### Default OpenAI-Compatible Provider
|
|
|
|
`openai-compatible` names PromptKit's current supported text Chat Completions
|
|
conventions. Preserve the existing shared behavior documented in the
|
|
[integration contract](../integrations/openai-compatible-chat.md), including:
|
|
|
|
- JSON POST requests to `/chat/completions` relative to the configured base;
|
|
- existing optional bearer-credential handling, including explicitly required
|
|
credential settings when selected;
|
|
- text messages, explicitly selected generation controls and structured-output
|
|
requests, and configured extra parameters;
|
|
- first-choice text extraction, existing supported token counts and their
|
|
presence-aware observation; and
|
|
- existing response validation, error handling, timeouts, cancellation, and
|
|
transient retries.
|
|
|
|
This selection supplies no default endpoint, model, API-key environment variable,
|
|
or sampling settings. It does not decode provider-reported monetary cost or
|
|
provide a default authentication-probe adapter. The backend zero-cost declaration
|
|
below remains available independently of provider selection. Without an explicit
|
|
authentication adapter, report the existing unsupported diagnostic outcome rather
|
|
than assuming authentication is inapplicable. Existing reachability and inference
|
|
diagnostics and explicit authentication adapters remain available.
|
|
|
|
"Lowest common denominator" identifies the shared protocol foundation; it does
|
|
not remove existing optional features or promise every backend supports them.
|
|
Continue sending caller-selected supported settings and return backend errors
|
|
normally, without capability negotiation or fallback requests. This is the
|
|
existing text Chat Completions scope, not a claim to support the full OpenAI API.
|
|
|
|
### vLLM Provider
|
|
|
|
`provider: vllm` selects the shared generation path plus vLLM-specific usage
|
|
normalization and a default authentication diagnostic. It supplies no default
|
|
endpoint, model, API-key environment variable, or sampling settings, and does not
|
|
create a backend definition automatically.
|
|
|
|
Preserve caller-selected reasoning settings, structured-output requests, and
|
|
extra parameters. Do not automatically change chat-template arguments, enable
|
|
thinking, or translate model-specific controls based on the provider label.
|
|
No separate generation protocol is needed; vLLM documents an
|
|
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
|
|
|
The token mappings are specified below. vLLM supplies no provider-reported cost;
|
|
cost remains unknown unless the backend explicitly declares the zero-cost
|
|
fallback below. Local hosting alone does not select that fallback. This provider
|
|
adds no price estimator, server administration, metrics collection, or automatic
|
|
capability/version probe.
|
|
|
|
### Smoke-Test Defaults And Explicit Overrides
|
|
|
|
When an authentication smoke test is explicitly requested, the selected provider
|
|
supplies its default probe convention. `provider: openrouter` supplies the
|
|
existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models`
|
|
convention described below. Preserve explicit
|
|
`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint`
|
|
settings as overrides:
|
|
|
|
- A supplied kind selects the diagnostic adapter instead of the provider default.
|
|
`kind: none` continues to declare authentication testing inapplicable.
|
|
- An omitted/null kind uses the provider default, if available. This permits an
|
|
endpoint override without repeating the provider's default kind.
|
|
- A supplied endpoint overrides the selected adapter's default destination.
|
|
An omitted/null endpoint uses that adapter's existing endpoint fallback.
|
|
|
|
An absent/null authentication block likewise leaves selection to the provider
|
|
default. Clearing the block removes its explicit overrides; `kind: none` is the
|
|
explicit way to declare authentication inapplicable despite a provider default.
|
|
Without a provider default or explicit adapter, retain existing unsupported
|
|
diagnostic behavior. Existing explicit adapters continue to work without a
|
|
provider declaration. Reachability and inference-default settings keep their
|
|
existing behavior.
|
|
|
|
These are conventions for explicitly requested diagnostics. Loading a backend,
|
|
constructing an engine, and ordinary generation do not initiate probes.
|
|
|
|
Resolve provider-supplied diagnostic defaults at the provider boundary without
|
|
writing them into the loaded configuration or recording them as explicit source
|
|
values. Configuration inspection preserves the distinction between configured
|
|
overrides and provider-supplied fallbacks, including their existing source
|
|
provenance.
|
|
|
|
### vLLM Authentication Diagnostic
|
|
|
|
The `vllm_models` adapter makes one GET request to `/models` relative to the
|
|
configured API base, supplying an available credential through the existing
|
|
credential rules. A base ending in `/v1` therefore produces `/v1/models`.
|
|
Existing explicit diagnostic endpoint overrides remain available. The adapter
|
|
can dispatch without a credential; selecting vLLM does not declare authentication
|
|
required or inapplicable.
|
|
|
|
Recognize HTTP 200 with a valid JSON object containing `object: "list"` and an
|
|
array-valued `data` as acceptance of the probe request, and a 401 as rejection.
|
|
An empty `data` array counts as acceptance; recognition does not require a
|
|
particular model or inspect model capabilities. Reuse the existing bounded
|
|
diagnostic decoding and framing rules. Unrecognized responses leave acceptance
|
|
unknown under the existing diagnostic reporting model. Report observed credential presence and
|
|
response facts without claiming that authentication is enforced or that models
|
|
listing establishes inference authorization. Preserve the shared override rules,
|
|
including `kind: none`; explicit `vllm_models` selection also works without a
|
|
`vllm` provider declaration, just as explicit OpenRouter adapters do.
|
|
|
|
vLLM documents `/v1/models` as protected when its API-key authentication is
|
|
configured, while `/health` does not establish authentication acceptance. See its
|
|
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
|
|
Retain existing reachability and inference diagnostic behavior rather than
|
|
adding health monitoring. These requests occur only during explicitly requested
|
|
diagnostics, never during ordinary generation or engine construction.
|
|
|
|
### Diagnostic Migration
|
|
|
|
This changes the current authentication metadata contract: an authentication
|
|
object currently requires `kind`, and clearing the object removes its configured
|
|
adapter. Under this roadmap, endpoint-only overrides become valid, and clearing
|
|
the object restores any selected provider's default. `kind: none` suppresses
|
|
that default. Explain these changes in the framework format documentation and
|
|
release notes, alongside the new provider field.
|
|
|
|
### Provider Behavior Ownership
|
|
|
|
Centralize provider-specific behavior within `internal/llm`, which already owns
|
|
OpenAI-compatible transport and OpenRouter authentication-response recognition.
|
|
Keep common request construction, credentials, HTTP execution, framing, and
|
|
common response fields shared. Provider-specific code owns additional response
|
|
metadata interpretation and authentication-probe conventions.
|
|
|
|
Use one provider-selection mechanism inside that package. Generation,
|
|
observation, and smoke-test coordination consume provider-neutral facts or invoke
|
|
generic diagnostic operations rather than repeating provider dispatch and
|
|
response interpretation. Explicit diagnostic adapter selection remains supported
|
|
at this boundary. Update the current OpenRouter-specific diagnostic collaborator
|
|
method to a generic operation as part of this consolidation.
|
|
|
|
Provider-specific files and straightforward dispatch are sufficient initially.
|
|
No new `internal/providers` package, registration framework, or broad interface
|
|
requiring every provider to implement every capability is needed.
|
|
This is configurable selection among PromptKit-supported provider implementations,
|
|
not a public API for downstream registration of additional implementations.
|
|
|
|
## Token And Cache Reporting
|
|
|
|
### OpenRouter Response Mapping
|
|
|
|
Use OpenRouter's documented Chat Completions usage fields. Its
|
|
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
|
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
|
|
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching)
|
|
define the following mapping, relative to `usage`:
|
|
|
|
| Quantity | Response field | PromptKit field |
|
|
| --- | --- | --- |
|
|
| Input tokens | `prompt_tokens` | Existing `PromptTokens` |
|
|
| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` |
|
|
| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` |
|
|
| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` |
|
|
| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` |
|
|
| Total tokens | `total_tokens` | Existing `TotalTokens` |
|
|
|
|
Preserve `CachedTokens` as the public field name and clarify its meaning as
|
|
cache-read tokens. Correct the OpenRouter cache-write mapping to the nested
|
|
field above; retain the generic provider's existing top-level handling. Do not
|
|
add the nested and top-level values together. OpenRouter uses its documented
|
|
nested value; an omitted nested value remains unknown.
|
|
|
|
Add `ReasoningTokens` to the public/domain token values, presence indicators,
|
|
and observed token usage, including conversions, result serialization, attempt
|
|
and terminal events, and group summaries. OpenRouter and vLLM supply this
|
|
additional built-in response interpretation; the generic provider preserves its
|
|
existing behavior. Injected clients may supply the canonical count and its
|
|
presence through the existing model-client boundary.
|
|
|
|
### vLLM Response Mapping And Availability
|
|
|
|
Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens`
|
|
counts. For `provider: vllm`, map these additional fields relative to `usage`:
|
|
|
|
| Response field | PromptKit field |
|
|
| --- | --- |
|
|
| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` |
|
|
| `prompt_tokens_details.cached_tokens` | `CachedTokens` |
|
|
| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` |
|
|
|
|
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
|
|
uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its
|
|
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
|
|
describe the creation count as prompt tokens counted as local prefix-cache
|
|
writes. Normalize that token activity to `CacheWriteTokens` without implying a
|
|
billable write charge. Use the selected provider's documented field, not a sum
|
|
of alternative cache-write fields; an absent vLLM creation count stays unknown.
|
|
|
|
Detail availability depends on server version and configuration. Current vLLM
|
|
gates prompt details on `--enable-prompt-tokens-details` and reasoning details on
|
|
a configured reasoning parser. Document these reporting settings, but do not
|
|
configure or discover them automatically. Older or differently configured
|
|
servers may omit the fields; accept those responses with the affected counts
|
|
unknown. Do not infer reasoning counts from generated text or cache writes from
|
|
input tokens minus cache reads.
|
|
|
|
Once normalized, these counts use the shared presence, aggregation, and
|
|
cache-hit rules below. No vLLM-specific result type, ratio, or group accounting
|
|
is needed.
|
|
|
|
### Count Relationships And Availability
|
|
|
|
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
|
|
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
|
|
vLLM likewise reports its reasoning detail within the generated-token usage.
|
|
Cache reads and writes describe input-token breakdowns. Present these as subsets
|
|
or details rather than additional consumption to add to input/output totals.
|
|
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
|
|
|
|
Consumers can derive non-reasoning output as completion tokens minus reasoning
|
|
tokens when both counts are available and consistent. No additional stored
|
|
non-reasoning count is needed in this scope. Capturing reasoning-token counts
|
|
does not capture reasoning text or change reasoning request settings.
|
|
|
|
Reuse existing per-field presence and aggregation rules: omitted/null means
|
|
unknown, explicit zero means known zero, and aggregate `Known` and `Complete`
|
|
flags describe the reported contributions across dispatched attempts. Include
|
|
initial generations, retries, and repairs through the existing observation
|
|
accounting, retaining terminal-only group aggregation and deduplication. Preserve
|
|
the existing distinction between integer-only result usage and presence-aware
|
|
observations; zero in `TokenUsage` alone does not establish provider reporting.
|
|
|
|
### Token Cache-Hit Ratio
|
|
|
|
Add this derived helper on the existing presence-aware usage type:
|
|
|
|
```go
|
|
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
|
|
```
|
|
|
|
The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero
|
|
to one. It measures the proportion of input tokens served from cache, not the
|
|
proportion of requests with a cache hit or monetary savings. The Boolean reports
|
|
whether a ratio is available; an unavailable result returns zero and false.
|
|
|
|
Return an available ratio only when both counts are known and complete, input
|
|
tokens are positive, and cache reads are between zero and input tokens inclusive.
|
|
Zero cache reads with known positive input yields an available zero ratio. Missing
|
|
or incomplete counts, zero input, or inconsistent counts leave it unavailable;
|
|
preserve the raw reported counts rather than clamping them or failing generation.
|
|
Cache-write availability does not affect this calculation.
|
|
|
|
For a run or group, divide the aggregated cache-read count by the aggregated
|
|
input count; do not average per-attempt or per-run percentages. Do not use
|
|
`reads / (reads + writes)`, because cache writes need not account for all
|
|
uncached input. The same helper works on attempts, terminal events, and group
|
|
summaries without storing another independently maintained field.
|
|
|
|
The ratio describes the included observations. A group ratio does not supersede
|
|
`TerminalCoverageComplete` or imply that all application work has finished.
|
|
For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
|
|
are known and complete, even if cache writes are unknown. Output 840 with
|
|
reasoning 600 still represents 840 output tokens, not 1,440.
|
|
|
|
## Optional Cost Reporting
|
|
|
|
### Initial Provider Support
|
|
|
|
Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter
|
|
defines as the amount charged to the account, denominated in USD. See its
|
|
[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
|
|
and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
|
|
Do not add `cost_details` breakdowns or combine upstream inference cost with the
|
|
top-level charge; those describe different facts and could double-count cost.
|
|
|
|
For the built-in client, select this interpretation through `provider: openrouter`,
|
|
regardless of backend identity or namespace. Do not infer USD semantics from an
|
|
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
|
Built-in provider-reported cost interpretation is an extension unique to
|
|
`openrouter` initially; `openai-compatible` and `vllm` do not interpret
|
|
`usage.cost`. Without an applicable zero-cost declaration, their built-in cost
|
|
remains unknown. No separate provider cost-decoding selector is needed.
|
|
Aggregation remains provider-neutral after the model-client boundary supplies
|
|
interpreted USD facts.
|
|
|
|
Use the cost included in the normal completion response. No additional usage
|
|
lookup, pricing request, or opt-in provider request parameter is needed.
|
|
|
|
### Explicit Backend Zero-Cost Declaration
|
|
|
|
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
|
|
|
|
```yaml
|
|
promptkit:
|
|
backends:
|
|
- id: local-inference
|
|
provider: vllm
|
|
endpoint: http://localhost:8000/v1
|
|
zero_cost: true
|
|
```
|
|
|
|
For each dispatched attempt, use a usable provider/model-client-reported cost
|
|
first, including an explicitly reported zero. If no usable reported cost is
|
|
available and the effective backend has `zero_cost: true`, supply USD zero with
|
|
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
|
|
it does not suppress a reported cost. Apply this policy to built-in and injected
|
|
client execution using the selected backend, regardless of its provider type.
|
|
|
|
The fallback also applies to dispatched attempts that fail, are canceled, or
|
|
produce no decodable response. It is an operator declaration, not a provider
|
|
observation. Non-dispatched work contributes no cost. Do not infer this setting
|
|
from a local address, provider name, token counts, or missing credentials, and
|
|
do not install it automatically on any backend. Without a selected backend
|
|
declaration, no configured-zero fallback applies.
|
|
|
|
Resolve the field through ordinary final ConfigKit composition and inheritance;
|
|
include it in typed backend helpers, configuration capture, and backend
|
|
inspection. Retain the effective policy through prepared execution without
|
|
reloading configuration. It is local accounting metadata, not an outbound
|
|
request parameter. `InspectBackend` shows the resolved declared setting,
|
|
preserving explicit `false`; omitted and null values can remain absent. Do not
|
|
manufacture an explicit false value for the runtime fallback or add null-tracking
|
|
machinery to resource inspection. `InspectConfiguration` retains ConfigKit's
|
|
source history and explicit/null/omitted distinctions. Source provenance belongs
|
|
to configuration inspection, not resource inspection.
|
|
|
|
This supports an explicit zero declaration, not configurable input/output or
|
|
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
|
|
cost estimation. It does not alter credentials, generation, or token reporting.
|
|
|
|
### Monetary Representation
|
|
|
|
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
|
as `CostUSD`, rather than introducing a generic amount with an implicit currency.
|
|
At the individual model-client response boundary, use an optional amount
|
|
(`*float64`): nil means unknown, and a pointer to zero means reported zero.
|
|
Injected clients may supply the same optional USD value without needing an
|
|
OpenRouter backend; no provider-specific interface is needed.
|
|
|
|
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
|
remain token counts. For run results, execution events, and group summaries,
|
|
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
|
`Complete bool`, following the existing observed-count conventions, plus the
|
|
source metadata below. Successful run results should expose cost even when no
|
|
observer was registered; observers also provide per-attempt and terminal cost,
|
|
including available facts on failure.
|
|
|
|
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
|
addition, and the public API lightweight. Do not round to cents or another fixed
|
|
scale during capture or aggregation; consumers choose display precision.
|
|
Decimal strings or a decimal arithmetic dependency would be appropriate for
|
|
exact financial ledgers, while fixed-scale integers would require a selected
|
|
precision and rounding policy. Neither is warranted for this reporting feature.
|
|
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
|
|
|
### Cost-Source Metadata
|
|
|
|
Give the presence-aware cost value a `Source` discriminator with these meanings:
|
|
|
|
- `provider`: a cost supplied by the provider or injected model-client response.
|
|
This records the reporting path, not independent verification of a charge.
|
|
- `configured_zero`: zero supplied by the explicit backend fallback.
|
|
- `mixed`: an aggregate includes both provider and configured-zero contributions.
|
|
- Empty: no known contribution has a source. This covers unknown cost and the
|
|
known, complete zero aggregate when no attempts were dispatched.
|
|
|
|
Derive aggregate source from the union of known contribution sources, including
|
|
zero-valued contributions. Unknown contributions supply no source but affect
|
|
completeness. A provider-only subtotal can therefore be incomplete without being
|
|
`mixed`. Carry this metadata consistently through attempt events, terminal
|
|
events, successful results, and group summaries. Reuse existing aggregation and
|
|
terminal deduplication rather than adding a parallel collector or cost history.
|
|
|
|
Consumers can distinguish reported charges, configured zeros, and mixed totals
|
|
without treating configured values as provider reports. Source metadata is
|
|
separate from configuration inspection and does not require provenance traversal
|
|
during execution.
|
|
|
|
### Presence, Aggregation, And Failures
|
|
|
|
- An absent or null provider cost is unknown unless the backend's explicit
|
|
zero-cost fallback supplies it. A reported numeric zero remains known zero
|
|
with a provider source, even when the fallback is enabled.
|
|
- On a dispatched attempt, a usable reported cost or configured-zero fallback
|
|
is known and complete for that attempt. Cost completeness is independent of
|
|
token-field completeness and does not assert provider reporting.
|
|
- Sum known costs across dispatched attempts, including initial generations,
|
|
retries, and repairs. `Known` means at least one attempt has an available cost
|
|
from either source; `Complete` means every dispatched attempt does under this
|
|
policy. Unknown values contribute no amount, but make the total incomplete.
|
|
With no dispatched attempts, follow the existing known, complete zero
|
|
convention, with an empty source.
|
|
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
|
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
|
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
|
from OpenRouter mixed with unreported local-backend cost is incomplete unless
|
|
those local attempts have an explicit configured-zero fallback. Combine
|
|
cost-source metadata as described above without changing token accounting.
|
|
- Keep valid cost observed from a decoded successful envelope when a later
|
|
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
|
error bodies continue to supply no provider usage, but the configured-zero
|
|
fallback still applies to dispatched attempts. Without that fallback,
|
|
cancellation or failure may leave incomplete cost reporting even when billable
|
|
upstream work occurred.
|
|
- Treat an unusable optional cost value, including a nonnumeric value or one
|
|
not representable as a finite `float64`, as unknown rather than making an
|
|
otherwise valid generation fail; the zero-cost fallback can then apply. This
|
|
does not relax existing response framing or content validation. An injected
|
|
client likewise cannot supply a nonfinite usable cost.
|
|
|
|
Completeness describes coverage by available reported or explicitly configured
|
|
cost, not verification of the provider's billing or proof that every provider
|
|
reported a charge. Preserve usable reported amounts without pricing inference or
|
|
reconciliation.
|
|
|
|
For a successful run, result cost and terminal-event cost describe the same
|
|
dispatched attempts and must agree on value, presence, completeness, and source.
|
|
Result cost must also be the same with observation enabled or disabled. For
|
|
example, an unreported failed attempt followed by a successful attempt reporting USD
|
|
0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
|
|
source when no zero-cost fallback applies. Apply the same accounting through
|
|
repairs and ordinary or prepared execution. An invocation with fully reported
|
|
OpenRouter charges and configured-zero local calls can have a complete subtotal
|
|
with a mixed source, without claiming all providers reported costs.
|
|
|
|
Group cost completeness concerns the terminal observations included in its
|
|
subtotal. If two completed runs reported all their costs but a third observed
|
|
run has no terminal event, the subtotal may have `Complete: true` while
|
|
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
|
|
coverage field; no additional completion flag or group lifecycle is needed.
|
|
|
|
## Accounting And Responsibility Boundaries
|
|
|
|
Reuse existing events, token presence/completeness, and group summaries, extending
|
|
them with the cost facts above. Available usage remains observable after an error
|
|
or cancellation under the existing execution-observation behavior; this does not
|
|
promise that upstream providers report usage for every attempt.
|
|
|
|
PromptKit owns observer selection, event generation, and aggregation. CommandKit
|
|
and other consumers own invocation lifecycle, CLI flags, presentation, output
|
|
delivery, and interpretation. This feature adds no CommandKit dependency or
|
|
application-specific concepts to PromptKit.
|
|
|
|
Smoke diagnostics retain their separate report/evidence model and are not
|
|
enrolled in ordinary execution observation by this change. Reasoning-text
|
|
capture, changes to reasoning or caching request behavior, additional built-in
|
|
cost providers, configurable token-rate estimation, fixed nonzero request
|
|
pricing, multiple currencies, currency conversion, pricing estimates,
|
|
billing reconciliation, persistent history, and logging infrastructure are
|
|
outside this scope.
|
|
|
|
## Work Needed And Completion Criteria
|
|
|
|
- Add the context helper and connect observer selection to ordinary and prepared
|
|
execution, keeping contextual selection separate from private run state.
|
|
- Preserve existing explicit selection and coordinator behavior while applying
|
|
the composition and lifetime rules above. Preserve existing observer selection
|
|
when no contextual observer is selected.
|
|
- Add `provider` to the backend configuration model, typed source helpers,
|
|
capture, inspection, and execution plumbing. Update the bundled OpenRouter
|
|
backend to select it explicitly. Retain the selection through prepared runs.
|
|
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
|
|
prepared-execution plumbing. Apply the fallback outside provider-specific
|
|
decoding so it works with built-in and injected clients. Verify composition
|
|
and inheritance, omitted/null/false behavior, explicit overrides, provider
|
|
independence, configuration-inspection provenance, and absence from provider
|
|
bodies. Verify that resource inspection preserves explicit false without
|
|
manufacturing values or source-history fields for omitted/null settings.
|
|
- Consolidate provider-specific response and authentication-probe behavior in
|
|
`internal/llm`; keep coordination and accounting provider-neutral. Cover
|
|
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
|
and existing explicit probes without a provider selection.
|
|
- Verify provider selection independently of backend names and namespaces,
|
|
including renamed and inherited backends, equivalent omitted/null/blank values,
|
|
trimming and case normalization, rejection of unsupported nonempty values,
|
|
and retention through endpoint overrides. Confirm provider metadata is not
|
|
serialized into the completion request body. Verify that explicit
|
|
`openai-compatible` and omitted/null/blank selection preserve the same existing
|
|
shared behavior without manufacturing explicit configuration provenance.
|
|
- Verify that provider diagnostic defaults do not become explicit values in the
|
|
loaded configuration or replace source provenance in inspection. Cover the
|
|
changed clearing behavior and endpoint-only authentication overrides.
|
|
- Carry OpenRouter reasoning-token counts through token values, presence,
|
|
results, events, and group summaries. Verify nested OpenRouter cache-write
|
|
decoding, including omitted/null/zero values and the nested value taking
|
|
precedence when a top-level value is also present. Preserve generic-provider
|
|
token mappings and existing field meanings.
|
|
- Add the `vllm` provider selection, reasoning and cache-detail mappings, and
|
|
`vllm_models` authentication adapter. Cover supplied/missing credentials,
|
|
recognized acceptance (including an empty models list), rejection,
|
|
unrecognized responses, provider defaults, and explicit kind/endpoint overrides,
|
|
including `none` and adapter selection
|
|
without a provider declaration. Verify that ordinary generation performs no
|
|
diagnostic requests and leaves monetary cost unknown unless the backend
|
|
explicitly declares the zero-cost fallback.
|
|
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
|
|
details, including responses representative of servers without detail
|
|
reporting enabled. Verify `created_cache_tokens` mapping independently of
|
|
OpenRouter's cache-write field and carry normalized counts through results,
|
|
events, and groups. Do not require a live server or version discovery.
|
|
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
|
|
incomplete counts, zero input, inconsistent counts, independence from
|
|
cache-write availability, and token-weighted aggregate ratios. Verify subset
|
|
accounting without adding reasoning or cached tokens to the reported totals.
|
|
- Carry optional OpenRouter cost through response decoding, internal/public
|
|
conversions, successful run results, attempt and terminal events, and existing
|
|
group summaries. Keep result cost available without registering an observer.
|
|
Preserve existing token fields and accounting.
|
|
- Carry cost-source metadata through results, events, conversions, and group
|
|
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
|
|
including zero-valued reported contributions and empty/non-dispatched cases.
|
|
Verify source union and terminal deduplication independently of cost value.
|
|
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
|
|
null, or unusable reported cost permits fallback. Cover dispatched failures,
|
|
cancellation, undecodable responses, retries, repairs, ordinary/prepared
|
|
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
|
|
and declared zero-cost local calls can produce complete cost totals while
|
|
token completeness remains independent.
|
|
- Verify that successful-result cost agrees with terminal-event cost and is
|
|
unchanged when observation is disabled, including the unreported-retry example
|
|
above. Verify group snapshots with fully reported completed runs alongside a
|
|
member without a terminal observation.
|
|
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
|
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
|
and independent scopes on a reused engine.
|
|
- Cover callback ordering, nested contextual registration, nil registration,
|
|
duplicate group enrollment, prepared-handle scope selection, and existing
|
|
preparation/discard/rejected-claim behavior.
|
|
- Verify aggregate accounting through retries and repairs, available usage after
|
|
failures or cancellation, and known zero versus unknown or incomplete counts.
|
|
Use offline fixtures and injected clients; live inference is unnecessary.
|
|
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
|
|
generic built-in selection, optional injected-client cost, mixed known
|
|
and unknown totals, repair/retry aggregation, and duplicate group registration.
|
|
Use tolerant comparisons where floating-point addition requires them; do not
|
|
turn telemetry into an exact-decimal guarantee.
|
|
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
|
aggregation example to the consumer observation guide, including cost
|
|
availability, units, completeness, and source. Explain that configured zeros
|
|
can make cost complete without provider reporting. Document the reasoning/cache
|
|
fields and cache-hit helper in their owning GoDoc and illustrate input/output
|
|
breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
|
|
OpenRouter cost decoding in the
|
|
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
|
and update internal observation and model-client documentation for changed flows.
|
|
Document `provider`, its smoke-test defaults and override semantics in the
|
|
framework format reference and relevant public GoDoc; update inspection and
|
|
diagnostic documentation with the implemented behavior. Document `zero_cost`
|
|
and its fallback precedence, inspection behavior, and prepared lifetime in the
|
|
relevant format, GoDoc, and consumer owners. Include the diagnostic
|
|
migration in release notes. Include vLLM usage availability, server reporting
|
|
settings, the meaning of cache creation, and the limits of its authentication
|
|
diagnostic in the relevant consumer and integration documentation.
|
|
|
|
## Implementation Planning Guidance
|
|
|
|
Organize the eventual implementation plan around two workstreams:
|
|
|
|
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
|
and nested execution boundaries and the existing group collector.
|
|
2. Backend provider selection and consolidation within `internal/llm`, including
|
|
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
|
|
extensions, provider-reported cost, and configured-zero fallback through
|
|
decoding, accounting, public results, events, and group summaries, plus the
|
|
derived cache-hit helper. Include the corresponding configuration and API
|
|
documentation.
|
|
|
|
The second area is more extensive than the context helper. Keep backend provider
|
|
configuration, diagnostic consolidation, token/cache reporting, and cost/source
|
|
accounting in separately reviewable implementation units rather than one large
|
|
implementation prompt.
|
|
Avoid coupling delivery of contextual observation to provider cost support.
|
|
|
|
Today, successful results accumulate generation responses in the runner, while
|
|
optional observation recorders account for dispatched attempts. Adding cost to
|
|
response objects alone does not preserve completeness across unreported retries.
|
|
Reuse a small accounting component for result and observation paths so they
|
|
apply the same rules, including when no observer is selected. Preserve existing
|
|
token accounting rather than expanding this work into a redesign of its public
|
|
contract.
|
|
|
|
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
|
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
|
overflowing value fail decoding of the entire response. Decode the optional
|
|
field separately so an unusable provider cost becomes unknown before applying
|
|
any configured-zero fallback, without changing existing framing or content
|
|
validation. This needs no general numeric-conversion layer.
|
|
|
|
Apply configured-zero fallback after decoding, outside provider-specific code,
|
|
so built-in and injected clients share the same policy. Aggregate cost sources
|
|
from known contributions even when their amounts are zero; a reported zero and
|
|
a configured zero together have a mixed source. Keep subtotal completeness
|
|
separate from terminal coverage, using the existing coverage field and the
|
|
examples above. These rules belong in shared accounting rather than separate
|
|
provider, result, and group implementations.
|
|
|
|
Use the completion criteria above to select focused offline checks for each
|
|
unit. This roadmap owns target behavior; a separate implementation plan owns
|
|
the detailed work sequence.
|