Add optional OpenRouter cost reporting to the observation roadmap

This commit is contained in:
2026-11-22 09:48:38 +00:00
parent 1070be8d89
commit 03209947f6

View File

@@ -7,7 +7,8 @@ Status: accepted scope; implementation pending.
Allow consumers to select execution observation once for an application-owned Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested, operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should concurrent, and multi-engine calls using that operation's context should
participate automatically. participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost support initially limited to OpenRouter.
The motivating consumer is CommandKit's optional `ckprompt` integration, which The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline. could use this facility to report token usage for an entire command or pipeline.
@@ -28,8 +29,9 @@ PromptKit v0.24.0 already provides:
execution, and [CoordinateRequest.Observer](../../coordinate.go) for execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs. coordinator-selected continuation runs.
The missing capability is contextual observer selection. Applications currently The missing capabilities are contextual observer selection and provider-reported
have to enroll each request explicitly. The existing cost accounting. Applications currently have to enroll each request explicitly,
and PromptKit does not expose provider charges. The existing
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) [consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
describes that workflow. describes that workflow.
@@ -46,7 +48,10 @@ or derived contexts that retain its values. It composes with any observer alread
associated with the context; it does not replace the inherited observer. A nil associated with the context; it does not replace the inherited observer. A nil
observer leaves the inherited selection unchanged. observer leaves the inherited selection unchanged.
No engine option, process-wide default, or new aggregation type is needed. No engine option, process-wide default, or replacement group type is needed.
Extend the existing result, event, and group surfaces with cost metadata as
described below.
The intended consumer workflow is: The intended consumer workflow is:
```go ```go
@@ -118,14 +123,90 @@ arbitrary Go function values.
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
terminal-event deduplication. Selecting the same group through multiple paths terminal-event deduplication. Selecting the same group through multiple paths
must not inflate its totals. Terminal usage is an aggregate, not an additional must not inflate its totals. Terminal usage is an aggregate, not an additional
contribution to be added to attempt usage. contribution to be added to attempt usage. These rules apply to cost as well as
token counts.
## Optional Provider-Reported Cost
### Initial Provider Support
Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter
defines as the amount charged to the account, denominated in USD. See its
[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
Do not add `cost_details` breakdowns or combine upstream inference cost with the
top-level charge; those describe different facts and could double-count cost.
For the built-in client, select this interpretation for the resolved
`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's
field named `cost`, or add endpoint discovery or verification. An endpoint
override on that backend continues to select its OpenRouter response contract.
Other built-in backend selections report unknown cost initially. Additional
provider adapters or cost-capability configuration are not needed for this scope.
Use the cost included in the normal completion response. No additional usage
lookup, pricing request, or opt-in provider request parameter is needed.
### Monetary Representation
Use `float64` amounts in USD. Make the unit explicit in public field names such
as `CostUSD`, rather than introducing a generic amount with an implicit currency.
At the individual model-client response boundary, use an optional amount
(`*float64`): nil means unknown, and a pointer to zero means reported zero.
Injected clients may supply the same optional USD value without needing an
OpenRouter backend; no provider-specific interface is needed.
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
remain token counts. For run results, execution events, and group summaries,
use a small presence-aware cost value with `Value float64`, `Known bool`, and
`Complete bool`, following the existing observed-count conventions. Successful
run results should expose cost even when no observer was registered; observers
also provide per-attempt and terminal cost, including available facts on failure.
Binary floating point is sufficient for operational telemetry and keeps decoding,
addition, and the public API lightweight. Do not round to cents or another fixed
scale during capture or aggregation; consumers choose display precision.
Decimal strings or a decimal arithmetic dependency would be appropriate for
exact financial ledgers, while fixed-scale integers would require a selected
precision and rounding policy. Neither is warranted for this reporting feature.
The API does not promise exact decimal arithmetic or invoice reconciliation.
### Presence, Aggregation, And Failures
- An absent or null cost is unknown. Explicit numeric zero is known zero.
A backend that reports no cost remains unknown even when a consumer knows it
has no external charge.
- On a dispatched attempt, a usable reported cost is known and complete for
that attempt. Cost completeness is independent of token-field completeness.
- Sum known costs across dispatched attempts, including initial generations,
retries, and repairs. `Known` means at least one attempt reported cost;
`Complete` means every dispatched attempt did. Unknown values contribute no
amount, but make the total incomplete. With no dispatched attempts, follow
the existing known, complete zero convention.
- Group summaries reuse existing terminal-only aggregation and deduplication:
sum known terminal values, OR their `Known` flags, and AND their `Complete`
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
from OpenRouter mixed with unreported local-backend cost is incomplete.
- Keep valid cost observed from a decoded successful envelope when a later
choice/content check fails, as with token usage. Invalid framing and non-2xx
error bodies continue to supply no usage. Cancellation or failure may leave
incomplete reporting even when billable upstream work occurred.
- Treat an unusable optional cost value, including a nonnumeric value or one
not representable as a finite `float64`, as unknown rather than making an
otherwise valid generation fail. This does not relax existing response framing
or content validation. An injected client likewise cannot supply a nonfinite
usable cost.
Completeness describes coverage of reported values, not verification of the
provider's billing. Preserve reported usable amounts without pricing inference
or reconciliation.
## Accounting And Responsibility Boundaries ## Accounting And Responsibility Boundaries
Reuse existing events, usage presence/completeness, and group summaries. Available Reuse existing events, token presence/completeness, and group summaries, extending
usage remains observable after an error or cancellation under the existing them with the cost facts above. Available usage remains observable after an error
execution-observation behavior; this does not promise that upstream providers or cancellation under the existing execution-observation behavior; this does not
report usage for every attempt. promise that upstream providers report usage for every attempt.
PromptKit owns observer selection, event generation, and aggregation. CommandKit PromptKit owns observer selection, event generation, and aggregation. CommandKit
and other consumers own invocation lifecycle, CLI flags, presentation, output and other consumers own invocation lifecycle, CLI flags, presentation, output
@@ -133,18 +214,22 @@ delivery, and interpretation. This feature adds no CommandKit dependency or
application-specific concepts to PromptKit. application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Additional usage enrolled in ordinary execution observation by this change. Reasoning-token
fields such as reasoning tokens and monetary cost, pricing estimates, billing capture, additional built-in cost providers, multiple currencies, currency
reconciliation, persistent history, and logging infrastructure are outside this conversion, pricing estimates, billing reconciliation, persistent history, and
scope. logging infrastructure are outside this scope.
## Work Needed And Completion Criteria ## Work Needed And Completion Criteria
- Add the context helper and connect observer selection to ordinary and prepared - Add the context helper and connect observer selection to ordinary and prepared
execution, keeping contextual selection separate from private run state. execution, keeping contextual selection separate from private run state.
- Preserve existing explicit selection and coordinator behavior while applying - Preserve existing explicit selection and coordinator behavior while applying
the composition and lifetime rules above. Keep existing execution behavior the composition and lifetime rules above. Preserve existing observer selection
unchanged when no contextual observer is selected. when no contextual observer is selected.
- Carry optional OpenRouter cost through response decoding, internal/public
conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer.
Preserve existing token fields and accounting.
- Cover ordinary and prepared execution, coordinated continuations, nested calls - Cover ordinary and prepared execution, coordinated continuations, nested calls
using derived contexts, concurrent calls, multiple engines sharing a scope, using derived contexts, concurrent calls, multiple engines sharing a scope,
and independent scopes on a reused engine. and independent scopes on a reused engine.
@@ -154,9 +239,16 @@ scope.
- Verify aggregate accounting through retries and repairs, available usage after - Verify aggregate accounting through retries and repairs, available usage after
failures or cancellation, and known zero versus unknown or incomplete counts. failures or cancellation, and known zero versus unknown or incomplete counts.
Use offline fixtures and injected clients; live inference is unnecessary. Use offline fixtures and injected clients; live inference is unnecessary.
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
non-OpenRouter built-in selection, optional injected-client cost, mixed known
and unknown totals, repair/retry aggregation, and duplicate group registration.
Use tolerant comparisons where floating-point addition requires them; do not
turn telemetry into an exact-decimal guarantee.
- Document the exact helper contract in GoDoc and add a focused invocation-level - Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide. Update internal aggregation example to the consumer observation guide, including cost
observation documentation where its selection flow changes. availability, units, and completeness. Document OpenRouter cost decoding in
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows.
This roadmap defines the target behavior. A separate implementation plan can This roadmap defines the target behavior. A separate implementation plan can
divide the work into appropriately sized stages. divide the work into appropriately sized stages.