Add reasoning and cache usage reporting to the roadmap

This commit is contained in:
2026-11-22 15:54:02 +00:00
parent 80649c579d
commit 23ad9e2a26

View File

@@ -8,7 +8,9 @@ Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested, operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should concurrent, and multi-engine calls using that operation's context should
participate automatically. Include optional provider-reported cost alongside participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost support initially limited to OpenRouter. token usage, with built-in cost support initially limited to OpenRouter. Extend
OpenRouter token reporting with reasoning counts, documented cache reads and
writes, and a derived token cache-hit ratio.
The motivating consumer is CommandKit's optional `ckprompt` integration, which The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline. could use this facility to report token usage for an entire command or pipeline.
@@ -29,15 +31,18 @@ PromptKit v0.24.0 already provides:
execution, and [CoordinateRequest.Observer](../../coordinate.go) for execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs. coordinator-selected continuation runs.
The missing capabilities are contextual observer selection and provider-reported The missing capabilities are contextual observer selection, provider-reported
cost accounting. Applications currently have to enroll each request explicitly, cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
and PromptKit does not expose provider charges. The existing OpenRouter cache writes also need their documented nested response mapping;
the current client reads a top-level `usage.cache_write_tokens` field instead.
Applications currently have to enroll each request explicitly, and PromptKit
does not expose provider charges. The existing
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) [consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
describes that workflow. describes that workflow.
## Target Public API ## Target Public API
Add one public helper: Add a public context helper:
```go ```go
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
@@ -49,8 +54,8 @@ associated with the context; it does not replace the inherited observer. A nil
observer leaves the inherited selection unchanged. observer leaves the inherited selection unchanged.
No engine option, process-wide default, or replacement group type is needed. No engine option, process-wide default, or replacement group type is needed.
Extend the existing result, event, and group surfaces with cost metadata as Extend the existing token types and result, event, and group surfaces with the
described below. token and cost metadata described below, plus a derived cache-hit helper.
The intended consumer workflow is: The intended consumer workflow is:
@@ -258,6 +263,91 @@ requiring every provider to implement every capability is needed.
This is configurable selection among PromptKit-supported provider implementations, This is configurable selection among PromptKit-supported provider implementations,
not a public API for downstream registration of additional implementations. not a public API for downstream registration of additional implementations.
## OpenRouter Token And Cache Reporting
### Response Mapping
Use OpenRouter's documented Chat Completions usage fields. Its
[response schema](https://openrouter.ai/docs/api_reference/overview),
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching)
define the following mapping, relative to `usage`:
| Quantity | Response field | PromptKit field |
| --- | --- | --- |
| Input tokens | `prompt_tokens` | Existing `PromptTokens` |
| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` |
| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` |
| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` |
| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` |
| Total tokens | `total_tokens` | Existing `TotalTokens` |
Preserve `CachedTokens` as the public field name and clarify its meaning as
cache-read tokens. Correct the OpenRouter cache-write mapping to the nested
field above; retain the generic provider's existing top-level handling. Do not
add the nested and top-level values together. OpenRouter uses its documented
nested value; an omitted nested value remains unknown.
Add `ReasoningTokens` to the public/domain token values, presence indicators,
and observed token usage, including conversions, result serialization, attempt
and terminal events, and group summaries. Initially, OpenRouter supplies this
additional built-in response interpretation; the generic provider preserves its
existing behavior. Injected clients may supply the canonical count and its
presence through the existing model-client boundary.
### Count Relationships And Availability
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
Cache reads and writes describe input-token breakdowns. Present these as subsets
or details rather than additional consumption to add to input/output totals.
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
Consumers can derive non-reasoning output as completion tokens minus reasoning
tokens when both counts are available and consistent. No additional stored
non-reasoning count is needed in this scope. Capturing reasoning-token counts
does not capture reasoning text or change reasoning request settings.
Reuse existing per-field presence and aggregation rules: omitted/null means
unknown, explicit zero means known zero, and aggregate `Known` and `Complete`
flags describe the reported contributions across dispatched attempts. Include
initial generations, retries, and repairs through the existing observation
accounting, retaining terminal-only group aggregation and deduplication. Preserve
the existing distinction between integer-only result usage and presence-aware
observations; zero in `TokenUsage` alone does not establish provider reporting.
### Token Cache-Hit Ratio
Add this derived helper on the existing presence-aware usage type:
```go
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
```
The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero
to one. It measures the proportion of input tokens served from cache, not the
proportion of requests with a cache hit or monetary savings. The Boolean reports
whether a ratio is available; an unavailable result returns zero and false.
Return an available ratio only when both counts are known and complete, input
tokens are positive, and cache reads are between zero and input tokens inclusive.
Zero cache reads with known positive input yields an available zero ratio. Missing
or incomplete counts, zero input, or inconsistent counts leave it unavailable;
preserve the raw reported counts rather than clamping them or failing generation.
Cache-write availability does not affect this calculation.
For a run or group, divide the aggregated cache-read count by the aggregated
input count; do not average per-attempt or per-run percentages. Do not use
`reads / (reads + writes)`, because cache writes need not account for all
uncached input. The same helper works on attempts, terminal events, and group
summaries without storing another independently maintained field.
The ratio describes the included observations. A group ratio does not supersede
`TerminalCoverageComplete` or imply that all application work has finished.
For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
are known and complete, even if cache writes are unknown. Output 840 with
reasoning 600 still represents 840 output tokens, not 1,440.
## Optional Provider-Reported Cost ## Optional Provider-Reported Cost
### Initial Provider Support ### Initial Provider Support
@@ -360,10 +450,11 @@ delivery, and interpretation. This feature adds no CommandKit dependency or
application-specific concepts to PromptKit. application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Reasoning-token enrolled in ordinary execution observation by this change. Reasoning-text
capture, additional built-in cost providers, multiple currencies, currency capture, changes to reasoning or caching request behavior, additional built-in
conversion, pricing estimates, billing reconciliation, persistent history, and cost providers, multiple currencies, currency conversion, pricing estimates,
logging infrastructure are outside this scope. billing reconciliation, persistent history, and logging infrastructure are
outside this scope.
## Work Needed And Completion Criteria ## Work Needed And Completion Criteria
@@ -389,6 +480,15 @@ logging infrastructure are outside this scope.
- Verify that provider diagnostic defaults do not become explicit values in the - Verify that provider diagnostic defaults do not become explicit values in the
loaded configuration or replace source provenance in inspection. Cover the loaded configuration or replace source provenance in inspection. Cover the
changed clearing behavior and endpoint-only authentication overrides. changed clearing behavior and endpoint-only authentication overrides.
- Carry OpenRouter reasoning-token counts through token values, presence,
results, events, and group summaries. Verify nested OpenRouter cache-write
decoding, including omitted/null/zero values and the nested value taking
precedence when a top-level value is also present. Preserve generic-provider
token mappings and existing field meanings.
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
incomplete counts, zero input, inconsistent counts, independence from
cache-write availability, and token-weighted aggregate ratios. Verify subset
accounting without adding reasoning or cached tokens to the reported totals.
- Carry optional OpenRouter cost through response decoding, internal/public - Carry optional OpenRouter cost through response decoding, internal/public
conversions, successful run results, attempt and terminal events, and existing conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer. group summaries. Keep result cost available without registering an observer.
@@ -413,7 +513,9 @@ logging infrastructure are outside this scope.
turn telemetry into an exact-decimal guarantee. turn telemetry into an exact-decimal guarantee.
- Document the exact helper contract in GoDoc and add a focused invocation-level - Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide, including cost aggregation example to the consumer observation guide, including cost
availability, units, and completeness. Document OpenRouter cost decoding in availability, units, and completeness. Document the reasoning/cache fields and
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
without double-counting. Document OpenRouter token and cost decoding in
the [integration contract](../integrations/openai-compatible-chat.md#response-handling), the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows. and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the Document `provider`, its smoke-test defaults and override semantics in the
@@ -428,13 +530,15 @@ Organize the eventual implementation plan around two workstreams:
1. Contextual observer selection, including ordinary, prepared, coordinated, 1. Contextual observer selection, including ordinary, prepared, coordinated,
and nested execution boundaries and the existing group collector. and nested execution boundaries and the existing group collector.
2. Backend provider selection and consolidation within `internal/llm`, including 2. Backend provider selection and consolidation within `internal/llm`, including
smoke-test defaults and explicit overrides, followed by provider-reported cost smoke-test defaults and explicit overrides, followed by OpenRouter token
through decoding, accounting, public results, events, and group summaries. extensions and provider-reported cost through decoding, accounting, public
results, events, and group summaries, plus the derived cache-hit helper.
Include the corresponding configuration and API documentation. Include the corresponding configuration and API documentation.
The second area is more extensive than the context helper. Keep backend provider The second area is more extensive than the context helper. Keep backend provider
configuration, diagnostic consolidation, and cost accounting in separately configuration, diagnostic consolidation, token/cache reporting, and cost
reviewable implementation units rather than one large implementation prompt. accounting in separately reviewable implementation units rather than one large
implementation prompt.
Complete the remaining provider discussion before preparing that plan; avoid Complete the remaining provider discussion before preparing that plan; avoid
coupling delivery of contextual observation to provider cost support. coupling delivery of contextual observation to provider cost support.