Add reasoning and cache usage reporting to the roadmap
This commit is contained in:
@@ -8,7 +8,9 @@ Allow consumers to select execution observation once for an application-owned
|
|||||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||||
concurrent, and multi-engine calls using that operation's context should
|
concurrent, and multi-engine calls using that operation's context should
|
||||||
participate automatically. Include optional provider-reported cost alongside
|
participate automatically. Include optional provider-reported cost alongside
|
||||||
token usage, with built-in cost support initially limited to OpenRouter.
|
token usage, with built-in cost support initially limited to OpenRouter. Extend
|
||||||
|
OpenRouter token reporting with reasoning counts, documented cache reads and
|
||||||
|
writes, and a derived token cache-hit ratio.
|
||||||
|
|
||||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||||
could use this facility to report token usage for an entire command or pipeline.
|
could use this facility to report token usage for an entire command or pipeline.
|
||||||
@@ -29,15 +31,18 @@ PromptKit v0.24.0 already provides:
|
|||||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||||
coordinator-selected continuation runs.
|
coordinator-selected continuation runs.
|
||||||
|
|
||||||
The missing capabilities are contextual observer selection and provider-reported
|
The missing capabilities are contextual observer selection, provider-reported
|
||||||
cost accounting. Applications currently have to enroll each request explicitly,
|
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
|
||||||
and PromptKit does not expose provider charges. The existing
|
OpenRouter cache writes also need their documented nested response mapping;
|
||||||
|
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
||||||
|
Applications currently have to enroll each request explicitly, and PromptKit
|
||||||
|
does not expose provider charges. The existing
|
||||||
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
||||||
describes that workflow.
|
describes that workflow.
|
||||||
|
|
||||||
## Target Public API
|
## Target Public API
|
||||||
|
|
||||||
Add one public helper:
|
Add a public context helper:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
||||||
@@ -49,8 +54,8 @@ associated with the context; it does not replace the inherited observer. A nil
|
|||||||
observer leaves the inherited selection unchanged.
|
observer leaves the inherited selection unchanged.
|
||||||
|
|
||||||
No engine option, process-wide default, or replacement group type is needed.
|
No engine option, process-wide default, or replacement group type is needed.
|
||||||
Extend the existing result, event, and group surfaces with cost metadata as
|
Extend the existing token types and result, event, and group surfaces with the
|
||||||
described below.
|
token and cost metadata described below, plus a derived cache-hit helper.
|
||||||
|
|
||||||
The intended consumer workflow is:
|
The intended consumer workflow is:
|
||||||
|
|
||||||
@@ -258,6 +263,91 @@ requiring every provider to implement every capability is needed.
|
|||||||
This is configurable selection among PromptKit-supported provider implementations,
|
This is configurable selection among PromptKit-supported provider implementations,
|
||||||
not a public API for downstream registration of additional implementations.
|
not a public API for downstream registration of additional implementations.
|
||||||
|
|
||||||
|
## OpenRouter Token And Cache Reporting
|
||||||
|
|
||||||
|
### Response Mapping
|
||||||
|
|
||||||
|
Use OpenRouter's documented Chat Completions usage fields. Its
|
||||||
|
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
||||||
|
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
|
||||||
|
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching)
|
||||||
|
define the following mapping, relative to `usage`:
|
||||||
|
|
||||||
|
| Quantity | Response field | PromptKit field |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Input tokens | `prompt_tokens` | Existing `PromptTokens` |
|
||||||
|
| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` |
|
||||||
|
| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` |
|
||||||
|
| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` |
|
||||||
|
| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` |
|
||||||
|
| Total tokens | `total_tokens` | Existing `TotalTokens` |
|
||||||
|
|
||||||
|
Preserve `CachedTokens` as the public field name and clarify its meaning as
|
||||||
|
cache-read tokens. Correct the OpenRouter cache-write mapping to the nested
|
||||||
|
field above; retain the generic provider's existing top-level handling. Do not
|
||||||
|
add the nested and top-level values together. OpenRouter uses its documented
|
||||||
|
nested value; an omitted nested value remains unknown.
|
||||||
|
|
||||||
|
Add `ReasoningTokens` to the public/domain token values, presence indicators,
|
||||||
|
and observed token usage, including conversions, result serialization, attempt
|
||||||
|
and terminal events, and group summaries. Initially, OpenRouter supplies this
|
||||||
|
additional built-in response interpretation; the generic provider preserves its
|
||||||
|
existing behavior. Injected clients may supply the canonical count and its
|
||||||
|
presence through the existing model-client boundary.
|
||||||
|
|
||||||
|
### Count Relationships And Availability
|
||||||
|
|
||||||
|
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
|
||||||
|
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
|
||||||
|
Cache reads and writes describe input-token breakdowns. Present these as subsets
|
||||||
|
or details rather than additional consumption to add to input/output totals.
|
||||||
|
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
|
||||||
|
|
||||||
|
Consumers can derive non-reasoning output as completion tokens minus reasoning
|
||||||
|
tokens when both counts are available and consistent. No additional stored
|
||||||
|
non-reasoning count is needed in this scope. Capturing reasoning-token counts
|
||||||
|
does not capture reasoning text or change reasoning request settings.
|
||||||
|
|
||||||
|
Reuse existing per-field presence and aggregation rules: omitted/null means
|
||||||
|
unknown, explicit zero means known zero, and aggregate `Known` and `Complete`
|
||||||
|
flags describe the reported contributions across dispatched attempts. Include
|
||||||
|
initial generations, retries, and repairs through the existing observation
|
||||||
|
accounting, retaining terminal-only group aggregation and deduplication. Preserve
|
||||||
|
the existing distinction between integer-only result usage and presence-aware
|
||||||
|
observations; zero in `TokenUsage` alone does not establish provider reporting.
|
||||||
|
|
||||||
|
### Token Cache-Hit Ratio
|
||||||
|
|
||||||
|
Add this derived helper on the existing presence-aware usage type:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
|
||||||
|
```
|
||||||
|
|
||||||
|
The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero
|
||||||
|
to one. It measures the proportion of input tokens served from cache, not the
|
||||||
|
proportion of requests with a cache hit or monetary savings. The Boolean reports
|
||||||
|
whether a ratio is available; an unavailable result returns zero and false.
|
||||||
|
|
||||||
|
Return an available ratio only when both counts are known and complete, input
|
||||||
|
tokens are positive, and cache reads are between zero and input tokens inclusive.
|
||||||
|
Zero cache reads with known positive input yields an available zero ratio. Missing
|
||||||
|
or incomplete counts, zero input, or inconsistent counts leave it unavailable;
|
||||||
|
preserve the raw reported counts rather than clamping them or failing generation.
|
||||||
|
Cache-write availability does not affect this calculation.
|
||||||
|
|
||||||
|
For a run or group, divide the aggregated cache-read count by the aggregated
|
||||||
|
input count; do not average per-attempt or per-run percentages. Do not use
|
||||||
|
`reads / (reads + writes)`, because cache writes need not account for all
|
||||||
|
uncached input. The same helper works on attempts, terminal events, and group
|
||||||
|
summaries without storing another independently maintained field.
|
||||||
|
|
||||||
|
The ratio describes the included observations. A group ratio does not supersede
|
||||||
|
`TerminalCoverageComplete` or imply that all application work has finished.
|
||||||
|
For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
|
||||||
|
are known and complete, even if cache writes are unknown. Output 840 with
|
||||||
|
reasoning 600 still represents 840 output tokens, not 1,440.
|
||||||
|
|
||||||
## Optional Provider-Reported Cost
|
## Optional Provider-Reported Cost
|
||||||
|
|
||||||
### Initial Provider Support
|
### Initial Provider Support
|
||||||
@@ -360,10 +450,11 @@ delivery, and interpretation. This feature adds no CommandKit dependency or
|
|||||||
application-specific concepts to PromptKit.
|
application-specific concepts to PromptKit.
|
||||||
|
|
||||||
Smoke diagnostics retain their separate report/evidence model and are not
|
Smoke diagnostics retain their separate report/evidence model and are not
|
||||||
enrolled in ordinary execution observation by this change. Reasoning-token
|
enrolled in ordinary execution observation by this change. Reasoning-text
|
||||||
capture, additional built-in cost providers, multiple currencies, currency
|
capture, changes to reasoning or caching request behavior, additional built-in
|
||||||
conversion, pricing estimates, billing reconciliation, persistent history, and
|
cost providers, multiple currencies, currency conversion, pricing estimates,
|
||||||
logging infrastructure are outside this scope.
|
billing reconciliation, persistent history, and logging infrastructure are
|
||||||
|
outside this scope.
|
||||||
|
|
||||||
## Work Needed And Completion Criteria
|
## Work Needed And Completion Criteria
|
||||||
|
|
||||||
@@ -389,6 +480,15 @@ logging infrastructure are outside this scope.
|
|||||||
- Verify that provider diagnostic defaults do not become explicit values in the
|
- Verify that provider diagnostic defaults do not become explicit values in the
|
||||||
loaded configuration or replace source provenance in inspection. Cover the
|
loaded configuration or replace source provenance in inspection. Cover the
|
||||||
changed clearing behavior and endpoint-only authentication overrides.
|
changed clearing behavior and endpoint-only authentication overrides.
|
||||||
|
- Carry OpenRouter reasoning-token counts through token values, presence,
|
||||||
|
results, events, and group summaries. Verify nested OpenRouter cache-write
|
||||||
|
decoding, including omitted/null/zero values and the nested value taking
|
||||||
|
precedence when a top-level value is also present. Preserve generic-provider
|
||||||
|
token mappings and existing field meanings.
|
||||||
|
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
|
||||||
|
incomplete counts, zero input, inconsistent counts, independence from
|
||||||
|
cache-write availability, and token-weighted aggregate ratios. Verify subset
|
||||||
|
accounting without adding reasoning or cached tokens to the reported totals.
|
||||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||||
conversions, successful run results, attempt and terminal events, and existing
|
conversions, successful run results, attempt and terminal events, and existing
|
||||||
group summaries. Keep result cost available without registering an observer.
|
group summaries. Keep result cost available without registering an observer.
|
||||||
@@ -413,7 +513,9 @@ logging infrastructure are outside this scope.
|
|||||||
turn telemetry into an exact-decimal guarantee.
|
turn telemetry into an exact-decimal guarantee.
|
||||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||||
aggregation example to the consumer observation guide, including cost
|
aggregation example to the consumer observation guide, including cost
|
||||||
availability, units, and completeness. Document OpenRouter cost decoding in
|
availability, units, and completeness. Document the reasoning/cache fields and
|
||||||
|
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
|
||||||
|
without double-counting. Document OpenRouter token and cost decoding in
|
||||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||||
and update internal observation and model-client documentation for changed flows.
|
and update internal observation and model-client documentation for changed flows.
|
||||||
Document `provider`, its smoke-test defaults and override semantics in the
|
Document `provider`, its smoke-test defaults and override semantics in the
|
||||||
@@ -428,13 +530,15 @@ Organize the eventual implementation plan around two workstreams:
|
|||||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||||
and nested execution boundaries and the existing group collector.
|
and nested execution boundaries and the existing group collector.
|
||||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||||
smoke-test defaults and explicit overrides, followed by provider-reported cost
|
smoke-test defaults and explicit overrides, followed by OpenRouter token
|
||||||
through decoding, accounting, public results, events, and group summaries.
|
extensions and provider-reported cost through decoding, accounting, public
|
||||||
|
results, events, and group summaries, plus the derived cache-hit helper.
|
||||||
Include the corresponding configuration and API documentation.
|
Include the corresponding configuration and API documentation.
|
||||||
|
|
||||||
The second area is more extensive than the context helper. Keep backend provider
|
The second area is more extensive than the context helper. Keep backend provider
|
||||||
configuration, diagnostic consolidation, and cost accounting in separately
|
configuration, diagnostic consolidation, token/cache reporting, and cost
|
||||||
reviewable implementation units rather than one large implementation prompt.
|
accounting in separately reviewable implementation units rather than one large
|
||||||
|
implementation prompt.
|
||||||
Complete the remaining provider discussion before preparing that plan; avoid
|
Complete the remaining provider discussion before preparing that plan; avoid
|
||||||
coupling delivery of contextual observation to provider cost support.
|
coupling delivery of contextual observation to provider cost support.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user