Add explicit zero-cost declarations and cost-source reporting to the roadmap

This commit is contained in:
2026-11-22 17:11:57 +00:00
parent 440d9a2c88
commit 4ee64b6eda

View File

@@ -8,9 +8,11 @@ Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested, operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should concurrent, and multi-engine calls using that operation's context should
participate automatically. Include optional provider-reported cost alongside participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost support initially limited to OpenRouter. Extend token usage, with built-in cost decoding initially limited to OpenRouter and an
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads explicit backend zero-cost fallback. Distinguish reported and configured cost
and writes, and a shared derived token cache-hit ratio. sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
reporting with reasoning counts, documented cache reads and writes, and a shared
derived token cache-hit ratio.
The motivating consumer is CommandKit's optional `ckprompt` integration, which The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline. could use this facility to report token usage for an entire command or pipeline.
@@ -31,8 +33,9 @@ PromptKit v0.24.0 already provides:
execution, and [CoordinateRequest.Observer](../../coordinate.go) for execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs. coordinator-selected continuation runs.
The missing capabilities are contextual observer selection, provider-reported The missing capabilities are contextual observer selection, cost accounting and
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper. source metadata, an explicit zero-cost declaration, reasoning-token counts, and
a presence-aware cache-hit helper.
OpenRouter cache writes also need their documented nested response mapping; OpenRouter cache writes also need their documented nested response mapping;
the current client reads a top-level `usage.cache_write_tokens` field instead. the current client reads a top-level `usage.cache_write_tokens` field instead.
Applications currently have to enroll each request explicitly, and PromptKit Applications currently have to enroll each request explicitly, and PromptKit
@@ -186,11 +189,12 @@ conventions. Preserve the existing shared behavior documented in the
transient retries. transient retries.
This selection supplies no default endpoint, model, API-key environment variable, This selection supplies no default endpoint, model, API-key environment variable,
or sampling settings. It does not interpret monetary cost or provide a default or sampling settings. It does not decode provider-reported monetary cost or
authentication-probe adapter. Without an explicit authentication adapter, report provide a default authentication-probe adapter. The backend zero-cost declaration
the existing unsupported diagnostic outcome rather than assuming authentication below remains available independently of provider selection. Without an explicit
is inapplicable. Existing reachability and inference diagnostics and explicit authentication adapter, report the existing unsupported diagnostic outcome rather
authentication adapters remain available. than assuming authentication is inapplicable. Existing reachability and inference
diagnostics and explicit authentication adapters remain available.
"Lowest common denominator" identifies the shared protocol foundation; it does "Lowest common denominator" identifies the shared protocol foundation; it does
not remove existing optional features or promise every backend supports them. not remove existing optional features or promise every backend supports them.
@@ -211,9 +215,11 @@ thinking, or translate model-specific controls based on the provider label.
No separate generation protocol is needed; vLLM documents an No separate generation protocol is needed; vLLM documents an
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). [OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
The token mappings are specified below. Monetary cost remains unknown, not zero, The token mappings are specified below. vLLM supplies no provider-reported cost;
including when vLLM is locally hosted. This provider adds no price estimator, cost remains unknown unless the backend explicitly declares the zero-cost
server administration, metrics collection, or automatic capability/version probe. fallback below. Local hosting alone does not select that fallback. This provider
adds no price estimator, server administration, metrics collection, or automatic
capability/version probe.
### Smoke-Test Defaults And Explicit Overrides ### Smoke-Test Defaults And Explicit Overrides
@@ -419,7 +425,7 @@ For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
are known and complete, even if cache writes are unknown. Output 840 with are known and complete, even if cache writes are unknown. Output 840 with
reasoning 600 still represents 840 output tokens, not 1,440. reasoning 600 still represents 840 output tokens, not 1,440.
## Optional Provider-Reported Cost ## Optional Cost Reporting
### Initial Provider Support ### Initial Provider Support
@@ -433,14 +439,54 @@ top-level charge; those describe different facts and could double-count cost.
For the built-in client, select this interpretation through `provider: openrouter`, For the built-in client, select this interpretation through `provider: openrouter`,
regardless of backend identity or namespace. Do not infer USD semantics from an regardless of backend identity or namespace. Do not infer USD semantics from an
arbitrary backend's field named `cost`, or add endpoint discovery or verification. arbitrary backend's field named `cost`, or add endpoint discovery or verification.
Built-in cost interpretation is an extension unique to `openrouter` initially; Built-in provider-reported cost interpretation is an extension unique to
`openai-compatible` and `vllm` leave cost unknown even if a response contains `openrouter` initially; `openai-compatible` and `vllm` do not interpret
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains `usage.cost`. Without an applicable zero-cost declaration, their built-in cost
provider-neutral after the model-client boundary supplies interpreted USD facts. remains unknown. No separate provider cost-decoding selector is needed.
Aggregation remains provider-neutral after the model-client boundary supplies
interpreted USD facts.
Use the cost included in the normal completion response. No additional usage Use the cost included in the normal completion response. No additional usage
lookup, pricing request, or opt-in provider request parameter is needed. lookup, pricing request, or opt-in provider request parameter is needed.
### Explicit Backend Zero-Cost Declaration
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
```yaml
promptkit:
backends:
- id: local-inference
provider: vllm
endpoint: http://localhost:8000/v1
zero_cost: true
```
For each dispatched attempt, use a usable provider/model-client-reported cost
first, including an explicitly reported zero. If no usable reported cost is
available and the effective backend has `zero_cost: true`, supply USD zero with
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
it does not suppress a reported cost. Apply this policy to built-in and injected
client execution using the selected backend, regardless of its provider type.
The fallback also applies to dispatched attempts that fail, are canceled, or
produce no decodable response. It is an operator declaration, not a provider
observation. Non-dispatched work contributes no cost. Do not infer this setting
from a local address, provider name, token counts, or missing credentials, and
do not install it automatically on any backend. Without a selected backend
declaration, no configured-zero fallback applies.
Resolve the field through ordinary final ConfigKit composition and inheritance;
include it in typed backend helpers, configuration capture, and backend
inspection. Retain the effective policy through prepared execution without
reloading configuration. It is local accounting metadata, not an outbound
request parameter. Preserve explicit/null/omitted source provenance in inspection
rather than manufacturing an explicit false value for the disabled fallback.
This supports an explicit zero declaration, not configurable input/output or
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
cost estimation. It does not alter credentials, generation, or token reporting.
### Monetary Representation ### Monetary Representation
Use `float64` amounts in USD. Make the unit explicit in public field names such Use `float64` amounts in USD. Make the unit explicit in public field names such
@@ -453,9 +499,10 @@ OpenRouter backend; no provider-specific interface is needed.
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
remain token counts. For run results, execution events, and group summaries, remain token counts. For run results, execution events, and group summaries,
use a small presence-aware cost value with `Value float64`, `Known bool`, and use a small presence-aware cost value with `Value float64`, `Known bool`, and
`Complete bool`, following the existing observed-count conventions. Successful `Complete bool`, following the existing observed-count conventions, plus the
run results should expose cost even when no observer was registered; observers source metadata below. Successful run results should expose cost even when no
also provide per-attempt and terminal cost, including available facts on failure. observer was registered; observers also provide per-attempt and terminal cost,
including available facts on failure.
Binary floating point is sufficient for operational telemetry and keeps decoding, Binary floating point is sufficient for operational telemetry and keeps decoding,
addition, and the public API lightweight. Do not round to cents or another fixed addition, and the public API lightweight. Do not round to cents or another fixed
@@ -465,42 +512,75 @@ exact financial ledgers, while fixed-scale integers would require a selected
precision and rounding policy. Neither is warranted for this reporting feature. precision and rounding policy. Neither is warranted for this reporting feature.
The API does not promise exact decimal arithmetic or invoice reconciliation. The API does not promise exact decimal arithmetic or invoice reconciliation.
### Cost-Source Metadata
Give the presence-aware cost value a `Source` discriminator with these meanings:
- `provider`: a cost supplied by the provider or injected model-client response.
This records the reporting path, not independent verification of a charge.
- `configured_zero`: zero supplied by the explicit backend fallback.
- `mixed`: an aggregate includes both provider and configured-zero contributions.
- Empty: no known contribution has a source. This covers unknown cost and the
known, complete zero aggregate when no attempts were dispatched.
Derive aggregate source from the union of known contribution sources, including
zero-valued contributions. Unknown contributions supply no source but affect
completeness. A provider-only subtotal can therefore be incomplete without being
`mixed`. Carry this metadata consistently through attempt events, terminal
events, successful results, and group summaries. Reuse existing aggregation and
terminal deduplication rather than adding a parallel collector or cost history.
Consumers can distinguish reported charges, configured zeros, and mixed totals
without treating configured values as provider reports. Source metadata is
separate from configuration inspection and does not require provenance traversal
during execution.
### Presence, Aggregation, And Failures ### Presence, Aggregation, And Failures
- An absent or null cost is unknown. Explicit numeric zero is known zero. - An absent or null provider cost is unknown unless the backend's explicit
A backend that reports no cost remains unknown even when a consumer knows it zero-cost fallback supplies it. A reported numeric zero remains known zero
has no external charge. with a provider source, even when the fallback is enabled.
- On a dispatched attempt, a usable reported cost is known and complete for - On a dispatched attempt, a usable reported cost or configured-zero fallback
that attempt. Cost completeness is independent of token-field completeness. is known and complete for that attempt. Cost completeness is independent of
token-field completeness and does not assert provider reporting.
- Sum known costs across dispatched attempts, including initial generations, - Sum known costs across dispatched attempts, including initial generations,
retries, and repairs. `Known` means at least one attempt reported cost; retries, and repairs. `Known` means at least one attempt has an available cost
`Complete` means every dispatched attempt did. Unknown values contribute no from either source; `Complete` means every dispatched attempt does under this
amount, but make the total incomplete. With no dispatched attempts, follow policy. Unknown values contribute no amount, but make the total incomplete.
the existing known, complete zero convention. With no dispatched attempts, follow the existing known, complete zero
convention, with an empty source.
- Group summaries reuse existing terminal-only aggregation and deduplication: - Group summaries reuse existing terminal-only aggregation and deduplication:
sum known terminal values, OR their `Known` flags, and AND their `Complete` sum known terminal values, OR their `Known` flags, and AND their `Complete`
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
from OpenRouter mixed with unreported local-backend cost is incomplete. from OpenRouter mixed with unreported local-backend cost is incomplete unless
those local attempts have an explicit configured-zero fallback. Combine
cost-source metadata as described above without changing token accounting.
- Keep valid cost observed from a decoded successful envelope when a later - Keep valid cost observed from a decoded successful envelope when a later
choice/content check fails, as with token usage. Invalid framing and non-2xx choice/content check fails, as with token usage. Invalid framing and non-2xx
error bodies continue to supply no usage. Cancellation or failure may leave error bodies continue to supply no provider usage, but the configured-zero
incomplete reporting even when billable upstream work occurred. fallback still applies to dispatched attempts. Without that fallback,
cancellation or failure may leave incomplete cost reporting even when billable
upstream work occurred.
- Treat an unusable optional cost value, including a nonnumeric value or one - Treat an unusable optional cost value, including a nonnumeric value or one
not representable as a finite `float64`, as unknown rather than making an not representable as a finite `float64`, as unknown rather than making an
otherwise valid generation fail. This does not relax existing response framing otherwise valid generation fail; the zero-cost fallback can then apply. This
or content validation. An injected client likewise cannot supply a nonfinite does not relax existing response framing or content validation. An injected
usable cost. client likewise cannot supply a nonfinite usable cost.
Completeness describes coverage of reported values, not verification of the Completeness describes coverage by available reported or explicitly configured
provider's billing. Preserve reported usable amounts without pricing inference cost, not verification of the provider's billing or proof that every provider
or reconciliation. reported a charge. Preserve usable reported amounts without pricing inference or
reconciliation.
For a successful run, result cost and terminal-event cost describe the same For a successful run, result cost and terminal-event cost describe the same
dispatched attempts and must agree on value, presence, and completeness. Result dispatched attempts and must agree on value, presence, completeness, and source.
cost must also be the same with observation enabled or disabled. For example, Result cost must also be the same with observation enabled or disabled. For
an unreported failed attempt followed by a successful attempt reporting USD example, an unreported failed attempt followed by a successful attempt reporting USD
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same 0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
accounting through repairs and ordinary or prepared execution. source when no zero-cost fallback applies. Apply the same accounting through
repairs and ordinary or prepared execution. An invocation with fully reported
OpenRouter charges and configured-zero local calls can have a complete subtotal
with a mixed source, without claiming all providers reported costs.
Group cost completeness concerns the terminal observations included in its Group cost completeness concerns the terminal observations included in its
subtotal. If two completed runs reported all their costs but a third observed subtotal. If two completed runs reported all their costs but a third observed
@@ -523,7 +603,8 @@ application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Reasoning-text enrolled in ordinary execution observation by this change. Reasoning-text
capture, changes to reasoning or caching request behavior, additional built-in capture, changes to reasoning or caching request behavior, additional built-in
cost providers, multiple currencies, currency conversion, pricing estimates, cost providers, configurable token-rate estimation, fixed nonzero request
pricing, multiple currencies, currency conversion, pricing estimates,
billing reconciliation, persistent history, and logging infrastructure are billing reconciliation, persistent history, and logging infrastructure are
outside this scope. outside this scope.
@@ -537,6 +618,11 @@ outside this scope.
- Add `provider` to the backend configuration model, typed source helpers, - Add `provider` to the backend configuration model, typed source helpers,
capture, inspection, and execution plumbing. Update the bundled OpenRouter capture, inspection, and execution plumbing. Update the bundled OpenRouter
backend to select it explicitly. Retain the selection through prepared runs. backend to select it explicitly. Retain the selection through prepared runs.
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
prepared-execution plumbing. Apply the fallback outside provider-specific
decoding so it works with built-in and injected clients. Verify composition
and inheritance, omitted/null/false behavior, explicit overrides, provider
independence, retained inspection provenance, and absence from provider bodies.
- Consolidate provider-specific response and authentication-probe behavior in - Consolidate provider-specific response and authentication-probe behavior in
`internal/llm`; keep coordination and accounting provider-neutral. Cover `internal/llm`; keep coordination and accounting provider-neutral. Cover
provider-default probes, explicit kind and endpoint overrides, `kind: none`, provider-default probes, explicit kind and endpoint overrides, `kind: none`,
@@ -561,7 +647,8 @@ outside this scope.
recognized acceptance, rejection, unrecognized responses, provider defaults, recognized acceptance, rejection, unrecognized responses, provider defaults,
and explicit kind/endpoint overrides, including `none` and adapter selection and explicit kind/endpoint overrides, including `none` and adapter selection
without a provider declaration. Verify that ordinary generation performs no without a provider declaration. Verify that ordinary generation performs no
diagnostic requests and leaves monetary cost unknown. diagnostic requests and leaves monetary cost unknown unless the backend
explicitly declares the zero-cost fallback.
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache - Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
details, including responses representative of servers without detail details, including responses representative of servers without detail
reporting enabled. Verify `created_cache_tokens` mapping independently of reporting enabled. Verify `created_cache_tokens` mapping independently of
@@ -575,6 +662,16 @@ outside this scope.
conversions, successful run results, attempt and terminal events, and existing conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer. group summaries. Keep result cost available without registering an observer.
Preserve existing token fields and accounting. Preserve existing token fields and accounting.
- Carry cost-source metadata through results, events, conversions, and group
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
including zero-valued reported contributions and empty/non-dispatched cases.
Verify source union and terminal deduplication independently of cost value.
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
null, or unusable reported cost permits fallback. Cover dispatched failures,
cancellation, undecodable responses, retries, repairs, ordinary/prepared
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
and declared zero-cost local calls can produce complete cost totals while
token completeness remains independent.
- Verify that successful-result cost agrees with terminal-event cost and is - Verify that successful-result cost agrees with terminal-event cost and is
unchanged when observation is disabled, including the unreported-retry example unchanged when observation is disabled, including the unreported-retry example
above. Verify group snapshots with fully reported completed runs alongside a above. Verify group snapshots with fully reported completed runs alongside a
@@ -595,15 +692,18 @@ outside this scope.
turn telemetry into an exact-decimal guarantee. turn telemetry into an exact-decimal guarantee.
- Document the exact helper contract in GoDoc and add a focused invocation-level - Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide, including cost aggregation example to the consumer observation guide, including cost
availability, units, and completeness. Document the reasoning/cache fields and availability, units, completeness, and source. Explain that configured zeros
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns can make cost complete without provider reporting. Document the reasoning/cache
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter fields and cache-hit helper in their owning GoDoc and illustrate input/output
cost decoding in the breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
OpenRouter cost decoding in the
[integration contract](../integrations/openai-compatible-chat.md#response-handling), [integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows. and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the Document `provider`, its smoke-test defaults and override semantics in the
framework format reference and relevant public GoDoc; update inspection and framework format reference and relevant public GoDoc; update inspection and
diagnostic documentation with the implemented behavior. Include the diagnostic diagnostic documentation with the implemented behavior. Document `zero_cost`
and its fallback precedence, inspection behavior, and prepared lifetime in the
relevant format, GoDoc, and consumer owners. Include the diagnostic
migration in release notes. Include vLLM usage availability, server reporting migration in release notes. Include vLLM usage availability, server reporting
settings, the meaning of cache creation, and the limits of its authentication settings, the meaning of cache creation, and the limits of its authentication
diagnostic in the relevant consumer and integration documentation. diagnostic in the relevant consumer and integration documentation.
@@ -616,12 +716,13 @@ Organize the eventual implementation plan around two workstreams:
and nested execution boundaries and the existing group collector. and nested execution boundaries and the existing group collector.
2. Backend provider selection and consolidation within `internal/llm`, including 2. Backend provider selection and consolidation within `internal/llm`, including
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
extensions and provider-reported cost through decoding, accounting, public extensions, provider-reported cost, and configured-zero fallback through
results, events, and group summaries, plus the derived cache-hit helper. decoding, accounting, public results, events, and group summaries, plus the
Include the corresponding configuration and API documentation. derived cache-hit helper. Include the corresponding configuration and API
documentation.
The second area is more extensive than the context helper. Keep backend provider The second area is more extensive than the context helper. Keep backend provider
configuration, diagnostic consolidation, token/cache reporting, and cost configuration, diagnostic consolidation, token/cache reporting, and cost/source
accounting in separately reviewable implementation units rather than one large accounting in separately reviewable implementation units rather than one large
implementation prompt. implementation prompt.
Avoid coupling delivery of contextual observation to provider cost support. Avoid coupling delivery of contextual observation to provider cost support.
@@ -637,8 +738,9 @@ contract.
Keep tolerant optional-cost decoding local to the model-client boundary. Adding Keep tolerant optional-cost decoding local to the model-client boundary. Adding
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
overflowing value fail decoding of the entire response. Decode the optional overflowing value fail decoding of the entire response. Decode the optional
field separately so an unusable cost becomes unknown without changing existing field separately so an unusable provider cost becomes unknown before applying
framing or content validation. This needs no general numeric-conversion layer. any configured-zero fallback, without changing existing framing or content
validation. This needs no general numeric-conversion layer.
Use the completion criteria above to select focused offline checks for each Use the completion criteria above to select focused offline checks for each
unit. This roadmap owns target behavior; a separate implementation plan owns unit. This roadmap owns target behavior; a separate implementation plan owns