Add explicit zero-cost declarations and cost-source reporting to the roadmap

This commit is contained in:
2026-11-22 17:11:57 +00:00
parent 440d9a2c88
commit 4ee64b6eda

View File

@@ -8,9 +8,11 @@ Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should
participate automatically. Include optional provider-reported cost alongside
token usage, with built-in cost support initially limited to OpenRouter. Extend
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads
and writes, and a shared derived token cache-hit ratio.
token usage, with built-in cost decoding initially limited to OpenRouter and an
explicit backend zero-cost fallback. Distinguish reported and configured cost
sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
reporting with reasoning counts, documented cache reads and writes, and a shared
derived token cache-hit ratio.
The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline.
@@ -31,8 +33,9 @@ PromptKit v0.24.0 already provides:
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs.
The missing capabilities are contextual observer selection, provider-reported
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
The missing capabilities are contextual observer selection, cost accounting and
source metadata, an explicit zero-cost declaration, reasoning-token counts, and
a presence-aware cache-hit helper.
OpenRouter cache writes also need their documented nested response mapping;
the current client reads a top-level `usage.cache_write_tokens` field instead.
Applications currently have to enroll each request explicitly, and PromptKit
@@ -186,11 +189,12 @@ conventions. Preserve the existing shared behavior documented in the
transient retries.
This selection supplies no default endpoint, model, API-key environment variable,
or sampling settings. It does not interpret monetary cost or provide a default
authentication-probe adapter. Without an explicit authentication adapter, report
the existing unsupported diagnostic outcome rather than assuming authentication
is inapplicable. Existing reachability and inference diagnostics and explicit
authentication adapters remain available.
or sampling settings. It does not decode provider-reported monetary cost or
provide a default authentication-probe adapter. The backend zero-cost declaration
below remains available independently of provider selection. Without an explicit
authentication adapter, report the existing unsupported diagnostic outcome rather
than assuming authentication is inapplicable. Existing reachability and inference
diagnostics and explicit authentication adapters remain available.
"Lowest common denominator" identifies the shared protocol foundation; it does
not remove existing optional features or promise every backend supports them.
@@ -211,9 +215,11 @@ thinking, or translate model-specific controls based on the provider label.
No separate generation protocol is needed; vLLM documents an
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
The token mappings are specified below. Monetary cost remains unknown, not zero,
including when vLLM is locally hosted. This provider adds no price estimator,
server administration, metrics collection, or automatic capability/version probe.
The token mappings are specified below. vLLM supplies no provider-reported cost;
cost remains unknown unless the backend explicitly declares the zero-cost
fallback below. Local hosting alone does not select that fallback. This provider
adds no price estimator, server administration, metrics collection, or automatic
capability/version probe.
### Smoke-Test Defaults And Explicit Overrides
@@ -419,7 +425,7 @@ For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
are known and complete, even if cache writes are unknown. Output 840 with
reasoning 600 still represents 840 output tokens, not 1,440.
## Optional Provider-Reported Cost
## Optional Cost Reporting
### Initial Provider Support
@@ -433,14 +439,54 @@ top-level charge; those describe different facts and could double-count cost.
For the built-in client, select this interpretation through `provider: openrouter`,
regardless of backend identity or namespace. Do not infer USD semantics from an
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
Built-in cost interpretation is an extension unique to `openrouter` initially;
`openai-compatible` and `vllm` leave cost unknown even if a response contains
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains
provider-neutral after the model-client boundary supplies interpreted USD facts.
Built-in provider-reported cost interpretation is an extension unique to
`openrouter` initially; `openai-compatible` and `vllm` do not interpret
`usage.cost`. Without an applicable zero-cost declaration, their built-in cost
remains unknown. No separate provider cost-decoding selector is needed.
Aggregation remains provider-neutral after the model-client boundary supplies
interpreted USD facts.
Use the cost included in the normal completion response. No additional usage
lookup, pricing request, or opt-in provider request parameter is needed.
### Explicit Backend Zero-Cost Declaration
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
```yaml
promptkit:
backends:
- id: local-inference
provider: vllm
endpoint: http://localhost:8000/v1
zero_cost: true
```
For each dispatched attempt, use a usable provider/model-client-reported cost
first, including an explicitly reported zero. If no usable reported cost is
available and the effective backend has `zero_cost: true`, supply USD zero with
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
it does not suppress a reported cost. Apply this policy to built-in and injected
client execution using the selected backend, regardless of its provider type.
The fallback also applies to dispatched attempts that fail, are canceled, or
produce no decodable response. It is an operator declaration, not a provider
observation. Non-dispatched work contributes no cost. Do not infer this setting
from a local address, provider name, token counts, or missing credentials, and
do not install it automatically on any backend. Without a selected backend
declaration, no configured-zero fallback applies.
Resolve the field through ordinary final ConfigKit composition and inheritance;
include it in typed backend helpers, configuration capture, and backend
inspection. Retain the effective policy through prepared execution without
reloading configuration. It is local accounting metadata, not an outbound
request parameter. Preserve explicit/null/omitted source provenance in inspection
rather than manufacturing an explicit false value for the disabled fallback.
This supports an explicit zero declaration, not configurable input/output or
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
cost estimation. It does not alter credentials, generation, or token reporting.
### Monetary Representation
Use `float64` amounts in USD. Make the unit explicit in public field names such
@@ -453,9 +499,10 @@ OpenRouter backend; no provider-specific interface is needed.
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
remain token counts. For run results, execution events, and group summaries,
use a small presence-aware cost value with `Value float64`, `Known bool`, and
`Complete bool`, following the existing observed-count conventions. Successful
run results should expose cost even when no observer was registered; observers
also provide per-attempt and terminal cost, including available facts on failure.
`Complete bool`, following the existing observed-count conventions, plus the
source metadata below. Successful run results should expose cost even when no
observer was registered; observers also provide per-attempt and terminal cost,
including available facts on failure.
Binary floating point is sufficient for operational telemetry and keeps decoding,
addition, and the public API lightweight. Do not round to cents or another fixed
@@ -465,42 +512,75 @@ exact financial ledgers, while fixed-scale integers would require a selected
precision and rounding policy. Neither is warranted for this reporting feature.
The API does not promise exact decimal arithmetic or invoice reconciliation.
### Cost-Source Metadata
Give the presence-aware cost value a `Source` discriminator with these meanings:
- `provider`: a cost supplied by the provider or injected model-client response.
This records the reporting path, not independent verification of a charge.
- `configured_zero`: zero supplied by the explicit backend fallback.
- `mixed`: an aggregate includes both provider and configured-zero contributions.
- Empty: no known contribution has a source. This covers unknown cost and the
known, complete zero aggregate when no attempts were dispatched.
Derive aggregate source from the union of known contribution sources, including
zero-valued contributions. Unknown contributions supply no source but affect
completeness. A provider-only subtotal can therefore be incomplete without being
`mixed`. Carry this metadata consistently through attempt events, terminal
events, successful results, and group summaries. Reuse existing aggregation and
terminal deduplication rather than adding a parallel collector or cost history.
Consumers can distinguish reported charges, configured zeros, and mixed totals
without treating configured values as provider reports. Source metadata is
separate from configuration inspection and does not require provenance traversal
during execution.
### Presence, Aggregation, And Failures
- An absent or null cost is unknown. Explicit numeric zero is known zero.
A backend that reports no cost remains unknown even when a consumer knows it
has no external charge.
- On a dispatched attempt, a usable reported cost is known and complete for
that attempt. Cost completeness is independent of token-field completeness.
- An absent or null provider cost is unknown unless the backend's explicit
zero-cost fallback supplies it. A reported numeric zero remains known zero
with a provider source, even when the fallback is enabled.
- On a dispatched attempt, a usable reported cost or configured-zero fallback
is known and complete for that attempt. Cost completeness is independent of
token-field completeness and does not assert provider reporting.
- Sum known costs across dispatched attempts, including initial generations,
retries, and repairs. `Known` means at least one attempt reported cost;
`Complete` means every dispatched attempt did. Unknown values contribute no
amount, but make the total incomplete. With no dispatched attempts, follow
the existing known, complete zero convention.
retries, and repairs. `Known` means at least one attempt has an available cost
from either source; `Complete` means every dispatched attempt does under this
policy. Unknown values contribute no amount, but make the total incomplete.
With no dispatched attempts, follow the existing known, complete zero
convention, with an empty source.
- Group summaries reuse existing terminal-only aggregation and deduplication:
sum known terminal values, OR their `Known` flags, and AND their `Complete`
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
from OpenRouter mixed with unreported local-backend cost is incomplete.
from OpenRouter mixed with unreported local-backend cost is incomplete unless
those local attempts have an explicit configured-zero fallback. Combine
cost-source metadata as described above without changing token accounting.
- Keep valid cost observed from a decoded successful envelope when a later
choice/content check fails, as with token usage. Invalid framing and non-2xx
error bodies continue to supply no usage. Cancellation or failure may leave
incomplete reporting even when billable upstream work occurred.
error bodies continue to supply no provider usage, but the configured-zero
fallback still applies to dispatched attempts. Without that fallback,
cancellation or failure may leave incomplete cost reporting even when billable
upstream work occurred.
- Treat an unusable optional cost value, including a nonnumeric value or one
not representable as a finite `float64`, as unknown rather than making an
otherwise valid generation fail. This does not relax existing response framing
or content validation. An injected client likewise cannot supply a nonfinite
usable cost.
otherwise valid generation fail; the zero-cost fallback can then apply. This
does not relax existing response framing or content validation. An injected
client likewise cannot supply a nonfinite usable cost.
Completeness describes coverage of reported values, not verification of the
provider's billing. Preserve reported usable amounts without pricing inference
or reconciliation.
Completeness describes coverage by available reported or explicitly configured
cost, not verification of the provider's billing or proof that every provider
reported a charge. Preserve usable reported amounts without pricing inference or
reconciliation.
For a successful run, result cost and terminal-event cost describe the same
dispatched attempts and must agree on value, presence, and completeness. Result
cost must also be the same with observation enabled or disabled. For example,
an unreported failed attempt followed by a successful attempt reporting USD
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
accounting through repairs and ordinary or prepared execution.
dispatched attempts and must agree on value, presence, completeness, and source.
Result cost must also be the same with observation enabled or disabled. For
example, an unreported failed attempt followed by a successful attempt reporting USD
0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
source when no zero-cost fallback applies. Apply the same accounting through
repairs and ordinary or prepared execution. An invocation with fully reported
OpenRouter charges and configured-zero local calls can have a complete subtotal
with a mixed source, without claiming all providers reported costs.
Group cost completeness concerns the terminal observations included in its
subtotal. If two completed runs reported all their costs but a third observed
@@ -523,7 +603,8 @@ application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Reasoning-text
capture, changes to reasoning or caching request behavior, additional built-in
cost providers, multiple currencies, currency conversion, pricing estimates,
cost providers, configurable token-rate estimation, fixed nonzero request
pricing, multiple currencies, currency conversion, pricing estimates,
billing reconciliation, persistent history, and logging infrastructure are
outside this scope.
@@ -537,6 +618,11 @@ outside this scope.
- Add `provider` to the backend configuration model, typed source helpers,
capture, inspection, and execution plumbing. Update the bundled OpenRouter
backend to select it explicitly. Retain the selection through prepared runs.
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
prepared-execution plumbing. Apply the fallback outside provider-specific
decoding so it works with built-in and injected clients. Verify composition
and inheritance, omitted/null/false behavior, explicit overrides, provider
independence, retained inspection provenance, and absence from provider bodies.
- Consolidate provider-specific response and authentication-probe behavior in
`internal/llm`; keep coordination and accounting provider-neutral. Cover
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
@@ -561,7 +647,8 @@ outside this scope.
recognized acceptance, rejection, unrecognized responses, provider defaults,
and explicit kind/endpoint overrides, including `none` and adapter selection
without a provider declaration. Verify that ordinary generation performs no
diagnostic requests and leaves monetary cost unknown.
diagnostic requests and leaves monetary cost unknown unless the backend
explicitly declares the zero-cost fallback.
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
details, including responses representative of servers without detail
reporting enabled. Verify `created_cache_tokens` mapping independently of
@@ -575,6 +662,16 @@ outside this scope.
conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer.
Preserve existing token fields and accounting.
- Carry cost-source metadata through results, events, conversions, and group
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
including zero-valued reported contributions and empty/non-dispatched cases.
Verify source union and terminal deduplication independently of cost value.
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
null, or unusable reported cost permits fallback. Cover dispatched failures,
cancellation, undecodable responses, retries, repairs, ordinary/prepared
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
and declared zero-cost local calls can produce complete cost totals while
token completeness remains independent.
- Verify that successful-result cost agrees with terminal-event cost and is
unchanged when observation is disabled, including the unreported-retry example
above. Verify group snapshots with fully reported completed runs alongside a
@@ -595,15 +692,18 @@ outside this scope.
turn telemetry into an exact-decimal guarantee.
- Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide, including cost
availability, units, and completeness. Document the reasoning/cache fields and
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter
cost decoding in the
availability, units, completeness, and source. Explain that configured zeros
can make cost complete without provider reporting. Document the reasoning/cache
fields and cache-hit helper in their owning GoDoc and illustrate input/output
breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
OpenRouter cost decoding in the
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows.
Document `provider`, its smoke-test defaults and override semantics in the
framework format reference and relevant public GoDoc; update inspection and
diagnostic documentation with the implemented behavior. Include the diagnostic
diagnostic documentation with the implemented behavior. Document `zero_cost`
and its fallback precedence, inspection behavior, and prepared lifetime in the
relevant format, GoDoc, and consumer owners. Include the diagnostic
migration in release notes. Include vLLM usage availability, server reporting
settings, the meaning of cache creation, and the limits of its authentication
diagnostic in the relevant consumer and integration documentation.
@@ -616,12 +716,13 @@ Organize the eventual implementation plan around two workstreams:
and nested execution boundaries and the existing group collector.
2. Backend provider selection and consolidation within `internal/llm`, including
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
extensions and provider-reported cost through decoding, accounting, public
results, events, and group summaries, plus the derived cache-hit helper.
Include the corresponding configuration and API documentation.
extensions, provider-reported cost, and configured-zero fallback through
decoding, accounting, public results, events, and group summaries, plus the
derived cache-hit helper. Include the corresponding configuration and API
documentation.
The second area is more extensive than the context helper. Keep backend provider
configuration, diagnostic consolidation, token/cache reporting, and cost
configuration, diagnostic consolidation, token/cache reporting, and cost/source
accounting in separately reviewable implementation units rather than one large
implementation prompt.
Avoid coupling delivery of contextual observation to provider cost support.
@@ -637,8 +738,9 @@ contract.
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
overflowing value fail decoding of the entire response. Decode the optional
field separately so an unusable cost becomes unknown without changing existing
framing or content validation. This needs no general numeric-conversion layer.
field separately so an unusable provider cost becomes unknown before applying
any configured-zero fallback, without changing existing framing or content
validation. This needs no general numeric-conversion layer.
Use the completion criteria above to select focused offline checks for each
unit. This roadmap owns target behavior; a separate implementation plan owns