Add explicit zero-cost declarations and cost-source reporting to the roadmap
This commit is contained in:
@@ -8,9 +8,11 @@ Allow consumers to select execution observation once for an application-owned
|
||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||
concurrent, and multi-engine calls using that operation's context should
|
||||
participate automatically. Include optional provider-reported cost alongside
|
||||
token usage, with built-in cost support initially limited to OpenRouter. Extend
|
||||
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads
|
||||
and writes, and a shared derived token cache-hit ratio.
|
||||
token usage, with built-in cost decoding initially limited to OpenRouter and an
|
||||
explicit backend zero-cost fallback. Distinguish reported and configured cost
|
||||
sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
|
||||
reporting with reasoning counts, documented cache reads and writes, and a shared
|
||||
derived token cache-hit ratio.
|
||||
|
||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||
could use this facility to report token usage for an entire command or pipeline.
|
||||
@@ -31,8 +33,9 @@ PromptKit v0.24.0 already provides:
|
||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||
coordinator-selected continuation runs.
|
||||
|
||||
The missing capabilities are contextual observer selection, provider-reported
|
||||
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
|
||||
The missing capabilities are contextual observer selection, cost accounting and
|
||||
source metadata, an explicit zero-cost declaration, reasoning-token counts, and
|
||||
a presence-aware cache-hit helper.
|
||||
OpenRouter cache writes also need their documented nested response mapping;
|
||||
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
||||
Applications currently have to enroll each request explicitly, and PromptKit
|
||||
@@ -186,11 +189,12 @@ conventions. Preserve the existing shared behavior documented in the
|
||||
transient retries.
|
||||
|
||||
This selection supplies no default endpoint, model, API-key environment variable,
|
||||
or sampling settings. It does not interpret monetary cost or provide a default
|
||||
authentication-probe adapter. Without an explicit authentication adapter, report
|
||||
the existing unsupported diagnostic outcome rather than assuming authentication
|
||||
is inapplicable. Existing reachability and inference diagnostics and explicit
|
||||
authentication adapters remain available.
|
||||
or sampling settings. It does not decode provider-reported monetary cost or
|
||||
provide a default authentication-probe adapter. The backend zero-cost declaration
|
||||
below remains available independently of provider selection. Without an explicit
|
||||
authentication adapter, report the existing unsupported diagnostic outcome rather
|
||||
than assuming authentication is inapplicable. Existing reachability and inference
|
||||
diagnostics and explicit authentication adapters remain available.
|
||||
|
||||
"Lowest common denominator" identifies the shared protocol foundation; it does
|
||||
not remove existing optional features or promise every backend supports them.
|
||||
@@ -211,9 +215,11 @@ thinking, or translate model-specific controls based on the provider label.
|
||||
No separate generation protocol is needed; vLLM documents an
|
||||
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
||||
|
||||
The token mappings are specified below. Monetary cost remains unknown, not zero,
|
||||
including when vLLM is locally hosted. This provider adds no price estimator,
|
||||
server administration, metrics collection, or automatic capability/version probe.
|
||||
The token mappings are specified below. vLLM supplies no provider-reported cost;
|
||||
cost remains unknown unless the backend explicitly declares the zero-cost
|
||||
fallback below. Local hosting alone does not select that fallback. This provider
|
||||
adds no price estimator, server administration, metrics collection, or automatic
|
||||
capability/version probe.
|
||||
|
||||
### Smoke-Test Defaults And Explicit Overrides
|
||||
|
||||
@@ -419,7 +425,7 @@ For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
|
||||
are known and complete, even if cache writes are unknown. Output 840 with
|
||||
reasoning 600 still represents 840 output tokens, not 1,440.
|
||||
|
||||
## Optional Provider-Reported Cost
|
||||
## Optional Cost Reporting
|
||||
|
||||
### Initial Provider Support
|
||||
|
||||
@@ -433,14 +439,54 @@ top-level charge; those describe different facts and could double-count cost.
|
||||
For the built-in client, select this interpretation through `provider: openrouter`,
|
||||
regardless of backend identity or namespace. Do not infer USD semantics from an
|
||||
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
||||
Built-in cost interpretation is an extension unique to `openrouter` initially;
|
||||
`openai-compatible` and `vllm` leave cost unknown even if a response contains
|
||||
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains
|
||||
provider-neutral after the model-client boundary supplies interpreted USD facts.
|
||||
Built-in provider-reported cost interpretation is an extension unique to
|
||||
`openrouter` initially; `openai-compatible` and `vllm` do not interpret
|
||||
`usage.cost`. Without an applicable zero-cost declaration, their built-in cost
|
||||
remains unknown. No separate provider cost-decoding selector is needed.
|
||||
Aggregation remains provider-neutral after the model-client boundary supplies
|
||||
interpreted USD facts.
|
||||
|
||||
Use the cost included in the normal completion response. No additional usage
|
||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||
|
||||
### Explicit Backend Zero-Cost Declaration
|
||||
|
||||
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
backends:
|
||||
- id: local-inference
|
||||
provider: vllm
|
||||
endpoint: http://localhost:8000/v1
|
||||
zero_cost: true
|
||||
```
|
||||
|
||||
For each dispatched attempt, use a usable provider/model-client-reported cost
|
||||
first, including an explicitly reported zero. If no usable reported cost is
|
||||
available and the effective backend has `zero_cost: true`, supply USD zero with
|
||||
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
|
||||
it does not suppress a reported cost. Apply this policy to built-in and injected
|
||||
client execution using the selected backend, regardless of its provider type.
|
||||
|
||||
The fallback also applies to dispatched attempts that fail, are canceled, or
|
||||
produce no decodable response. It is an operator declaration, not a provider
|
||||
observation. Non-dispatched work contributes no cost. Do not infer this setting
|
||||
from a local address, provider name, token counts, or missing credentials, and
|
||||
do not install it automatically on any backend. Without a selected backend
|
||||
declaration, no configured-zero fallback applies.
|
||||
|
||||
Resolve the field through ordinary final ConfigKit composition and inheritance;
|
||||
include it in typed backend helpers, configuration capture, and backend
|
||||
inspection. Retain the effective policy through prepared execution without
|
||||
reloading configuration. It is local accounting metadata, not an outbound
|
||||
request parameter. Preserve explicit/null/omitted source provenance in inspection
|
||||
rather than manufacturing an explicit false value for the disabled fallback.
|
||||
|
||||
This supports an explicit zero declaration, not configurable input/output or
|
||||
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
|
||||
cost estimation. It does not alter credentials, generation, or token reporting.
|
||||
|
||||
### Monetary Representation
|
||||
|
||||
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
||||
@@ -453,9 +499,10 @@ OpenRouter backend; no provider-specific interface is needed.
|
||||
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
||||
remain token counts. For run results, execution events, and group summaries,
|
||||
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
||||
`Complete bool`, following the existing observed-count conventions. Successful
|
||||
run results should expose cost even when no observer was registered; observers
|
||||
also provide per-attempt and terminal cost, including available facts on failure.
|
||||
`Complete bool`, following the existing observed-count conventions, plus the
|
||||
source metadata below. Successful run results should expose cost even when no
|
||||
observer was registered; observers also provide per-attempt and terminal cost,
|
||||
including available facts on failure.
|
||||
|
||||
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
||||
addition, and the public API lightweight. Do not round to cents or another fixed
|
||||
@@ -465,42 +512,75 @@ exact financial ledgers, while fixed-scale integers would require a selected
|
||||
precision and rounding policy. Neither is warranted for this reporting feature.
|
||||
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
||||
|
||||
### Cost-Source Metadata
|
||||
|
||||
Give the presence-aware cost value a `Source` discriminator with these meanings:
|
||||
|
||||
- `provider`: a cost supplied by the provider or injected model-client response.
|
||||
This records the reporting path, not independent verification of a charge.
|
||||
- `configured_zero`: zero supplied by the explicit backend fallback.
|
||||
- `mixed`: an aggregate includes both provider and configured-zero contributions.
|
||||
- Empty: no known contribution has a source. This covers unknown cost and the
|
||||
known, complete zero aggregate when no attempts were dispatched.
|
||||
|
||||
Derive aggregate source from the union of known contribution sources, including
|
||||
zero-valued contributions. Unknown contributions supply no source but affect
|
||||
completeness. A provider-only subtotal can therefore be incomplete without being
|
||||
`mixed`. Carry this metadata consistently through attempt events, terminal
|
||||
events, successful results, and group summaries. Reuse existing aggregation and
|
||||
terminal deduplication rather than adding a parallel collector or cost history.
|
||||
|
||||
Consumers can distinguish reported charges, configured zeros, and mixed totals
|
||||
without treating configured values as provider reports. Source metadata is
|
||||
separate from configuration inspection and does not require provenance traversal
|
||||
during execution.
|
||||
|
||||
### Presence, Aggregation, And Failures
|
||||
|
||||
- An absent or null cost is unknown. Explicit numeric zero is known zero.
|
||||
A backend that reports no cost remains unknown even when a consumer knows it
|
||||
has no external charge.
|
||||
- On a dispatched attempt, a usable reported cost is known and complete for
|
||||
that attempt. Cost completeness is independent of token-field completeness.
|
||||
- An absent or null provider cost is unknown unless the backend's explicit
|
||||
zero-cost fallback supplies it. A reported numeric zero remains known zero
|
||||
with a provider source, even when the fallback is enabled.
|
||||
- On a dispatched attempt, a usable reported cost or configured-zero fallback
|
||||
is known and complete for that attempt. Cost completeness is independent of
|
||||
token-field completeness and does not assert provider reporting.
|
||||
- Sum known costs across dispatched attempts, including initial generations,
|
||||
retries, and repairs. `Known` means at least one attempt reported cost;
|
||||
`Complete` means every dispatched attempt did. Unknown values contribute no
|
||||
amount, but make the total incomplete. With no dispatched attempts, follow
|
||||
the existing known, complete zero convention.
|
||||
retries, and repairs. `Known` means at least one attempt has an available cost
|
||||
from either source; `Complete` means every dispatched attempt does under this
|
||||
policy. Unknown values contribute no amount, but make the total incomplete.
|
||||
With no dispatched attempts, follow the existing known, complete zero
|
||||
convention, with an empty source.
|
||||
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
||||
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
||||
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
||||
from OpenRouter mixed with unreported local-backend cost is incomplete.
|
||||
from OpenRouter mixed with unreported local-backend cost is incomplete unless
|
||||
those local attempts have an explicit configured-zero fallback. Combine
|
||||
cost-source metadata as described above without changing token accounting.
|
||||
- Keep valid cost observed from a decoded successful envelope when a later
|
||||
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
||||
error bodies continue to supply no usage. Cancellation or failure may leave
|
||||
incomplete reporting even when billable upstream work occurred.
|
||||
error bodies continue to supply no provider usage, but the configured-zero
|
||||
fallback still applies to dispatched attempts. Without that fallback,
|
||||
cancellation or failure may leave incomplete cost reporting even when billable
|
||||
upstream work occurred.
|
||||
- Treat an unusable optional cost value, including a nonnumeric value or one
|
||||
not representable as a finite `float64`, as unknown rather than making an
|
||||
otherwise valid generation fail. This does not relax existing response framing
|
||||
or content validation. An injected client likewise cannot supply a nonfinite
|
||||
usable cost.
|
||||
otherwise valid generation fail; the zero-cost fallback can then apply. This
|
||||
does not relax existing response framing or content validation. An injected
|
||||
client likewise cannot supply a nonfinite usable cost.
|
||||
|
||||
Completeness describes coverage of reported values, not verification of the
|
||||
provider's billing. Preserve reported usable amounts without pricing inference
|
||||
or reconciliation.
|
||||
Completeness describes coverage by available reported or explicitly configured
|
||||
cost, not verification of the provider's billing or proof that every provider
|
||||
reported a charge. Preserve usable reported amounts without pricing inference or
|
||||
reconciliation.
|
||||
|
||||
For a successful run, result cost and terminal-event cost describe the same
|
||||
dispatched attempts and must agree on value, presence, and completeness. Result
|
||||
cost must also be the same with observation enabled or disabled. For example,
|
||||
an unreported failed attempt followed by a successful attempt reporting USD
|
||||
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
|
||||
accounting through repairs and ordinary or prepared execution.
|
||||
dispatched attempts and must agree on value, presence, completeness, and source.
|
||||
Result cost must also be the same with observation enabled or disabled. For
|
||||
example, an unreported failed attempt followed by a successful attempt reporting USD
|
||||
0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
|
||||
source when no zero-cost fallback applies. Apply the same accounting through
|
||||
repairs and ordinary or prepared execution. An invocation with fully reported
|
||||
OpenRouter charges and configured-zero local calls can have a complete subtotal
|
||||
with a mixed source, without claiming all providers reported costs.
|
||||
|
||||
Group cost completeness concerns the terminal observations included in its
|
||||
subtotal. If two completed runs reported all their costs but a third observed
|
||||
@@ -523,7 +603,8 @@ application-specific concepts to PromptKit.
|
||||
Smoke diagnostics retain their separate report/evidence model and are not
|
||||
enrolled in ordinary execution observation by this change. Reasoning-text
|
||||
capture, changes to reasoning or caching request behavior, additional built-in
|
||||
cost providers, multiple currencies, currency conversion, pricing estimates,
|
||||
cost providers, configurable token-rate estimation, fixed nonzero request
|
||||
pricing, multiple currencies, currency conversion, pricing estimates,
|
||||
billing reconciliation, persistent history, and logging infrastructure are
|
||||
outside this scope.
|
||||
|
||||
@@ -537,6 +618,11 @@ outside this scope.
|
||||
- Add `provider` to the backend configuration model, typed source helpers,
|
||||
capture, inspection, and execution plumbing. Update the bundled OpenRouter
|
||||
backend to select it explicitly. Retain the selection through prepared runs.
|
||||
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
|
||||
prepared-execution plumbing. Apply the fallback outside provider-specific
|
||||
decoding so it works with built-in and injected clients. Verify composition
|
||||
and inheritance, omitted/null/false behavior, explicit overrides, provider
|
||||
independence, retained inspection provenance, and absence from provider bodies.
|
||||
- Consolidate provider-specific response and authentication-probe behavior in
|
||||
`internal/llm`; keep coordination and accounting provider-neutral. Cover
|
||||
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
||||
@@ -561,7 +647,8 @@ outside this scope.
|
||||
recognized acceptance, rejection, unrecognized responses, provider defaults,
|
||||
and explicit kind/endpoint overrides, including `none` and adapter selection
|
||||
without a provider declaration. Verify that ordinary generation performs no
|
||||
diagnostic requests and leaves monetary cost unknown.
|
||||
diagnostic requests and leaves monetary cost unknown unless the backend
|
||||
explicitly declares the zero-cost fallback.
|
||||
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
|
||||
details, including responses representative of servers without detail
|
||||
reporting enabled. Verify `created_cache_tokens` mapping independently of
|
||||
@@ -575,6 +662,16 @@ outside this scope.
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
Preserve existing token fields and accounting.
|
||||
- Carry cost-source metadata through results, events, conversions, and group
|
||||
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
|
||||
including zero-valued reported contributions and empty/non-dispatched cases.
|
||||
Verify source union and terminal deduplication independently of cost value.
|
||||
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
|
||||
null, or unusable reported cost permits fallback. Cover dispatched failures,
|
||||
cancellation, undecodable responses, retries, repairs, ordinary/prepared
|
||||
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
|
||||
and declared zero-cost local calls can produce complete cost totals while
|
||||
token completeness remains independent.
|
||||
- Verify that successful-result cost agrees with terminal-event cost and is
|
||||
unchanged when observation is disabled, including the unreported-retry example
|
||||
above. Verify group snapshots with fully reported completed runs alongside a
|
||||
@@ -595,15 +692,18 @@ outside this scope.
|
||||
turn telemetry into an exact-decimal guarantee.
|
||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||
aggregation example to the consumer observation guide, including cost
|
||||
availability, units, and completeness. Document the reasoning/cache fields and
|
||||
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
|
||||
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter
|
||||
cost decoding in the
|
||||
availability, units, completeness, and source. Explain that configured zeros
|
||||
can make cost complete without provider reporting. Document the reasoning/cache
|
||||
fields and cache-hit helper in their owning GoDoc and illustrate input/output
|
||||
breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
|
||||
OpenRouter cost decoding in the
|
||||
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
Document `provider`, its smoke-test defaults and override semantics in the
|
||||
framework format reference and relevant public GoDoc; update inspection and
|
||||
diagnostic documentation with the implemented behavior. Include the diagnostic
|
||||
diagnostic documentation with the implemented behavior. Document `zero_cost`
|
||||
and its fallback precedence, inspection behavior, and prepared lifetime in the
|
||||
relevant format, GoDoc, and consumer owners. Include the diagnostic
|
||||
migration in release notes. Include vLLM usage availability, server reporting
|
||||
settings, the meaning of cache creation, and the limits of its authentication
|
||||
diagnostic in the relevant consumer and integration documentation.
|
||||
@@ -616,12 +716,13 @@ Organize the eventual implementation plan around two workstreams:
|
||||
and nested execution boundaries and the existing group collector.
|
||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
|
||||
extensions and provider-reported cost through decoding, accounting, public
|
||||
results, events, and group summaries, plus the derived cache-hit helper.
|
||||
Include the corresponding configuration and API documentation.
|
||||
extensions, provider-reported cost, and configured-zero fallback through
|
||||
decoding, accounting, public results, events, and group summaries, plus the
|
||||
derived cache-hit helper. Include the corresponding configuration and API
|
||||
documentation.
|
||||
|
||||
The second area is more extensive than the context helper. Keep backend provider
|
||||
configuration, diagnostic consolidation, token/cache reporting, and cost
|
||||
configuration, diagnostic consolidation, token/cache reporting, and cost/source
|
||||
accounting in separately reviewable implementation units rather than one large
|
||||
implementation prompt.
|
||||
Avoid coupling delivery of contextual observation to provider cost support.
|
||||
@@ -637,8 +738,9 @@ contract.
|
||||
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
||||
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
||||
overflowing value fail decoding of the entire response. Decode the optional
|
||||
field separately so an unusable cost becomes unknown without changing existing
|
||||
framing or content validation. This needs no general numeric-conversion layer.
|
||||
field separately so an unusable provider cost becomes unknown before applying
|
||||
any configured-zero fallback, without changing existing framing or content
|
||||
validation. This needs no general numeric-conversion layer.
|
||||
|
||||
Use the completion criteria above to select focused offline checks for each
|
||||
unit. This roadmap owns target behavior; a separate implementation plan owns
|
||||
|
||||
Reference in New Issue
Block a user