Clarify observation accounting and implementation planning

This commit is contained in:
2026-11-22 10:02:22 +00:00
parent 03209947f6
commit 0564c8086c

View File

@@ -144,6 +144,12 @@ override on that backend continues to select its OpenRouter response contract.
Other built-in backend selections report unknown cost initially. Additional Other built-in backend selections report unknown cost initially. Additional
provider adapters or cost-capability configuration are not needed for this scope. provider adapters or cost-capability configuration are not needed for this scope.
This baseline selects by exact backend identity: a separately named or
collection-qualified backend pointing to OpenRouter would report unknown cost.
Backend configuration as an alternative to identity-based selection is now an
open discussion; settle that selection before implementing provider cost support.
The contextual-observation stage does not depend on that decision.
Use the cost included in the normal completion response. No additional usage Use the cost included in the normal completion response. No additional usage
lookup, pricing request, or opt-in provider request parameter is needed. lookup, pricing request, or opt-in provider request parameter is needed.
@@ -201,6 +207,19 @@ Completeness describes coverage of reported values, not verification of the
provider's billing. Preserve reported usable amounts without pricing inference provider's billing. Preserve reported usable amounts without pricing inference
or reconciliation. or reconciliation.
For a successful run, result cost and terminal-event cost describe the same
dispatched attempts and must agree on value, presence, and completeness. Result
cost must also be the same with observation enabled or disabled. For example,
an unreported failed attempt followed by a successful attempt reporting USD
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
accounting through repairs and ordinary or prepared execution.
Group cost completeness concerns the terminal observations included in its
subtotal. If two completed runs reported all their costs but a third observed
run has no terminal event, the subtotal may have `Complete: true` while
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
coverage field; no additional completion flag or group lifecycle is needed.
## Accounting And Responsibility Boundaries ## Accounting And Responsibility Boundaries
Reuse existing events, token presence/completeness, and group summaries, extending Reuse existing events, token presence/completeness, and group summaries, extending
@@ -230,6 +249,10 @@ logging infrastructure are outside this scope.
conversions, successful run results, attempt and terminal events, and existing conversions, successful run results, attempt and terminal events, and existing
group summaries. Keep result cost available without registering an observer. group summaries. Keep result cost available without registering an observer.
Preserve existing token fields and accounting. Preserve existing token fields and accounting.
- Verify that successful-result cost agrees with terminal-event cost and is
unchanged when observation is disabled, including the unreported-retry example
above. Verify group snapshots with fully reported completed runs alongside a
member without a terminal observation.
- Cover ordinary and prepared execution, coordinated continuations, nested calls - Cover ordinary and prepared execution, coordinated continuations, nested calls
using derived contexts, concurrent calls, multiple engines sharing a scope, using derived contexts, concurrent calls, multiple engines sharing a scope,
and independent scopes on a reused engine. and independent scopes on a reused engine.
@@ -250,5 +273,33 @@ logging infrastructure are outside this scope.
the [integration contract](../integrations/openai-compatible-chat.md#response-handling), the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
and update internal observation and model-client documentation for changed flows. and update internal observation and model-client documentation for changed flows.
This roadmap defines the target behavior. A separate implementation plan can ## Implementation Planning Guidance
divide the work into appropriately sized stages.
Plan two separately reviewable stages:
1. Contextual observer selection, including ordinary, prepared, coordinated,
and nested execution boundaries and the existing group collector.
2. Provider-reported cost, carried through decoding, accounting, public results,
events, and group summaries, with the corresponding documentation.
The second stage is more extensive than the context helper. Resolve the backend
selection discussion before finalizing that stage; avoid coupling delivery of
contextual observation to provider cost support.
Today, successful results accumulate generation responses in the runner, while
optional observation recorders account for dispatched attempts. Adding cost to
response objects alone does not preserve completeness across unreported retries.
Reuse a small accounting component for result and observation paths so they
apply the same rules, including when no observer is selected. Preserve existing
token accounting rather than expanding this work into a redesign of its public
contract.
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
overflowing value fail decoding of the entire response. Decode the optional
field separately so an unusable cost becomes unknown without changing existing
framing or content validation. This needs no general numeric-conversion layer.
Use the completion criteria above to select focused offline checks for each
stage. This roadmap owns target behavior; a separate implementation plan owns
the detailed work sequence.