Clarify observation accounting and implementation planning
This commit is contained in:
@@ -144,6 +144,12 @@ override on that backend continues to select its OpenRouter response contract.
|
|||||||
Other built-in backend selections report unknown cost initially. Additional
|
Other built-in backend selections report unknown cost initially. Additional
|
||||||
provider adapters or cost-capability configuration are not needed for this scope.
|
provider adapters or cost-capability configuration are not needed for this scope.
|
||||||
|
|
||||||
|
This baseline selects by exact backend identity: a separately named or
|
||||||
|
collection-qualified backend pointing to OpenRouter would report unknown cost.
|
||||||
|
Backend configuration as an alternative to identity-based selection is now an
|
||||||
|
open discussion; settle that selection before implementing provider cost support.
|
||||||
|
The contextual-observation stage does not depend on that decision.
|
||||||
|
|
||||||
Use the cost included in the normal completion response. No additional usage
|
Use the cost included in the normal completion response. No additional usage
|
||||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||||
|
|
||||||
@@ -201,6 +207,19 @@ Completeness describes coverage of reported values, not verification of the
|
|||||||
provider's billing. Preserve reported usable amounts without pricing inference
|
provider's billing. Preserve reported usable amounts without pricing inference
|
||||||
or reconciliation.
|
or reconciliation.
|
||||||
|
|
||||||
|
For a successful run, result cost and terminal-event cost describe the same
|
||||||
|
dispatched attempts and must agree on value, presence, and completeness. Result
|
||||||
|
cost must also be the same with observation enabled or disabled. For example,
|
||||||
|
an unreported failed attempt followed by a successful attempt reporting USD
|
||||||
|
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
|
||||||
|
accounting through repairs and ordinary or prepared execution.
|
||||||
|
|
||||||
|
Group cost completeness concerns the terminal observations included in its
|
||||||
|
subtotal. If two completed runs reported all their costs but a third observed
|
||||||
|
run has no terminal event, the subtotal may have `Complete: true` while
|
||||||
|
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
|
||||||
|
coverage field; no additional completion flag or group lifecycle is needed.
|
||||||
|
|
||||||
## Accounting And Responsibility Boundaries
|
## Accounting And Responsibility Boundaries
|
||||||
|
|
||||||
Reuse existing events, token presence/completeness, and group summaries, extending
|
Reuse existing events, token presence/completeness, and group summaries, extending
|
||||||
@@ -230,6 +249,10 @@ logging infrastructure are outside this scope.
|
|||||||
conversions, successful run results, attempt and terminal events, and existing
|
conversions, successful run results, attempt and terminal events, and existing
|
||||||
group summaries. Keep result cost available without registering an observer.
|
group summaries. Keep result cost available without registering an observer.
|
||||||
Preserve existing token fields and accounting.
|
Preserve existing token fields and accounting.
|
||||||
|
- Verify that successful-result cost agrees with terminal-event cost and is
|
||||||
|
unchanged when observation is disabled, including the unreported-retry example
|
||||||
|
above. Verify group snapshots with fully reported completed runs alongside a
|
||||||
|
member without a terminal observation.
|
||||||
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
||||||
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
||||||
and independent scopes on a reused engine.
|
and independent scopes on a reused engine.
|
||||||
@@ -250,5 +273,33 @@ logging infrastructure are outside this scope.
|
|||||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||||
and update internal observation and model-client documentation for changed flows.
|
and update internal observation and model-client documentation for changed flows.
|
||||||
|
|
||||||
This roadmap defines the target behavior. A separate implementation plan can
|
## Implementation Planning Guidance
|
||||||
divide the work into appropriately sized stages.
|
|
||||||
|
Plan two separately reviewable stages:
|
||||||
|
|
||||||
|
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||||
|
and nested execution boundaries and the existing group collector.
|
||||||
|
2. Provider-reported cost, carried through decoding, accounting, public results,
|
||||||
|
events, and group summaries, with the corresponding documentation.
|
||||||
|
|
||||||
|
The second stage is more extensive than the context helper. Resolve the backend
|
||||||
|
selection discussion before finalizing that stage; avoid coupling delivery of
|
||||||
|
contextual observation to provider cost support.
|
||||||
|
|
||||||
|
Today, successful results accumulate generation responses in the runner, while
|
||||||
|
optional observation recorders account for dispatched attempts. Adding cost to
|
||||||
|
response objects alone does not preserve completeness across unreported retries.
|
||||||
|
Reuse a small accounting component for result and observation paths so they
|
||||||
|
apply the same rules, including when no observer is selected. Preserve existing
|
||||||
|
token accounting rather than expanding this work into a redesign of its public
|
||||||
|
contract.
|
||||||
|
|
||||||
|
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
||||||
|
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
||||||
|
overflowing value fail decoding of the entire response. Decode the optional
|
||||||
|
field separately so an unusable cost becomes unknown without changing existing
|
||||||
|
framing or content validation. This needs no general numeric-conversion layer.
|
||||||
|
|
||||||
|
Use the completion criteria above to select focused offline checks for each
|
||||||
|
stage. This roadmap owns target behavior; a separate implementation plan owns
|
||||||
|
the detailed work sequence.
|
||||||
|
|||||||
Reference in New Issue
Block a user