Document invocation observation and cost reporting

This commit is contained in:
2026-11-22 18:38:11 +00:00
parent d4509b163e
commit 20c5ea47af
22 changed files with 231 additions and 52 deletions

View File

@@ -728,6 +728,40 @@ counts, but its zero counts remain unknown to observations; supply explicit
presence when measured zeros matter. Existing `RunResult.Usage` sums returned
values independently of observation presence.
For an application-owned invocation spanning several calls or engines, attach
an observer once to a context and propagate it to those calls:
```go
var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
result, err := engine.Run(ctx, request)
// After joining all application-owned work, inspect group.Snapshot(), even on error.
```
Explicit request observers run first, followed by contextual observers in
registration order. Coordinated continuations pick up the context through
`Run`; prepared handles retain the request observer but use the execution
context's observers, not the preparation context's. Each nested call has its
own run ID and accounting. Duplicate registration can deliver duplicate events;
groups count each run's first terminal once.
Reasoning tokens are a subset of completion tokens; cache reads and writes are
input details, not extra tokens. `ObservedTokenUsage.CacheHitRatio()` derives
the cache-read fraction only when input and read totals are known, complete,
positive-denominator, and consistent. It divides aggregate counts rather than
averaging per-run ratios and is independent of cache writes.
`RunResult.CostUSD` is available even without an observer; attempt and terminal
events and group snapshots also report cost. `Known` denotes an available
subtotal, `Complete` covers dispatched attempts, and `Source` distinguishes
provider reports, an explicitly configured zero fallback, and mixed sources.
Reported cost, including zero, takes precedence over fallback. Unknown failed
attempts make a subtotal incomplete; configured zero may cover them. Terminal
events can report cost when a failed operation returns no result. Neither
complete reporting nor provider attribution verifies an invoice; consumers
choose presentation precision. See [backend definitions](../formats.md#backend-definitions)
and the [integration contract](../integrations/openai-compatible-chat.md#response-handling).
Callbacks run synchronously, in order for one operation, and can overlap for
different operations. Synchronize shared callback state and avoid blocking on
work that needs the same run's admission slot. Callback panics propagate;
@@ -879,15 +913,15 @@ outcome, err := engine.Coordinate(ctx, promptkit.CoordinateRequest{
})
```
`assess` is application code and explicitly selects its observer on any nested
run. A completed turn releases its admission lease before the callback, so that
`assess` is application code and can select its observer explicitly or inherit
one from the context. A completed turn releases its admission lease before the callback, so that
assessment may use the same backend. `stopped` and `exhausted` describe control
flow, not content acceptance. Cancellation before callback delivery can return one
completed response in `Undelivered`; errors retain their ordinary identities.
### Collect Selected Execution Facts
Groups are independent of continuation and enrollment is explicit:
Groups are independent of continuation and can be enrolled explicitly:
```go
var group, validatorGroup promptkit.ExecutionGroup
@@ -904,7 +938,7 @@ _, _ = summary, validatorSummary
```
Only the first terminal event per run contributes counts and usage; coverage of
currently observed members is separate from token-usage completeness. Empty
currently observed members is separate from token and cost completeness. Empty
groups report complete, known zero totals, while members without terminals do
not invent totals. Overlapping groups are not generally additive. See the
[offline example](../../examples/go-library/observe/main.go) for complete setup.