Document invocation observation and cost reporting
This commit is contained in:
@@ -728,6 +728,40 @@ counts, but its zero counts remain unknown to observations; supply explicit
|
||||
presence when measured zeros matter. Existing `RunResult.Usage` sums returned
|
||||
values independently of observation presence.
|
||||
|
||||
For an application-owned invocation spanning several calls or engines, attach
|
||||
an observer once to a context and propagate it to those calls:
|
||||
|
||||
```go
|
||||
var group promptkit.ExecutionGroup
|
||||
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
|
||||
result, err := engine.Run(ctx, request)
|
||||
// After joining all application-owned work, inspect group.Snapshot(), even on error.
|
||||
```
|
||||
|
||||
Explicit request observers run first, followed by contextual observers in
|
||||
registration order. Coordinated continuations pick up the context through
|
||||
`Run`; prepared handles retain the request observer but use the execution
|
||||
context's observers, not the preparation context's. Each nested call has its
|
||||
own run ID and accounting. Duplicate registration can deliver duplicate events;
|
||||
groups count each run's first terminal once.
|
||||
|
||||
Reasoning tokens are a subset of completion tokens; cache reads and writes are
|
||||
input details, not extra tokens. `ObservedTokenUsage.CacheHitRatio()` derives
|
||||
the cache-read fraction only when input and read totals are known, complete,
|
||||
positive-denominator, and consistent. It divides aggregate counts rather than
|
||||
averaging per-run ratios and is independent of cache writes.
|
||||
|
||||
`RunResult.CostUSD` is available even without an observer; attempt and terminal
|
||||
events and group snapshots also report cost. `Known` denotes an available
|
||||
subtotal, `Complete` covers dispatched attempts, and `Source` distinguishes
|
||||
provider reports, an explicitly configured zero fallback, and mixed sources.
|
||||
Reported cost, including zero, takes precedence over fallback. Unknown failed
|
||||
attempts make a subtotal incomplete; configured zero may cover them. Terminal
|
||||
events can report cost when a failed operation returns no result. Neither
|
||||
complete reporting nor provider attribution verifies an invoice; consumers
|
||||
choose presentation precision. See [backend definitions](../formats.md#backend-definitions)
|
||||
and the [integration contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
|
||||
Callbacks run synchronously, in order for one operation, and can overlap for
|
||||
different operations. Synchronize shared callback state and avoid blocking on
|
||||
work that needs the same run's admission slot. Callback panics propagate;
|
||||
@@ -879,15 +913,15 @@ outcome, err := engine.Coordinate(ctx, promptkit.CoordinateRequest{
|
||||
})
|
||||
```
|
||||
|
||||
`assess` is application code and explicitly selects its observer on any nested
|
||||
run. A completed turn releases its admission lease before the callback, so that
|
||||
`assess` is application code and can select its observer explicitly or inherit
|
||||
one from the context. A completed turn releases its admission lease before the callback, so that
|
||||
assessment may use the same backend. `stopped` and `exhausted` describe control
|
||||
flow, not content acceptance. Cancellation before callback delivery can return one
|
||||
completed response in `Undelivered`; errors retain their ordinary identities.
|
||||
|
||||
### Collect Selected Execution Facts
|
||||
|
||||
Groups are independent of continuation and enrollment is explicit:
|
||||
Groups are independent of continuation and can be enrolled explicitly:
|
||||
|
||||
```go
|
||||
var group, validatorGroup promptkit.ExecutionGroup
|
||||
@@ -904,7 +938,7 @@ _, _ = summary, validatorSummary
|
||||
```
|
||||
|
||||
Only the first terminal event per run contributes counts and usage; coverage of
|
||||
currently observed members is separate from token-usage completeness. Empty
|
||||
currently observed members is separate from token and cost completeness. Empty
|
||||
groups report complete, known zero totals, while members without terminals do
|
||||
not invent totals. Overlapping groups are not generally additive. See the
|
||||
[offline example](../../examples/go-library/observe/main.go) for complete setup.
|
||||
|
||||
Reference in New Issue
Block a user