306 lines
16 KiB
Markdown
306 lines
16 KiB
Markdown
# Invocation-Scoped Execution Observation
|
|
|
|
Status: accepted scope; implementation pending.
|
|
|
|
## Purpose
|
|
|
|
Allow consumers to select execution observation once for an application-owned
|
|
operation, rather than attach an observer to every PromptKit request. Nested,
|
|
concurrent, and multi-engine calls using that operation's context should
|
|
participate automatically. Include optional provider-reported cost alongside
|
|
token usage, with built-in cost support initially limited to OpenRouter.
|
|
|
|
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
|
could use this facility to report token usage for an entire command or pipeline.
|
|
PromptKit supplies application-neutral observation and aggregation; consumers
|
|
choose how to present or use the results.
|
|
|
|
## Current Baseline
|
|
|
|
PromptKit v0.24.0 already provides:
|
|
|
|
- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous
|
|
metadata callbacks and ordered composition.
|
|
- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached
|
|
snapshots, and aggregation of the first terminal event for each observed run.
|
|
- Presence-aware prompt, completion, total, cached, and cache-write token counts,
|
|
including aggregate accounting across retries and repairs.
|
|
- Explicit selection through `RunRequest.Observer`, retained for prepared
|
|
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
|
coordinator-selected continuation runs.
|
|
|
|
The missing capabilities are contextual observer selection and provider-reported
|
|
cost accounting. Applications currently have to enroll each request explicitly,
|
|
and PromptKit does not expose provider charges. The existing
|
|
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
|
describes that workflow.
|
|
|
|
## Target Public API
|
|
|
|
Add one public helper:
|
|
|
|
```go
|
|
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
|
```
|
|
|
|
The returned context associates the observer with executions using that context
|
|
or derived contexts that retain its values. It composes with any observer already
|
|
associated with the context; it does not replace the inherited observer. A nil
|
|
observer leaves the inherited selection unchanged.
|
|
|
|
No engine option, process-wide default, or replacement group type is needed.
|
|
Extend the existing result, event, and group surfaces with cost metadata as
|
|
described below.
|
|
|
|
The intended consumer workflow is:
|
|
|
|
```go
|
|
var group promptkit.ExecutionGroup
|
|
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
|
|
|
|
err := runApplicationPipeline(ctx)
|
|
// After all application-owned work has finished, including on failure:
|
|
summary := group.Snapshot()
|
|
```
|
|
|
|
The consumer owns context propagation, waiting for its work, and using the
|
|
snapshot and error. Calls using unrelated contexts are not automatically
|
|
enrolled. Group coverage continues to describe observed members, not future
|
|
work or overall application progress.
|
|
|
|
## Observer Selection And Delivery
|
|
|
|
### Composition
|
|
|
|
Preserve existing explicit observers. Deliver each event to the explicit request
|
|
observer first, then any coordinator observer already composed with it, then
|
|
contextual observers in registration order, from inherited to newly added.
|
|
|
|
Preserve the existing synchronous callback and panic behavior. Concurrent
|
|
executions may invoke a shared observer concurrently; consumers remain
|
|
responsible for synchronizing their callback state. `ExecutionGroup` already
|
|
supports concurrent use.
|
|
|
|
### Ordinary And Nested Execution
|
|
|
|
`Run` selects contextual observation from its invocation context. Each execution
|
|
has its own observation identity, recorder, and accounting, including a nested
|
|
execution using a context passed to an application callback or injected client.
|
|
|
|
Inherited observer selection is distinct from private execution state. Existing
|
|
clearing of an enclosing execution's private state at a public execution boundary
|
|
must preserve the contextual observer selection. Inheriting observation does not
|
|
reuse an enclosing run's recorder, totals, or lifecycle.
|
|
|
|
### Prepared Execution
|
|
|
|
`RunPrepared` selects contextual observation from the context supplied when the
|
|
handle is executed. A handle prepared in one scope and executed in another
|
|
participates in the execution scope, not the preparation scope.
|
|
|
|
The explicit `RunRequest.Observer` remains retained by a successfully prepared
|
|
handle as today and composes with the execution-time contextual observers.
|
|
Preparation itself emits no execution events. Discarded handles and rejected
|
|
execution claims retain their existing observation behavior.
|
|
|
|
### Coordinated Execution
|
|
|
|
Continuation runs receive contextual observation through their ordinary `Run`
|
|
boundary. `CoordinateRequest.Observer` continues to select continuation runs
|
|
only. Additional calls made by application callbacks participate when they use
|
|
the scoped context, without being automatically selected by the coordinator's
|
|
explicit observer.
|
|
|
|
Contextual observers are incorporated once per execution, rather than being
|
|
re-added by coordination or another forwarding layer.
|
|
|
|
### Duplicate Registration And Accounting
|
|
|
|
Registering the same callback through multiple explicit or contextual paths may
|
|
deliver the same event more than once. Do not attempt to identify or deduplicate
|
|
arbitrary Go function values.
|
|
|
|
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
|
|
terminal-event deduplication. Selecting the same group through multiple paths
|
|
must not inflate its totals. Terminal usage is an aggregate, not an additional
|
|
contribution to be added to attempt usage. These rules apply to cost as well as
|
|
token counts.
|
|
|
|
## Optional Provider-Reported Cost
|
|
|
|
### Initial Provider Support
|
|
|
|
Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter
|
|
defines as the amount charged to the account, denominated in USD. See its
|
|
[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
|
|
and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
|
|
Do not add `cost_details` breakdowns or combine upstream inference cost with the
|
|
top-level charge; those describe different facts and could double-count cost.
|
|
|
|
For the built-in client, select this interpretation for the resolved
|
|
`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's
|
|
field named `cost`, or add endpoint discovery or verification. An endpoint
|
|
override on that backend continues to select its OpenRouter response contract.
|
|
Other built-in backend selections report unknown cost initially. Additional
|
|
provider adapters or cost-capability configuration are not needed for this scope.
|
|
|
|
This baseline selects by exact backend identity: a separately named or
|
|
collection-qualified backend pointing to OpenRouter would report unknown cost.
|
|
Backend configuration as an alternative to identity-based selection is now an
|
|
open discussion; settle that selection before implementing provider cost support.
|
|
The contextual-observation stage does not depend on that decision.
|
|
|
|
Use the cost included in the normal completion response. No additional usage
|
|
lookup, pricing request, or opt-in provider request parameter is needed.
|
|
|
|
### Monetary Representation
|
|
|
|
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
|
as `CostUSD`, rather than introducing a generic amount with an implicit currency.
|
|
At the individual model-client response boundary, use an optional amount
|
|
(`*float64`): nil means unknown, and a pointer to zero means reported zero.
|
|
Injected clients may supply the same optional USD value without needing an
|
|
OpenRouter backend; no provider-specific interface is needed.
|
|
|
|
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
|
remain token counts. For run results, execution events, and group summaries,
|
|
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
|
`Complete bool`, following the existing observed-count conventions. Successful
|
|
run results should expose cost even when no observer was registered; observers
|
|
also provide per-attempt and terminal cost, including available facts on failure.
|
|
|
|
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
|
addition, and the public API lightweight. Do not round to cents or another fixed
|
|
scale during capture or aggregation; consumers choose display precision.
|
|
Decimal strings or a decimal arithmetic dependency would be appropriate for
|
|
exact financial ledgers, while fixed-scale integers would require a selected
|
|
precision and rounding policy. Neither is warranted for this reporting feature.
|
|
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
|
|
|
### Presence, Aggregation, And Failures
|
|
|
|
- An absent or null cost is unknown. Explicit numeric zero is known zero.
|
|
A backend that reports no cost remains unknown even when a consumer knows it
|
|
has no external charge.
|
|
- On a dispatched attempt, a usable reported cost is known and complete for
|
|
that attempt. Cost completeness is independent of token-field completeness.
|
|
- Sum known costs across dispatched attempts, including initial generations,
|
|
retries, and repairs. `Known` means at least one attempt reported cost;
|
|
`Complete` means every dispatched attempt did. Unknown values contribute no
|
|
amount, but make the total incomplete. With no dispatched attempts, follow
|
|
the existing known, complete zero convention.
|
|
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
|
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
|
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
|
from OpenRouter mixed with unreported local-backend cost is incomplete.
|
|
- Keep valid cost observed from a decoded successful envelope when a later
|
|
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
|
error bodies continue to supply no usage. Cancellation or failure may leave
|
|
incomplete reporting even when billable upstream work occurred.
|
|
- Treat an unusable optional cost value, including a nonnumeric value or one
|
|
not representable as a finite `float64`, as unknown rather than making an
|
|
otherwise valid generation fail. This does not relax existing response framing
|
|
or content validation. An injected client likewise cannot supply a nonfinite
|
|
usable cost.
|
|
|
|
Completeness describes coverage of reported values, not verification of the
|
|
provider's billing. Preserve reported usable amounts without pricing inference
|
|
or reconciliation.
|
|
|
|
For a successful run, result cost and terminal-event cost describe the same
|
|
dispatched attempts and must agree on value, presence, and completeness. Result
|
|
cost must also be the same with observation enabled or disabled. For example,
|
|
an unreported failed attempt followed by a successful attempt reporting USD
|
|
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
|
|
accounting through repairs and ordinary or prepared execution.
|
|
|
|
Group cost completeness concerns the terminal observations included in its
|
|
subtotal. If two completed runs reported all their costs but a third observed
|
|
run has no terminal event, the subtotal may have `Complete: true` while
|
|
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
|
|
coverage field; no additional completion flag or group lifecycle is needed.
|
|
|
|
## Accounting And Responsibility Boundaries
|
|
|
|
Reuse existing events, token presence/completeness, and group summaries, extending
|
|
them with the cost facts above. Available usage remains observable after an error
|
|
or cancellation under the existing execution-observation behavior; this does not
|
|
promise that upstream providers report usage for every attempt.
|
|
|
|
PromptKit owns observer selection, event generation, and aggregation. CommandKit
|
|
and other consumers own invocation lifecycle, CLI flags, presentation, output
|
|
delivery, and interpretation. This feature adds no CommandKit dependency or
|
|
application-specific concepts to PromptKit.
|
|
|
|
Smoke diagnostics retain their separate report/evidence model and are not
|
|
enrolled in ordinary execution observation by this change. Reasoning-token
|
|
capture, additional built-in cost providers, multiple currencies, currency
|
|
conversion, pricing estimates, billing reconciliation, persistent history, and
|
|
logging infrastructure are outside this scope.
|
|
|
|
## Work Needed And Completion Criteria
|
|
|
|
- Add the context helper and connect observer selection to ordinary and prepared
|
|
execution, keeping contextual selection separate from private run state.
|
|
- Preserve existing explicit selection and coordinator behavior while applying
|
|
the composition and lifetime rules above. Preserve existing observer selection
|
|
when no contextual observer is selected.
|
|
- Carry optional OpenRouter cost through response decoding, internal/public
|
|
conversions, successful run results, attempt and terminal events, and existing
|
|
group summaries. Keep result cost available without registering an observer.
|
|
Preserve existing token fields and accounting.
|
|
- Verify that successful-result cost agrees with terminal-event cost and is
|
|
unchanged when observation is disabled, including the unreported-retry example
|
|
above. Verify group snapshots with fully reported completed runs alongside a
|
|
member without a terminal observation.
|
|
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
|
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
|
and independent scopes on a reused engine.
|
|
- Cover callback ordering, nested contextual registration, nil registration,
|
|
duplicate group enrollment, prepared-handle scope selection, and existing
|
|
preparation/discard/rejected-claim behavior.
|
|
- Verify aggregate accounting through retries and repairs, available usage after
|
|
failures or cancellation, and known zero versus unknown or incomplete counts.
|
|
Use offline fixtures and injected clients; live inference is unnecessary.
|
|
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
|
|
non-OpenRouter built-in selection, optional injected-client cost, mixed known
|
|
and unknown totals, repair/retry aggregation, and duplicate group registration.
|
|
Use tolerant comparisons where floating-point addition requires them; do not
|
|
turn telemetry into an exact-decimal guarantee.
|
|
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
|
aggregation example to the consumer observation guide, including cost
|
|
availability, units, and completeness. Document OpenRouter cost decoding in
|
|
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
|
and update internal observation and model-client documentation for changed flows.
|
|
|
|
## Implementation Planning Guidance
|
|
|
|
Plan two separately reviewable stages:
|
|
|
|
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
|
and nested execution boundaries and the existing group collector.
|
|
2. Provider-reported cost, carried through decoding, accounting, public results,
|
|
events, and group summaries, with the corresponding documentation.
|
|
|
|
The second stage is more extensive than the context helper. Resolve the backend
|
|
selection discussion before finalizing that stage; avoid coupling delivery of
|
|
contextual observation to provider cost support.
|
|
|
|
Today, successful results accumulate generation responses in the runner, while
|
|
optional observation recorders account for dispatched attempts. Adding cost to
|
|
response objects alone does not preserve completeness across unreported retries.
|
|
Reuse a small accounting component for result and observation paths so they
|
|
apply the same rules, including when no observer is selected. Preserve existing
|
|
token accounting rather than expanding this work into a redesign of its public
|
|
contract.
|
|
|
|
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
|
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
|
overflowing value fail decoding of the entire response. Decode the optional
|
|
field separately so an unusable cost becomes unknown without changing existing
|
|
framing or content validation. This needs no general numeric-conversion layer.
|
|
|
|
Use the completion criteria above to select focused offline checks for each
|
|
stage. This roadmap owns target behavior; a separate implementation plan owns
|
|
the detailed work sequence.
|