Define invocation-scoped observation roadmap
This commit is contained in:
162
docs/roadmap/invocation-scoped-token-usage.md
Normal file
162
docs/roadmap/invocation-scoped-token-usage.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Invocation-Scoped Execution Observation
|
||||
|
||||
Status: accepted scope; implementation pending.
|
||||
|
||||
## Purpose
|
||||
|
||||
Allow consumers to select execution observation once for an application-owned
|
||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||
concurrent, and multi-engine calls using that operation's context should
|
||||
participate automatically.
|
||||
|
||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||
could use this facility to report token usage for an entire command or pipeline.
|
||||
PromptKit supplies application-neutral observation and aggregation; consumers
|
||||
choose how to present or use the results.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
PromptKit v0.24.0 already provides:
|
||||
|
||||
- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous
|
||||
metadata callbacks and ordered composition.
|
||||
- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached
|
||||
snapshots, and aggregation of the first terminal event for each observed run.
|
||||
- Presence-aware prompt, completion, total, cached, and cache-write token counts,
|
||||
including aggregate accounting across retries and repairs.
|
||||
- Explicit selection through `RunRequest.Observer`, retained for prepared
|
||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||
coordinator-selected continuation runs.
|
||||
|
||||
The missing capability is contextual observer selection. Applications currently
|
||||
have to enroll each request explicitly. The existing
|
||||
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
||||
describes that workflow.
|
||||
|
||||
## Target Public API
|
||||
|
||||
Add one public helper:
|
||||
|
||||
```go
|
||||
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
||||
```
|
||||
|
||||
The returned context associates the observer with executions using that context
|
||||
or derived contexts that retain its values. It composes with any observer already
|
||||
associated with the context; it does not replace the inherited observer. A nil
|
||||
observer leaves the inherited selection unchanged.
|
||||
|
||||
No engine option, process-wide default, or new aggregation type is needed.
|
||||
The intended consumer workflow is:
|
||||
|
||||
```go
|
||||
var group promptkit.ExecutionGroup
|
||||
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
|
||||
|
||||
err := runApplicationPipeline(ctx)
|
||||
// After all application-owned work has finished, including on failure:
|
||||
summary := group.Snapshot()
|
||||
```
|
||||
|
||||
The consumer owns context propagation, waiting for its work, and using the
|
||||
snapshot and error. Calls using unrelated contexts are not automatically
|
||||
enrolled. Group coverage continues to describe observed members, not future
|
||||
work or overall application progress.
|
||||
|
||||
## Observer Selection And Delivery
|
||||
|
||||
### Composition
|
||||
|
||||
Preserve existing explicit observers. Deliver each event to the explicit request
|
||||
observer first, then any coordinator observer already composed with it, then
|
||||
contextual observers in registration order, from inherited to newly added.
|
||||
|
||||
Preserve the existing synchronous callback and panic behavior. Concurrent
|
||||
executions may invoke a shared observer concurrently; consumers remain
|
||||
responsible for synchronizing their callback state. `ExecutionGroup` already
|
||||
supports concurrent use.
|
||||
|
||||
### Ordinary And Nested Execution
|
||||
|
||||
`Run` selects contextual observation from its invocation context. Each execution
|
||||
has its own observation identity, recorder, and accounting, including a nested
|
||||
execution using a context passed to an application callback or injected client.
|
||||
|
||||
Inherited observer selection is distinct from private execution state. Existing
|
||||
clearing of an enclosing execution's private state at a public execution boundary
|
||||
must preserve the contextual observer selection. Inheriting observation does not
|
||||
reuse an enclosing run's recorder, totals, or lifecycle.
|
||||
|
||||
### Prepared Execution
|
||||
|
||||
`RunPrepared` selects contextual observation from the context supplied when the
|
||||
handle is executed. A handle prepared in one scope and executed in another
|
||||
participates in the execution scope, not the preparation scope.
|
||||
|
||||
The explicit `RunRequest.Observer` remains retained by a successfully prepared
|
||||
handle as today and composes with the execution-time contextual observers.
|
||||
Preparation itself emits no execution events. Discarded handles and rejected
|
||||
execution claims retain their existing observation behavior.
|
||||
|
||||
### Coordinated Execution
|
||||
|
||||
Continuation runs receive contextual observation through their ordinary `Run`
|
||||
boundary. `CoordinateRequest.Observer` continues to select continuation runs
|
||||
only. Additional calls made by application callbacks participate when they use
|
||||
the scoped context, without being automatically selected by the coordinator's
|
||||
explicit observer.
|
||||
|
||||
Contextual observers are incorporated once per execution, rather than being
|
||||
re-added by coordination or another forwarding layer.
|
||||
|
||||
### Duplicate Registration And Accounting
|
||||
|
||||
Registering the same callback through multiple explicit or contextual paths may
|
||||
deliver the same event more than once. Do not attempt to identify or deduplicate
|
||||
arbitrary Go function values.
|
||||
|
||||
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
|
||||
terminal-event deduplication. Selecting the same group through multiple paths
|
||||
must not inflate its totals. Terminal usage is an aggregate, not an additional
|
||||
contribution to be added to attempt usage.
|
||||
|
||||
## Accounting And Responsibility Boundaries
|
||||
|
||||
Reuse existing events, usage presence/completeness, and group summaries. Available
|
||||
usage remains observable after an error or cancellation under the existing
|
||||
execution-observation behavior; this does not promise that upstream providers
|
||||
report usage for every attempt.
|
||||
|
||||
PromptKit owns observer selection, event generation, and aggregation. CommandKit
|
||||
and other consumers own invocation lifecycle, CLI flags, presentation, output
|
||||
delivery, and interpretation. This feature adds no CommandKit dependency or
|
||||
application-specific concepts to PromptKit.
|
||||
|
||||
Smoke diagnostics retain their separate report/evidence model and are not
|
||||
enrolled in ordinary execution observation by this change. Additional usage
|
||||
fields such as reasoning tokens and monetary cost, pricing estimates, billing
|
||||
reconciliation, persistent history, and logging infrastructure are outside this
|
||||
scope.
|
||||
|
||||
## Work Needed And Completion Criteria
|
||||
|
||||
- Add the context helper and connect observer selection to ordinary and prepared
|
||||
execution, keeping contextual selection separate from private run state.
|
||||
- Preserve existing explicit selection and coordinator behavior while applying
|
||||
the composition and lifetime rules above. Keep existing execution behavior
|
||||
unchanged when no contextual observer is selected.
|
||||
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
||||
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
||||
and independent scopes on a reused engine.
|
||||
- Cover callback ordering, nested contextual registration, nil registration,
|
||||
duplicate group enrollment, prepared-handle scope selection, and existing
|
||||
preparation/discard/rejected-claim behavior.
|
||||
- Verify aggregate accounting through retries and repairs, available usage after
|
||||
failures or cancellation, and known zero versus unknown or incomplete counts.
|
||||
Use offline fixtures and injected clients; live inference is unnecessary.
|
||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||
aggregation example to the consumer observation guide. Update internal
|
||||
observation documentation where its selection flow changes.
|
||||
|
||||
This roadmap defines the target behavior. A separate implementation plan can
|
||||
divide the work into appropriately sized stages.
|
||||
Reference in New Issue
Block a user