Define invocation-scoped observation roadmap

This commit is contained in:
2026-11-22 09:39:27 +00:00
parent 5ea60e9b96
commit 1070be8d89

View File

@@ -0,0 +1,162 @@
# Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending.
## Purpose
Allow consumers to select execution observation once for an application-owned
operation, rather than attach an observer to every PromptKit request. Nested,
concurrent, and multi-engine calls using that operation's context should
participate automatically.
The motivating consumer is CommandKit's optional `ckprompt` integration, which
could use this facility to report token usage for an entire command or pipeline.
PromptKit supplies application-neutral observation and aggregation; consumers
choose how to present or use the results.
## Current Baseline
PromptKit v0.24.0 already provides:
- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous
metadata callbacks and ordered composition.
- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached
snapshots, and aggregation of the first terminal event for each observed run.
- Presence-aware prompt, completion, total, cached, and cache-write token counts,
including aggregate accounting across retries and repairs.
- Explicit selection through `RunRequest.Observer`, retained for prepared
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
coordinator-selected continuation runs.
The missing capability is contextual observer selection. Applications currently
have to enroll each request explicitly. The existing
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
describes that workflow.
## Target Public API
Add one public helper:
```go
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
```
The returned context associates the observer with executions using that context
or derived contexts that retain its values. It composes with any observer already
associated with the context; it does not replace the inherited observer. A nil
observer leaves the inherited selection unchanged.
No engine option, process-wide default, or new aggregation type is needed.
The intended consumer workflow is:
```go
var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
err := runApplicationPipeline(ctx)
// After all application-owned work has finished, including on failure:
summary := group.Snapshot()
```
The consumer owns context propagation, waiting for its work, and using the
snapshot and error. Calls using unrelated contexts are not automatically
enrolled. Group coverage continues to describe observed members, not future
work or overall application progress.
## Observer Selection And Delivery
### Composition
Preserve existing explicit observers. Deliver each event to the explicit request
observer first, then any coordinator observer already composed with it, then
contextual observers in registration order, from inherited to newly added.
Preserve the existing synchronous callback and panic behavior. Concurrent
executions may invoke a shared observer concurrently; consumers remain
responsible for synchronizing their callback state. `ExecutionGroup` already
supports concurrent use.
### Ordinary And Nested Execution
`Run` selects contextual observation from its invocation context. Each execution
has its own observation identity, recorder, and accounting, including a nested
execution using a context passed to an application callback or injected client.
Inherited observer selection is distinct from private execution state. Existing
clearing of an enclosing execution's private state at a public execution boundary
must preserve the contextual observer selection. Inheriting observation does not
reuse an enclosing run's recorder, totals, or lifecycle.
### Prepared Execution
`RunPrepared` selects contextual observation from the context supplied when the
handle is executed. A handle prepared in one scope and executed in another
participates in the execution scope, not the preparation scope.
The explicit `RunRequest.Observer` remains retained by a successfully prepared
handle as today and composes with the execution-time contextual observers.
Preparation itself emits no execution events. Discarded handles and rejected
execution claims retain their existing observation behavior.
### Coordinated Execution
Continuation runs receive contextual observation through their ordinary `Run`
boundary. `CoordinateRequest.Observer` continues to select continuation runs
only. Additional calls made by application callbacks participate when they use
the scoped context, without being automatically selected by the coordinator's
explicit observer.
Contextual observers are incorporated once per execution, rather than being
re-added by coordination or another forwarding layer.
### Duplicate Registration And Accounting
Registering the same callback through multiple explicit or contextual paths may
deliver the same event more than once. Do not attempt to identify or deduplicate
arbitrary Go function values.
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
terminal-event deduplication. Selecting the same group through multiple paths
must not inflate its totals. Terminal usage is an aggregate, not an additional
contribution to be added to attempt usage.
## Accounting And Responsibility Boundaries
Reuse existing events, usage presence/completeness, and group summaries. Available
usage remains observable after an error or cancellation under the existing
execution-observation behavior; this does not promise that upstream providers
report usage for every attempt.
PromptKit owns observer selection, event generation, and aggregation. CommandKit
and other consumers own invocation lifecycle, CLI flags, presentation, output
delivery, and interpretation. This feature adds no CommandKit dependency or
application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation by this change. Additional usage
fields such as reasoning tokens and monetary cost, pricing estimates, billing
reconciliation, persistent history, and logging infrastructure are outside this
scope.
## Work Needed And Completion Criteria
- Add the context helper and connect observer selection to ordinary and prepared
execution, keeping contextual selection separate from private run state.
- Preserve existing explicit selection and coordinator behavior while applying
the composition and lifetime rules above. Keep existing execution behavior
unchanged when no contextual observer is selected.
- Cover ordinary and prepared execution, coordinated continuations, nested calls
using derived contexts, concurrent calls, multiple engines sharing a scope,
and independent scopes on a reused engine.
- Cover callback ordering, nested contextual registration, nil registration,
duplicate group enrollment, prepared-handle scope selection, and existing
preparation/discard/rejected-claim behavior.
- Verify aggregate accounting through retries and repairs, available usage after
failures or cancellation, and known zero versus unknown or incomplete counts.
Use offline fixtures and injected clients; live inference is unnecessary.
- Document the exact helper contract in GoDoc and add a focused invocation-level
aggregation example to the consumer observation guide. Update internal
observation documentation where its selection flow changes.
This roadmap defines the target behavior. A separate implementation plan can
divide the work into appropriately sized stages.