7.4 KiB
Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending.
Purpose
Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically.
The motivating consumer is CommandKit's optional ckprompt integration, which
could use this facility to report token usage for an entire command or pipeline.
PromptKit supplies application-neutral observation and aggregation; consumers
choose how to present or use the results.
Current Baseline
PromptKit v0.24.0 already provides:
- ExecutionObserver and ComposeObservers for synchronous metadata callbacks and ordered composition.
- ExecutionGroup for concurrent collection, detached snapshots, and aggregation of the first terminal event for each observed run.
- Presence-aware prompt, completion, total, cached, and cache-write token counts, including aggregate accounting across retries and repairs.
- Explicit selection through
RunRequest.Observer, retained for prepared execution, and CoordinateRequest.Observer for coordinator-selected continuation runs.
The missing capability is contextual observer selection. Applications currently have to enroll each request explicitly. The existing consumer observation guide describes that workflow.
Target Public API
Add one public helper:
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
The returned context associates the observer with executions using that context or derived contexts that retain its values. It composes with any observer already associated with the context; it does not replace the inherited observer. A nil observer leaves the inherited selection unchanged.
No engine option, process-wide default, or new aggregation type is needed. The intended consumer workflow is:
var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
err := runApplicationPipeline(ctx)
// After all application-owned work has finished, including on failure:
summary := group.Snapshot()
The consumer owns context propagation, waiting for its work, and using the snapshot and error. Calls using unrelated contexts are not automatically enrolled. Group coverage continues to describe observed members, not future work or overall application progress.
Observer Selection And Delivery
Composition
Preserve existing explicit observers. Deliver each event to the explicit request observer first, then any coordinator observer already composed with it, then contextual observers in registration order, from inherited to newly added.
Preserve the existing synchronous callback and panic behavior. Concurrent
executions may invoke a shared observer concurrently; consumers remain
responsible for synchronizing their callback state. ExecutionGroup already
supports concurrent use.
Ordinary And Nested Execution
Run selects contextual observation from its invocation context. Each execution
has its own observation identity, recorder, and accounting, including a nested
execution using a context passed to an application callback or injected client.
Inherited observer selection is distinct from private execution state. Existing clearing of an enclosing execution's private state at a public execution boundary must preserve the contextual observer selection. Inheriting observation does not reuse an enclosing run's recorder, totals, or lifecycle.
Prepared Execution
RunPrepared selects contextual observation from the context supplied when the
handle is executed. A handle prepared in one scope and executed in another
participates in the execution scope, not the preparation scope.
The explicit RunRequest.Observer remains retained by a successfully prepared
handle as today and composes with the execution-time contextual observers.
Preparation itself emits no execution events. Discarded handles and rejected
execution claims retain their existing observation behavior.
Coordinated Execution
Continuation runs receive contextual observation through their ordinary Run
boundary. CoordinateRequest.Observer continues to select continuation runs
only. Additional calls made by application callbacks participate when they use
the scoped context, without being automatically selected by the coordinator's
explicit observer.
Contextual observers are incorporated once per execution, rather than being re-added by coordination or another forwarding layer.
Duplicate Registration And Accounting
Registering the same callback through multiple explicit or contextual paths may deliver the same event more than once. Do not attempt to identify or deduplicate arbitrary Go function values.
Retain one execution lifecycle per run and reuse ExecutionGroup's existing
terminal-event deduplication. Selecting the same group through multiple paths
must not inflate its totals. Terminal usage is an aggregate, not an additional
contribution to be added to attempt usage.
Accounting And Responsibility Boundaries
Reuse existing events, usage presence/completeness, and group summaries. Available usage remains observable after an error or cancellation under the existing execution-observation behavior; this does not promise that upstream providers report usage for every attempt.
PromptKit owns observer selection, event generation, and aggregation. CommandKit and other consumers own invocation lifecycle, CLI flags, presentation, output delivery, and interpretation. This feature adds no CommandKit dependency or application-specific concepts to PromptKit.
Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation by this change. Additional usage fields such as reasoning tokens and monetary cost, pricing estimates, billing reconciliation, persistent history, and logging infrastructure are outside this scope.
Work Needed And Completion Criteria
- Add the context helper and connect observer selection to ordinary and prepared execution, keeping contextual selection separate from private run state.
- Preserve existing explicit selection and coordinator behavior while applying the composition and lifetime rules above. Keep existing execution behavior unchanged when no contextual observer is selected.
- Cover ordinary and prepared execution, coordinated continuations, nested calls using derived contexts, concurrent calls, multiple engines sharing a scope, and independent scopes on a reused engine.
- Cover callback ordering, nested contextual registration, nil registration, duplicate group enrollment, prepared-handle scope selection, and existing preparation/discard/rejected-claim behavior.
- Verify aggregate accounting through retries and repairs, available usage after failures or cancellation, and known zero versus unknown or incomplete counts. Use offline fixtures and injected clients; live inference is unnecessary.
- Document the exact helper contract in GoDoc and add a focused invocation-level aggregation example to the consumer observation guide. Update internal observation documentation where its selection flow changes.
This roadmap defines the target behavior. A separate implementation plan can divide the work into appropriately sized stages.