# Invocation-Scoped Execution Observation Status: agreed direction; vLLM scope under discussion; implementation pending. ## Purpose Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically. Include optional provider-reported cost alongside token usage, with built-in cost support initially limited to OpenRouter. Extend OpenRouter token reporting with reasoning counts, documented cache reads and writes, and a derived token cache-hit ratio. The motivating consumer is CommandKit's optional `ckprompt` integration, which could use this facility to report token usage for an entire command or pipeline. PromptKit supplies application-neutral observation and aggregation; consumers choose how to present or use the results. ## Current Baseline PromptKit v0.24.0 already provides: - [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous metadata callbacks and ordered composition. - [ExecutionGroup](../../execution_group.go) for concurrent collection, detached snapshots, and aggregation of the first terminal event for each observed run. - Presence-aware prompt, completion, total, cached, and cache-write token counts, including aggregate accounting across retries and repairs. - Explicit selection through `RunRequest.Observer`, retained for prepared execution, and [CoordinateRequest.Observer](../../coordinate.go) for coordinator-selected continuation runs. The missing capabilities are contextual observer selection, provider-reported cost accounting, reasoning-token counts, and a presence-aware cache-hit helper. OpenRouter cache writes also need their documented nested response mapping; the current client reads a top-level `usage.cache_write_tokens` field instead. Applications currently have to enroll each request explicitly, and PromptKit does not expose provider charges. The existing [consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) describes that workflow. ## Target Public API Add a public context helper: ```go func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context ``` The returned context associates the observer with executions using that context or derived contexts that retain its values. It composes with any observer already associated with the context; it does not replace the inherited observer. A nil observer leaves the inherited selection unchanged. No engine option, process-wide default, or replacement group type is needed. Extend the existing token types and result, event, and group surfaces with the token and cost metadata described below, plus a derived cache-hit helper. The intended consumer workflow is: ```go var group promptkit.ExecutionGroup ctx = promptkit.WithExecutionObserver(ctx, group.Observe) err := runApplicationPipeline(ctx) // After all application-owned work has finished, including on failure: summary := group.Snapshot() ``` The consumer owns context propagation, waiting for its work, and using the snapshot and error. Calls using unrelated contexts are not automatically enrolled. Group coverage continues to describe observed members, not future work or overall application progress. ## Observer Selection And Delivery ### Composition Preserve existing explicit observers. Deliver each event to the explicit request observer first, then any coordinator observer already composed with it, then contextual observers in registration order, from inherited to newly added. Preserve the existing synchronous callback and panic behavior. Concurrent executions may invoke a shared observer concurrently; consumers remain responsible for synchronizing their callback state. `ExecutionGroup` already supports concurrent use. ### Ordinary And Nested Execution `Run` selects contextual observation from its invocation context. Each execution has its own observation identity, recorder, and accounting, including a nested execution using a context passed to an application callback or injected client. Inherited observer selection is distinct from private execution state. Existing clearing of an enclosing execution's private state at a public execution boundary must preserve the contextual observer selection. Inheriting observation does not reuse an enclosing run's recorder, totals, or lifecycle. ### Prepared Execution `RunPrepared` selects contextual observation from the context supplied when the handle is executed. A handle prepared in one scope and executed in another participates in the execution scope, not the preparation scope. The explicit `RunRequest.Observer` remains retained by a successfully prepared handle as today and composes with the execution-time contextual observers. Preparation itself emits no execution events. Discarded handles and rejected execution claims retain their existing observation behavior. ### Coordinated Execution Continuation runs receive contextual observation through their ordinary `Run` boundary. `CoordinateRequest.Observer` continues to select continuation runs only. Additional calls made by application callbacks participate when they use the scoped context, without being automatically selected by the coordinator's explicit observer. Contextual observers are incorporated once per execution, rather than being re-added by coordination or another forwarding layer. ### Duplicate Registration And Accounting Registering the same callback through multiple explicit or contextual paths may deliver the same event more than once. Do not attempt to identify or deduplicate arbitrary Go function values. Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing terminal-event deduplication. Selecting the same group through multiple paths must not inflate its totals. Terminal usage is an aggregate, not an additional contribution to be added to attempt usage. These rules apply to cost as well as token counts. ## Backend Provider Selection Add an optional backend `provider` field identifying the API implementation whose conventions PromptKit should use. It identifies the service contacted, not the organization that trained the selected model. Backend IDs remain resource identities; names, namespaces, and endpoint hostnames do not select provider behavior. ```yaml promptkit: backends: - id: research-router provider: openrouter endpoint: https://openrouter.ai/api/v1 api_key_env: OPENROUTER_API_KEY ``` Normalize the final effective provider value by trimming whitespace and lowercasing. Omitted, null, empty, and whitespace-only values are equivalent. Reject an unsupported nonempty normalized value as an ordinary configuration error rather than silently using generic behavior. The agreed provider selections are `openai-compatible` and `openrouter`; the possible `vllm` addition remains under discussion. Omitted/null/blank provider values select the same effective behavior as explicit `provider: openai-compatible`. This is a runtime fallback, not an explicitly sourced configuration value. Backend inspection retains the distinction between an explicit provider declaration and an omitted/null/blank declaration. The `openrouter` provider reuses the shared OpenAI-compatible foundation and adds the OpenRouter response-metadata and diagnostic conventions below. Before finalizing this roadmap, discuss the conventions and scope of a `vllm` selection; this document does not yet select a vLLM contract. Resolve the field through ordinary ConfigKit composition and inheritance. Carry the selected value through configuration capture and prepared execution, expose it in backend inspection, and declare it in the bundled OpenRouter backend. It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a provider does not choose a model or alter the configured endpoint. An endpoint override retains the provider selection. ### Default OpenAI-Compatible Provider `openai-compatible` names PromptKit's current supported text Chat Completions conventions. Preserve the existing shared behavior documented in the [integration contract](../integrations/openai-compatible-chat.md), including: - JSON POST requests to `/chat/completions` relative to the configured base; - existing optional bearer-credential handling, including explicitly required credential settings when selected; - text messages, explicitly selected generation controls and structured-output requests, and configured extra parameters; - first-choice text extraction, existing supported token counts and their presence-aware observation; and - existing response validation, error handling, timeouts, cancellation, and transient retries. This selection supplies no default endpoint, model, API-key environment variable, or sampling settings. It does not interpret monetary cost or provide a default authentication-probe adapter. Without an explicit authentication adapter, report the existing unsupported diagnostic outcome rather than assuming authentication is inapplicable. Existing reachability and inference diagnostics and explicit authentication adapters remain available. "Lowest common denominator" identifies the shared protocol foundation; it does not remove existing optional features or promise every backend supports them. Continue sending caller-selected supported settings and return backend errors normally, without capability negotiation or fallback requests. This is the existing text Chat Completions scope, not a claim to support the full OpenAI API. ### Smoke-Test Defaults And Explicit Overrides When an authentication smoke test is explicitly requested, the selected provider supplies its default probe convention. `provider: openrouter` supplies the existing `openrouter_key` convention. Preserve explicit `smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint` settings as overrides: - A supplied kind selects the diagnostic adapter instead of the provider default. `kind: none` continues to declare authentication testing inapplicable. - An omitted/null kind uses the provider default, if available. This permits an endpoint override without repeating the provider's default kind. - A supplied endpoint overrides the selected adapter's default destination. An omitted/null endpoint uses that adapter's existing endpoint fallback. An absent/null authentication block likewise leaves selection to the provider default. Clearing the block removes its explicit overrides; `kind: none` is the explicit way to declare authentication inapplicable despite a provider default. Without a provider default or explicit adapter, retain existing unsupported diagnostic behavior. Existing explicit adapters continue to work without a provider declaration. Reachability and inference-default settings keep their existing behavior. These are conventions for explicitly requested diagnostics. Loading a backend, constructing an engine, and ordinary generation do not initiate probes. Resolve provider-supplied diagnostic defaults at the provider boundary without writing them into the loaded configuration or recording them as explicit source values. Configuration inspection preserves the distinction between configured overrides and provider-supplied fallbacks, including their existing source provenance. ### Diagnostic Migration This changes the current authentication metadata contract: an authentication object currently requires `kind`, and clearing the object removes its configured adapter. Under this roadmap, endpoint-only overrides become valid, and clearing the object restores any selected provider's default. `kind: none` suppresses that default. Explain these changes in the framework format documentation and release notes, alongside the new provider field. ### Provider Behavior Ownership Centralize provider-specific behavior within `internal/llm`, which already owns OpenAI-compatible transport and OpenRouter authentication-response recognition. Keep common request construction, credentials, HTTP execution, framing, and common response fields shared. Provider-specific code owns additional response metadata interpretation and authentication-probe conventions. Use one provider-selection mechanism inside that package. Generation, observation, and smoke-test coordination consume provider-neutral facts or invoke generic diagnostic operations rather than repeating provider dispatch and response interpretation. Explicit diagnostic adapter selection remains supported at this boundary. Update the current OpenRouter-specific diagnostic collaborator method to a generic operation as part of this consolidation. Provider-specific files and straightforward dispatch are sufficient initially. No new `internal/providers` package, registration framework, or broad interface requiring every provider to implement every capability is needed. This is configurable selection among PromptKit-supported provider implementations, not a public API for downstream registration of additional implementations. ## OpenRouter Token And Cache Reporting ### Response Mapping Use OpenRouter's documented Chat Completions usage fields. Its [response schema](https://openrouter.ai/docs/api_reference/overview), [usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting), and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching) define the following mapping, relative to `usage`: | Quantity | Response field | PromptKit field | | --- | --- | --- | | Input tokens | `prompt_tokens` | Existing `PromptTokens` | | Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` | | Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` | | Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` | | Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` | | Total tokens | `total_tokens` | Existing `TotalTokens` | Preserve `CachedTokens` as the public field name and clarify its meaning as cache-read tokens. Correct the OpenRouter cache-write mapping to the nested field above; retain the generic provider's existing top-level handling. Do not add the nested and top-level values together. OpenRouter uses its documented nested value; an omitted nested value remains unknown. Add `ReasoningTokens` to the public/domain token values, presence indicators, and observed token usage, including conversions, result serialization, attempt and terminal events, and group summaries. Initially, OpenRouter supplies this additional built-in response interpretation; the generic provider preserves its existing behavior. Injected clients may supply the canonical count and its presence through the existing model-client boundary. ### Count Relationships And Availability Reasoning tokens are included in completion tokens, as explained in OpenRouter's [reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens). Cache reads and writes describe input-token breakdowns. Present these as subsets or details rather than additional consumption to add to input/output totals. Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it. Consumers can derive non-reasoning output as completion tokens minus reasoning tokens when both counts are available and consistent. No additional stored non-reasoning count is needed in this scope. Capturing reasoning-token counts does not capture reasoning text or change reasoning request settings. Reuse existing per-field presence and aggregation rules: omitted/null means unknown, explicit zero means known zero, and aggregate `Known` and `Complete` flags describe the reported contributions across dispatched attempts. Include initial generations, retries, and repairs through the existing observation accounting, retaining terminal-only group aggregation and deduplication. Preserve the existing distinction between integer-only result usage and presence-aware observations; zero in `TokenUsage` alone does not establish provider reporting. ### Token Cache-Hit Ratio Add this derived helper on the existing presence-aware usage type: ```go func (u ObservedTokenUsage) CacheHitRatio() (float64, bool) ``` The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero to one. It measures the proportion of input tokens served from cache, not the proportion of requests with a cache hit or monetary savings. The Boolean reports whether a ratio is available; an unavailable result returns zero and false. Return an available ratio only when both counts are known and complete, input tokens are positive, and cache reads are between zero and input tokens inclusive. Zero cache reads with known positive input yields an available zero ratio. Missing or incomplete counts, zero input, or inconsistent counts leave it unavailable; preserve the raw reported counts rather than clamping them or failing generation. Cache-write availability does not affect this calculation. For a run or group, divide the aggregated cache-read count by the aggregated input count; do not average per-attempt or per-run percentages. Do not use `reads / (reads + writes)`, because cache writes need not account for all uncached input. The same helper works on attempts, terminal events, and group summaries without storing another independently maintained field. The ratio describes the included observations. A group ratio does not supersede `TerminalCoverageComplete` or imply that all application work has finished. For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts are known and complete, even if cache writes are unknown. Output 840 with reasoning 600 still represents 840 output tokens, not 1,440. ## Optional Provider-Reported Cost ### Initial Provider Support Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter defines as the amount charged to the account, denominated in USD. See its [usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting) and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/). Do not add `cost_details` breakdowns or combine upstream inference cost with the top-level charge; those describe different facts and could double-count cost. For the built-in client, select this interpretation through `provider: openrouter`, regardless of backend identity or namespace. Do not infer USD semantics from an arbitrary backend's field named `cost`, or add endpoint discovery or verification. Built-in cost interpretation is an extension unique to `openrouter` initially; `openai-compatible` leaves cost unknown even if a response contains `usage.cost`. No separate cost-reporting selector is needed. Aggregation remains provider-neutral after the model-client boundary supplies interpreted USD facts. Use the cost included in the normal completion response. No additional usage lookup, pricing request, or opt-in provider request parameter is needed. ### Monetary Representation Use `float64` amounts in USD. Make the unit explicit in public field names such as `CostUSD`, rather than introducing a generic amount with an implicit currency. At the individual model-client response boundary, use an optional amount (`*float64`): nil means unknown, and a pointer to zero means reported zero. Injected clients may supply the same optional USD value without needing an OpenRouter backend; no provider-specific interface is needed. Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields remain token counts. For run results, execution events, and group summaries, use a small presence-aware cost value with `Value float64`, `Known bool`, and `Complete bool`, following the existing observed-count conventions. Successful run results should expose cost even when no observer was registered; observers also provide per-attempt and terminal cost, including available facts on failure. Binary floating point is sufficient for operational telemetry and keeps decoding, addition, and the public API lightweight. Do not round to cents or another fixed scale during capture or aggregation; consumers choose display precision. Decimal strings or a decimal arithmetic dependency would be appropriate for exact financial ledgers, while fixed-scale integers would require a selected precision and rounding policy. Neither is warranted for this reporting feature. The API does not promise exact decimal arithmetic or invoice reconciliation. ### Presence, Aggregation, And Failures - An absent or null cost is unknown. Explicit numeric zero is known zero. A backend that reports no cost remains unknown even when a consumer knows it has no external charge. - On a dispatched attempt, a usable reported cost is known and complete for that attempt. Cost completeness is independent of token-field completeness. - Sum known costs across dispatched attempts, including initial generations, retries, and repairs. `Known` means at least one attempt reported cost; `Complete` means every dispatched attempt did. Unknown values contribute no amount, but make the total incomplete. With no dispatched attempts, follow the existing known, complete zero convention. - Group summaries reuse existing terminal-only aggregation and deduplication: sum known terminal values, OR their `Known` flags, and AND their `Complete` flags. Retain existing empty-group and terminal-coverage semantics. A subtotal from OpenRouter mixed with unreported local-backend cost is incomplete. - Keep valid cost observed from a decoded successful envelope when a later choice/content check fails, as with token usage. Invalid framing and non-2xx error bodies continue to supply no usage. Cancellation or failure may leave incomplete reporting even when billable upstream work occurred. - Treat an unusable optional cost value, including a nonnumeric value or one not representable as a finite `float64`, as unknown rather than making an otherwise valid generation fail. This does not relax existing response framing or content validation. An injected client likewise cannot supply a nonfinite usable cost. Completeness describes coverage of reported values, not verification of the provider's billing. Preserve reported usable amounts without pricing inference or reconciliation. For a successful run, result cost and terminal-event cost describe the same dispatched attempts and must agree on value, presence, and completeness. Result cost must also be the same with observation enabled or disabled. For example, an unreported failed attempt followed by a successful attempt reporting USD 0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same accounting through repairs and ordinary or prepared execution. Group cost completeness concerns the terminal observations included in its subtotal. If two completed runs reported all their costs but a third observed run has no terminal event, the subtotal may have `Complete: true` while `TerminalCoverageComplete` is false. Preserve this distinction using the existing coverage field; no additional completion flag or group lifecycle is needed. ## Accounting And Responsibility Boundaries Reuse existing events, token presence/completeness, and group summaries, extending them with the cost facts above. Available usage remains observable after an error or cancellation under the existing execution-observation behavior; this does not promise that upstream providers report usage for every attempt. PromptKit owns observer selection, event generation, and aggregation. CommandKit and other consumers own invocation lifecycle, CLI flags, presentation, output delivery, and interpretation. This feature adds no CommandKit dependency or application-specific concepts to PromptKit. Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation by this change. Reasoning-text capture, changes to reasoning or caching request behavior, additional built-in cost providers, multiple currencies, currency conversion, pricing estimates, billing reconciliation, persistent history, and logging infrastructure are outside this scope. ## Work Needed And Completion Criteria - Add the context helper and connect observer selection to ordinary and prepared execution, keeping contextual selection separate from private run state. - Preserve existing explicit selection and coordinator behavior while applying the composition and lifetime rules above. Preserve existing observer selection when no contextual observer is selected. - Add `provider` to the backend configuration model, typed source helpers, capture, inspection, and execution plumbing. Update the bundled OpenRouter backend to select it explicitly. Retain the selection through prepared runs. - Consolidate provider-specific response and authentication-probe behavior in `internal/llm`; keep coordination and accounting provider-neutral. Cover provider-default probes, explicit kind and endpoint overrides, `kind: none`, and existing explicit probes without a provider selection. - Verify provider selection independently of backend names and namespaces, including renamed and inherited backends, equivalent omitted/null/blank values, trimming and case normalization, rejection of unsupported nonempty values, and retention through endpoint overrides. Confirm provider metadata is not serialized into the completion request body. Verify that explicit `openai-compatible` and omitted/null/blank selection preserve the same existing shared behavior without manufacturing explicit configuration provenance. - Verify that provider diagnostic defaults do not become explicit values in the loaded configuration or replace source provenance in inspection. Cover the changed clearing behavior and endpoint-only authentication overrides. - Carry OpenRouter reasoning-token counts through token values, presence, results, events, and group summaries. Verify nested OpenRouter cache-write decoding, including omitted/null/zero values and the nested value taking precedence when a top-level value is also present. Preserve generic-provider token mappings and existing field meanings. - Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and incomplete counts, zero input, inconsistent counts, independence from cache-write availability, and token-weighted aggregate ratios. Verify subset accounting without adding reasoning or cached tokens to the reported totals. - Carry optional OpenRouter cost through response decoding, internal/public conversions, successful run results, attempt and terminal events, and existing group summaries. Keep result cost available without registering an observer. Preserve existing token fields and accounting. - Verify that successful-result cost agrees with terminal-event cost and is unchanged when observation is disabled, including the unreported-retry example above. Verify group snapshots with fully reported completed runs alongside a member without a terminal observation. - Cover ordinary and prepared execution, coordinated continuations, nested calls using derived contexts, concurrent calls, multiple engines sharing a scope, and independent scopes on a reused engine. - Cover callback ordering, nested contextual registration, nil registration, duplicate group enrollment, prepared-handle scope selection, and existing preparation/discard/rejected-claim behavior. - Verify aggregate accounting through retries and repairs, available usage after failures or cancellation, and known zero versus unknown or incomplete counts. Use offline fixtures and injected clients; live inference is unnecessary. - Cover fractional USD values, explicit zero, omitted/null and unusable cost, generic built-in selection, optional injected-client cost, mixed known and unknown totals, repair/retry aggregation, and duplicate group registration. Use tolerant comparisons where floating-point addition requires them; do not turn telemetry into an exact-decimal guarantee. - Document the exact helper contract in GoDoc and add a focused invocation-level aggregation example to the consumer observation guide, including cost availability, units, and completeness. Document the reasoning/cache fields and cache-hit helper in their owning GoDoc and illustrate input/output breakdowns without double-counting. Document OpenRouter token and cost decoding in the [integration contract](../integrations/openai-compatible-chat.md#response-handling), and update internal observation and model-client documentation for changed flows. Document `provider`, its smoke-test defaults and override semantics in the framework format reference and relevant public GoDoc; update inspection and diagnostic documentation with the implemented behavior. Include the diagnostic migration in release notes. ## Implementation Planning Guidance Organize the eventual implementation plan around two workstreams: 1. Contextual observer selection, including ordinary, prepared, coordinated, and nested execution boundaries and the existing group collector. 2. Backend provider selection and consolidation within `internal/llm`, including smoke-test defaults and explicit overrides, followed by OpenRouter token extensions and provider-reported cost through decoding, accounting, public results, events, and group summaries, plus the derived cache-hit helper. Include the corresponding configuration and API documentation. The second area is more extensive than the context helper. Keep backend provider configuration, diagnostic consolidation, token/cache reporting, and cost accounting in separately reviewable implementation units rather than one large implementation prompt. Complete the remaining provider discussion before preparing that plan; avoid coupling delivery of contextual observation to provider cost support. Today, successful results accumulate generation responses in the runner, while optional observation recorders account for dispatched attempts. Adding cost to response objects alone does not preserve completeness across unreported retries. Reuse a small accounting component for result and observation paths so they apply the same rules, including when no observer is selected. Preserve existing token accounting rather than expanding this work into a redesign of its public contract. Keep tolerant optional-cost decoding local to the model-client boundary. Adding a plain `*float64` JSON field to the typed envelope would make a nonnumeric or overflowing value fail decoding of the entire response. Decode the optional field separately so an unusable cost becomes unknown without changing existing framing or content validation. This needs no general numeric-conversion layer. Use the completion criteria above to select focused offline checks for each unit. This roadmap owns target behavior; a separate implementation plan owns the detailed work sequence. ## Topics For Further Discussion Before finalizing the roadmap, discuss a `vllm` provider with vLLM-specific conventions, identifying the response and diagnostic behavior to support from reviewed upstream contracts and fixtures. The `openai-compatible` default, normalization, unsupported-value handling, override precedence, and configuration-inspection boundaries above are agreed decisions. The vLLM discussion does not yet add provider behavior or an implementation commitment.