27 KiB
Invocation-Scoped Execution Observation
Status: accepted scope; implementation complete. Current behavior is documented in the linked canonical contracts; roadmap retirement is separate housekeeping.
Purpose And Scope
Let consumers select observation once for an application-owned operation and obtain usage across its nested, concurrent, coordinated, and multi-engine PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a token cache-hit ratio, and optional USD cost with its source and completeness.
CommandKit's optional ckprompt integration motivates this work, but the API
remains application-neutral. PromptKit owns observation, provider response
interpretation, and aggregation. Consumers own context propagation, waiting for
their work, invocation lifecycle, presentation, and interpretation.
This document defines the target state and the changes required to reach it. A separate implementation plan will own sequencing and prompt-sized stages.
Current Baseline
PromptKit v0.24.0 already provides execution observers and ordered composition, a concurrency-safe execution group, and request-level observation. Groups aggregate the first terminal event for each observed run. Existing observation distinguishes known, unknown, and incomplete prompt, completion, total, cached, and cache-write token counts across retries and repairs.
Today, consumers explicitly enroll requests. Prepared handles retain the request
observer; coordinator observers select continuation runs.
There is no contextual selection, reasoning-token count, cache-hit helper, or
monetary cost reporting. OpenRouter cache writes also need their documented
nested mapping: the current client reads usage.cache_write_tokens instead.
Contextual Observer Selection
Add the public helper:
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
The returned context composes the supplied observer with any inherited contextual observers. A nil observer leaves that selection unchanged. Executions using the context, or derived contexts retaining its values, participate automatically; unrelated contexts do not. No engine option or process-wide default is needed.
A consumer can create an invocation-owned group, associate group.Observe with
the context, run its pipeline, and snapshot the group after its owned work
finishes, including when the pipeline returns an error. Group coverage describes
observed members, not future work or overall application progress.
Delivery And Execution Boundaries
Runselects contextual observers from its invocation context.RunPreparedselects them from its execution context, not the context used to prepare the handle. The explicitly supplied request observer remains retained by the handle and composes with those execution-time observers.- Preparation emits no execution events. Discarded handles and rejected execution claims retain their existing observation behavior.
- Coordinated continuations acquire contextual observation through their
ordinary
Runboundary.CoordinateRequest.Observerstill selects continuation runs only; additional calls made by application callbacks participate when they use the scoped context. - Every execution, including a nested one using a callback or injected-client context, has its own identity, recorder, and totals. Clearing an enclosing execution's private state at a public boundary preserves contextual observer selection, not the enclosing recorder.
Deliver events to explicit request observers first, then any coordinator observer already composed with them, then contextual observers from inherited to newly registered. Incorporate contextual selection once per execution, not again in forwarding or coordination layers.
Preserve synchronous callback delivery and panic behavior. Concurrent runs may invoke a shared observer concurrently; consumers synchronize their callback state. The existing group remains concurrency-safe.
Registering the same callback through multiple paths may deliver duplicate events; arbitrary Go functions are not deduplicated. Retain one lifecycle per execution and reuse the group's terminal-event deduplication. Terminal totals are aggregates, not additional contributions to add to attempt totals.
Backend Provider And Accounting Metadata
Add two optional backend fields:
| Field | Meaning |
|---|---|
provider |
Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. |
zero_cost |
Enables a configured-zero cost fallback when no usable reported cost is available. |
For example:
promptkit:
backends:
- id: research-router
provider: openrouter
endpoint: https://openrouter.ai/api/v1
api_key_env: OPENROUTER_API_KEY
- id: local-inference
provider: vllm
endpoint: http://localhost:8000/v1
zero_cost: true
Trim and lowercase the final effective provider value. Omitted, null, empty, and
whitespace-only values select openai-compatible. Reject unsupported nonempty
normalized values as ordinary configuration errors. The supported selections are
openai-compatible, openrouter, and vllm.
zero_cost is Boolean. Omitted, null, or false disables its fallback; true
enables it. It is independent of provider selection and does not suppress a
usable reported cost.
Resolve both fields through ordinary final ConfigKit composition and inheritance.
Carry their effective selections through capture and prepared execution without
reloading configuration. Declare provider: openrouter in the bundled
OpenRouter backend. Do not automatically declare any backend zero-cost.
These fields are local metadata, not outbound request parameters or entries
added to extra_params. Provider selection does not choose an endpoint or model,
and endpoint overrides retain the selected provider and accounting policy.
Inspection Boundary
InspectBackend shows the normalized/resolved declared provider and zero-cost
setting, preserving explicit false for zero_cost. Omitted/null declarations
may remain absent, and blank provider values need no extra history representation.
Do not manufacture an explicit openai-compatible declaration or false
zero_cost value from runtime defaults.
InspectConfiguration owns ConfigKit source history and explicit/null/omitted
distinctions. Resource inspection does not acquire source-history fields or
null-tracking machinery. Provider diagnostic defaults likewise remain runtime
fallbacks, not values written into loaded configuration or recorded as explicit
source declarations.
Provider Conventions
All selections reuse the existing
OpenAI-compatible text Chat Completions foundation:
base-relative /chat/completions JSON POST requests, existing credential rules,
text messages, selected generation controls and structured output, extra
parameters, first-choice content extraction, common token fields, and existing
response validation, errors, cancellation, timeouts, and transient retries.
| Provider | Additional response interpretation | Default authentication adapter |
|---|---|---|
openai-compatible |
Existing token mappings; no provider-reported monetary cost decoding. | None. |
openrouter |
OpenRouter reasoning/cache details and usage.cost in USD. |
openrouter_key. |
vllm |
vLLM reasoning/cache details; no provider-reported monetary cost decoding. | vllm_models. |
The default provider names PromptKit's current supported conventions, not the entire OpenAI API. "Lowest common denominator" does not strip existing optional features or promise every backend supports them. Continue sending selected supported settings and return backend errors normally.
Provider selection supplies no default endpoint, model, credential environment variable, or sampling settings, and does not create backend definitions. Keep vLLM reasoning, chat-template, and other model-specific controls caller-selected; do not automatically enable thinking or translate those controls. No new generation protocol, capability negotiation, or version discovery is needed. vLLM documents its OpenAI-compatible interface.
Authentication Diagnostics
Provider defaults apply only to explicitly requested authentication diagnostics. Loading configuration, constructing an engine, and ordinary generation do not initiate probes. Existing reachability and inference-default behavior remains.
Explicit smoke_tests.authentication metadata overrides provider defaults:
- A supplied
kindselects the adapter;nonedeclares authentication testing inapplicable. - Omitted/null
kinduses the provider default, allowing endpoint-only overrides. - A supplied endpoint overrides the selected adapter's destination; omitted/null uses that adapter's existing fallback.
- An absent/null authentication block removes its overrides and leaves provider
defaults applicable.
kind: none, not clearing the block, suppresses them.
Without a provider default or explicit adapter, preserve the existing unsupported diagnostic outcome rather than assuming authentication is inapplicable. Explicit adapters remain selectable without a matching provider declaration.
Retain openrouter_key behavior. The new vllm_models adapter makes one GET to
/models relative to the configured API base, supplying an available credential
under existing rules and allowing dispatch without one. A base ending in /v1
therefore produces /v1/models; explicit diagnostic endpoint overrides apply.
For vllm_models, HTTP 200 with a valid JSON object containing object: "list"
and array-valued data means acceptance of the probe request. An empty array is
accepted; recognition does not inspect model capabilities or require a particular
model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown.
Reuse existing bounded diagnostic decoding and framing. Report observed
credential presence and response facts without claiming authentication is
enforced or models-list acceptance establishes inference authorization.
vLLM documents /v1/models as protected when API-key authentication is configured;
/health is not an authentication check. See its
authentication documentation.
No health-monitoring facility is added here.
Token And Cache Reporting
Provider Response Mapping
Use common usage.prompt_tokens, usage.completion_tokens, and
usage.total_tokens for PromptTokens, CompletionTokens, and TotalTokens.
Normalize these additional provider details, relative to usage:
| Public field | OpenRouter | vLLM |
|---|---|---|
New ReasoningTokens |
completion_tokens_details.reasoning_tokens |
completion_tokens_details.reasoning_tokens |
Existing CachedTokens |
prompt_tokens_details.cached_tokens |
prompt_tokens_details.cached_tokens |
Existing CacheWriteTokens |
prompt_tokens_details.cache_write_tokens |
prompt_tokens_details.created_cache_tokens |
OpenRouter documents these in its
response schema,
usage-accounting guide,
and caching guide.
vLLM's response construction
uses created_cache_tokens; its
output definitions
describe local prefix-cache writes, not a billable write charge.
Preserve CachedTokens as the public name and clarify its meaning as cache reads.
Use each provider's documented cache-write field, not a sum or guessed alias;
a missing nested value is unknown even if an alternative field is present.
Retain the generic provider's existing mappings, including its top-level
usage.cache_write_tokens handling.
Add ReasoningTokens to public/domain token values, presence indicators, observed
usage, conversions, and result serialization, and carry it through attempt events,
terminal events, and group summaries. Injected clients may supply canonical
counts and presence through the existing model-client boundary, retaining its
legacy UsagePresence fallback; explicit presence distinguishes measured zero.
Availability And Optional Metadata
Omitted/null details are unknown, not zero. Explicit integer zero is known zero. For the newly interpreted provider-specific reasoning and cache-write details, wrong-type, nonintegral, or unrepresentable integer values are also unknown. An unusable new detail does not invalidate otherwise usable content or erase independently available usage. Preserve existing common-field decoding and response framing/content validation; this is a targeted optional-metadata rule, not a general relaxation of the response contract. Keep this interpretation local to the model-client boundary without a general numeric-conversion framework.
vLLM detail availability depends on version and configuration: prompt details
currently require --enable-prompt-tokens-details, and reasoning details depend
on a configured reasoning parser. Document those reporting settings without
configuring or discovering them automatically. Older or differently configured
servers may omit details and still produce usable responses. Do not estimate
reasoning counts from text or cache writes from input minus cache reads.
Count Relationships And Aggregation
Reasoning is included in completion tokens, not additional output to add to them. OpenRouter explains this in its reasoning documentation; vLLM similarly includes reasoning detail within generated-token usage. Cache reads and writes are input-token details, not extra input consumption. Preserve the reported total without recomputation or reconciliation.
Consumers can derive non-reasoning output from consistent, known completion and reasoning counts; no additional stored count is needed. Capturing counts does not capture reasoning text or change generation/caching settings.
Reuse existing per-field token accounting: aggregates sum known contributions,
Known identifies available reporting, and Complete describes its coverage
across dispatched attempts. Preserve initial-generation, retry, and repair
observation accounting and terminal-only group deduplication. Existing integer
result usage remains distinct from presence-aware observations; a zero result
field alone does not establish that the provider reported it.
Token Cache-Hit Ratio
Add a derived helper, not another stored total:
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
Compute CachedTokens / PromptTokens as a fraction from zero to one. This is the
proportion of input tokens served from cache, not request hit frequency or cost
savings. For aggregates, divide summed reads by summed input; do not average
individual percentages or use reads / (reads + writes).
Return an available ratio only when input and cache-read counts are known and complete, input is positive, and reads lie between zero and input inclusive. Known zero reads with positive input yields an available zero. Missing, incomplete, zero-denominator, or inconsistent data yields zero and false while preserving raw counts; it does not fail generation or clamp reported values. Cache-write availability is irrelevant.
The same helper serves attempts, terminals, and group summaries. A group ratio describes its included observations and does not supersede terminal coverage or imply all application work has finished.
Cost Reporting
Reported Cost And Configured-Zero Precedence
Initially, only the OpenRouter built-in provider interprets usage.cost as the
account charge in USD. See its
usage-accounting reference
and USD explanation.
Do not add cost_details amounts to that charge or interpret arbitrary providers'
fields named cost. The generic and vLLM providers do not decode monetary cost.
Use the cost included in the ordinary response; no extra request or usage lookup
is needed. At the model-client boundary, represent an optional USD amount as
*float64: nil means unknown; a pointer to zero means reported zero. Injected
clients may supply the same value regardless of provider selection.
Resolve cost at completion of each dispatched attempt, outside provider-specific decoding and whether or not decoding occurred or succeeded:
- A usable provider/model-client-reported cost takes precedence, including zero.
- Otherwise,
zero_cost: trueon the effective backend supplies configured USD zero. - Otherwise, cost remains unknown.
This applies to built-in and injected clients, including failed or canceled dispatched attempts and undecodable responses. Without a selected backend declaration, no configured-zero fallback applies. Non-dispatched work contributes no cost. Do not infer zero cost from local hosting, provider name, credentials, or token counts, or install the declaration automatically on bundled backends.
An unusable reported amount, including a nonnumeric value or one not representable
as a finite float64, is unknown before fallback; it does not make an otherwise
usable generation fail. Keep available cost from a decoded successful envelope
when a later choice/content check fails. Invalid framing and non-2xx bodies supply
no provider usage, although the configured-zero fallback still applies to a
dispatched attempt. Existing framing, content, and error handling remain intact.
Monetary Representation And Sources
Expose cost separately from token counts and their presence indicators. Use
explicit USD field naming such as CostUSD and a small public cost value for
results, events, and group summaries with:
Value float64: the available USD amount or subtotal;Known bool: availability;Complete bool: coverage under the selected cost policy; andSource: the contribution-source discriminator below.
| Source | Meaning |
|---|---|
provider |
Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. |
configured_zero |
Supplied by the explicit backend fallback. |
mixed |
An aggregate includes both reported and configured-zero contributions. |
| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. |
Use binary floating point for operational telemetry, without rounding to cents or another fixed scale during capture or aggregation. Consumers choose display precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency rounding, pricing inference, and invoice reconciliation are not part of this work.
Attempt, Run, And Group Accounting
A dispatched attempt with usable reported cost or configured zero is known and
complete for cost, independently of token completeness. Sum known amounts across
all dispatched attempts, including retries and repairs. Aggregate Known means
at least one known contribution; Complete means every dispatched attempt has a
cost from either source. Unknown contributions add no amount and make coverage
incomplete. Completeness does not assert that every provider reported a charge.
Aggregate sources by union of known contributions, including zero amounts.
Unknown contributions add no source: a provider-only subtotal can be incomplete
without becoming mixed. A reported zero plus a configured zero is mixed.
Successful results expose cost even without an observer. Result and terminal cost agree on amount, availability, completeness, and source, and enabling observation does not change result cost. For example, an unreported failed attempt followed by a reported USD 0.002 yields an incomplete provider-source subtotal when no zero-cost fallback applies.
Groups aggregate terminal costs only, counting each run's first terminal once.
Combine known amounts, OR Known, AND Complete, and union sources. Preserve
existing empty/no-terminal group semantics: a completed execution with no
dispatched attempts yields known, complete zero with empty source; an empty group
does likewise, while a group with members but no terminals has unknown,
incomplete usage.
Cost completeness concerns included terminal observations, independently of
TerminalCoverageComplete. Two completed runs with full cost coverage can yield
a complete subtotal while a third observed run lacks a terminal. Conversely,
fully reported OpenRouter runs and explicitly zero-cost local runs can yield a
complete, mixed-source total. Neither case verifies a provider invoice or future
application progress.
Internal Responsibilities And Scope Limits
Provider-specific metadata interpretation and authentication conventions belong
in internal/llm, sharing common transport, request construction, credentials,
framing, and response fields. Use one provider-selection mechanism there.
Generation/observation coordination consumes normalized facts or generic
diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator
method with a generic operation while retaining explicit adapter selection.
Provider-specific files and straightforward dispatch suffice; no public provider
registration API, new internal/providers package, or broad mandatory-capability
interface is needed. This is configuration-driven selection among supported
implementations, not downstream registration of implementations.
Reuse a small shared cost-accounting component for result and observation paths so dispatched attempts, fallback, and source union follow the same rules even when observation is disabled. Provider code interprets wire metadata; common accounting owns fallback and aggregation. No ConfigKit provenance traversal belongs in ordinary execution.
Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation. CLI integration, output delivery, persistent history, logging infrastructure, reasoning-text capture, changes to reasoning/caching request behavior, server administration/metrics, additional built-in cost decoders, multiple currencies, configurable token rates, fixed nonzero request prices, and cost estimation remain outside scope.
Work Needed And Acceptance
The following areas describe required changes, not an execution order.
- Extend the public context API and ordinary/prepared execution wiring while preserving independent nested lifecycles, explicit observers, coordinator selection, and existing preparation/discard/rejected-claim behavior.
- Extend backend settings, typed helpers, final-value normalization/validation,
capture, prepared metadata, and inspection for
providerandzero_cost. Update the bundled OpenRouter declaration. Preserve configuration provenance and resource-inspection simplicity, and keep metadata out of request bodies. - Consolidate provider behavior in
internal/llm, adding vLLM usage normalization and the models-list diagnostic alongside OpenRouter interpretation and the shared default path. Preserve common decoding while tolerating unusable new optional details and cost locally to the model-client boundary. - Extend public/domain token types, presence, serialization, conversions, events, and groups with reasoning counts and correct provider-specific cache writes. Add the presence-aware cache-hit helper without changing existing token meanings or introducing a separate ratio accumulator.
- Carry optional reported cost and configured-zero policy through attempt completion and shared accounting into results, events, and groups, including source metadata and observer-independent result accounting.
Select focused offline checks under the testing policy. Use real internal collaborators and controlled HTTP/injected-client boundaries, with proportionate public contract coverage. Important behaviors include:
- Context inheritance, registration order, nil/duplicate enrollment, concurrent calls, multiple engines, independent scopes, nested calls, and execution-time selection for a handle prepared elsewhere.
- Provider selection independent of backend identity; normalization and unsupported value handling; omitted/null/blank defaults; inheritance, endpoint overrides, explicit false, and inspection without manufactured values or source history.
- Provider-default and explicit diagnostic selection, endpoint-only overrides,
clearing metadata versus
kind: none, vLLM acceptance including an empty list, rejection, unrecognized/framing-limited responses, and optional credentials. - Provider token mappings, absent/null/zero details, malformed new optional counts without generation failure, unchanged common-field handling, and responses representative of vLLM servers with detail reporting disabled.
- Cache-hit availability, inconsistent counts without generation failure, token-weighted aggregates, independence from cache writes, and no double-counting of reasoning or cache details.
- Reported cost precedence, configured-zero fallback on dispatched failure and cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources, zero-valued contributions, and independently incomplete token reporting.
- Result/terminal cost agreement with observation enabled or disabled, duplicate terminal deduplication, empty/no-terminal groups, and subtotal completeness distinct from terminal coverage.
Synthetic fixtures and offline examples suffice; live providers are not required. Keep ownership localized and avoid repeating the same assertions across layers.
Documentation And Migration
When implemented, exact exported contracts belong in GoDoc; the consumer guide should show invocation-scoped aggregation, reasoning/cache breakdowns without double-counting, and cost availability, source, and completeness. Document provider wire mappings, optional-metadata handling, and diagnostic response recognition in the integration contract. The framework format reference owns the new backend fields, default selection, inheritance, and diagnostic overrides. Update internal documentation for changed ownership and flows.
Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache creation meaning without promising availability on every deployment. Consumers own presentation. Models-list acceptance does not establish inference authorization, and complete cost metadata does not independently verify billing.
Release notes should call out that authentication objects no longer require
kind when a provider default supplies it, endpoint-only overrides are supported,
and clearing an authentication block restores provider defaults. kind: none
suppresses that default. Preserve existing explicit adapters without requiring a
provider declaration.