Document invocation observation and cost reporting

This commit is contained in:
2026-11-22 18:38:11 +00:00
parent d4509b163e
commit 20c5ea47af
22 changed files with 231 additions and 52 deletions

View File

@@ -728,6 +728,40 @@ counts, but its zero counts remain unknown to observations; supply explicit
presence when measured zeros matter. Existing `RunResult.Usage` sums returned
values independently of observation presence.
For an application-owned invocation spanning several calls or engines, attach
an observer once to a context and propagate it to those calls:
```go
var group promptkit.ExecutionGroup
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
result, err := engine.Run(ctx, request)
// After joining all application-owned work, inspect group.Snapshot(), even on error.
```
Explicit request observers run first, followed by contextual observers in
registration order. Coordinated continuations pick up the context through
`Run`; prepared handles retain the request observer but use the execution
context's observers, not the preparation context's. Each nested call has its
own run ID and accounting. Duplicate registration can deliver duplicate events;
groups count each run's first terminal once.
Reasoning tokens are a subset of completion tokens; cache reads and writes are
input details, not extra tokens. `ObservedTokenUsage.CacheHitRatio()` derives
the cache-read fraction only when input and read totals are known, complete,
positive-denominator, and consistent. It divides aggregate counts rather than
averaging per-run ratios and is independent of cache writes.
`RunResult.CostUSD` is available even without an observer; attempt and terminal
events and group snapshots also report cost. `Known` denotes an available
subtotal, `Complete` covers dispatched attempts, and `Source` distinguishes
provider reports, an explicitly configured zero fallback, and mixed sources.
Reported cost, including zero, takes precedence over fallback. Unknown failed
attempts make a subtotal incomplete; configured zero may cover them. Terminal
events can report cost when a failed operation returns no result. Neither
complete reporting nor provider attribution verifies an invoice; consumers
choose presentation precision. See [backend definitions](../formats.md#backend-definitions)
and the [integration contract](../integrations/openai-compatible-chat.md#response-handling).
Callbacks run synchronously, in order for one operation, and can overlap for
different operations. Synchronize shared callback state and avoid blocking on
work that needs the same run's admission slot. Callback panics propagate;
@@ -879,15 +913,15 @@ outcome, err := engine.Coordinate(ctx, promptkit.CoordinateRequest{
})
```
`assess` is application code and explicitly selects its observer on any nested
run. A completed turn releases its admission lease before the callback, so that
`assess` is application code and can select its observer explicitly or inherit
one from the context. A completed turn releases its admission lease before the callback, so that
assessment may use the same backend. `stopped` and `exhausted` describe control
flow, not content acceptance. Cancellation before callback delivery can return one
completed response in `Undelivered`; errors retain their ordinary identities.
### Collect Selected Execution Facts
Groups are independent of continuation and enrollment is explicit:
Groups are independent of continuation and can be enrolled explicitly:
```go
var group, validatorGroup promptkit.ExecutionGroup
@@ -904,7 +938,7 @@ _, _ = summary, validatorSummary
```
Only the first terminal event per run contributes counts and usage; coverage of
currently observed members is separate from token-usage completeness. Empty
currently observed members is separate from token and cost completeness. Empty
groups report complete, known zero totals, while members without terminals do
not invent totals. Overlapping groups are not generally additive. See the
[offline example](../../examples/go-library/observe/main.go) for complete setup.

View File

@@ -223,7 +223,7 @@ Backend definitions are supplied through caller-selected configuration
sources. The conventional backend IDs are names only; PromptKit does not
install definitions for them.
A backend definition may have `endpoint`, `api_key_env`,
A backend definition may have `provider`, `zero_cost`, `endpoint`, `api_key_env`,
`concurrency_limit`, `queue_capacity`, `extra_params`, and `smoke_tests`. The
final endpoint is required and must be an absolute HTTP(S) URL with a host, no
user information, query, or fragment. An omitted or null API-key environment
@@ -237,6 +237,17 @@ unrecognized backend field is application data and is not automatically redacted
by Promptkit. Applications own sensitivity declarations and disclosure policy
for their extension fields.
`provider` selects response and diagnostic conventions independently of the
backend ID, endpoint, or model. Its final value is trimmed and lowercased;
omitted, null, or blank selects `openai-compatible` at runtime. Supported
nonblank values are `openai-compatible`, `openrouter`, and `vllm`. `zero_cost`
is an optional Boolean: true supplies a zero-cost fallback when a dispatched
attempt has no usable reported cost; omitted, null, and false disable it.
Neither setting is sent to the provider. They follow normal layering and
inheritance, and endpoint overrides retain the selected backend's policy.
Resource inspection shows declared normalized values without inventing runtime
defaults; configuration inspection retains source history.
`smoke_tests` is optional backend metadata for explicit diagnostic operations.
It may contain `default_profile`, `reachability`, and `authentication`:
@@ -274,6 +285,10 @@ Without a selected adapter, authentication diagnostics report an unsupported
selection. An endpoint-only authentication
block overrides the destination of a provider default; `kind: none` suppresses
that default.
Previously, a supplied authentication object required `kind`. Endpoint-only
objects now use a provider default where available; clearing the object
restores that default rather than suppressing it. Use `kind: none` to suppress
authentication testing.
These values are configuration only: engine construction does not contact a
backend.
@@ -364,7 +379,7 @@ does not disable the caller context or HTTP transport timeout.
### Bundled OpenRouter Configuration
The default bundle installs root backend `openrouter`, endpoint
The default bundle installs root backend `openrouter`, provider `openrouter`, endpoint
`https://openrouter.ai/api/v1`, optional credential environment name
`OPENROUTER_API_KEY`, and smoke metadata selecting `gpt-luna-latest` with the
`openrouter_key` authentication adapter. Existing diagnostic endpoint fallback

View File

@@ -173,6 +173,10 @@ otherwise usable content. Existing common count decoding remains strict.
OpenRouter's `usage.cost` is an optional USD amount; malformed or non-finite
amounts are ignored without invalidating content. The generic and vLLM
providers do not interpret that field.
vLLM prompt-token details depend on server configuration, including
`--enable-prompt-tokens-details`; reasoning details depend on a configured
reasoning parser. Servers without these details remain usable. vLLM cache
writes describe local prefix-cache creation, not a billable write charge.
Each common token field retains whether the provider reported a
non-null integer, including zero. Omitted and `null` fields are unknown for

View File

@@ -79,7 +79,7 @@ other backend IDs.
The coordinator does not retry injected clients; it normalizes a nil successful
response as malformed generation. The built-in HTTP client remains responsible
for provider transport behavior. The runner, rather than the
capacity layer, sums all five usage fields from the initial response and every
capacity layer, sums usage fields from the initial response and every
completed repair response into the successful run result.
An invocation-local observer receives attempt events after active permit

View File

@@ -35,6 +35,12 @@ unknown usage. Transport, timeout, malformed, and noneligible status failures
stop the sequence without retry exhaustion. Context is checked before each
wait, acquisition, and dispatch.
At every dispatched completion the coordinator resolves any reported cost
against the frozen backend zero-cost policy and adds it once to the run's
[cost accumulator](observation.md), independently of observer selection.
Injected responses may retain reported cost alongside an error. Undispatched
acquisitions and retry waits contribute no cost.
The public client adapter translates explicitly supplied public transport and
malformed-response identities into internal categories while retaining the
original error chain. This keeps injected attempt classifications consistent

View File

@@ -142,6 +142,14 @@ known. Strictly decoded usage can still be observed if later choice/content
checks fail; invalid framing or a non-success error body contributes none.
Existing successful-result accounting remains value-based.
The captured provider selects additional response details and authentication
diagnostic defaults. Optional new reasoning, cache-write, and OpenRouter cost
fields are decoded independently, without relaxing common-field or framing
validation. Cost remains an optional raw amount at this boundary; shared
accounting applies configured-zero fallback after dispatch. The integration
contract specifies the [wire mappings](../integrations/openai-compatible-chat.md#response-handling)
and [probe recognition](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics).
For a non-success response, `ProviderHTTPError` retains the HTTP status and
only normalized detail from the bounded recognized envelope. It retains
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;

View File

@@ -21,7 +21,7 @@ contributor workflow and validation.
| `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
| `internal/domain` | Defines internal framework values for requests, immutable conversation exchanges, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, validation, and source-neutral smoke diagnostics, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, output contracts, and resource-identity syntax and qualification. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [exchange storage](../../internal/domain/exchange.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
| `internal/engine` | Assembles an already captured configuration into shared capacity, rendering, validation, generation, repair, runner, and smoke-test collaborators. Immutable namespace views share those collaborators and qualify selections without rewriting captured references. Built-in engines provide the diagnostic HTTP collaborator; injected-client engines do not. It delegates resource inspection and listing to the capture. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#typed-configuration-loading), [internal runner](runner.md) |
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, ordinary attempt observations, and invocation-local evidence for diagnostics. | [Internal generation coordination](generation.md) |
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded retries, per-attempt cost, ordinary observations, and separate diagnostic evidence. | [Internal generation coordination](generation.md) |
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible provider values, and detaches previously accepted values without repeating validation. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
@@ -30,8 +30,8 @@ contributor workflow and validation.
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data, and conservatively enumerates dependencies from captured template syntax. | [Go-template renderer](../../internal/prompt/go_renderer.go), [requirement discovery](../../internal/prompt/requirements.go) |
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
| `internal/validate` | Creates operation-local validation plans from inline schema documents or bound filesystem resources and validates basic, JSON, and JSON Schema output. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded response decoding, authentication, deadline handling, per-invocation HTTP evidence and built-in diagnostic probes, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, synchronous metadata event delivery and composition, and explicit terminal-based group aggregation; smoke diagnostics keep their private evidence outside ordinary observer events. | [Internal execution observation](observation.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible requests, provider-specific response interpretation and authentication probes, bounded decoding, deadlines, and diagnostic evidence. | [Internal model client](llm.md) |
| `internal/observation` | Owns invocation-local run IDs, presence-aware token and cost accounting, contextual observer selection, synchronous event delivery, and terminal-based group aggregation; smoke diagnostics retain separate evidence. | [Internal execution observation](observation.md) |
| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair, advances a single conversation branch, and runs explicit smoke diagnostics with private cases, shared generation/capacity, and optional built-in HTTP probes. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
The internal engine assembles these internal components without exposing their

View File

@@ -216,10 +216,16 @@ ordinary `Run` reuses its run ID and carries it to the generation coordinator
through context. Prompt and profile identities are enriched only after their
respective resolutions succeed, before admission. The facade starts observation
before request conversion and emits its terminal event after public error
mapping and after the runner's deferred admission release. Preparation and
prepared execution follows the same terminal mapping after a successful claim;
mapping and after the runner's deferred admission release. Preparation emits no
events; prepared execution follows the same terminal mapping after a successful claim;
invalid or discarded handles do not acquire an observer.
Ordinary runs establish fresh private cost state before observation; a claimed
prepared execution establishes its own state after claim. Contextual observers
remain available at public boundaries, while enclosing recorder and cost state
are masked. Prepared handles retain the request observer and frozen provider
policy, not preparation-context selection or an accumulator.
The facade establishes an optional operation scope for ordinary runs before
observation and preparation. A claimed prepared execution establishes its own
scope from the retained duration; preparation and idle time do not consume it.
@@ -273,11 +279,13 @@ run identifier, and UTC timing. The same effective session reaches initial
generation and any repair attempt through the rendered prompt. The same
effective target and presence metadata, including backend identity and direct
credential during execution, reaches generation and every repair attempt.
Result usage is the field-wise sum of all five usage values from the initial
Result usage is the field-wise sum of usage values from the initial
response and every completed repair response. Final raw output, artifact, and
validation state still come from the last candidate. A repair error returns no
partial run result or partial result usage, although the invocation-local
observer retains completed-attempt usage and reports it in the terminal event.
Result cost instead snapshots the shared accumulator across all dispatched
attempts, including retries and repairs, whether or not observation is enabled.
## Failure Categories

View File

@@ -417,6 +417,8 @@ implementation has reached its final buildable state.
## Stage 15 — Canonical Documentation And Final Verification
**Completed.**
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and
sources, client ownership, and observer/prepared lifetimes. Update stale
authentication Kind requirements in shared settings and public docs.

View File

@@ -1,6 +1,7 @@
# Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending.
Status: accepted scope; implementation complete. Current behavior is documented
in the linked canonical contracts; roadmap retirement is separate housekeeping.
## Purpose And Scope