Prepare the v0.25.0 release documentation
This commit is contained in:
@@ -45,6 +45,12 @@ boundary and constraints that framework work must preserve.
|
||||
|
||||
## Release Guidance
|
||||
|
||||
The [v0.25.0 release notes](docs/releases/v0.25.0.md) cover invocation-scoped
|
||||
observation, provider conventions, reasoning and cache usage, and optional USD
|
||||
cost reporting. Upgrading consumers should review provider selection,
|
||||
authentication diagnostic defaults, and the corrected OpenRouter cache-write
|
||||
mapping.
|
||||
|
||||
The [v0.24.0 release notes](docs/releases/v0.24.0.md) cover application-owned
|
||||
configuration loading, prompt-owned schemas, default output repair, and the
|
||||
prepared message-content hash. Upgrading consumers should review the breaking
|
||||
|
||||
88
docs/releases/v0.25.0.md
Normal file
88
docs/releases/v0.25.0.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Promptkit v0.25.0
|
||||
|
||||
This minor release adds invocation-scoped execution observation, provider-specific
|
||||
reasoning and cache usage, and optional USD cost reporting. Backend definitions
|
||||
can now select API conventions independently of their names and endpoints.
|
||||
Consumers upgrading from v0.24.x should review the configuration and accounting
|
||||
changes below.
|
||||
|
||||
## Added And Changed
|
||||
|
||||
- `WithExecutionObserver` selects observation through a caller-owned context
|
||||
across nested, concurrent, coordinated, and multi-engine calls. Request
|
||||
observers run before contextual observers. Prepared handles retain their
|
||||
explicit request observer but select contextual observers when executed,
|
||||
not when prepared. Each execution retains independent accounting; groups
|
||||
count each run's first terminal once. See the
|
||||
[observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
||||
and [offline example](../../examples/go-library/observe/main.go).
|
||||
- Backend `provider` selects `openai-compatible`, `openrouter`, or `vllm`
|
||||
conventions. Omitted, null, or blank selects the generic OpenAI-compatible
|
||||
behavior. The bundled OpenRouter backend explicitly selects `openrouter`.
|
||||
`InspectBackend` exposes declared provider and `zero_cost` settings without
|
||||
manufacturing runtime defaults. See the
|
||||
[backend reference](../formats.md#backend-definitions).
|
||||
- Usage values, presence indicators, observations, and group summaries include
|
||||
`ReasoningTokens`. OpenRouter and vLLM decoding includes their respective
|
||||
reasoning, cache-read, and cache-write fields. Reasoning is a subset of
|
||||
completion tokens; cache activity does not add to reported token totals.
|
||||
`ObservedTokenUsage.CacheHitRatio` derives a ratio from complete, consistent
|
||||
input and cache-read totals. See the
|
||||
[response contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
- `RunResult.CostUSD`, execution events, and group summaries report a USD
|
||||
subtotal with availability, completeness, and source metadata. OpenRouter
|
||||
supplies optional `usage.cost`; injected clients can supply
|
||||
`GenerateResponse.CostUSD`. Explicit backend `zero_cost: true` supplies a
|
||||
zero fallback when no usable amount is reported. A reported amount, including
|
||||
zero, takes precedence. See the [cost GoDoc](../../cost.go).
|
||||
- Authentication diagnostics can use provider defaults: `openrouter_key` for
|
||||
OpenRouter and the new `vllm_models` adapter for vLLM. The latter probes
|
||||
base-relative `/models` and reports recognized response facts without
|
||||
asserting authentication enforcement or inference authorization. See the
|
||||
[diagnostic integration contract](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics).
|
||||
|
||||
## Compatibility And Upgrade
|
||||
|
||||
- For application-defined OpenRouter backends, set `provider: openrouter` to
|
||||
enable OpenRouter-specific usage and cost decoding and its authentication
|
||||
default. Backend IDs and endpoint hostnames do not select conventions.
|
||||
Generic backends retain their existing request and token conventions and
|
||||
do not decode monetary cost. Unsupported nonblank provider names are
|
||||
configuration errors; supplied names are trimmed and lowercased.
|
||||
- OpenRouter cache-write reporting now reads
|
||||
`usage.prompt_tokens_details.cache_write_tokens`, not the generic top-level
|
||||
`usage.cache_write_tokens` field. vLLM uses
|
||||
`usage.prompt_tokens_details.created_cache_tokens`. Missing or unusable new
|
||||
details remain unknown rather than invalidating otherwise usable content.
|
||||
Availability depends on the provider and server settings; consult the
|
||||
[response contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
- A supplied authentication object no longer requires `kind`: an omitted or
|
||||
null kind uses the provider default. Clearing the authentication block also
|
||||
restores that default. Set `kind: none` to declare authentication testing
|
||||
inapplicable. The generic provider has no authentication default; explicit
|
||||
adapters remain supported independently of provider selection. Probes still
|
||||
run only when explicitly requested. See the
|
||||
[backend reference](../formats.md#backend-definitions).
|
||||
- Cost includes all dispatched attempts, including transient retries and
|
||||
structural repairs, and is available on successful results without an
|
||||
observer. On operational failure, observations can retain cost even though
|
||||
no partial result is returned. Existing `RunResult.Usage` continues to sum
|
||||
successful generation responses; presence-aware observations cover all
|
||||
dispatched attempts. Do not add terminal totals to attempt totals.
|
||||
- Unknown cost is distinct from reported or configured zero. No backend is
|
||||
automatically declared zero-cost, and no token-rate estimation is performed.
|
||||
Group cost completeness describes included terminals, independently of
|
||||
terminal coverage. Consumers own presentation and interpretation; complete
|
||||
reporting does not verify billing.
|
||||
- Result JSON adds `cost_usd`, and token JSON adds `reasoning_tokens`.
|
||||
Update strict decoders if necessary. Older result JSON without cost decodes
|
||||
to unknown cost. Consumers using positional literals for expanded public
|
||||
structs may need to update them. See the [public types](../../types.go).
|
||||
|
||||
The Go consumer minimum remains 1.27.0; dependencies are unchanged from v0.24.0.
|
||||
|
||||
```sh
|
||||
go get go.maximumdirect.net/promptkit@v0.25.0
|
||||
```
|
||||
|
||||
[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/COPYRIGHT) | [LICENSE](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/LICENSE) | [ATTRIBUTION](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/docs/oss/attribution.md)
|
||||
@@ -1,448 +0,0 @@
|
||||
# Invocation Observation And Usage Implementation Plan
|
||||
|
||||
## Objective And Working Rules
|
||||
|
||||
Implement the accepted [target-state roadmap](invocation-scoped-token-usage.md).
|
||||
It owns scope, consumer intent, and reporting policy. This plan owns concrete
|
||||
implementation decisions and the ordered work sequence. Baseline: `a47df4e`.
|
||||
|
||||
Each numbered stage is one Luna-sized implementation prompt. Execute them in
|
||||
order within one end-to-end sprint. Intermediate stages may leave unresolved
|
||||
references or failing compilation; do not add compatibility scaffolding merely
|
||||
to keep those intermediate states green. There are no intermediate human-review,
|
||||
external-publication, or live-provider gates. Run focused checks when their
|
||||
dependencies are ready; the completed sprint must pass the final checks.
|
||||
|
||||
Read [development guidance](../development.md) and the applicable agent,
|
||||
architecture, testing, and documentation policies. Follow
|
||||
[ADR 0001](../adr/0001-configuration-boundary.md): interpretation and validation
|
||||
use final ConfigKit values, not raw sources or historical inspection. Follow the
|
||||
shared-settings facade exception without introducing public aliases for runtime
|
||||
implementation types. Preserve unrelated work and follow the agent commit policy.
|
||||
|
||||
Planning and implementing are separate tasks. This document does not authorize
|
||||
a release, dependency upgrade, CLI implementation, or changes to published notes.
|
||||
|
||||
## Selected Design
|
||||
|
||||
- Store contextual observers as internal `domain.ExecutionObserver` callbacks
|
||||
under a context key separate from the recorder, claim sink, and cost state.
|
||||
The public helper wraps public event conversion at registration. Internal
|
||||
composition preserves explicit-first and inherited-to-new contextual ordering.
|
||||
- Add `Provider *string` and `ZeroCost *bool` to shared backend settings and
|
||||
`BackendConfig`, with ordinary `provider`/`zero_cost` field names. Resource
|
||||
inspection returns declared normalized values; configuration inspection owns
|
||||
history. Neither inspection API needs a new provenance model.
|
||||
- Capture runtime `Provider string` and `ZeroCost bool` on `domain.Backend` and
|
||||
carry them on internal `domain.ExecutionTarget` as local metadata excluded
|
||||
from JSON/YAML. Empty provider resolves to `openai-compatible` at runtime.
|
||||
Keep these fields out of the root execution-target API and override API:
|
||||
backend inspection is the requested public configuration surface. Endpoint-only
|
||||
profiles use the generic provider and disabled zero fallback. Existing
|
||||
model/endpoint overrides retain the selected backend's metadata.
|
||||
- Keep provider ID recognition, response interpretation, and authentication
|
||||
adapter resolution in `internal/llm`, using simple functions/dispatch. Config
|
||||
validation may call that package's supported-name checks as it already does
|
||||
for reserved request fields. No public provider registration mechanism or new
|
||||
providers package is needed.
|
||||
- Add `ReasoningTokens int` to token values, `ReasoningTokens bool` to presence,
|
||||
and `ReasoningTokens ObservedTokenCount` to observed usage. Public token JSON
|
||||
uses `reasoning_tokens`. Add the roadmap's two helper methods/functions exactly
|
||||
as specified; do not add aliases for cache reads or a stored cache-hit ratio.
|
||||
- Define public `CostSource string` constants `CostSourceNone = ""`,
|
||||
`CostSourceProvider = "provider"`, `CostSourceConfiguredZero = "configured_zero"`,
|
||||
and `CostSourceMixed = "mixed"`. Define `ObservedCost` with `Value float64`,
|
||||
`Known bool`, `Complete bool`, and `Source CostSource`, JSON-tagged `value`,
|
||||
`known`, `complete`, and `source`, without omitting zero fields.
|
||||
- Add `GenerateResponse.CostUSD *float64` (`json:"cost_usd,omitempty"`) and
|
||||
`RunResult.CostUSD ObservedCost` (`json:"cost_usd"`). Add `CostUSD ObservedCost`
|
||||
to `ExecutionEvent` and `ExecutionGroupSummary`; the latter uses `cost_usd`.
|
||||
Define equivalent internal domain values and explicit facade conversions.
|
||||
Older result JSON lacking cost decodes to unknown cost, not reported zero.
|
||||
- Own pure cost resolution/merge and run-local cost state in
|
||||
`internal/observation`. Cost is collected once per dispatched attempt regardless
|
||||
of observer selection. The recorder references that same run accumulator for
|
||||
terminal cost; it must not independently add attempts a second time. Groups
|
||||
reuse the merge rules but remain terminal-only collectors.
|
||||
- New optional reasoning/cache-write details use local tolerant integer decoding;
|
||||
cost uses local tolerant finite-float decoding. A typed `*int`/`*float64` field
|
||||
in the main envelope would fail the whole decode for malformed optional data.
|
||||
Preserve existing common token decoding, framing, content validation, and
|
||||
error identities. Do not add a general numeric coercion framework.
|
||||
|
||||
## Stage 1 — Context Helper And Ordinary Run Selection
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch `observations.go`, the root `Engine.Run` in `engine.go`, and context support
|
||||
in `internal/observation/recorder.go` or a focused adjacent file.
|
||||
|
||||
- Implement `WithExecutionObserver(context.Context, ExecutionObserver)
|
||||
context.Context` using a private internal observer-selection key. Nil is a
|
||||
no-op. Compose inherited callbacks before the new callback using the existing
|
||||
composition facility; keep registration state immutable.
|
||||
- In ordinary `Run`, compose a converted explicit request observer with the
|
||||
contextual selection before deciding whether to create a recorder. Preserve
|
||||
current start timing, run IDs, public error mapping, timeout finalization,
|
||||
permit release, and terminal delivery. With neither observer, preserve the
|
||||
existing no-observation path.
|
||||
- Do not mask the contextual key in `WithoutExecutionState`; continue masking
|
||||
inherited recorder and claim state. Each nested public call creates its own
|
||||
recorder. Do not modify `Coordinate` to append contextual selection: its
|
||||
explicit request/coordinator composition already leads to ordinary `Run`.
|
||||
- Add focused public tests for no observer, contextual-only selection, explicit
|
||||
plus nested context ordering, nil registration, unrelated contexts, and two
|
||||
independent run IDs from a nested injected-client call. Group duplication
|
||||
uses existing terminal deduplication, not callback equality checks.
|
||||
|
||||
Validation when buildable: root observation/boundary contract tests and focused
|
||||
internal observer tests. No provider requests are needed.
|
||||
|
||||
## Stage 2 — Execution-Time Context For Prepared Handles
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch root `RunPrepared`, internal `usecase/prepared_execution.go`, and observer
|
||||
conversion only where required.
|
||||
|
||||
- Retain only the explicitly supplied request observer in the prepared payload.
|
||||
Do not snapshot contextual observers during `PrepareExecution` or capture
|
||||
preparation contexts in the handle.
|
||||
- After a successful claim, compose that observer with the contextual selection
|
||||
from the execution context before recorder creation. Preserve the private
|
||||
claim sink so root terminal delivery still follows public error mapping and
|
||||
finalized timeout attribution.
|
||||
- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle
|
||||
rejection, and existing payload clearing. Do not expose private callbacks in
|
||||
`Details` or create a second lifecycle around the claimed execution.
|
||||
- Test a handle prepared in scope A and executed in B: retained explicit
|
||||
observer and B receive events; A does not. Include a context-only execution,
|
||||
rejected claims, and a continuation selected by `Coordinate` to protect its
|
||||
explicit-before-context order.
|
||||
|
||||
Validation: prepared execution and coordinate observation contract tests.
|
||||
|
||||
## Stage 3 — Backend Configuration And Provider Selection
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch `internal/settings/settings.go`, `backends.go`, configuration ownership,
|
||||
resolution/validation code, and the provider-selection functions in `internal/llm`.
|
||||
|
||||
- Add the two pointer fields and presence-preserving source-helper encoding.
|
||||
Nil omits a field; false and blank pointers remain supplied values. Raw
|
||||
ConfigKit mappings remain the way to supply explicit null.
|
||||
- Normalize supplied provider strings with trimming/lowercasing in the existing
|
||||
final resolution callback and record normalizations using existing ConfigKit
|
||||
APIs. Do not materialize the runtime default into the settings object.
|
||||
- Recognize exactly `openai-compatible`, `openrouter`, and `vllm`; empty means
|
||||
generic. Reject other nonempty values through existing final configuration
|
||||
validation/error wrapping. Do not validate overwritten contributions.
|
||||
- Validate `zero_cost` as the typed Boolean setting; absent/null/false disables
|
||||
fallback. Do not infer it or make `LocalBackend` set it automatically.
|
||||
- Relax authentication-object validation to allow absent/null kind, retain
|
||||
existing kind normalization and supported explicit kinds, and add
|
||||
`vllm_models`. A missing kind with no provider default remains an unsupported
|
||||
diagnostic selection, not a new load-time requirement for a provider.
|
||||
- Cover layered final-value validation, inheritance, null clearing, normalization,
|
||||
unsupported names, helper presence, and absence of invented defaults.
|
||||
|
||||
Validation: focused configuration/settings/source-helper tests.
|
||||
|
||||
## Stage 4 — Runtime Capture, Freezing, And Backend Inspection
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch `internal/configuration/targets.go`, `ownership.go`, `resources.go`, domain
|
||||
backend/target values and clones, and `bundled/openrouter.yaml` under that package.
|
||||
|
||||
- Copy declared settings into owned records and inspection results. Preserve
|
||||
explicit false while allowing absent/null collapse in resource inspection.
|
||||
Leave full null/omission history to `InspectConfiguration`.
|
||||
- Capture effective provider and zero policy on the internal backend and the
|
||||
backend-derived execution target. Ensure scalar copying through profile
|
||||
capture, runtime overrides, prepared details/payloads, and repair requests.
|
||||
Retain backend metadata when a profile/request overrides only its endpoint.
|
||||
- Keep root public execution targets and provider HTTP bodies unchanged. Audit
|
||||
serialization/copy helpers so excluded internal fields are not accidentally
|
||||
lost by a JSON-based copy. Do not recapture policy from configuration during
|
||||
`RunPrepared` or traverse provenance during execution.
|
||||
- Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost
|
||||
defaults or new backend instances. Keep API keys and default profiles unchanged.
|
||||
- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only
|
||||
profiles, frozen prepared policy, and inspection without manufactured values.
|
||||
|
||||
Validation: configuration capture, backend inspection, bundle, and preparation tests.
|
||||
|
||||
## Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults
|
||||
|
||||
**Completed.**
|
||||
|
||||
Current provider branches occur in `internal/usecase/smoke_authentication.go` and
|
||||
`smoke_selection.go`, including `ProbeOpenRouterKey` and endpoint derivation.
|
||||
|
||||
- Replace the OpenRouter-specific collaborator operation with a generic
|
||||
authentication-probe operation owned by `internal/llm`. Carry a resolved
|
||||
adapter selection to it; leave usecase coordination provider-neutral.
|
||||
- Resolve adapter kind and destination with pure provider-boundary functions.
|
||||
Explicit kind wins, absent/null kind uses the provider default, and `none`
|
||||
is inapplicable. Explicit endpoint wins; otherwise use the adapter-relative
|
||||
fallback. Missing selection remains unsupported. Do not rewrite loaded
|
||||
settings, captured declared smoke metadata, or inspection provenance.
|
||||
- OpenRouter defaults to `openrouter_key`; generic has no default. Preserve
|
||||
explicit OpenRouter adapters on any backend and existing HTTP evidence,
|
||||
bounded recognition, credential, cancellation, and overflow handling.
|
||||
- Clear an authentication block to restore the provider default; an endpoint-only
|
||||
block may use that default. Preserve reachability and inference selection.
|
||||
- Update affected interfaces, engine assembly, and test doubles. Test default,
|
||||
explicit/none, endpoint-only, cleared-block, and unsupported combinations at
|
||||
their owning boundaries rather than duplicating every permutation publicly.
|
||||
|
||||
Validation: smoke selection/authentication tests and existing OpenRouter contract tests.
|
||||
|
||||
## Stage 6 — vLLM Models Authentication Adapter
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Implement `vllm_models` beside the OpenRouter adapter, sharing generic HTTP
|
||||
execution and bounded recognition helpers rather than duplicating transport.
|
||||
Select it by default for `vllm` and allow explicit use on other providers.
|
||||
- Resolve `models` beneath the configured API base, preserving base prefixes;
|
||||
honor an explicit full diagnostic endpoint. Do not hard-code a host or probe
|
||||
`/health`. Use existing optional credential rules without forcing a key.
|
||||
- Recognize exactly the roadmap's HTTP 200 object/list/data-array shape,
|
||||
including empty arrays. HTTP 401 is rejection; other/unrecognized responses
|
||||
follow existing unknown/failure reporting. Do not inspect list members for
|
||||
model existence, inference permissions, or authentication enforcement.
|
||||
- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other
|
||||
statuses, optional credentials, overridden destinations, and `kind: none`.
|
||||
Ensure unsupported injected HTTP diagnostics still do not fall back to a
|
||||
built-in transport.
|
||||
|
||||
Validation: focused adapter and smoke coordinator tests with controlled HTTP.
|
||||
|
||||
## Stage 7 — Reasoning Values And Provider-Specific Token Decoding
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch public/domain token declarations, facade conversions, and
|
||||
`internal/llm/openai_compatible_client.go` plus focused provider files.
|
||||
|
||||
- Add `ReasoningTokens` to `TokenUsage`, `TokenUsagePresence`, and observed usage
|
||||
on both sides of the facade; include JSON field/conversions and mutable-response
|
||||
ownership where applicable. Do not rename `CachedTokens`.
|
||||
- Keep common counts and cached reads on their existing decode path. For
|
||||
OpenRouter, read reasoning from `completion_tokens_details.reasoning_tokens`
|
||||
and writes from `prompt_tokens_details.cache_write_tokens`; for vLLM, read
|
||||
the same reasoning path and `prompt_tokens_details.created_cache_tokens`.
|
||||
Generic keeps its existing top-level write mapping and does not interpret the
|
||||
new reasoning field. Ignore other providers' alternative write locations;
|
||||
do not add/fallback between the nested and top-level values.
|
||||
- Decode newly interpreted integer fields independently with local raw JSON
|
||||
helpers. Absent/null, wrong-type, fractional, or int-overflow values become
|
||||
unknown. Do not coerce strings. Retain independently valid fields; adding a
|
||||
typed optional field must not create a new whole-response failure. Existing
|
||||
malformed common fields and framing/content failures retain their behavior.
|
||||
- Preserve decoded usage on later missing-content/choice failure as today.
|
||||
Do not infer missing reasoning or writes, enforce new nonnegative restrictions,
|
||||
change generation controls, or collect reasoning text.
|
||||
- Test provider-specific raw fixtures with conflicting write locations,
|
||||
partial/malformed new fields, known zero, and disabled-detail vLLM responses.
|
||||
Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap.
|
||||
|
||||
Validation: model-client parsing tests and public token JSON round trips.
|
||||
|
||||
## Stage 8 — Token Aggregation And Cache-Hit Helper
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch observation usage accumulators/recorder/group, runner `addTokenUsage`,
|
||||
public event/group conversions, and related fixtures.
|
||||
|
||||
- Include reasoning counts in existing successful-result summation, attempt and
|
||||
terminal usage, group merging, and explicit/legacy injected presence behavior.
|
||||
Do not replace legacy result token sums with all-attempt observation sums.
|
||||
- Implement `ObservedTokenUsage.CacheHitRatio() (float64, bool)` using the
|
||||
roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid
|
||||
data returns `(0, false)`; known zero reads with positive input returns `(0, true)`.
|
||||
Divide totals as floats, not integer division. Do not persist or average ratios.
|
||||
- Cover per-field incomplete reasoning, injected known zero, retry/repair sums,
|
||||
duplicate group enrollment, weighted aggregate ratios, zero denominator,
|
||||
inconsistent counts without generation failure, and independence from writes.
|
||||
- Preserve reported total tokens: reasoning and cache details are subsets, not
|
||||
extra values to add into input/output/total counts.
|
||||
|
||||
Validation: focused token accumulator/group tests and public observation contracts.
|
||||
|
||||
## Stage 9 — Public Cost Values And Shared Cost Algebra
|
||||
|
||||
**Completed.**
|
||||
|
||||
Create public cost declarations in a focused root file and matching domain
|
||||
values; add the selected fields to responses, results, events, and summaries.
|
||||
|
||||
- Implement the selected cost/source declarations and conversions. Copy optional
|
||||
amount pointers at the injected-client conversion boundary so caller mutation
|
||||
cannot change retained data. Include cost in custom result JSON marshal/unmarshal
|
||||
paths as needed, preserving existing duration and omission behavior elsewhere.
|
||||
- Implement pure internal cost resolution: finite reported amount first, then
|
||||
configured zero, else unknown. A reported zero remains provider-sourced.
|
||||
Negative finite reported amounts are not newly prohibited. Unknown cost is
|
||||
zero-valued with false Known/Complete and empty source.
|
||||
- Implement one small `CostAccumulator` in `internal/observation` that merges
|
||||
known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot
|
||||
is known/complete zero; its first unknown contribution must reset that empty
|
||||
special case rather than accidentally making unknown cost known. Include
|
||||
known zero contributions in source union. Unknown contributions add no source.
|
||||
- Allow merging aggregate costs for groups with the same helper; leave the
|
||||
existing no-terminal group special case to group coverage logic.
|
||||
- Unit-test precedence, source union including zeros, all-unknown/empty cases,
|
||||
completeness, and approximate floating-point sums. Add JSON tests for old
|
||||
result JSON without cost, current round trips, and optional client zero.
|
||||
|
||||
Validation: pure accounting and public serialization tests; no pricing library.
|
||||
|
||||
## Stage 10 — Optional Cost At The Model-Client Boundary
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Decode OpenRouter `usage.cost` independently using local tolerant finite-float
|
||||
handling. Missing/null/string/object/overflow values are unknown; invalid
|
||||
framing or common envelope fields retain existing failures. Generic/vLLM do
|
||||
not interpret the field. Do not consume `cost_details` or send a new usage flag.
|
||||
- Carry raw optional cost on built-in `AttemptOutcome` and `GenerateResponse`
|
||||
even when a later decoded-content check fails. Non-2xx/invalid framing remain
|
||||
without provider usage. Do not apply zero-cost fallback inside provider code.
|
||||
- Preserve optional cost supplied through an injected client. The adapter already
|
||||
converts non-nil responses alongside errors: retain cost for accounting even
|
||||
then, without changing existing token treatment or public error/result rules.
|
||||
Normalize nonfinite injected cost to unknown before applying fallback.
|
||||
- Test cost independent of token fields, usable zero, malformed optional cost
|
||||
without generation failure, ignored cost on generic/vLLM, and reported cost
|
||||
retained alongside a classified injected error.
|
||||
|
||||
Validation: provider decoder and injected-adapter tests.
|
||||
|
||||
## Stage 11 — Per-Attempt Cost Collection And Observation
|
||||
|
||||
**Completed.**
|
||||
|
||||
Touch generation `recordAttempt` and built-in/injected execution paths,
|
||||
observation attempt data/events, recorder terminal cost, and group aggregation.
|
||||
|
||||
- At every attempt completion, if dispatched, resolve raw cost against the
|
||||
frozen target's ZeroCost and add it exactly once to an invocation accumulator.
|
||||
Do this for transport errors, eligible retry statuses, cancellation after
|
||||
dispatch, malformed bodies, and missing content as well as success. A failed
|
||||
acquisition/pre-dispatch cancellation contributes nothing.
|
||||
- Keep cost collection independent of the optional recorder and diagnostic
|
||||
evidence capture flag. Preserve permit-release-before-callback behavior.
|
||||
Do not count retry waits, terminal events, or generation responses again.
|
||||
- Carry resolved cost on attempt events. Recorder terminal cost reads the same
|
||||
accumulator; groups merge only first terminal costs via shared algebra.
|
||||
Extend public event/group projection without changing existing token totals.
|
||||
- Introduce private cost-state context access. Extend `WithoutExecutionState` to
|
||||
mask inherited cost state, but not contextual observer selection. Smoke
|
||||
diagnostics must continue masking private execution accounting and emit no
|
||||
ordinary lifecycle or enclosing-run cost contributions.
|
||||
- Exercise controlled retry/error sequences and zero-cost fallback without an
|
||||
observer. Test duplicate terminal delivery and independence of group terminal
|
||||
coverage from cost completeness. Do not retain an attempt history for cost.
|
||||
|
||||
Validation: generation, observation recorder/group, and diagnostic-boundary tests.
|
||||
|
||||
## Stage 12 — Ordinary Run Cost Lifetime And Result Accounting
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Create/install a new private accumulator at the public ordinary Run boundary,
|
||||
before optional recorder creation. Pass that same instance to the recorder.
|
||||
Ensure direct internal runner execution creates a local accumulator if one
|
||||
was not supplied, without replacing one supplied by the owning facade.
|
||||
- Retain it through initial generation and all repair calls. Set successful
|
||||
`domain.RunResult.CostUSD` from its snapshot, not by summing responses. Keep
|
||||
the legacy `Usage` success-response sum and error handling unchanged.
|
||||
- Ensure errors still return no partial public `RunResult`; available cost is
|
||||
reported in terminal observations. Runs rejected before dispatch have the
|
||||
established empty aggregate, not a configured-zero contribution source.
|
||||
- Test a retry with unreported cost followed by a reported charge, mixed sources
|
||||
within repairs, reported-cost precedence over zero policy, and identical
|
||||
successful cost metadata with observation enabled/disabled.
|
||||
|
||||
Validation: assembled ordinary run/reliable execution contract tests.
|
||||
|
||||
## Stage 13 — Prepared Cost Lifetime And Nested Isolation
|
||||
|
||||
**Completed.**
|
||||
|
||||
- In internal RunPrepared, after successful claim create/install its own cost
|
||||
accumulator before recorder creation and generation. Use the same result and
|
||||
terminal accounting path as ordinary Run. Do not store an accumulator or
|
||||
contextual observer in the reusable preparation details/private ready payload.
|
||||
- The claimed handle uses its frozen backend Provider/ZeroCost, even when run
|
||||
from another invocation scope. Preserve admission, deadline attribution,
|
||||
single-claim semantics, discard behavior, and cleared callbacks/credentials.
|
||||
- Verify ordinary/prepared child runs launched from injected clients or
|
||||
prepared callbacks do not inherit the parent's accumulator, even when both
|
||||
inherit one contextual group observer. Each terminal contributes once to the
|
||||
invocation total; parent cost includes only its own dispatched generations.
|
||||
- Cover prepared zero-cost failure and repair, rejected claims without events,
|
||||
independent contexts on a reused engine, and diagnostic calls nested in a run
|
||||
without cost/event leakage.
|
||||
|
||||
Validation: prepared and nested-boundary contracts, including race-enabled focused tests.
|
||||
|
||||
## Stage 14 — End-To-End Consumer Regression Coverage
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Add one representative offline invocation workflow using two engines and
|
||||
concurrent runs under one group. Use controlled transport/injected clients
|
||||
for OpenRouter reported cost, vLLM details/configured zero, a retry, and a
|
||||
failed operation after available usage. Join application-owned goroutines
|
||||
before taking the final snapshot.
|
||||
- Assert independently predictable terminal membership, sums, sources, token
|
||||
presence/completeness, and weighted ratio (or its unavailability when an
|
||||
attempt leaves input/cache-read reporting incomplete). Compare floating-point
|
||||
sums with tolerance; do not assert map order, run-ID bytes, or exact timing.
|
||||
- Cover a snapshot with completed fully costed members and one started member:
|
||||
cost completeness concerns the included terminals while terminal coverage is
|
||||
false. Cover empty and members-with-no-terminals snapshots separately.
|
||||
- Reuse narrower tests from earlier stages rather than reproduce every case in
|
||||
this workflow. Keep observations metadata-only; do not add bodies, credentials,
|
||||
URLs, or error objects to the event contract.
|
||||
|
||||
Validation: focused new integration tests plus `go test -race ./...` when the
|
||||
implementation has reached its final buildable state.
|
||||
|
||||
## Stage 15 — Canonical Documentation And Final Verification
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and
|
||||
sources, client ownership, and observer/prepared lifetimes. Update stale
|
||||
authentication Kind requirements in shared settings and public docs.
|
||||
- Update the consumer guide and maintained offline observation example for
|
||||
invocation scoping, cache/reasoning subsets, known/complete cost, configured
|
||||
zeros, sources, and failure-path reporting. Maintain semantic example tests.
|
||||
- Update `docs/formats.md`, the OpenAI-compatible integration contract, and
|
||||
internal overview/observation/generation/model-client/runner documentation for
|
||||
actual ownership and flow. Explain provider defaults versus declared inspection,
|
||||
vLLM reporting settings, and the exact diagnostic response recognition.
|
||||
- Document migration from required authentication kind and block-clearing
|
||||
semantics. Record the change in the next unpublished release note when such a
|
||||
note exists; otherwise leave release-note creation to release preparation.
|
||||
Do not edit published v0.24.0 notes or tag/publish anything in this sprint.
|
||||
- Use the newest already installed Go toolchain per the development guide,
|
||||
`GOTOOLCHAIN=local`, and `GOWORK=off`. Format changed Go files, run
|
||||
`go test -race ./...`, `go vet ./...`, and `git diff --check`. Reuse successful
|
||||
results for unchanged content; do not add redundant builds or live-provider
|
||||
checks. Verify local Markdown links and maintained examples through their
|
||||
normal offline tests. Review the entire diff for unrelated changes and secrets.
|
||||
- Audit completion against every target-state section, especially no-observer
|
||||
cost, failed dispatched attempts, optional-field tolerance, prepared policy
|
||||
freezing, and nested/diagnostic separation. Resolve remaining implementation
|
||||
defects before reporting sprint completion. Roadmap retirement is later
|
||||
housekeeping, not part of this implementation plan.
|
||||
|
||||
There are no unresolved product decisions required to begin these stages.
|
||||
@@ -1,497 +0,0 @@
|
||||
# Invocation-Scoped Execution Observation
|
||||
|
||||
Status: accepted scope; implementation complete. Current behavior is documented
|
||||
in the linked canonical contracts; roadmap retirement is separate housekeeping.
|
||||
|
||||
## Purpose And Scope
|
||||
|
||||
Let consumers select observation once for an application-owned operation and
|
||||
obtain usage across its nested, concurrent, coordinated, and multi-engine
|
||||
PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a
|
||||
token cache-hit ratio, and optional USD cost with its source and completeness.
|
||||
|
||||
CommandKit's optional `ckprompt` integration motivates this work, but the API
|
||||
remains application-neutral. PromptKit owns observation, provider response
|
||||
interpretation, and aggregation. Consumers own context propagation, waiting for
|
||||
their work, invocation lifecycle, presentation, and interpretation.
|
||||
|
||||
This document defines the target state and the changes required to reach it.
|
||||
A separate implementation plan will own sequencing and prompt-sized stages.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
PromptKit v0.24.0 already provides
|
||||
[execution observers and ordered composition](../../observations.go),
|
||||
a concurrency-safe [execution group](../../execution_group.go), and
|
||||
[request-level observation](../consumers/pkg-promptkit.md#observe-an-ordinary-run).
|
||||
Groups aggregate the first terminal event for each observed run. Existing
|
||||
observation distinguishes known, unknown, and incomplete prompt, completion,
|
||||
total, cached, and cache-write token counts across retries and repairs.
|
||||
|
||||
Today, consumers explicitly enroll requests. Prepared handles retain the request
|
||||
observer; [coordinator observers](../../coordinate.go) select continuation runs.
|
||||
There is no contextual selection, reasoning-token count, cache-hit helper, or
|
||||
monetary cost reporting. OpenRouter cache writes also need their documented
|
||||
nested mapping: the current client reads `usage.cache_write_tokens` instead.
|
||||
|
||||
## Contextual Observer Selection
|
||||
|
||||
Add the public helper:
|
||||
|
||||
```go
|
||||
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
||||
```
|
||||
|
||||
The returned context composes the supplied observer with any inherited contextual
|
||||
observers. A nil observer leaves that selection unchanged. Executions using the
|
||||
context, or derived contexts retaining its values, participate automatically;
|
||||
unrelated contexts do not. No engine option or process-wide default is needed.
|
||||
|
||||
A consumer can create an invocation-owned group, associate `group.Observe` with
|
||||
the context, run its pipeline, and snapshot the group after its owned work
|
||||
finishes, including when the pipeline returns an error. Group coverage describes
|
||||
observed members, not future work or overall application progress.
|
||||
|
||||
### Delivery And Execution Boundaries
|
||||
|
||||
- `Run` selects contextual observers from its invocation context.
|
||||
- `RunPrepared` selects them from its execution context, not the context used
|
||||
to prepare the handle. The explicitly supplied request observer remains
|
||||
retained by the handle and composes with those execution-time observers.
|
||||
- Preparation emits no execution events. Discarded handles and rejected
|
||||
execution claims retain their existing observation behavior.
|
||||
- Coordinated continuations acquire contextual observation through their
|
||||
ordinary `Run` boundary. `CoordinateRequest.Observer` still selects
|
||||
continuation runs only; additional calls made by application callbacks
|
||||
participate when they use the scoped context.
|
||||
- Every execution, including a nested one using a callback or injected-client
|
||||
context, has its own identity, recorder, and totals. Clearing an enclosing
|
||||
execution's private state at a public boundary preserves contextual observer
|
||||
selection, not the enclosing recorder.
|
||||
|
||||
Deliver events to explicit request observers first, then any coordinator observer
|
||||
already composed with them, then contextual observers from inherited to newly
|
||||
registered. Incorporate contextual selection once per execution, not again in
|
||||
forwarding or coordination layers.
|
||||
|
||||
Preserve synchronous callback delivery and panic behavior. Concurrent runs may
|
||||
invoke a shared observer concurrently; consumers synchronize their callback state.
|
||||
The existing group remains concurrency-safe.
|
||||
|
||||
Registering the same callback through multiple paths may deliver duplicate
|
||||
events; arbitrary Go functions are not deduplicated. Retain one lifecycle per
|
||||
execution and reuse the group's terminal-event deduplication. Terminal totals
|
||||
are aggregates, not additional contributions to add to attempt totals.
|
||||
|
||||
## Backend Provider And Accounting Metadata
|
||||
|
||||
Add two optional backend fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `provider` | Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. |
|
||||
| `zero_cost` | Enables a configured-zero cost fallback when no usable reported cost is available. |
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
backends:
|
||||
- id: research-router
|
||||
provider: openrouter
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
- id: local-inference
|
||||
provider: vllm
|
||||
endpoint: http://localhost:8000/v1
|
||||
zero_cost: true
|
||||
```
|
||||
|
||||
Trim and lowercase the final effective provider value. Omitted, null, empty, and
|
||||
whitespace-only values select `openai-compatible`. Reject unsupported nonempty
|
||||
normalized values as ordinary configuration errors. The supported selections are
|
||||
`openai-compatible`, `openrouter`, and `vllm`.
|
||||
|
||||
`zero_cost` is Boolean. Omitted, null, or false disables its fallback; true
|
||||
enables it. It is independent of provider selection and does not suppress a
|
||||
usable reported cost.
|
||||
|
||||
Resolve both fields through ordinary final ConfigKit composition and inheritance.
|
||||
Carry their effective selections through capture and prepared execution without
|
||||
reloading configuration. Declare `provider: openrouter` in the bundled
|
||||
OpenRouter backend. Do not automatically declare any backend zero-cost.
|
||||
|
||||
These fields are local metadata, not outbound request parameters or entries
|
||||
added to `extra_params`. Provider selection does not choose an endpoint or model,
|
||||
and endpoint overrides retain the selected provider and accounting policy.
|
||||
|
||||
### Inspection Boundary
|
||||
|
||||
`InspectBackend` shows the normalized/resolved declared provider and zero-cost
|
||||
setting, preserving explicit false for `zero_cost`. Omitted/null declarations
|
||||
may remain absent, and blank provider values need no extra history representation.
|
||||
Do not manufacture an explicit `openai-compatible` declaration or false
|
||||
`zero_cost` value from runtime defaults.
|
||||
|
||||
`InspectConfiguration` owns ConfigKit source history and explicit/null/omitted
|
||||
distinctions. Resource inspection does not acquire source-history fields or
|
||||
null-tracking machinery. Provider diagnostic defaults likewise remain runtime
|
||||
fallbacks, not values written into loaded configuration or recorded as explicit
|
||||
source declarations.
|
||||
|
||||
## Provider Conventions
|
||||
|
||||
All selections reuse the existing
|
||||
[OpenAI-compatible text Chat Completions foundation](../integrations/openai-compatible-chat.md):
|
||||
base-relative `/chat/completions` JSON POST requests, existing credential rules,
|
||||
text messages, selected generation controls and structured output, extra
|
||||
parameters, first-choice content extraction, common token fields, and existing
|
||||
response validation, errors, cancellation, timeouts, and transient retries.
|
||||
|
||||
| Provider | Additional response interpretation | Default authentication adapter |
|
||||
| --- | --- | --- |
|
||||
| `openai-compatible` | Existing token mappings; no provider-reported monetary cost decoding. | None. |
|
||||
| `openrouter` | OpenRouter reasoning/cache details and `usage.cost` in USD. | `openrouter_key`. |
|
||||
| `vllm` | vLLM reasoning/cache details; no provider-reported monetary cost decoding. | `vllm_models`. |
|
||||
|
||||
The default provider names PromptKit's current supported conventions, not the
|
||||
entire OpenAI API. "Lowest common denominator" does not strip existing optional
|
||||
features or promise every backend supports them. Continue sending selected
|
||||
supported settings and return backend errors normally.
|
||||
|
||||
Provider selection supplies no default endpoint, model, credential environment
|
||||
variable, or sampling settings, and does not create backend definitions. Keep
|
||||
vLLM reasoning, chat-template, and other model-specific controls caller-selected;
|
||||
do not automatically enable thinking or translate those controls. No new
|
||||
generation protocol, capability negotiation, or version discovery is needed.
|
||||
vLLM documents its
|
||||
[OpenAI-compatible interface](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
||||
|
||||
### Authentication Diagnostics
|
||||
|
||||
Provider defaults apply only to explicitly requested authentication diagnostics.
|
||||
Loading configuration, constructing an engine, and ordinary generation do not
|
||||
initiate probes. Existing reachability and inference-default behavior remains.
|
||||
|
||||
Explicit `smoke_tests.authentication` metadata overrides provider defaults:
|
||||
|
||||
- A supplied `kind` selects the adapter; `none` declares authentication testing
|
||||
inapplicable.
|
||||
- Omitted/null `kind` uses the provider default, allowing endpoint-only overrides.
|
||||
- A supplied endpoint overrides the selected adapter's destination; omitted/null
|
||||
uses that adapter's existing fallback.
|
||||
- An absent/null authentication block removes its overrides and leaves provider
|
||||
defaults applicable. `kind: none`, not clearing the block, suppresses them.
|
||||
|
||||
Without a provider default or explicit adapter, preserve the existing unsupported
|
||||
diagnostic outcome rather than assuming authentication is inapplicable. Explicit
|
||||
adapters remain selectable without a matching provider declaration.
|
||||
|
||||
Retain `openrouter_key` behavior. The new `vllm_models` adapter makes one GET to
|
||||
`/models` relative to the configured API base, supplying an available credential
|
||||
under existing rules and allowing dispatch without one. A base ending in `/v1`
|
||||
therefore produces `/v1/models`; explicit diagnostic endpoint overrides apply.
|
||||
|
||||
For `vllm_models`, HTTP 200 with a valid JSON object containing `object: "list"`
|
||||
and array-valued `data` means acceptance of the probe request. An empty array is
|
||||
accepted; recognition does not inspect model capabilities or require a particular
|
||||
model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown.
|
||||
Reuse existing bounded diagnostic decoding and framing. Report observed
|
||||
credential presence and response facts without claiming authentication is
|
||||
enforced or models-list acceptance establishes inference authorization.
|
||||
|
||||
vLLM documents `/v1/models` as protected when API-key authentication is configured;
|
||||
`/health` is not an authentication check. See its
|
||||
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
|
||||
No health-monitoring facility is added here.
|
||||
|
||||
## Token And Cache Reporting
|
||||
|
||||
### Provider Response Mapping
|
||||
|
||||
Use common `usage.prompt_tokens`, `usage.completion_tokens`, and
|
||||
`usage.total_tokens` for `PromptTokens`, `CompletionTokens`, and `TotalTokens`.
|
||||
Normalize these additional provider details, relative to `usage`:
|
||||
|
||||
| Public field | OpenRouter | vLLM |
|
||||
| --- | --- | --- |
|
||||
| New `ReasoningTokens` | `completion_tokens_details.reasoning_tokens` | `completion_tokens_details.reasoning_tokens` |
|
||||
| Existing `CachedTokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cached_tokens` |
|
||||
| Existing `CacheWriteTokens` | `prompt_tokens_details.cache_write_tokens` | `prompt_tokens_details.created_cache_tokens` |
|
||||
|
||||
OpenRouter documents these in its
|
||||
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
||||
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
|
||||
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching).
|
||||
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
|
||||
uses `created_cache_tokens`; its
|
||||
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
|
||||
describe local prefix-cache writes, not a billable write charge.
|
||||
|
||||
Preserve `CachedTokens` as the public name and clarify its meaning as cache reads.
|
||||
Use each provider's documented cache-write field, not a sum or guessed alias;
|
||||
a missing nested value is unknown even if an alternative field is present.
|
||||
Retain the generic provider's existing mappings, including its top-level
|
||||
`usage.cache_write_tokens` handling.
|
||||
|
||||
Add `ReasoningTokens` to public/domain token values, presence indicators, observed
|
||||
usage, conversions, and result serialization, and carry it through attempt events,
|
||||
terminal events, and group summaries. Injected clients may supply canonical
|
||||
counts and presence through the existing model-client boundary, retaining its
|
||||
legacy `UsagePresence` fallback; explicit presence distinguishes measured zero.
|
||||
|
||||
### Availability And Optional Metadata
|
||||
|
||||
Omitted/null details are unknown, not zero. Explicit integer zero is known zero.
|
||||
For the newly interpreted provider-specific reasoning and cache-write details,
|
||||
wrong-type, nonintegral, or unrepresentable integer values are also unknown.
|
||||
An unusable new detail does not invalidate otherwise usable content or erase
|
||||
independently available usage. Preserve existing common-field decoding and
|
||||
response framing/content validation; this is a targeted optional-metadata rule,
|
||||
not a general relaxation of the response contract. Keep this interpretation local
|
||||
to the model-client boundary without a general numeric-conversion framework.
|
||||
|
||||
vLLM detail availability depends on version and configuration: prompt details
|
||||
currently require `--enable-prompt-tokens-details`, and reasoning details depend
|
||||
on a configured reasoning parser. Document those reporting settings without
|
||||
configuring or discovering them automatically. Older or differently configured
|
||||
servers may omit details and still produce usable responses. Do not estimate
|
||||
reasoning counts from text or cache writes from input minus cache reads.
|
||||
|
||||
### Count Relationships And Aggregation
|
||||
|
||||
Reasoning is included in completion tokens, not additional output to add to them.
|
||||
OpenRouter explains this in its
|
||||
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens);
|
||||
vLLM similarly includes reasoning detail within generated-token usage. Cache reads
|
||||
and writes are input-token details, not extra input consumption. Preserve the
|
||||
reported total without recomputation or reconciliation.
|
||||
|
||||
Consumers can derive non-reasoning output from consistent, known completion and
|
||||
reasoning counts; no additional stored count is needed. Capturing counts does
|
||||
not capture reasoning text or change generation/caching settings.
|
||||
|
||||
Reuse existing per-field token accounting: aggregates sum known contributions,
|
||||
`Known` identifies available reporting, and `Complete` describes its coverage
|
||||
across dispatched attempts. Preserve initial-generation, retry, and repair
|
||||
observation accounting and terminal-only group deduplication. Existing integer
|
||||
result usage remains distinct from presence-aware observations; a zero result
|
||||
field alone does not establish that the provider reported it.
|
||||
|
||||
### Token Cache-Hit Ratio
|
||||
|
||||
Add a derived helper, not another stored total:
|
||||
|
||||
```go
|
||||
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
|
||||
```
|
||||
|
||||
Compute `CachedTokens / PromptTokens` as a fraction from zero to one. This is the
|
||||
proportion of input tokens served from cache, not request hit frequency or cost
|
||||
savings. For aggregates, divide summed reads by summed input; do not average
|
||||
individual percentages or use `reads / (reads + writes)`.
|
||||
|
||||
Return an available ratio only when input and cache-read counts are known and
|
||||
complete, input is positive, and reads lie between zero and input inclusive.
|
||||
Known zero reads with positive input yields an available zero. Missing,
|
||||
incomplete, zero-denominator, or inconsistent data yields zero and false while
|
||||
preserving raw counts; it does not fail generation or clamp reported values.
|
||||
Cache-write availability is irrelevant.
|
||||
|
||||
The same helper serves attempts, terminals, and group summaries. A group ratio
|
||||
describes its included observations and does not supersede terminal coverage or
|
||||
imply all application work has finished.
|
||||
|
||||
## Cost Reporting
|
||||
|
||||
### Reported Cost And Configured-Zero Precedence
|
||||
|
||||
Initially, only the OpenRouter built-in provider interprets `usage.cost` as the
|
||||
account charge in USD. See its
|
||||
[usage-accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
|
||||
and [USD explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
|
||||
Do not add `cost_details` amounts to that charge or interpret arbitrary providers'
|
||||
fields named `cost`. The generic and vLLM providers do not decode monetary cost.
|
||||
|
||||
Use the cost included in the ordinary response; no extra request or usage lookup
|
||||
is needed. At the model-client boundary, represent an optional USD amount as
|
||||
`*float64`: nil means unknown; a pointer to zero means reported zero. Injected
|
||||
clients may supply the same value regardless of provider selection.
|
||||
|
||||
Resolve cost at completion of each dispatched attempt, outside provider-specific
|
||||
decoding and whether or not decoding occurred or succeeded:
|
||||
|
||||
- A usable provider/model-client-reported cost takes precedence, including zero.
|
||||
- Otherwise, `zero_cost: true` on the effective backend supplies configured USD zero.
|
||||
- Otherwise, cost remains unknown.
|
||||
|
||||
This applies to built-in and injected clients, including failed or canceled
|
||||
dispatched attempts and undecodable responses. Without a selected backend
|
||||
declaration, no configured-zero fallback applies. Non-dispatched work contributes
|
||||
no cost. Do not infer zero cost from local hosting, provider name, credentials,
|
||||
or token counts, or install the declaration automatically on bundled backends.
|
||||
|
||||
An unusable reported amount, including a nonnumeric value or one not representable
|
||||
as a finite `float64`, is unknown before fallback; it does not make an otherwise
|
||||
usable generation fail. Keep available cost from a decoded successful envelope
|
||||
when a later choice/content check fails. Invalid framing and non-2xx bodies supply
|
||||
no provider usage, although the configured-zero fallback still applies to a
|
||||
dispatched attempt. Existing framing, content, and error handling remain intact.
|
||||
|
||||
### Monetary Representation And Sources
|
||||
|
||||
Expose cost separately from token counts and their presence indicators. Use
|
||||
explicit USD field naming such as `CostUSD` and a small public cost value for
|
||||
results, events, and group summaries with:
|
||||
|
||||
- `Value float64`: the available USD amount or subtotal;
|
||||
- `Known bool`: availability;
|
||||
- `Complete bool`: coverage under the selected cost policy; and
|
||||
- `Source`: the contribution-source discriminator below.
|
||||
|
||||
| Source | Meaning |
|
||||
| --- | --- |
|
||||
| `provider` | Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. |
|
||||
| `configured_zero` | Supplied by the explicit backend fallback. |
|
||||
| `mixed` | An aggregate includes both reported and configured-zero contributions. |
|
||||
| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. |
|
||||
|
||||
Use binary floating point for operational telemetry, without rounding to cents
|
||||
or another fixed scale during capture or aggregation. Consumers choose display
|
||||
precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency
|
||||
rounding, pricing inference, and invoice reconciliation are not part of this work.
|
||||
|
||||
### Attempt, Run, And Group Accounting
|
||||
|
||||
A dispatched attempt with usable reported cost or configured zero is known and
|
||||
complete for cost, independently of token completeness. Sum known amounts across
|
||||
all dispatched attempts, including retries and repairs. Aggregate `Known` means
|
||||
at least one known contribution; `Complete` means every dispatched attempt has a
|
||||
cost from either source. Unknown contributions add no amount and make coverage
|
||||
incomplete. Completeness does not assert that every provider reported a charge.
|
||||
|
||||
Aggregate sources by union of known contributions, including zero amounts.
|
||||
Unknown contributions add no source: a provider-only subtotal can be incomplete
|
||||
without becoming `mixed`. A reported zero plus a configured zero is `mixed`.
|
||||
|
||||
Successful results expose cost even without an observer. Result and terminal
|
||||
cost agree on amount, availability, completeness, and source, and enabling
|
||||
observation does not change result cost. For example, an unreported failed
|
||||
attempt followed by a reported USD 0.002 yields an incomplete provider-source
|
||||
subtotal when no zero-cost fallback applies.
|
||||
|
||||
Groups aggregate terminal costs only, counting each run's first terminal once.
|
||||
Combine known amounts, OR `Known`, AND `Complete`, and union sources. Preserve
|
||||
existing empty/no-terminal group semantics: a completed execution with no
|
||||
dispatched attempts yields known, complete zero with empty source; an empty group
|
||||
does likewise, while a group with members but no terminals has unknown,
|
||||
incomplete usage.
|
||||
|
||||
Cost completeness concerns included terminal observations, independently of
|
||||
`TerminalCoverageComplete`. Two completed runs with full cost coverage can yield
|
||||
a complete subtotal while a third observed run lacks a terminal. Conversely,
|
||||
fully reported OpenRouter runs and explicitly zero-cost local runs can yield a
|
||||
complete, mixed-source total. Neither case verifies a provider invoice or future
|
||||
application progress.
|
||||
|
||||
## Internal Responsibilities And Scope Limits
|
||||
|
||||
Provider-specific metadata interpretation and authentication conventions belong
|
||||
in `internal/llm`, sharing common transport, request construction, credentials,
|
||||
framing, and response fields. Use one provider-selection mechanism there.
|
||||
Generation/observation coordination consumes normalized facts or generic
|
||||
diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator
|
||||
method with a generic operation while retaining explicit adapter selection.
|
||||
|
||||
Provider-specific files and straightforward dispatch suffice; no public provider
|
||||
registration API, new `internal/providers` package, or broad mandatory-capability
|
||||
interface is needed. This is configuration-driven selection among supported
|
||||
implementations, not downstream registration of implementations.
|
||||
|
||||
Reuse a small shared cost-accounting component for result and observation paths
|
||||
so dispatched attempts, fallback, and source union follow the same rules even
|
||||
when observation is disabled. Provider code interprets wire metadata; common
|
||||
accounting owns fallback and aggregation. No ConfigKit provenance traversal
|
||||
belongs in ordinary execution.
|
||||
|
||||
Smoke diagnostics retain their separate report/evidence model and are not
|
||||
enrolled in ordinary execution observation. CLI integration, output delivery,
|
||||
persistent history, logging infrastructure, reasoning-text capture, changes to
|
||||
reasoning/caching request behavior, server administration/metrics, additional
|
||||
built-in cost decoders, multiple currencies, configurable token rates, fixed
|
||||
nonzero request prices, and cost estimation remain outside scope.
|
||||
|
||||
## Work Needed And Acceptance
|
||||
|
||||
The following areas describe required changes, not an execution order.
|
||||
|
||||
- Extend the public context API and ordinary/prepared execution wiring while
|
||||
preserving independent nested lifecycles, explicit observers, coordinator
|
||||
selection, and existing preparation/discard/rejected-claim behavior.
|
||||
- Extend backend settings, typed helpers, final-value normalization/validation,
|
||||
capture, prepared metadata, and inspection for `provider` and `zero_cost`.
|
||||
Update the bundled OpenRouter declaration. Preserve configuration provenance
|
||||
and resource-inspection simplicity, and keep metadata out of request bodies.
|
||||
- Consolidate provider behavior in `internal/llm`, adding vLLM usage normalization
|
||||
and the models-list diagnostic alongside OpenRouter interpretation and the
|
||||
shared default path. Preserve common decoding while tolerating unusable new
|
||||
optional details and cost locally to the model-client boundary.
|
||||
- Extend public/domain token types, presence, serialization, conversions, events,
|
||||
and groups with reasoning counts and correct provider-specific cache writes.
|
||||
Add the presence-aware cache-hit helper without changing existing token
|
||||
meanings or introducing a separate ratio accumulator.
|
||||
- Carry optional reported cost and configured-zero policy through attempt
|
||||
completion and shared accounting into results, events, and groups, including
|
||||
source metadata and observer-independent result accounting.
|
||||
|
||||
Select focused offline checks under the [testing policy](../policy/testing.md).
|
||||
Use real internal collaborators and controlled HTTP/injected-client boundaries,
|
||||
with proportionate public contract coverage. Important behaviors include:
|
||||
|
||||
- Context inheritance, registration order, nil/duplicate enrollment, concurrent
|
||||
calls, multiple engines, independent scopes, nested calls, and execution-time
|
||||
selection for a handle prepared elsewhere.
|
||||
- Provider selection independent of backend identity; normalization and unsupported
|
||||
value handling; omitted/null/blank defaults; inheritance, endpoint overrides,
|
||||
explicit false, and inspection without manufactured values or source history.
|
||||
- Provider-default and explicit diagnostic selection, endpoint-only overrides,
|
||||
clearing metadata versus `kind: none`, vLLM acceptance including an empty list,
|
||||
rejection, unrecognized/framing-limited responses, and optional credentials.
|
||||
- Provider token mappings, absent/null/zero details, malformed new optional counts
|
||||
without generation failure, unchanged common-field handling, and responses
|
||||
representative of vLLM servers with detail reporting disabled.
|
||||
- Cache-hit availability, inconsistent counts without generation failure,
|
||||
token-weighted aggregates, independence from cache writes, and no double-counting
|
||||
of reasoning or cache details.
|
||||
- Reported cost precedence, configured-zero fallback on dispatched failure and
|
||||
cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources,
|
||||
zero-valued contributions, and independently incomplete token reporting.
|
||||
- Result/terminal cost agreement with observation enabled or disabled, duplicate
|
||||
terminal deduplication, empty/no-terminal groups, and subtotal completeness
|
||||
distinct from terminal coverage.
|
||||
|
||||
Synthetic fixtures and offline examples suffice; live providers are not required.
|
||||
Keep ownership localized and avoid repeating the same assertions across layers.
|
||||
|
||||
## Documentation And Migration
|
||||
|
||||
When implemented, exact exported contracts belong in GoDoc; the
|
||||
[consumer guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) should show
|
||||
invocation-scoped aggregation, reasoning/cache breakdowns without double-counting,
|
||||
and cost availability, source, and completeness. Document provider wire mappings,
|
||||
optional-metadata handling, and diagnostic response recognition in the
|
||||
[integration contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
The [framework format reference](../formats.md#backend-definitions) owns the new
|
||||
backend fields, default selection, inheritance, and diagnostic overrides.
|
||||
Update internal documentation for changed ownership and flows.
|
||||
|
||||
Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache
|
||||
creation meaning without promising availability on every deployment. Consumers
|
||||
own presentation. Models-list acceptance does not establish inference
|
||||
authorization, and complete cost metadata does not independently verify billing.
|
||||
|
||||
Release notes should call out that authentication objects no longer require
|
||||
`kind` when a provider default supplies it, endpoint-only overrides are supported,
|
||||
and clearing an authentication block restores provider defaults. `kind: none`
|
||||
suppresses that default. Preserve existing explicit adapters without requiring a
|
||||
provider declaration.
|
||||
Reference in New Issue
Block a user