2026-11-22 18:01:31 +00:00
|
|
|
# Invocation Observation And Usage Implementation Plan
|
|
|
|
|
|
|
|
|
|
## Objective And Working Rules
|
|
|
|
|
|
|
|
|
|
Implement the accepted [target-state roadmap](invocation-scoped-token-usage.md).
|
|
|
|
|
It owns scope, consumer intent, and reporting policy. This plan owns concrete
|
|
|
|
|
implementation decisions and the ordered work sequence. Baseline: `a47df4e`.
|
|
|
|
|
|
|
|
|
|
Each numbered stage is one Luna-sized implementation prompt. Execute them in
|
|
|
|
|
order within one end-to-end sprint. Intermediate stages may leave unresolved
|
|
|
|
|
references or failing compilation; do not add compatibility scaffolding merely
|
|
|
|
|
to keep those intermediate states green. There are no intermediate human-review,
|
|
|
|
|
external-publication, or live-provider gates. Run focused checks when their
|
|
|
|
|
dependencies are ready; the completed sprint must pass the final checks.
|
|
|
|
|
|
|
|
|
|
Read [development guidance](../development.md) and the applicable agent,
|
|
|
|
|
architecture, testing, and documentation policies. Follow
|
|
|
|
|
[ADR 0001](../adr/0001-configuration-boundary.md): interpretation and validation
|
|
|
|
|
use final ConfigKit values, not raw sources or historical inspection. Follow the
|
|
|
|
|
shared-settings facade exception without introducing public aliases for runtime
|
|
|
|
|
implementation types. Preserve unrelated work and follow the agent commit policy.
|
|
|
|
|
|
|
|
|
|
Planning and implementing are separate tasks. This document does not authorize
|
|
|
|
|
a release, dependency upgrade, CLI implementation, or changes to published notes.
|
|
|
|
|
|
|
|
|
|
## Selected Design
|
|
|
|
|
|
|
|
|
|
- Store contextual observers as internal `domain.ExecutionObserver` callbacks
|
|
|
|
|
under a context key separate from the recorder, claim sink, and cost state.
|
|
|
|
|
The public helper wraps public event conversion at registration. Internal
|
|
|
|
|
composition preserves explicit-first and inherited-to-new contextual ordering.
|
|
|
|
|
- Add `Provider *string` and `ZeroCost *bool` to shared backend settings and
|
|
|
|
|
`BackendConfig`, with ordinary `provider`/`zero_cost` field names. Resource
|
|
|
|
|
inspection returns declared normalized values; configuration inspection owns
|
|
|
|
|
history. Neither inspection API needs a new provenance model.
|
|
|
|
|
- Capture runtime `Provider string` and `ZeroCost bool` on `domain.Backend` and
|
|
|
|
|
carry them on internal `domain.ExecutionTarget` as local metadata excluded
|
|
|
|
|
from JSON/YAML. Empty provider resolves to `openai-compatible` at runtime.
|
|
|
|
|
Keep these fields out of the root execution-target API and override API:
|
|
|
|
|
backend inspection is the requested public configuration surface. Endpoint-only
|
|
|
|
|
profiles use the generic provider and disabled zero fallback. Existing
|
|
|
|
|
model/endpoint overrides retain the selected backend's metadata.
|
|
|
|
|
- Keep provider ID recognition, response interpretation, and authentication
|
|
|
|
|
adapter resolution in `internal/llm`, using simple functions/dispatch. Config
|
|
|
|
|
validation may call that package's supported-name checks as it already does
|
|
|
|
|
for reserved request fields. No public provider registration mechanism or new
|
|
|
|
|
providers package is needed.
|
|
|
|
|
- Add `ReasoningTokens int` to token values, `ReasoningTokens bool` to presence,
|
|
|
|
|
and `ReasoningTokens ObservedTokenCount` to observed usage. Public token JSON
|
|
|
|
|
uses `reasoning_tokens`. Add the roadmap's two helper methods/functions exactly
|
|
|
|
|
as specified; do not add aliases for cache reads or a stored cache-hit ratio.
|
|
|
|
|
- Define public `CostSource string` constants `CostSourceNone = ""`,
|
|
|
|
|
`CostSourceProvider = "provider"`, `CostSourceConfiguredZero = "configured_zero"`,
|
|
|
|
|
and `CostSourceMixed = "mixed"`. Define `ObservedCost` with `Value float64`,
|
|
|
|
|
`Known bool`, `Complete bool`, and `Source CostSource`, JSON-tagged `value`,
|
|
|
|
|
`known`, `complete`, and `source`, without omitting zero fields.
|
|
|
|
|
- Add `GenerateResponse.CostUSD *float64` (`json:"cost_usd,omitempty"`) and
|
|
|
|
|
`RunResult.CostUSD ObservedCost` (`json:"cost_usd"`). Add `CostUSD ObservedCost`
|
|
|
|
|
to `ExecutionEvent` and `ExecutionGroupSummary`; the latter uses `cost_usd`.
|
|
|
|
|
Define equivalent internal domain values and explicit facade conversions.
|
|
|
|
|
Older result JSON lacking cost decodes to unknown cost, not reported zero.
|
|
|
|
|
- Own pure cost resolution/merge and run-local cost state in
|
|
|
|
|
`internal/observation`. Cost is collected once per dispatched attempt regardless
|
|
|
|
|
of observer selection. The recorder references that same run accumulator for
|
|
|
|
|
terminal cost; it must not independently add attempts a second time. Groups
|
|
|
|
|
reuse the merge rules but remain terminal-only collectors.
|
|
|
|
|
- New optional reasoning/cache-write details use local tolerant integer decoding;
|
|
|
|
|
cost uses local tolerant finite-float decoding. A typed `*int`/`*float64` field
|
|
|
|
|
in the main envelope would fail the whole decode for malformed optional data.
|
|
|
|
|
Preserve existing common token decoding, framing, content validation, and
|
|
|
|
|
error identities. Do not add a general numeric coercion framework.
|
|
|
|
|
|
|
|
|
|
## Stage 1 — Context Helper And Ordinary Run Selection
|
|
|
|
|
|
2026-11-22 18:07:17 +00:00
|
|
|
**Completed.**
|
|
|
|
|
|
2026-11-22 18:01:31 +00:00
|
|
|
Touch `observations.go`, the root `Engine.Run` in `engine.go`, and context support
|
|
|
|
|
in `internal/observation/recorder.go` or a focused adjacent file.
|
|
|
|
|
|
|
|
|
|
- Implement `WithExecutionObserver(context.Context, ExecutionObserver)
|
|
|
|
|
context.Context` using a private internal observer-selection key. Nil is a
|
|
|
|
|
no-op. Compose inherited callbacks before the new callback using the existing
|
|
|
|
|
composition facility; keep registration state immutable.
|
|
|
|
|
- In ordinary `Run`, compose a converted explicit request observer with the
|
|
|
|
|
contextual selection before deciding whether to create a recorder. Preserve
|
|
|
|
|
current start timing, run IDs, public error mapping, timeout finalization,
|
|
|
|
|
permit release, and terminal delivery. With neither observer, preserve the
|
|
|
|
|
existing no-observation path.
|
|
|
|
|
- Do not mask the contextual key in `WithoutExecutionState`; continue masking
|
|
|
|
|
inherited recorder and claim state. Each nested public call creates its own
|
|
|
|
|
recorder. Do not modify `Coordinate` to append contextual selection: its
|
|
|
|
|
explicit request/coordinator composition already leads to ordinary `Run`.
|
|
|
|
|
- Add focused public tests for no observer, contextual-only selection, explicit
|
|
|
|
|
plus nested context ordering, nil registration, unrelated contexts, and two
|
|
|
|
|
independent run IDs from a nested injected-client call. Group duplication
|
|
|
|
|
uses existing terminal deduplication, not callback equality checks.
|
|
|
|
|
|
|
|
|
|
Validation when buildable: root observation/boundary contract tests and focused
|
|
|
|
|
internal observer tests. No provider requests are needed.
|
|
|
|
|
|
|
|
|
|
## Stage 2 — Execution-Time Context For Prepared Handles
|
|
|
|
|
|
2026-11-22 18:08:28 +00:00
|
|
|
**Completed.**
|
|
|
|
|
|
2026-11-22 18:01:31 +00:00
|
|
|
Touch root `RunPrepared`, internal `usecase/prepared_execution.go`, and observer
|
|
|
|
|
conversion only where required.
|
|
|
|
|
|
|
|
|
|
- Retain only the explicitly supplied request observer in the prepared payload.
|
|
|
|
|
Do not snapshot contextual observers during `PrepareExecution` or capture
|
|
|
|
|
preparation contexts in the handle.
|
|
|
|
|
- After a successful claim, compose that observer with the contextual selection
|
|
|
|
|
from the execution context before recorder creation. Preserve the private
|
|
|
|
|
claim sink so root terminal delivery still follows public error mapping and
|
|
|
|
|
finalized timeout attribution.
|
|
|
|
|
- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle
|
|
|
|
|
rejection, and existing payload clearing. Do not expose private callbacks in
|
|
|
|
|
`Details` or create a second lifecycle around the claimed execution.
|
|
|
|
|
- Test a handle prepared in scope A and executed in B: retained explicit
|
|
|
|
|
observer and B receive events; A does not. Include a context-only execution,
|
|
|
|
|
rejected claims, and a continuation selected by `Coordinate` to protect its
|
|
|
|
|
explicit-before-context order.
|
|
|
|
|
|
|
|
|
|
Validation: prepared execution and coordinate observation contract tests.
|
|
|
|
|
|
|
|
|
|
## Stage 3 — Backend Configuration And Provider Selection
|
|
|
|
|
|
2026-11-22 18:11:10 +00:00
|
|
|
**Completed.**
|
|
|
|
|
|
2026-11-22 18:01:31 +00:00
|
|
|
Touch `internal/settings/settings.go`, `backends.go`, configuration ownership,
|
|
|
|
|
resolution/validation code, and the provider-selection functions in `internal/llm`.
|
|
|
|
|
|
|
|
|
|
- Add the two pointer fields and presence-preserving source-helper encoding.
|
|
|
|
|
Nil omits a field; false and blank pointers remain supplied values. Raw
|
|
|
|
|
ConfigKit mappings remain the way to supply explicit null.
|
|
|
|
|
- Normalize supplied provider strings with trimming/lowercasing in the existing
|
|
|
|
|
final resolution callback and record normalizations using existing ConfigKit
|
|
|
|
|
APIs. Do not materialize the runtime default into the settings object.
|
|
|
|
|
- Recognize exactly `openai-compatible`, `openrouter`, and `vllm`; empty means
|
|
|
|
|
generic. Reject other nonempty values through existing final configuration
|
|
|
|
|
validation/error wrapping. Do not validate overwritten contributions.
|
|
|
|
|
- Validate `zero_cost` as the typed Boolean setting; absent/null/false disables
|
|
|
|
|
fallback. Do not infer it or make `LocalBackend` set it automatically.
|
|
|
|
|
- Relax authentication-object validation to allow absent/null kind, retain
|
|
|
|
|
existing kind normalization and supported explicit kinds, and add
|
|
|
|
|
`vllm_models`. A missing kind with no provider default remains an unsupported
|
|
|
|
|
diagnostic selection, not a new load-time requirement for a provider.
|
|
|
|
|
- Cover layered final-value validation, inheritance, null clearing, normalization,
|
|
|
|
|
unsupported names, helper presence, and absence of invented defaults.
|
|
|
|
|
|
|
|
|
|
Validation: focused configuration/settings/source-helper tests.
|
|
|
|
|
|
|
|
|
|
## Stage 4 — Runtime Capture, Freezing, And Backend Inspection
|
|
|
|
|
|
|
|
|
|
Touch `internal/configuration/targets.go`, `ownership.go`, `resources.go`, domain
|
|
|
|
|
backend/target values and clones, and `bundled/openrouter.yaml` under that package.
|
|
|
|
|
|
|
|
|
|
- Copy declared settings into owned records and inspection results. Preserve
|
|
|
|
|
explicit false while allowing absent/null collapse in resource inspection.
|
|
|
|
|
Leave full null/omission history to `InspectConfiguration`.
|
|
|
|
|
- Capture effective provider and zero policy on the internal backend and the
|
|
|
|
|
backend-derived execution target. Ensure scalar copying through profile
|
|
|
|
|
capture, runtime overrides, prepared details/payloads, and repair requests.
|
|
|
|
|
Retain backend metadata when a profile/request overrides only its endpoint.
|
|
|
|
|
- Keep root public execution targets and provider HTTP bodies unchanged. Audit
|
|
|
|
|
serialization/copy helpers so excluded internal fields are not accidentally
|
|
|
|
|
lost by a JSON-based copy. Do not recapture policy from configuration during
|
|
|
|
|
`RunPrepared` or traverse provenance during execution.
|
|
|
|
|
- Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost
|
|
|
|
|
defaults or new backend instances. Keep API keys and default profiles unchanged.
|
|
|
|
|
- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only
|
|
|
|
|
profiles, frozen prepared policy, and inspection without manufactured values.
|
|
|
|
|
|
|
|
|
|
Validation: configuration capture, backend inspection, bundle, and preparation tests.
|
|
|
|
|
|
|
|
|
|
## Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults
|
|
|
|
|
|
|
|
|
|
Current provider branches occur in `internal/usecase/smoke_authentication.go` and
|
|
|
|
|
`smoke_selection.go`, including `ProbeOpenRouterKey` and endpoint derivation.
|
|
|
|
|
|
|
|
|
|
- Replace the OpenRouter-specific collaborator operation with a generic
|
|
|
|
|
authentication-probe operation owned by `internal/llm`. Carry a resolved
|
|
|
|
|
adapter selection to it; leave usecase coordination provider-neutral.
|
|
|
|
|
- Resolve adapter kind and destination with pure provider-boundary functions.
|
|
|
|
|
Explicit kind wins, absent/null kind uses the provider default, and `none`
|
|
|
|
|
is inapplicable. Explicit endpoint wins; otherwise use the adapter-relative
|
|
|
|
|
fallback. Missing selection remains unsupported. Do not rewrite loaded
|
|
|
|
|
settings, captured declared smoke metadata, or inspection provenance.
|
|
|
|
|
- OpenRouter defaults to `openrouter_key`; generic has no default. Preserve
|
|
|
|
|
explicit OpenRouter adapters on any backend and existing HTTP evidence,
|
|
|
|
|
bounded recognition, credential, cancellation, and overflow handling.
|
|
|
|
|
- Clear an authentication block to restore the provider default; an endpoint-only
|
|
|
|
|
block may use that default. Preserve reachability and inference selection.
|
|
|
|
|
- Update affected interfaces, engine assembly, and test doubles. Test default,
|
|
|
|
|
explicit/none, endpoint-only, cleared-block, and unsupported combinations at
|
|
|
|
|
their owning boundaries rather than duplicating every permutation publicly.
|
|
|
|
|
|
|
|
|
|
Validation: smoke selection/authentication tests and existing OpenRouter contract tests.
|
|
|
|
|
|
|
|
|
|
## Stage 6 — vLLM Models Authentication Adapter
|
|
|
|
|
|
|
|
|
|
- Implement `vllm_models` beside the OpenRouter adapter, sharing generic HTTP
|
|
|
|
|
execution and bounded recognition helpers rather than duplicating transport.
|
|
|
|
|
Select it by default for `vllm` and allow explicit use on other providers.
|
|
|
|
|
- Resolve `models` beneath the configured API base, preserving base prefixes;
|
|
|
|
|
honor an explicit full diagnostic endpoint. Do not hard-code a host or probe
|
|
|
|
|
`/health`. Use existing optional credential rules without forcing a key.
|
|
|
|
|
- Recognize exactly the roadmap's HTTP 200 object/list/data-array shape,
|
|
|
|
|
including empty arrays. HTTP 401 is rejection; other/unrecognized responses
|
|
|
|
|
follow existing unknown/failure reporting. Do not inspect list members for
|
|
|
|
|
model existence, inference permissions, or authentication enforcement.
|
|
|
|
|
- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other
|
|
|
|
|
statuses, optional credentials, overridden destinations, and `kind: none`.
|
|
|
|
|
Ensure unsupported injected HTTP diagnostics still do not fall back to a
|
|
|
|
|
built-in transport.
|
|
|
|
|
|
|
|
|
|
Validation: focused adapter and smoke coordinator tests with controlled HTTP.
|
|
|
|
|
|
|
|
|
|
## Stage 7 — Reasoning Values And Provider-Specific Token Decoding
|
|
|
|
|
|
|
|
|
|
Touch public/domain token declarations, facade conversions, and
|
|
|
|
|
`internal/llm/openai_compatible_client.go` plus focused provider files.
|
|
|
|
|
|
|
|
|
|
- Add `ReasoningTokens` to `TokenUsage`, `TokenUsagePresence`, and observed usage
|
|
|
|
|
on both sides of the facade; include JSON field/conversions and mutable-response
|
|
|
|
|
ownership where applicable. Do not rename `CachedTokens`.
|
|
|
|
|
- Keep common counts and cached reads on their existing decode path. For
|
|
|
|
|
OpenRouter, read reasoning from `completion_tokens_details.reasoning_tokens`
|
|
|
|
|
and writes from `prompt_tokens_details.cache_write_tokens`; for vLLM, read
|
|
|
|
|
the same reasoning path and `prompt_tokens_details.created_cache_tokens`.
|
|
|
|
|
Generic keeps its existing top-level write mapping and does not interpret the
|
|
|
|
|
new reasoning field. Ignore other providers' alternative write locations;
|
|
|
|
|
do not add/fallback between the nested and top-level values.
|
|
|
|
|
- Decode newly interpreted integer fields independently with local raw JSON
|
|
|
|
|
helpers. Absent/null, wrong-type, fractional, or int-overflow values become
|
|
|
|
|
unknown. Do not coerce strings. Retain independently valid fields; adding a
|
|
|
|
|
typed optional field must not create a new whole-response failure. Existing
|
|
|
|
|
malformed common fields and framing/content failures retain their behavior.
|
|
|
|
|
- Preserve decoded usage on later missing-content/choice failure as today.
|
|
|
|
|
Do not infer missing reasoning or writes, enforce new nonnegative restrictions,
|
|
|
|
|
change generation controls, or collect reasoning text.
|
|
|
|
|
- Test provider-specific raw fixtures with conflicting write locations,
|
|
|
|
|
partial/malformed new fields, known zero, and disabled-detail vLLM responses.
|
|
|
|
|
Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap.
|
|
|
|
|
|
|
|
|
|
Validation: model-client parsing tests and public token JSON round trips.
|
|
|
|
|
|
|
|
|
|
## Stage 8 — Token Aggregation And Cache-Hit Helper
|
|
|
|
|
|
|
|
|
|
Touch observation usage accumulators/recorder/group, runner `addTokenUsage`,
|
|
|
|
|
public event/group conversions, and related fixtures.
|
|
|
|
|
|
|
|
|
|
- Include reasoning counts in existing successful-result summation, attempt and
|
|
|
|
|
terminal usage, group merging, and explicit/legacy injected presence behavior.
|
|
|
|
|
Do not replace legacy result token sums with all-attempt observation sums.
|
|
|
|
|
- Implement `ObservedTokenUsage.CacheHitRatio() (float64, bool)` using the
|
|
|
|
|
roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid
|
|
|
|
|
data returns `(0, false)`; known zero reads with positive input returns `(0, true)`.
|
|
|
|
|
Divide totals as floats, not integer division. Do not persist or average ratios.
|
|
|
|
|
- Cover per-field incomplete reasoning, injected known zero, retry/repair sums,
|
|
|
|
|
duplicate group enrollment, weighted aggregate ratios, zero denominator,
|
|
|
|
|
inconsistent counts without generation failure, and independence from writes.
|
|
|
|
|
- Preserve reported total tokens: reasoning and cache details are subsets, not
|
|
|
|
|
extra values to add into input/output/total counts.
|
|
|
|
|
|
|
|
|
|
Validation: focused token accumulator/group tests and public observation contracts.
|
|
|
|
|
|
|
|
|
|
## Stage 9 — Public Cost Values And Shared Cost Algebra
|
|
|
|
|
|
|
|
|
|
Create public cost declarations in a focused root file and matching domain
|
|
|
|
|
values; add the selected fields to responses, results, events, and summaries.
|
|
|
|
|
|
|
|
|
|
- Implement the selected cost/source declarations and conversions. Copy optional
|
|
|
|
|
amount pointers at the injected-client conversion boundary so caller mutation
|
|
|
|
|
cannot change retained data. Include cost in custom result JSON marshal/unmarshal
|
|
|
|
|
paths as needed, preserving existing duration and omission behavior elsewhere.
|
|
|
|
|
- Implement pure internal cost resolution: finite reported amount first, then
|
|
|
|
|
configured zero, else unknown. A reported zero remains provider-sourced.
|
|
|
|
|
Negative finite reported amounts are not newly prohibited. Unknown cost is
|
|
|
|
|
zero-valued with false Known/Complete and empty source.
|
|
|
|
|
- Implement one small `CostAccumulator` in `internal/observation` that merges
|
|
|
|
|
known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot
|
|
|
|
|
is known/complete zero; its first unknown contribution must reset that empty
|
|
|
|
|
special case rather than accidentally making unknown cost known. Include
|
|
|
|
|
known zero contributions in source union. Unknown contributions add no source.
|
|
|
|
|
- Allow merging aggregate costs for groups with the same helper; leave the
|
|
|
|
|
existing no-terminal group special case to group coverage logic.
|
|
|
|
|
- Unit-test precedence, source union including zeros, all-unknown/empty cases,
|
|
|
|
|
completeness, and approximate floating-point sums. Add JSON tests for old
|
|
|
|
|
result JSON without cost, current round trips, and optional client zero.
|
|
|
|
|
|
|
|
|
|
Validation: pure accounting and public serialization tests; no pricing library.
|
|
|
|
|
|
|
|
|
|
## Stage 10 — Optional Cost At The Model-Client Boundary
|
|
|
|
|
|
|
|
|
|
- Decode OpenRouter `usage.cost` independently using local tolerant finite-float
|
|
|
|
|
handling. Missing/null/string/object/overflow values are unknown; invalid
|
|
|
|
|
framing or common envelope fields retain existing failures. Generic/vLLM do
|
|
|
|
|
not interpret the field. Do not consume `cost_details` or send a new usage flag.
|
|
|
|
|
- Carry raw optional cost on built-in `AttemptOutcome` and `GenerateResponse`
|
|
|
|
|
even when a later decoded-content check fails. Non-2xx/invalid framing remain
|
|
|
|
|
without provider usage. Do not apply zero-cost fallback inside provider code.
|
|
|
|
|
- Preserve optional cost supplied through an injected client. The adapter already
|
|
|
|
|
converts non-nil responses alongside errors: retain cost for accounting even
|
|
|
|
|
then, without changing existing token treatment or public error/result rules.
|
|
|
|
|
Normalize nonfinite injected cost to unknown before applying fallback.
|
|
|
|
|
- Test cost independent of token fields, usable zero, malformed optional cost
|
|
|
|
|
without generation failure, ignored cost on generic/vLLM, and reported cost
|
|
|
|
|
retained alongside a classified injected error.
|
|
|
|
|
|
|
|
|
|
Validation: provider decoder and injected-adapter tests.
|
|
|
|
|
|
|
|
|
|
## Stage 11 — Per-Attempt Cost Collection And Observation
|
|
|
|
|
|
|
|
|
|
Touch generation `recordAttempt` and built-in/injected execution paths,
|
|
|
|
|
observation attempt data/events, recorder terminal cost, and group aggregation.
|
|
|
|
|
|
|
|
|
|
- At every attempt completion, if dispatched, resolve raw cost against the
|
|
|
|
|
frozen target's ZeroCost and add it exactly once to an invocation accumulator.
|
|
|
|
|
Do this for transport errors, eligible retry statuses, cancellation after
|
|
|
|
|
dispatch, malformed bodies, and missing content as well as success. A failed
|
|
|
|
|
acquisition/pre-dispatch cancellation contributes nothing.
|
|
|
|
|
- Keep cost collection independent of the optional recorder and diagnostic
|
|
|
|
|
evidence capture flag. Preserve permit-release-before-callback behavior.
|
|
|
|
|
Do not count retry waits, terminal events, or generation responses again.
|
|
|
|
|
- Carry resolved cost on attempt events. Recorder terminal cost reads the same
|
|
|
|
|
accumulator; groups merge only first terminal costs via shared algebra.
|
|
|
|
|
Extend public event/group projection without changing existing token totals.
|
|
|
|
|
- Introduce private cost-state context access. Extend `WithoutExecutionState` to
|
|
|
|
|
mask inherited cost state, but not contextual observer selection. Smoke
|
|
|
|
|
diagnostics must continue masking private execution accounting and emit no
|
|
|
|
|
ordinary lifecycle or enclosing-run cost contributions.
|
|
|
|
|
- Exercise controlled retry/error sequences and zero-cost fallback without an
|
|
|
|
|
observer. Test duplicate terminal delivery and independence of group terminal
|
|
|
|
|
coverage from cost completeness. Do not retain an attempt history for cost.
|
|
|
|
|
|
|
|
|
|
Validation: generation, observation recorder/group, and diagnostic-boundary tests.
|
|
|
|
|
|
|
|
|
|
## Stage 12 — Ordinary Run Cost Lifetime And Result Accounting
|
|
|
|
|
|
|
|
|
|
- Create/install a new private accumulator at the public ordinary Run boundary,
|
|
|
|
|
before optional recorder creation. Pass that same instance to the recorder.
|
|
|
|
|
Ensure direct internal runner execution creates a local accumulator if one
|
|
|
|
|
was not supplied, without replacing one supplied by the owning facade.
|
|
|
|
|
- Retain it through initial generation and all repair calls. Set successful
|
|
|
|
|
`domain.RunResult.CostUSD` from its snapshot, not by summing responses. Keep
|
|
|
|
|
the legacy `Usage` success-response sum and error handling unchanged.
|
|
|
|
|
- Ensure errors still return no partial public `RunResult`; available cost is
|
|
|
|
|
reported in terminal observations. Runs rejected before dispatch have the
|
|
|
|
|
established empty aggregate, not a configured-zero contribution source.
|
|
|
|
|
- Test a retry with unreported cost followed by a reported charge, mixed sources
|
|
|
|
|
within repairs, reported-cost precedence over zero policy, and identical
|
|
|
|
|
successful cost metadata with observation enabled/disabled.
|
|
|
|
|
|
|
|
|
|
Validation: assembled ordinary run/reliable execution contract tests.
|
|
|
|
|
|
|
|
|
|
## Stage 13 — Prepared Cost Lifetime And Nested Isolation
|
|
|
|
|
|
|
|
|
|
- In internal RunPrepared, after successful claim create/install its own cost
|
|
|
|
|
accumulator before recorder creation and generation. Use the same result and
|
|
|
|
|
terminal accounting path as ordinary Run. Do not store an accumulator or
|
|
|
|
|
contextual observer in the reusable preparation details/private ready payload.
|
|
|
|
|
- The claimed handle uses its frozen backend Provider/ZeroCost, even when run
|
|
|
|
|
from another invocation scope. Preserve admission, deadline attribution,
|
|
|
|
|
single-claim semantics, discard behavior, and cleared callbacks/credentials.
|
|
|
|
|
- Verify ordinary/prepared child runs launched from injected clients or
|
|
|
|
|
prepared callbacks do not inherit the parent's accumulator, even when both
|
|
|
|
|
inherit one contextual group observer. Each terminal contributes once to the
|
|
|
|
|
invocation total; parent cost includes only its own dispatched generations.
|
|
|
|
|
- Cover prepared zero-cost failure and repair, rejected claims without events,
|
|
|
|
|
independent contexts on a reused engine, and diagnostic calls nested in a run
|
|
|
|
|
without cost/event leakage.
|
|
|
|
|
|
|
|
|
|
Validation: prepared and nested-boundary contracts, including race-enabled focused tests.
|
|
|
|
|
|
|
|
|
|
## Stage 14 — End-To-End Consumer Regression Coverage
|
|
|
|
|
|
|
|
|
|
- Add one representative offline invocation workflow using two engines and
|
|
|
|
|
concurrent runs under one group. Use controlled transport/injected clients
|
|
|
|
|
for OpenRouter reported cost, vLLM details/configured zero, a retry, and a
|
|
|
|
|
failed operation after available usage. Join application-owned goroutines
|
|
|
|
|
before taking the final snapshot.
|
|
|
|
|
- Assert independently predictable terminal membership, sums, sources, token
|
|
|
|
|
presence/completeness, and weighted ratio (or its unavailability when an
|
|
|
|
|
attempt leaves input/cache-read reporting incomplete). Compare floating-point
|
|
|
|
|
sums with tolerance; do not assert map order, run-ID bytes, or exact timing.
|
|
|
|
|
- Cover a snapshot with completed fully costed members and one started member:
|
|
|
|
|
cost completeness concerns the included terminals while terminal coverage is
|
|
|
|
|
false. Cover empty and members-with-no-terminals snapshots separately.
|
|
|
|
|
- Reuse narrower tests from earlier stages rather than reproduce every case in
|
|
|
|
|
this workflow. Keep observations metadata-only; do not add bodies, credentials,
|
|
|
|
|
URLs, or error objects to the event contract.
|
|
|
|
|
|
|
|
|
|
Validation: focused new integration tests plus `go test -race ./...` when the
|
|
|
|
|
implementation has reached its final buildable state.
|
|
|
|
|
|
|
|
|
|
## Stage 15 — Canonical Documentation And Final Verification
|
|
|
|
|
|
|
|
|
|
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and
|
|
|
|
|
sources, client ownership, and observer/prepared lifetimes. Update stale
|
|
|
|
|
authentication Kind requirements in shared settings and public docs.
|
|
|
|
|
- Update the consumer guide and maintained offline observation example for
|
|
|
|
|
invocation scoping, cache/reasoning subsets, known/complete cost, configured
|
|
|
|
|
zeros, sources, and failure-path reporting. Maintain semantic example tests.
|
|
|
|
|
- Update `docs/formats.md`, the OpenAI-compatible integration contract, and
|
|
|
|
|
internal overview/observation/generation/model-client/runner documentation for
|
|
|
|
|
actual ownership and flow. Explain provider defaults versus declared inspection,
|
|
|
|
|
vLLM reporting settings, and the exact diagnostic response recognition.
|
|
|
|
|
- Document migration from required authentication kind and block-clearing
|
|
|
|
|
semantics. Record the change in the next unpublished release note when such a
|
|
|
|
|
note exists; otherwise leave release-note creation to release preparation.
|
|
|
|
|
Do not edit published v0.24.0 notes or tag/publish anything in this sprint.
|
|
|
|
|
- Use the newest already installed Go toolchain per the development guide,
|
|
|
|
|
`GOTOOLCHAIN=local`, and `GOWORK=off`. Format changed Go files, run
|
|
|
|
|
`go test -race ./...`, `go vet ./...`, and `git diff --check`. Reuse successful
|
|
|
|
|
results for unchanged content; do not add redundant builds or live-provider
|
|
|
|
|
checks. Verify local Markdown links and maintained examples through their
|
|
|
|
|
normal offline tests. Review the entire diff for unrelated changes and secrets.
|
|
|
|
|
- Audit completion against every target-state section, especially no-observer
|
|
|
|
|
cost, failed dispatched attempts, optional-field tolerance, prepared policy
|
|
|
|
|
freezing, and nested/diagnostic separation. Resolve remaining implementation
|
|
|
|
|
defects before reporting sprint completion. Roadmap retirement is later
|
|
|
|
|
housekeeping, not part of this implementation plan.
|
|
|
|
|
|
|
|
|
|
There are no unresolved product decisions required to begin these stages.
|