26 KiB
Invocation Observation And Usage Implementation Plan
Objective And Working Rules
Implement the accepted target-state roadmap.
It owns scope, consumer intent, and reporting policy. This plan owns concrete
implementation decisions and the ordered work sequence. Baseline: a47df4e.
Each numbered stage is one Luna-sized implementation prompt. Execute them in order within one end-to-end sprint. Intermediate stages may leave unresolved references or failing compilation; do not add compatibility scaffolding merely to keep those intermediate states green. There are no intermediate human-review, external-publication, or live-provider gates. Run focused checks when their dependencies are ready; the completed sprint must pass the final checks.
Read development guidance and the applicable agent, architecture, testing, and documentation policies. Follow ADR 0001: interpretation and validation use final ConfigKit values, not raw sources or historical inspection. Follow the shared-settings facade exception without introducing public aliases for runtime implementation types. Preserve unrelated work and follow the agent commit policy.
Planning and implementing are separate tasks. This document does not authorize a release, dependency upgrade, CLI implementation, or changes to published notes.
Selected Design
- Store contextual observers as internal
domain.ExecutionObservercallbacks under a context key separate from the recorder, claim sink, and cost state. The public helper wraps public event conversion at registration. Internal composition preserves explicit-first and inherited-to-new contextual ordering. - Add
Provider *stringandZeroCost *boolto shared backend settings andBackendConfig, with ordinaryprovider/zero_costfield names. Resource inspection returns declared normalized values; configuration inspection owns history. Neither inspection API needs a new provenance model. - Capture runtime
Provider stringandZeroCost boolondomain.Backendand carry them on internaldomain.ExecutionTargetas local metadata excluded from JSON/YAML. Empty provider resolves toopenai-compatibleat runtime. Keep these fields out of the root execution-target API and override API: backend inspection is the requested public configuration surface. Endpoint-only profiles use the generic provider and disabled zero fallback. Existing model/endpoint overrides retain the selected backend's metadata. - Keep provider ID recognition, response interpretation, and authentication
adapter resolution in
internal/llm, using simple functions/dispatch. Config validation may call that package's supported-name checks as it already does for reserved request fields. No public provider registration mechanism or new providers package is needed. - Add
ReasoningTokens intto token values,ReasoningTokens boolto presence, andReasoningTokens ObservedTokenCountto observed usage. Public token JSON usesreasoning_tokens. Add the roadmap's two helper methods/functions exactly as specified; do not add aliases for cache reads or a stored cache-hit ratio. - Define public
CostSource stringconstantsCostSourceNone = "",CostSourceProvider = "provider",CostSourceConfiguredZero = "configured_zero", andCostSourceMixed = "mixed". DefineObservedCostwithValue float64,Known bool,Complete bool, andSource CostSource, JSON-taggedvalue,known,complete, andsource, without omitting zero fields. - Add
GenerateResponse.CostUSD *float64(json:"cost_usd,omitempty") andRunResult.CostUSD ObservedCost(json:"cost_usd"). AddCostUSD ObservedCosttoExecutionEventandExecutionGroupSummary; the latter usescost_usd. Define equivalent internal domain values and explicit facade conversions. Older result JSON lacking cost decodes to unknown cost, not reported zero. - Own pure cost resolution/merge and run-local cost state in
internal/observation. Cost is collected once per dispatched attempt regardless of observer selection. The recorder references that same run accumulator for terminal cost; it must not independently add attempts a second time. Groups reuse the merge rules but remain terminal-only collectors. - New optional reasoning/cache-write details use local tolerant integer decoding;
cost uses local tolerant finite-float decoding. A typed
*int/*float64field in the main envelope would fail the whole decode for malformed optional data. Preserve existing common token decoding, framing, content validation, and error identities. Do not add a general numeric coercion framework.
Stage 1 — Context Helper And Ordinary Run Selection
Completed.
Touch observations.go, the root Engine.Run in engine.go, and context support
in internal/observation/recorder.go or a focused adjacent file.
- Implement
WithExecutionObserver(context.Context, ExecutionObserver) context.Contextusing a private internal observer-selection key. Nil is a no-op. Compose inherited callbacks before the new callback using the existing composition facility; keep registration state immutable. - In ordinary
Run, compose a converted explicit request observer with the contextual selection before deciding whether to create a recorder. Preserve current start timing, run IDs, public error mapping, timeout finalization, permit release, and terminal delivery. With neither observer, preserve the existing no-observation path. - Do not mask the contextual key in
WithoutExecutionState; continue masking inherited recorder and claim state. Each nested public call creates its own recorder. Do not modifyCoordinateto append contextual selection: its explicit request/coordinator composition already leads to ordinaryRun. - Add focused public tests for no observer, contextual-only selection, explicit plus nested context ordering, nil registration, unrelated contexts, and two independent run IDs from a nested injected-client call. Group duplication uses existing terminal deduplication, not callback equality checks.
Validation when buildable: root observation/boundary contract tests and focused internal observer tests. No provider requests are needed.
Stage 2 — Execution-Time Context For Prepared Handles
Completed.
Touch root RunPrepared, internal usecase/prepared_execution.go, and observer
conversion only where required.
- Retain only the explicitly supplied request observer in the prepared payload.
Do not snapshot contextual observers during
PrepareExecutionor capture preparation contexts in the handle. - After a successful claim, compose that observer with the contextual selection from the execution context before recorder creation. Preserve the private claim sink so root terminal delivery still follows public error mapping and finalized timeout attribution.
- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle
rejection, and existing payload clearing. Do not expose private callbacks in
Detailsor create a second lifecycle around the claimed execution. - Test a handle prepared in scope A and executed in B: retained explicit
observer and B receive events; A does not. Include a context-only execution,
rejected claims, and a continuation selected by
Coordinateto protect its explicit-before-context order.
Validation: prepared execution and coordinate observation contract tests.
Stage 3 — Backend Configuration And Provider Selection
Completed.
Touch internal/settings/settings.go, backends.go, configuration ownership,
resolution/validation code, and the provider-selection functions in internal/llm.
- Add the two pointer fields and presence-preserving source-helper encoding. Nil omits a field; false and blank pointers remain supplied values. Raw ConfigKit mappings remain the way to supply explicit null.
- Normalize supplied provider strings with trimming/lowercasing in the existing final resolution callback and record normalizations using existing ConfigKit APIs. Do not materialize the runtime default into the settings object.
- Recognize exactly
openai-compatible,openrouter, andvllm; empty means generic. Reject other nonempty values through existing final configuration validation/error wrapping. Do not validate overwritten contributions. - Validate
zero_costas the typed Boolean setting; absent/null/false disables fallback. Do not infer it or makeLocalBackendset it automatically. - Relax authentication-object validation to allow absent/null kind, retain
existing kind normalization and supported explicit kinds, and add
vllm_models. A missing kind with no provider default remains an unsupported diagnostic selection, not a new load-time requirement for a provider. - Cover layered final-value validation, inheritance, null clearing, normalization, unsupported names, helper presence, and absence of invented defaults.
Validation: focused configuration/settings/source-helper tests.
Stage 4 — Runtime Capture, Freezing, And Backend Inspection
Completed.
Touch internal/configuration/targets.go, ownership.go, resources.go, domain
backend/target values and clones, and bundled/openrouter.yaml under that package.
- Copy declared settings into owned records and inspection results. Preserve
explicit false while allowing absent/null collapse in resource inspection.
Leave full null/omission history to
InspectConfiguration. - Capture effective provider and zero policy on the internal backend and the backend-derived execution target. Ensure scalar copying through profile capture, runtime overrides, prepared details/payloads, and repair requests. Retain backend metadata when a profile/request overrides only its endpoint.
- Keep root public execution targets and provider HTTP bodies unchanged. Audit
serialization/copy helpers so excluded internal fields are not accidentally
lost by a JSON-based copy. Do not recapture policy from configuration during
RunPreparedor traverse provenance during execution. - Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost defaults or new backend instances. Keep API keys and default profiles unchanged.
- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only profiles, frozen prepared policy, and inspection without manufactured values.
Validation: configuration capture, backend inspection, bundle, and preparation tests.
Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults
Completed.
Current provider branches occur in internal/usecase/smoke_authentication.go and
smoke_selection.go, including ProbeOpenRouterKey and endpoint derivation.
- Replace the OpenRouter-specific collaborator operation with a generic
authentication-probe operation owned by
internal/llm. Carry a resolved adapter selection to it; leave usecase coordination provider-neutral. - Resolve adapter kind and destination with pure provider-boundary functions.
Explicit kind wins, absent/null kind uses the provider default, and
noneis inapplicable. Explicit endpoint wins; otherwise use the adapter-relative fallback. Missing selection remains unsupported. Do not rewrite loaded settings, captured declared smoke metadata, or inspection provenance. - OpenRouter defaults to
openrouter_key; generic has no default. Preserve explicit OpenRouter adapters on any backend and existing HTTP evidence, bounded recognition, credential, cancellation, and overflow handling. - Clear an authentication block to restore the provider default; an endpoint-only block may use that default. Preserve reachability and inference selection.
- Update affected interfaces, engine assembly, and test doubles. Test default, explicit/none, endpoint-only, cleared-block, and unsupported combinations at their owning boundaries rather than duplicating every permutation publicly.
Validation: smoke selection/authentication tests and existing OpenRouter contract tests.
Stage 6 — vLLM Models Authentication Adapter
Completed.
- Implement
vllm_modelsbeside the OpenRouter adapter, sharing generic HTTP execution and bounded recognition helpers rather than duplicating transport. Select it by default forvllmand allow explicit use on other providers. - Resolve
modelsbeneath the configured API base, preserving base prefixes; honor an explicit full diagnostic endpoint. Do not hard-code a host or probe/health. Use existing optional credential rules without forcing a key. - Recognize exactly the roadmap's HTTP 200 object/list/data-array shape, including empty arrays. HTTP 401 is rejection; other/unrecognized responses follow existing unknown/failure reporting. Do not inspect list members for model existence, inference permissions, or authentication enforcement.
- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other
statuses, optional credentials, overridden destinations, and
kind: none. Ensure unsupported injected HTTP diagnostics still do not fall back to a built-in transport.
Validation: focused adapter and smoke coordinator tests with controlled HTTP.
Stage 7 — Reasoning Values And Provider-Specific Token Decoding
Completed.
Touch public/domain token declarations, facade conversions, and
internal/llm/openai_compatible_client.go plus focused provider files.
- Add
ReasoningTokenstoTokenUsage,TokenUsagePresence, and observed usage on both sides of the facade; include JSON field/conversions and mutable-response ownership where applicable. Do not renameCachedTokens. - Keep common counts and cached reads on their existing decode path. For
OpenRouter, read reasoning from
completion_tokens_details.reasoning_tokensand writes fromprompt_tokens_details.cache_write_tokens; for vLLM, read the same reasoning path andprompt_tokens_details.created_cache_tokens. Generic keeps its existing top-level write mapping and does not interpret the new reasoning field. Ignore other providers' alternative write locations; do not add/fallback between the nested and top-level values. - Decode newly interpreted integer fields independently with local raw JSON helpers. Absent/null, wrong-type, fractional, or int-overflow values become unknown. Do not coerce strings. Retain independently valid fields; adding a typed optional field must not create a new whole-response failure. Existing malformed common fields and framing/content failures retain their behavior.
- Preserve decoded usage on later missing-content/choice failure as today. Do not infer missing reasoning or writes, enforce new nonnegative restrictions, change generation controls, or collect reasoning text.
- Test provider-specific raw fixtures with conflicting write locations, partial/malformed new fields, known zero, and disabled-detail vLLM responses. Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap.
Validation: model-client parsing tests and public token JSON round trips.
Stage 8 — Token Aggregation And Cache-Hit Helper
Completed.
Touch observation usage accumulators/recorder/group, runner addTokenUsage,
public event/group conversions, and related fixtures.
- Include reasoning counts in existing successful-result summation, attempt and terminal usage, group merging, and explicit/legacy injected presence behavior. Do not replace legacy result token sums with all-attempt observation sums.
- Implement
ObservedTokenUsage.CacheHitRatio() (float64, bool)using the roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid data returns(0, false); known zero reads with positive input returns(0, true). Divide totals as floats, not integer division. Do not persist or average ratios. - Cover per-field incomplete reasoning, injected known zero, retry/repair sums, duplicate group enrollment, weighted aggregate ratios, zero denominator, inconsistent counts without generation failure, and independence from writes.
- Preserve reported total tokens: reasoning and cache details are subsets, not extra values to add into input/output/total counts.
Validation: focused token accumulator/group tests and public observation contracts.
Stage 9 — Public Cost Values And Shared Cost Algebra
Completed.
Create public cost declarations in a focused root file and matching domain values; add the selected fields to responses, results, events, and summaries.
- Implement the selected cost/source declarations and conversions. Copy optional amount pointers at the injected-client conversion boundary so caller mutation cannot change retained data. Include cost in custom result JSON marshal/unmarshal paths as needed, preserving existing duration and omission behavior elsewhere.
- Implement pure internal cost resolution: finite reported amount first, then configured zero, else unknown. A reported zero remains provider-sourced. Negative finite reported amounts are not newly prohibited. Unknown cost is zero-valued with false Known/Complete and empty source.
- Implement one small
CostAccumulatorininternal/observationthat merges known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot is known/complete zero; its first unknown contribution must reset that empty special case rather than accidentally making unknown cost known. Include known zero contributions in source union. Unknown contributions add no source. - Allow merging aggregate costs for groups with the same helper; leave the existing no-terminal group special case to group coverage logic.
- Unit-test precedence, source union including zeros, all-unknown/empty cases, completeness, and approximate floating-point sums. Add JSON tests for old result JSON without cost, current round trips, and optional client zero.
Validation: pure accounting and public serialization tests; no pricing library.
Stage 10 — Optional Cost At The Model-Client Boundary
Completed.
- Decode OpenRouter
usage.costindependently using local tolerant finite-float handling. Missing/null/string/object/overflow values are unknown; invalid framing or common envelope fields retain existing failures. Generic/vLLM do not interpret the field. Do not consumecost_detailsor send a new usage flag. - Carry raw optional cost on built-in
AttemptOutcomeandGenerateResponseeven when a later decoded-content check fails. Non-2xx/invalid framing remain without provider usage. Do not apply zero-cost fallback inside provider code. - Preserve optional cost supplied through an injected client. The adapter already converts non-nil responses alongside errors: retain cost for accounting even then, without changing existing token treatment or public error/result rules. Normalize nonfinite injected cost to unknown before applying fallback.
- Test cost independent of token fields, usable zero, malformed optional cost without generation failure, ignored cost on generic/vLLM, and reported cost retained alongside a classified injected error.
Validation: provider decoder and injected-adapter tests.
Stage 11 — Per-Attempt Cost Collection And Observation
Completed.
Touch generation recordAttempt and built-in/injected execution paths,
observation attempt data/events, recorder terminal cost, and group aggregation.
- At every attempt completion, if dispatched, resolve raw cost against the frozen target's ZeroCost and add it exactly once to an invocation accumulator. Do this for transport errors, eligible retry statuses, cancellation after dispatch, malformed bodies, and missing content as well as success. A failed acquisition/pre-dispatch cancellation contributes nothing.
- Keep cost collection independent of the optional recorder and diagnostic evidence capture flag. Preserve permit-release-before-callback behavior. Do not count retry waits, terminal events, or generation responses again.
- Carry resolved cost on attempt events. Recorder terminal cost reads the same accumulator; groups merge only first terminal costs via shared algebra. Extend public event/group projection without changing existing token totals.
- Introduce private cost-state context access. Extend
WithoutExecutionStateto mask inherited cost state, but not contextual observer selection. Smoke diagnostics must continue masking private execution accounting and emit no ordinary lifecycle or enclosing-run cost contributions. - Exercise controlled retry/error sequences and zero-cost fallback without an observer. Test duplicate terminal delivery and independence of group terminal coverage from cost completeness. Do not retain an attempt history for cost.
Validation: generation, observation recorder/group, and diagnostic-boundary tests.
Stage 12 — Ordinary Run Cost Lifetime And Result Accounting
- Create/install a new private accumulator at the public ordinary Run boundary, before optional recorder creation. Pass that same instance to the recorder. Ensure direct internal runner execution creates a local accumulator if one was not supplied, without replacing one supplied by the owning facade.
- Retain it through initial generation and all repair calls. Set successful
domain.RunResult.CostUSDfrom its snapshot, not by summing responses. Keep the legacyUsagesuccess-response sum and error handling unchanged. - Ensure errors still return no partial public
RunResult; available cost is reported in terminal observations. Runs rejected before dispatch have the established empty aggregate, not a configured-zero contribution source. - Test a retry with unreported cost followed by a reported charge, mixed sources within repairs, reported-cost precedence over zero policy, and identical successful cost metadata with observation enabled/disabled.
Validation: assembled ordinary run/reliable execution contract tests.
Stage 13 — Prepared Cost Lifetime And Nested Isolation
- In internal RunPrepared, after successful claim create/install its own cost accumulator before recorder creation and generation. Use the same result and terminal accounting path as ordinary Run. Do not store an accumulator or contextual observer in the reusable preparation details/private ready payload.
- The claimed handle uses its frozen backend Provider/ZeroCost, even when run from another invocation scope. Preserve admission, deadline attribution, single-claim semantics, discard behavior, and cleared callbacks/credentials.
- Verify ordinary/prepared child runs launched from injected clients or prepared callbacks do not inherit the parent's accumulator, even when both inherit one contextual group observer. Each terminal contributes once to the invocation total; parent cost includes only its own dispatched generations.
- Cover prepared zero-cost failure and repair, rejected claims without events, independent contexts on a reused engine, and diagnostic calls nested in a run without cost/event leakage.
Validation: prepared and nested-boundary contracts, including race-enabled focused tests.
Stage 14 — End-To-End Consumer Regression Coverage
- Add one representative offline invocation workflow using two engines and concurrent runs under one group. Use controlled transport/injected clients for OpenRouter reported cost, vLLM details/configured zero, a retry, and a failed operation after available usage. Join application-owned goroutines before taking the final snapshot.
- Assert independently predictable terminal membership, sums, sources, token presence/completeness, and weighted ratio (or its unavailability when an attempt leaves input/cache-read reporting incomplete). Compare floating-point sums with tolerance; do not assert map order, run-ID bytes, or exact timing.
- Cover a snapshot with completed fully costed members and one started member: cost completeness concerns the included terminals while terminal coverage is false. Cover empty and members-with-no-terminals snapshots separately.
- Reuse narrower tests from earlier stages rather than reproduce every case in this workflow. Keep observations metadata-only; do not add bodies, credentials, URLs, or error objects to the event contract.
Validation: focused new integration tests plus go test -race ./... when the
implementation has reached its final buildable state.
Stage 15 — Canonical Documentation And Final Verification
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and sources, client ownership, and observer/prepared lifetimes. Update stale authentication Kind requirements in shared settings and public docs.
- Update the consumer guide and maintained offline observation example for invocation scoping, cache/reasoning subsets, known/complete cost, configured zeros, sources, and failure-path reporting. Maintain semantic example tests.
- Update
docs/formats.md, the OpenAI-compatible integration contract, and internal overview/observation/generation/model-client/runner documentation for actual ownership and flow. Explain provider defaults versus declared inspection, vLLM reporting settings, and the exact diagnostic response recognition. - Document migration from required authentication kind and block-clearing semantics. Record the change in the next unpublished release note when such a note exists; otherwise leave release-note creation to release preparation. Do not edit published v0.24.0 notes or tag/publish anything in this sprint.
- Use the newest already installed Go toolchain per the development guide,
GOTOOLCHAIN=local, andGOWORK=off. Format changed Go files, rungo test -race ./...,go vet ./..., andgit diff --check. Reuse successful results for unchanged content; do not add redundant builds or live-provider checks. Verify local Markdown links and maintained examples through their normal offline tests. Review the entire diff for unrelated changes and secrets. - Audit completion against every target-state section, especially no-observer cost, failed dispatched attempts, optional-field tolerance, prepared policy freezing, and nested/diagnostic separation. Resolve remaining implementation defects before reporting sprint completion. Roadmap retirement is later housekeeping, not part of this implementation plan.
There are no unresolved product decisions required to begin these stages.