From f762c0358e6154d58f70d95ca2397f2ae6db81cc Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 18:01:31 +0000 Subject: [PATCH] Plan invocation observation and usage implementation --- docs/roadmap/implementation.md | 418 +++++++++++++++++++++++++++++++++ 1 file changed, 418 insertions(+) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..752829e --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,418 @@ +# Invocation Observation And Usage Implementation Plan + +## Objective And Working Rules + +Implement the accepted [target-state roadmap](invocation-scoped-token-usage.md). +It owns scope, consumer intent, and reporting policy. This plan owns concrete +implementation decisions and the ordered work sequence. Baseline: `a47df4e`. + +Each numbered stage is one Luna-sized implementation prompt. Execute them in +order within one end-to-end sprint. Intermediate stages may leave unresolved +references or failing compilation; do not add compatibility scaffolding merely +to keep those intermediate states green. There are no intermediate human-review, +external-publication, or live-provider gates. Run focused checks when their +dependencies are ready; the completed sprint must pass the final checks. + +Read [development guidance](../development.md) and the applicable agent, +architecture, testing, and documentation policies. Follow +[ADR 0001](../adr/0001-configuration-boundary.md): interpretation and validation +use final ConfigKit values, not raw sources or historical inspection. Follow the +shared-settings facade exception without introducing public aliases for runtime +implementation types. Preserve unrelated work and follow the agent commit policy. + +Planning and implementing are separate tasks. This document does not authorize +a release, dependency upgrade, CLI implementation, or changes to published notes. + +## Selected Design + +- Store contextual observers as internal `domain.ExecutionObserver` callbacks + under a context key separate from the recorder, claim sink, and cost state. + The public helper wraps public event conversion at registration. Internal + composition preserves explicit-first and inherited-to-new contextual ordering. +- Add `Provider *string` and `ZeroCost *bool` to shared backend settings and + `BackendConfig`, with ordinary `provider`/`zero_cost` field names. Resource + inspection returns declared normalized values; configuration inspection owns + history. Neither inspection API needs a new provenance model. +- Capture runtime `Provider string` and `ZeroCost bool` on `domain.Backend` and + carry them on internal `domain.ExecutionTarget` as local metadata excluded + from JSON/YAML. Empty provider resolves to `openai-compatible` at runtime. + Keep these fields out of the root execution-target API and override API: + backend inspection is the requested public configuration surface. Endpoint-only + profiles use the generic provider and disabled zero fallback. Existing + model/endpoint overrides retain the selected backend's metadata. +- Keep provider ID recognition, response interpretation, and authentication + adapter resolution in `internal/llm`, using simple functions/dispatch. Config + validation may call that package's supported-name checks as it already does + for reserved request fields. No public provider registration mechanism or new + providers package is needed. +- Add `ReasoningTokens int` to token values, `ReasoningTokens bool` to presence, + and `ReasoningTokens ObservedTokenCount` to observed usage. Public token JSON + uses `reasoning_tokens`. Add the roadmap's two helper methods/functions exactly + as specified; do not add aliases for cache reads or a stored cache-hit ratio. +- Define public `CostSource string` constants `CostSourceNone = ""`, + `CostSourceProvider = "provider"`, `CostSourceConfiguredZero = "configured_zero"`, + and `CostSourceMixed = "mixed"`. Define `ObservedCost` with `Value float64`, + `Known bool`, `Complete bool`, and `Source CostSource`, JSON-tagged `value`, + `known`, `complete`, and `source`, without omitting zero fields. +- Add `GenerateResponse.CostUSD *float64` (`json:"cost_usd,omitempty"`) and + `RunResult.CostUSD ObservedCost` (`json:"cost_usd"`). Add `CostUSD ObservedCost` + to `ExecutionEvent` and `ExecutionGroupSummary`; the latter uses `cost_usd`. + Define equivalent internal domain values and explicit facade conversions. + Older result JSON lacking cost decodes to unknown cost, not reported zero. +- Own pure cost resolution/merge and run-local cost state in + `internal/observation`. Cost is collected once per dispatched attempt regardless + of observer selection. The recorder references that same run accumulator for + terminal cost; it must not independently add attempts a second time. Groups + reuse the merge rules but remain terminal-only collectors. +- New optional reasoning/cache-write details use local tolerant integer decoding; + cost uses local tolerant finite-float decoding. A typed `*int`/`*float64` field + in the main envelope would fail the whole decode for malformed optional data. + Preserve existing common token decoding, framing, content validation, and + error identities. Do not add a general numeric coercion framework. + +## Stage 1 — Context Helper And Ordinary Run Selection + +Touch `observations.go`, the root `Engine.Run` in `engine.go`, and context support +in `internal/observation/recorder.go` or a focused adjacent file. + +- Implement `WithExecutionObserver(context.Context, ExecutionObserver) + context.Context` using a private internal observer-selection key. Nil is a + no-op. Compose inherited callbacks before the new callback using the existing + composition facility; keep registration state immutable. +- In ordinary `Run`, compose a converted explicit request observer with the + contextual selection before deciding whether to create a recorder. Preserve + current start timing, run IDs, public error mapping, timeout finalization, + permit release, and terminal delivery. With neither observer, preserve the + existing no-observation path. +- Do not mask the contextual key in `WithoutExecutionState`; continue masking + inherited recorder and claim state. Each nested public call creates its own + recorder. Do not modify `Coordinate` to append contextual selection: its + explicit request/coordinator composition already leads to ordinary `Run`. +- Add focused public tests for no observer, contextual-only selection, explicit + plus nested context ordering, nil registration, unrelated contexts, and two + independent run IDs from a nested injected-client call. Group duplication + uses existing terminal deduplication, not callback equality checks. + +Validation when buildable: root observation/boundary contract tests and focused +internal observer tests. No provider requests are needed. + +## Stage 2 — Execution-Time Context For Prepared Handles + +Touch root `RunPrepared`, internal `usecase/prepared_execution.go`, and observer +conversion only where required. + +- Retain only the explicitly supplied request observer in the prepared payload. + Do not snapshot contextual observers during `PrepareExecution` or capture + preparation contexts in the handle. +- After a successful claim, compose that observer with the contextual selection + from the execution context before recorder creation. Preserve the private + claim sink so root terminal delivery still follows public error mapping and + finalized timeout attribution. +- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle + rejection, and existing payload clearing. Do not expose private callbacks in + `Details` or create a second lifecycle around the claimed execution. +- Test a handle prepared in scope A and executed in B: retained explicit + observer and B receive events; A does not. Include a context-only execution, + rejected claims, and a continuation selected by `Coordinate` to protect its + explicit-before-context order. + +Validation: prepared execution and coordinate observation contract tests. + +## Stage 3 — Backend Configuration And Provider Selection + +Touch `internal/settings/settings.go`, `backends.go`, configuration ownership, +resolution/validation code, and the provider-selection functions in `internal/llm`. + +- Add the two pointer fields and presence-preserving source-helper encoding. + Nil omits a field; false and blank pointers remain supplied values. Raw + ConfigKit mappings remain the way to supply explicit null. +- Normalize supplied provider strings with trimming/lowercasing in the existing + final resolution callback and record normalizations using existing ConfigKit + APIs. Do not materialize the runtime default into the settings object. +- Recognize exactly `openai-compatible`, `openrouter`, and `vllm`; empty means + generic. Reject other nonempty values through existing final configuration + validation/error wrapping. Do not validate overwritten contributions. +- Validate `zero_cost` as the typed Boolean setting; absent/null/false disables + fallback. Do not infer it or make `LocalBackend` set it automatically. +- Relax authentication-object validation to allow absent/null kind, retain + existing kind normalization and supported explicit kinds, and add + `vllm_models`. A missing kind with no provider default remains an unsupported + diagnostic selection, not a new load-time requirement for a provider. +- Cover layered final-value validation, inheritance, null clearing, normalization, + unsupported names, helper presence, and absence of invented defaults. + +Validation: focused configuration/settings/source-helper tests. + +## Stage 4 — Runtime Capture, Freezing, And Backend Inspection + +Touch `internal/configuration/targets.go`, `ownership.go`, `resources.go`, domain +backend/target values and clones, and `bundled/openrouter.yaml` under that package. + +- Copy declared settings into owned records and inspection results. Preserve + explicit false while allowing absent/null collapse in resource inspection. + Leave full null/omission history to `InspectConfiguration`. +- Capture effective provider and zero policy on the internal backend and the + backend-derived execution target. Ensure scalar copying through profile + capture, runtime overrides, prepared details/payloads, and repair requests. + Retain backend metadata when a profile/request overrides only its endpoint. +- Keep root public execution targets and provider HTTP bodies unchanged. Audit + serialization/copy helpers so excluded internal fields are not accidentally + lost by a JSON-based copy. Do not recapture policy from configuration during + `RunPrepared` or traverse provenance during execution. +- Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost + defaults or new backend instances. Keep API keys and default profiles unchanged. +- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only + profiles, frozen prepared policy, and inspection without manufactured values. + +Validation: configuration capture, backend inspection, bundle, and preparation tests. + +## Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults + +Current provider branches occur in `internal/usecase/smoke_authentication.go` and +`smoke_selection.go`, including `ProbeOpenRouterKey` and endpoint derivation. + +- Replace the OpenRouter-specific collaborator operation with a generic + authentication-probe operation owned by `internal/llm`. Carry a resolved + adapter selection to it; leave usecase coordination provider-neutral. +- Resolve adapter kind and destination with pure provider-boundary functions. + Explicit kind wins, absent/null kind uses the provider default, and `none` + is inapplicable. Explicit endpoint wins; otherwise use the adapter-relative + fallback. Missing selection remains unsupported. Do not rewrite loaded + settings, captured declared smoke metadata, or inspection provenance. +- OpenRouter defaults to `openrouter_key`; generic has no default. Preserve + explicit OpenRouter adapters on any backend and existing HTTP evidence, + bounded recognition, credential, cancellation, and overflow handling. +- Clear an authentication block to restore the provider default; an endpoint-only + block may use that default. Preserve reachability and inference selection. +- Update affected interfaces, engine assembly, and test doubles. Test default, + explicit/none, endpoint-only, cleared-block, and unsupported combinations at + their owning boundaries rather than duplicating every permutation publicly. + +Validation: smoke selection/authentication tests and existing OpenRouter contract tests. + +## Stage 6 — vLLM Models Authentication Adapter + +- Implement `vllm_models` beside the OpenRouter adapter, sharing generic HTTP + execution and bounded recognition helpers rather than duplicating transport. + Select it by default for `vllm` and allow explicit use on other providers. +- Resolve `models` beneath the configured API base, preserving base prefixes; + honor an explicit full diagnostic endpoint. Do not hard-code a host or probe + `/health`. Use existing optional credential rules without forcing a key. +- Recognize exactly the roadmap's HTTP 200 object/list/data-array shape, + including empty arrays. HTTP 401 is rejection; other/unrecognized responses + follow existing unknown/failure reporting. Do not inspect list members for + model existence, inference permissions, or authentication enforcement. +- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other + statuses, optional credentials, overridden destinations, and `kind: none`. + Ensure unsupported injected HTTP diagnostics still do not fall back to a + built-in transport. + +Validation: focused adapter and smoke coordinator tests with controlled HTTP. + +## Stage 7 — Reasoning Values And Provider-Specific Token Decoding + +Touch public/domain token declarations, facade conversions, and +`internal/llm/openai_compatible_client.go` plus focused provider files. + +- Add `ReasoningTokens` to `TokenUsage`, `TokenUsagePresence`, and observed usage + on both sides of the facade; include JSON field/conversions and mutable-response + ownership where applicable. Do not rename `CachedTokens`. +- Keep common counts and cached reads on their existing decode path. For + OpenRouter, read reasoning from `completion_tokens_details.reasoning_tokens` + and writes from `prompt_tokens_details.cache_write_tokens`; for vLLM, read + the same reasoning path and `prompt_tokens_details.created_cache_tokens`. + Generic keeps its existing top-level write mapping and does not interpret the + new reasoning field. Ignore other providers' alternative write locations; + do not add/fallback between the nested and top-level values. +- Decode newly interpreted integer fields independently with local raw JSON + helpers. Absent/null, wrong-type, fractional, or int-overflow values become + unknown. Do not coerce strings. Retain independently valid fields; adding a + typed optional field must not create a new whole-response failure. Existing + malformed common fields and framing/content failures retain their behavior. +- Preserve decoded usage on later missing-content/choice failure as today. + Do not infer missing reasoning or writes, enforce new nonnegative restrictions, + change generation controls, or collect reasoning text. +- Test provider-specific raw fixtures with conflicting write locations, + partial/malformed new fields, known zero, and disabled-detail vLLM responses. + Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap. + +Validation: model-client parsing tests and public token JSON round trips. + +## Stage 8 — Token Aggregation And Cache-Hit Helper + +Touch observation usage accumulators/recorder/group, runner `addTokenUsage`, +public event/group conversions, and related fixtures. + +- Include reasoning counts in existing successful-result summation, attempt and + terminal usage, group merging, and explicit/legacy injected presence behavior. + Do not replace legacy result token sums with all-attempt observation sums. +- Implement `ObservedTokenUsage.CacheHitRatio() (float64, bool)` using the + roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid + data returns `(0, false)`; known zero reads with positive input returns `(0, true)`. + Divide totals as floats, not integer division. Do not persist or average ratios. +- Cover per-field incomplete reasoning, injected known zero, retry/repair sums, + duplicate group enrollment, weighted aggregate ratios, zero denominator, + inconsistent counts without generation failure, and independence from writes. +- Preserve reported total tokens: reasoning and cache details are subsets, not + extra values to add into input/output/total counts. + +Validation: focused token accumulator/group tests and public observation contracts. + +## Stage 9 — Public Cost Values And Shared Cost Algebra + +Create public cost declarations in a focused root file and matching domain +values; add the selected fields to responses, results, events, and summaries. + +- Implement the selected cost/source declarations and conversions. Copy optional + amount pointers at the injected-client conversion boundary so caller mutation + cannot change retained data. Include cost in custom result JSON marshal/unmarshal + paths as needed, preserving existing duration and omission behavior elsewhere. +- Implement pure internal cost resolution: finite reported amount first, then + configured zero, else unknown. A reported zero remains provider-sourced. + Negative finite reported amounts are not newly prohibited. Unknown cost is + zero-valued with false Known/Complete and empty source. +- Implement one small `CostAccumulator` in `internal/observation` that merges + known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot + is known/complete zero; its first unknown contribution must reset that empty + special case rather than accidentally making unknown cost known. Include + known zero contributions in source union. Unknown contributions add no source. +- Allow merging aggregate costs for groups with the same helper; leave the + existing no-terminal group special case to group coverage logic. +- Unit-test precedence, source union including zeros, all-unknown/empty cases, + completeness, and approximate floating-point sums. Add JSON tests for old + result JSON without cost, current round trips, and optional client zero. + +Validation: pure accounting and public serialization tests; no pricing library. + +## Stage 10 — Optional Cost At The Model-Client Boundary + +- Decode OpenRouter `usage.cost` independently using local tolerant finite-float + handling. Missing/null/string/object/overflow values are unknown; invalid + framing or common envelope fields retain existing failures. Generic/vLLM do + not interpret the field. Do not consume `cost_details` or send a new usage flag. +- Carry raw optional cost on built-in `AttemptOutcome` and `GenerateResponse` + even when a later decoded-content check fails. Non-2xx/invalid framing remain + without provider usage. Do not apply zero-cost fallback inside provider code. +- Preserve optional cost supplied through an injected client. The adapter already + converts non-nil responses alongside errors: retain cost for accounting even + then, without changing existing token treatment or public error/result rules. + Normalize nonfinite injected cost to unknown before applying fallback. +- Test cost independent of token fields, usable zero, malformed optional cost + without generation failure, ignored cost on generic/vLLM, and reported cost + retained alongside a classified injected error. + +Validation: provider decoder and injected-adapter tests. + +## Stage 11 — Per-Attempt Cost Collection And Observation + +Touch generation `recordAttempt` and built-in/injected execution paths, +observation attempt data/events, recorder terminal cost, and group aggregation. + +- At every attempt completion, if dispatched, resolve raw cost against the + frozen target's ZeroCost and add it exactly once to an invocation accumulator. + Do this for transport errors, eligible retry statuses, cancellation after + dispatch, malformed bodies, and missing content as well as success. A failed + acquisition/pre-dispatch cancellation contributes nothing. +- Keep cost collection independent of the optional recorder and diagnostic + evidence capture flag. Preserve permit-release-before-callback behavior. + Do not count retry waits, terminal events, or generation responses again. +- Carry resolved cost on attempt events. Recorder terminal cost reads the same + accumulator; groups merge only first terminal costs via shared algebra. + Extend public event/group projection without changing existing token totals. +- Introduce private cost-state context access. Extend `WithoutExecutionState` to + mask inherited cost state, but not contextual observer selection. Smoke + diagnostics must continue masking private execution accounting and emit no + ordinary lifecycle or enclosing-run cost contributions. +- Exercise controlled retry/error sequences and zero-cost fallback without an + observer. Test duplicate terminal delivery and independence of group terminal + coverage from cost completeness. Do not retain an attempt history for cost. + +Validation: generation, observation recorder/group, and diagnostic-boundary tests. + +## Stage 12 — Ordinary Run Cost Lifetime And Result Accounting + +- Create/install a new private accumulator at the public ordinary Run boundary, + before optional recorder creation. Pass that same instance to the recorder. + Ensure direct internal runner execution creates a local accumulator if one + was not supplied, without replacing one supplied by the owning facade. +- Retain it through initial generation and all repair calls. Set successful + `domain.RunResult.CostUSD` from its snapshot, not by summing responses. Keep + the legacy `Usage` success-response sum and error handling unchanged. +- Ensure errors still return no partial public `RunResult`; available cost is + reported in terminal observations. Runs rejected before dispatch have the + established empty aggregate, not a configured-zero contribution source. +- Test a retry with unreported cost followed by a reported charge, mixed sources + within repairs, reported-cost precedence over zero policy, and identical + successful cost metadata with observation enabled/disabled. + +Validation: assembled ordinary run/reliable execution contract tests. + +## Stage 13 — Prepared Cost Lifetime And Nested Isolation + +- In internal RunPrepared, after successful claim create/install its own cost + accumulator before recorder creation and generation. Use the same result and + terminal accounting path as ordinary Run. Do not store an accumulator or + contextual observer in the reusable preparation details/private ready payload. +- The claimed handle uses its frozen backend Provider/ZeroCost, even when run + from another invocation scope. Preserve admission, deadline attribution, + single-claim semantics, discard behavior, and cleared callbacks/credentials. +- Verify ordinary/prepared child runs launched from injected clients or + prepared callbacks do not inherit the parent's accumulator, even when both + inherit one contextual group observer. Each terminal contributes once to the + invocation total; parent cost includes only its own dispatched generations. +- Cover prepared zero-cost failure and repair, rejected claims without events, + independent contexts on a reused engine, and diagnostic calls nested in a run + without cost/event leakage. + +Validation: prepared and nested-boundary contracts, including race-enabled focused tests. + +## Stage 14 — End-To-End Consumer Regression Coverage + +- Add one representative offline invocation workflow using two engines and + concurrent runs under one group. Use controlled transport/injected clients + for OpenRouter reported cost, vLLM details/configured zero, a retry, and a + failed operation after available usage. Join application-owned goroutines + before taking the final snapshot. +- Assert independently predictable terminal membership, sums, sources, token + presence/completeness, and weighted ratio (or its unavailability when an + attempt leaves input/cache-read reporting incomplete). Compare floating-point + sums with tolerance; do not assert map order, run-ID bytes, or exact timing. +- Cover a snapshot with completed fully costed members and one started member: + cost completeness concerns the included terminals while terminal coverage is + false. Cover empty and members-with-no-terminals snapshots separately. +- Reuse narrower tests from earlier stages rather than reproduce every case in + this workflow. Keep observations metadata-only; do not add bodies, credentials, + URLs, or error objects to the event contract. + +Validation: focused new integration tests plus `go test -race ./...` when the +implementation has reached its final buildable state. + +## Stage 15 — Canonical Documentation And Final Verification + +- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and + sources, client ownership, and observer/prepared lifetimes. Update stale + authentication Kind requirements in shared settings and public docs. +- Update the consumer guide and maintained offline observation example for + invocation scoping, cache/reasoning subsets, known/complete cost, configured + zeros, sources, and failure-path reporting. Maintain semantic example tests. +- Update `docs/formats.md`, the OpenAI-compatible integration contract, and + internal overview/observation/generation/model-client/runner documentation for + actual ownership and flow. Explain provider defaults versus declared inspection, + vLLM reporting settings, and the exact diagnostic response recognition. +- Document migration from required authentication kind and block-clearing + semantics. Record the change in the next unpublished release note when such a + note exists; otherwise leave release-note creation to release preparation. + Do not edit published v0.24.0 notes or tag/publish anything in this sprint. +- Use the newest already installed Go toolchain per the development guide, + `GOTOOLCHAIN=local`, and `GOWORK=off`. Format changed Go files, run + `go test -race ./...`, `go vet ./...`, and `git diff --check`. Reuse successful + results for unchanged content; do not add redundant builds or live-provider + checks. Verify local Markdown links and maintained examples through their + normal offline tests. Review the entire diff for unrelated changes and secrets. +- Audit completion against every target-state section, especially no-observer + cost, failed dispatched attempts, optional-field tolerance, prepared policy + freezing, and nested/diagnostic separation. Resolve remaining implementation + defects before reporting sprint completion. Roadmap retirement is later + housekeeping, not part of this implementation plan. + +There are no unresolved product decisions required to begin these stages.