Files
promptkit/docs/roadmap/implementation.md

419 lines
26 KiB
Markdown
Raw Normal View History

# Invocation Observation And Usage Implementation Plan
## Objective And Working Rules
Implement the accepted [target-state roadmap](invocation-scoped-token-usage.md).
It owns scope, consumer intent, and reporting policy. This plan owns concrete
implementation decisions and the ordered work sequence. Baseline: `a47df4e`.
Each numbered stage is one Luna-sized implementation prompt. Execute them in
order within one end-to-end sprint. Intermediate stages may leave unresolved
references or failing compilation; do not add compatibility scaffolding merely
to keep those intermediate states green. There are no intermediate human-review,
external-publication, or live-provider gates. Run focused checks when their
dependencies are ready; the completed sprint must pass the final checks.
Read [development guidance](../development.md) and the applicable agent,
architecture, testing, and documentation policies. Follow
[ADR 0001](../adr/0001-configuration-boundary.md): interpretation and validation
use final ConfigKit values, not raw sources or historical inspection. Follow the
shared-settings facade exception without introducing public aliases for runtime
implementation types. Preserve unrelated work and follow the agent commit policy.
Planning and implementing are separate tasks. This document does not authorize
a release, dependency upgrade, CLI implementation, or changes to published notes.
## Selected Design
- Store contextual observers as internal `domain.ExecutionObserver` callbacks
under a context key separate from the recorder, claim sink, and cost state.
The public helper wraps public event conversion at registration. Internal
composition preserves explicit-first and inherited-to-new contextual ordering.
- Add `Provider *string` and `ZeroCost *bool` to shared backend settings and
`BackendConfig`, with ordinary `provider`/`zero_cost` field names. Resource
inspection returns declared normalized values; configuration inspection owns
history. Neither inspection API needs a new provenance model.
- Capture runtime `Provider string` and `ZeroCost bool` on `domain.Backend` and
carry them on internal `domain.ExecutionTarget` as local metadata excluded
from JSON/YAML. Empty provider resolves to `openai-compatible` at runtime.
Keep these fields out of the root execution-target API and override API:
backend inspection is the requested public configuration surface. Endpoint-only
profiles use the generic provider and disabled zero fallback. Existing
model/endpoint overrides retain the selected backend's metadata.
- Keep provider ID recognition, response interpretation, and authentication
adapter resolution in `internal/llm`, using simple functions/dispatch. Config
validation may call that package's supported-name checks as it already does
for reserved request fields. No public provider registration mechanism or new
providers package is needed.
- Add `ReasoningTokens int` to token values, `ReasoningTokens bool` to presence,
and `ReasoningTokens ObservedTokenCount` to observed usage. Public token JSON
uses `reasoning_tokens`. Add the roadmap's two helper methods/functions exactly
as specified; do not add aliases for cache reads or a stored cache-hit ratio.
- Define public `CostSource string` constants `CostSourceNone = ""`,
`CostSourceProvider = "provider"`, `CostSourceConfiguredZero = "configured_zero"`,
and `CostSourceMixed = "mixed"`. Define `ObservedCost` with `Value float64`,
`Known bool`, `Complete bool`, and `Source CostSource`, JSON-tagged `value`,
`known`, `complete`, and `source`, without omitting zero fields.
- Add `GenerateResponse.CostUSD *float64` (`json:"cost_usd,omitempty"`) and
`RunResult.CostUSD ObservedCost` (`json:"cost_usd"`). Add `CostUSD ObservedCost`
to `ExecutionEvent` and `ExecutionGroupSummary`; the latter uses `cost_usd`.
Define equivalent internal domain values and explicit facade conversions.
Older result JSON lacking cost decodes to unknown cost, not reported zero.
- Own pure cost resolution/merge and run-local cost state in
`internal/observation`. Cost is collected once per dispatched attempt regardless
of observer selection. The recorder references that same run accumulator for
terminal cost; it must not independently add attempts a second time. Groups
reuse the merge rules but remain terminal-only collectors.
- New optional reasoning/cache-write details use local tolerant integer decoding;
cost uses local tolerant finite-float decoding. A typed `*int`/`*float64` field
in the main envelope would fail the whole decode for malformed optional data.
Preserve existing common token decoding, framing, content validation, and
error identities. Do not add a general numeric coercion framework.
## Stage 1 — Context Helper And Ordinary Run Selection
Touch `observations.go`, the root `Engine.Run` in `engine.go`, and context support
in `internal/observation/recorder.go` or a focused adjacent file.
- Implement `WithExecutionObserver(context.Context, ExecutionObserver)
context.Context` using a private internal observer-selection key. Nil is a
no-op. Compose inherited callbacks before the new callback using the existing
composition facility; keep registration state immutable.
- In ordinary `Run`, compose a converted explicit request observer with the
contextual selection before deciding whether to create a recorder. Preserve
current start timing, run IDs, public error mapping, timeout finalization,
permit release, and terminal delivery. With neither observer, preserve the
existing no-observation path.
- Do not mask the contextual key in `WithoutExecutionState`; continue masking
inherited recorder and claim state. Each nested public call creates its own
recorder. Do not modify `Coordinate` to append contextual selection: its
explicit request/coordinator composition already leads to ordinary `Run`.
- Add focused public tests for no observer, contextual-only selection, explicit
plus nested context ordering, nil registration, unrelated contexts, and two
independent run IDs from a nested injected-client call. Group duplication
uses existing terminal deduplication, not callback equality checks.
Validation when buildable: root observation/boundary contract tests and focused
internal observer tests. No provider requests are needed.
## Stage 2 — Execution-Time Context For Prepared Handles
Touch root `RunPrepared`, internal `usecase/prepared_execution.go`, and observer
conversion only where required.
- Retain only the explicitly supplied request observer in the prepared payload.
Do not snapshot contextual observers during `PrepareExecution` or capture
preparation contexts in the handle.
- After a successful claim, compose that observer with the contextual selection
from the execution context before recorder creation. Preserve the private
claim sink so root terminal delivery still follows public error mapping and
finalized timeout attribution.
- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle
rejection, and existing payload clearing. Do not expose private callbacks in
`Details` or create a second lifecycle around the claimed execution.
- Test a handle prepared in scope A and executed in B: retained explicit
observer and B receive events; A does not. Include a context-only execution,
rejected claims, and a continuation selected by `Coordinate` to protect its
explicit-before-context order.
Validation: prepared execution and coordinate observation contract tests.
## Stage 3 — Backend Configuration And Provider Selection
Touch `internal/settings/settings.go`, `backends.go`, configuration ownership,
resolution/validation code, and the provider-selection functions in `internal/llm`.
- Add the two pointer fields and presence-preserving source-helper encoding.
Nil omits a field; false and blank pointers remain supplied values. Raw
ConfigKit mappings remain the way to supply explicit null.
- Normalize supplied provider strings with trimming/lowercasing in the existing
final resolution callback and record normalizations using existing ConfigKit
APIs. Do not materialize the runtime default into the settings object.
- Recognize exactly `openai-compatible`, `openrouter`, and `vllm`; empty means
generic. Reject other nonempty values through existing final configuration
validation/error wrapping. Do not validate overwritten contributions.
- Validate `zero_cost` as the typed Boolean setting; absent/null/false disables
fallback. Do not infer it or make `LocalBackend` set it automatically.
- Relax authentication-object validation to allow absent/null kind, retain
existing kind normalization and supported explicit kinds, and add
`vllm_models`. A missing kind with no provider default remains an unsupported
diagnostic selection, not a new load-time requirement for a provider.
- Cover layered final-value validation, inheritance, null clearing, normalization,
unsupported names, helper presence, and absence of invented defaults.
Validation: focused configuration/settings/source-helper tests.
## Stage 4 — Runtime Capture, Freezing, And Backend Inspection
Touch `internal/configuration/targets.go`, `ownership.go`, `resources.go`, domain
backend/target values and clones, and `bundled/openrouter.yaml` under that package.
- Copy declared settings into owned records and inspection results. Preserve
explicit false while allowing absent/null collapse in resource inspection.
Leave full null/omission history to `InspectConfiguration`.
- Capture effective provider and zero policy on the internal backend and the
backend-derived execution target. Ensure scalar copying through profile
capture, runtime overrides, prepared details/payloads, and repair requests.
Retain backend metadata when a profile/request overrides only its endpoint.
- Keep root public execution targets and provider HTTP bodies unchanged. Audit
serialization/copy helpers so excluded internal fields are not accidentally
lost by a JSON-based copy. Do not recapture policy from configuration during
`RunPrepared` or traverse provenance during execution.
- Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost
defaults or new backend instances. Keep API keys and default profiles unchanged.
- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only
profiles, frozen prepared policy, and inspection without manufactured values.
Validation: configuration capture, backend inspection, bundle, and preparation tests.
## Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults
Current provider branches occur in `internal/usecase/smoke_authentication.go` and
`smoke_selection.go`, including `ProbeOpenRouterKey` and endpoint derivation.
- Replace the OpenRouter-specific collaborator operation with a generic
authentication-probe operation owned by `internal/llm`. Carry a resolved
adapter selection to it; leave usecase coordination provider-neutral.
- Resolve adapter kind and destination with pure provider-boundary functions.
Explicit kind wins, absent/null kind uses the provider default, and `none`
is inapplicable. Explicit endpoint wins; otherwise use the adapter-relative
fallback. Missing selection remains unsupported. Do not rewrite loaded
settings, captured declared smoke metadata, or inspection provenance.
- OpenRouter defaults to `openrouter_key`; generic has no default. Preserve
explicit OpenRouter adapters on any backend and existing HTTP evidence,
bounded recognition, credential, cancellation, and overflow handling.
- Clear an authentication block to restore the provider default; an endpoint-only
block may use that default. Preserve reachability and inference selection.
- Update affected interfaces, engine assembly, and test doubles. Test default,
explicit/none, endpoint-only, cleared-block, and unsupported combinations at
their owning boundaries rather than duplicating every permutation publicly.
Validation: smoke selection/authentication tests and existing OpenRouter contract tests.
## Stage 6 — vLLM Models Authentication Adapter
- Implement `vllm_models` beside the OpenRouter adapter, sharing generic HTTP
execution and bounded recognition helpers rather than duplicating transport.
Select it by default for `vllm` and allow explicit use on other providers.
- Resolve `models` beneath the configured API base, preserving base prefixes;
honor an explicit full diagnostic endpoint. Do not hard-code a host or probe
`/health`. Use existing optional credential rules without forcing a key.
- Recognize exactly the roadmap's HTTP 200 object/list/data-array shape,
including empty arrays. HTTP 401 is rejection; other/unrecognized responses
follow existing unknown/failure reporting. Do not inspect list members for
model existence, inference permissions, or authentication enforcement.
- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other
statuses, optional credentials, overridden destinations, and `kind: none`.
Ensure unsupported injected HTTP diagnostics still do not fall back to a
built-in transport.
Validation: focused adapter and smoke coordinator tests with controlled HTTP.
## Stage 7 — Reasoning Values And Provider-Specific Token Decoding
Touch public/domain token declarations, facade conversions, and
`internal/llm/openai_compatible_client.go` plus focused provider files.
- Add `ReasoningTokens` to `TokenUsage`, `TokenUsagePresence`, and observed usage
on both sides of the facade; include JSON field/conversions and mutable-response
ownership where applicable. Do not rename `CachedTokens`.
- Keep common counts and cached reads on their existing decode path. For
OpenRouter, read reasoning from `completion_tokens_details.reasoning_tokens`
and writes from `prompt_tokens_details.cache_write_tokens`; for vLLM, read
the same reasoning path and `prompt_tokens_details.created_cache_tokens`.
Generic keeps its existing top-level write mapping and does not interpret the
new reasoning field. Ignore other providers' alternative write locations;
do not add/fallback between the nested and top-level values.
- Decode newly interpreted integer fields independently with local raw JSON
helpers. Absent/null, wrong-type, fractional, or int-overflow values become
unknown. Do not coerce strings. Retain independently valid fields; adding a
typed optional field must not create a new whole-response failure. Existing
malformed common fields and framing/content failures retain their behavior.
- Preserve decoded usage on later missing-content/choice failure as today.
Do not infer missing reasoning or writes, enforce new nonnegative restrictions,
change generation controls, or collect reasoning text.
- Test provider-specific raw fixtures with conflicting write locations,
partial/malformed new fields, known zero, and disabled-detail vLLM responses.
Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap.
Validation: model-client parsing tests and public token JSON round trips.
## Stage 8 — Token Aggregation And Cache-Hit Helper
Touch observation usage accumulators/recorder/group, runner `addTokenUsage`,
public event/group conversions, and related fixtures.
- Include reasoning counts in existing successful-result summation, attempt and
terminal usage, group merging, and explicit/legacy injected presence behavior.
Do not replace legacy result token sums with all-attempt observation sums.
- Implement `ObservedTokenUsage.CacheHitRatio() (float64, bool)` using the
roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid
data returns `(0, false)`; known zero reads with positive input returns `(0, true)`.
Divide totals as floats, not integer division. Do not persist or average ratios.
- Cover per-field incomplete reasoning, injected known zero, retry/repair sums,
duplicate group enrollment, weighted aggregate ratios, zero denominator,
inconsistent counts without generation failure, and independence from writes.
- Preserve reported total tokens: reasoning and cache details are subsets, not
extra values to add into input/output/total counts.
Validation: focused token accumulator/group tests and public observation contracts.
## Stage 9 — Public Cost Values And Shared Cost Algebra
Create public cost declarations in a focused root file and matching domain
values; add the selected fields to responses, results, events, and summaries.
- Implement the selected cost/source declarations and conversions. Copy optional
amount pointers at the injected-client conversion boundary so caller mutation
cannot change retained data. Include cost in custom result JSON marshal/unmarshal
paths as needed, preserving existing duration and omission behavior elsewhere.
- Implement pure internal cost resolution: finite reported amount first, then
configured zero, else unknown. A reported zero remains provider-sourced.
Negative finite reported amounts are not newly prohibited. Unknown cost is
zero-valued with false Known/Complete and empty source.
- Implement one small `CostAccumulator` in `internal/observation` that merges
known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot
is known/complete zero; its first unknown contribution must reset that empty
special case rather than accidentally making unknown cost known. Include
known zero contributions in source union. Unknown contributions add no source.
- Allow merging aggregate costs for groups with the same helper; leave the
existing no-terminal group special case to group coverage logic.
- Unit-test precedence, source union including zeros, all-unknown/empty cases,
completeness, and approximate floating-point sums. Add JSON tests for old
result JSON without cost, current round trips, and optional client zero.
Validation: pure accounting and public serialization tests; no pricing library.
## Stage 10 — Optional Cost At The Model-Client Boundary
- Decode OpenRouter `usage.cost` independently using local tolerant finite-float
handling. Missing/null/string/object/overflow values are unknown; invalid
framing or common envelope fields retain existing failures. Generic/vLLM do
not interpret the field. Do not consume `cost_details` or send a new usage flag.
- Carry raw optional cost on built-in `AttemptOutcome` and `GenerateResponse`
even when a later decoded-content check fails. Non-2xx/invalid framing remain
without provider usage. Do not apply zero-cost fallback inside provider code.
- Preserve optional cost supplied through an injected client. The adapter already
converts non-nil responses alongside errors: retain cost for accounting even
then, without changing existing token treatment or public error/result rules.
Normalize nonfinite injected cost to unknown before applying fallback.
- Test cost independent of token fields, usable zero, malformed optional cost
without generation failure, ignored cost on generic/vLLM, and reported cost
retained alongside a classified injected error.
Validation: provider decoder and injected-adapter tests.
## Stage 11 — Per-Attempt Cost Collection And Observation
Touch generation `recordAttempt` and built-in/injected execution paths,
observation attempt data/events, recorder terminal cost, and group aggregation.
- At every attempt completion, if dispatched, resolve raw cost against the
frozen target's ZeroCost and add it exactly once to an invocation accumulator.
Do this for transport errors, eligible retry statuses, cancellation after
dispatch, malformed bodies, and missing content as well as success. A failed
acquisition/pre-dispatch cancellation contributes nothing.
- Keep cost collection independent of the optional recorder and diagnostic
evidence capture flag. Preserve permit-release-before-callback behavior.
Do not count retry waits, terminal events, or generation responses again.
- Carry resolved cost on attempt events. Recorder terminal cost reads the same
accumulator; groups merge only first terminal costs via shared algebra.
Extend public event/group projection without changing existing token totals.
- Introduce private cost-state context access. Extend `WithoutExecutionState` to
mask inherited cost state, but not contextual observer selection. Smoke
diagnostics must continue masking private execution accounting and emit no
ordinary lifecycle or enclosing-run cost contributions.
- Exercise controlled retry/error sequences and zero-cost fallback without an
observer. Test duplicate terminal delivery and independence of group terminal
coverage from cost completeness. Do not retain an attempt history for cost.
Validation: generation, observation recorder/group, and diagnostic-boundary tests.
## Stage 12 — Ordinary Run Cost Lifetime And Result Accounting
- Create/install a new private accumulator at the public ordinary Run boundary,
before optional recorder creation. Pass that same instance to the recorder.
Ensure direct internal runner execution creates a local accumulator if one
was not supplied, without replacing one supplied by the owning facade.
- Retain it through initial generation and all repair calls. Set successful
`domain.RunResult.CostUSD` from its snapshot, not by summing responses. Keep
the legacy `Usage` success-response sum and error handling unchanged.
- Ensure errors still return no partial public `RunResult`; available cost is
reported in terminal observations. Runs rejected before dispatch have the
established empty aggregate, not a configured-zero contribution source.
- Test a retry with unreported cost followed by a reported charge, mixed sources
within repairs, reported-cost precedence over zero policy, and identical
successful cost metadata with observation enabled/disabled.
Validation: assembled ordinary run/reliable execution contract tests.
## Stage 13 — Prepared Cost Lifetime And Nested Isolation
- In internal RunPrepared, after successful claim create/install its own cost
accumulator before recorder creation and generation. Use the same result and
terminal accounting path as ordinary Run. Do not store an accumulator or
contextual observer in the reusable preparation details/private ready payload.
- The claimed handle uses its frozen backend Provider/ZeroCost, even when run
from another invocation scope. Preserve admission, deadline attribution,
single-claim semantics, discard behavior, and cleared callbacks/credentials.
- Verify ordinary/prepared child runs launched from injected clients or
prepared callbacks do not inherit the parent's accumulator, even when both
inherit one contextual group observer. Each terminal contributes once to the
invocation total; parent cost includes only its own dispatched generations.
- Cover prepared zero-cost failure and repair, rejected claims without events,
independent contexts on a reused engine, and diagnostic calls nested in a run
without cost/event leakage.
Validation: prepared and nested-boundary contracts, including race-enabled focused tests.
## Stage 14 — End-To-End Consumer Regression Coverage
- Add one representative offline invocation workflow using two engines and
concurrent runs under one group. Use controlled transport/injected clients
for OpenRouter reported cost, vLLM details/configured zero, a retry, and a
failed operation after available usage. Join application-owned goroutines
before taking the final snapshot.
- Assert independently predictable terminal membership, sums, sources, token
presence/completeness, and weighted ratio (or its unavailability when an
attempt leaves input/cache-read reporting incomplete). Compare floating-point
sums with tolerance; do not assert map order, run-ID bytes, or exact timing.
- Cover a snapshot with completed fully costed members and one started member:
cost completeness concerns the included terminals while terminal coverage is
false. Cover empty and members-with-no-terminals snapshots separately.
- Reuse narrower tests from earlier stages rather than reproduce every case in
this workflow. Keep observations metadata-only; do not add bodies, credentials,
URLs, or error objects to the event contract.
Validation: focused new integration tests plus `go test -race ./...` when the
implementation has reached its final buildable state.
## Stage 15 — Canonical Documentation And Final Verification
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and
sources, client ownership, and observer/prepared lifetimes. Update stale
authentication Kind requirements in shared settings and public docs.
- Update the consumer guide and maintained offline observation example for
invocation scoping, cache/reasoning subsets, known/complete cost, configured
zeros, sources, and failure-path reporting. Maintain semantic example tests.
- Update `docs/formats.md`, the OpenAI-compatible integration contract, and
internal overview/observation/generation/model-client/runner documentation for
actual ownership and flow. Explain provider defaults versus declared inspection,
vLLM reporting settings, and the exact diagnostic response recognition.
- Document migration from required authentication kind and block-clearing
semantics. Record the change in the next unpublished release note when such a
note exists; otherwise leave release-note creation to release preparation.
Do not edit published v0.24.0 notes or tag/publish anything in this sprint.
- Use the newest already installed Go toolchain per the development guide,
`GOTOOLCHAIN=local`, and `GOWORK=off`. Format changed Go files, run
`go test -race ./...`, `go vet ./...`, and `git diff --check`. Reuse successful
results for unchanged content; do not add redundant builds or live-provider
checks. Verify local Markdown links and maintained examples through their
normal offline tests. Review the entire diff for unrelated changes and secrets.
- Audit completion against every target-state section, especially no-observer
cost, failed dispatched attempts, optional-field tolerance, prepared policy
freezing, and nested/diagnostic separation. Resolve remaining implementation
defects before reporting sprint completion. Roadmap retirement is later
housekeeping, not part of this implementation plan.
There are no unresolved product decisions required to begin these stages.