diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 4f9a87a..469a278 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -2,9 +2,10 @@ **Updated:** 2026-11-11 -**Status:** Two open findings from the stage-by-stage review of the -[implementation plan](implementation.md), against committed revision `aa8073c`. -This document records required follow-up work; it does not implement the fixes. +**Status:** Four open findings from the stage-by-stage review of the +[implementation plan](implementation.md), against implementation revision +`aa8073c`. R1 and R2 were confirmed in the follow-up audit, which added R3 and +R4. ## Review Result @@ -16,10 +17,10 @@ fresh validation and enter deterministic application under the existing conflict checks and stage barrier. Ordinary reports summarize branches; debug artifacts retain detailed evidence. -The review found two gaps in outcome attribution and retained diagnostic evidence. -Neither reproduced case applies an invalid correction or changes the transcript -incorrectly. Both should be addressed before treating reporting and diagnostics -as complete. +The review found gaps in outcome attribution, retained diagnostic evidence, +and replay of configured cache settings. The default validation path also +repeats replacement previews across validators. The reproduced cases preserve +correction acceptance and application. No additional strengthening of invariants, contracts, prohibitions, or guarantees was identified in the reviewed implementation. In particular, the new semantic @@ -103,36 +104,92 @@ in its single final debug row. Verify that it remains `approved_skipped`, has no applied-edit fields, and does not create a duplicate ledger row or an additional operational error. +### R3 — Preserve Cache Settings When Replaying Repair Turns (P3) + +**Affected plan:** Stage 3, items 3–5. + +**Evidence:** [The adapter](../../internal/framework/llm/adapter.go#L117) appends +the current repair instruction as native PromptKit messages. It retains that +instruction for the next attempt through `convertMessages`, which copies only +role and content. On the next attempt the adapter reconstructs those prior +turns without their `CacheControl`. Operator prompt replacements already +support PromptKit's per-message `cache_control` field. + +**Reproduced behavior:** Override the grammar repair prompt with a user message +using `cache_control: {type: ephemeral, ttl: 1h}`, then execute two repair +turns on the same branch. The first call sends the instruction as a cached text +block. The second call sends the same earlier instruction as a plain string, +while the new instruction still carries the cache settings. The transport +confirmed this difference with a local fixture. + +**Required change:** Preserve native cache metadata for rendered instructions +carried into later turns. Keep its conversion in the adapter boundary and leave +wire encoding to PromptKit. + +**Acceptance:** Extend the two-turn adapter test with a cache-controlled +operator repair prompt. Verify that the previous instruction retains its +cache settings, text, and role when replayed. The test should check the +outbound request without depending on provider caching behavior. + +### R4 — Reuse Replacement Previews Across Feedback Checks (P3) + +**Affected plan:** Stage 5, items 1–2. + +**Evidence:** [feedbackCandidates](../../internal/framework/runner/feedback.go#L30) +calls `feedbackPreview` for every candidate at every validator, including +shape and confidence checks that never use it. Each call scans the section for +the target, then builds the replacement preview. The +[runner](../../internal/framework/runner/runner.go#L744) previews the candidate +again for ambiguity, and [validator request construction](../../internal/framework/validators/llm_validators.go#L269) +rebuilds the preview for the LLM payload. The default positive repair budget +adds this work even when all proposals pass the initial chain. + +**Measured behavior:** An offline benchmark with 100 segments and 100 valid +proposals exercised the same five deterministic validators at budgets zero +and one, with identical approvals and no repairs. On the review machine, +validation rose from about 0.28 to 0.50 ms and allocated bytes from about 0.40 to +0.68 MB per section. With 1,000 segments and proposals, the same probe rose from +about 4.2 to 8.0 ms and from 5.4 to 7.8 MB. These measurements isolate local +validation work. + +**Required change:** Reuse the existing replacement preview across applicability, +ambiguity, and validation payload construction for a given candidate version +where the rules use the same before/after text. Reuse a section target index and +compute previews only for checks that need them. Recompute the preview for a +revision, and preserve the separate application check against the current text. + +**Acceptance:** Keep the existing independent-feedback and zero-budget coverage +passing, including ambiguous spans, blank results, invalid confidence, and +protection exceptions. Re-run a focused offline benchmark to verify reduced +scanning and allocation in the feedback path. + ## Stage Review Coverage | Stage | Evidence reviewed | Result | | --- | --- | --- | | 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding | | 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding | -| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, failure classification, native structural-repair accounting | No additional finding | +| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, cache control, failure classification, native structural-repair accounting | R3 | | 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding | -| 5 — Findings | Applicability selection, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | No additional finding | +| 5 — Findings | Applicability selection, preview reuse, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | R4 | | 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 | | 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 | | 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 | -| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass; add regressions for R1 and R2 | +| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass | ## Validation Performed -- `go test ./...` passed. -- `go test -race ./internal/framework/runner ./internal/framework/llm ./cmd/audita` - passed. -- `go build -o /tmp/audita-review ./cmd/audita` passed. +- The fresh `go test -count=1 ./...` run passed. +- Fresh race runs passed for the runner, LLM adapter, process reporting, and CLI. +- The CLI build passed. - The repair-preview command in [the example guide](../../examples/README.md) successfully rendered the grammar repair instruction and schema without a provider call. Maintained-example integration tests also passed in the full suite. -- All 67 local links and anchors in the 13 documentation files changed by the - implementation resolved. -- Two temporary Go-overlay probes reproduced R1 and R2 against the committed - code. Their assertions failed as described above. They were not added to the - repository; the existing suite does not detect these gaps. +- The implementation documentation passed its link review. The follow-up + checked the links in this roadmap and ran `git diff --check`. +- Temporary Go overlays reproduced the defects in R1–R3 and measured the + validation overhead in R4 against the committed implementation. -Resolve both findings with focused regressions, then repeat the affected report, -ledger, runner, and assembled-command checks. Retain the current architecture and -accepted operator choices while closing these evidence gaps. +Resolve the findings with focused checks at their owning boundaries. Retain the +current architecture and accepted operator choices.