Record correction repair integration findings

This commit is contained in:
2026-11-11 12:11:36 +00:00
parent e029730c79
commit d3fbddce11

View File

@@ -2,9 +2,10 @@
**Updated:** 2026-11-11
**Status:** Two open findings from the stage-by-stage review of the
[implementation plan](implementation.md), against committed revision `aa8073c`.
This document records required follow-up work; it does not implement the fixes.
**Status:** Four open findings from the stage-by-stage review of the
[implementation plan](implementation.md), against implementation revision
`aa8073c`. R1 and R2 were confirmed in the follow-up audit, which added R3 and
R4.
## Review Result
@@ -16,10 +17,10 @@ fresh validation and enter deterministic application under the existing conflict
checks and stage barrier. Ordinary reports summarize branches; debug artifacts
retain detailed evidence.
The review found two gaps in outcome attribution and retained diagnostic evidence.
Neither reproduced case applies an invalid correction or changes the transcript
incorrectly. Both should be addressed before treating reporting and diagnostics
as complete.
The review found gaps in outcome attribution, retained diagnostic evidence,
and replay of configured cache settings. The default validation path also
repeats replacement previews across validators. The reproduced cases preserve
correction acceptance and application.
No additional strengthening of invariants, contracts, prohibitions, or guarantees
was identified in the reviewed implementation. In particular, the new semantic
@@ -103,36 +104,92 @@ in its single final debug row. Verify that it remains `approved_skipped`, has no
applied-edit fields, and does not create a duplicate ledger row or an additional
operational error.
### R3 — Preserve Cache Settings When Replaying Repair Turns (P3)
**Affected plan:** Stage 3, items 3–5.
**Evidence:** [The adapter](../../internal/framework/llm/adapter.go#L117) appends
the current repair instruction as native PromptKit messages. It retains that
instruction for the next attempt through `convertMessages`, which copies only
role and content. On the next attempt the adapter reconstructs those prior
turns without their `CacheControl`. Operator prompt replacements already
support PromptKit's per-message `cache_control` field.
**Reproduced behavior:** Override the grammar repair prompt with a user message
using `cache_control: {type: ephemeral, ttl: 1h}`, then execute two repair
turns on the same branch. The first call sends the instruction as a cached text
block. The second call sends the same earlier instruction as a plain string,
while the new instruction still carries the cache settings. The transport
confirmed this difference with a local fixture.
**Required change:** Preserve native cache metadata for rendered instructions
carried into later turns. Keep its conversion in the adapter boundary and leave
wire encoding to PromptKit.
**Acceptance:** Extend the two-turn adapter test with a cache-controlled
operator repair prompt. Verify that the previous instruction retains its
cache settings, text, and role when replayed. The test should check the
outbound request without depending on provider caching behavior.
### R4 — Reuse Replacement Previews Across Feedback Checks (P3)
**Affected plan:** Stage 5, items 1–2.
**Evidence:** [feedbackCandidates](../../internal/framework/runner/feedback.go#L30)
calls `feedbackPreview` for every candidate at every validator, including
shape and confidence checks that never use it. Each call scans the section for
the target, then builds the replacement preview. The
[runner](../../internal/framework/runner/runner.go#L744) previews the candidate
again for ambiguity, and [validator request construction](../../internal/framework/validators/llm_validators.go#L269)
rebuilds the preview for the LLM payload. The default positive repair budget
adds this work even when all proposals pass the initial chain.
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
proposals exercised the same five deterministic validators at budgets zero
and one, with identical approvals and no repairs. On the review machine,
validation rose from about 0.28 to 0.50 ms and allocated bytes from about 0.40 to
0.68 MB per section. With 1,000 segments and proposals, the same probe rose from
about 4.2 to 8.0 ms and from 5.4 to 7.8 MB. These measurements isolate local
validation work.
**Required change:** Reuse the existing replacement preview across applicability,
ambiguity, and validation payload construction for a given candidate version
where the rules use the same before/after text. Reuse a section target index and
compute previews only for checks that need them. Recompute the preview for a
revision, and preserve the separate application check against the current text.
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
passing, including ambiguous spans, blank results, invalid confidence, and
protection exceptions. Re-run a focused offline benchmark to verify reduced
scanning and allocation in the feedback path.
## Stage Review Coverage
| Stage | Evidence reviewed | Result |
| --- | --- | --- |
| 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding |
| 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding |
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, failure classification, native structural-repair accounting | No additional finding |
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, cache control, failure classification, native structural-repair accounting | R3 |
| 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding |
| 5 — Findings | Applicability selection, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | No additional finding |
| 5 — Findings | Applicability selection, preview reuse, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | R4 |
| 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 |
| 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 |
| 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 |
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass; add regressions for R1 and R2 |
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass |
## Validation Performed
- `go test ./...` passed.
- `go test -race ./internal/framework/runner ./internal/framework/llm ./cmd/audita`
passed.
- `go build -o /tmp/audita-review ./cmd/audita` passed.
- The fresh `go test -count=1 ./...` run passed.
- Fresh race runs passed for the runner, LLM adapter, process reporting, and CLI.
- The CLI build passed.
- The repair-preview command in [the example guide](../../examples/README.md)
successfully rendered the grammar repair instruction and schema without a
provider call. Maintained-example integration tests also passed in the full
suite.
- All 67 local links and anchors in the 13 documentation files changed by the
implementation resolved.
- Two temporary Go-overlay probes reproduced R1 and R2 against the committed
code. Their assertions failed as described above. They were not added to the
repository; the existing suite does not detect these gaps.
- The implementation documentation passed its link review. The follow-up
checked the links in this roadmap and ran `git diff --check`.
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
validation overhead in R4 against the committed implementation.
Resolve both findings with focused regressions, then repeat the affected report,
ledger, runner, and assembled-command checks. Retain the current architecture and
accepted operator choices while closing these evidence gaps.
Resolve the findings with focused checks at their owning boundaries. Retain the
current architecture and accepted operator choices.