Record correction repair integration findings
This commit is contained in:
@@ -2,9 +2,10 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Two open findings from the stage-by-stage review of the
|
||||
[implementation plan](implementation.md), against committed revision `aa8073c`.
|
||||
This document records required follow-up work; it does not implement the fixes.
|
||||
**Status:** Four open findings from the stage-by-stage review of the
|
||||
[implementation plan](implementation.md), against implementation revision
|
||||
`aa8073c`. R1 and R2 were confirmed in the follow-up audit, which added R3 and
|
||||
R4.
|
||||
|
||||
## Review Result
|
||||
|
||||
@@ -16,10 +17,10 @@ fresh validation and enter deterministic application under the existing conflict
|
||||
checks and stage barrier. Ordinary reports summarize branches; debug artifacts
|
||||
retain detailed evidence.
|
||||
|
||||
The review found two gaps in outcome attribution and retained diagnostic evidence.
|
||||
Neither reproduced case applies an invalid correction or changes the transcript
|
||||
incorrectly. Both should be addressed before treating reporting and diagnostics
|
||||
as complete.
|
||||
The review found gaps in outcome attribution, retained diagnostic evidence,
|
||||
and replay of configured cache settings. The default validation path also
|
||||
repeats replacement previews across validators. The reproduced cases preserve
|
||||
correction acceptance and application.
|
||||
|
||||
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
||||
was identified in the reviewed implementation. In particular, the new semantic
|
||||
@@ -103,36 +104,92 @@ in its single final debug row. Verify that it remains `approved_skipped`, has no
|
||||
applied-edit fields, and does not create a duplicate ledger row or an additional
|
||||
operational error.
|
||||
|
||||
### R3 — Preserve Cache Settings When Replaying Repair Turns (P3)
|
||||
|
||||
**Affected plan:** Stage 3, items 3–5.
|
||||
|
||||
**Evidence:** [The adapter](../../internal/framework/llm/adapter.go#L117) appends
|
||||
the current repair instruction as native PromptKit messages. It retains that
|
||||
instruction for the next attempt through `convertMessages`, which copies only
|
||||
role and content. On the next attempt the adapter reconstructs those prior
|
||||
turns without their `CacheControl`. Operator prompt replacements already
|
||||
support PromptKit's per-message `cache_control` field.
|
||||
|
||||
**Reproduced behavior:** Override the grammar repair prompt with a user message
|
||||
using `cache_control: {type: ephemeral, ttl: 1h}`, then execute two repair
|
||||
turns on the same branch. The first call sends the instruction as a cached text
|
||||
block. The second call sends the same earlier instruction as a plain string,
|
||||
while the new instruction still carries the cache settings. The transport
|
||||
confirmed this difference with a local fixture.
|
||||
|
||||
**Required change:** Preserve native cache metadata for rendered instructions
|
||||
carried into later turns. Keep its conversion in the adapter boundary and leave
|
||||
wire encoding to PromptKit.
|
||||
|
||||
**Acceptance:** Extend the two-turn adapter test with a cache-controlled
|
||||
operator repair prompt. Verify that the previous instruction retains its
|
||||
cache settings, text, and role when replayed. The test should check the
|
||||
outbound request without depending on provider caching behavior.
|
||||
|
||||
### R4 — Reuse Replacement Previews Across Feedback Checks (P3)
|
||||
|
||||
**Affected plan:** Stage 5, items 1–2.
|
||||
|
||||
**Evidence:** [feedbackCandidates](../../internal/framework/runner/feedback.go#L30)
|
||||
calls `feedbackPreview` for every candidate at every validator, including
|
||||
shape and confidence checks that never use it. Each call scans the section for
|
||||
the target, then builds the replacement preview. The
|
||||
[runner](../../internal/framework/runner/runner.go#L744) previews the candidate
|
||||
again for ambiguity, and [validator request construction](../../internal/framework/validators/llm_validators.go#L269)
|
||||
rebuilds the preview for the LLM payload. The default positive repair budget
|
||||
adds this work even when all proposals pass the initial chain.
|
||||
|
||||
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
|
||||
proposals exercised the same five deterministic validators at budgets zero
|
||||
and one, with identical approvals and no repairs. On the review machine,
|
||||
validation rose from about 0.28 to 0.50 ms and allocated bytes from about 0.40 to
|
||||
0.68 MB per section. With 1,000 segments and proposals, the same probe rose from
|
||||
about 4.2 to 8.0 ms and from 5.4 to 7.8 MB. These measurements isolate local
|
||||
validation work.
|
||||
|
||||
**Required change:** Reuse the existing replacement preview across applicability,
|
||||
ambiguity, and validation payload construction for a given candidate version
|
||||
where the rules use the same before/after text. Reuse a section target index and
|
||||
compute previews only for checks that need them. Recompute the preview for a
|
||||
revision, and preserve the separate application check against the current text.
|
||||
|
||||
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
|
||||
passing, including ambiguous spans, blank results, invalid confidence, and
|
||||
protection exceptions. Re-run a focused offline benchmark to verify reduced
|
||||
scanning and allocation in the feedback path.
|
||||
|
||||
## Stage Review Coverage
|
||||
|
||||
| Stage | Evidence reviewed | Result |
|
||||
| --- | --- | --- |
|
||||
| 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding |
|
||||
| 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding |
|
||||
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, failure classification, native structural-repair accounting | No additional finding |
|
||||
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, cache control, failure classification, native structural-repair accounting | R3 |
|
||||
| 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding |
|
||||
| 5 — Findings | Applicability selection, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | No additional finding |
|
||||
| 5 — Findings | Applicability selection, preview reuse, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | R4 |
|
||||
| 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 |
|
||||
| 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 |
|
||||
| 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 |
|
||||
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass; add regressions for R1 and R2 |
|
||||
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass |
|
||||
|
||||
## Validation Performed
|
||||
|
||||
- `go test ./...` passed.
|
||||
- `go test -race ./internal/framework/runner ./internal/framework/llm ./cmd/audita`
|
||||
passed.
|
||||
- `go build -o /tmp/audita-review ./cmd/audita` passed.
|
||||
- The fresh `go test -count=1 ./...` run passed.
|
||||
- Fresh race runs passed for the runner, LLM adapter, process reporting, and CLI.
|
||||
- The CLI build passed.
|
||||
- The repair-preview command in [the example guide](../../examples/README.md)
|
||||
successfully rendered the grammar repair instruction and schema without a
|
||||
provider call. Maintained-example integration tests also passed in the full
|
||||
suite.
|
||||
- All 67 local links and anchors in the 13 documentation files changed by the
|
||||
implementation resolved.
|
||||
- Two temporary Go-overlay probes reproduced R1 and R2 against the committed
|
||||
code. Their assertions failed as described above. They were not added to the
|
||||
repository; the existing suite does not detect these gaps.
|
||||
- The implementation documentation passed its link review. The follow-up
|
||||
checked the links in this roadmap and ran `git diff --check`.
|
||||
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
|
||||
validation overhead in R4 against the committed implementation.
|
||||
|
||||
Resolve both findings with focused regressions, then repeat the affected report,
|
||||
ledger, runner, and assembled-command checks. Retain the current architecture and
|
||||
accepted operator choices while closing these evidence gaps.
|
||||
Resolve the findings with focused checks at their owning boundaries. Retain the
|
||||
current architecture and accepted operator choices.
|
||||
|
||||
Reference in New Issue
Block a user