Record correction repair integration findings
This commit is contained in:
@@ -2,9 +2,10 @@
|
|||||||
|
|
||||||
**Updated:** 2026-11-11
|
**Updated:** 2026-11-11
|
||||||
|
|
||||||
**Status:** Two open findings from the stage-by-stage review of the
|
**Status:** Four open findings from the stage-by-stage review of the
|
||||||
[implementation plan](implementation.md), against committed revision `aa8073c`.
|
[implementation plan](implementation.md), against implementation revision
|
||||||
This document records required follow-up work; it does not implement the fixes.
|
`aa8073c`. R1 and R2 were confirmed in the follow-up audit, which added R3 and
|
||||||
|
R4.
|
||||||
|
|
||||||
## Review Result
|
## Review Result
|
||||||
|
|
||||||
@@ -16,10 +17,10 @@ fresh validation and enter deterministic application under the existing conflict
|
|||||||
checks and stage barrier. Ordinary reports summarize branches; debug artifacts
|
checks and stage barrier. Ordinary reports summarize branches; debug artifacts
|
||||||
retain detailed evidence.
|
retain detailed evidence.
|
||||||
|
|
||||||
The review found two gaps in outcome attribution and retained diagnostic evidence.
|
The review found gaps in outcome attribution, retained diagnostic evidence,
|
||||||
Neither reproduced case applies an invalid correction or changes the transcript
|
and replay of configured cache settings. The default validation path also
|
||||||
incorrectly. Both should be addressed before treating reporting and diagnostics
|
repeats replacement previews across validators. The reproduced cases preserve
|
||||||
as complete.
|
correction acceptance and application.
|
||||||
|
|
||||||
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
||||||
was identified in the reviewed implementation. In particular, the new semantic
|
was identified in the reviewed implementation. In particular, the new semantic
|
||||||
@@ -103,36 +104,92 @@ in its single final debug row. Verify that it remains `approved_skipped`, has no
|
|||||||
applied-edit fields, and does not create a duplicate ledger row or an additional
|
applied-edit fields, and does not create a duplicate ledger row or an additional
|
||||||
operational error.
|
operational error.
|
||||||
|
|
||||||
|
### R3 — Preserve Cache Settings When Replaying Repair Turns (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 3, items 3–5.
|
||||||
|
|
||||||
|
**Evidence:** [The adapter](../../internal/framework/llm/adapter.go#L117) appends
|
||||||
|
the current repair instruction as native PromptKit messages. It retains that
|
||||||
|
instruction for the next attempt through `convertMessages`, which copies only
|
||||||
|
role and content. On the next attempt the adapter reconstructs those prior
|
||||||
|
turns without their `CacheControl`. Operator prompt replacements already
|
||||||
|
support PromptKit's per-message `cache_control` field.
|
||||||
|
|
||||||
|
**Reproduced behavior:** Override the grammar repair prompt with a user message
|
||||||
|
using `cache_control: {type: ephemeral, ttl: 1h}`, then execute two repair
|
||||||
|
turns on the same branch. The first call sends the instruction as a cached text
|
||||||
|
block. The second call sends the same earlier instruction as a plain string,
|
||||||
|
while the new instruction still carries the cache settings. The transport
|
||||||
|
confirmed this difference with a local fixture.
|
||||||
|
|
||||||
|
**Required change:** Preserve native cache metadata for rendered instructions
|
||||||
|
carried into later turns. Keep its conversion in the adapter boundary and leave
|
||||||
|
wire encoding to PromptKit.
|
||||||
|
|
||||||
|
**Acceptance:** Extend the two-turn adapter test with a cache-controlled
|
||||||
|
operator repair prompt. Verify that the previous instruction retains its
|
||||||
|
cache settings, text, and role when replayed. The test should check the
|
||||||
|
outbound request without depending on provider caching behavior.
|
||||||
|
|
||||||
|
### R4 — Reuse Replacement Previews Across Feedback Checks (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 5, items 1–2.
|
||||||
|
|
||||||
|
**Evidence:** [feedbackCandidates](../../internal/framework/runner/feedback.go#L30)
|
||||||
|
calls `feedbackPreview` for every candidate at every validator, including
|
||||||
|
shape and confidence checks that never use it. Each call scans the section for
|
||||||
|
the target, then builds the replacement preview. The
|
||||||
|
[runner](../../internal/framework/runner/runner.go#L744) previews the candidate
|
||||||
|
again for ambiguity, and [validator request construction](../../internal/framework/validators/llm_validators.go#L269)
|
||||||
|
rebuilds the preview for the LLM payload. The default positive repair budget
|
||||||
|
adds this work even when all proposals pass the initial chain.
|
||||||
|
|
||||||
|
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
|
||||||
|
proposals exercised the same five deterministic validators at budgets zero
|
||||||
|
and one, with identical approvals and no repairs. On the review machine,
|
||||||
|
validation rose from about 0.28 to 0.50 ms and allocated bytes from about 0.40 to
|
||||||
|
0.68 MB per section. With 1,000 segments and proposals, the same probe rose from
|
||||||
|
about 4.2 to 8.0 ms and from 5.4 to 7.8 MB. These measurements isolate local
|
||||||
|
validation work.
|
||||||
|
|
||||||
|
**Required change:** Reuse the existing replacement preview across applicability,
|
||||||
|
ambiguity, and validation payload construction for a given candidate version
|
||||||
|
where the rules use the same before/after text. Reuse a section target index and
|
||||||
|
compute previews only for checks that need them. Recompute the preview for a
|
||||||
|
revision, and preserve the separate application check against the current text.
|
||||||
|
|
||||||
|
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
|
||||||
|
passing, including ambiguous spans, blank results, invalid confidence, and
|
||||||
|
protection exceptions. Re-run a focused offline benchmark to verify reduced
|
||||||
|
scanning and allocation in the feedback path.
|
||||||
|
|
||||||
## Stage Review Coverage
|
## Stage Review Coverage
|
||||||
|
|
||||||
| Stage | Evidence reviewed | Result |
|
| Stage | Evidence reviewed | Result |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding |
|
| 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding |
|
||||||
| 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding |
|
| 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding |
|
||||||
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, failure classification, native structural-repair accounting | No additional finding |
|
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, cache control, failure classification, native structural-repair accounting | R3 |
|
||||||
| 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding |
|
| 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding |
|
||||||
| 5 — Findings | Applicability selection, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | No additional finding |
|
| 5 — Findings | Applicability selection, preview reuse, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | R4 |
|
||||||
| 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 |
|
| 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 |
|
||||||
| 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 |
|
| 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 |
|
||||||
| 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 |
|
| 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 |
|
||||||
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass; add regressions for R1 and R2 |
|
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass |
|
||||||
|
|
||||||
## Validation Performed
|
## Validation Performed
|
||||||
|
|
||||||
- `go test ./...` passed.
|
- The fresh `go test -count=1 ./...` run passed.
|
||||||
- `go test -race ./internal/framework/runner ./internal/framework/llm ./cmd/audita`
|
- Fresh race runs passed for the runner, LLM adapter, process reporting, and CLI.
|
||||||
passed.
|
- The CLI build passed.
|
||||||
- `go build -o /tmp/audita-review ./cmd/audita` passed.
|
|
||||||
- The repair-preview command in [the example guide](../../examples/README.md)
|
- The repair-preview command in [the example guide](../../examples/README.md)
|
||||||
successfully rendered the grammar repair instruction and schema without a
|
successfully rendered the grammar repair instruction and schema without a
|
||||||
provider call. Maintained-example integration tests also passed in the full
|
provider call. Maintained-example integration tests also passed in the full
|
||||||
suite.
|
suite.
|
||||||
- All 67 local links and anchors in the 13 documentation files changed by the
|
- The implementation documentation passed its link review. The follow-up
|
||||||
implementation resolved.
|
checked the links in this roadmap and ran `git diff --check`.
|
||||||
- Two temporary Go-overlay probes reproduced R1 and R2 against the committed
|
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
|
||||||
code. Their assertions failed as described above. They were not added to the
|
validation overhead in R4 against the committed implementation.
|
||||||
repository; the existing suite does not detect these gaps.
|
|
||||||
|
|
||||||
Resolve both findings with focused regressions, then repeat the affected report,
|
Resolve the findings with focused checks at their owning boundaries. Retain the
|
||||||
ledger, runner, and assembled-command checks. Retain the current architecture and
|
current architecture and accepted operator choices.
|
||||||
accepted operator choices while closing these evidence gaps.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user