eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Plan correction repair implementation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-11 04:00:47 +00:00
parent b036336e2a
commit 93a080059d
2 changed files with 372 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

367
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,367 @@
# Correction Repair Implementation Plan
Implement the stages in order, one assignment at a time. The [feature
roadmap](correction-repair.md) owns the intended behavior and operator choices.
The implementation baseline is `b036336`, with PromptKit v0.21.0 and ConfigKit
v1.2.0. Start with the [contributor guide](../development.md) and read the
applicable policies and component contracts as the work reaches them.
This is one end-to-end sprint. Intermediate stages may leave callers or tests
pending their later stages. Carry those changes forward directly and run a
focused check when its boundary is assembled. Complete the assembled checks in
Stage 9. Keep the existing fixed chunks, stage barriers, module scopes, provider
admission, and apply-time safeguards. Decision recovery for unavailable
validator responses remains [deferred](future.md#domain-aware-response-recovery).
## Stage 1 — Define the repair response and prove native conversation composition
1. In `internal/framework/responseschema`, register `CorrectionRepairKey` with
schema ID `audita.correction_repair`, name `audita_correction_repair_v1`, and
asset `correction_repair.v1.json`. Use a closed JSON object with three
required fields below.
- `proposal_ref` is the one-based position in the **original correction
array**, including items the application later rejects. Each turn echoes
the same short reference for that proposal.
- `action` is `revise` or `withdraw`.
- `correction` is a nullable object. Its object shape uses the existing
correction item fields `id`, `original_text`, `corrected_text`, and
`confidence`. The `id` refers to a segment position in the original
section, independent of `proposal_ref`. Keep the existing correction
field constraints, including an empty replacement and a whitespace-only
original span.
2. Decode the response at the repair boundary. Require the expected reference
and the consistent action/payload pair (`revise` with an object,
`withdraw` with null). Treat an inconsistent pair or wrong reference as an
unusable response with the existing `invalid_output` reason. Keep source
span, segment applicability, and confidence threshold decisions with domain
validation. PromptKit owns structural validation and its configured repair
attempts.
3. Add a focused offline test at the PromptKit integration boundary, using the
new schema and a fixture with a multi-message proposal and a separate repair
definition. Use the original proposal ID and variables as the base of each
native request. Render the repair definition with `Engine.Prepare`. Append
the full assistant response from the proposal call, then the rendered
repair instruction using `RunRequest.AppendedMessages`. Set `Validation`
to the repair schema and execute via `PrepareExecution` and `RunPrepared`.
Exercise a second turn by appending the prior instruction and its full
assistant response once, followed by the newly rendered instruction. The
fixture should check the actual messages and schema received by a controlled
transport, including use of a separate repair profile.
4. Use this test to prove that the pinned API can reproduce the initial
prepared prefix and use a different output contract. The native renderer
uses the captured prompt and supplied template variables, and appends the
already-rendered suffix after that definition's messages. Check roles,
contents, and ordering in the prepared request. If the test exposes a
genuine gap, record the failing case and reproducer for a focused upstream
change. Keep rendering and execution with PromptKit.
## Stage 2 — Add and register the four module repair prompts
1. Add the definitions `audita/modules.glossary.repair`,
`audita/modules.homophones.repair`, `audita/modules.spoken_word.repair`,
and `audita/modules.grammar.repair` beside the corresponding proposals in
`internal/prompts/assets/prompts`. Register them through `internal/prompts`
and the existing embedded PromptKit underlay. Each built-in module supplies
its repair prompt ID through a small `RepairPromptID() string` capability.
Retain the existing proposal interface for callers that only generate
candidates.
2. Use a user message in each shipped repair definition. Set its native output
contract to JSON schema using `correction_repair.v1.json` so standard
PromptKit inspection can prepare it with its own inputs. Put shared response
instructions in `RepairContract` and expose that fragment through the
existing shared-variable helpers. Render `RepairRequestJSON` from the
current proposal reference, current local segment reference, original and
replacement spans, confidence, and ordered findings. For findings, include
validator key, reason code, and available explanation. Convey an unavailable
assessment as such using safe facts from its decision or operation.
3. Make the instruction unambiguous after the earlier correction-set exchange.
Identify which proposal and current version it addresses, ask for one
revision or withdrawal, and describe the separate proposal and segment
references. Keep each module's existing edit scope, literal matching,
glossary/protection rules, and span deletion semantics. Treat transcript,
earlier model responses, and validator explanations as data in the prompt.
4. Cover schema shape, action and reference correspondence, empty-string
deletion, and real rendering of all four registered assets. Use the standard
inspection path for a replacement definition and a configured message file
through the existing source-loading path. The assembled branch test will
cover execution with that replacement.
## Stage 3 — Carry the original exchange through proposal generation and the adapter
1. Use `internal/framework/structuredoutput` for internal conversation data
shared by the role client and runner. Extend the existing structured request
with optional repair replay data. Keep the repair definition as the
requested prompt ID in Audita and carry the original proposal ID, frozen
effective variables, and original prepared messages in a shared read-only
exchange. Retain the full final assistant content used to decode the
original correction set, including when PromptKit structurally repaired
earlier output. Capture the application-level exchange at the existing
prepared execution boundary.
2. Pass the exchange from `GenerateCandidates` through
`ExecuteModuleProposal` and `contracts.ProposalResult` to the section
worker in `runner`. Capture it on a successful proposal response when the
effective repair budget is positive, independently of debug settings.
Reuse the existing request-local segment mapping and keep each proposal's
one-based position in the response aligned with its runner-assigned
proposal index. Hold one initial exchange per section and share it among
branches. Keep the exchange out of ordinary serialized results and release
it when the section's repair work finishes.
3. In `internal/framework/llm/adapter.go`, render the current repair definition
against the repair profile via the same engine and append its rendered
message block to the branch suffix. Form the final request from the
captured proposal prompt ID and variables, the original assistant response,
prior turns from that branch, and the current instruction. Pass the repair
profile, per-call repair schema, timeout, transient retry budget, and
structural repair budget through the existing role client. Preserve
the original message roles and contents. Use the prepared handle that
passed the original-prefix comparison for execution. Treat a preparation
invariant failure as an application failure and discard the unused handle.
4. Return the rendered repair instruction block and the final response for
this turn, so a subsequent attempt appends that exact exchange once. Keep
the repair definition ID in debug attribution alongside native prepared
metadata for the base prompt. Reuse typed failure and terminal count
handling. Retain the existing distinction between caller cancellation,
local setup or invariant errors, and provider errors. An upstream API
rejection of a repair request follows the recoverable branch path even
when PromptKit supplies no usable model output.
5. Exercise the role client with the test from Stage 1 using a real registered
repair asset and an operator replacement. Verify full history, isolated
turns, selected repair backend/profile, and counts for one logical attempt
that uses a native retry or structural repair. Verify capture without
`--debug` and ordinary proposal behavior with repair disabled.
## Stage 4 — Resolve configuration and wire the repair role
1. In `internal/core/config`, add `llm.correction_repair_attempts` as a
nonnegative integer, default `1`, and `llm.correction_repair_profile` as an
optional profile selection. Reuse the stage `llm` block. Omitted stage
attempts inherit the effective root value. An explicit stage zero clears
the inherited budget. Use ConfigKit's existing scalar validation and
presence rules, with no extra maximum for the new attempt count.
2. In `resolveSettings`, follow the recorded validation-profile pattern.
Resolve the effective stage proposal profile first. An explicit repair
profile selects that profile. An omitted stage repair profile inherits
an explicit nonnull root repair profile, otherwise it falls back to the
stage's effective proposal profile. An explicit stage null clears the root
override and uses that same fallback. At root, omission or null falls back
to the effective root proposal profile. Preserve recorded origins and
detached profile pointers. Use existing YAML, overlays, and native
application configuration inspection.
3. Extend `CheckReadiness` and the CLI processing wiring in
`internal/cli/process_inputs.go` and `process_pipeline.go`. Check the selected
repair profile and backend only for stages with a positive budget. The
existing captured engine validates registered prompt assets and their
overrides during loading. Retain that path for repair prompts. Allocate
a repair role client in the module run spec only for enabled stages,
using the same runtime, stage execution settings, admission, and optional
observation hooks. Unused repair profile IDs require no readiness check
when repair is disabled. A normalization-only run has no repair role.
4. Test root/stage inheritance including different proposal profiles, null and
explicit overrides, zero and positive budgets, native effective inspection,
and readiness before output preparation. Use the existing configuration
version and presence conventions.
## Stage 5 — Collect usable findings while preserving approval semantics
1. Refactor `validateSectionCandidates` in `internal/framework/runner` so the
initial pass and later revisions use the same validation code. Keep the
existing shortlist flow when the effective repair budget is zero. In
feedback mode, track decisions by proposal index and version. Use
the current validator order and batching. Retain approval eligibility only
for candidates that satisfy all required checks. Collect additional findings
separately for candidates already rejected.
2. Make applicability explicit for the built-in checks using the current
candidate and the original section.
- Shape can evaluate malformed candidates. No-effect can evaluate literal
equality on a usable span, and confidence can evaluate a valid score
independently of target or span placement.
- Original-text presence needs a meaningful target and nonempty span.
Use the existing replacement preview and `ambiguous_original_text` reason
to provide an actionable finding when the span has multiple matches. In
feedback mode, record it before withholding the candidate from semantic
validation. A shape verdict for an unmapped reference already supplies
target feedback.
- Evaluate protection where the existing rule has usable before/after
evidence. Preserve active glossary match boundaries and spoken-word
repetition rules when they require full-segment context. Run the
nonempty-result guard when a replacement can be located, including a
preview that would leave a blank segment.
- Reuse the preview for semantic validators when it supplies usable
before/after text. Continue across independent semantic checks after
an actionable rejection or unavailable decision when the candidate still
supplies that input. Preserve the existing model batching budget.
3. Apply internal decision cardinality to the candidates actually submitted to
each validator. A skipped check supplies no decision. Keep missing,
duplicate, and unusable model decisions as unavailable under existing
reconciliation. Retain recoverable operation evidence. Route fatal errors
and cancellation through the existing peer cancellation and join path.
4. Build feedback from validator decisions and safe optional evidence that the
validator owns. Reuse existing reason codes and messages. Supply specific
evidence where a reason alone gives too little direction, such as the
applicable confidence threshold or a protected spelling when identified.
Use normal LLM decision explanations for semantic judgments. Preserve
debug redaction and avoid putting raw provider errors into feedback.
5. Test two independent meaningful findings after one rejects, a usable
rejected candidate that reaches the semantic checks, invalid target and
ambiguous span handling, and a no-effect candidate with no repair work.
Confirm that an unavailable decision alone remains unavailable and that
the zero-budget pass calls later validators with the existing shortlist.
## Stage 6 — Run independent repair branches and apply accepted revisions
1. Extend the section validation worker in `runModulePipeline` after its
initial pass. Keep the approved candidates as they are. For each candidate
with an actionable rejection, initialize an independent branch on the
section exchange. No-effect and unavailable-only outcomes finish without a
branch. When an actionable rejection coexists with unavailable validation,
use the known findings and show the missing assessment as unavailable.
2. Visit repairable candidates in proposal-index order within each section
worker. Reuse existing section concurrency and engine admission. Each
logical repair call consumes one attempt from that proposal's budget.
Track attempts separately from native transient retries, structural
repairs, and revalidation calls. Send the full conversation to the selected
provider without local size checks or truncation. The validation batching
budget still applies to validator calls. End the branch on a provider error
after applicable native retries, including context-window rejection. Leave
other proposals eligible to continue. Keep the existing fatal/cancellation path for app
failures and parent cancellation.
3. Accept a valid withdrawal as a terminal correction outcome. For a
structurally usable revision with a matching proposal reference, map its
segment position using the original section, retaining the original
stage, chunk, and proposal index. Allow a changed target or span inside
that section. Run the full applicable chain from scratch against the
original section. Place a fully approved revision into the original
candidate's application position. A still-actionable rejection can take
another turn if budget remains. Use only the current version's findings
for the next instruction, retaining the earlier turns in that branch's
history.
4. End a rejected revision that has no further actionable finding with a
rejected disposition, and record an actionable rejection at the budget
boundary as exhausted. An unavailable-only revalidation, recoverable
repair-call failure, or unusable response ends the branch as unavailable.
Use the interrupted disposition when fatal failure or cancellation leaves a
branch in progress or prevents an approved revision from reaching
application. Preserve already-terminal dispositions. A revision needs its
own approval, and each response can supply at most one candidate.
5. Merge initial approvals and accepted revisions into the existing section
application path, keeping ascending proposal indices and all apply-time
checks against the text produced by earlier edits. Resolve applied versus
skipped status from the actual application result. Keep the stage-level
commit barrier. Count filtering once for a defective original that remains
excluded, using its original filtering reasons. An approved revision takes
that candidate's place in the existing application accounting. Attribute
defects in a later repair response to its own operation. Keep one terminal
proposal outcome per branch, with superseded versions available as debug
evidence.
6. Test a response with approved and rejected proposals on the same segment,
a successful retargeted revision, a valid span deletion, withdrawal,
exhaustion across two attempts, and overlapping edits where a repaired
candidate passes validation and then skips at application. Check unrelated
approvals survive a recoverable provider failure. Confirm revalidation
revisits a validator that approved an earlier version.
## Stage 7 — Account for operations and publish compact ordinary outcomes
1. Add `correction_repair` as an operation kind in
`internal/framework/outcomes`. For each repair call retain its proposal
identity, attempt number, operation counts, and safe terminal reason.
Attribute revalidation calls to the proposal and attempt internally so
their local batch indices can repeat safely. Keep the normal proposal
and initial validation accounting, including proposal response counts.
Extend deterministic report sorting
to visit initial proposal and validation evidence first, then repairs by
proposal, attempt, and validator/batch order.
2. In `processreport`, `reporting`, and the runner's work-unit results,
count repair and revalidation executions as logical operations, including
failed provider requests. Keep existing accounting for pre-call failures.
Use the existing work-unit error paths for recoverable model
failures and the run-level path for fatal or canceled work. Keep failures
from earlier attempts visible when later work succeeds. A usable revision
or withdrawal is a successful model operation. Use existing native retry
and structural-repair reasons where applicable. Add `correction_revised`
to the repair operation that yields
a fully approved revision. This recovery identifies validation success.
Actual application is reflected in the final branch disposition and the
existing correction count. A clean withdrawal or ordinary rejected revision
adds no error or recovery by itself.
3. Add a single optional **Correction repair** section to `report.md`, with
one row per branch that attempted a repair. Use the same stage and chunk
numbering as the matrices, the original internal proposal index, the
semantic attempt count, and a final outcome. Use `applied`,
`approved_skipped`, `withdrawn`, `rejected`, `exhausted`, `unavailable`,
and `interrupted` as applicable. Derive the row from compact work-unit facts
after application or from the available interrupted evidence. Sort by stage,
chunk, and proposal index. Keep the section focused on these fields.
4. Keep the existing three output files, current YAML envelope versions, and
existing YAML fields. Document `correction_repair` and
`correction_revised` as additions to the operation and reason vocabulary.
Keep compact branch rows in the Markdown report. Reporting-disabled runs
still compute the existing CLI counts. Keep debug capture independently
selectable.
5. Verify the report for successful application, approval followed by skip,
withdrawal, exhaustion, provider failure, and partial evidence on fatal
interruption. Check that semantic attempts, actual edits, and PromptKit
retry/repair counts are distinct and that both YAML summaries agree with
the CLI on the counts they share.
## Stage 8 — Retain optional repair evidence and preserve ledger lineage
1. Extend the current interaction writers and logical execution observation
data with a repair role, original proposal index, and one-based semantic
attempt number. Use generated stage instance, section, proposal, and
attempt coordinates for repair filenames and include validator and batch
coordinates for revalidation. Original stage labels remain data. Reuse the
existing diagnostics failure and credential redaction path, including the
selected repair role's prepared credential references.
2. Save each repair attempt's rendered conversation, response, feedback,
and validation evidence through existing debug JSON artifacts. Keep these
writes conditional on an active debug directory and keep runtime repair
functional when preparation or writing of optional artifacts fails.
3. Extend `processreport.BuildCorrectionLedger` to emit one final row per
repaired proposal keyed by the original stage instance and proposal
index. Represent the original proposed edit and the final applied edit
separately, including when the revision retargets the proposal. Use
compact optional repair attempt metadata and artifact references to
connect versions and their respective decisions. Keep initial proposal
rejections as evidence within that lineage. Preserve the existing ledger
behavior for untouched candidates and the indexed interrupted join.
4. Test several repairs against a single segment and revalidation batches
that reuse local indices, then verify unique debug artifact names and
unambiguous ledger attribution. Check the terminal row and prior
decisions on interruption. Check redaction with synthetic credentials in
captured interaction material and verify that ordinary reports contain
only safe outcome data.
## Stage 9 — Validate the assembled command and update canonical documentation
1. Exercise the normal processing command with controlled provider responses,
real configuration and prompt loading, and the output session. Cover a
mixed proposal set with two independent repair branches, a second turn
on one branch, provider context rejection on the other, full revalidation,
and preserved original approvals. Inspect the wire order and contract.
Run the relevant behavior with debug off and with an operator repair prompt
replacement under the usual underlay. Cover zero budget and independent
report/debug selection. Reuse lower-level cases for individual failure and
disposition variants.
2. Add a controlled cancellation case during a repair or its revalidation.
Confirm active work joins, no completed transcript is published, and
failed reports preserve available partial evidence. Use the existing
subprocess fixtures and output assertions where they express the contract.
3. Update the owning [configuration](../config.md),
[pipeline](../internal/pipeline.md), [modules](../internal/modules.md),
[validators](../internal/validators.md), [prompts](../internal/prompts.md),
[LLM runtime](../internal/llm-runtime.md), and
[output files](../integrations/output-files.md) guides. Explain the default
attempt, per-stage control, profile fallback, complete replay to the
selected repair provider, provider failure behavior, and compact reporting.
Add minimal configuration or preview examples where needed in the existing
maintained material. Update [operations](../operations.md) if the new debug
evidence needs operator guidance. Keep the deferred decision-recovery
entry aligned with the implemented scope.
4. Validate examples and local links. Run the full offline suite and the
assembled CLI build under the [contributor guide](../development.md),
including race detection for the section workers. Check the focused
provider and ledger regressions in the normal suite. The sprint ends
with the implementation and its evidence.