audita/docs/roadmap/correction-repair.md

Correction Repair After Validation

Updated: 2026-11-11

Status: Feature direction and all six decisions accepted; implementation pending. No open feature decisions remain. Configuration names, concrete schema fields, and adapter details remain implementation-planning work. This document owns the target and remaining work, not a release date or implementation status for the completed library adoption.

Objective

Give a rejected correction an opportunity to improve using specific validator feedback, while preserving proposals that have already passed validation. Each correction module supplies a repair prompt. Each validator supplies findings within its own remit. Audita coordinates bounded revision conversations and acceptance through the existing PromptKit runtime.

The unit of repair is an individual correction proposal. Multiple proposals against one transcript segment remain separately identifiable. A model may revise the selected proposal or explicitly withdraw it. Every revised proposal must pass the full applicable validation chain before it becomes eligible for application.

This adds domain revision after rejection. It is distinct from PromptKit's structural output repair and from obtaining an unavailable validator decision. The latter remains separate deferred work.

Current Baseline

Reviewed application revision: 8946a82. Recheck the owning components when implementation begins, including any subsequent correctness-review fixes.

• The pipeline partitions the transcript once and runs ordered stages. Proposals can be generated concurrently; indexing and application remain deterministic. Approved corrections are applied in proposal order, with existing replacement and conflict checks.
• Proposal generation returns a correction set, potentially with several proposals per segment. Model-facing segment references are request-local and are mapped to transcript identities. The four modules are glossary, homophones, spoken-word cleanup, and grammar.
• The validator runtime runs deterministic checks before LLM-backed checks and filters rejected candidates before later validators. Decisions already carry proposal identity, disposition, reason code, and message. Unavailable decisions are distinguished from editorial rejection. Validator chains are currently selected by module, not individually configured by operators.
• The LLM adapter uses PromptKit v0.21.0 prepared execution with ConfigKit v1.2.0. It captures prepared-request data and final response text. Proposal-generation results carry candidates, operation outcomes, and artifact references, but do not retain a conversation for domain revision.
• Embedded prompts live in the audita collection and support operator replacement through native underlays. Audita supplies response schemas separately. Full interaction artifacts are optional debug output.
• Existing llm.repair_attempts controls structural repairs. It is not a budget for the proposed domain revision loop. No such loop is implemented.

Targeted End State

Proposal Lifecycle

1. Generate the initial correction set using the stage's instruction, transcript section, relevant glossary/protection data, and response schema.
2. Validate candidates, collecting findings from checks that can meaningfully evaluate them. Checks that cannot run do not become invented rejection reasons.
3. Retain approved proposals for application. Give each rejected proposal a repair branch containing its conversation and findings. Budget and eligibility rules determine whether another request can run.
4. Request revision or withdrawal of that one proposal. An accepted withdrawal ends the branch without an edit. A revised proposal returns to the complete applicable validator chain; approvals of a previous version do not transfer.
5. Continue until approval, withdrawal, budget exhaustion, or a classified failure. Preserve independently accepted work under the existing partial-success rules.
6. Apply accepted corrections in deterministic proposal order, retaining the existing apply-time checks for stale spans, protection, and conflicts.

Withdrawal or exhaustion contributes no edit from that proposal. Other accepted proposals against the same segment may still change its text. Retaining an approval does not guarantee application when another edit makes it inapplicable. This feature does not add a second repair loop for application-time conflicts.

Each revision remains associated with the original proposal and its processing stage and section. Repair responses must identify the requested proposal; unrelated edits do not become additional candidates. Revisions may correct the proposed span or target reference within the original section context, as specified in the accepted target decision. Exact internal types and model-facing reference spelling are implementation choices, not new globally unique identity requirements.

Full Conversation and Independent Branches

The first repair request contains the complete initial rendered request messages, with their original roles and order, followed by the complete assistant response that supplied the decoded proposal set, then the targeted repair instruction. Retain the original section and request inputs for this purpose; do not rebuild the conversation from a transcript already changed by other corrections.

Every proposal branches independently from that same initial exchange. Later attempts append that proposal's previous repair response and new feedback to its own branch. Repairs of other proposals never become turns in that conversation. Keep approved proposals outside the set being reconsidered. Branch scheduling must not change proposal attribution or final application order.

Here, conversation history means the application-level message exchange. Preserve the full response used for candidate decoding rather than serializing only the failed proposal as if it were the model's original answer. Provider transport attempts and PromptKit's internal structural-repair operations remain accounted for separately. Do not manufacture hidden provider messages or reasoning.

Keep required conversation state in memory when repair is enabled, independently of debug artifact saving. Its lifetime need only cover the associated work; this does not introduce durable conversation storage or restart/resume support. Send the full replay and feedback under the revision output contract, subject to the selected provider's request limits.

Module Repair Prompts and Response Contract

Each built-in correction module owns a corresponding repair definition in the audita namespace, alongside its proposal definition. Use the existing embedded filesystem and ConfigKit underlay mechanism so operators can replace repair prompts without rebuilding Audita. Shared instruction fragments may be reused; module-specific instructions stay with the module's prompt.

The repair instruction identifies the proposal, repeats its original and proposed replacement text, presents the findings, and asks for a revision or withdrawal. It explains that only this proposal is under review and states the response contract for this turn. This must remain clear even though the initial prompt requested a complete correction set. Later turns refer explicitly to the current proposal version rather than relying on an ambiguous "last message" reference.

Use a structured response that distinguishes revision from withdrawal and carries the requested proposal reference. A revision contains the correction data needed by the existing validators. Empty replacement text remains a possible span deletion; it must not double as a withdrawal signal. The response schema owns shape validation; Audita owns correspondence and domain acceptance. Exact field names can be selected with the schema implementation.

Validator-Owned Feedback

Validators contribute their own actionable findings: validator identity, stable reason code, an explanation, and relevant evidence or constraints where available. Build on existing decision data rather than creating a parallel reason registry. For example, a protection finding can identify the spelling affected, while an editorial finding can explain the meaning change it detected.

The module repair template owns presentation of the collected findings. Validators do not construct chat histories or own the correction response schema. Order feedback by the applicable validator chain for understandable reports and prompts. Model-backed validators should contribute their assessment through the existing structured decision path.

Collect failures from all applicable checks that can meaningfully evaluate the candidate. A malformed target may prevent contextual checks; a provider failure means the assessment is unavailable, not that the edit violated a domain rule. Apply the accepted eligibility and continuation policy. Making validator chains operator-configurable is not required by this feature.

Ownership and Library Integration

Audita owns proposal lineage, semantic feedback, branch state, revision budgets, revalidation, and transcript application. PromptKit continues to own rendering, profile/backend selection, transport retry, structural repair, and per-backend admission. Configuration continues through ConfigKit's native resolution and the existing root/stage scope conventions. Standard CommandKit inspection and request commands expose the configured prompt assets through their normal APIs.

PromptKit already supports appending rendered assistant and corrective messages. Use its public preparation and execution APIs to render the module repair definition and assemble the intended conversation. Appended messages are already rendered; appending template text alone will not expand its variables. Confirm the full-history path and per-call response-schema selection with a focused integration example before settling adapter details. If that exposes a genuine upstream gap, document a small reproducer and request focused support rather than introducing another renderer or provider client.

Maintain the existing distinctions between recoverable model-operation failures, fatal application/invariant failures, and cancellation. Domain revision attempts are additional logical calls, each potentially containing PromptKit retries and structural repairs; account for these separately. Share existing backend admission and cancellation instead of adding a repair-specific global concurrency limiter. Use the accepted reporting scope.

Work From the Baseline

1. Prove the request path. Exercise native message composition with an original multi-proposal response, a rendered module repair instruction, a revision response schema, and a subsequent branch turn. Confirm operator prompt replacements and the same behavior with debug disabled.
2. Carry conversation state through the existing boundaries. Extend framework/structuredoutput, the LLM adapter, proposal-generation results, and module/framework contracts as needed to retain and replay the original request and full response. Preserve source-reference mapping and metadata. Use compact lineage data and shared immutable initial context where practical; no particular storage layout or new package hierarchy is prescribed.
3. Add module repair assets and the response schema. Extend the prompt registry, embedded definitions, shared fragments where useful, and framework/responseschema. Provide revision/withdrawal decoding and correspondence checks without changing ordinary span-deletion semantics.
4. Adapt feedback collection. Update validator findings where their current messages are insufficient for repair. When repair is enabled, collect the agreed applicable findings and distinguish failures, unavailable decisions, and checks that could not run. With an effective budget of zero, use the existing validator filtering. Keep complete validation mandatory for approval.
5. Implement the bounded branch loop. Integrate revision calls and full revalidation into the current stage/section flow. Preserve accepted proposals, branch isolation, deterministic application, cancellation, and existing apply-time checks. Unsuccessful branches must not discard unrelated accepted work. This does not require redesigning stage barriers or chunk scheduling.
6. Wire configuration and readiness. Add the accepted budget and profile settings with native presence and stage inheritance. Keep semantic attempts distinct from structural llm.repair_attempts. Preflight checks any additional required role/prompt readiness when enabled, without making disabled repair require an unused profile. Let existing inspection expose resolved settings.
7. Extend outcomes and documentation. Implement the accepted reporting scope through ordinary reporting, correction ledgers, and optional redacted debug artifacts. Update the canonical configuration, pipeline, module, validator, prompt, runtime, and output documentation when behavior lands. Record significant ownership decisions through the documentation policy's ADR process if warranted; routine API and template choices need no ADR.

These are work dependencies for later implementation planning. They do not mandate separate commits, a general workflow engine, or a new release process.

Acceptance and Completion

Use controlled model responses and real loading/rendering at integration boundaries. Cover the following behavior with proportionate tests:

• One approved and one rejected proposal from the same response: repair only the rejected proposal, then apply the accepted results in deterministic order. Include several proposals against one segment and an overlapping-edit case.
• Full initial messages and full response retained, correct roles and ordering, explicit proposal selection, independent branches, and a second repair turn containing only its own branch's history. Exercise debug enabled and disabled.
• Repair success, explicit withdrawal, budget exhaustion, the default of one semantic attempt, additional configured attempts, and global or stage-level zero disabling both repair and extra feedback collection. Distinguish withdrawal from an empty-string deletion proposal.
• Validator-owned findings collected under the accepted continuation policy; inapplicable or unavailable checks do not masquerade as editorial rejection. Cover known rejection alongside an unavailable assessment, unavailable assessment alone, and a no-effect proposal finishing without repair. Revised candidates must pass all applicable checks, including checks that approved their earlier versions.
• Wrong proposal references and unusable repair responses cannot introduce unrelated edits or approval. Preserve ordinary identity reconciliation and structural validation; no generic salvage of malformed JSON is implied.
• A revision can correct its source span or segment reference within the original request's section, preserving proposal lineage and passing ordinary target mapping and full validation.
• A provider error on a repair request, including a context-size rejection, fails only that proposal after any applicable PromptKit retries. Other approved work remains eligible for application. Cover structural-repair exhaustion and cancellation under the existing failure boundaries. Account for semantic attempts, structural repairs, and transport retries separately.
• Embedded module repair prompts and operator replacements work through the shared engine. Root and stage settings follow the accepted scope and fallback rules, including inheritance of the effective proposal profile after stage settings are combined and explicit repair-profile overrides. Verify that ordinary reporting records one compact outcome per repair branch and that optional redacted debug output carries the detailed attempt history.

Run focused checks during development and assembled application validation at completion under the testing policy. Representative transcript comparisons can help tune prompts and defaults, but this roadmap sets no numerical quality threshold or requirement for paid provider calls. Completion requires the agreed loop and accepted policies, acceptance evidence, and updated canonical documentation. Successful schema validation alone is not evidence that a revised correction is acceptable.

Accepted Decisions

Semantic Repair Budget and Configuration Scope

Allow one additional semantic repair attempt per rejected proposal by default, configurable at the root and per stage. Zero disables both the repair loop and extra feedback collection. A revision that fails validation can continue when the operator configures more attempts. Use a distinct setting from structural llm.repair_attempts. Retain existing per-call timeouts and parent cancellation, and document that each semantic attempt may include transport retries and structural repairs. This provides useful feedback-driven revision with a simple finite bound. The default and scope are accepted; exact configuration names remain an implementation-planning choice.

Provider Request Limits

Submit the full repair conversation and let the selected provider enforce its request limits. If a provider error prevents revision after any applicable PromptKit retries, including when the conversation exceeds the provider's context window, record the repair as unavailable and end that proposal's branch. Continue processing other proposals. Audita's existing fatal application error and cancellation behavior still applies.

Repair Execution Profile

Inherit the effective proposal profile by default, with an optional root/per-stage repair-profile override. Resolve fallback after stage settings are combined, following the existing validation-profile pattern. This keeps ordinary configuration small while allowing a different model for a difficult revision. Backend capacity remains shared through PromptKit.

Feedback Eligibility and Unavailable Validation

When repair is enabled, collect all meaningful deterministic failures, then run the applicable semantic validators when the candidate has the usable target and before/after data they need, even if another independent check rejected it. With an effective repair budget of zero, later validators receive only the candidates that advance under the current validation flow. An actionable rejection may enter repair; no-effect proposals can finish without repair because they contribute no edit. Build a small applicability map from the actual checks rather than running validators on unusable inputs or encoding universal dependencies between them.

Keep unavailable validation as a separate operation outcome. If a proposal also has an actionable rejection, its repair may proceed using the known findings, with unavailable assessments identified as such; any revision still needs a full successful validation chain. An unavailable assessment by itself should retain the current unavailable outcome, leaving decision-recovery calls to the deferred feature. This produces useful feedback without treating infrastructure failure as evidence against the proposed wording.

Ordinary Reports and Attempt History

Use one compact repair-outcomes section in ordinary reporting. Give each repair branch one entry with the original proposal reference, its attempt count, and final disposition. Use the existing work-unit context for stage and chunk identity. Make it clear whether an approved revision was applied. Record operational errors and recoveries through the existing report framework and keep the current exit-status policy. Editorial rejection and voluntary withdrawal are correction outcomes. Keep conversation text and per-attempt details in optional redacted debug artifacts. Choose the report location, fields, and any schema version change during implementation planning.

Revising the Span or Target Reference

Allow the revised proposal to correct its original-text span and segment reference within the original request's section, while retaining its proposal lineage. Apply the ordinary target mapping, replacement policy, and full validation to the new candidate. This permits repair of an incorrect source span or model-facing target reference, both of which can cause validation failures. It adds no new transcript-wide targeting mechanism.