audita/docs/roadmap/correction-repair.md

Correction Repair After Validation

Updated: 2026-11-10

Status: Feature direction agreed; implementation pending. Open decisions below require resolution before the dependent implementation work. This document owns the target and remaining work, not a release date or implementation status for the completed library adoption.

Objective

Give a rejected correction an opportunity to improve using specific validator feedback, while preserving proposals that have already passed validation. Each correction module supplies a repair prompt. Each validator supplies findings within its own remit. Audita coordinates bounded revision conversations and acceptance through the existing PromptKit runtime.

The unit of repair is an individual correction proposal. Multiple proposals against one transcript segment remain separately identifiable. A model may revise the selected proposal or explicitly withdraw it. Every revised proposal must pass the full applicable validation chain before it becomes eligible for application.

This adds domain revision after rejection. It is distinct from PromptKit's structural output repair and from obtaining an unavailable validator decision. The latter remains separate deferred work.

Current Baseline

Reviewed application revision: 8946a82. Recheck the owning components when implementation begins, including any subsequent correctness-review fixes.

• The pipeline partitions the transcript once and runs ordered stages. Proposals can be generated concurrently; indexing and application remain deterministic. Approved corrections are applied in proposal order, with existing replacement and conflict checks.
• Proposal generation returns a correction set, potentially with several proposals per segment. Model-facing segment references are request-local and are mapped to transcript identities. The four modules are glossary, homophones, spoken-word cleanup, and grammar.
• The validator runtime runs deterministic checks before LLM-backed checks and filters rejected candidates before later validators. Decisions already carry proposal identity, disposition, reason code, and message. Unavailable decisions are distinguished from editorial rejection. Validator chains are currently selected by module, not individually configured by operators.
• The LLM adapter uses PromptKit v0.21.0 prepared execution with ConfigKit v1.2.0. It captures prepared-request data and final response text. Proposal-generation results carry candidates, operation outcomes, and artifact references, but do not retain a conversation for domain revision.
• Embedded prompts live in the audita collection and support operator replacement through native underlays. Audita supplies response schemas separately. Full interaction artifacts are optional debug output.
• Existing llm.repair_attempts controls structural repairs. It is not a budget for the proposed domain revision loop. No such loop is implemented.

Targeted End State

Proposal Lifecycle

1. Generate the initial correction set using the stage's instruction, transcript section, relevant glossary/protection data, and response schema.
2. Validate candidates, collecting findings from checks that can meaningfully evaluate them. Checks that cannot run do not become invented rejection reasons.
3. Retain approved proposals for application. Give each rejected proposal a repair branch containing its conversation and findings. Budget and eligibility rules determine whether another request can run.
4. Request revision or withdrawal of that one proposal. An accepted withdrawal ends the branch without an edit. A revised proposal returns to the complete applicable validator chain; approvals of a previous version do not transfer.
5. Continue until approval, withdrawal, budget exhaustion, or a classified failure. Preserve independently accepted work under the existing partial-success rules.
6. Apply accepted corrections in deterministic proposal order, retaining the existing apply-time checks for stale spans, protection, and conflicts.

Withdrawal or exhaustion contributes no edit from that proposal. Other accepted proposals against the same segment may still change its text. Retaining an approval does not guarantee application when another edit makes it inapplicable. This feature does not add a second repair loop for application-time conflicts.

Each revision remains associated with the original proposal and its processing stage and section. Repair responses must identify the requested proposal; unrelated edits do not become additional candidates. OD-6 addresses whether revisions can change the proposed span or target reference within the original section context; the ordinary mapping and validators still determine applicability. Exact internal types and model-facing reference spelling are implementation choices, not new globally unique identity requirements.

Full Conversation and Independent Branches

The first repair request contains the complete initial rendered request messages, with their original roles and order, followed by the complete assistant response that supplied the decoded proposal set, then the targeted repair instruction. Retain the original section and request inputs for this purpose; do not rebuild the conversation from a transcript already changed by other corrections.

Every proposal branches independently from that same initial exchange. Later attempts append that proposal's previous repair response and new feedback to its own branch. Repairs of other proposals never become turns in that conversation. Keep approved proposals outside the set being reconsidered. Branch scheduling must not change proposal attribution or final application order.

Here, conversation history means the application-level message exchange. Preserve the full response used for candidate decoding rather than serializing only the failed proposal as if it were the model's original answer. Provider transport attempts and PromptKit's internal structural-repair operations remain accounted for separately. Do not manufacture hidden provider messages or reasoning.

Keep required conversation state in memory when repair is enabled, independently of debug artifact saving. Its lifetime need only cover the associated work; this does not introduce durable conversation storage or restart/resume support. Count the complete replay, feedback, and applicable output contract when assessing request size. Context-limit handling is OD-2 below.

Module Repair Prompts and Response Contract

Each built-in correction module owns a corresponding repair definition in the audita namespace, alongside its proposal definition. Use the existing embedded filesystem and ConfigKit underlay mechanism so operators can replace repair prompts without rebuilding Audita. Shared instruction fragments may be reused; module-specific instructions stay with the module's prompt.

The repair instruction identifies the proposal, repeats its original and proposed replacement text, presents the findings, and asks for a revision or withdrawal. It explains that only this proposal is under review and states the response contract for this turn. This must remain clear even though the initial prompt requested a complete correction set. Later turns refer explicitly to the current proposal version rather than relying on an ambiguous "last message" reference.

Use a structured response that distinguishes revision from withdrawal and carries the requested proposal reference. A revision contains the correction data needed by the existing validators. Empty replacement text remains a possible span deletion; it must not double as a withdrawal signal. The response schema owns shape validation; Audita owns correspondence and domain acceptance. Exact field names can be selected with the schema implementation.

Validator-Owned Feedback

Validators contribute their own actionable findings: validator identity, stable reason code, an explanation, and relevant evidence or constraints where available. Build on existing decision data rather than creating a parallel reason registry. For example, a protection finding can identify the spelling affected, while an editorial finding can explain the meaning change it detected.

The module repair template owns presentation of the collected findings. Validators do not construct chat histories or own the correction response schema. Order feedback by the applicable validator chain for understandable reports and prompts. Model-backed validators should contribute their assessment through the existing structured decision path.

Collect failures from all applicable checks that can meaningfully evaluate the candidate. A malformed target may prevent contextual checks; a provider failure means the assessment is unavailable, not that the edit violated a domain rule. OD-4 settles the eligibility and continuation policy. Making validator chains operator-configurable is not required by this feature.

Ownership and Library Integration

Audita owns proposal lineage, semantic feedback, branch state, revision budgets, revalidation, and transcript application. PromptKit continues to own rendering, profile/backend selection, transport retry, structural repair, and per-backend admission. Configuration continues through ConfigKit's native resolution and the existing root/stage scope conventions. Standard CommandKit inspection and request commands expose the configured prompt assets through their normal APIs.

PromptKit already supports appending rendered assistant and corrective messages. Use its public preparation and execution APIs to render the module repair definition and assemble the intended conversation. Appended messages are already rendered; appending template text alone will not expand its variables. Confirm the full-history path and per-call response-schema selection with a focused integration example before settling adapter details. If that exposes a genuine upstream gap, document a small reproducer and request focused support rather than introducing another renderer or provider client.

Maintain the existing distinctions between recoverable model-operation failures, fatal application/invariant failures, and cancellation. Domain revision attempts are additional logical calls, each potentially containing PromptKit retries and structural repairs; account for these separately. Share existing backend admission and cancellation instead of adding a repair-specific global concurrency limiter. Reporting choices remain in OD-5.

Work From the Baseline

1. Resolve the open policies and prove the request path. Settle the decisions below. Exercise native message composition with an original multi-proposal response, a rendered module repair instruction, a revision response schema, and a subsequent branch turn. Confirm operator prompt replacements and the same behavior with debug disabled.
2. Carry conversation state through the existing boundaries. Extend framework/structuredoutput, the LLM adapter, proposal-generation results, and module/framework contracts as needed to retain and replay the original request and full response. Preserve source-reference mapping and metadata. Use compact lineage data and shared immutable initial context where practical; no particular storage layout or new package hierarchy is prescribed.
3. Add module repair assets and the response schema. Extend the prompt registry, embedded definitions, shared fragments where useful, and framework/responseschema. Provide revision/withdrawal decoding and correspondence checks without changing ordinary span-deletion semantics.
4. Adapt feedback collection. Update validator findings where their current messages are insufficient for repair. Change runner filtering to collect the agreed applicable findings, distinguishing failures, unavailable decisions, and checks that could not run. Keep complete validation mandatory for approval.
5. Implement the bounded branch loop. Integrate revision calls and full revalidation into the current stage/section flow. Preserve accepted proposals, branch isolation, deterministic application, cancellation, and existing apply-time checks. Unsuccessful branches must not discard unrelated accepted work. This does not require redesigning stage barriers or chunk scheduling.
6. Wire configuration and readiness. Add the selected budget/profile settings with native presence and stage inheritance. Keep semantic attempts distinct from structural llm.repair_attempts. Preflight checks any additional required role/prompt readiness when enabled, without making disabled repair require an unused profile. Let existing inspection expose resolved settings.
7. Extend outcomes and documentation. Implement OD-5 through ordinary reporting, correction ledgers, and optional redacted debug artifacts. Update the canonical configuration, pipeline, module, validator, prompt, runtime, and output documentation when behavior lands. Record significant ownership decisions through the documentation policy's ADR process if warranted; routine API and template choices need no ADR.

These are work dependencies for later implementation planning. They do not mandate separate commits, a general workflow engine, or a new release process.

Acceptance and Completion

Use controlled model responses and real loading/rendering at integration boundaries. Cover the following behavior with proportionate tests:

• One approved and one rejected proposal from the same response: repair only the rejected proposal, then apply the accepted results in deterministic order. Include several proposals against one segment and an overlapping-edit case.
• Full initial messages and full response retained, correct roles and ordering, explicit proposal selection, independent branches, and a second repair turn containing only its own branch's history. Exercise debug enabled and disabled.
• Repair success, explicit withdrawal, budget exhaustion, and configured zero attempts. Distinguish withdrawal from an empty-string deletion proposal.
• Validator-owned findings collected under the chosen continuation policy; inapplicable or unavailable checks do not masquerade as editorial rejection. Revised candidates must pass all applicable checks, including checks that approved their earlier versions.
• Wrong proposal references and unusable repair responses cannot introduce unrelated edits or approval. Preserve ordinary identity reconciliation and structural validation; no generic salvage of malformed JSON is implied.
• Context-size handling, provider failure, structural-repair exhaustion, and cancellation follow the selected policies while preserving existing partial-success and fatal-error boundaries. Account for semantic attempts, structural repairs, and transport retries without conflating them.
• Embedded module repair prompts and operator replacements work through the shared engine. Root and stage settings follow the selected scope and fallback rules. Reports identify attempts and final outcomes without presenting every attempt as a separate applied correction or exposing raw history by default.

Run focused checks during development and assembled application validation at completion under the testing policy. Representative transcript comparisons can help tune prompts and defaults, but this roadmap sets no numerical quality threshold or requirement for paid provider calls. Completion requires the agreed loop, resolved open policies, acceptance evidence, and updated canonical documentation. Successful schema validation alone is not evidence that a revised correction is acceptable.

Open Decisions

OD-1: Semantic Repair Budget and Configuration Scope

Recommendation: start with one additional semantic repair attempt per rejected proposal, with a root default and per-stage override; zero disables the loop. A revision that fails validation can continue when the operator configures more attempts. Use a distinct setting from structural llm.repair_attempts. Retain existing per-call timeouts and parent cancellation, and document that each semantic attempt may include transport retries and structural repairs. This provides useful feedback-driven revision with a simple finite bound. The proposed default and setting names are not yet accepted configuration.

Alternative: default to zero and require explicit enabling per run or stage. This avoids additional model work until operators choose it and supports gradual evaluation, but leaves the new behavior inactive in ordinary use. A shared per-stage call budget is another viable model for bounding aggregate work; it offers tighter run-cost control but introduces competition between proposals and requires a deterministic allocation policy.

OD-2: Full History That Does Not Fit

Recommendation: account for the complete conversation and feedback using the supported request-sizing facilities; if a repair cannot fit the selected limit, end that branch with a reported size failure and keep unrelated accepted work. Preserve existing context-estimation limits rather than promising exact provider token accounting. This honors the agreed full-history design without silently truncating context or changing the execution profile.

Alternative: support an explicitly configured fallback repair profile with a larger context allowance. It can preserve full history and rescue otherwise blocked corrections, but adds profile selection, readiness, and failure-handling work and may change cost and model behavior. Smaller initial chunks are also available through existing configuration, at the expense of reduced proposal context; automatic rechunking is not necessary for the initial feature.

OD-3: Repair Execution Profile

Recommendation: inherit the effective proposal profile by default, with an optional root/per-stage repair-profile override. Resolve fallback after stage settings are combined, following the existing validation-profile pattern. This keeps ordinary configuration small while allowing a different model for a difficult revision. Backend capacity remains shared through PromptKit.

Alternative: always use the proposal profile, with no separate selector. This is simpler and keeps model behavior consistent across the conversation, but prevents targeted use of a more capable or larger-context repair model. Neither choice requires a separate backend or credential registry.

OD-4: Feedback Eligibility and Unavailable Validation

Recommendation: collect all meaningful deterministic failures, then run the applicable semantic validators when the candidate has the usable target and before/after data they need, even if another independent check rejected it. An actionable rejection may enter repair; no-effect proposals can finish without repair because they contribute no edit. Build a small applicability map from the actual checks rather than running validators on unusable inputs or encoding universal dependencies between them.

Keep unavailable validation as a separate operation outcome. If a proposal also has an actionable rejection, its repair may proceed using the known findings, with unavailable assessments identified as such; any revision still needs a full successful validation chain. An unavailable assessment by itself should retain the current unavailable outcome, leaving decision-recovery calls to the deferred feature. This produces useful feedback without treating infrastructure failure as evidence against the proposed wording.

Alternative: collect the same applicable findings but require all applicable validator operations to finish successfully before starting a repair. If any assessment is unavailable, keep the candidate unavailable for this stage. This gives each repair a complete assessment and avoids spending calls on a candidate whose review could not finish, but sacrifices opportunities to address a known rejection while another validator is temporarily unavailable. Both approaches preserve full validation as a condition of eventual approval.

OD-5: Ordinary Reports and Attempt History

Recommendation: retain compact proposal lineage, attempt counts, and final dispositions in ordinary results, with meaningful errors and recoveries in the existing report framework. Distinguish approval, actual application, withdrawal, and exhausted or unavailable repair. Keep full conversations and detailed per-attempt evidence in optional redacted debug artifacts. A rejected proposal or voluntary withdrawal is not automatically an operational error. This makes outcomes explainable without turning ordinary reports into conversation dumps. Choose concrete fields and any output-schema version change during implementation planning, preserving the existing failure and exit-status policy unless an explicitly identified requirement calls for a change.

Alternative: include every revision and validator finding in the ordinary correction ledger. This improves analysis without enabling debug capture, but increases report size, retained transcript content, and public schema surface. It requires clear grouping so attempted revisions are not counted as independent corrections or successful recoveries.

OD-6: Revising the Span or Target Reference

Recommendation: allow the revised proposal to correct its original-text span and segment reference within the original request's section, while retaining its proposal lineage. Apply the ordinary target mapping, replacement policy, and full validation to the new candidate. This permits repair of an incorrect source span or model-facing target reference, both of which can cause validation failures. It adds no new transcript-wide targeting mechanism.

Alternative: keep a resolvable target segment fixed, allowing changes to the span and replacement only within that segment; terminate invalid-target proposals without domain repair. This makes continuity easier to interpret and reduces the chance of revising a different edit, but cannot repair misidentified targets. The restriction would be a deliberate feature choice, not an invariant inferred from the current proposal representation.