From d91bb658c9aa4ca1309ea9ead5480e415e5a1a97 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 00:05:04 +0000 Subject: [PATCH] Plan correction repair after validation --- docs/roadmap/correction-repair.md | 371 ++++++++++++++++++++++++++++++ docs/roadmap/future.md | 25 +- 2 files changed, 386 insertions(+), 10 deletions(-) create mode 100644 docs/roadmap/correction-repair.md diff --git a/docs/roadmap/correction-repair.md b/docs/roadmap/correction-repair.md new file mode 100644 index 0000000..678047b --- /dev/null +++ b/docs/roadmap/correction-repair.md @@ -0,0 +1,371 @@ +# Correction Repair After Validation + +**Updated:** 2026-11-10 + +**Status:** Feature direction agreed; implementation pending. Open decisions +below require resolution before the dependent implementation work. This document +owns the target and remaining work, not a release date or implementation status +for the completed library adoption. + +## Objective + +Give a rejected correction an opportunity to improve using specific validator +feedback, while preserving proposals that have already passed validation. Each +correction module supplies a repair prompt. Each validator supplies findings +within its own remit. Audita coordinates bounded revision conversations and +acceptance through the existing PromptKit runtime. + +The unit of repair is an individual correction proposal. Multiple proposals +against one transcript segment remain separately identifiable. A model may revise +the selected proposal or explicitly withdraw it. Every revised proposal must pass +the full applicable validation chain before it becomes eligible for application. + +This adds domain revision after rejection. It is distinct from PromptKit's +structural output repair and from obtaining an unavailable validator decision. +The latter remains separate [deferred work](future.md#domain-aware-response-recovery). + +## Current Baseline + +Reviewed application revision: `8946a82`. Recheck the owning components when +implementation begins, including any subsequent correctness-review fixes. + +- The [pipeline](../internal/pipeline.md) partitions the transcript once and runs + ordered stages. Proposals can be generated concurrently; indexing and + application remain deterministic. Approved corrections are applied in proposal + order, with existing replacement and conflict checks. +- [Proposal generation](../internal/modules.md#proposal-generation-ownership) + returns a correction set, potentially with several proposals per segment. + Model-facing segment references are request-local and are mapped to transcript + identities. The four modules are glossary, homophones, spoken-word cleanup, + and grammar. +- The [validator runtime](../internal/validators.md) runs deterministic checks + before LLM-backed checks and filters rejected candidates before later + validators. Decisions already carry proposal identity, disposition, reason code, + and message. Unavailable decisions are distinguished from editorial rejection. + Validator chains are currently selected by module, not individually configured + by operators. +- The [LLM adapter](../internal/llm-runtime.md) uses PromptKit v0.21.0 prepared + execution with ConfigKit v1.2.0. It captures prepared-request data and final + response text. Proposal-generation results carry candidates, operation outcomes, + and artifact references, but do not retain a conversation for domain revision. +- [Embedded prompts](../internal/prompts.md) live in the `audita` collection and + support operator replacement through native underlays. Audita supplies response + schemas separately. Full interaction artifacts are optional debug output. +- Existing `llm.repair_attempts` controls structural repairs. It is not a budget + for the proposed domain revision loop. No such loop is implemented. + +## Targeted End State + +### Proposal Lifecycle + +1. Generate the initial correction set using the stage's instruction, transcript + section, relevant glossary/protection data, and response schema. +2. Validate candidates, collecting findings from checks that can meaningfully + evaluate them. Checks that cannot run do not become invented rejection reasons. +3. Retain approved proposals for application. Give each rejected proposal a repair + branch containing its conversation and findings. Budget and eligibility rules + determine whether another request can run. +4. Request revision or withdrawal of that one proposal. An accepted withdrawal + ends the branch without an edit. A revised proposal returns to the complete + applicable validator chain; approvals of a previous version do not transfer. +5. Continue until approval, withdrawal, budget exhaustion, or a classified failure. + Preserve independently accepted work under the existing partial-success rules. +6. Apply accepted corrections in deterministic proposal order, retaining the + existing apply-time checks for stale spans, protection, and conflicts. + +Withdrawal or exhaustion contributes no edit from that proposal. Other accepted +proposals against the same segment may still change its text. Retaining an +approval does not guarantee application when another edit makes it inapplicable. +This feature does not add a second repair loop for application-time conflicts. + +Each revision remains associated with the original proposal and its processing +stage and section. Repair responses must identify the requested proposal; unrelated +edits do not become additional candidates. OD-6 addresses whether revisions can +change the proposed span or target reference within the original section context; +the ordinary mapping and validators still determine applicability. +Exact internal types and model-facing reference spelling are implementation +choices, not new globally unique identity requirements. + +### Full Conversation and Independent Branches + +The first repair request contains the complete initial rendered request messages, +with their original roles and order, followed by the complete assistant response +that supplied the decoded proposal set, then the targeted repair instruction. +Retain the original section and request inputs for this purpose; do not rebuild +the conversation from a transcript already changed by other corrections. + +Every proposal branches independently from that same initial exchange. Later +attempts append that proposal's previous repair response and new feedback to its +own branch. Repairs of other proposals never become turns in that conversation. +Keep approved proposals outside the set being reconsidered. Branch scheduling +must not change proposal attribution or final application order. + +Here, conversation history means the application-level message exchange. Preserve +the full response used for candidate decoding rather than serializing only the +failed proposal as if it were the model's original answer. Provider transport +attempts and PromptKit's internal structural-repair operations remain accounted +for separately. Do not manufacture hidden provider messages or reasoning. + +Keep required conversation state in memory when repair is enabled, independently +of debug artifact saving. Its lifetime need only cover the associated work; this +does not introduce durable conversation storage or restart/resume support. Count +the complete replay, feedback, and applicable output contract when assessing +request size. Context-limit handling is OD-2 below. + +### Module Repair Prompts and Response Contract + +Each built-in correction module owns a corresponding repair definition in the +`audita` namespace, alongside its proposal definition. Use the existing embedded +filesystem and ConfigKit underlay mechanism so operators can replace repair +prompts without rebuilding Audita. Shared instruction fragments may be reused; +module-specific instructions stay with the module's prompt. + +The repair instruction identifies the proposal, repeats its original and proposed +replacement text, presents the findings, and asks for a revision or withdrawal. +It explains that only this proposal is under review and states the response +contract for this turn. This must remain clear even though the initial prompt +requested a complete correction set. Later turns refer explicitly to the current +proposal version rather than relying on an ambiguous "last message" reference. + +Use a structured response that distinguishes revision from withdrawal and carries +the requested proposal reference. A revision contains the correction data needed +by the existing validators. Empty replacement text remains a possible span +deletion; it must not double as a withdrawal signal. The response schema owns +shape validation; Audita owns correspondence and domain acceptance. Exact field +names can be selected with the schema implementation. + +### Validator-Owned Feedback + +Validators contribute their own actionable findings: validator identity, stable +reason code, an explanation, and relevant evidence or constraints where available. +Build on existing decision data rather than creating a parallel reason registry. +For example, a protection finding can identify the spelling affected, while an +editorial finding can explain the meaning change it detected. + +The module repair template owns presentation of the collected findings. +Validators do not construct chat histories or own the correction response schema. +Order feedback by the applicable validator chain for understandable reports and +prompts. Model-backed validators should contribute their assessment through the +existing structured decision path. + +Collect failures from all applicable checks that can meaningfully evaluate the +candidate. A malformed target may prevent contextual checks; a provider failure +means the assessment is unavailable, not that the edit violated a domain rule. +OD-4 settles the eligibility and continuation policy. Making validator chains +operator-configurable is not required by this feature. + +### Ownership and Library Integration + +Audita owns proposal lineage, semantic feedback, branch state, revision budgets, +revalidation, and transcript application. PromptKit continues to own rendering, +profile/backend selection, transport retry, structural repair, and per-backend +admission. Configuration continues through ConfigKit's native resolution and +the existing root/stage scope conventions. Standard CommandKit inspection and +request commands expose the configured prompt assets through their normal APIs. + +PromptKit already supports +[appending rendered assistant and corrective messages](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/docs/consumers/pkg-promptkit.md#append-already-rendered-messages). +Use its public preparation and execution APIs to render the module repair +definition and assemble the intended conversation. Appended messages are already +rendered; appending template text alone will not expand its variables. Confirm +the full-history path and per-call response-schema selection with a focused +integration example before settling adapter details. If that exposes a genuine +upstream gap, document a small reproducer and request focused support rather than +introducing another renderer or provider client. + +Maintain the existing distinctions between recoverable model-operation failures, +fatal application/invariant failures, and cancellation. Domain revision attempts +are additional logical calls, each potentially containing PromptKit retries and +structural repairs; account for these separately. Share existing backend admission +and cancellation instead of adding a repair-specific global concurrency limiter. +Reporting choices remain in OD-5. + +## Work From the Baseline + +1. **Resolve the open policies and prove the request path.** Settle the decisions + below. Exercise native message composition with an original multi-proposal + response, a rendered module repair instruction, a revision response schema, + and a subsequent branch turn. Confirm operator prompt replacements and the + same behavior with debug disabled. +2. **Carry conversation state through the existing boundaries.** Extend + `framework/structuredoutput`, the LLM adapter, proposal-generation results, + and module/framework contracts as needed to retain and replay the original + request and full response. Preserve source-reference mapping and metadata. + Use compact lineage data and shared immutable initial context where practical; + no particular storage layout or new package hierarchy is prescribed. +3. **Add module repair assets and the response schema.** Extend the prompt + registry, embedded definitions, shared fragments where useful, and + `framework/responseschema`. Provide revision/withdrawal decoding and + correspondence checks without changing ordinary span-deletion semantics. +4. **Adapt feedback collection.** Update validator findings where their current + messages are insufficient for repair. Change runner filtering to collect the + agreed applicable findings, distinguishing failures, unavailable decisions, + and checks that could not run. Keep complete validation mandatory for approval. +5. **Implement the bounded branch loop.** Integrate revision calls and full + revalidation into the current stage/section flow. Preserve accepted proposals, + branch isolation, deterministic application, cancellation, and existing + apply-time checks. Unsuccessful branches must not discard unrelated accepted + work. This does not require redesigning stage barriers or chunk scheduling. +6. **Wire configuration and readiness.** Add the selected budget/profile settings + with native presence and stage inheritance. Keep semantic attempts distinct + from structural `llm.repair_attempts`. Preflight checks any additional required + role/prompt readiness when enabled, without making disabled repair require + an unused profile. Let existing inspection expose resolved settings. +7. **Extend outcomes and documentation.** Implement OD-5 through ordinary + reporting, correction ledgers, and optional redacted debug artifacts. Update + the canonical [configuration](../config.md), [pipeline](../internal/pipeline.md), + [module](../internal/modules.md), [validator](../internal/validators.md), + [prompt](../internal/prompts.md), [runtime](../internal/llm-runtime.md), and + [output](../integrations/output-files.md) documentation when behavior lands. + Record significant ownership decisions through the documentation policy's + ADR process if warranted; routine API and template choices need no ADR. + +These are work dependencies for later implementation planning. They do not +mandate separate commits, a general workflow engine, or a new release process. + +## Acceptance and Completion + +Use controlled model responses and real loading/rendering at integration +boundaries. Cover the following behavior with proportionate tests: + +- One approved and one rejected proposal from the same response: repair only + the rejected proposal, then apply the accepted results in deterministic order. + Include several proposals against one segment and an overlapping-edit case. +- Full initial messages and full response retained, correct roles and ordering, + explicit proposal selection, independent branches, and a second repair turn + containing only its own branch's history. Exercise debug enabled and disabled. +- Repair success, explicit withdrawal, budget exhaustion, and configured zero + attempts. Distinguish withdrawal from an empty-string deletion proposal. +- Validator-owned findings collected under the chosen continuation policy; + inapplicable or unavailable checks do not masquerade as editorial rejection. + Revised candidates must pass all applicable checks, including checks that + approved their earlier versions. +- Wrong proposal references and unusable repair responses cannot introduce + unrelated edits or approval. Preserve ordinary identity reconciliation and + structural validation; no generic salvage of malformed JSON is implied. +- Context-size handling, provider failure, structural-repair exhaustion, and + cancellation follow the selected policies while preserving existing + partial-success and fatal-error boundaries. Account for semantic attempts, + structural repairs, and transport retries without conflating them. +- Embedded module repair prompts and operator replacements work through the + shared engine. Root and stage settings follow the selected scope and fallback + rules. Reports identify attempts and final outcomes without presenting every + attempt as a separate applied correction or exposing raw history by default. + +Run focused checks during development and assembled application validation at +completion under the [testing policy](../policy/testing.md). Representative +transcript comparisons can help tune prompts and defaults, but this roadmap +sets no numerical quality threshold or requirement for paid provider calls. +Completion requires the agreed loop, resolved open policies, acceptance evidence, +and updated canonical documentation. Successful schema validation alone is not +evidence that a revised correction is acceptable. + +## Open Decisions + +### OD-1: Semantic Repair Budget and Configuration Scope + +**Recommendation:** start with one additional semantic repair attempt per +rejected proposal, with a root default and per-stage override; zero disables the +loop. A revision that fails validation can continue when the operator configures +more attempts. Use a distinct setting from structural `llm.repair_attempts`. +Retain existing per-call timeouts and parent cancellation, and document that +each semantic attempt may include transport retries and structural repairs. +This provides useful feedback-driven revision with a simple finite bound. +The proposed default and setting names are not yet accepted configuration. + +**Alternative:** default to zero and require explicit enabling per run or stage. +This avoids additional model work until operators choose it and supports gradual +evaluation, but leaves the new behavior inactive in ordinary use. A shared +per-stage call budget is another viable model for bounding aggregate work; it +offers tighter run-cost control but introduces competition between proposals +and requires a deterministic allocation policy. + +### OD-2: Full History That Does Not Fit + +**Recommendation:** account for the complete conversation and feedback using +the supported request-sizing facilities; if a repair cannot fit the selected +limit, end that branch with a reported size failure and keep unrelated accepted +work. Preserve existing context-estimation limits rather than promising exact +provider token accounting. This honors the agreed full-history design without +silently truncating context or changing the execution profile. + +**Alternative:** support an explicitly configured fallback repair profile with +a larger context allowance. It can preserve full history and rescue otherwise +blocked corrections, but adds profile selection, readiness, and failure-handling +work and may change cost and model behavior. Smaller initial chunks are also +available through existing configuration, at the expense of reduced proposal +context; automatic rechunking is not necessary for the initial feature. + +### OD-3: Repair Execution Profile + +**Recommendation:** inherit the effective proposal profile by default, with an +optional root/per-stage repair-profile override. Resolve fallback after stage +settings are combined, following the existing validation-profile pattern. This +keeps ordinary configuration small while allowing a different model for a +difficult revision. Backend capacity remains shared through PromptKit. + +**Alternative:** always use the proposal profile, with no separate selector. +This is simpler and keeps model behavior consistent across the conversation, +but prevents targeted use of a more capable or larger-context repair model. +Neither choice requires a separate backend or credential registry. + +### OD-4: Feedback Eligibility and Unavailable Validation + +**Recommendation:** collect all meaningful deterministic failures, then run the +applicable semantic validators when the candidate has the usable target and +before/after data they need, even if another independent check rejected it. +An actionable rejection may enter repair; no-effect proposals can finish without +repair because they contribute no edit. Build a small applicability map from +the actual checks rather than running validators on unusable inputs or encoding +universal dependencies between them. + +Keep unavailable validation as a separate operation outcome. If a proposal also +has an actionable rejection, its repair may proceed using the known findings, +with unavailable assessments identified as such; any revision still needs a full +successful validation chain. An unavailable assessment by itself should retain +the current unavailable outcome, leaving decision-recovery calls to the deferred +feature. This produces useful feedback without treating infrastructure failure +as evidence against the proposed wording. + +**Alternative:** collect the same applicable findings but require all applicable +validator operations to finish successfully before starting a repair. If any +assessment is unavailable, keep the candidate unavailable for this stage. This +gives each repair a complete assessment and avoids spending calls on a candidate +whose review could not finish, but sacrifices opportunities to address a known +rejection while another validator is temporarily unavailable. Both approaches +preserve full validation as a condition of eventual approval. + +### OD-5: Ordinary Reports and Attempt History + +**Recommendation:** retain compact proposal lineage, attempt counts, and final +dispositions in ordinary results, with meaningful errors and recoveries in the +existing report framework. Distinguish approval, actual application, withdrawal, +and exhausted or unavailable repair. Keep full conversations and detailed +per-attempt evidence in optional redacted debug artifacts. A rejected proposal +or voluntary withdrawal is not automatically an operational error. This makes +outcomes explainable without turning ordinary reports into conversation dumps. +Choose concrete fields and any output-schema version change during implementation +planning, preserving the existing failure and exit-status policy unless an +explicitly identified requirement calls for a change. + +**Alternative:** include every revision and validator finding in the ordinary +correction ledger. This improves analysis without enabling debug capture, but +increases report size, retained transcript content, and public schema surface. +It requires clear grouping so attempted revisions are not counted as independent +corrections or successful recoveries. + +### OD-6: Revising the Span or Target Reference + +**Recommendation:** allow the revised proposal to correct its original-text span +and segment reference within the original request's section, while retaining its +proposal lineage. Apply the ordinary target mapping, replacement policy, and full +validation to the new candidate. This permits repair of an incorrect source span +or model-facing target reference, both of which can cause validation failures. +It adds no new transcript-wide targeting mechanism. + +**Alternative:** keep a resolvable target segment fixed, allowing changes to the +span and replacement only within that segment; terminate invalid-target proposals +without domain repair. This makes continuity easier to interpret and reduces +the chance of revising a different edit, but cannot repair misidentified targets. +The restriction would be a deliberate feature choice, not an invariant inferred +from the current proposal representation. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 1b0fcf7..14b21dc 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -1,6 +1,6 @@ # Deferred Feature Work -**Last updated:** 2026-10-11 +**Last updated:** 2026-11-10 **Status:** Deferred or exploratory work; not implemented or scheduled. These entries preserve work beyond completed feature and library upgrades. @@ -10,10 +10,11 @@ implementing every capability. ## Domain-Aware Response Recovery The implemented [partial-success decision reconciliation](../internal/validators.md#llm-validator-batching) -preserves usable decisions without another model call. Guided recovery remains -deferred pending evidence from failures that remain. General per-entry -salvage from structurally invalid responses is also a separate design question; -the accepted reconciliation feature retains whole-response structural validation. +preserves usable decisions without another model call. Revision of rejected +correction proposals is now an agreed feature direction in the +[correction-repair roadmap](correction-repair.md), with its own open decisions. +That work includes substantive validator feedback and explicit proposal withdrawal. +It is distinct from recovering an unavailable validator decision. Consider bounded feedback and regeneration for model responses that satisfy the structural schema but leave requested decisions unavailable after reconciliation. @@ -30,11 +31,15 @@ secrets, and how regeneration fits bounded budgets and operation deadlines. Reus shared library capabilities where appropriate; do not introduce an uncoordinated outer retry loop that multiplies existing retry/repair budgets. -Ordinary editorial rejection is a valid decision, not a reason to regenerate until -an edit is approved. Recovery must preserve required validation, candidate identity, -partial-success safety, and truthful error/recovery reporting. Validate with -controlled responses that distinguish recoverable protocol defects from valid -negative decisions. Exact APIs, budgets, and sequencing remain undecided. +Editorial rejection remains a valid decision about the evaluated proposal. The +correction-repair roadmap owns any subsequent revision of that proposal and full +revalidation; this deferred entry concerns unavailable decisions. Decision recovery +must preserve required validation, candidate identity, partial-success safety, +and truthful error/recovery reporting. Validate with controlled responses that +distinguish protocol defects from valid negative decisions. Exact APIs, budgets, +and sequencing remain undecided. General per-entry salvage from structurally +invalid responses also remains a separate design question; current reconciliation +retains whole-response structural validation. ## Independent Whole-Pipeline Traversal by Chunk