diff --git a/docs/roadmap/correction-repair.md b/docs/roadmap/correction-repair.md index 678047b..b21fda6 100644 --- a/docs/roadmap/correction-repair.md +++ b/docs/roadmap/correction-repair.md @@ -1,9 +1,10 @@ # Correction Repair After Validation -**Updated:** 2026-11-10 +**Updated:** 2026-11-11 -**Status:** Feature direction agreed; implementation pending. Open decisions -below require resolution before the dependent implementation work. This document +**Status:** Feature direction and all six decisions accepted; implementation +pending. No open feature decisions remain. Configuration names, concrete schema +fields, and adapter details remain implementation-planning work. This document owns the target and remaining work, not a release date or implementation status for the completed library adoption. @@ -80,9 +81,9 @@ This feature does not add a second repair loop for application-time conflicts. Each revision remains associated with the original proposal and its processing stage and section. Repair responses must identify the requested proposal; unrelated -edits do not become additional candidates. OD-6 addresses whether revisions can -change the proposed span or target reference within the original section context; -the ordinary mapping and validators still determine applicability. +edits do not become additional candidates. Revisions may correct the proposed +span or target reference within the original section context, as specified in +[the accepted target decision](#revising-the-span-or-target-reference). Exact internal types and model-facing reference spelling are implementation choices, not new globally unique identity requirements. @@ -110,7 +111,8 @@ Keep required conversation state in memory when repair is enabled, independently of debug artifact saving. Its lifetime need only cover the associated work; this does not introduce durable conversation storage or restart/resume support. Count the complete replay, feedback, and applicable output contract when assessing -request size. Context-limit handling is OD-2 below. +request size. Apply the accepted +[context-limit behavior](#full-history-that-does-not-fit). ### Module Repair Prompts and Response Contract @@ -151,8 +153,9 @@ existing structured decision path. Collect failures from all applicable checks that can meaningfully evaluate the candidate. A malformed target may prevent contextual checks; a provider failure means the assessment is unavailable, not that the edit violated a domain rule. -OD-4 settles the eligibility and continuation policy. Making validator chains -operator-configurable is not required by this feature. +Apply the accepted +[eligibility and continuation policy](#feedback-eligibility-and-unavailable-validation). +Making validator chains operator-configurable is not required by this feature. ### Ownership and Library Integration @@ -178,15 +181,14 @@ fatal application/invariant failures, and cancellation. Domain revision attempts are additional logical calls, each potentially containing PromptKit retries and structural repairs; account for these separately. Share existing backend admission and cancellation instead of adding a repair-specific global concurrency limiter. -Reporting choices remain in OD-5. +Use the accepted [reporting scope](#ordinary-reports-and-attempt-history). ## Work From the Baseline -1. **Resolve the open policies and prove the request path.** Settle the decisions - below. Exercise native message composition with an original multi-proposal - response, a rendered module repair instruction, a revision response schema, - and a subsequent branch turn. Confirm operator prompt replacements and the - same behavior with debug disabled. +1. **Prove the request path.** Exercise native message composition with an + original multi-proposal response, a rendered module repair instruction, a + revision response schema, and a subsequent branch turn. Confirm operator + prompt replacements and the same behavior with debug disabled. 2. **Carry conversation state through the existing boundaries.** Extend `framework/structuredoutput`, the LLM adapter, proposal-generation results, and module/framework contracts as needed to retain and replay the original @@ -206,13 +208,16 @@ Reporting choices remain in OD-5. branch isolation, deterministic application, cancellation, and existing apply-time checks. Unsuccessful branches must not discard unrelated accepted work. This does not require redesigning stage barriers or chunk scheduling. -6. **Wire configuration and readiness.** Add the selected budget/profile settings - with native presence and stage inheritance. Keep semantic attempts distinct - from structural `llm.repair_attempts`. Preflight checks any additional required - role/prompt readiness when enabled, without making disabled repair require - an unused profile. Let existing inspection expose resolved settings. -7. **Extend outcomes and documentation.** Implement OD-5 through ordinary - reporting, correction ledgers, and optional redacted debug artifacts. Update +6. **Wire configuration and readiness.** Add the accepted + [budget](#semantic-repair-budget-and-configuration-scope) + and [profile](#repair-execution-profile) settings with native presence and stage + inheritance. Keep semantic attempts distinct from structural + `llm.repair_attempts`. Preflight checks any additional required role/prompt + readiness when enabled, without making disabled repair require an unused + profile. Let existing inspection expose resolved settings. +7. **Extend outcomes and documentation.** Implement the accepted reporting scope + through ordinary reporting, correction ledgers, and optional redacted debug + artifacts. Update the canonical [configuration](../config.md), [pipeline](../internal/pipeline.md), [module](../internal/modules.md), [validator](../internal/validators.md), [prompt](../internal/prompts.md), [runtime](../internal/llm-runtime.md), and @@ -234,85 +239,78 @@ boundaries. Cover the following behavior with proportionate tests: - Full initial messages and full response retained, correct roles and ordering, explicit proposal selection, independent branches, and a second repair turn containing only its own branch's history. Exercise debug enabled and disabled. -- Repair success, explicit withdrawal, budget exhaustion, and configured zero - attempts. Distinguish withdrawal from an empty-string deletion proposal. -- Validator-owned findings collected under the chosen continuation policy; +- Repair success, explicit withdrawal, budget exhaustion, the default of one + semantic attempt, additional configured attempts, and global or stage-level + zero disabling repair. Distinguish withdrawal from an empty-string deletion + proposal. +- Validator-owned findings collected under the accepted continuation policy; inapplicable or unavailable checks do not masquerade as editorial rejection. + Cover known rejection alongside an unavailable assessment, unavailable + assessment alone, and a no-effect proposal finishing without repair. Revised candidates must pass all applicable checks, including checks that approved their earlier versions. - Wrong proposal references and unusable repair responses cannot introduce unrelated edits or approval. Preserve ordinary identity reconciliation and structural validation; no generic salvage of malformed JSON is implied. +- A revision can correct its source span or segment reference within the original + request's section, preserving proposal lineage and passing ordinary target + mapping and full validation. - Context-size handling, provider failure, structural-repair exhaustion, and - cancellation follow the selected policies while preserving existing + cancellation follow the accepted policies while preserving existing partial-success and fatal-error boundaries. Account for semantic attempts, structural repairs, and transport retries without conflating them. + An oversized conversation ends only the affected branch with a reported size + failure, preserving unrelated accepted work. - Embedded module repair prompts and operator replacements work through the - shared engine. Root and stage settings follow the selected scope and fallback - rules. Reports identify attempts and final outcomes without presenting every - attempt as a separate applied correction or exposing raw history by default. + shared engine. Root and stage settings follow the accepted scope and fallback + rules, including inheritance of the effective proposal profile after stage + settings are combined and explicit repair-profile overrides. Reports identify + attempts and final outcomes without presenting every attempt as a separate + applied correction or exposing raw history by default. Run focused checks during development and assembled application validation at completion under the [testing policy](../policy/testing.md). Representative transcript comparisons can help tune prompts and defaults, but this roadmap sets no numerical quality threshold or requirement for paid provider calls. -Completion requires the agreed loop, resolved open policies, acceptance evidence, +Completion requires the agreed loop and accepted policies, acceptance evidence, and updated canonical documentation. Successful schema validation alone is not evidence that a revised correction is acceptable. -## Open Decisions +## Accepted Decisions -### OD-1: Semantic Repair Budget and Configuration Scope +### Semantic Repair Budget and Configuration Scope -**Recommendation:** start with one additional semantic repair attempt per -rejected proposal, with a root default and per-stage override; zero disables the -loop. A revision that fails validation can continue when the operator configures +Allow one additional semantic repair attempt per rejected proposal by default, +configurable at the root and per stage; zero disables the loop. A revision that +fails validation can continue when the operator configures more attempts. Use a distinct setting from structural `llm.repair_attempts`. Retain existing per-call timeouts and parent cancellation, and document that each semantic attempt may include transport retries and structural repairs. This provides useful feedback-driven revision with a simple finite bound. -The proposed default and setting names are not yet accepted configuration. +The default and scope are accepted; exact configuration names remain an +implementation-planning choice. -**Alternative:** default to zero and require explicit enabling per run or stage. -This avoids additional model work until operators choose it and supports gradual -evaluation, but leaves the new behavior inactive in ordinary use. A shared -per-stage call budget is another viable model for bounding aggregate work; it -offers tighter run-cost control but introduces competition between proposals -and requires a deterministic allocation policy. +### Full History That Does Not Fit -### OD-2: Full History That Does Not Fit - -**Recommendation:** account for the complete conversation and feedback using -the supported request-sizing facilities; if a repair cannot fit the selected +Account for the complete conversation and feedback using the supported +request-sizing facilities; if a repair cannot fit the selected limit, end that branch with a reported size failure and keep unrelated accepted work. Preserve existing context-estimation limits rather than promising exact provider token accounting. This honors the agreed full-history design without silently truncating context or changing the execution profile. -**Alternative:** support an explicitly configured fallback repair profile with -a larger context allowance. It can preserve full history and rescue otherwise -blocked corrections, but adds profile selection, readiness, and failure-handling -work and may change cost and model behavior. Smaller initial chunks are also -available through existing configuration, at the expense of reduced proposal -context; automatic rechunking is not necessary for the initial feature. +### Repair Execution Profile -### OD-3: Repair Execution Profile - -**Recommendation:** inherit the effective proposal profile by default, with an -optional root/per-stage repair-profile override. Resolve fallback after stage +Inherit the effective proposal profile by default, with an optional root/per-stage +repair-profile override. Resolve fallback after stage settings are combined, following the existing validation-profile pattern. This keeps ordinary configuration small while allowing a different model for a difficult revision. Backend capacity remains shared through PromptKit. -**Alternative:** always use the proposal profile, with no separate selector. -This is simpler and keeps model behavior consistent across the conversation, -but prevents targeted use of a more capable or larger-context repair model. -Neither choice requires a separate backend or credential registry. +### Feedback Eligibility and Unavailable Validation -### OD-4: Feedback Eligibility and Unavailable Validation - -**Recommendation:** collect all meaningful deterministic failures, then run the -applicable semantic validators when the candidate has the usable target and +Collect all meaningful deterministic failures, then run the applicable semantic +validators when the candidate has the usable target and before/after data they need, even if another independent check rejected it. An actionable rejection may enter repair; no-effect proposals can finish without repair because they contribute no edit. Build a small applicability map from @@ -327,18 +325,10 @@ the current unavailable outcome, leaving decision-recovery calls to the deferred feature. This produces useful feedback without treating infrastructure failure as evidence against the proposed wording. -**Alternative:** collect the same applicable findings but require all applicable -validator operations to finish successfully before starting a repair. If any -assessment is unavailable, keep the candidate unavailable for this stage. This -gives each repair a complete assessment and avoids spending calls on a candidate -whose review could not finish, but sacrifices opportunities to address a known -rejection while another validator is temporarily unavailable. Both approaches -preserve full validation as a condition of eventual approval. +### Ordinary Reports and Attempt History -### OD-5: Ordinary Reports and Attempt History - -**Recommendation:** retain compact proposal lineage, attempt counts, and final -dispositions in ordinary results, with meaningful errors and recoveries in the +Retain compact proposal lineage, attempt counts, and final dispositions in +ordinary results, with meaningful errors and recoveries in the existing report framework. Distinguish approval, actual application, withdrawal, and exhausted or unavailable repair. Keep full conversations and detailed per-attempt evidence in optional redacted debug artifacts. A rejected proposal @@ -348,24 +338,11 @@ Choose concrete fields and any output-schema version change during implementatio planning, preserving the existing failure and exit-status policy unless an explicitly identified requirement calls for a change. -**Alternative:** include every revision and validator finding in the ordinary -correction ledger. This improves analysis without enabling debug capture, but -increases report size, retained transcript content, and public schema surface. -It requires clear grouping so attempted revisions are not counted as independent -corrections or successful recoveries. +### Revising the Span or Target Reference -### OD-6: Revising the Span or Target Reference - -**Recommendation:** allow the revised proposal to correct its original-text span -and segment reference within the original request's section, while retaining its +Allow the revised proposal to correct its original-text span and segment +reference within the original request's section, while retaining its proposal lineage. Apply the ordinary target mapping, replacement policy, and full validation to the new candidate. This permits repair of an incorrect source span or model-facing target reference, both of which can cause validation failures. It adds no new transcript-wide targeting mechanism. - -**Alternative:** keep a resolvable target segment fixed, allowing changes to the -span and replacement only within that segment; terminate invalid-target proposals -without domain repair. This makes continuity easier to interpret and reduces -the chance of revising a different edit, but cannot repair misidentified targets. -The restriction would be a deliberate feature choice, not an invariant inferred -from the current proposal representation. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 14b21dc..63278a4 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -1,6 +1,6 @@ # Deferred Feature Work -**Last updated:** 2026-11-10 +**Last updated:** 2026-11-11 **Status:** Deferred or exploratory work; not implemented or scheduled. These entries preserve work beyond completed feature and library upgrades. @@ -12,7 +12,7 @@ implementing every capability. The implemented [partial-success decision reconciliation](../internal/validators.md#llm-validator-batching) preserves usable decisions without another model call. Revision of rejected correction proposals is now an agreed feature direction in the -[correction-repair roadmap](correction-repair.md), with its own open decisions. +[correction-repair roadmap](correction-repair.md), with its feature decisions accepted. That work includes substantive validator feedback and explicit proposal withdrawal. It is distinct from recovering an unavailable validator decision.