Record accepted correction repair decisions

This commit is contained in:
2026-11-11 03:36:26 +00:00
parent d91bb658c9
commit 4f525e07cc
2 changed files with 70 additions and 93 deletions

View File

@@ -1,9 +1,10 @@
# Correction Repair After Validation
**Updated:** 2026-11-10
**Updated:** 2026-11-11
**Status:** Feature direction agreed; implementation pending. Open decisions
below require resolution before the dependent implementation work. This document
**Status:** Feature direction and all six decisions accepted; implementation
pending. No open feature decisions remain. Configuration names, concrete schema
fields, and adapter details remain implementation-planning work. This document
owns the target and remaining work, not a release date or implementation status
for the completed library adoption.
@@ -80,9 +81,9 @@ This feature does not add a second repair loop for application-time conflicts.
Each revision remains associated with the original proposal and its processing
stage and section. Repair responses must identify the requested proposal; unrelated
edits do not become additional candidates. OD-6 addresses whether revisions can
change the proposed span or target reference within the original section context;
the ordinary mapping and validators still determine applicability.
edits do not become additional candidates. Revisions may correct the proposed
span or target reference within the original section context, as specified in
[the accepted target decision](#revising-the-span-or-target-reference).
Exact internal types and model-facing reference spelling are implementation
choices, not new globally unique identity requirements.
@@ -110,7 +111,8 @@ Keep required conversation state in memory when repair is enabled, independently
of debug artifact saving. Its lifetime need only cover the associated work; this
does not introduce durable conversation storage or restart/resume support. Count
the complete replay, feedback, and applicable output contract when assessing
request size. Context-limit handling is OD-2 below.
request size. Apply the accepted
[context-limit behavior](#full-history-that-does-not-fit).
### Module Repair Prompts and Response Contract
@@ -151,8 +153,9 @@ existing structured decision path.
Collect failures from all applicable checks that can meaningfully evaluate the
candidate. A malformed target may prevent contextual checks; a provider failure
means the assessment is unavailable, not that the edit violated a domain rule.
OD-4 settles the eligibility and continuation policy. Making validator chains
operator-configurable is not required by this feature.
Apply the accepted
[eligibility and continuation policy](#feedback-eligibility-and-unavailable-validation).
Making validator chains operator-configurable is not required by this feature.
### Ownership and Library Integration
@@ -178,15 +181,14 @@ fatal application/invariant failures, and cancellation. Domain revision attempts
are additional logical calls, each potentially containing PromptKit retries and
structural repairs; account for these separately. Share existing backend admission
and cancellation instead of adding a repair-specific global concurrency limiter.
Reporting choices remain in OD-5.
Use the accepted [reporting scope](#ordinary-reports-and-attempt-history).
## Work From the Baseline
1. **Resolve the open policies and prove the request path.** Settle the decisions
below. Exercise native message composition with an original multi-proposal
response, a rendered module repair instruction, a revision response schema,
and a subsequent branch turn. Confirm operator prompt replacements and the
same behavior with debug disabled.
1. **Prove the request path.** Exercise native message composition with an
original multi-proposal response, a rendered module repair instruction, a
revision response schema, and a subsequent branch turn. Confirm operator
prompt replacements and the same behavior with debug disabled.
2. **Carry conversation state through the existing boundaries.** Extend
`framework/structuredoutput`, the LLM adapter, proposal-generation results,
and module/framework contracts as needed to retain and replay the original
@@ -206,13 +208,16 @@ Reporting choices remain in OD-5.
branch isolation, deterministic application, cancellation, and existing
apply-time checks. Unsuccessful branches must not discard unrelated accepted
work. This does not require redesigning stage barriers or chunk scheduling.
6. **Wire configuration and readiness.** Add the selected budget/profile settings
with native presence and stage inheritance. Keep semantic attempts distinct
from structural `llm.repair_attempts`. Preflight checks any additional required
role/prompt readiness when enabled, without making disabled repair require
an unused profile. Let existing inspection expose resolved settings.
7. **Extend outcomes and documentation.** Implement OD-5 through ordinary
reporting, correction ledgers, and optional redacted debug artifacts. Update
6. **Wire configuration and readiness.** Add the accepted
[budget](#semantic-repair-budget-and-configuration-scope)
and [profile](#repair-execution-profile) settings with native presence and stage
inheritance. Keep semantic attempts distinct from structural
`llm.repair_attempts`. Preflight checks any additional required role/prompt
readiness when enabled, without making disabled repair require an unused
profile. Let existing inspection expose resolved settings.
7. **Extend outcomes and documentation.** Implement the accepted reporting scope
through ordinary reporting, correction ledgers, and optional redacted debug
artifacts. Update
the canonical [configuration](../config.md), [pipeline](../internal/pipeline.md),
[module](../internal/modules.md), [validator](../internal/validators.md),
[prompt](../internal/prompts.md), [runtime](../internal/llm-runtime.md), and
@@ -234,85 +239,78 @@ boundaries. Cover the following behavior with proportionate tests:
- Full initial messages and full response retained, correct roles and ordering,
explicit proposal selection, independent branches, and a second repair turn
containing only its own branch's history. Exercise debug enabled and disabled.
- Repair success, explicit withdrawal, budget exhaustion, and configured zero
attempts. Distinguish withdrawal from an empty-string deletion proposal.
- Validator-owned findings collected under the chosen continuation policy;
- Repair success, explicit withdrawal, budget exhaustion, the default of one
semantic attempt, additional configured attempts, and global or stage-level
zero disabling repair. Distinguish withdrawal from an empty-string deletion
proposal.
- Validator-owned findings collected under the accepted continuation policy;
inapplicable or unavailable checks do not masquerade as editorial rejection.
Cover known rejection alongside an unavailable assessment, unavailable
assessment alone, and a no-effect proposal finishing without repair.
Revised candidates must pass all applicable checks, including checks that
approved their earlier versions.
- Wrong proposal references and unusable repair responses cannot introduce
unrelated edits or approval. Preserve ordinary identity reconciliation and
structural validation; no generic salvage of malformed JSON is implied.
- A revision can correct its source span or segment reference within the original
request's section, preserving proposal lineage and passing ordinary target
mapping and full validation.
- Context-size handling, provider failure, structural-repair exhaustion, and
cancellation follow the selected policies while preserving existing
cancellation follow the accepted policies while preserving existing
partial-success and fatal-error boundaries. Account for semantic attempts,
structural repairs, and transport retries without conflating them.
An oversized conversation ends only the affected branch with a reported size
failure, preserving unrelated accepted work.
- Embedded module repair prompts and operator replacements work through the
shared engine. Root and stage settings follow the selected scope and fallback
rules. Reports identify attempts and final outcomes without presenting every
attempt as a separate applied correction or exposing raw history by default.
shared engine. Root and stage settings follow the accepted scope and fallback
rules, including inheritance of the effective proposal profile after stage
settings are combined and explicit repair-profile overrides. Reports identify
attempts and final outcomes without presenting every attempt as a separate
applied correction or exposing raw history by default.
Run focused checks during development and assembled application validation at
completion under the [testing policy](../policy/testing.md). Representative
transcript comparisons can help tune prompts and defaults, but this roadmap
sets no numerical quality threshold or requirement for paid provider calls.
Completion requires the agreed loop, resolved open policies, acceptance evidence,
Completion requires the agreed loop and accepted policies, acceptance evidence,
and updated canonical documentation. Successful schema validation alone is not
evidence that a revised correction is acceptable.
## Open Decisions
## Accepted Decisions
### OD-1: Semantic Repair Budget and Configuration Scope
### Semantic Repair Budget and Configuration Scope
**Recommendation:** start with one additional semantic repair attempt per
rejected proposal, with a root default and per-stage override; zero disables the
loop. A revision that fails validation can continue when the operator configures
Allow one additional semantic repair attempt per rejected proposal by default,
configurable at the root and per stage; zero disables the loop. A revision that
fails validation can continue when the operator configures
more attempts. Use a distinct setting from structural `llm.repair_attempts`.
Retain existing per-call timeouts and parent cancellation, and document that
each semantic attempt may include transport retries and structural repairs.
This provides useful feedback-driven revision with a simple finite bound.
The proposed default and setting names are not yet accepted configuration.
The default and scope are accepted; exact configuration names remain an
implementation-planning choice.
**Alternative:** default to zero and require explicit enabling per run or stage.
This avoids additional model work until operators choose it and supports gradual
evaluation, but leaves the new behavior inactive in ordinary use. A shared
per-stage call budget is another viable model for bounding aggregate work; it
offers tighter run-cost control but introduces competition between proposals
and requires a deterministic allocation policy.
### Full History That Does Not Fit
### OD-2: Full History That Does Not Fit
**Recommendation:** account for the complete conversation and feedback using
the supported request-sizing facilities; if a repair cannot fit the selected
Account for the complete conversation and feedback using the supported
request-sizing facilities; if a repair cannot fit the selected
limit, end that branch with a reported size failure and keep unrelated accepted
work. Preserve existing context-estimation limits rather than promising exact
provider token accounting. This honors the agreed full-history design without
silently truncating context or changing the execution profile.
**Alternative:** support an explicitly configured fallback repair profile with
a larger context allowance. It can preserve full history and rescue otherwise
blocked corrections, but adds profile selection, readiness, and failure-handling
work and may change cost and model behavior. Smaller initial chunks are also
available through existing configuration, at the expense of reduced proposal
context; automatic rechunking is not necessary for the initial feature.
### Repair Execution Profile
### OD-3: Repair Execution Profile
**Recommendation:** inherit the effective proposal profile by default, with an
optional root/per-stage repair-profile override. Resolve fallback after stage
Inherit the effective proposal profile by default, with an optional root/per-stage
repair-profile override. Resolve fallback after stage
settings are combined, following the existing validation-profile pattern. This
keeps ordinary configuration small while allowing a different model for a
difficult revision. Backend capacity remains shared through PromptKit.
**Alternative:** always use the proposal profile, with no separate selector.
This is simpler and keeps model behavior consistent across the conversation,
but prevents targeted use of a more capable or larger-context repair model.
Neither choice requires a separate backend or credential registry.
### Feedback Eligibility and Unavailable Validation
### OD-4: Feedback Eligibility and Unavailable Validation
**Recommendation:** collect all meaningful deterministic failures, then run the
applicable semantic validators when the candidate has the usable target and
Collect all meaningful deterministic failures, then run the applicable semantic
validators when the candidate has the usable target and
before/after data they need, even if another independent check rejected it.
An actionable rejection may enter repair; no-effect proposals can finish without
repair because they contribute no edit. Build a small applicability map from
@@ -327,18 +325,10 @@ the current unavailable outcome, leaving decision-recovery calls to the deferred
feature. This produces useful feedback without treating infrastructure failure
as evidence against the proposed wording.
**Alternative:** collect the same applicable findings but require all applicable
validator operations to finish successfully before starting a repair. If any
assessment is unavailable, keep the candidate unavailable for this stage. This
gives each repair a complete assessment and avoids spending calls on a candidate
whose review could not finish, but sacrifices opportunities to address a known
rejection while another validator is temporarily unavailable. Both approaches
preserve full validation as a condition of eventual approval.
### Ordinary Reports and Attempt History
### OD-5: Ordinary Reports and Attempt History
**Recommendation:** retain compact proposal lineage, attempt counts, and final
dispositions in ordinary results, with meaningful errors and recoveries in the
Retain compact proposal lineage, attempt counts, and final dispositions in
ordinary results, with meaningful errors and recoveries in the
existing report framework. Distinguish approval, actual application, withdrawal,
and exhausted or unavailable repair. Keep full conversations and detailed
per-attempt evidence in optional redacted debug artifacts. A rejected proposal
@@ -348,24 +338,11 @@ Choose concrete fields and any output-schema version change during implementatio
planning, preserving the existing failure and exit-status policy unless an
explicitly identified requirement calls for a change.
**Alternative:** include every revision and validator finding in the ordinary
correction ledger. This improves analysis without enabling debug capture, but
increases report size, retained transcript content, and public schema surface.
It requires clear grouping so attempted revisions are not counted as independent
corrections or successful recoveries.
### Revising the Span or Target Reference
### OD-6: Revising the Span or Target Reference
**Recommendation:** allow the revised proposal to correct its original-text span
and segment reference within the original request's section, while retaining its
Allow the revised proposal to correct its original-text span and segment
reference within the original request's section, while retaining its
proposal lineage. Apply the ordinary target mapping, replacement policy, and full
validation to the new candidate. This permits repair of an incorrect source span
or model-facing target reference, both of which can cause validation failures.
It adds no new transcript-wide targeting mechanism.
**Alternative:** keep a resolvable target segment fixed, allowing changes to the
span and replacement only within that segment; terminate invalid-target proposals
without domain repair. This makes continuity easier to interpret and reduces
the chance of revising a different edit, but cannot repair misidentified targets.
The restriction would be a deliberate feature choice, not an invariant inferred
from the current proposal representation.

View File

@@ -1,6 +1,6 @@
# Deferred Feature Work
**Last updated:** 2026-11-10
**Last updated:** 2026-11-11
**Status:** Deferred or exploratory work; not implemented or scheduled. These
entries preserve work beyond completed feature and library upgrades.
@@ -12,7 +12,7 @@ implementing every capability.
The implemented [partial-success decision reconciliation](../internal/validators.md#llm-validator-batching)
preserves usable decisions without another model call. Revision of rejected
correction proposals is now an agreed feature direction in the
[correction-repair roadmap](correction-repair.md), with its own open decisions.
[correction-repair roadmap](correction-repair.md), with its feature decisions accepted.
That work includes substantive validator feedback and explicit proposal withdrawal.
It is distinct from recovering an unavailable validator decision.