Record accepted correction repair decisions
This commit is contained in:
@@ -1,9 +1,10 @@
|
||||
# Correction Repair After Validation
|
||||
|
||||
**Updated:** 2026-11-10
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Feature direction agreed; implementation pending. Open decisions
|
||||
below require resolution before the dependent implementation work. This document
|
||||
**Status:** Feature direction and all six decisions accepted; implementation
|
||||
pending. No open feature decisions remain. Configuration names, concrete schema
|
||||
fields, and adapter details remain implementation-planning work. This document
|
||||
owns the target and remaining work, not a release date or implementation status
|
||||
for the completed library adoption.
|
||||
|
||||
@@ -80,9 +81,9 @@ This feature does not add a second repair loop for application-time conflicts.
|
||||
|
||||
Each revision remains associated with the original proposal and its processing
|
||||
stage and section. Repair responses must identify the requested proposal; unrelated
|
||||
edits do not become additional candidates. OD-6 addresses whether revisions can
|
||||
change the proposed span or target reference within the original section context;
|
||||
the ordinary mapping and validators still determine applicability.
|
||||
edits do not become additional candidates. Revisions may correct the proposed
|
||||
span or target reference within the original section context, as specified in
|
||||
[the accepted target decision](#revising-the-span-or-target-reference).
|
||||
Exact internal types and model-facing reference spelling are implementation
|
||||
choices, not new globally unique identity requirements.
|
||||
|
||||
@@ -110,7 +111,8 @@ Keep required conversation state in memory when repair is enabled, independently
|
||||
of debug artifact saving. Its lifetime need only cover the associated work; this
|
||||
does not introduce durable conversation storage or restart/resume support. Count
|
||||
the complete replay, feedback, and applicable output contract when assessing
|
||||
request size. Context-limit handling is OD-2 below.
|
||||
request size. Apply the accepted
|
||||
[context-limit behavior](#full-history-that-does-not-fit).
|
||||
|
||||
### Module Repair Prompts and Response Contract
|
||||
|
||||
@@ -151,8 +153,9 @@ existing structured decision path.
|
||||
Collect failures from all applicable checks that can meaningfully evaluate the
|
||||
candidate. A malformed target may prevent contextual checks; a provider failure
|
||||
means the assessment is unavailable, not that the edit violated a domain rule.
|
||||
OD-4 settles the eligibility and continuation policy. Making validator chains
|
||||
operator-configurable is not required by this feature.
|
||||
Apply the accepted
|
||||
[eligibility and continuation policy](#feedback-eligibility-and-unavailable-validation).
|
||||
Making validator chains operator-configurable is not required by this feature.
|
||||
|
||||
### Ownership and Library Integration
|
||||
|
||||
@@ -178,15 +181,14 @@ fatal application/invariant failures, and cancellation. Domain revision attempts
|
||||
are additional logical calls, each potentially containing PromptKit retries and
|
||||
structural repairs; account for these separately. Share existing backend admission
|
||||
and cancellation instead of adding a repair-specific global concurrency limiter.
|
||||
Reporting choices remain in OD-5.
|
||||
Use the accepted [reporting scope](#ordinary-reports-and-attempt-history).
|
||||
|
||||
## Work From the Baseline
|
||||
|
||||
1. **Resolve the open policies and prove the request path.** Settle the decisions
|
||||
below. Exercise native message composition with an original multi-proposal
|
||||
response, a rendered module repair instruction, a revision response schema,
|
||||
and a subsequent branch turn. Confirm operator prompt replacements and the
|
||||
same behavior with debug disabled.
|
||||
1. **Prove the request path.** Exercise native message composition with an
|
||||
original multi-proposal response, a rendered module repair instruction, a
|
||||
revision response schema, and a subsequent branch turn. Confirm operator
|
||||
prompt replacements and the same behavior with debug disabled.
|
||||
2. **Carry conversation state through the existing boundaries.** Extend
|
||||
`framework/structuredoutput`, the LLM adapter, proposal-generation results,
|
||||
and module/framework contracts as needed to retain and replay the original
|
||||
@@ -206,13 +208,16 @@ Reporting choices remain in OD-5.
|
||||
branch isolation, deterministic application, cancellation, and existing
|
||||
apply-time checks. Unsuccessful branches must not discard unrelated accepted
|
||||
work. This does not require redesigning stage barriers or chunk scheduling.
|
||||
6. **Wire configuration and readiness.** Add the selected budget/profile settings
|
||||
with native presence and stage inheritance. Keep semantic attempts distinct
|
||||
from structural `llm.repair_attempts`. Preflight checks any additional required
|
||||
role/prompt readiness when enabled, without making disabled repair require
|
||||
an unused profile. Let existing inspection expose resolved settings.
|
||||
7. **Extend outcomes and documentation.** Implement OD-5 through ordinary
|
||||
reporting, correction ledgers, and optional redacted debug artifacts. Update
|
||||
6. **Wire configuration and readiness.** Add the accepted
|
||||
[budget](#semantic-repair-budget-and-configuration-scope)
|
||||
and [profile](#repair-execution-profile) settings with native presence and stage
|
||||
inheritance. Keep semantic attempts distinct from structural
|
||||
`llm.repair_attempts`. Preflight checks any additional required role/prompt
|
||||
readiness when enabled, without making disabled repair require an unused
|
||||
profile. Let existing inspection expose resolved settings.
|
||||
7. **Extend outcomes and documentation.** Implement the accepted reporting scope
|
||||
through ordinary reporting, correction ledgers, and optional redacted debug
|
||||
artifacts. Update
|
||||
the canonical [configuration](../config.md), [pipeline](../internal/pipeline.md),
|
||||
[module](../internal/modules.md), [validator](../internal/validators.md),
|
||||
[prompt](../internal/prompts.md), [runtime](../internal/llm-runtime.md), and
|
||||
@@ -234,85 +239,78 @@ boundaries. Cover the following behavior with proportionate tests:
|
||||
- Full initial messages and full response retained, correct roles and ordering,
|
||||
explicit proposal selection, independent branches, and a second repair turn
|
||||
containing only its own branch's history. Exercise debug enabled and disabled.
|
||||
- Repair success, explicit withdrawal, budget exhaustion, and configured zero
|
||||
attempts. Distinguish withdrawal from an empty-string deletion proposal.
|
||||
- Validator-owned findings collected under the chosen continuation policy;
|
||||
- Repair success, explicit withdrawal, budget exhaustion, the default of one
|
||||
semantic attempt, additional configured attempts, and global or stage-level
|
||||
zero disabling repair. Distinguish withdrawal from an empty-string deletion
|
||||
proposal.
|
||||
- Validator-owned findings collected under the accepted continuation policy;
|
||||
inapplicable or unavailable checks do not masquerade as editorial rejection.
|
||||
Cover known rejection alongside an unavailable assessment, unavailable
|
||||
assessment alone, and a no-effect proposal finishing without repair.
|
||||
Revised candidates must pass all applicable checks, including checks that
|
||||
approved their earlier versions.
|
||||
- Wrong proposal references and unusable repair responses cannot introduce
|
||||
unrelated edits or approval. Preserve ordinary identity reconciliation and
|
||||
structural validation; no generic salvage of malformed JSON is implied.
|
||||
- A revision can correct its source span or segment reference within the original
|
||||
request's section, preserving proposal lineage and passing ordinary target
|
||||
mapping and full validation.
|
||||
- Context-size handling, provider failure, structural-repair exhaustion, and
|
||||
cancellation follow the selected policies while preserving existing
|
||||
cancellation follow the accepted policies while preserving existing
|
||||
partial-success and fatal-error boundaries. Account for semantic attempts,
|
||||
structural repairs, and transport retries without conflating them.
|
||||
An oversized conversation ends only the affected branch with a reported size
|
||||
failure, preserving unrelated accepted work.
|
||||
- Embedded module repair prompts and operator replacements work through the
|
||||
shared engine. Root and stage settings follow the selected scope and fallback
|
||||
rules. Reports identify attempts and final outcomes without presenting every
|
||||
attempt as a separate applied correction or exposing raw history by default.
|
||||
shared engine. Root and stage settings follow the accepted scope and fallback
|
||||
rules, including inheritance of the effective proposal profile after stage
|
||||
settings are combined and explicit repair-profile overrides. Reports identify
|
||||
attempts and final outcomes without presenting every attempt as a separate
|
||||
applied correction or exposing raw history by default.
|
||||
|
||||
Run focused checks during development and assembled application validation at
|
||||
completion under the [testing policy](../policy/testing.md). Representative
|
||||
transcript comparisons can help tune prompts and defaults, but this roadmap
|
||||
sets no numerical quality threshold or requirement for paid provider calls.
|
||||
Completion requires the agreed loop, resolved open policies, acceptance evidence,
|
||||
Completion requires the agreed loop and accepted policies, acceptance evidence,
|
||||
and updated canonical documentation. Successful schema validation alone is not
|
||||
evidence that a revised correction is acceptable.
|
||||
|
||||
## Open Decisions
|
||||
## Accepted Decisions
|
||||
|
||||
### OD-1: Semantic Repair Budget and Configuration Scope
|
||||
### Semantic Repair Budget and Configuration Scope
|
||||
|
||||
**Recommendation:** start with one additional semantic repair attempt per
|
||||
rejected proposal, with a root default and per-stage override; zero disables the
|
||||
loop. A revision that fails validation can continue when the operator configures
|
||||
Allow one additional semantic repair attempt per rejected proposal by default,
|
||||
configurable at the root and per stage; zero disables the loop. A revision that
|
||||
fails validation can continue when the operator configures
|
||||
more attempts. Use a distinct setting from structural `llm.repair_attempts`.
|
||||
Retain existing per-call timeouts and parent cancellation, and document that
|
||||
each semantic attempt may include transport retries and structural repairs.
|
||||
This provides useful feedback-driven revision with a simple finite bound.
|
||||
The proposed default and setting names are not yet accepted configuration.
|
||||
The default and scope are accepted; exact configuration names remain an
|
||||
implementation-planning choice.
|
||||
|
||||
**Alternative:** default to zero and require explicit enabling per run or stage.
|
||||
This avoids additional model work until operators choose it and supports gradual
|
||||
evaluation, but leaves the new behavior inactive in ordinary use. A shared
|
||||
per-stage call budget is another viable model for bounding aggregate work; it
|
||||
offers tighter run-cost control but introduces competition between proposals
|
||||
and requires a deterministic allocation policy.
|
||||
### Full History That Does Not Fit
|
||||
|
||||
### OD-2: Full History That Does Not Fit
|
||||
|
||||
**Recommendation:** account for the complete conversation and feedback using
|
||||
the supported request-sizing facilities; if a repair cannot fit the selected
|
||||
Account for the complete conversation and feedback using the supported
|
||||
request-sizing facilities; if a repair cannot fit the selected
|
||||
limit, end that branch with a reported size failure and keep unrelated accepted
|
||||
work. Preserve existing context-estimation limits rather than promising exact
|
||||
provider token accounting. This honors the agreed full-history design without
|
||||
silently truncating context or changing the execution profile.
|
||||
|
||||
**Alternative:** support an explicitly configured fallback repair profile with
|
||||
a larger context allowance. It can preserve full history and rescue otherwise
|
||||
blocked corrections, but adds profile selection, readiness, and failure-handling
|
||||
work and may change cost and model behavior. Smaller initial chunks are also
|
||||
available through existing configuration, at the expense of reduced proposal
|
||||
context; automatic rechunking is not necessary for the initial feature.
|
||||
### Repair Execution Profile
|
||||
|
||||
### OD-3: Repair Execution Profile
|
||||
|
||||
**Recommendation:** inherit the effective proposal profile by default, with an
|
||||
optional root/per-stage repair-profile override. Resolve fallback after stage
|
||||
Inherit the effective proposal profile by default, with an optional root/per-stage
|
||||
repair-profile override. Resolve fallback after stage
|
||||
settings are combined, following the existing validation-profile pattern. This
|
||||
keeps ordinary configuration small while allowing a different model for a
|
||||
difficult revision. Backend capacity remains shared through PromptKit.
|
||||
|
||||
**Alternative:** always use the proposal profile, with no separate selector.
|
||||
This is simpler and keeps model behavior consistent across the conversation,
|
||||
but prevents targeted use of a more capable or larger-context repair model.
|
||||
Neither choice requires a separate backend or credential registry.
|
||||
### Feedback Eligibility and Unavailable Validation
|
||||
|
||||
### OD-4: Feedback Eligibility and Unavailable Validation
|
||||
|
||||
**Recommendation:** collect all meaningful deterministic failures, then run the
|
||||
applicable semantic validators when the candidate has the usable target and
|
||||
Collect all meaningful deterministic failures, then run the applicable semantic
|
||||
validators when the candidate has the usable target and
|
||||
before/after data they need, even if another independent check rejected it.
|
||||
An actionable rejection may enter repair; no-effect proposals can finish without
|
||||
repair because they contribute no edit. Build a small applicability map from
|
||||
@@ -327,18 +325,10 @@ the current unavailable outcome, leaving decision-recovery calls to the deferred
|
||||
feature. This produces useful feedback without treating infrastructure failure
|
||||
as evidence against the proposed wording.
|
||||
|
||||
**Alternative:** collect the same applicable findings but require all applicable
|
||||
validator operations to finish successfully before starting a repair. If any
|
||||
assessment is unavailable, keep the candidate unavailable for this stage. This
|
||||
gives each repair a complete assessment and avoids spending calls on a candidate
|
||||
whose review could not finish, but sacrifices opportunities to address a known
|
||||
rejection while another validator is temporarily unavailable. Both approaches
|
||||
preserve full validation as a condition of eventual approval.
|
||||
### Ordinary Reports and Attempt History
|
||||
|
||||
### OD-5: Ordinary Reports and Attempt History
|
||||
|
||||
**Recommendation:** retain compact proposal lineage, attempt counts, and final
|
||||
dispositions in ordinary results, with meaningful errors and recoveries in the
|
||||
Retain compact proposal lineage, attempt counts, and final dispositions in
|
||||
ordinary results, with meaningful errors and recoveries in the
|
||||
existing report framework. Distinguish approval, actual application, withdrawal,
|
||||
and exhausted or unavailable repair. Keep full conversations and detailed
|
||||
per-attempt evidence in optional redacted debug artifacts. A rejected proposal
|
||||
@@ -348,24 +338,11 @@ Choose concrete fields and any output-schema version change during implementatio
|
||||
planning, preserving the existing failure and exit-status policy unless an
|
||||
explicitly identified requirement calls for a change.
|
||||
|
||||
**Alternative:** include every revision and validator finding in the ordinary
|
||||
correction ledger. This improves analysis without enabling debug capture, but
|
||||
increases report size, retained transcript content, and public schema surface.
|
||||
It requires clear grouping so attempted revisions are not counted as independent
|
||||
corrections or successful recoveries.
|
||||
### Revising the Span or Target Reference
|
||||
|
||||
### OD-6: Revising the Span or Target Reference
|
||||
|
||||
**Recommendation:** allow the revised proposal to correct its original-text span
|
||||
and segment reference within the original request's section, while retaining its
|
||||
Allow the revised proposal to correct its original-text span and segment
|
||||
reference within the original request's section, while retaining its
|
||||
proposal lineage. Apply the ordinary target mapping, replacement policy, and full
|
||||
validation to the new candidate. This permits repair of an incorrect source span
|
||||
or model-facing target reference, both of which can cause validation failures.
|
||||
It adds no new transcript-wide targeting mechanism.
|
||||
|
||||
**Alternative:** keep a resolvable target segment fixed, allowing changes to the
|
||||
span and replacement only within that segment; terminate invalid-target proposals
|
||||
without domain repair. This makes continuity easier to interpret and reduces
|
||||
the chance of revising a different edit, but cannot repair misidentified targets.
|
||||
The restriction would be a deliberate feature choice, not an invariant inferred
|
||||
from the current proposal representation.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Deferred Feature Work
|
||||
|
||||
**Last updated:** 2026-11-10
|
||||
**Last updated:** 2026-11-11
|
||||
|
||||
**Status:** Deferred or exploratory work; not implemented or scheduled. These
|
||||
entries preserve work beyond completed feature and library upgrades.
|
||||
@@ -12,7 +12,7 @@ implementing every capability.
|
||||
The implemented [partial-success decision reconciliation](../internal/validators.md#llm-validator-batching)
|
||||
preserves usable decisions without another model call. Revision of rejected
|
||||
correction proposals is now an agreed feature direction in the
|
||||
[correction-repair roadmap](correction-repair.md), with its own open decisions.
|
||||
[correction-repair roadmap](correction-repair.md), with its feature decisions accepted.
|
||||
That work includes substantive validator feedback and explicit proposal withdrawal.
|
||||
It is distinct from recovering an unavailable validator decision.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user