Remove completed correction repair planning documents
This commit is contained in:
@@ -1,292 +0,0 @@
|
||||
# Correction Repair After Validation
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Implemented. See the [implementation plan](implementation.md)
|
||||
and the [pipeline guide](../internal/pipeline.md).
|
||||
|
||||
## Objective
|
||||
|
||||
Give a rejected correction an opportunity to improve using specific validator
|
||||
feedback, while preserving proposals that have already passed validation. Each
|
||||
correction module supplies a repair prompt. Each validator supplies findings
|
||||
within its own remit. Audita coordinates bounded revision conversations and
|
||||
acceptance through the existing PromptKit runtime.
|
||||
|
||||
The unit of repair is an individual correction proposal. Multiple proposals
|
||||
against one transcript segment remain separately identifiable. A model may revise
|
||||
the selected proposal or explicitly withdraw it. Every revised proposal must pass
|
||||
the full applicable validation chain before it becomes eligible for application.
|
||||
|
||||
This adds domain revision after rejection. It is distinct from PromptKit's
|
||||
structural output repair and from obtaining an unavailable validator decision.
|
||||
The latter remains separate [deferred work](future.md#domain-aware-response-recovery).
|
||||
|
||||
## Current Baseline
|
||||
|
||||
Reviewed application revision: `8946a82`. Recheck the owning components when
|
||||
implementation begins, including any subsequent correctness-review fixes.
|
||||
|
||||
- The [pipeline](../internal/pipeline.md) partitions the transcript once and runs
|
||||
ordered stages. Proposals can be generated concurrently; indexing and
|
||||
application remain deterministic. Approved corrections are applied in proposal
|
||||
order, with existing replacement and conflict checks.
|
||||
- [Proposal generation](../internal/modules.md#proposal-generation-ownership)
|
||||
returns a correction set, potentially with several proposals per segment.
|
||||
Model-facing segment references are request-local and are mapped to transcript
|
||||
identities. The four modules are glossary, homophones, spoken-word cleanup,
|
||||
and grammar.
|
||||
- The [validator runtime](../internal/validators.md) runs deterministic checks
|
||||
before LLM-backed checks and filters rejected candidates before later
|
||||
validators. Decisions already carry proposal identity, disposition, reason code,
|
||||
and message. Unavailable decisions are distinguished from editorial rejection.
|
||||
Validator chains are currently selected by module, not individually configured
|
||||
by operators.
|
||||
- The [LLM adapter](../internal/llm-runtime.md) uses PromptKit v0.21.0 prepared
|
||||
execution with ConfigKit v1.2.0. It captures prepared-request data and final
|
||||
response text. Proposal-generation results carry candidates, operation outcomes,
|
||||
and artifact references, but do not retain a conversation for domain revision.
|
||||
- [Embedded prompts](../internal/prompts.md) live in the `audita` collection and
|
||||
support operator replacement through native underlays. Audita supplies response
|
||||
schemas separately. Full interaction artifacts are optional debug output.
|
||||
- Existing `llm.repair_attempts` controls structural repairs. It is not a budget
|
||||
for the proposed domain revision loop. No such loop is implemented.
|
||||
|
||||
## Targeted End State
|
||||
|
||||
### Proposal Lifecycle
|
||||
|
||||
1. Generate the initial correction set using the stage's instruction, transcript
|
||||
section, relevant glossary/protection data, and response schema.
|
||||
2. Validate candidates, collecting findings from checks that can meaningfully
|
||||
evaluate them. Checks that cannot run do not become invented rejection reasons.
|
||||
3. Retain approved proposals for application. Give each rejected proposal a repair
|
||||
branch containing its conversation and findings. Budget and eligibility rules
|
||||
determine whether another request can run.
|
||||
4. Request revision or withdrawal of that one proposal. An accepted withdrawal
|
||||
ends the branch without an edit. A revised proposal returns to the complete
|
||||
applicable validator chain; approvals of a previous version do not transfer.
|
||||
5. Continue until approval, withdrawal, budget exhaustion, or a classified failure.
|
||||
Preserve independently accepted work under the existing partial-success rules.
|
||||
6. Apply accepted corrections in deterministic proposal order, retaining the
|
||||
existing apply-time checks for stale spans, protection, and conflicts.
|
||||
|
||||
Withdrawal or exhaustion contributes no edit from that proposal. Other accepted
|
||||
proposals against the same segment may still change its text. Retaining an
|
||||
approval does not guarantee application when another edit makes it inapplicable.
|
||||
This feature does not add a second repair loop for application-time conflicts.
|
||||
|
||||
Each revision remains associated with the original proposal and its processing
|
||||
stage and section. Repair responses must identify the requested proposal; unrelated
|
||||
edits do not become additional candidates. Revisions may correct the proposed
|
||||
span or target reference within the original section context, as specified in
|
||||
[the accepted target decision](#revising-the-span-or-target-reference).
|
||||
Exact internal types and model-facing reference spelling are implementation
|
||||
choices, not new globally unique identity requirements.
|
||||
|
||||
### Full Conversation and Independent Branches
|
||||
|
||||
The first repair request contains the complete initial rendered request messages,
|
||||
with their original roles and order, followed by the complete assistant response
|
||||
that supplied the decoded proposal set, then the targeted repair instruction.
|
||||
Retain the original section and request inputs for this purpose; do not rebuild
|
||||
the conversation from a transcript already changed by other corrections.
|
||||
|
||||
Every proposal branches independently from that same initial exchange. Later
|
||||
attempts append that proposal's previous repair response and new feedback to its
|
||||
own branch. Repairs of other proposals never become turns in that conversation.
|
||||
Keep approved proposals outside the set being reconsidered. Branch scheduling
|
||||
must not change proposal attribution or final application order.
|
||||
|
||||
Here, conversation history means the application-level message exchange. Preserve
|
||||
the full response used for candidate decoding rather than serializing only the
|
||||
failed proposal as if it were the model's original answer. Provider transport
|
||||
attempts and PromptKit's internal structural-repair operations remain accounted
|
||||
for separately. Do not manufacture hidden provider messages or reasoning.
|
||||
|
||||
Keep required conversation state in memory when repair is enabled, independently
|
||||
of debug artifact saving. Its lifetime need only cover the associated work; this
|
||||
does not introduce durable conversation storage or restart/resume support. Send
|
||||
the full replay and feedback under the revision output contract, subject to the
|
||||
selected provider's [request limits](#provider-request-limits).
|
||||
|
||||
### Module Repair Prompts and Response Contract
|
||||
|
||||
Each built-in correction module owns a corresponding repair definition in the
|
||||
`audita` namespace, alongside its proposal definition. Use the existing embedded
|
||||
filesystem and ConfigKit underlay mechanism so operators can replace repair
|
||||
prompts without rebuilding Audita. Shared instruction fragments may be reused;
|
||||
module-specific instructions stay with the module's prompt.
|
||||
|
||||
The repair instruction identifies the proposal, repeats its original and proposed
|
||||
replacement text, presents the findings, and asks for a revision or withdrawal.
|
||||
It explains that only this proposal is under review and states the response
|
||||
contract for this turn. This must remain clear even though the initial prompt
|
||||
requested a complete correction set. Later turns refer explicitly to the current
|
||||
proposal version rather than relying on an ambiguous "last message" reference.
|
||||
|
||||
Use a structured response that distinguishes revision from withdrawal and carries
|
||||
the requested proposal reference. A revision contains the correction data needed
|
||||
by the existing validators. Empty replacement text remains a possible span
|
||||
deletion; it must not double as a withdrawal signal. The response schema owns
|
||||
shape validation. Audita owns correspondence and domain acceptance.
|
||||
|
||||
### Validator-Owned Feedback
|
||||
|
||||
Validators contribute their own actionable findings: validator identity, stable
|
||||
reason code, an explanation, and relevant evidence or constraints where available.
|
||||
Build on existing decision data rather than creating a parallel reason registry.
|
||||
For example, a protection finding can identify the spelling affected, while an
|
||||
editorial finding can explain the meaning change it detected.
|
||||
|
||||
The module repair template owns presentation of the collected findings.
|
||||
Validators do not construct chat histories or own the correction response schema.
|
||||
Order feedback by the applicable validator chain for understandable reports and
|
||||
prompts. Model-backed validators should contribute their assessment through the
|
||||
existing structured decision path.
|
||||
|
||||
Collect failures from all applicable checks that can meaningfully evaluate the
|
||||
candidate. A malformed target may prevent contextual checks; a provider failure
|
||||
means the assessment is unavailable, not that the edit violated a domain rule.
|
||||
Apply the accepted
|
||||
[eligibility and continuation policy](#feedback-eligibility-and-unavailable-validation).
|
||||
Making validator chains operator-configurable is not required by this feature.
|
||||
|
||||
### Ownership and Library Integration
|
||||
|
||||
Audita owns proposal lineage, semantic feedback, branch state, revision budgets,
|
||||
revalidation, and transcript application. PromptKit continues to own rendering,
|
||||
profile/backend selection, transport retry, structural repair, and per-backend
|
||||
admission. Configuration continues through ConfigKit's native resolution and
|
||||
the existing root/stage scope conventions. Standard CommandKit inspection and
|
||||
request commands expose the configured prompt assets through their normal APIs.
|
||||
|
||||
PromptKit already supports
|
||||
[appending rendered assistant and corrective messages](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/docs/consumers/pkg-promptkit.md#append-already-rendered-messages).
|
||||
Use its public preparation and execution APIs to render the module repair
|
||||
definition and assemble the intended conversation from rendered messages. Use
|
||||
its per-call schema selection for the revision contract.
|
||||
|
||||
Maintain the existing distinctions between recoverable model-operation failures,
|
||||
fatal application/invariant failures, and cancellation. Domain revision attempts
|
||||
are additional logical calls, each potentially containing PromptKit retries and
|
||||
structural repairs; account for these separately. Share existing backend admission
|
||||
and cancellation instead of adding a repair-specific global concurrency limiter.
|
||||
Use the accepted [reporting scope](#ordinary-reports-and-attempt-history).
|
||||
|
||||
## Acceptance and Completion
|
||||
|
||||
Use controlled model responses and real loading/rendering at integration
|
||||
boundaries. Cover the following behavior with proportionate tests:
|
||||
|
||||
- One approved and one rejected proposal from the same response: repair only
|
||||
the rejected proposal, then apply the accepted results in deterministic order.
|
||||
Include several proposals against one segment and an overlapping-edit case.
|
||||
- Full initial messages and full response retained, correct roles and ordering,
|
||||
explicit proposal selection, independent branches, and a second repair turn
|
||||
containing only its own branch's history. Exercise debug enabled and disabled.
|
||||
- Repair success, explicit withdrawal, budget exhaustion, the default of one
|
||||
semantic attempt, additional configured attempts, and global or stage-level
|
||||
zero disabling both repair and extra feedback collection. Distinguish
|
||||
withdrawal from an empty-string deletion proposal.
|
||||
- Validator-owned findings collected under the accepted continuation policy;
|
||||
inapplicable or unavailable checks do not masquerade as editorial rejection.
|
||||
Cover known rejection alongside an unavailable assessment, unavailable
|
||||
assessment alone, and a no-effect proposal finishing without repair.
|
||||
Revised candidates must pass all applicable checks, including checks that
|
||||
approved their earlier versions.
|
||||
- Wrong proposal references and unusable repair responses cannot introduce
|
||||
unrelated edits or approval. Preserve ordinary identity reconciliation and
|
||||
structural validation; no generic salvage of malformed JSON is implied.
|
||||
- A revision can correct its source span or segment reference within the original
|
||||
request's section, preserving proposal lineage and passing ordinary target
|
||||
mapping and full validation.
|
||||
- A provider error on a repair request, including a context-size rejection,
|
||||
fails only that proposal after any applicable PromptKit retries. Other approved
|
||||
work remains eligible for application. Cover structural-repair exhaustion and
|
||||
cancellation under the existing failure boundaries. Account for semantic
|
||||
attempts, structural repairs, and transport retries separately.
|
||||
- Embedded module repair prompts and operator replacements work through the
|
||||
shared engine. Root and stage settings follow the accepted scope and fallback
|
||||
rules, including inheritance of the effective proposal profile after stage
|
||||
settings are combined and explicit repair-profile overrides. Verify that
|
||||
ordinary reporting records one compact outcome per repair branch and that
|
||||
optional redacted debug output carries the detailed attempt history.
|
||||
|
||||
Run focused checks during development and assembled application validation at
|
||||
completion under the [testing policy](../policy/testing.md). Representative
|
||||
transcript comparisons can help tune prompts and defaults, but this roadmap
|
||||
sets no numerical quality threshold or requirement for paid provider calls.
|
||||
Completion requires the agreed loop and accepted policies, acceptance evidence,
|
||||
and updated canonical documentation. Successful schema validation alone is not
|
||||
evidence that a revised correction is acceptable.
|
||||
|
||||
## Accepted Decisions
|
||||
|
||||
### Semantic Repair Budget and Configuration Scope
|
||||
|
||||
Allow one additional semantic repair attempt per rejected proposal by default,
|
||||
configurable at the root and per stage. Zero disables both the repair loop and
|
||||
extra feedback collection. A revision that fails validation can continue when
|
||||
the operator configures more attempts. Use a distinct setting from structural
|
||||
`llm.repair_attempts`.
|
||||
Retain existing per-call timeouts and parent cancellation, and document that
|
||||
each semantic attempt may include transport retries and structural repairs.
|
||||
This provides useful feedback-driven revision with a simple finite bound.
|
||||
|
||||
### Provider Request Limits
|
||||
|
||||
Submit the full repair conversation and let the selected provider enforce its
|
||||
request limits. If a provider error prevents revision after any applicable
|
||||
PromptKit retries, including when the conversation exceeds the provider's context
|
||||
window, record the repair as unavailable and end that proposal's branch.
|
||||
Continue processing other proposals. Audita's existing fatal application error
|
||||
and cancellation behavior still applies.
|
||||
|
||||
### Repair Execution Profile
|
||||
|
||||
Inherit the effective proposal profile by default, with an optional root/per-stage
|
||||
repair-profile override. Resolve fallback after stage
|
||||
settings are combined, following the existing validation-profile pattern. This
|
||||
keeps ordinary configuration small while allowing a different model for a
|
||||
difficult revision. Backend capacity remains shared through PromptKit.
|
||||
|
||||
### Feedback Eligibility and Unavailable Validation
|
||||
|
||||
When repair is enabled, collect all meaningful deterministic failures, then run
|
||||
the applicable semantic validators when the candidate has the usable target and
|
||||
before/after data they need, even if another independent check rejected it. With
|
||||
an effective repair budget of zero, later validators receive only the candidates
|
||||
that advance under the current validation flow.
|
||||
An actionable rejection may enter repair; no-effect proposals can finish without
|
||||
repair because they contribute no edit. Build a small applicability map from
|
||||
the actual checks rather than running validators on unusable inputs or encoding
|
||||
universal dependencies between them.
|
||||
|
||||
Keep unavailable validation as a separate operation outcome. If a proposal also
|
||||
has an actionable rejection, its repair may proceed using the known findings,
|
||||
with unavailable assessments identified as such; any revision still needs a full
|
||||
successful validation chain. An unavailable assessment by itself should retain
|
||||
the current unavailable outcome, leaving decision-recovery calls to the deferred
|
||||
feature. This produces useful feedback without treating infrastructure failure
|
||||
as evidence against the proposed wording.
|
||||
|
||||
### Ordinary Reports and Attempt History
|
||||
|
||||
Use one compact repair-outcomes section in ordinary reporting. Give each repair
|
||||
branch one entry with the original proposal reference, its attempt count, and
|
||||
final disposition. Use the existing work-unit context for stage and chunk
|
||||
identity. Make it clear whether an approved revision was applied. Record
|
||||
operational errors and recoveries through the existing report framework and keep
|
||||
the current exit-status policy.
|
||||
Editorial rejection and voluntary withdrawal are correction outcomes. Keep
|
||||
conversation text and per-attempt details in optional redacted debug artifacts.
|
||||
|
||||
### Revising the Span or Target Reference
|
||||
|
||||
Allow the revised proposal to correct its original-text span and segment
|
||||
reference within the original request's section, while retaining its
|
||||
proposal lineage. Apply the ordinary target mapping, replacement policy, and full
|
||||
validation to the new candidate. This permits repair of an incorrect source span
|
||||
or model-facing target reference, both of which can cause validation failures.
|
||||
It adds no new transcript-wide targeting mechanism.
|
||||
@@ -1,385 +0,0 @@
|
||||
# Correction Repair Implementation Plan
|
||||
|
||||
Implement the stages in order, one assignment at a time. The [feature
|
||||
roadmap](correction-repair.md) owns the intended behavior and operator choices.
|
||||
The implementation baseline is `b036336`, with PromptKit v0.21.0 and ConfigKit
|
||||
v1.2.0. Start with the [contributor guide](../development.md) and read the
|
||||
applicable policies and component contracts as the work reaches them.
|
||||
|
||||
This is one end-to-end sprint. Intermediate stages may leave callers or tests
|
||||
pending their later stages. Carry those changes forward directly and run a
|
||||
focused check when its boundary is assembled. Complete the assembled checks in
|
||||
Stage 9. Keep the existing fixed chunks, stage barriers, module scopes, provider
|
||||
admission, and apply-time safeguards. Decision recovery for unavailable
|
||||
validator responses remains [deferred](future.md#domain-aware-response-recovery).
|
||||
|
||||
## Stage 1 — Define the repair response and prove native conversation composition
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. In `internal/framework/responseschema`, register `CorrectionRepairKey` with
|
||||
schema ID `audita.correction_repair`, name `audita_correction_repair_v1`, and
|
||||
asset `correction_repair.v1.json`. Use a closed JSON object with three
|
||||
required fields below.
|
||||
|
||||
- `proposal_ref` is the one-based position in the **original correction
|
||||
array**, including items the application later rejects. Each turn echoes
|
||||
the same short reference for that proposal.
|
||||
- `action` is `revise` or `withdraw`.
|
||||
- `correction` is a nullable object. Its object shape uses the existing
|
||||
correction item fields `id`, `original_text`, `corrected_text`, and
|
||||
`confidence`. The `id` refers to a segment position in the original
|
||||
section, independent of `proposal_ref`. Keep the existing correction
|
||||
field constraints, including an empty replacement and a whitespace-only
|
||||
original span.
|
||||
2. Decode the response at the repair boundary. Require the expected reference
|
||||
and the consistent action/payload pair (`revise` with an object,
|
||||
`withdraw` with null). Treat an inconsistent pair or wrong reference as an
|
||||
unusable response with the existing `invalid_output` reason. Keep source
|
||||
span, segment applicability, and confidence threshold decisions with domain
|
||||
validation. PromptKit owns structural validation and its configured repair
|
||||
attempts.
|
||||
3. Add a focused offline test at the PromptKit integration boundary, using the
|
||||
new schema and a fixture with a multi-message proposal and a separate repair
|
||||
definition. Use the original proposal ID and variables as the base of each
|
||||
native request. Render the repair definition with `Engine.Prepare`. Append
|
||||
the full assistant response from the proposal call, then the rendered
|
||||
repair instruction using `RunRequest.AppendedMessages`. Set `Validation`
|
||||
to the repair schema and execute via `PrepareExecution` and `RunPrepared`.
|
||||
Exercise a second turn by appending the prior instruction and its full
|
||||
assistant response once, followed by the newly rendered instruction. The
|
||||
fixture should check the actual messages and schema received by a controlled
|
||||
transport, including use of a separate repair profile.
|
||||
4. Use this test to prove that the pinned API can reproduce the initial
|
||||
prepared prefix and use a different output contract. The native renderer
|
||||
uses the captured prompt and supplied template variables, and appends the
|
||||
already-rendered suffix after that definition's messages. Check roles,
|
||||
contents, and ordering in the prepared request. If the test exposes a
|
||||
genuine gap, record the failing case and reproducer for a focused upstream
|
||||
change. Keep rendering and execution with PromptKit.
|
||||
|
||||
## Stage 2 — Add and register the four module repair prompts
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Add the definitions `audita/modules.glossary.repair`,
|
||||
`audita/modules.homophones.repair`, `audita/modules.spoken_word.repair`,
|
||||
and `audita/modules.grammar.repair` beside the corresponding proposals in
|
||||
`internal/prompts/assets/prompts`. Register them through `internal/prompts`
|
||||
and the existing embedded PromptKit underlay. Each built-in module supplies
|
||||
its repair prompt ID through a small `RepairPromptID() string` capability.
|
||||
Retain the existing proposal interface for callers that only generate
|
||||
candidates.
|
||||
2. Use a user message in each shipped repair definition. Set its native output
|
||||
contract to JSON schema using `correction_repair.v1.json` so standard
|
||||
PromptKit inspection can prepare it with its own inputs. Put shared response
|
||||
instructions in `RepairContract` and expose that fragment through the
|
||||
existing shared-variable helpers. Render `RepairRequestJSON` from the
|
||||
current proposal reference, current local segment reference, original and
|
||||
replacement spans, confidence, and ordered findings. For findings, include
|
||||
validator key, reason code, and available explanation. Convey an unavailable
|
||||
assessment as such using safe facts from its decision or operation.
|
||||
3. Make the instruction unambiguous after the earlier correction-set exchange.
|
||||
Identify which proposal and current version it addresses, ask for one
|
||||
revision or withdrawal, and describe the separate proposal and segment
|
||||
references. Keep each module's existing edit scope, literal matching,
|
||||
glossary/protection rules, and span deletion semantics. Treat transcript,
|
||||
earlier model responses, and validator explanations as data in the prompt.
|
||||
4. Cover schema shape, action and reference correspondence, empty-string
|
||||
deletion, and real rendering of all four registered assets. Use the standard
|
||||
inspection path for a replacement definition and a configured message file
|
||||
through the existing source-loading path. The assembled branch test will
|
||||
cover execution with that replacement.
|
||||
|
||||
## Stage 3 — Carry the original exchange through proposal generation and the adapter
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Use `internal/framework/structuredoutput` for internal conversation data
|
||||
shared by the role client and runner. Extend the existing structured request
|
||||
with optional repair replay data. Keep the repair definition as the
|
||||
requested prompt ID in Audita and carry the original proposal ID, frozen
|
||||
effective variables, and original prepared messages in a shared read-only
|
||||
exchange. Retain the full final assistant content used to decode the
|
||||
original correction set, including when PromptKit structurally repaired
|
||||
earlier output. Capture the application-level exchange at the existing
|
||||
prepared execution boundary.
|
||||
2. Pass the exchange from `GenerateCandidates` through
|
||||
`ExecuteModuleProposal` and `contracts.ProposalResult` to the section
|
||||
worker in `runner`. Capture it on a successful proposal response when the
|
||||
effective repair budget is positive, independently of debug settings.
|
||||
Reuse the existing request-local segment mapping and keep each proposal's
|
||||
one-based position in the response aligned with its runner-assigned
|
||||
proposal index. Hold one initial exchange per section and share it among
|
||||
branches. Keep the exchange out of ordinary serialized results and release
|
||||
it when the section's repair work finishes.
|
||||
3. In `internal/framework/llm/adapter.go`, render the current repair definition
|
||||
against the repair profile via the same engine and append its rendered
|
||||
message block to the branch suffix. Form the final request from the
|
||||
captured proposal prompt ID and variables, the original assistant response,
|
||||
prior turns from that branch, and the current instruction. Pass the repair
|
||||
profile, per-call repair schema, timeout, transient retry budget, and
|
||||
structural repair budget through the existing role client. Preserve
|
||||
the original message roles and contents. Use the prepared handle that
|
||||
passed the original-prefix comparison for execution. Treat a preparation
|
||||
invariant failure as an application failure and discard the unused handle.
|
||||
4. Return the rendered repair instruction block and the final response for
|
||||
this turn, so a subsequent attempt appends that exact exchange once. Keep
|
||||
the repair definition ID in debug attribution alongside native prepared
|
||||
metadata for the base prompt. Reuse typed failure and terminal count
|
||||
handling. Retain the existing distinction between caller cancellation,
|
||||
local setup or invariant errors, and provider errors. An upstream API
|
||||
rejection of a repair request follows the recoverable branch path even
|
||||
when PromptKit supplies no usable model output.
|
||||
5. Exercise the role client with the test from Stage 1 using a real registered
|
||||
repair asset and an operator replacement. Verify full history, isolated
|
||||
turns, selected repair backend/profile, and counts for one logical attempt
|
||||
that uses a native retry or structural repair. Verify capture without
|
||||
`--debug` and ordinary proposal behavior with repair disabled.
|
||||
|
||||
## Stage 4 — Resolve configuration and wire the repair role
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. In `internal/core/config`, add `llm.correction_repair_attempts` as a
|
||||
nonnegative integer, default `1`, and `llm.correction_repair_profile` as an
|
||||
optional profile selection. Reuse the stage `llm` block. Omitted stage
|
||||
attempts inherit the effective root value. An explicit stage zero clears
|
||||
the inherited budget. Use ConfigKit's existing scalar validation and
|
||||
presence rules, with no extra maximum for the new attempt count.
|
||||
2. In `resolveSettings`, follow the recorded validation-profile pattern.
|
||||
Resolve the effective stage proposal profile first. An explicit repair
|
||||
profile selects that profile. An omitted stage repair profile inherits
|
||||
an explicit nonnull root repair profile, otherwise it falls back to the
|
||||
stage's effective proposal profile. An explicit stage null clears the root
|
||||
override and uses that same fallback. At root, omission or null falls back
|
||||
to the effective root proposal profile. Preserve recorded origins and
|
||||
detached profile pointers. Use existing YAML, overlays, and native
|
||||
application configuration inspection.
|
||||
3. Extend `CheckReadiness` and the CLI processing wiring in
|
||||
`internal/cli/process_inputs.go` and `process_pipeline.go`. Check the selected
|
||||
repair profile and backend only for stages with a positive budget. The
|
||||
existing captured engine validates registered prompt assets and their
|
||||
overrides during loading. Retain that path for repair prompts. Allocate
|
||||
a repair role client in the module run spec only for enabled stages,
|
||||
using the same runtime, stage execution settings, admission, and optional
|
||||
observation hooks. Unused repair profile IDs require no readiness check
|
||||
when repair is disabled. A normalization-only run has no repair role.
|
||||
4. Test root/stage inheritance including different proposal profiles, null and
|
||||
explicit overrides, zero and positive budgets, native effective inspection,
|
||||
and readiness before output preparation. Use the existing configuration
|
||||
version and presence conventions.
|
||||
|
||||
## Stage 5 — Collect usable findings while preserving approval semantics
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Refactor `validateSectionCandidates` in `internal/framework/runner` so the
|
||||
initial pass and later revisions use the same validation code. Keep the
|
||||
existing shortlist flow when the effective repair budget is zero. In
|
||||
feedback mode, track decisions by proposal index and version. Use
|
||||
the current validator order and batching. Retain approval eligibility only
|
||||
for candidates that satisfy all required checks. Collect additional findings
|
||||
separately for candidates already rejected.
|
||||
2. Make applicability explicit for the built-in checks using the current
|
||||
candidate and the original section.
|
||||
|
||||
- Shape can evaluate malformed candidates. No-effect can evaluate literal
|
||||
equality on a usable span, and confidence can evaluate a valid score
|
||||
independently of target or span placement.
|
||||
- Original-text presence needs a meaningful target and nonempty span.
|
||||
Use the existing replacement preview and `ambiguous_original_text` reason
|
||||
to provide an actionable finding when the span has multiple matches. In
|
||||
feedback mode, record it before withholding the candidate from semantic
|
||||
validation. A shape verdict for an unmapped reference already supplies
|
||||
target feedback.
|
||||
- Evaluate protection where the existing rule has usable before/after
|
||||
evidence. Preserve active glossary match boundaries and spoken-word
|
||||
repetition rules when they require full-segment context. Run the
|
||||
nonempty-result guard when a replacement can be located, including a
|
||||
preview that would leave a blank segment.
|
||||
- Reuse the preview for semantic validators when it supplies usable
|
||||
before/after text. Continue across independent semantic checks after
|
||||
an actionable rejection or unavailable decision when the candidate still
|
||||
supplies that input. Preserve the existing model batching budget.
|
||||
3. Apply internal decision cardinality to the candidates actually submitted to
|
||||
each validator. A skipped check supplies no decision. Keep missing,
|
||||
duplicate, and unusable model decisions as unavailable under existing
|
||||
reconciliation. Retain recoverable operation evidence. Route fatal errors
|
||||
and cancellation through the existing peer cancellation and join path.
|
||||
4. Build feedback from validator decisions and safe optional evidence that the
|
||||
validator owns. Reuse existing reason codes and messages. Supply specific
|
||||
evidence where a reason alone gives too little direction, such as the
|
||||
applicable confidence threshold or a protected spelling when identified.
|
||||
Use normal LLM decision explanations for semantic judgments. Preserve
|
||||
debug redaction and avoid putting raw provider errors into feedback.
|
||||
5. Test two independent meaningful findings after one rejects, a usable
|
||||
rejected candidate that reaches the semantic checks, invalid target and
|
||||
ambiguous span handling, and a no-effect candidate with no repair work.
|
||||
Confirm that an unavailable decision alone remains unavailable and that
|
||||
the zero-budget pass calls later validators with the existing shortlist.
|
||||
|
||||
## Stage 6 — Run independent repair branches and apply accepted revisions
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Extend the section validation worker in `runModulePipeline` after its
|
||||
initial pass. Keep the approved candidates as they are. For each candidate
|
||||
with an actionable rejection, initialize an independent branch on the
|
||||
section exchange. No-effect and unavailable-only outcomes finish without a
|
||||
branch. When an actionable rejection coexists with unavailable validation,
|
||||
use the known findings and show the missing assessment as unavailable.
|
||||
2. Visit repairable candidates in proposal-index order within each section
|
||||
worker. Reuse existing section concurrency and engine admission. Each
|
||||
logical repair call consumes one attempt from that proposal's budget.
|
||||
Track attempts separately from native transient retries, structural
|
||||
repairs, and revalidation calls. Send the full conversation to the selected
|
||||
provider without local size checks or truncation. The validation batching
|
||||
budget still applies to validator calls. End the branch on a provider error
|
||||
after applicable native retries, including context-window rejection. Leave
|
||||
other proposals eligible to continue. Keep the existing fatal/cancellation path for app
|
||||
failures and parent cancellation.
|
||||
3. Accept a valid withdrawal as a terminal correction outcome. For a
|
||||
structurally usable revision with a matching proposal reference, map its
|
||||
segment position using the original section, retaining the original
|
||||
stage, chunk, and proposal index. Allow a changed target or span inside
|
||||
that section. Run the full applicable chain from scratch against the
|
||||
original section. Place a fully approved revision into the original
|
||||
candidate's application position. A still-actionable rejection can take
|
||||
another turn if budget remains. Use only the current version's findings
|
||||
for the next instruction, retaining the earlier turns in that branch's
|
||||
history.
|
||||
4. End a rejected revision that has no further actionable finding with a
|
||||
rejected disposition, and record an actionable rejection at the budget
|
||||
boundary as exhausted. An unavailable-only revalidation, recoverable
|
||||
repair-call failure, or unusable response ends the branch as unavailable.
|
||||
Use the interrupted disposition when fatal failure or cancellation leaves a
|
||||
branch in progress or prevents an approved revision from reaching
|
||||
application. Preserve already-terminal dispositions. A revision needs its
|
||||
own approval, and each response can supply at most one candidate.
|
||||
5. Merge initial approvals and accepted revisions into the existing section
|
||||
application path, keeping ascending proposal indices and all apply-time
|
||||
checks against the text produced by earlier edits. Resolve applied versus
|
||||
skipped status from the actual application result. Keep the stage-level
|
||||
commit barrier. Count filtering once for a defective original that remains
|
||||
excluded, using its original filtering reasons. An approved revision takes
|
||||
that candidate's place in the existing application accounting. Attribute
|
||||
defects in a later repair response to its own operation. Keep one terminal
|
||||
proposal outcome per branch, with superseded versions available as debug
|
||||
evidence.
|
||||
6. Test a response with approved and rejected proposals on the same segment,
|
||||
a successful retargeted revision, a valid span deletion, withdrawal,
|
||||
exhaustion across two attempts, and overlapping edits where a repaired
|
||||
candidate passes validation and then skips at application. Check unrelated
|
||||
approvals survive a recoverable provider failure. Confirm revalidation
|
||||
revisits a validator that approved an earlier version.
|
||||
|
||||
## Stage 7 — Account for operations and publish compact ordinary outcomes
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Add `correction_repair` as an operation kind in
|
||||
`internal/framework/outcomes`. For each repair call retain its proposal
|
||||
identity, attempt number, operation counts, and safe terminal reason.
|
||||
Attribute revalidation calls to the proposal and attempt internally so
|
||||
their local batch indices can repeat safely. Keep the normal proposal
|
||||
and initial validation accounting, including proposal response counts.
|
||||
Extend deterministic report sorting
|
||||
to visit initial proposal and validation evidence first, then repairs by
|
||||
proposal, attempt, and validator/batch order.
|
||||
2. In `processreport`, `reporting`, and the runner's work-unit results,
|
||||
count repair and revalidation executions as logical operations, including
|
||||
failed provider requests. Keep existing accounting for pre-call failures.
|
||||
Use the existing work-unit error paths for recoverable model
|
||||
failures and the run-level path for fatal or canceled work. Keep failures
|
||||
from earlier attempts visible when later work succeeds. A usable revision
|
||||
or withdrawal is a successful model operation. Use existing native retry
|
||||
and structural-repair reasons where applicable. Add `correction_revised`
|
||||
to the repair operation that yields
|
||||
a fully approved revision. This recovery identifies validation success.
|
||||
Actual application is reflected in the final branch disposition and the
|
||||
existing correction count. A clean withdrawal or ordinary rejected revision
|
||||
adds no error or recovery by itself.
|
||||
3. Add a single optional **Correction repair** section to `report.md`, with
|
||||
one row per branch that attempted a repair. Use the same stage and chunk
|
||||
numbering as the matrices, the original internal proposal index, the
|
||||
semantic attempt count, and a final outcome. Use `applied`,
|
||||
`approved_skipped`, `withdrawn`, `rejected`, `exhausted`, `unavailable`,
|
||||
and `interrupted` as applicable. Derive the row from compact work-unit facts
|
||||
after application or from the available interrupted evidence. Sort by stage,
|
||||
chunk, and proposal index. Keep the section focused on these fields.
|
||||
4. Keep the existing three output files, current YAML envelope versions, and
|
||||
existing YAML fields. Document `correction_repair` and
|
||||
`correction_revised` as additions to the operation and reason vocabulary.
|
||||
Keep compact branch rows in the Markdown report. Reporting-disabled runs
|
||||
still compute the existing CLI counts. Keep debug capture independently
|
||||
selectable.
|
||||
5. Verify the report for successful application, approval followed by skip,
|
||||
withdrawal, exhaustion, provider failure, and partial evidence on fatal
|
||||
interruption. Check that semantic attempts, actual edits, and PromptKit
|
||||
retry/repair counts are distinct and that both YAML summaries agree with
|
||||
the CLI on the counts they share.
|
||||
|
||||
## Stage 8 — Retain optional repair evidence and preserve ledger lineage
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Extend the current interaction writers and logical execution observation
|
||||
data with a repair role, original proposal index, and one-based semantic
|
||||
attempt number. Use generated stage instance, section, proposal, and
|
||||
attempt coordinates for repair filenames and include validator and batch
|
||||
coordinates for revalidation. Original stage labels remain data. Reuse the
|
||||
existing diagnostics failure and credential redaction path, including the
|
||||
selected repair role's prepared credential references.
|
||||
2. Save each repair attempt's rendered conversation, response, feedback,
|
||||
and validation evidence through existing debug JSON artifacts. Keep these
|
||||
writes conditional on an active debug directory and keep runtime repair
|
||||
functional when preparation or writing of optional artifacts fails.
|
||||
3. Extend `processreport.BuildCorrectionLedger` to emit one final row per
|
||||
repaired proposal keyed by the original stage instance and proposal
|
||||
index. Represent the original proposed edit and the final applied edit
|
||||
separately, including when the revision retargets the proposal. Use
|
||||
compact optional repair attempt metadata and artifact references to
|
||||
connect versions and their respective decisions. Keep initial proposal
|
||||
rejections as evidence within that lineage. Preserve the existing ledger
|
||||
behavior for untouched candidates and the indexed interrupted join.
|
||||
4. Test several repairs against a single segment and revalidation batches
|
||||
that reuse local indices, then verify unique debug artifact names and
|
||||
unambiguous ledger attribution. Check the terminal row and prior
|
||||
decisions on interruption. Check redaction with synthetic credentials in
|
||||
captured interaction material and verify that ordinary reports contain
|
||||
only safe outcome data.
|
||||
|
||||
## Stage 9 — Validate the assembled command and update canonical documentation
|
||||
|
||||
**Completed.**
|
||||
|
||||
1. Exercise the normal processing command with controlled provider responses,
|
||||
real configuration and prompt loading, and the output session. Cover a
|
||||
mixed proposal set with two independent repair branches, a second turn
|
||||
on one branch, provider context rejection on the other, full revalidation,
|
||||
and preserved original approvals. Inspect the wire order and contract.
|
||||
Run the relevant behavior with debug off and with an operator repair prompt
|
||||
replacement under the usual underlay. Cover zero budget and independent
|
||||
report/debug selection. Reuse lower-level cases for individual failure and
|
||||
disposition variants.
|
||||
2. Add a controlled cancellation case during a repair or its revalidation.
|
||||
Confirm active work joins, no completed transcript is published, and
|
||||
failed reports preserve available partial evidence. Use the existing
|
||||
subprocess fixtures and output assertions where they express the contract.
|
||||
3. Update the owning [configuration](../config.md),
|
||||
[pipeline](../internal/pipeline.md), [modules](../internal/modules.md),
|
||||
[validators](../internal/validators.md), [prompts](../internal/prompts.md),
|
||||
[LLM runtime](../internal/llm-runtime.md), and
|
||||
[output files](../integrations/output-files.md) guides. Explain the default
|
||||
attempt, per-stage control, profile fallback, complete replay to the
|
||||
selected repair provider, provider failure behavior, and compact reporting.
|
||||
Add minimal configuration or preview examples where needed in the existing
|
||||
maintained material. Update [operations](../operations.md) if the new debug
|
||||
evidence needs operator guidance. Keep the deferred decision-recovery
|
||||
entry aligned with the implemented scope.
|
||||
4. Validate examples and local links. Run the full offline suite and the
|
||||
assembled CLI build under the [contributor guide](../development.md),
|
||||
including race detection for the section workers. Check the focused
|
||||
provider and ledger regressions in the normal suite. The sprint ends
|
||||
with the implementation and its evidence.
|
||||
@@ -1,359 +0,0 @@
|
||||
# Correction Repair Review Remediation
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** All nine findings resolved. Findings refer to implementation revision
|
||||
`aa8073c`. See the [implementation plan](implementation.md).
|
||||
|
||||
## Review Result
|
||||
|
||||
The intended processing architecture is present: module-owned repair prompts,
|
||||
validator-owned findings, full original exchanges with independent proposal
|
||||
branches, ConfigKit-resolved budgets and profiles, and PromptKit-owned rendering,
|
||||
execution, retries, structural repair, and backend admission. Revisions undergo
|
||||
fresh validation and enter deterministic application under the existing conflict
|
||||
checks and stage barrier. Ordinary reports summarize branches; debug artifacts
|
||||
retain detailed evidence.
|
||||
|
||||
The review found gaps in outcome attribution, retained diagnostic evidence,
|
||||
replay of configured cache settings, and protected-alias feedback. The
|
||||
algorithmic audit found repeated section scans, quadratic proposal bookkeeping,
|
||||
and debug-only history retained during ordinary runs. These are local
|
||||
implementation costs. The plan deliberately calls for full conversation
|
||||
replay and fresh validation of each revision.
|
||||
|
||||
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
||||
was identified in the reviewed implementation. In particular, the new semantic
|
||||
budget has no additional maximum, zero disables extra feedback and repair,
|
||||
revisions may change their target within the original section, empty replacements
|
||||
and whitespace spans remain supported, and provider request limits are not
|
||||
replaced by a new local context-size restriction.
|
||||
|
||||
## Findings
|
||||
|
||||
### R1 — Preserve Defect Attribution for Revised Proposals (P2)
|
||||
|
||||
**Completed.** Revision defects now appear on the originating repair recovery. Subprocess tests cover exhaustion, later approval, and ordinary rejection.
|
||||
|
||||
**Affected plan:** Stage 6, item 5; Stage 7, items 1–2.
|
||||
|
||||
**Evidence:** In [repairCandidates](../../internal/framework/runner/repair.go),
|
||||
lines 178–191 retain revalidation decisions, rejections, and operations, but drop
|
||||
`validated.filtered`. `RepairAttempt` has no corresponding defect data, and
|
||||
`repairOperations` (lines 230–249) consequently emits no defect attribution for
|
||||
that revision. `mergeFilteredProposals` in
|
||||
[runner.go](../../internal/framework/runner/runner.go) attributes original
|
||||
proposal defects only; it cannot recover the discarded revision evidence.
|
||||
|
||||
**Reproduced behavior:** Begin with one segment, `cat`, and an initially
|
||||
low-confidence proposal to replace it with `Cat`. Supply a structurally valid
|
||||
repair with `proposal_ref: 1`, `action: revise`, and correction
|
||||
`{id: 99, original_text: cat, corrected_text: Cat, confidence: 0.99}`.
|
||||
Revalidation correctly rejects the out-of-section target with
|
||||
`invalid_target_segment_id`, and the branch exhausts its one-attempt budget.
|
||||
However, its sole `correction_repair` operation is recorded as successful with
|
||||
no failure reason, filtered defect, or recovery reason. The defect is visible
|
||||
only in retained validation/debug evidence; ordinary operation reporting cannot
|
||||
attribute it to the repair response. This also loses the defect's operation
|
||||
attribution when a later attempt succeeds.
|
||||
|
||||
**Required change:** Carry the defective revision's evidence into its originating
|
||||
repair attempt and operation, following the plan's distinction between original
|
||||
proposal filtering and defects introduced by a repair response. Preserve the
|
||||
existing treatment of ordinary editorial rejection, low confidence, withdrawal,
|
||||
and application conflicts; those are not automatically operational errors.
|
||||
Keep original proposal-response counts separate and avoid attributing the later
|
||||
defect to the original response or counting it twice.
|
||||
|
||||
**Acceptance:** Add a regression covering a schema-valid revision with an invalid
|
||||
target or missing source span, both at budget exhaustion and before a later
|
||||
successful repair. Verify that the specific defect remains attributable to its
|
||||
repair operation in ordinary reporting, while proposal-response totals, semantic
|
||||
attempt counts, and the final branch disposition remain correct. Include an
|
||||
ordinary rejected revision to protect the existing distinction from response
|
||||
defects.
|
||||
|
||||
### R2 — Retain Application Skip Reasons in Repaired Ledger Entries (P2)
|
||||
|
||||
**Completed.** The debug ledger joins the application skip reason and message by stage and proposal. Ledger and overlap tests pass.
|
||||
|
||||
**Affected plan:** Stage 6, item 5; Stage 8, item 3.
|
||||
|
||||
**Evidence:** [BuildCorrectionLedger](../../internal/framework/processreport/correction_ledger.go)
|
||||
skips every repaired proposal in the `module.SkippedChanges` loop (lines 131–134).
|
||||
Its replacement repair-row construction (lines 194–209) copies the branch
|
||||
disposition and attempt history, but never joins the actual skipped application
|
||||
result. `DispositionReasonCode` and `DispositionMessage` are therefore empty for
|
||||
`approved_skipped` repair rows, although they are retained for unrepaired skipped
|
||||
proposals.
|
||||
|
||||
**Reproduced behavior:** An earlier accepted proposal changes `cat` to `dog`.
|
||||
A later repair validates a revision of that same original span to `fox`, then
|
||||
correctly skips at application because the earlier edit removed its source span.
|
||||
The runner retains the actual `SkippedChange`, but the repaired proposal's debug
|
||||
ledger row retains only `approved_skipped` and its pre-application decisions.
|
||||
A focused ledger probe confirms that a supplied skip reason and message disappear
|
||||
from the serialized entry. Repair interaction artifacts are written before
|
||||
application, so they cannot supply the missing application result.
|
||||
|
||||
**Required change:** Join the final skipped change to the repaired ledger entry
|
||||
using the existing stage-instance/proposal identity, and populate the existing
|
||||
disposition reason and message fields. Preserve one final row per repaired
|
||||
proposal, its original proposal data, and its revision history. The compact
|
||||
ordinary report can retain its existing fields.
|
||||
|
||||
**Acceptance:** Extend the existing overlapping-edit runner/ledger coverage so a
|
||||
validated but skipped revision retains the actual application reason and message
|
||||
in its single final debug row. Verify that it remains `approved_skipped`, has no
|
||||
applied-edit fields, and does not create a duplicate ledger row or an additional
|
||||
operational error.
|
||||
|
||||
### R3 — Preserve Cache Settings When Replaying Repair Turns (P3)
|
||||
|
||||
**Completed.** Replay retains each rendered instruction cache setting. The two-turn adapter test verifies the outbound role, text, and cache metadata.
|
||||
|
||||
**Affected plan:** Stage 3, items 3–5.
|
||||
|
||||
**Evidence:** [The adapter](../../internal/framework/llm/adapter.go#L117) appends
|
||||
the current repair instruction as native PromptKit messages. It retains that
|
||||
instruction for the next attempt through `convertMessages`, which copies only
|
||||
role and content. On the next attempt the adapter reconstructs those prior
|
||||
turns without their `CacheControl`. Operator prompt replacements already
|
||||
support PromptKit's per-message `cache_control` field.
|
||||
|
||||
**Reproduced behavior:** Override the grammar repair prompt with a user message
|
||||
using `cache_control: {type: ephemeral, ttl: 1h}`, then execute two repair
|
||||
turns on the same branch. The first call sends the instruction as a cached text
|
||||
block. The second call sends the same earlier instruction as a plain string,
|
||||
while the new instruction still carries the cache settings. The transport
|
||||
confirmed this difference with a local fixture.
|
||||
|
||||
**Required change:** Preserve native cache metadata for rendered instructions
|
||||
carried into later turns. Keep its conversion in the adapter boundary and leave
|
||||
wire encoding to PromptKit.
|
||||
|
||||
**Acceptance:** Extend the two-turn adapter test with a cache-controlled
|
||||
operator repair prompt. Verify that the previous instruction retains its
|
||||
cache settings, text, and role when replayed. The test should check the
|
||||
outbound request without depending on provider caching behavior.
|
||||
|
||||
### R4 — Reuse Replacement Previews Across Feedback Checks (P3)
|
||||
|
||||
**Completed.** Each source section shares one target index across validation passes. Preview-dependent checks reuse one lazy preview per version. Targeted tests passed, and the offline benchmark showed lower allocation counts at both sizes.
|
||||
|
||||
**Affected plan:** Stage 5, items 1–2.
|
||||
|
||||
**Evidence:** [feedbackCandidates](../../internal/framework/runner/feedback.go#L30)
|
||||
calls `feedbackPreview` for every candidate at every validator, including
|
||||
shape and confidence checks that never use it. Each call scans the section for
|
||||
the target, then builds the replacement preview. The
|
||||
[runner](../../internal/framework/runner/runner.go#L744) previews the candidate
|
||||
again for ambiguity, and [validator request construction](../../internal/framework/validators/llm_validators.go#L269)
|
||||
rebuilds the preview for the LLM payload. The default positive repair budget
|
||||
adds this work even when all proposals pass the initial chain.
|
||||
|
||||
**Complexity:** For a section with `S` segments and `M` candidate versions
|
||||
(initial proposals and revisions), the feedback selector alone can perform
|
||||
`O(V × M × S)` target lookup work across `V` validators. Each singleton
|
||||
revalidation also rebuilds section-wide indexes inside presence, nonempty-result,
|
||||
and semantic validators. With a shared section index and one preview per
|
||||
version that needs it, lookup and candidate traversal take expected
|
||||
`O(S + V × M)` time, plus the actual text and payload work. That removes the
|
||||
quadratic lookup term when proposals grow with section size.
|
||||
|
||||
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
|
||||
proposals exercised the same five deterministic validators at budgets zero
|
||||
and one, with identical approvals and no repairs. On the review machine,
|
||||
validation rose from about 0.28 to 0.50 ms and allocated bytes from about 0.40 to
|
||||
0.68 MB per section. With 1,000 segments and proposals, the same probe rose from
|
||||
about 4.2 to 8.0 ms and from 5.4 to 7.8 MB. These measurements isolate local
|
||||
validation work.
|
||||
|
||||
**Required change:** Reuse the existing replacement preview across applicability,
|
||||
ambiguity, and validation payload construction for a given candidate version
|
||||
where the rules use the same before/after text. Reuse a section target index and
|
||||
compute previews only for checks that need them. Recompute the preview for a
|
||||
revision, and preserve the separate application check against the current text.
|
||||
Pass the same read-only section lookup into singleton revalidation so validators
|
||||
can use the original target and its neighbors without rebuilding their indexes.
|
||||
|
||||
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
|
||||
passing, including ambiguous spans, blank results, invalid confidence, and
|
||||
protection exceptions. Re-run a focused offline benchmark to verify reduced
|
||||
scanning and allocation in the feedback path.
|
||||
|
||||
### R5 — Index Initial Findings and Compact Filtering Once (P3)
|
||||
|
||||
**Completed.** Initial findings are grouped by proposal in validator order. Superseded filtering records are removed once, including on partial returns. Runner and report tests pass.
|
||||
|
||||
**Affected plan:** Stage 6, items 1 and 5.
|
||||
|
||||
**Evidence:** [repairCandidates](../../internal/framework/runner/repair.go#L59)
|
||||
calls `actionable` with the full initial rejection list for every proposal.
|
||||
Each eligible proposal then calls `repairFindings` over the same full list.
|
||||
Every successful revision also walks and compacts the original filtered-defect
|
||||
slice. [Findings extraction](../../internal/framework/runner/feedback.go#L72)
|
||||
already appends in validator order.
|
||||
|
||||
**Complexity:** With `P` original proposals and `D` initial rejection
|
||||
records, branch selection and initial feedback can take `Θ(P × D)` time.
|
||||
If a validation outage leaves one unavailable decision per proposal, this
|
||||
already takes quadratic time before the runner determines no repairs can start.
|
||||
With `Q` approved repairs and `F` original filtering records, repeated
|
||||
compaction can add `Θ(Q × F)` work in the worst case. Both products can be
|
||||
quadratic in proposal count even with a fixed validator chain.
|
||||
|
||||
**Refactoring:** Group initial rejections by proposal index in a single pass.
|
||||
Walk the proposals in their existing order and read only each proposal's
|
||||
findings. Record the approved replacements in a set and filter the original
|
||||
defect list once when returning the section result, including partial results
|
||||
on error. The combined bookkeeping then takes expected `O(P + D + F)` time
|
||||
apart from the individual repair calls and revalidations.
|
||||
|
||||
**Acceptance:** Preserve validator order within findings, proposal order
|
||||
across branches, no-effect and unavailable-only handling, and original
|
||||
filtering evidence for candidates that remain excluded.
|
||||
|
||||
### R6 — Index Application Results for Repair Outcomes (P3)
|
||||
|
||||
**Completed.** Application outcomes use a stage-local index built once. The runner test covers an original approval, a skipped revision, and an applied revision in the same stage.
|
||||
|
||||
**Affected plan:** Stage 6, item 5, and Stage 7, item 3.
|
||||
|
||||
**Evidence:** After applying the stage, [Runner.Run](../../internal/framework/runner/runner.go#L244)
|
||||
scans the stage's entire applied-change slice for each approved repair. A
|
||||
repair that was skipped scans the whole slice.
|
||||
|
||||
**Complexity:** With `B` repair branches, `Q` approved repairs, and `A`
|
||||
applied changes in the stage, outcome classification costs
|
||||
`O(B + Q × A)` beyond walking the work units. It can be quadratic even when
|
||||
every approved repair is applied and the matches occur at successive positions.
|
||||
|
||||
**Refactoring:** Index the application results once by the existing stage-local
|
||||
proposal index, then look up each approved repair. This takes expected
|
||||
`O(A + B)` time and `O(A)` auxiliary space for an applied-index set. An
|
||||
index carrying skip details could also support the evidence join in R2.
|
||||
|
||||
**Acceptance:** Keep disposition based on actual application. Cover a stage
|
||||
with an ordinary approval, an applied revision, and an approved revision
|
||||
skipped by the existing conflict checks.
|
||||
|
||||
### R7 — Keep Detailed Repair History in Debug Runs (P3)
|
||||
|
||||
**Completed.** Ordinary runs retain a compact attempt count and disposition. Debug runs retain the revision history. Runner, reporting, and CLI tests pass with debug on and off.
|
||||
|
||||
**Affected plan:** Stage 7, item 3, and Stage 8, items 2–3.
|
||||
|
||||
**Evidence:** [repairCandidates](../../internal/framework/runner/repair.go#L178)
|
||||
stores the revision, decisions, rejections, and operation slice on every
|
||||
attempt, even when debug capture is disabled. The run retains those attempts in
|
||||
its [work units](../../internal/framework/runner/runner.go#L597). The
|
||||
[ordinary report](../../internal/framework/processreport/report.go#L126)
|
||||
needs a branch's proposal index, attempt count, and disposition. It counts
|
||||
executions from the work unit's separate operation list. The debug ledger reads
|
||||
the revision, decisions, and artifact references. Rejections are read during
|
||||
the interaction write, and the saved attempt-level operation slice has no
|
||||
consumer.
|
||||
|
||||
**Complexity:** For `K` attempts, the runner retains `O(K + H)` additional
|
||||
history across the run, where `H` is the size of the retained revision text and
|
||||
validator records. With debug off, that history can be reduced to
|
||||
`O(B)` branch summaries for `B` branches after ordinary operation facts
|
||||
have been emitted. Those operation facts still scale with the number of calls.
|
||||
The conversation and findings needed for the next attempt remain branch-local
|
||||
working state.
|
||||
|
||||
**Refactoring:** Keep compact outcome fields separate from optional ledger
|
||||
history. Retain detailed records for the ledger when debug capture is active.
|
||||
Use rejections locally for feedback and artifact writes and keep one copy of
|
||||
operation facts for ordinary accounting. Pass the final revision through the
|
||||
existing approved-candidate path for application.
|
||||
|
||||
**Acceptance:** Preserve counts and compact outcomes with debug off, including
|
||||
when ordinary reporting is disabled. Debug runs retain per-attempt decisions
|
||||
and available artifact paths on success and interruption.
|
||||
|
||||
### R8 — Select a Protected-Spelling Finding in One Pass (P3)
|
||||
|
||||
**Completed.** Protection checks select a deterministic spelling in one pass without sorting. Validator and runner tests pass.
|
||||
|
||||
**Affected plan:** Stage 5, item 4.
|
||||
|
||||
**Evidence:** The [protection checks](../../internal/framework/validators/protected_terms.go#L98)
|
||||
collect every violating spelling into a slice, sort it, and read only the first
|
||||
value. This code runs during validation and through the application-time
|
||||
protection guard.
|
||||
|
||||
**Complexity:** For `K` reported violations, selection uses up to
|
||||
`O(K log K)` string comparisons and `O(K)` extra storage. A running
|
||||
lexical minimum needs `O(K)` comparisons and constant auxiliary space.
|
||||
The matching and identity checks still do their existing work.
|
||||
|
||||
**Refactoring:** Select the same deterministic representative as violations
|
||||
are detected, using the evidence corrected in R9. Preserve the precedence
|
||||
of identity loss over capitalization and the existing rejection rules.
|
||||
|
||||
### R9 — Name the Affected Protected Alias (P3)
|
||||
|
||||
**Completed.** Protection feedback identifies a spelling whose occurrence count fell. The alias regression and existing spoken-word protection tests pass.
|
||||
|
||||
**Affected plan:** Stage 5, item 4.
|
||||
|
||||
**Evidence:** When an identity loses a mention, the
|
||||
[identity-preservation check](../../internal/framework/validators/protected_terms.go#L98)
|
||||
uses the first prior occurrence's spelling for its finding. An identity can
|
||||
have multiple protected aliases. The first spelling can still be present
|
||||
after the edit.
|
||||
|
||||
**Reproduced behavior:** Protect both `Mira` and `Captain Mira` under one
|
||||
identity. In a glossary stage, replace `Captain Mira` with `the captain` in
|
||||
`Mira spoke to Captain Mira`. The validator correctly rejects the edit but
|
||||
says it may alter `Mira`, the spelling that stayed. The same misleading
|
||||
message becomes repair feedback.
|
||||
|
||||
**Refactoring:** Derive a specific spelling from before/after occurrence
|
||||
counts for the identity that lost a mention, or describe the identity loss
|
||||
when a spelling cannot be identified reliably. Counting spelling deficits
|
||||
can be done in expected linear time in matched occurrences. Preserve the
|
||||
existing identity-level protection decision and spoken-word repetition
|
||||
exception.
|
||||
|
||||
**Acceptance:** Exercise two aliases of one identity. The finding identifies
|
||||
the removed alias or the affected identity without attributing the edit to
|
||||
the unchanged spelling.
|
||||
|
||||
## Stage Review Coverage
|
||||
|
||||
| Stage | Evidence reviewed | Result |
|
||||
| --- | --- | --- |
|
||||
| 1 — Response and native composition | Repair schema, typed decode, correspondence, native multi-turn test | Response processing scales with response size. |
|
||||
| 2 — Module prompts | Four registered definitions, shared contract, rendering and operator replacements | Per-request feedback serialization scales with supplied content. |
|
||||
| 3 — Exchange and adapter | Shared section exchange, prepared prefix, branch history, native failure handling | R3. Cost grows with the sum of the full requests sent, as specified in the plan. |
|
||||
| 4 — Configuration and readiness | Root/stage resolution, zero and positive budgets, profile readiness, role creation | A fixed number of role selections per stage. Readiness deduplicates profile inspections. |
|
||||
| 5 — Findings | Applicability, previews, cardinality, protection, semantic input and batching | R4, R8, R9. Decision cardinality uses indexed checks. |
|
||||
| 6 — Branch execution and application | Eligibility, versioned revalidation, defect attribution, application outcomes and cancellation | R1, R4–R6. Application evidence feeds R2. |
|
||||
| 7 — Ordinary outcomes | Operation aggregation and sorting, report-disabled summaries, compact branch rows | R1. New presentation work scans and sorts the recorded facts. |
|
||||
| 8 — Debug and ledger | Attempt retention, optional artifacts, redaction, indexed lineage and interrupted joins | R2, R7. The ledger joins decisions and branches by identity. |
|
||||
| 9 — Assembled command and docs | Controlled-provider subprocess flows, cancellation, output selection, documentation and preview | Existing tests cover the principal execution paths. The recorded findings identify the gaps. |
|
||||
|
||||
## Validation Performed
|
||||
|
||||
- The fresh `go test -count=1 ./...` run passed.
|
||||
- Fresh race runs passed for the runner, LLM adapter, process reporting, and CLI.
|
||||
- The CLI build passed.
|
||||
- The repair-preview command in [the example guide](../../examples/README.md)
|
||||
successfully rendered the grammar repair instruction and schema without a
|
||||
provider call. Maintained-example integration tests also passed in the full
|
||||
suite.
|
||||
- The implementation documentation passed its link review. The follow-up
|
||||
checked the links in this roadmap and ran `git diff --check`.
|
||||
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
|
||||
validation overhead in R4 against the committed implementation.
|
||||
- The algorithmic audit derived scaling from the current source and its
|
||||
callers. A temporary Go overlay reproduced the wrong-alias finding in R9.
|
||||
- Fresh tests passed for runner, validators, process reporting, LLM adapter,
|
||||
preflight, response schemas, proposal generation, configuration, and prompts
|
||||
during the algorithmic audit.
|
||||
- After remediation, the fresh full test suite, targeted race checks for the
|
||||
runner, validators, process reporting, LLM adapter, and CLI, and the CLI build
|
||||
passed.
|
||||
Reference in New Issue
Block a user