Record correction repair scaling findings
This commit is contained in:
@@ -2,10 +2,9 @@
|
|||||||
|
|
||||||
**Updated:** 2026-11-11
|
**Updated:** 2026-11-11
|
||||||
|
|
||||||
**Status:** Four open findings from the stage-by-stage review of the
|
**Status:** Nine open findings against implementation revision `aa8073c`. The
|
||||||
[implementation plan](implementation.md), against implementation revision
|
algorithmic audit expanded R4 and added R5–R9. See the
|
||||||
`aa8073c`. R1 and R2 were confirmed in the follow-up audit, which added R3 and
|
[implementation plan](implementation.md).
|
||||||
R4.
|
|
||||||
|
|
||||||
## Review Result
|
## Review Result
|
||||||
|
|
||||||
@@ -18,9 +17,11 @@ checks and stage barrier. Ordinary reports summarize branches; debug artifacts
|
|||||||
retain detailed evidence.
|
retain detailed evidence.
|
||||||
|
|
||||||
The review found gaps in outcome attribution, retained diagnostic evidence,
|
The review found gaps in outcome attribution, retained diagnostic evidence,
|
||||||
and replay of configured cache settings. The default validation path also
|
replay of configured cache settings, and protected-alias feedback. The
|
||||||
repeats replacement previews across validators. The reproduced cases preserve
|
algorithmic audit found repeated section scans, quadratic proposal bookkeeping,
|
||||||
correction acceptance and application.
|
and debug-only history retained during ordinary runs. These are local
|
||||||
|
implementation costs. The plan deliberately calls for full conversation
|
||||||
|
replay and fresh validation of each revision.
|
||||||
|
|
||||||
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
No additional strengthening of invariants, contracts, prohibitions, or guarantees
|
||||||
was identified in the reviewed implementation. In particular, the new semantic
|
was identified in the reviewed implementation. In particular, the new semantic
|
||||||
@@ -144,6 +145,15 @@ again for ambiguity, and [validator request construction](../../internal/framewo
|
|||||||
rebuilds the preview for the LLM payload. The default positive repair budget
|
rebuilds the preview for the LLM payload. The default positive repair budget
|
||||||
adds this work even when all proposals pass the initial chain.
|
adds this work even when all proposals pass the initial chain.
|
||||||
|
|
||||||
|
**Complexity:** For a section with `S` segments and `M` candidate versions
|
||||||
|
(initial proposals and revisions), the feedback selector alone can perform
|
||||||
|
`O(V × M × S)` target lookup work across `V` validators. Each singleton
|
||||||
|
revalidation also rebuilds section-wide indexes inside presence, nonempty-result,
|
||||||
|
and semantic validators. With a shared section index and one preview per
|
||||||
|
version that needs it, lookup and candidate traversal take expected
|
||||||
|
`O(S + V × M)` time, plus the actual text and payload work. That removes the
|
||||||
|
quadratic lookup term when proposals grow with section size.
|
||||||
|
|
||||||
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
|
**Measured behavior:** An offline benchmark with 100 segments and 100 valid
|
||||||
proposals exercised the same five deterministic validators at budgets zero
|
proposals exercised the same five deterministic validators at budgets zero
|
||||||
and one, with identical approvals and no repairs. On the review machine,
|
and one, with identical approvals and no repairs. On the review machine,
|
||||||
@@ -157,25 +167,157 @@ ambiguity, and validation payload construction for a given candidate version
|
|||||||
where the rules use the same before/after text. Reuse a section target index and
|
where the rules use the same before/after text. Reuse a section target index and
|
||||||
compute previews only for checks that need them. Recompute the preview for a
|
compute previews only for checks that need them. Recompute the preview for a
|
||||||
revision, and preserve the separate application check against the current text.
|
revision, and preserve the separate application check against the current text.
|
||||||
|
Pass the same read-only section lookup into singleton revalidation so validators
|
||||||
|
can use the original target and its neighbors without rebuilding their indexes.
|
||||||
|
|
||||||
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
|
**Acceptance:** Keep the existing independent-feedback and zero-budget coverage
|
||||||
passing, including ambiguous spans, blank results, invalid confidence, and
|
passing, including ambiguous spans, blank results, invalid confidence, and
|
||||||
protection exceptions. Re-run a focused offline benchmark to verify reduced
|
protection exceptions. Re-run a focused offline benchmark to verify reduced
|
||||||
scanning and allocation in the feedback path.
|
scanning and allocation in the feedback path.
|
||||||
|
|
||||||
|
### R5 — Index Initial Findings and Compact Filtering Once (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 6, items 1 and 5.
|
||||||
|
|
||||||
|
**Evidence:** [repairCandidates](../../internal/framework/runner/repair.go#L59)
|
||||||
|
calls `actionable` with the full initial rejection list for every proposal.
|
||||||
|
Each eligible proposal then calls `repairFindings` over the same full list.
|
||||||
|
Every successful revision also walks and compacts the original filtered-defect
|
||||||
|
slice. [Findings extraction](../../internal/framework/runner/feedback.go#L72)
|
||||||
|
already appends in validator order.
|
||||||
|
|
||||||
|
**Complexity:** With `P` original proposals and `D` initial rejection
|
||||||
|
records, branch selection and initial feedback can take `Θ(P × D)` time.
|
||||||
|
If a validation outage leaves one unavailable decision per proposal, this
|
||||||
|
already takes quadratic time before the runner determines no repairs can start.
|
||||||
|
With `Q` approved repairs and `F` original filtering records, repeated
|
||||||
|
compaction can add `Θ(Q × F)` work in the worst case. Both products can be
|
||||||
|
quadratic in proposal count even with a fixed validator chain.
|
||||||
|
|
||||||
|
**Refactoring:** Group initial rejections by proposal index in a single pass.
|
||||||
|
Walk the proposals in their existing order and read only each proposal's
|
||||||
|
findings. Record the approved replacements in a set and filter the original
|
||||||
|
defect list once when returning the section result, including partial results
|
||||||
|
on error. The combined bookkeeping then takes expected `O(P + D + F)` time
|
||||||
|
apart from the individual repair calls and revalidations.
|
||||||
|
|
||||||
|
**Acceptance:** Preserve validator order within findings, proposal order
|
||||||
|
across branches, no-effect and unavailable-only handling, and original
|
||||||
|
filtering evidence for candidates that remain excluded.
|
||||||
|
|
||||||
|
### R6 — Index Application Results for Repair Outcomes (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 6, item 5, and Stage 7, item 3.
|
||||||
|
|
||||||
|
**Evidence:** After applying the stage, [Runner.Run](../../internal/framework/runner/runner.go#L244)
|
||||||
|
scans the stage's entire applied-change slice for each approved repair. A
|
||||||
|
repair that was skipped scans the whole slice.
|
||||||
|
|
||||||
|
**Complexity:** With `B` repair branches, `Q` approved repairs, and `A`
|
||||||
|
applied changes in the stage, outcome classification costs
|
||||||
|
`O(B + Q × A)` beyond walking the work units. It can be quadratic even when
|
||||||
|
every approved repair is applied and the matches occur at successive positions.
|
||||||
|
|
||||||
|
**Refactoring:** Index the application results once by the existing stage-local
|
||||||
|
proposal index, then look up each approved repair. This takes expected
|
||||||
|
`O(A + B)` time and `O(A)` auxiliary space for an applied-index set. An
|
||||||
|
index carrying skip details could also support the evidence join in R2.
|
||||||
|
|
||||||
|
**Acceptance:** Keep disposition based on actual application. Cover a stage
|
||||||
|
with an ordinary approval, an applied revision, and an approved revision
|
||||||
|
skipped by the existing conflict checks.
|
||||||
|
|
||||||
|
### R7 — Keep Detailed Repair History in Debug Runs (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 7, item 3, and Stage 8, items 2–3.
|
||||||
|
|
||||||
|
**Evidence:** [repairCandidates](../../internal/framework/runner/repair.go#L178)
|
||||||
|
stores the revision, decisions, rejections, and operation slice on every
|
||||||
|
attempt, even when debug capture is disabled. The run retains those attempts in
|
||||||
|
its [work units](../../internal/framework/runner/runner.go#L597). The
|
||||||
|
[ordinary report](../../internal/framework/processreport/report.go#L126)
|
||||||
|
needs a branch's proposal index, attempt count, and disposition. It counts
|
||||||
|
executions from the work unit's separate operation list. The debug ledger reads
|
||||||
|
the revision, decisions, and artifact references. Rejections are read during
|
||||||
|
the interaction write, and the saved attempt-level operation slice has no
|
||||||
|
consumer.
|
||||||
|
|
||||||
|
**Complexity:** For `K` attempts, the runner retains `O(K + H)` additional
|
||||||
|
history across the run, where `H` is the size of the retained revision text and
|
||||||
|
validator records. With debug off, that history can be reduced to
|
||||||
|
`O(B)` branch summaries for `B` branches after ordinary operation facts
|
||||||
|
have been emitted. Those operation facts still scale with the number of calls.
|
||||||
|
The conversation and findings needed for the next attempt remain branch-local
|
||||||
|
working state.
|
||||||
|
|
||||||
|
**Refactoring:** Keep compact outcome fields separate from optional ledger
|
||||||
|
history. Retain detailed records for the ledger when debug capture is active.
|
||||||
|
Use rejections locally for feedback and artifact writes and keep one copy of
|
||||||
|
operation facts for ordinary accounting. Pass the final revision through the
|
||||||
|
existing approved-candidate path for application.
|
||||||
|
|
||||||
|
**Acceptance:** Preserve counts and compact outcomes with debug off, including
|
||||||
|
when ordinary reporting is disabled. Debug runs retain per-attempt decisions
|
||||||
|
and available artifact paths on success and interruption.
|
||||||
|
|
||||||
|
### R8 — Select a Protected-Spelling Finding in One Pass (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 5, item 4.
|
||||||
|
|
||||||
|
**Evidence:** The [protection checks](../../internal/framework/validators/protected_terms.go#L98)
|
||||||
|
collect every violating spelling into a slice, sort it, and read only the first
|
||||||
|
value. This code runs during validation and through the application-time
|
||||||
|
protection guard.
|
||||||
|
|
||||||
|
**Complexity:** For `K` reported violations, selection uses up to
|
||||||
|
`O(K log K)` string comparisons and `O(K)` extra storage. A running
|
||||||
|
lexical minimum needs `O(K)` comparisons and constant auxiliary space.
|
||||||
|
The matching and identity checks still do their existing work.
|
||||||
|
|
||||||
|
**Refactoring:** Select the same deterministic representative as violations
|
||||||
|
are detected, using the evidence corrected in R9. Preserve the precedence
|
||||||
|
of identity loss over capitalization and the existing rejection rules.
|
||||||
|
|
||||||
|
### R9 — Name the Affected Protected Alias (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 5, item 4.
|
||||||
|
|
||||||
|
**Evidence:** When an identity loses a mention, the
|
||||||
|
[identity-preservation check](../../internal/framework/validators/protected_terms.go#L98)
|
||||||
|
uses the first prior occurrence's spelling for its finding. An identity can
|
||||||
|
have multiple protected aliases. The first spelling can still be present
|
||||||
|
after the edit.
|
||||||
|
|
||||||
|
**Reproduced behavior:** Protect both `Mira` and `Captain Mira` under one
|
||||||
|
identity. In a glossary stage, replace `Captain Mira` with `the captain` in
|
||||||
|
`Mira spoke to Captain Mira`. The validator correctly rejects the edit but
|
||||||
|
says it may alter `Mira`, the spelling that stayed. The same misleading
|
||||||
|
message becomes repair feedback.
|
||||||
|
|
||||||
|
**Refactoring:** Derive a specific spelling from before/after occurrence
|
||||||
|
counts for the identity that lost a mention, or describe the identity loss
|
||||||
|
when a spelling cannot be identified reliably. Counting spelling deficits
|
||||||
|
can be done in expected linear time in matched occurrences. Preserve the
|
||||||
|
existing identity-level protection decision and spoken-word repetition
|
||||||
|
exception.
|
||||||
|
|
||||||
|
**Acceptance:** Exercise two aliases of one identity. The finding identifies
|
||||||
|
the removed alias or the affected identity without attributing the edit to
|
||||||
|
the unchanged spelling.
|
||||||
|
|
||||||
## Stage Review Coverage
|
## Stage Review Coverage
|
||||||
|
|
||||||
| Stage | Evidence reviewed | Result |
|
| Stage | Evidence reviewed | Result |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 1 — Response and native composition | Repair schema, correspondence decoder, native multi-turn wire/schema test with a separate repair profile | No additional finding |
|
| 1 — Response and native composition | Repair schema, typed decode, correspondence, native multi-turn test | Response processing scales with response size. |
|
||||||
| 2 — Module prompts | Four registered assets, shared contract, rendering tests, operator replacement and message-file loading | No additional finding |
|
| 2 — Module prompts | Four registered definitions, shared contract, rendering and operator replacements | Per-request feedback serialization scales with supplied content. |
|
||||||
| 3 — Exchange and adapter | Capture through proposal contracts, frozen variables, prefix comparison, independent suffixes, cache control, failure classification, native structural-repair accounting | R3 |
|
| 3 — Exchange and adapter | Shared section exchange, prepared prefix, branch history, native failure handling | R3. Cost grows with the sum of the full requests sent, as specified in the plan. |
|
||||||
| 4 — Configuration and readiness | Root/stage resolution, null fallback, zero and positive budgets, recorded resolution, disabled-profile readiness, CLI role wiring | No additional finding |
|
| 4 — Configuration and readiness | Root/stage resolution, zero and positive budgets, profile readiness, role creation | A fixed number of role selections per stage. Readiness deduplicates profile inspections. |
|
||||||
| 5 — Findings | Applicability selection, preview reuse, independent deterministic/semantic findings, unavailable assessments, no-effect handling, zero-budget shortlist, cardinality and protection checks | R4 |
|
| 5 — Findings | Applicability, previews, cardinality, protection, semantic input and batching | R4, R8, R9. Decision cardinality uses indexed checks. |
|
||||||
| 6 — Branch execution and application | Per-proposal budget, retargeting, deletion, withdrawal, exhaustion, cancellation, full revalidation, deterministic application and stage commit | R1; application evidence feeds R2 |
|
| 6 — Branch execution and application | Eligibility, versioned revalidation, defect attribution, application outcomes and cancellation | R1, R4–R6. Application evidence feeds R2. |
|
||||||
| 7 — Ordinary outcomes | Operation ordering, logical/native counts, compact branch rows, report-disabled summaries, error/recovery mapping | R1 |
|
| 7 — Ordinary outcomes | Operation aggregation and sorting, report-disabled summaries, compact branch rows | R1. New presentation work scans and sorts the recorded facts. |
|
||||||
| 8 — Debug and ledger | Artifact coordinates, redaction, optional-write failures, original/revised identities, interrupted evidence, application-result join | R2 |
|
| 8 — Debug and ledger | Attempt retention, optional artifacts, redaction, indexed lineage and interrupted joins | R2, R7. The ledger joins decisions and branches by identity. |
|
||||||
| 9 — Assembled command and docs | Controlled-provider subprocess flows, prompt override, cancellation, output selections, canonical documentation and preview example | Existing checks pass |
|
| 9 — Assembled command and docs | Controlled-provider subprocess flows, cancellation, output selection, documentation and preview | Existing tests cover the principal execution paths. The recorded findings identify the gaps. |
|
||||||
|
|
||||||
## Validation Performed
|
## Validation Performed
|
||||||
|
|
||||||
@@ -190,6 +332,11 @@ scanning and allocation in the feedback path.
|
|||||||
checked the links in this roadmap and ran `git diff --check`.
|
checked the links in this roadmap and ran `git diff --check`.
|
||||||
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
|
- Temporary Go overlays reproduced the defects in R1–R3 and measured the
|
||||||
validation overhead in R4 against the committed implementation.
|
validation overhead in R4 against the committed implementation.
|
||||||
|
- The algorithmic audit derived scaling from the current source and its
|
||||||
|
callers. A temporary Go overlay reproduced the wrong-alias finding in R9.
|
||||||
|
- Fresh tests passed for runner, validators, process reporting, LLM adapter,
|
||||||
|
preflight, response schemas, proposal generation, configuration, and prompts
|
||||||
|
during the algorithmic audit.
|
||||||
|
|
||||||
Resolve the findings with focused checks at their owning boundaries. Retain the
|
Resolve the findings with focused checks at their owning boundaries. Retain the
|
||||||
current architecture and accepted operator choices.
|
current architecture and accepted operator choices.
|
||||||
|
|||||||
Reference in New Issue
Block a user