From f99fd993b4e028823d4d0314d4de18fff2fbb14d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 17 Nov 2026 03:49:35 +0000 Subject: [PATCH] Record algorithmic reliability audit findings --- docs/roadmap/remediation.md | 199 +++++++++++++++++++++++++++++++++++- 1 file changed, 196 insertions(+), 3 deletions(-) diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 159d330..bb5ee86 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -2,10 +2,10 @@ **Reviewed revision:** `e8c0708` -**Status:** Eight open findings from the review of all 20 +**Status:** Fourteen open findings from review of the 20 [implementation stages](implementation.md) against the -[target roadmap](reliability-and-maintainability.md) and the follow-up audit -of how those stages interact with existing callers and contracts. +[target roadmap](reliability-and-maintainability.md), including follow-up +audits of component interactions and algorithmic costs. ## Review Result @@ -34,6 +34,16 @@ validator chains, model execution, publication, and report and debug readers. Additional probes reproduced the case-variant input behavior in R4 and the cross-component failures in R6–R8. +The algorithmic audit kept the selected stage window, full repair history, +and chunk-partitioning algorithm. The normalization merge builds text and +categories incrementally after sorting, and the explicit partitioner advances +through the weights in linear passes. The remaining opportunities are +primarily in repeated validation and glossary preparation, queued debug +snapshots, and evidence retained after it has served its purpose. R9 was +reproduced through the PromptKit role adapter with a synthetic provider. +R10–R14 follow from source analysis. Focused offline tests passed for +normalization, chunking, preflight, validators, the runner, and reports. + ## Findings ### R1 — Include Later Delivery Failures in Writable Reports (P2) @@ -244,3 +254,186 @@ protection when a located edit produces an empty preview. Verify consistent local attribution, no unsupported protection approval, and no model validation call for the inapplicable candidate. Keep repair available in the positive-budget case. + +### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2) + +**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9 +already handles the corresponding validator references. + +**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20) +and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9) +specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21) +and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24) +use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184) +classifies a conversion error after schema validation as recoverable invalid +output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123) +then returns an operation failure with no candidates. + +**Reproduced behavior:** A synthetic provider returned two proposals with +references `1` and `2.0`. PromptKit accepted the response in one request with +zero structural repairs. Audita returned `invalid_output` at typed decode. +The control case with `1` and `2` decoded both proposals. The repair decoder +has the same conversion for `proposal_ref` and the revision's `id`. + +**Required change:** Decode wire references as exact numeric tokens and +reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9) +before local position or correspondence checks. Preserve the distinction +between segment and proposal references, existing out-of-range handling, and +the original response in debug capture. + +**Acceptance:** Exercise the actual structured paths with an integer and an +equivalent decimal or exponent form. A valid sibling survives proposal +decoding, and the repair revision reaches the same revalidation as its integer +form. Retain existing handling for a mismatched repair reference. + +### R10 — Fit Required Validation Data Before Collecting Optional Context (P3) + +**Affected plan:** Stages 8 and 10. + +**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251) +builds optional context for every previewable candidate before +[fitting the payload](../../internal/framework/validators/llm_budget.go#L50). +[Context construction](../../internal/framework/validators/llm_context.go#L39) +rescans the original and adjacent segments, finds matches in the proposed +text, sorts matched entry positions, and builds the entry slice. The fitter +can discard all optional context while retaining the original slice's +backing array on a fitted item. + +**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B` +bytes of shared source and neighboring text, the unchanged text is scanned +`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate +add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work +happens even when required fields consume the budget. Distinct proposed texts +still need their own match evidence when optional context is used. + +**Recommended change:** Fit the required fields before collecting optional +context for a candidate. Reuse identity membership for unchanged text within +the section and combine it with the corrected-text matches only when needed. +Keep existing glossary and neighbor order, exact serialized budget accounting, +and oversized-input behavior. Drop unused optional storage before dispatch. + +**Acceptance:** Compare the emitted payload to the existing serialized-budget +oracle for usable and oversized candidates. A controlled no-room case should +skip optional matching. Sibling candidates should share unchanged-text lookup +while still receiving definitions specific to their revisions. + +### R11 — Reuse Glossary Selections Before Building Their Views (P3) + +**Affected plan:** Stage 11, item 2. + +**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115) +clones selected glossary entries and aliases and derives protection terms for +every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152) +can reuse prompt variables and validator views only after all those copies +have been built. The retained specs still hold separate entry and term slices +for identical selections. + +**Cost:** With `S` stages sharing a selection of `G` entries and spelling +contributions, materializing and retaining those selection data costs +`Θ(S × G)`. Sharing immutable selection data brings this portion to +`Θ(G + S)` for repeated fixed-length selection keys. + +**Recommended change:** Resolve and prepare the shared glossary data on the +first occurrence of an ordered context/protection selection and exemption. +Reuse the complete read-only selection in later specs. Keep stage identity, +first-failure attribution, and copy-on-mutation at consumer boundaries. + +**Acceptance:** Exercise repeated selections, a reordered selection, and a +glossary-stage exemption. Reused selections should supply the same data to +proposal, validation, and application as the existing independent views. + +### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3) + +**Affected plan:** Stages 3, 14, 16, and 18. + +**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151) +converts every turn's full rendered message list to a new slice. Each +[repair builder](../../internal/framework/runner/repair.go#L194) captures that +snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22) +holds the builders until ordered proposal indices are assigned, including +during earlier sections' model work. + +**Cost:** A branch with `H` starting messages and `T` successful turns that +add a constant number of messages per turn can retain `Θ(T × H + T²)` message +descriptors at once. Content strings may share backing storage. The window +bounds the number of sections held, and a single section can still contain +many turns and branches. Full-history requests and self-contained debug files +require repeated serialization by design. They can be serialized one at a +time without keeping every prepared list live. + +**Recommended change:** Keep compact evidence pending ordered identity +assignment. For example, stage the request payload under section-local +coordinates while it is live and use its final name when indices are assigned. +An in-memory approach can share immutable history prefixes and expand each +snapshot at write time if it can reproduce the exact rendered messages. +Retain full model context, nonfatal capture, and self-contained final files. + +**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section. +Compare its decoded request bodies, final artifact names, and ledger paths +with the current representation, including a capture failure followed by a +successful write. + +### R13 — Retain Only Operations Used by Ordinary Reports (P3) + +**Affected plan:** Stages 12, 17, and 18. + +**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632) +counts each work unit and retains all its operations when reports are enabled. +It also [stores every candidate index](../../internal/framework/runner/runner.go#L548) +on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79) +uses only recoverable failures and successful recoveries. Failure projection +uses affected indices and typed facts. Recovery projection uses counts and +filtered evidence. Neither uses the cause stored on each operation, which is +needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22). + +**Cost:** The retained operation records, unused successful proposal index +lists, and causes account for `Θ(M + P + B)` storage, for `M` operation +records, `P` successful proposal indices, and `B` bytes of retained causes. +PromptKit's non-success HTTP error can include a bounded +provider diagnostic in each cause. Even an all-success run with no recoveries +keeps and later sorts operation rows that produce no error or recovery entry. +The report matrix itself still needs a row per work unit. + +**Recommended change:** After counting and resolving application filtering and +failure attribution, retain compact facts only for operations that feed the +reports. Keep typed initiating and peer causes where they belong on execution +failures. Debug execution observations and correction-ledger inputs have +their own consumers. Avoid retaining proposal index lists on operations that +never project those lists. + +**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals, +and debug association across existing evidence modes. Include clean successes, +both initial and repair filtering, and an interrupted run with available origin. + +### R14 — Reuse Original Applicability When Attributing Application Skips (P3) + +**Affected plan:** Stages 8 and 17, item 3. + +**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59) +checks each proposal against evolving segment text. On a defective-looking +skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953) +previews the proposal again against the original section to distinguish a +stale conflict from a defective response. The built-in validation path has +already determined the latter preview's applicability. Its result is discarded +before the attribution pass. + +**Cost:** For `K` conflicting proposals against one original segment of `B` +bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of +original-text previews, even if application quickly rejected those proposals +against the changed text. This is separate from the checks required against +current text. When a validator chain can approve originally inapplicable +revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46) +can also scan `O(F × M)` operations for `F` such filtered revisions and `M` +operations in the same unit. Built-in validators normally reject those revisions +before application. + +**Recommended change:** Carry a compact original-applicability fact for +approved candidates when the validator already computed it. Recompute only +when no such fact is available. If the repair attribution branch needs changes, +index or group originating repair operations once per affected unit. Preserve +ordered application, current-text protection checks, and the distinction +between response filtering and staleness. + +**Acceptance:** Keep both a stale-conflict case with initially valid proposals +and an originally inapplicable approval from a controlled validator. Verify +the skip, correction counts, and filtering or recovery attribution.