Record algorithmic reliability audit findings

This commit is contained in:
2026-11-17 03:49:35 +00:00
parent 1c90fedb1e
commit f99fd993b4

View File

@@ -2,10 +2,10 @@
**Reviewed revision:** `e8c0708` **Reviewed revision:** `e8c0708`
**Status:** Eight open findings from the review of all 20 **Status:** Fourteen open findings from review of the 20
[implementation stages](implementation.md) against the [implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md) and the follow-up audit [target roadmap](reliability-and-maintainability.md), including follow-up
of how those stages interact with existing callers and contracts. audits of component interactions and algorithmic costs.
## Review Result ## Review Result
@@ -34,6 +34,16 @@ validator chains, model execution, publication, and report and debug readers.
Additional probes reproduced the case-variant input behavior in R4 and the Additional probes reproduced the case-variant input behavior in R4 and the
cross-component failures in R6–R8. cross-component failures in R6–R8.
The algorithmic audit kept the selected stage window, full repair history,
and chunk-partitioning algorithm. The normalization merge builds text and
categories incrementally after sorting, and the explicit partitioner advances
through the weights in linear passes. The remaining opportunities are
primarily in repeated validation and glossary preparation, queued debug
snapshots, and evidence retained after it has served its purpose. R9 was
reproduced through the PromptKit role adapter with a synthetic provider.
R10–R14 follow from source analysis. Focused offline tests passed for
normalization, chunking, preflight, validators, the runner, and reports.
## Findings ## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2) ### R1 — Include Later Delivery Failures in Writable Reports (P2)
@@ -244,3 +254,186 @@ protection when a located edit produces an empty preview.
Verify consistent local attribution, no unsupported protection approval, Verify consistent local attribution, no unsupported protection approval,
and no model validation call for the inapplicable candidate. Keep repair and no model validation call for the inapplicable candidate. Keep repair
available in the positive-budget case. available in the positive-budget case.
### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9
already handles the corresponding validator references.
**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20)
and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9)
specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21)
and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24)
use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184)
classifies a conversion error after schema validation as recoverable invalid
output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123)
then returns an operation failure with no candidates.
**Reproduced behavior:** A synthetic provider returned two proposals with
references `1` and `2.0`. PromptKit accepted the response in one request with
zero structural repairs. Audita returned `invalid_output` at typed decode.
The control case with `1` and `2` decoded both proposals. The repair decoder
has the same conversion for `proposal_ref` and the revision's `id`.
**Required change:** Decode wire references as exact numeric tokens and
reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9)
before local position or correspondence checks. Preserve the distinction
between segment and proposal references, existing out-of-range handling, and
the original response in debug capture.
**Acceptance:** Exercise the actual structured paths with an integer and an
equivalent decimal or exponent form. A valid sibling survives proposal
decoding, and the repair revision reaches the same revalidation as its integer
form. Retain existing handling for a mismatched repair reference.
### R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
**Affected plan:** Stages 8 and 10.
**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251)
builds optional context for every previewable candidate before
[fitting the payload](../../internal/framework/validators/llm_budget.go#L50).
[Context construction](../../internal/framework/validators/llm_context.go#L39)
rescans the original and adjacent segments, finds matches in the proposed
text, sorts matched entry positions, and builds the entry slice. The fitter
can discard all optional context while retaining the original slice's
backing array on a fitted item.
**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B`
bytes of shared source and neighboring text, the unchanged text is scanned
`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate
add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work
happens even when required fields consume the budget. Distinct proposed texts
still need their own match evidence when optional context is used.
**Recommended change:** Fit the required fields before collecting optional
context for a candidate. Reuse identity membership for unchanged text within
the section and combine it with the corrected-text matches only when needed.
Keep existing glossary and neighbor order, exact serialized budget accounting,
and oversized-input behavior. Drop unused optional storage before dispatch.
**Acceptance:** Compare the emitted payload to the existing serialized-budget
oracle for usable and oversized candidates. A controlled no-room case should
skip optional matching. Sibling candidates should share unchanged-text lookup
while still receiving definitions specific to their revisions.
### R11 — Reuse Glossary Selections Before Building Their Views (P3)
**Affected plan:** Stage 11, item 2.
**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115)
clones selected glossary entries and aliases and derives protection terms for
every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152)
can reuse prompt variables and validator views only after all those copies
have been built. The retained specs still hold separate entry and term slices
for identical selections.
**Cost:** With `S` stages sharing a selection of `G` entries and spelling
contributions, materializing and retaining those selection data costs
`Θ(S × G)`. Sharing immutable selection data brings this portion to
`Θ(G + S)` for repeated fixed-length selection keys.
**Recommended change:** Resolve and prepare the shared glossary data on the
first occurrence of an ordered context/protection selection and exemption.
Reuse the complete read-only selection in later specs. Keep stage identity,
first-failure attribution, and copy-on-mutation at consumer boundaries.
**Acceptance:** Exercise repeated selections, a reordered selection, and a
glossary-stage exemption. Reused selections should supply the same data to
proposal, validation, and application as the existing independent views.
### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
**Affected plan:** Stages 3, 14, 16, and 18.
**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151)
converts every turn's full rendered message list to a new slice. Each
[repair builder](../../internal/framework/runner/repair.go#L194) captures that
snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22)
holds the builders until ordered proposal indices are assigned, including
during earlier sections' model work.
**Cost:** A branch with `H` starting messages and `T` successful turns that
add a constant number of messages per turn can retain `Θ(T × H + T²)` message
descriptors at once. Content strings may share backing storage. The window
bounds the number of sections held, and a single section can still contain
many turns and branches. Full-history requests and self-contained debug files
require repeated serialization by design. They can be serialized one at a
time without keeping every prepared list live.
**Recommended change:** Keep compact evidence pending ordered identity
assignment. For example, stage the request payload under section-local
coordinates while it is live and use its final name when indices are assigned.
An in-memory approach can share immutable history prefixes and expand each
snapshot at write time if it can reproduce the exact rendered messages.
Retain full model context, nonfatal capture, and self-contained final files.
**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section.
Compare its decoded request bodies, final artifact names, and ledger paths
with the current representation, including a capture failure followed by a
successful write.
### R13 — Retain Only Operations Used by Ordinary Reports (P3)
**Affected plan:** Stages 12, 17, and 18.
**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632)
counts each work unit and retains all its operations when reports are enabled.
It also [stores every candidate index](../../internal/framework/runner/runner.go#L548)
on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79)
uses only recoverable failures and successful recoveries. Failure projection
uses affected indices and typed facts. Recovery projection uses counts and
filtered evidence. Neither uses the cause stored on each operation, which is
needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22).
**Cost:** The retained operation records, unused successful proposal index
lists, and causes account for `Θ(M + P + B)` storage, for `M` operation
records, `P` successful proposal indices, and `B` bytes of retained causes.
PromptKit's non-success HTTP error can include a bounded
provider diagnostic in each cause. Even an all-success run with no recoveries
keeps and later sorts operation rows that produce no error or recovery entry.
The report matrix itself still needs a row per work unit.
**Recommended change:** After counting and resolving application filtering and
failure attribution, retain compact facts only for operations that feed the
reports. Keep typed initiating and peer causes where they belong on execution
failures. Debug execution observations and correction-ledger inputs have
their own consumers. Avoid retaining proposal index lists on operations that
never project those lists.
**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals,
and debug association across existing evidence modes. Include clean successes,
both initial and repair filtering, and an interrupted run with available origin.
### R14 — Reuse Original Applicability When Attributing Application Skips (P3)
**Affected plan:** Stages 8 and 17, item 3.
**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59)
checks each proposal against evolving segment text. On a defective-looking
skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953)
previews the proposal again against the original section to distinguish a
stale conflict from a defective response. The built-in validation path has
already determined the latter preview's applicability. Its result is discarded
before the attribution pass.
**Cost:** For `K` conflicting proposals against one original segment of `B`
bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of
original-text previews, even if application quickly rejected those proposals
against the changed text. This is separate from the checks required against
current text. When a validator chain can approve originally inapplicable
revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46)
can also scan `O(F × M)` operations for `F` such filtered revisions and `M`
operations in the same unit. Built-in validators normally reject those revisions
before application.
**Recommended change:** Carry a compact original-applicability fact for
approved candidates when the validator already computed it. Recompute only
when no such fact is available. If the repair attribution branch needs changes,
index or group originating repair operations once per affected unit. Preserve
ordered application, current-text protection checks, and the distinction
between response filtering and staleness.
**Acceptance:** Keep both a stale-conflict case with initially valid proposals
and an originally inapplicable approval from a controlled validator. Verify
the skip, correction counts, and filtering or recovery attribution.