Record algorithmic reliability audit findings
This commit is contained in:
@@ -2,10 +2,10 @@
|
|||||||
|
|
||||||
**Reviewed revision:** `e8c0708`
|
**Reviewed revision:** `e8c0708`
|
||||||
|
|
||||||
**Status:** Eight open findings from the review of all 20
|
**Status:** Fourteen open findings from review of the 20
|
||||||
[implementation stages](implementation.md) against the
|
[implementation stages](implementation.md) against the
|
||||||
[target roadmap](reliability-and-maintainability.md) and the follow-up audit
|
[target roadmap](reliability-and-maintainability.md), including follow-up
|
||||||
of how those stages interact with existing callers and contracts.
|
audits of component interactions and algorithmic costs.
|
||||||
|
|
||||||
## Review Result
|
## Review Result
|
||||||
|
|
||||||
@@ -34,6 +34,16 @@ validator chains, model execution, publication, and report and debug readers.
|
|||||||
Additional probes reproduced the case-variant input behavior in R4 and the
|
Additional probes reproduced the case-variant input behavior in R4 and the
|
||||||
cross-component failures in R6–R8.
|
cross-component failures in R6–R8.
|
||||||
|
|
||||||
|
The algorithmic audit kept the selected stage window, full repair history,
|
||||||
|
and chunk-partitioning algorithm. The normalization merge builds text and
|
||||||
|
categories incrementally after sorting, and the explicit partitioner advances
|
||||||
|
through the weights in linear passes. The remaining opportunities are
|
||||||
|
primarily in repeated validation and glossary preparation, queued debug
|
||||||
|
snapshots, and evidence retained after it has served its purpose. R9 was
|
||||||
|
reproduced through the PromptKit role adapter with a synthetic provider.
|
||||||
|
R10–R14 follow from source analysis. Focused offline tests passed for
|
||||||
|
normalization, chunking, preflight, validators, the runner, and reports.
|
||||||
|
|
||||||
## Findings
|
## Findings
|
||||||
|
|
||||||
### R1 — Include Later Delivery Failures in Writable Reports (P2)
|
### R1 — Include Later Delivery Failures in Writable Reports (P2)
|
||||||
@@ -244,3 +254,186 @@ protection when a located edit produces an empty preview.
|
|||||||
Verify consistent local attribution, no unsupported protection approval,
|
Verify consistent local attribution, no unsupported protection approval,
|
||||||
and no model validation call for the inapplicable candidate. Keep repair
|
and no model validation call for the inapplicable candidate. Keep repair
|
||||||
available in the positive-budget case.
|
available in the positive-budget case.
|
||||||
|
|
||||||
|
### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
|
||||||
|
|
||||||
|
**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9
|
||||||
|
already handles the corresponding validator references.
|
||||||
|
|
||||||
|
**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20)
|
||||||
|
and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9)
|
||||||
|
specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21)
|
||||||
|
and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24)
|
||||||
|
use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184)
|
||||||
|
classifies a conversion error after schema validation as recoverable invalid
|
||||||
|
output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123)
|
||||||
|
then returns an operation failure with no candidates.
|
||||||
|
|
||||||
|
**Reproduced behavior:** A synthetic provider returned two proposals with
|
||||||
|
references `1` and `2.0`. PromptKit accepted the response in one request with
|
||||||
|
zero structural repairs. Audita returned `invalid_output` at typed decode.
|
||||||
|
The control case with `1` and `2` decoded both proposals. The repair decoder
|
||||||
|
has the same conversion for `proposal_ref` and the revision's `id`.
|
||||||
|
|
||||||
|
**Required change:** Decode wire references as exact numeric tokens and
|
||||||
|
reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9)
|
||||||
|
before local position or correspondence checks. Preserve the distinction
|
||||||
|
between segment and proposal references, existing out-of-range handling, and
|
||||||
|
the original response in debug capture.
|
||||||
|
|
||||||
|
**Acceptance:** Exercise the actual structured paths with an integer and an
|
||||||
|
equivalent decimal or exponent form. A valid sibling survives proposal
|
||||||
|
decoding, and the repair revision reaches the same revalidation as its integer
|
||||||
|
form. Retain existing handling for a mismatched repair reference.
|
||||||
|
|
||||||
|
### R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stages 8 and 10.
|
||||||
|
|
||||||
|
**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251)
|
||||||
|
builds optional context for every previewable candidate before
|
||||||
|
[fitting the payload](../../internal/framework/validators/llm_budget.go#L50).
|
||||||
|
[Context construction](../../internal/framework/validators/llm_context.go#L39)
|
||||||
|
rescans the original and adjacent segments, finds matches in the proposed
|
||||||
|
text, sorts matched entry positions, and builds the entry slice. The fitter
|
||||||
|
can discard all optional context while retaining the original slice's
|
||||||
|
backing array on a fitted item.
|
||||||
|
|
||||||
|
**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B`
|
||||||
|
bytes of shared source and neighboring text, the unchanged text is scanned
|
||||||
|
`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate
|
||||||
|
add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work
|
||||||
|
happens even when required fields consume the budget. Distinct proposed texts
|
||||||
|
still need their own match evidence when optional context is used.
|
||||||
|
|
||||||
|
**Recommended change:** Fit the required fields before collecting optional
|
||||||
|
context for a candidate. Reuse identity membership for unchanged text within
|
||||||
|
the section and combine it with the corrected-text matches only when needed.
|
||||||
|
Keep existing glossary and neighbor order, exact serialized budget accounting,
|
||||||
|
and oversized-input behavior. Drop unused optional storage before dispatch.
|
||||||
|
|
||||||
|
**Acceptance:** Compare the emitted payload to the existing serialized-budget
|
||||||
|
oracle for usable and oversized candidates. A controlled no-room case should
|
||||||
|
skip optional matching. Sibling candidates should share unchanged-text lookup
|
||||||
|
while still receiving definitions specific to their revisions.
|
||||||
|
|
||||||
|
### R11 — Reuse Glossary Selections Before Building Their Views (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stage 11, item 2.
|
||||||
|
|
||||||
|
**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115)
|
||||||
|
clones selected glossary entries and aliases and derives protection terms for
|
||||||
|
every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152)
|
||||||
|
can reuse prompt variables and validator views only after all those copies
|
||||||
|
have been built. The retained specs still hold separate entry and term slices
|
||||||
|
for identical selections.
|
||||||
|
|
||||||
|
**Cost:** With `S` stages sharing a selection of `G` entries and spelling
|
||||||
|
contributions, materializing and retaining those selection data costs
|
||||||
|
`Θ(S × G)`. Sharing immutable selection data brings this portion to
|
||||||
|
`Θ(G + S)` for repeated fixed-length selection keys.
|
||||||
|
|
||||||
|
**Recommended change:** Resolve and prepare the shared glossary data on the
|
||||||
|
first occurrence of an ordered context/protection selection and exemption.
|
||||||
|
Reuse the complete read-only selection in later specs. Keep stage identity,
|
||||||
|
first-failure attribution, and copy-on-mutation at consumer boundaries.
|
||||||
|
|
||||||
|
**Acceptance:** Exercise repeated selections, a reordered selection, and a
|
||||||
|
glossary-stage exemption. Reused selections should supply the same data to
|
||||||
|
proposal, validation, and application as the existing independent views.
|
||||||
|
|
||||||
|
### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stages 3, 14, 16, and 18.
|
||||||
|
|
||||||
|
**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151)
|
||||||
|
converts every turn's full rendered message list to a new slice. Each
|
||||||
|
[repair builder](../../internal/framework/runner/repair.go#L194) captures that
|
||||||
|
snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22)
|
||||||
|
holds the builders until ordered proposal indices are assigned, including
|
||||||
|
during earlier sections' model work.
|
||||||
|
|
||||||
|
**Cost:** A branch with `H` starting messages and `T` successful turns that
|
||||||
|
add a constant number of messages per turn can retain `Θ(T × H + T²)` message
|
||||||
|
descriptors at once. Content strings may share backing storage. The window
|
||||||
|
bounds the number of sections held, and a single section can still contain
|
||||||
|
many turns and branches. Full-history requests and self-contained debug files
|
||||||
|
require repeated serialization by design. They can be serialized one at a
|
||||||
|
time without keeping every prepared list live.
|
||||||
|
|
||||||
|
**Recommended change:** Keep compact evidence pending ordered identity
|
||||||
|
assignment. For example, stage the request payload under section-local
|
||||||
|
coordinates while it is live and use its final name when indices are assigned.
|
||||||
|
An in-memory approach can share immutable history prefixes and expand each
|
||||||
|
snapshot at write time if it can reproduce the exact rendered messages.
|
||||||
|
Retain full model context, nonfatal capture, and self-contained final files.
|
||||||
|
|
||||||
|
**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section.
|
||||||
|
Compare its decoded request bodies, final artifact names, and ledger paths
|
||||||
|
with the current representation, including a capture failure followed by a
|
||||||
|
successful write.
|
||||||
|
|
||||||
|
### R13 — Retain Only Operations Used by Ordinary Reports (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stages 12, 17, and 18.
|
||||||
|
|
||||||
|
**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632)
|
||||||
|
counts each work unit and retains all its operations when reports are enabled.
|
||||||
|
It also [stores every candidate index](../../internal/framework/runner/runner.go#L548)
|
||||||
|
on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79)
|
||||||
|
uses only recoverable failures and successful recoveries. Failure projection
|
||||||
|
uses affected indices and typed facts. Recovery projection uses counts and
|
||||||
|
filtered evidence. Neither uses the cause stored on each operation, which is
|
||||||
|
needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22).
|
||||||
|
|
||||||
|
**Cost:** The retained operation records, unused successful proposal index
|
||||||
|
lists, and causes account for `Θ(M + P + B)` storage, for `M` operation
|
||||||
|
records, `P` successful proposal indices, and `B` bytes of retained causes.
|
||||||
|
PromptKit's non-success HTTP error can include a bounded
|
||||||
|
provider diagnostic in each cause. Even an all-success run with no recoveries
|
||||||
|
keeps and later sorts operation rows that produce no error or recovery entry.
|
||||||
|
The report matrix itself still needs a row per work unit.
|
||||||
|
|
||||||
|
**Recommended change:** After counting and resolving application filtering and
|
||||||
|
failure attribution, retain compact facts only for operations that feed the
|
||||||
|
reports. Keep typed initiating and peer causes where they belong on execution
|
||||||
|
failures. Debug execution observations and correction-ledger inputs have
|
||||||
|
their own consumers. Avoid retaining proposal index lists on operations that
|
||||||
|
never project those lists.
|
||||||
|
|
||||||
|
**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals,
|
||||||
|
and debug association across existing evidence modes. Include clean successes,
|
||||||
|
both initial and repair filtering, and an interrupted run with available origin.
|
||||||
|
|
||||||
|
### R14 — Reuse Original Applicability When Attributing Application Skips (P3)
|
||||||
|
|
||||||
|
**Affected plan:** Stages 8 and 17, item 3.
|
||||||
|
|
||||||
|
**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59)
|
||||||
|
checks each proposal against evolving segment text. On a defective-looking
|
||||||
|
skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953)
|
||||||
|
previews the proposal again against the original section to distinguish a
|
||||||
|
stale conflict from a defective response. The built-in validation path has
|
||||||
|
already determined the latter preview's applicability. Its result is discarded
|
||||||
|
before the attribution pass.
|
||||||
|
|
||||||
|
**Cost:** For `K` conflicting proposals against one original segment of `B`
|
||||||
|
bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of
|
||||||
|
original-text previews, even if application quickly rejected those proposals
|
||||||
|
against the changed text. This is separate from the checks required against
|
||||||
|
current text. When a validator chain can approve originally inapplicable
|
||||||
|
revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46)
|
||||||
|
can also scan `O(F × M)` operations for `F` such filtered revisions and `M`
|
||||||
|
operations in the same unit. Built-in validators normally reject those revisions
|
||||||
|
before application.
|
||||||
|
|
||||||
|
**Recommended change:** Carry a compact original-applicability fact for
|
||||||
|
approved candidates when the validator already computed it. Recompute only
|
||||||
|
when no such fact is available. If the repair attribution branch needs changes,
|
||||||
|
index or group originating repair operations once per affected unit. Preserve
|
||||||
|
ordered application, current-text protection checks, and the distinction
|
||||||
|
between response filtering and staleness.
|
||||||
|
|
||||||
|
**Acceptance:** Keep both a stale-conflict case with initially valid proposals
|
||||||
|
and an originally inapplicable approval from a controlled validator. Verify
|
||||||
|
the skip, correction counts, and filtering or recovery attribution.
|
||||||
|
|||||||
Reference in New Issue
Block a user