Files
audita/docs/roadmap/remediation.md

463 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Reliability and Maintainability Review Remediation
**Reviewed revision:** `e8c0708`
**Status:** Findings and remediation progress from review of the 20
[implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md), including follow-up
audits of component interactions and algorithmic costs.
## Review Result
The sprint established the target architecture. ConfigKit and native
CommandKit integrations handle configuration and inspection, and preflight
hands checked processing selections to the runner. Audita decodes and
normalizes transcripts directly and partitions them once. A bounded section
window allows proposal, validation, and repair to overlap within a stage.
The runner assigns final proposal indices in section order and commits edits
at the stage boundary. PromptKit owns model execution, backend admission,
retries, and repair history. Required publication uses checked destinations.
Ordinary reports use typed processing facts, while optional debug capture
retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks
for the runner, LLM adapter, validators, report projection, diagnostics,
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
under the race detector. A built CLI returned status 2 with source diagnostics
for malformed PromptKit and application configuration across the delegated
commands and processing cases in the plan. Focused review probes reproduced
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in
validator chains, model execution, publication, and report and debug readers.
Additional probes reproduced the case-variant input behavior in R4 and the
cross-component failures in R6–R8.
The algorithmic audit kept the selected stage window, full repair history,
and chunk-partitioning algorithm. The normalization merge builds text and
categories incrementally after sorting, and the explicit partitioner advances
through the weights in linear passes. The remaining opportunities are
primarily in repeated validation and glossary preparation, queued debug
snapshots, and evidence retained after it has served its purpose. R9 was
reproduced through the PromptKit role adapter with a synthetic provider.
R10–R14 follow from source analysis. Focused offline tests passed for
normalization, chunking, preflight, validators, the runner, and reports.
## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2)
**Complete.** Later report writes include newly observed delivery failures.
One best-effort pass updates earlier reports with stale evidence. Diagnostics
tests cover both initially failed runs and failures that arise during
publication of a completed run.
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
renders a failed bundle using the failures accumulated at that point. Later
writes append to the returned error list, but the remaining reports and the
best-effort pass reuse the same bytes. The
[report projection](../../internal/framework/processreport/failures.go#L41)
can include secondary file failures only if publication supplies them.
**Reproduced behavior:** Start finalization with a processing failure. Make
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
**Required change:** Use the freshest observed failures at the report
publication boundary and make a bounded best-effort update of already-written
reports when their failure evidence becomes stale. Retain existing evidence
if rendering the updated failed bundle fails. Keep transcript-first,
nontransactional publication.
**Acceptance:** Extend the output-session regression to inspect the facts
rendered into a surviving report when an earlier failure is followed by a
second write failure. Check failure attribution and counts as well as status.
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
**Complete.** Version 1 retains the `work_unit` and `run` scopes. The
initiating run record carries available peer details in an optional
`interruptions` sequence. The mixed-run YAML and Markdown test verifies
peer attribution and unchanged counts.
**Affected plan:** Stage 17, item 1.
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
adds a top-level `scope: interrupted` record to the
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
The baseline contract described `work_unit` and `run` records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
also preserves the documented output contract for minor and patch releases.
**Required change:** Keep the existing top-level variants. Carry available
peer interruption details as optional evidence on an existing record, for
example an optional sequence on the initiating run failure. Keep the primary
origin and peer origins separately identifiable and preserve the work-unit and
run-error counts. Treat a new top-level variant as an explicit contract change
if that representation is preferred.
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
Check that a consumer of the existing record variants can still read the
report, and that the peer's available phase, reason, and origin remain present.
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
**Complete.** The operation and run report use the same typed reason fallback.
The regression covers a missing reason with a wrapped deadline, retained
operation origin and cause, and distinct parent cancellation.
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
uses `internal_failure` when a typed model failure has no reason and retains
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
**Reproduced behavior:** A typed fatal failure with an empty reason and a
wrapped `context.DeadlineExceeded` is recorded on the operation as
`fatal` / `internal_failure` and serialized in the run report as
`canceled`. The cause is available, but it does not make this a parent
cancellation.
**Required change:** Share the typed reason projection and its empty-reason
fallback. Derive a reason from the raw cause only when a typed failure is absent.
**Acceptance:** Verify matching fallbacks for the operation and the run record
and retain the phase, origin, and original cause. Keep genuine parent
cancellation distinct.
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
**Complete.** Each segment resolves duplicate keys before field conversion
and processes case variants in the baseline key order. The decoder keeps
numeric tokens intact. Schema tests cover superseded and effective type
mismatches, both case-variant orders, and exact IDs in both envelopes.
**Affected plan:** Stage 5, item 1.
**Evidence:** The baseline selected the final value of an exact duplicate JSON
member when it decoded the segment into a generic object, before applying
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
now applies the field type to every occurrence. An earlier type error survives
even when the final value is valid. The refactor was meant to retain accepted
inputs while improving numeric fidelity.
**Reproduced behavior:** The baseline at `d226bdd` accepts
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
speaker `A`. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing `"text":"expected utterance"` followed
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
normalization. The new decoder passes `other metadata`. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
**Required change:** Preserve the effective-member behavior for existing
inputs without reviving whole-transcript generic decoding or float conversion
for IDs. Keep the compatibility handling focused on this changed boundary.
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
in the effective member, plus the case-variant example in both member orders.
Keep the existing exact-ID and unknown-field tests for both input envelopes.
### R5 — Align Roadmap Status and Runtime Guidance (P3)
**Complete.** The roadmap points to the landed implementation and follow-up
review. The pipeline guide describes available peer evidence in reporting mode.
The CLI guide renders the preflight sentence normally.
**Affected plan:** Stages 15, 19, and 20.
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
describes the implementation as pending despite the completed sprint.
The [pipeline guide](../internal/pipeline.md) says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
literal `\n` between “selected” and “profiles.”
**Required change:** Mark the roadmap with the landed sprint and this review,
while retaining its historical baseline context. Describe interruption detail
in terms of outcomes actually available and selected for retention. Fix the
CLI sentence and keep public guidance focused on the operator-visible behavior.
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
reflects the evidence modes, and the CLI paragraph renders normally.
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
**Affected plan:** Stages 12 and 19.
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
calls [progress rendering](../../internal/cli/process_progress.go#L34)
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
PromptKit v0.23.1 starts the operation deadline before emitting the start
event, so a blocked or slow write to CommandKit's diagnostic stream spends
that operation's budget before the provider request starts.
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported `operation_timeout`, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
**Required change:** Deliver progress outside PromptKit's timed model-call
observation path. Keep native observations for execution facts and use
application lifecycle boundaries for synchronous progress delivery, or another
delivery mechanism that keeps stderr waits out of the per-operation budget.
Preserve CommandKit's routing and presentation status, and keep the parent
context authoritative for cancellation of the invocation.
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
Verify that it still dispatches and applies the same correction within the
model's own budget, and retain live progress and presentation-failure coverage.
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
**Affected plan:** Stages 1, 11, and 20.
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing [process entry point](../../internal/cli/process.go#L27)
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
unconditionally returns a usage outcome for those errors.
**Reproduced behavior:** A readable main file beside an unreadable automatic
`prompts.yml` produces `source.permission` in all three commands.
`promptkit list prompts` and `config show --scope promptkit` return status 1.
`process` returns status 2 and prints usage guidance for the same loading
failure.
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
**Acceptance:** Compare `process` and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
### R8 — Use the Same Span Applicability With and Without Repair (P3)
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
rejects an ambiguous require-unique span only when independent repair evidence
is enabled. Without repair, the existing shortlist forwards the candidate to
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
which records approval even when it cannot obtain a usable preview. The
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
later rejects that same local ambiguity before dispatching a provider call.
**Reproduced behavior:** In the built-in grammar stage, offer to replace
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
protected. With a zero repair budget, the recorded decisions show protection
approval and `editorial_review` rejection for `ambiguous_original_text`.
With repair enabled, `original_text_presence` records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
**Required change:** Give the local applicability check ownership of
require-unique ambiguity in both paths. Keep independent evidence and repair
attempts conditional on the configured budget. Retain the ability to assess
protection when a located edit produces an empty preview.
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
Verify consistent local attribution, no unsupported protection approval,
and no model validation call for the inapplicable candidate. Keep repair
available in the positive-budget case.
### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9
already handles the corresponding validator references.
**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20)
and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9)
specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21)
and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24)
use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184)
classifies a conversion error after schema validation as recoverable invalid
output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123)
then returns an operation failure with no candidates.
**Reproduced behavior:** A synthetic provider returned two proposals with
references `1` and `2.0`. PromptKit accepted the response in one request with
zero structural repairs. Audita returned `invalid_output` at typed decode.
The control case with `1` and `2` decoded both proposals. The repair decoder
has the same conversion for `proposal_ref` and the revision's `id`.
**Required change:** Decode wire references as exact numeric tokens and
reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9)
before local position or correspondence checks. Preserve the distinction
between segment and proposal references, existing out-of-range handling, and
the original response in debug capture.
**Acceptance:** Exercise the actual structured paths with an integer and an
equivalent decimal or exponent form. A valid sibling survives proposal
decoding, and the repair revision reaches the same revalidation as its integer
form. Retain existing handling for a mismatched repair reference.
### R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
**Affected plan:** Stages 8 and 10.
**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251)
builds optional context for every previewable candidate before
[fitting the payload](../../internal/framework/validators/llm_budget.go#L50).
[Context construction](../../internal/framework/validators/llm_context.go#L39)
rescans the original and adjacent segments, finds matches in the proposed
text, sorts matched entry positions, and builds the entry slice. The fitter
can discard all optional context while retaining the original slice's
backing array on a fitted item.
**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B`
bytes of shared source and neighboring text, the unchanged text is scanned
`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate
add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work
happens even when required fields consume the budget. Distinct proposed texts
still need their own match evidence when optional context is used.
**Recommended change:** Fit the required fields before collecting optional
context for a candidate. Reuse identity membership for unchanged text within
the section and combine it with the corrected-text matches only when needed.
Keep existing glossary and neighbor order, exact serialized budget accounting,
and oversized-input behavior. Drop unused optional storage before dispatch.
**Acceptance:** Compare the emitted payload to the existing serialized-budget
oracle for usable and oversized candidates. A controlled no-room case should
skip optional matching. Sibling candidates should share unchanged-text lookup
while still receiving definitions specific to their revisions.
### R11 — Reuse Glossary Selections Before Building Their Views (P3)
**Affected plan:** Stage 11, item 2.
**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115)
clones selected glossary entries and aliases and derives protection terms for
every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152)
can reuse prompt variables and validator views only after all those copies
have been built. The retained specs still hold separate entry and term slices
for identical selections.
**Cost:** With `S` stages sharing a selection of `G` entries and spelling
contributions, materializing and retaining those selection data costs
`Θ(S × G)`. Sharing immutable selection data brings this portion to
`Θ(G + S)` for repeated fixed-length selection keys.
**Recommended change:** Resolve and prepare the shared glossary data on the
first occurrence of an ordered context/protection selection and exemption.
Reuse the complete read-only selection in later specs. Keep stage identity,
first-failure attribution, and copy-on-mutation at consumer boundaries.
**Acceptance:** Exercise repeated selections, a reordered selection, and a
glossary-stage exemption. Reused selections should supply the same data to
proposal, validation, and application as the existing independent views.
### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
**Affected plan:** Stages 3, 14, 16, and 18.
**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151)
converts every turn's full rendered message list to a new slice. Each
[repair builder](../../internal/framework/runner/repair.go#L194) captures that
snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22)
holds the builders until ordered proposal indices are assigned, including
during earlier sections' model work.
**Cost:** A branch with `H` starting messages and `T` successful turns that
add a constant number of messages per turn can retain `Θ(T × H + T²)` message
descriptors at once. Content strings may share backing storage. The window
bounds the number of sections held, and a single section can still contain
many turns and branches. Full-history requests and self-contained debug files
require repeated serialization by design. They can be serialized one at a
time without keeping every prepared list live.
**Recommended change:** Keep compact evidence pending ordered identity
assignment. For example, stage the request payload under section-local
coordinates while it is live and use its final name when indices are assigned.
An in-memory approach can share immutable history prefixes and expand each
snapshot at write time if it can reproduce the exact rendered messages.
Retain full model context, nonfatal capture, and self-contained final files.
**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section.
Compare its decoded request bodies, final artifact names, and ledger paths
with the current representation, including a capture failure followed by a
successful write.
### R13 — Retain Only Operations Used by Ordinary Reports (P3)
**Affected plan:** Stages 12, 17, and 18.
**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632)
counts each work unit and retains all its operations when reports are enabled.
It also [stores every candidate index](../../internal/framework/runner/runner.go#L548)
on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79)
uses only recoverable failures and successful recoveries. Failure projection
uses affected indices and typed facts. Recovery projection uses counts and
filtered evidence. Neither uses the cause stored on each operation, which is
needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22).
**Cost:** The retained operation records, unused successful proposal index
lists, and causes account for `Θ(M + P + B)` storage, for `M` operation
records, `P` successful proposal indices, and `B` bytes of retained causes.
PromptKit's non-success HTTP error can include a bounded
provider diagnostic in each cause. Even an all-success run with no recoveries
keeps and later sorts operation rows that produce no error or recovery entry.
The report matrix itself still needs a row per work unit.
**Recommended change:** After counting and resolving application filtering and
failure attribution, retain compact facts only for operations that feed the
reports. Keep typed initiating and peer causes where they belong on execution
failures. Debug execution observations and correction-ledger inputs have
their own consumers. Avoid retaining proposal index lists on operations that
never project those lists.
**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals,
and debug association across existing evidence modes. Include clean successes,
both initial and repair filtering, and an interrupted run with available origin.
### R14 — Reuse Original Applicability When Attributing Application Skips (P3)
**Affected plan:** Stages 8 and 17, item 3.
**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59)
checks each proposal against evolving segment text. On a defective-looking
skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953)
previews the proposal again against the original section to distinguish a
stale conflict from a defective response. The built-in validation path has
already determined the latter preview's applicability. Its result is discarded
before the attribution pass.
**Cost:** For `K` conflicting proposals against one original segment of `B`
bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of
original-text previews, even if application quickly rejected those proposals
against the changed text. This is separate from the checks required against
current text. When a validator chain can approve originally inapplicable
revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46)
can also scan `O(F × M)` operations for `F` such filtered revisions and `M`
operations in the same unit. Built-in validators normally reject those revisions
before application.
**Recommended change:** Carry a compact original-applicability fact for
approved candidates when the validator already computed it. Recompute only
when no such fact is available. If the repair attribution branch needs changes,
index or group originating repair operations once per affected unit. Preserve
ordered application, current-text protection checks, and the distinction
between response filtering and staleness.
**Acceptance:** Keep both a stale-conflict case with initially valid proposals
and an originally inapplicable approval from a controlled validator. Verify
the skip, correction counts, and filtering or recovery attribution.