diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md new file mode 100644 index 0000000..d2bedee --- /dev/null +++ b/docs/roadmap/remediation.md @@ -0,0 +1,150 @@ +# Reliability and Maintainability Review Remediation + +**Reviewed revision:** `e8c0708` + +**Status:** Five open findings from the review of all 20 +[implementation stages](implementation.md) against the +[target roadmap](reliability-and-maintainability.md). + +## Review Result + +The sprint established the target architecture. ConfigKit and native +CommandKit integrations handle configuration and inspection, and preflight +hands checked processing selections to the runner. Audita decodes and +normalizes transcripts directly and partitions them once. A bounded section +window allows proposal, validation, and repair to overlap within a stage. +The runner assigns final proposal indices in section order and commits edits +at the stage boundary. PromptKit owns model execution, backend admission, +retries, and repair history. Required publication uses checked destinations. +Ordinary reports use typed processing facts, while optional debug capture +retains interactions and decision history. CommandKit renders live progress. + +The offline CLI build and full Go suite passed, as did targeted race checks +for the runner, LLM adapter, validators, report projection, diagnostics, +glossary, and CLI. The ordered repair artifact test also passed 150 repetitions +under the race detector. A built CLI returned status 2 with source diagnostics +for malformed PromptKit and application configuration across the delegated +commands and processing cases in the plan. Focused review probes reproduced +R1, R3, and R4. R2 and R5 follow from the versioned output contract and current +documentation. + +## Findings + +### R1 — Include Later Delivery Failures in Writable Reports (P2) + +**Affected plan:** Stage 4, item 3, and Stage 17, item 1. + +**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311) +renders a failed bundle using the failures accumulated at that point. Later +writes append to the returned error list, but the remaining reports and the +best-effort pass reuse the same bytes. The +[report projection](../../internal/framework/processreport/failures.go#L41) +can include secondary file failures only if publication supplies them. + +**Reproduced behavior:** Start finalization with a processing failure. Make +`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed. +The returned error contains the processing failure and both delivery failures. +The renderer receives only the processing failure, so the writable report +omits the two already-observed failures. The same stale-bundle problem can +follow a report error that first switches a successful run to failed status. + +**Required change:** Use the freshest observed failures at the report +publication boundary and make a bounded best-effort update of already-written +reports when their failure evidence becomes stale. Retain existing evidence +if rendering the updated failed bundle fails. Keep transcript-first, +nontransactional publication. + +**Acceptance:** Extend the output-session regression to inspect the facts +rendered into a surviving report when an earlier failure is followed by a +second write failure. Check failure attribution and counts as well as status. + +### R2 — Preserve the Version 1 Error Record Discriminator (P2) + +**Affected plan:** Stage 17, item 1. + +**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108) +adds a top-level `scope: interrupted` record to the +[version 1 error document](../integrations/output-files.md#error-and-recovery-records). +The baseline contract described `work_unit` and `run` records. An exhaustive +reader of those variants encounters a third variant under the same version. +The target asks to retain peer attribution and existing counts and schema. +Audita's [release policy](../policy/release.md#versioning-and-compatibility) +also preserves the documented output contract for minor and patch releases. + +**Required change:** Keep the existing top-level variants. Carry available +peer interruption details as optional evidence on an existing record, for +example an optional sequence on the initiating run failure. Keep the primary +origin and peer origins separately identifiable and preserve the work-unit and +run-error counts. Treat a new top-level variant as an explicit contract change +if that representation is preferred. + +**Acceptance:** Exercise a fatal operation and interrupted peer in one run. +Check that a consumer of the existing record variants can still read the +report, and that the peer's available phase, reason, and origin remain present. + +### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3) + +**Affected plan:** Stage 12, item 2, and Stage 17, item 1. + +**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42) +uses `internal_failure` when a typed model failure has no reason and retains +its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16) +recognizes a typed failure only when its reason is nonempty. It can then derive +a different reason from a wrapped cause or the processing phase. + +**Reproduced behavior:** A typed fatal failure with an empty reason and a +wrapped `context.DeadlineExceeded` is recorded on the operation as +`fatal` / `internal_failure` and serialized in the run report as +`canceled`. The cause is available, but it does not make this a parent +cancellation. + +**Required change:** Share the typed reason projection and its empty-reason +fallback. Derive a reason from the raw cause only when a typed failure is absent. + +**Acceptance:** Verify matching fallbacks for the operation and the run record +and retain the phase, origin, and original cause. Keep genuine parent +cancellation distinct. + +### R4 — Preserve Effective Transcript Members During Typed Decoding (P3) + +**Affected plan:** Stage 5, item 1. + +**Evidence:** The baseline selected the final value of an exact duplicate JSON +member when it decoded the segment into a generic object, before applying +field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39) +now applies the field type to every occurrence. An earlier type error survives +even when the final value is valid. The refactor was meant to retain accepted +inputs while improving numeric fidelity. + +**Reproduced behavior:** The baseline at `d226bdd` accepts +`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads +speaker `A`. The sprint revision rejects the same input because of the +superseded numeric value. + +**Required change:** Preserve the effective-member behavior for existing +inputs without reviving whole-transcript generic decoding or float conversion +for IDs. Keep the compatibility handling focused on this changed boundary. + +**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch +in the effective member. Keep the existing exact-ID and unknown-field tests +for both input envelopes. + +### R5 — Align Roadmap Status and Runtime Guidance (P3) + +**Affected plan:** Stages 15, 19, and 20. + +**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still +describes the implementation as pending despite the completed sprint. +The [pipeline guide](../internal/pipeline.md) says work-unit +evidence preserves each peer's interruption or error. The runner preserves +available outcomes and projects detailed work units according to output +selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a +literal `\n` between “selected” and “profiles.” + +**Required change:** Mark the roadmap with the landed sprint and this review, +while retaining its historical baseline context. Describe interruption detail +in terms of outcomes actually available and selected for retention. Fix the +CLI sentence and keep public guidance focused on the operator-visible behavior. + +**Acceptance:** The roadmap and implementation status agree, the pipeline guide +reflects the evidence modes, and the CLI paragraph renders normally.