Record reliability sprint review findings
This commit is contained in:
150
docs/roadmap/remediation.md
Normal file
150
docs/roadmap/remediation.md
Normal file
@@ -0,0 +1,150 @@
|
||||
# Reliability and Maintainability Review Remediation
|
||||
|
||||
**Reviewed revision:** `e8c0708`
|
||||
|
||||
**Status:** Five open findings from the review of all 20
|
||||
[implementation stages](implementation.md) against the
|
||||
[target roadmap](reliability-and-maintainability.md).
|
||||
|
||||
## Review Result
|
||||
|
||||
The sprint established the target architecture. ConfigKit and native
|
||||
CommandKit integrations handle configuration and inspection, and preflight
|
||||
hands checked processing selections to the runner. Audita decodes and
|
||||
normalizes transcripts directly and partitions them once. A bounded section
|
||||
window allows proposal, validation, and repair to overlap within a stage.
|
||||
The runner assigns final proposal indices in section order and commits edits
|
||||
at the stage boundary. PromptKit owns model execution, backend admission,
|
||||
retries, and repair history. Required publication uses checked destinations.
|
||||
Ordinary reports use typed processing facts, while optional debug capture
|
||||
retains interactions and decision history. CommandKit renders live progress.
|
||||
|
||||
The offline CLI build and full Go suite passed, as did targeted race checks
|
||||
for the runner, LLM adapter, validators, report projection, diagnostics,
|
||||
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
|
||||
under the race detector. A built CLI returned status 2 with source diagnostics
|
||||
for malformed PromptKit and application configuration across the delegated
|
||||
commands and processing cases in the plan. Focused review probes reproduced
|
||||
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
|
||||
documentation.
|
||||
|
||||
## Findings
|
||||
|
||||
### R1 — Include Later Delivery Failures in Writable Reports (P2)
|
||||
|
||||
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
|
||||
|
||||
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
|
||||
renders a failed bundle using the failures accumulated at that point. Later
|
||||
writes append to the returned error list, but the remaining reports and the
|
||||
best-effort pass reuse the same bytes. The
|
||||
[report projection](../../internal/framework/processreport/failures.go#L41)
|
||||
can include secondary file failures only if publication supplies them.
|
||||
|
||||
**Reproduced behavior:** Start finalization with a processing failure. Make
|
||||
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
|
||||
The returned error contains the processing failure and both delivery failures.
|
||||
The renderer receives only the processing failure, so the writable report
|
||||
omits the two already-observed failures. The same stale-bundle problem can
|
||||
follow a report error that first switches a successful run to failed status.
|
||||
|
||||
**Required change:** Use the freshest observed failures at the report
|
||||
publication boundary and make a bounded best-effort update of already-written
|
||||
reports when their failure evidence becomes stale. Retain existing evidence
|
||||
if rendering the updated failed bundle fails. Keep transcript-first,
|
||||
nontransactional publication.
|
||||
|
||||
**Acceptance:** Extend the output-session regression to inspect the facts
|
||||
rendered into a surviving report when an earlier failure is followed by a
|
||||
second write failure. Check failure attribution and counts as well as status.
|
||||
|
||||
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
|
||||
|
||||
**Affected plan:** Stage 17, item 1.
|
||||
|
||||
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
|
||||
adds a top-level `scope: interrupted` record to the
|
||||
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
|
||||
The baseline contract described `work_unit` and `run` records. An exhaustive
|
||||
reader of those variants encounters a third variant under the same version.
|
||||
The target asks to retain peer attribution and existing counts and schema.
|
||||
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
|
||||
also preserves the documented output contract for minor and patch releases.
|
||||
|
||||
**Required change:** Keep the existing top-level variants. Carry available
|
||||
peer interruption details as optional evidence on an existing record, for
|
||||
example an optional sequence on the initiating run failure. Keep the primary
|
||||
origin and peer origins separately identifiable and preserve the work-unit and
|
||||
run-error counts. Treat a new top-level variant as an explicit contract change
|
||||
if that representation is preferred.
|
||||
|
||||
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
|
||||
Check that a consumer of the existing record variants can still read the
|
||||
report, and that the peer's available phase, reason, and origin remain present.
|
||||
|
||||
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
|
||||
|
||||
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
|
||||
|
||||
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
|
||||
uses `internal_failure` when a typed model failure has no reason and retains
|
||||
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
|
||||
recognizes a typed failure only when its reason is nonempty. It can then derive
|
||||
a different reason from a wrapped cause or the processing phase.
|
||||
|
||||
**Reproduced behavior:** A typed fatal failure with an empty reason and a
|
||||
wrapped `context.DeadlineExceeded` is recorded on the operation as
|
||||
`fatal` / `internal_failure` and serialized in the run report as
|
||||
`canceled`. The cause is available, but it does not make this a parent
|
||||
cancellation.
|
||||
|
||||
**Required change:** Share the typed reason projection and its empty-reason
|
||||
fallback. Derive a reason from the raw cause only when a typed failure is absent.
|
||||
|
||||
**Acceptance:** Verify matching fallbacks for the operation and the run record
|
||||
and retain the phase, origin, and original cause. Keep genuine parent
|
||||
cancellation distinct.
|
||||
|
||||
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
|
||||
|
||||
**Affected plan:** Stage 5, item 1.
|
||||
|
||||
**Evidence:** The baseline selected the final value of an exact duplicate JSON
|
||||
member when it decoded the segment into a generic object, before applying
|
||||
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
|
||||
now applies the field type to every occurrence. An earlier type error survives
|
||||
even when the final value is valid. The refactor was meant to retain accepted
|
||||
inputs while improving numeric fidelity.
|
||||
|
||||
**Reproduced behavior:** The baseline at `d226bdd` accepts
|
||||
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
|
||||
speaker `A`. The sprint revision rejects the same input because of the
|
||||
superseded numeric value.
|
||||
|
||||
**Required change:** Preserve the effective-member behavior for existing
|
||||
inputs without reviving whole-transcript generic decoding or float conversion
|
||||
for IDs. Keep the compatibility handling focused on this changed boundary.
|
||||
|
||||
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
|
||||
in the effective member. Keep the existing exact-ID and unknown-field tests
|
||||
for both input envelopes.
|
||||
|
||||
### R5 — Align Roadmap Status and Runtime Guidance (P3)
|
||||
|
||||
**Affected plan:** Stages 15, 19, and 20.
|
||||
|
||||
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
|
||||
describes the implementation as pending despite the completed sprint.
|
||||
The [pipeline guide](../internal/pipeline.md) says work-unit
|
||||
evidence preserves each peer's interruption or error. The runner preserves
|
||||
available outcomes and projects detailed work units according to output
|
||||
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
|
||||
literal `\n` between “selected” and “profiles.”
|
||||
|
||||
**Required change:** Mark the roadmap with the landed sprint and this review,
|
||||
while retaining its historical baseline context. Describe interruption detail
|
||||
in terms of outcomes actually available and selected for retention. Fix the
|
||||
CLI sentence and keep public guidance focused on the operator-visible behavior.
|
||||
|
||||
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
|
||||
reflects the evidence modes, and the CLI paragraph renders normally.
|
||||
Reference in New Issue
Block a user