Files
audita/docs/roadmap/remediation.md

247 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Reliability and Maintainability Review Remediation
**Reviewed revision:** `e8c0708`
**Status:** Eight open findings from the review of all 20
[implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md) and the follow-up audit
of how those stages interact with existing callers and contracts.
## Review Result
The sprint established the target architecture. ConfigKit and native
CommandKit integrations handle configuration and inspection, and preflight
hands checked processing selections to the runner. Audita decodes and
normalizes transcripts directly and partitions them once. A bounded section
window allows proposal, validation, and repair to overlap within a stage.
The runner assigns final proposal indices in section order and commits edits
at the stage boundary. PromptKit owns model execution, backend admission,
retries, and repair history. Required publication uses checked destinations.
Ordinary reports use typed processing facts, while optional debug capture
retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks
for the runner, LLM adapter, validators, report projection, diagnostics,
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
under the race detector. A built CLI returned status 2 with source diagnostics
for malformed PromptKit and application configuration across the delegated
commands and processing cases in the plan. Focused review probes reproduced
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in
validator chains, model execution, publication, and report and debug readers.
Additional probes reproduced the case-variant input behavior in R4 and the
cross-component failures in R6–R8.
## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2)
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
renders a failed bundle using the failures accumulated at that point. Later
writes append to the returned error list, but the remaining reports and the
best-effort pass reuse the same bytes. The
[report projection](../../internal/framework/processreport/failures.go#L41)
can include secondary file failures only if publication supplies them.
**Reproduced behavior:** Start finalization with a processing failure. Make
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
**Required change:** Use the freshest observed failures at the report
publication boundary and make a bounded best-effort update of already-written
reports when their failure evidence becomes stale. Retain existing evidence
if rendering the updated failed bundle fails. Keep transcript-first,
nontransactional publication.
**Acceptance:** Extend the output-session regression to inspect the facts
rendered into a surviving report when an earlier failure is followed by a
second write failure. Check failure attribution and counts as well as status.
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
**Affected plan:** Stage 17, item 1.
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
adds a top-level `scope: interrupted` record to the
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
The baseline contract described `work_unit` and `run` records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
also preserves the documented output contract for minor and patch releases.
**Required change:** Keep the existing top-level variants. Carry available
peer interruption details as optional evidence on an existing record, for
example an optional sequence on the initiating run failure. Keep the primary
origin and peer origins separately identifiable and preserve the work-unit and
run-error counts. Treat a new top-level variant as an explicit contract change
if that representation is preferred.
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
Check that a consumer of the existing record variants can still read the
report, and that the peer's available phase, reason, and origin remain present.
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
uses `internal_failure` when a typed model failure has no reason and retains
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
**Reproduced behavior:** A typed fatal failure with an empty reason and a
wrapped `context.DeadlineExceeded` is recorded on the operation as
`fatal` / `internal_failure` and serialized in the run report as
`canceled`. The cause is available, but it does not make this a parent
cancellation.
**Required change:** Share the typed reason projection and its empty-reason
fallback. Derive a reason from the raw cause only when a typed failure is absent.
**Acceptance:** Verify matching fallbacks for the operation and the run record
and retain the phase, origin, and original cause. Keep genuine parent
cancellation distinct.
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
**Affected plan:** Stage 5, item 1.
**Evidence:** The baseline selected the final value of an exact duplicate JSON
member when it decoded the segment into a generic object, before applying
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
now applies the field type to every occurrence. An earlier type error survives
even when the final value is valid. The refactor was meant to retain accepted
inputs while improving numeric fidelity.
**Reproduced behavior:** The baseline at `d226bdd` accepts
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
speaker `A`. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing `"text":"expected utterance"` followed
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
normalization. The new decoder passes `other metadata`. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
**Required change:** Preserve the effective-member behavior for existing
inputs without reviving whole-transcript generic decoding or float conversion
for IDs. Keep the compatibility handling focused on this changed boundary.
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
in the effective member, plus the case-variant example in both member orders.
Keep the existing exact-ID and unknown-field tests for both input envelopes.
### R5 — Align Roadmap Status and Runtime Guidance (P3)
**Affected plan:** Stages 15, 19, and 20.
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
describes the implementation as pending despite the completed sprint.
The [pipeline guide](../internal/pipeline.md) says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
literal `\n` between “selected” and “profiles.”
**Required change:** Mark the roadmap with the landed sprint and this review,
while retaining its historical baseline context. Describe interruption detail
in terms of outcomes actually available and selected for retention. Fix the
CLI sentence and keep public guidance focused on the operator-visible behavior.
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
reflects the evidence modes, and the CLI paragraph renders normally.
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
**Affected plan:** Stages 12 and 19.
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
calls [progress rendering](../../internal/cli/process_progress.go#L34)
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
PromptKit v0.23.1 starts the operation deadline before emitting the start
event, so a blocked or slow write to CommandKit's diagnostic stream spends
that operation's budget before the provider request starts.
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported `operation_timeout`, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
**Required change:** Deliver progress outside PromptKit's timed model-call
observation path. Keep native observations for execution facts and use
application lifecycle boundaries for synchronous progress delivery, or another
delivery mechanism that keeps stderr waits out of the per-operation budget.
Preserve CommandKit's routing and presentation status, and keep the parent
context authoritative for cancellation of the invocation.
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
Verify that it still dispatches and applies the same correction within the
model's own budget, and retain live progress and presentation-failure coverage.
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
**Affected plan:** Stages 1, 11, and 20.
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing [process entry point](../../internal/cli/process.go#L27)
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
unconditionally returns a usage outcome for those errors.
**Reproduced behavior:** A readable main file beside an unreadable automatic
`prompts.yml` produces `source.permission` in all three commands.
`promptkit list prompts` and `config show --scope promptkit` return status 1.
`process` returns status 2 and prints usage guidance for the same loading
failure.
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
**Acceptance:** Compare `process` and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
### R8 — Use the Same Span Applicability With and Without Repair (P3)
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
rejects an ambiguous require-unique span only when independent repair evidence
is enabled. Without repair, the existing shortlist forwards the candidate to
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
which records approval even when it cannot obtain a usable preview. The
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
later rejects that same local ambiguity before dispatching a provider call.
**Reproduced behavior:** In the built-in grammar stage, offer to replace
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
protected. With a zero repair budget, the recorded decisions show protection
approval and `editorial_review` rejection for `ambiguous_original_text`.
With repair enabled, `original_text_presence` records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
**Required change:** Give the local applicability check ownership of
require-unique ambiguity in both paths. Keep independent evidence and repair
attempts conditional on the configured budget. Retain the ability to assess
protection when a located edit produces an empty preview.
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
Verify consistent local attribution, no unsupported protection approval,
and no model validation call for the inapplicable candidate. Keep repair
available in the positive-budget case.