13 KiB
Reliability and Maintainability Review Remediation
Reviewed revision: e8c0708
Status: Eight open findings from the review of all 20 implementation stages against the target roadmap and the follow-up audit of how those stages interact with existing callers and contracts.
Review Result
The sprint established the target architecture. ConfigKit and native CommandKit integrations handle configuration and inspection, and preflight hands checked processing selections to the runner. Audita decodes and normalizes transcripts directly and partitions them once. A bounded section window allows proposal, validation, and repair to overlap within a stage. The runner assigns final proposal indices in section order and commits edits at the stage boundary. PromptKit owns model execution, backend admission, retries, and repair history. Required publication uses checked destinations. Ordinary reports use typed processing facts, while optional debug capture retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks for the runner, LLM adapter, validators, report projection, diagnostics, glossary, and CLI. The ordered repair artifact test also passed 150 repetitions under the race detector. A built CLI returned status 2 with source diagnostics for malformed PromptKit and application configuration across the delegated commands and processing cases in the plan. Focused review probes reproduced R1, R3, and R4. R2 and R5 follow from the versioned output contract and current documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in validator chains, model execution, publication, and report and debug readers. Additional probes reproduced the case-variant input behavior in R4 and the cross-component failures in R6–R8.
Findings
R1 — Include Later Delivery Failures in Writable Reports (P2)
Affected plan: Stage 4, item 3, and Stage 17, item 1.
Evidence: Report publication renders a failed bundle using the failures accumulated at that point. Later writes append to the returned error list, but the remaining reports and the best-effort pass reuse the same bytes. The report projection can include secondary file failures only if publication supplies them.
Reproduced behavior: Start finalization with a processing failure. Make
errors.yml and recoveries.yml unwritable and let report.md succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
Required change: Use the freshest observed failures at the report publication boundary and make a bounded best-effort update of already-written reports when their failure evidence becomes stale. Retain existing evidence if rendering the updated failed bundle fails. Keep transcript-first, nontransactional publication.
Acceptance: Extend the output-session regression to inspect the facts rendered into a surviving report when an earlier failure is followed by a second write failure. Check failure attribution and counts as well as status.
R2 — Preserve the Version 1 Error Record Discriminator (P2)
Affected plan: Stage 17, item 1.
Evidence: Report projection
adds a top-level scope: interrupted record to the
version 1 error document.
The baseline contract described work_unit and run records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's release policy
also preserves the documented output contract for minor and patch releases.
Required change: Keep the existing top-level variants. Carry available peer interruption details as optional evidence on an existing record, for example an optional sequence on the initiating run failure. Keep the primary origin and peer origins separately identifiable and preserve the work-unit and run-error counts. Treat a new top-level variant as an explicit contract change if that representation is preferred.
Acceptance: Exercise a fatal operation and interrupted peer in one run. Check that a consumer of the existing record variants can still read the report, and that the peer's available phase, reason, and origin remain present.
R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
Affected plan: Stage 12, item 2, and Stage 17, item 1.
Evidence: Operation projection
uses internal_failure when a typed model failure has no reason and retains
its original class and cause. Run-failure projection
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
Reproduced behavior: A typed fatal failure with an empty reason and a
wrapped context.DeadlineExceeded is recorded on the operation as
fatal / internal_failure and serialized in the run report as
canceled. The cause is available, but it does not make this a parent
cancellation.
Required change: Share the typed reason projection and its empty-reason fallback. Derive a reason from the raw cause only when a typed failure is absent.
Acceptance: Verify matching fallbacks for the operation and the run record and retain the phase, origin, and original cause. Keep genuine parent cancellation distinct.
R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
Affected plan: Stage 5, item 1.
Evidence: The baseline selected the final value of an exact duplicate JSON member when it decoded the segment into a generic object, before applying field types. Typed transcript decoding now applies the field type to every occurrence. An earlier type error survives even when the final value is valid. The refactor was meant to retain accepted inputs while improving numeric fidelity.
Reproduced behavior: The baseline at d226bdd accepts
[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}] and reads
speaker A. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing "text":"expected utterance" followed
by "TEXT":"other metadata", the baseline passes expected utterance to
normalization. The new decoder passes other metadata. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
Required change: Preserve the effective-member behavior for existing inputs without reviving whole-transcript generic decoding or float conversion for IDs. Keep the compatibility handling focused on this changed boundary.
Acceptance: Cover an earlier superseded type mismatch and a type mismatch in the effective member, plus the case-variant example in both member orders. Keep the existing exact-ID and unknown-field tests for both input envelopes.
R5 — Align Roadmap Status and Runtime Guidance (P3)
Affected plan: Stages 15, 19, and 20.
Evidence: The feature roadmap still
describes the implementation as pending despite the completed sprint.
The pipeline guide says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The CLI guide contains a
literal \n between “selected” and “profiles.”
Required change: Mark the roadmap with the landed sprint and this review, while retaining its historical baseline context. Describe interruption detail in terms of outcomes actually available and selected for retention. Fix the CLI sentence and keep public guidance focused on the operator-visible behavior.
Acceptance: The roadmap and implementation status agree, the pipeline guide reflects the evidence modes, and the CLI paragraph renders normally.
R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
Affected plan: Stages 12 and 19.
Evidence: Pipeline wiring calls progress rendering synchronously from the adapter's native observer. PromptKit v0.23.1 starts the operation deadline before emitting the start event, so a blocked or slow write to CommandKit's diagnostic stream spends that operation's budget before the provider request starts.
Reproduced behavior: With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
hello world to Hello world. With --progress and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported operation_timeout, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
Required change: Deliver progress outside PromptKit's timed model-call observation path. Keep native observations for execution facts and use application lifecycle boundaries for synchronous progress delivery, or another delivery mechanism that keeps stderr waits out of the per-operation budget. Preserve CommandKit's routing and presentation status, and keep the parent context authoritative for cancellation of the invocation.
Acceptance: Vary only the delay on the progress stream of a controlled run. Verify that it still dispatches and applies the same correction within the model's own budget, and retain live progress and presentation-failure coverage.
R7 — Classify Configuration Acquisition Errors Consistently (P2)
Affected plan: Stages 1, 11, and 20.
Evidence: The adopted ckconfig v0.8.0 and ckprompt v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing process entry point
uses ckconfig.AdaptDiagnostics for every ConfigKit.LoadError, then
unconditionally returns a usage outcome for those errors.
Reproduced behavior: A readable main file beside an unreadable automatic
prompts.yml produces source.permission in all three commands.
promptkit list prompts and config show --scope promptkit return status 1.
process returns status 2 and prints usage guidance for the same loading
failure.
Required change: Use the shared ckconfig.ClassifyConfigurationError
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
Acceptance: Compare process and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
R8 — Use the Same Span Applicability With and Without Repair (P3)
Affected plan: Stages 8 and 9, with ledger projection in Stage 17.
Evidence: The original-text validator rejects an ambiguous require-unique span only when independent repair evidence is enabled. Without repair, the existing shortlist forwards the candidate to the protected-term validator, which records approval even when it cannot obtain a usable preview. The LLM validator's request builder later rejects that same local ambiguity before dispatching a provider call.
Reproduced behavior: In the built-in grammar stage, offer to replace
Mage Armor with ward in Mage Armor and Mage Armor, with Mage Armor
protected. With a zero repair budget, the recorded decisions show protection
approval and editorial_review rejection for ambiguous_original_text.
With repair enabled, original_text_presence records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
Required change: Give the local applicability check ownership of require-unique ambiguity in both paths. Keep independent evidence and repair attempts conditional on the configured budget. Retain the ability to assess protection when a located edit produces an empty preview.
Acceptance: Use the built-in chain with zero and positive repair budgets. Verify consistent local attribution, no unsupported protection approval, and no model validation call for the inapplicable candidate. Keep repair available in the positive-budget case.