From 1c90fedb1e8999a2c5a8344f9fb11c93dece53a0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 17 Nov 2026 02:00:31 +0000 Subject: [PATCH] Record reliability integration audit findings --- docs/roadmap/remediation.md | 104 ++++++++++++++++++++++++++++++++++-- 1 file changed, 100 insertions(+), 4 deletions(-) diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index d2bedee..159d330 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -2,9 +2,10 @@ **Reviewed revision:** `e8c0708` -**Status:** Five open findings from the review of all 20 +**Status:** Eight open findings from the review of all 20 [implementation stages](implementation.md) against the -[target roadmap](reliability-and-maintainability.md). +[target roadmap](reliability-and-maintainability.md) and the follow-up audit +of how those stages interact with existing callers and contracts. ## Review Result @@ -28,6 +29,11 @@ commands and processing cases in the plan. Focused review probes reproduced R1, R3, and R4. R2 and R5 follow from the versioned output contract and current documentation. +The follow-up audit traced the new paths through CLI dispatch, the built-in +validator chains, model execution, publication, and report and debug readers. +Additional probes reproduced the case-variant input behavior in R4 and the +cross-component failures in R6–R8. + ## Findings ### R1 — Include Later Delivery Failures in Writable Reports (P2) @@ -121,13 +127,20 @@ inputs while improving numeric fidelity. speaker `A`. The sprint revision rejects the same input because of the superseded numeric value. +The follow-up audit also found a silent change when known member names differ +only by case. For a segment containing `"text":"expected utterance"` followed +by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to +normalization. The new decoder passes `other metadata`. Both accept the input. +The old map round trip ordered the member names before typed decoding. Direct +decoding applies them in input order. + **Required change:** Preserve the effective-member behavior for existing inputs without reviving whole-transcript generic decoding or float conversion for IDs. Keep the compatibility handling focused on this changed boundary. **Acceptance:** Cover an earlier superseded type mismatch and a type mismatch -in the effective member. Keep the existing exact-ID and unknown-field tests -for both input envelopes. +in the effective member, plus the case-variant example in both member orders. +Keep the existing exact-ID and unknown-field tests for both input envelopes. ### R5 — Align Roadmap Status and Runtime Guidance (P3) @@ -148,3 +161,86 @@ CLI sentence and keep public guidance focused on the operator-visible behavior. **Acceptance:** The roadmap and implementation status agree, the pipeline guide reflects the evidence modes, and the CLI paragraph renders normally. + +### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2) + +**Affected plan:** Stages 12 and 19. + +**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30) +calls [progress rendering](../../internal/cli/process_progress.go#L34) +synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139). +PromptKit v0.23.1 starts the operation deadline before emitting the start +event, so a blocked or slow write to CommandKit's diagnostic stream spends +that operation's budget before the provider request starts. + +**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate +loopback provider, the ordinary run dispatched three requests and changed +`hello world` to `Hello world`. With `--progress` and a diagnostic writer that +delayed the proposal-start message by 750 ms, Audita dispatched zero requests, +reported `operation_timeout`, and returned the uncorrected transcript with +status 0 under the existing recoverable-failure policy. The writer eventually +succeeded in delivering the message. + +**Required change:** Deliver progress outside PromptKit's timed model-call +observation path. Keep native observations for execution facts and use +application lifecycle boundaries for synchronous progress delivery, or another +delivery mechanism that keeps stderr waits out of the per-operation budget. +Preserve CommandKit's routing and presentation status, and keep the parent +context authoritative for cancellation of the invocation. + +**Acceptance:** Vary only the delay on the progress stream of a controlled run. +Verify that it still dispatches and applies the same correction within the +model's own budget, and retain live progress and presentation-failure coverage. + +### R7 — Classify Configuration Acquisition Errors Consistently (P2) + +**Affected plan:** Stages 1, 11, and 20. + +**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use +configuration diagnostics to distinguish invalid content from operational +errors. The existing [process entry point](../../internal/cli/process.go#L27) +uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then +unconditionally returns a usage outcome for those errors. + +**Reproduced behavior:** A readable main file beside an unreadable automatic +`prompts.yml` produces `source.permission` in all three commands. +`promptkit list prompts` and `config show --scope promptkit` return status 1. +`process` returns status 2 and prints usage guidance for the same loading +failure. + +**Required change:** Use the shared `ckconfig.ClassifyConfigurationError` +policy at the processing configuration boundary. Preserve typed causes and +source diagnostics for both outcome classes. + +**Acceptance:** Compare `process` and representative delegated commands for an +operational acquisition failure and malformed YAML. Check status 1 and status +2 respectively, and preserve cancellation precedence and the diagnostic source. + +### R8 — Use the Same Span Applicability With and Without Repair (P3) + +**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17. + +**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38) +rejects an ambiguous require-unique span only when independent repair evidence +is enabled. Without repair, the existing shortlist forwards the candidate to +the [protected-term validator](../../internal/framework/validators/deterministic.go#L106), +which records approval even when it cannot obtain a usable preview. The +[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251) +later rejects that same local ambiguity before dispatching a provider call. + +**Reproduced behavior:** In the built-in grammar stage, offer to replace +`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor` +protected. With a zero repair budget, the recorded decisions show protection +approval and `editorial_review` rejection for `ambiguous_original_text`. +With repair enabled, `original_text_presence` records the ambiguity and +protection emits no decision. Neither run applies the ambiguous proposal. + +**Required change:** Give the local applicability check ownership of +require-unique ambiguity in both paths. Keep independent evidence and repair +attempts conditional on the configured budget. Retain the ability to assess +protection when a located edit produces an empty preview. + +**Acceptance:** Use the built-in chain with zero and positive repair budgets. +Verify consistent local attribution, no unsupported protection approval, +and no model validation call for the inapplicable candidate. Keep repair +available in the positive-budget case.