Record reliability integration audit findings

This commit is contained in:
2026-11-17 02:00:31 +00:00
parent 087e388a03
commit 1c90fedb1e

View File

@@ -2,9 +2,10 @@
**Reviewed revision:** `e8c0708`
**Status:** Five open findings from the review of all 20
**Status:** Eight open findings from the review of all 20
[implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md).
[target roadmap](reliability-and-maintainability.md) and the follow-up audit
of how those stages interact with existing callers and contracts.
## Review Result
@@ -28,6 +29,11 @@ commands and processing cases in the plan. Focused review probes reproduced
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in
validator chains, model execution, publication, and report and debug readers.
Additional probes reproduced the case-variant input behavior in R4 and the
cross-component failures in R6–R8.
## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2)
@@ -121,13 +127,20 @@ inputs while improving numeric fidelity.
speaker `A`. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing `"text":"expected utterance"` followed
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
normalization. The new decoder passes `other metadata`. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
**Required change:** Preserve the effective-member behavior for existing
inputs without reviving whole-transcript generic decoding or float conversion
for IDs. Keep the compatibility handling focused on this changed boundary.
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
in the effective member. Keep the existing exact-ID and unknown-field tests
for both input envelopes.
in the effective member, plus the case-variant example in both member orders.
Keep the existing exact-ID and unknown-field tests for both input envelopes.
### R5 — Align Roadmap Status and Runtime Guidance (P3)
@@ -148,3 +161,86 @@ CLI sentence and keep public guidance focused on the operator-visible behavior.
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
reflects the evidence modes, and the CLI paragraph renders normally.
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
**Affected plan:** Stages 12 and 19.
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
calls [progress rendering](../../internal/cli/process_progress.go#L34)
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
PromptKit v0.23.1 starts the operation deadline before emitting the start
event, so a blocked or slow write to CommandKit's diagnostic stream spends
that operation's budget before the provider request starts.
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported `operation_timeout`, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
**Required change:** Deliver progress outside PromptKit's timed model-call
observation path. Keep native observations for execution facts and use
application lifecycle boundaries for synchronous progress delivery, or another
delivery mechanism that keeps stderr waits out of the per-operation budget.
Preserve CommandKit's routing and presentation status, and keep the parent
context authoritative for cancellation of the invocation.
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
Verify that it still dispatches and applies the same correction within the
model's own budget, and retain live progress and presentation-failure coverage.
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
**Affected plan:** Stages 1, 11, and 20.
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing [process entry point](../../internal/cli/process.go#L27)
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
unconditionally returns a usage outcome for those errors.
**Reproduced behavior:** A readable main file beside an unreadable automatic
`prompts.yml` produces `source.permission` in all three commands.
`promptkit list prompts` and `config show --scope promptkit` return status 1.
`process` returns status 2 and prints usage guidance for the same loading
failure.
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
**Acceptance:** Compare `process` and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
### R8 — Use the Same Span Applicability With and Without Repair (P3)
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
rejects an ambiguous require-unique span only when independent repair evidence
is enabled. Without repair, the existing shortlist forwards the candidate to
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
which records approval even when it cannot obtain a usable preview. The
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
later rejects that same local ambiguity before dispatching a provider call.
**Reproduced behavior:** In the built-in grammar stage, offer to replace
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
protected. With a zero repair budget, the recorded decisions show protection
approval and `editorial_review` rejection for `ambiguous_original_text`.
With repair enabled, `original_text_presence` records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
**Required change:** Give the local applicability check ownership of
require-unique ambiguity in both paths. Keep independent evidence and repair
attempts conditional on the configured budget. Retain the ability to assess
protection when a located edit produces an empty preview.
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
Verify consistent local attribution, no unsupported protection approval,
and no model validation call for the inapplicable candidate. Keep repair
available in the positive-budget case.