Record reliability integration audit findings
This commit is contained in:
@@ -2,9 +2,10 @@
|
||||
|
||||
**Reviewed revision:** `e8c0708`
|
||||
|
||||
**Status:** Five open findings from the review of all 20
|
||||
**Status:** Eight open findings from the review of all 20
|
||||
[implementation stages](implementation.md) against the
|
||||
[target roadmap](reliability-and-maintainability.md).
|
||||
[target roadmap](reliability-and-maintainability.md) and the follow-up audit
|
||||
of how those stages interact with existing callers and contracts.
|
||||
|
||||
## Review Result
|
||||
|
||||
@@ -28,6 +29,11 @@ commands and processing cases in the plan. Focused review probes reproduced
|
||||
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
|
||||
documentation.
|
||||
|
||||
The follow-up audit traced the new paths through CLI dispatch, the built-in
|
||||
validator chains, model execution, publication, and report and debug readers.
|
||||
Additional probes reproduced the case-variant input behavior in R4 and the
|
||||
cross-component failures in R6–R8.
|
||||
|
||||
## Findings
|
||||
|
||||
### R1 — Include Later Delivery Failures in Writable Reports (P2)
|
||||
@@ -121,13 +127,20 @@ inputs while improving numeric fidelity.
|
||||
speaker `A`. The sprint revision rejects the same input because of the
|
||||
superseded numeric value.
|
||||
|
||||
The follow-up audit also found a silent change when known member names differ
|
||||
only by case. For a segment containing `"text":"expected utterance"` followed
|
||||
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
|
||||
normalization. The new decoder passes `other metadata`. Both accept the input.
|
||||
The old map round trip ordered the member names before typed decoding. Direct
|
||||
decoding applies them in input order.
|
||||
|
||||
**Required change:** Preserve the effective-member behavior for existing
|
||||
inputs without reviving whole-transcript generic decoding or float conversion
|
||||
for IDs. Keep the compatibility handling focused on this changed boundary.
|
||||
|
||||
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
|
||||
in the effective member. Keep the existing exact-ID and unknown-field tests
|
||||
for both input envelopes.
|
||||
in the effective member, plus the case-variant example in both member orders.
|
||||
Keep the existing exact-ID and unknown-field tests for both input envelopes.
|
||||
|
||||
### R5 — Align Roadmap Status and Runtime Guidance (P3)
|
||||
|
||||
@@ -148,3 +161,86 @@ CLI sentence and keep public guidance focused on the operator-visible behavior.
|
||||
|
||||
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
|
||||
reflects the evidence modes, and the CLI paragraph renders normally.
|
||||
|
||||
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
|
||||
|
||||
**Affected plan:** Stages 12 and 19.
|
||||
|
||||
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
|
||||
calls [progress rendering](../../internal/cli/process_progress.go#L34)
|
||||
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
|
||||
PromptKit v0.23.1 starts the operation deadline before emitting the start
|
||||
event, so a blocked or slow write to CommandKit's diagnostic stream spends
|
||||
that operation's budget before the provider request starts.
|
||||
|
||||
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
|
||||
loopback provider, the ordinary run dispatched three requests and changed
|
||||
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
|
||||
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
|
||||
reported `operation_timeout`, and returned the uncorrected transcript with
|
||||
status 0 under the existing recoverable-failure policy. The writer eventually
|
||||
succeeded in delivering the message.
|
||||
|
||||
**Required change:** Deliver progress outside PromptKit's timed model-call
|
||||
observation path. Keep native observations for execution facts and use
|
||||
application lifecycle boundaries for synchronous progress delivery, or another
|
||||
delivery mechanism that keeps stderr waits out of the per-operation budget.
|
||||
Preserve CommandKit's routing and presentation status, and keep the parent
|
||||
context authoritative for cancellation of the invocation.
|
||||
|
||||
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
|
||||
Verify that it still dispatches and applies the same correction within the
|
||||
model's own budget, and retain live progress and presentation-failure coverage.
|
||||
|
||||
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
|
||||
|
||||
**Affected plan:** Stages 1, 11, and 20.
|
||||
|
||||
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
|
||||
configuration diagnostics to distinguish invalid content from operational
|
||||
errors. The existing [process entry point](../../internal/cli/process.go#L27)
|
||||
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
|
||||
unconditionally returns a usage outcome for those errors.
|
||||
|
||||
**Reproduced behavior:** A readable main file beside an unreadable automatic
|
||||
`prompts.yml` produces `source.permission` in all three commands.
|
||||
`promptkit list prompts` and `config show --scope promptkit` return status 1.
|
||||
`process` returns status 2 and prints usage guidance for the same loading
|
||||
failure.
|
||||
|
||||
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
|
||||
policy at the processing configuration boundary. Preserve typed causes and
|
||||
source diagnostics for both outcome classes.
|
||||
|
||||
**Acceptance:** Compare `process` and representative delegated commands for an
|
||||
operational acquisition failure and malformed YAML. Check status 1 and status
|
||||
2 respectively, and preserve cancellation precedence and the diagnostic source.
|
||||
|
||||
### R8 — Use the Same Span Applicability With and Without Repair (P3)
|
||||
|
||||
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
|
||||
|
||||
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
|
||||
rejects an ambiguous require-unique span only when independent repair evidence
|
||||
is enabled. Without repair, the existing shortlist forwards the candidate to
|
||||
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
|
||||
which records approval even when it cannot obtain a usable preview. The
|
||||
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
|
||||
later rejects that same local ambiguity before dispatching a provider call.
|
||||
|
||||
**Reproduced behavior:** In the built-in grammar stage, offer to replace
|
||||
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
|
||||
protected. With a zero repair budget, the recorded decisions show protection
|
||||
approval and `editorial_review` rejection for `ambiguous_original_text`.
|
||||
With repair enabled, `original_text_presence` records the ambiguity and
|
||||
protection emits no decision. Neither run applies the ambiguous proposal.
|
||||
|
||||
**Required change:** Give the local applicability check ownership of
|
||||
require-unique ambiguity in both paths. Keep independent evidence and repair
|
||||
attempts conditional on the configured budget. Retain the ability to assess
|
||||
protection when a located edit produces an empty preview.
|
||||
|
||||
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
|
||||
Verify consistent local attribution, no unsupported protection approval,
|
||||
and no model validation call for the inapplicable candidate. Keep repair
|
||||
available in the positive-budget case.
|
||||
|
||||
Reference in New Issue
Block a user