Record error handling audit findings
This commit is contained in:
249
docs/roadmap/audit/06-error-handling.md
Normal file
249
docs/roadmap/audit/06-error-handling.md
Normal file
@@ -0,0 +1,249 @@
|
||||
# Error-handling audit
|
||||
|
||||
Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt
|
||||
configuration, transcript parsing and chunking, model and validator execution,
|
||||
correction repair, output publication, and release bootstrap. It used the
|
||||
checked-in contracts, the pinned dependency source, existing tests, and
|
||||
disposable failure probes.
|
||||
|
||||
## Architecture and intended behavior
|
||||
|
||||
Audita is a single-process Go CLI. CommandKit routes commands and presents
|
||||
outcomes. ConfigKit loads and validates application settings. On `process`, the
|
||||
CLI parses the transcript and prepares the selected outputs before normalizing
|
||||
and running the pipeline. PromptKit owns preparation, backend admission, HTTP,
|
||||
transient retry, schema validation, and structural repair. Audita's role adapter
|
||||
maps those errors to a class and a safe reason while preserving the typed cause.
|
||||
The runner isolates recoverable operations, cancels and joins peers after a
|
||||
fatal error, and returns partial facts for reports. The CLI checks for
|
||||
cancellation, delivers the transcript and reports, and selects the exit status.
|
||||
[CLI outcomes](../../cli.md#inspection-and-outcomes)
|
||||
· [Pipeline](../../internal/pipeline.md)
|
||||
· [Model integration](../../integrations/openai-compatible-llm.md)
|
||||
· [Reporting and output ownership](../../internal/diagnostics-reporting.md)
|
||||
|
||||
The audit treated recoverable model errors, rejected corrections, and unavailable
|
||||
validator assessments according to their separate contracts. Existing tests
|
||||
confirm that one unavailable validation batch can leave other batches and later
|
||||
modules usable, and that parent cancellation can stop active and waiting work
|
||||
without publishing a transcript. Optional debug warnings can accompany exit
|
||||
`0`, and a transcript can remain published if a later report write fails. These
|
||||
are selected behaviors. The report boundary intentionally omits raw provider
|
||||
causes, and input failures before initialization can leave no reports.
|
||||
[Validation contract](../../internal/validators.md#llm-validator-batching)
|
||||
· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091)
|
||||
· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
|
||||
· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179)
|
||||
· [Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
· [Early failure contract](../../integrations/subprocess.md#result-handling)
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Finalization publishes completed reports after it knows delivery failed
|
||||
|
||||
The CLI prepares a `completed` bundle for a successful pipeline and passes it
|
||||
to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output
|
||||
session before writing reports. Even if a transcript write or cleanup has
|
||||
already failed, it writes the initial `completed` bundle and only then asks for
|
||||
and writes a failed-status bundle. A disposable probe supplied a short stdout
|
||||
writer. The failed-bundle callback received a known transcript write failure
|
||||
and could already read `completed` in all three report files. `Finish` returned
|
||||
the short-write error and the second pass rewrote the reports to `failed`.
|
||||
[CLI bundle selection](../../../internal/cli/process_output.go#L27)
|
||||
· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273)
|
||||
· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318)
|
||||
|
||||
Reports should reflect a failure that is known before the first report write.
|
||||
The existing nontransactional contract allows a later report failure to leave
|
||||
mixed evidence. In this case the initial success claim is avoidable, and a
|
||||
reader can observe it or it can persist if the failure rewrite cannot finish.
|
||||
Select the failed bundle before publishing reports when transcript delivery or
|
||||
cleanup has already failed. Keep a best-effort rewrite for errors discovered
|
||||
during report publication, retaining the original error and any later failures.
|
||||
If a failed bundle cannot be rendered, surface that error and preserve the
|
||||
last written evidence. The initialization callback currently discards its
|
||||
rendering error, and the finalization callback falls back to the previous
|
||||
bundle, which can still say `completed`.
|
||||
The missing phase for delivery errors is a separate, previously recorded issue.
|
||||
[Output lifecycle contract](../../operations.md#outputs-and-lifecycle)
|
||||
· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105)
|
||||
· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46)
|
||||
· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322)
|
||||
· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records)
|
||||
|
||||
### 2. A provider request timeout is reported as the logical operation deadline
|
||||
|
||||
The profile's `timeout_seconds` bounds one provider attempt. The distinct
|
||||
`llm.operation_timeout` applies to preparation, capacity waiting, retries,
|
||||
generation, and structural repair. Pinned PromptKit deliberately reserves
|
||||
`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as
|
||||
`ErrTransport` with the underlying deadline cause. After checking the native
|
||||
operation sentinel, Audita checks a generic `context.DeadlineExceeded` before
|
||||
`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason.
|
||||
In PromptKit, the distinction is documented in `engine.go` and implemented in
|
||||
`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`.
|
||||
[Timeout settings](../../migrations/v2.md)
|
||||
· [Native runtime contract](../../integrations/openai-compatible-llm.md)
|
||||
· [Classifier](../../../internal/framework/llm/adapter.go#L228)
|
||||
· [Pinned PromptKit](../../../go.mod#L11)
|
||||
|
||||
A loopback provider withheld its response beyond a one-second profile timeout.
|
||||
With a ten-second operation budget and zero retries, the CLI finished after
|
||||
about one second. It returned `0` with a completed transcript and reported
|
||||
the failed proposal as `operation_timeout`. Continuing the run is consistent
|
||||
with both timeout classes being recoverable. The reason directs an operator
|
||||
toward the logical operation budget when the attempt deadline expired. Use
|
||||
the native sentinel for whole-operation timeouts, preserving parent cancellation
|
||||
precedence. Classify an HTTP-only timeout as transport failure or adopt a
|
||||
distinct safe reason for that cause if consumers need the distinction.
|
||||
[Report message mapping](../../../internal/framework/processreport/report.go#L224)
|
||||
· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
|
||||
|
||||
### 3. Missing required credentials produce a generic setup message
|
||||
|
||||
The pinned PromptKit use case identifies a missing required environment
|
||||
credential through `ErrAPIKeyEnvMissing` and retains its configured variable
|
||||
name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a
|
||||
generic safe message before delivering the operation result to the domain. A
|
||||
synthetic profile with a required unset variable stopped before HTTP and
|
||||
exited `1`. Stderr ended with `structured model setup failed`, the YAML report said
|
||||
`Processing setup failed.`, and the interaction error artifact repeated the
|
||||
generic adapter message. The cause is available to Go callers via `Unwrap`,
|
||||
but the operator-facing process path does not report the missing requirement.
|
||||
[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243)
|
||||
· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17)
|
||||
· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231)
|
||||
· [Final CLI failure](../../../internal/cli/process_output.go#L61)
|
||||
|
||||
The fatal class and exclusion of raw provider text are appropriate. The safe
|
||||
diagnostic can still identify an unset required credential and the affected role
|
||||
or profile. Map the native credential sentinel to an actionable safe message,
|
||||
and include the configured variable name if obtained from safe profile or
|
||||
backend metadata. Carry the typed cause internally and retain preparation at
|
||||
the existing execution boundary, where role selection and credential validation
|
||||
already occur.
|
||||
[Credential ownership](../../internal/llm-runtime.md)
|
||||
· [Recovery workflow](../../operations.md#recovery)
|
||||
|
||||
### 4. Fatal model errors lose their chunk and operation coordinates
|
||||
|
||||
Proposal and validation code records an operation reference even on fatal
|
||||
returns. It can identify the section, role, validator, and batch. The runner
|
||||
stores these operations per work unit, but its first-error path retains only
|
||||
the error value. `executePipeline` reconstructs the fatal report origin from
|
||||
the failed module without identifying which section or operation stopped it.
|
||||
The report builder supports these coordinates for run-scope records. In the
|
||||
missing-credential probe the run record included `grammar_1` and the stage
|
||||
index, with no `chunk_index` or `operation`, even though the pre-dispatch
|
||||
proposal invocation had both. Fatal validator batches lose their available
|
||||
validator and batch coordinates at the same boundary.
|
||||
[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116)
|
||||
· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176)
|
||||
· [First error and cancellation](../../../internal/framework/runner/runner.go#L477)
|
||||
· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60)
|
||||
· [Run origin serialization](../../../internal/framework/processreport/report.go#L155)
|
||||
|
||||
Retain the primary error and its available `outcomes.Reference` together when
|
||||
the runner selects the failure that stops the module. This lets the CLI identify
|
||||
the failed operation without guessing from canceled peers or counting the fatal
|
||||
error again as a recoverable work-unit error. Use module-level context for
|
||||
setup and application errors with no narrower origin.
|
||||
[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records)
|
||||
|
||||
### 5. Invalid processing selections use the runtime failure exit status
|
||||
|
||||
The CLI contract gives exit `2` to invalid input or usage and exit `1` to
|
||||
runtime or output failure. `loadProcessInputs` maps malformed input and
|
||||
configuration to usage errors. Processing readiness is checked later inside
|
||||
`preflight.Prepare`, where a missing selected profile, a profile reference
|
||||
that cannot be resolved, or an empty selected glossary for a required glossary
|
||||
module can fail. `runProcess` treats every error at this boundary as a runtime
|
||||
failure, together with output and cancellation errors. A probe selecting an
|
||||
unknown proposal profile returned `1` with a specific profile diagnostic, no
|
||||
provider calls, and no reports or transcript. The existing integration test
|
||||
for a similarly invalid profile inheritance reference expects `2`.
|
||||
[Exit status contract](../../cli.md#inspection-and-outcomes)
|
||||
· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41)
|
||||
· [Selection validation](../../../internal/framework/preflight/preflight.go#L96)
|
||||
· [CLI status selection](../../../internal/cli/process.go#L24)
|
||||
· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122)
|
||||
|
||||
Distinguish rejected processing selections from context and destination failures
|
||||
with a typed readiness error or a separate input-check step. Return usage for
|
||||
invalid operator selections and preserve failure for cancellation, I/O, and
|
||||
internal defects. Retain the existing early check and the ability to inspect
|
||||
configuration without requiring a processing-ready pipeline.
|
||||
[Inspection contract](../../cli.md#inspection-and-outcomes)
|
||||
|
||||
### 6. An infeasible chunk target is recorded as an internal processing failure
|
||||
|
||||
A positive `chunking.target_sections` can be valid configuration and infeasible
|
||||
for the normalized input. The chunker rejects that combination and the runner
|
||||
wraps the error with `chunk normalized transcript`. The CLI's run record defaults
|
||||
to `phase: processing`, `reason: internal_failure`, and a generic message for
|
||||
errors outside the structured model and cancellation classes. With one usable
|
||||
segment and an explicit target of two, a CLI probe produced that failed report,
|
||||
exited `1`, and made no model calls. This failure should remain visible when
|
||||
reports are enabled because it occurs after initialization. The reported phase
|
||||
and reason currently misidentify an operator-correctable constraint as an
|
||||
internal defect in processing.
|
||||
[Chunking settings](../../config.md#settings)
|
||||
· [Feasibility check](../../../internal/core/chunking/sections.go#L143)
|
||||
· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126)
|
||||
· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60)
|
||||
· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244)
|
||||
|
||||
Carry a typed chunking failure through the runner and map infeasible parameters
|
||||
to `phase: chunking` and an input reason. Keep a separate classification for an
|
||||
unexpected failure in the chunker. Align the exit mapping with the input
|
||||
contract if the run distinguishes these classes at that boundary. The earlier
|
||||
correctness finding about feasible targets being rejected concerns a separate
|
||||
chunking decision.
|
||||
[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records)
|
||||
· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets)
|
||||
|
||||
### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty
|
||||
|
||||
The release job checks `test -z "$(git status --porcelain --untracked-files=all)"`
|
||||
under `set -eu`. The `test` command's exit status controls whether the shell
|
||||
continues. If Git emits a diagnostic on stderr and fails with empty stdout,
|
||||
the test succeeds. A shell probe substituted a status command that exited
|
||||
`128` with no stdout and confirmed that execution continued after the check.
|
||||
The expected bootstrap behavior is to stop before dependency preparation if
|
||||
cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared`
|
||||
independently checks the status command's error before delivery, limiting this
|
||||
finding to the earlier bootstrap gate and diagnosis.
|
||||
[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17)
|
||||
· [Release procedure](../../release.md#binary-delivery)
|
||||
· [Pinned ReleaseKit](../../../go.mod#L12)
|
||||
|
||||
Assign the status output in a separate command and check the command result
|
||||
before testing the content. Preserve the existing verification in ReleaseKit.
|
||||
|
||||
## Related failure paths already recorded
|
||||
|
||||
- Cancellation during FIFO or stdout delivery can be missed after the last
|
||||
context check. A blocked FIFO input can also defer cancellation before
|
||||
output preflight. The runner has a separate cancellation window before
|
||||
committing module changes.
|
||||
[Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
|
||||
· [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
|
||||
· [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary)
|
||||
- The failure path can drop completed proposal evidence behind an earlier
|
||||
section. The internal preflight seam can also overwrite initialized reports
|
||||
with empty files when a failed renderer is absent.
|
||||
[Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
|
||||
· [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence)
|
||||
- Output cleanup can discard removal errors or forget a staging path before a
|
||||
later cleanup attempt. This differs from intentionally retaining requested
|
||||
debug evidence or keeping a transcript already published successfully.
|
||||
[Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
|
||||
|
||||
## Verification
|
||||
|
||||
The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a
|
||||
temporary CLI build, a synthetic loopback provider, a disposable output-session
|
||||
caller, and a shell simulation. They verified the failure conditions described
|
||||
above without persisting tests or program changes. The release check was
|
||||
reviewed locally against the pinned ReleaseKit implementation and exercised at
|
||||
the shell expression, without running publication.
|
||||
Reference in New Issue
Block a user