Files
audita/docs/roadmap/audit/06-error-handling.md

250 lines
16 KiB
Markdown

# Error-handling audit
Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt
configuration, transcript parsing and chunking, model and validator execution,
correction repair, output publication, and release bootstrap. It used the
checked-in contracts, the pinned dependency source, existing tests, and
disposable failure probes.
## Architecture and intended behavior
Audita is a single-process Go CLI. CommandKit routes commands and presents
outcomes. ConfigKit loads and validates application settings. On `process`, the
CLI parses the transcript and prepares the selected outputs before normalizing
and running the pipeline. PromptKit owns preparation, backend admission, HTTP,
transient retry, schema validation, and structural repair. Audita's role adapter
maps those errors to a class and a safe reason while preserving the typed cause.
The runner isolates recoverable operations, cancels and joins peers after a
fatal error, and returns partial facts for reports. The CLI checks for
cancellation, delivers the transcript and reports, and selects the exit status.
[CLI outcomes](../../cli.md#inspection-and-outcomes)
· [Pipeline](../../internal/pipeline.md)
· [Model integration](../../integrations/openai-compatible-llm.md)
· [Reporting and output ownership](../../internal/diagnostics-reporting.md)
The audit treated recoverable model errors, rejected corrections, and unavailable
validator assessments according to their separate contracts. Existing tests
confirm that one unavailable validation batch can leave other batches and later
modules usable, and that parent cancellation can stop active and waiting work
without publishing a transcript. Optional debug warnings can accompany exit
`0`, and a transcript can remain published if a later report write fails. These
are selected behaviors. The report boundary intentionally omits raw provider
causes, and input failures before initialization can leave no reports.
[Validation contract](../../internal/validators.md#llm-validator-batching)
· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091)
· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179)
· [Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Early failure contract](../../integrations/subprocess.md#result-handling)
## Findings
### 1. Finalization publishes completed reports after it knows delivery failed
The CLI prepares a `completed` bundle for a successful pipeline and passes it
to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output
session before writing reports. Even if a transcript write or cleanup has
already failed, it writes the initial `completed` bundle and only then asks for
and writes a failed-status bundle. A disposable probe supplied a short stdout
writer. The failed-bundle callback received a known transcript write failure
and could already read `completed` in all three report files. `Finish` returned
the short-write error and the second pass rewrote the reports to `failed`.
[CLI bundle selection](../../../internal/cli/process_output.go#L27)
· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273)
· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318)
Reports should reflect a failure that is known before the first report write.
The existing nontransactional contract allows a later report failure to leave
mixed evidence. In this case the initial success claim is avoidable, and a
reader can observe it or it can persist if the failure rewrite cannot finish.
Select the failed bundle before publishing reports when transcript delivery or
cleanup has already failed. Keep a best-effort rewrite for errors discovered
during report publication, retaining the original error and any later failures.
If a failed bundle cannot be rendered, surface that error and preserve the
last written evidence. The initialization callback currently discards its
rendering error, and the finalization callback falls back to the previous
bundle, which can still say `completed`.
The missing phase for delivery errors is a separate, previously recorded issue.
[Output lifecycle contract](../../operations.md#outputs-and-lifecycle)
· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105)
· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46)
· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322)
· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records)
### 2. A provider request timeout is reported as the logical operation deadline
The profile's `timeout_seconds` bounds one provider attempt. The distinct
`llm.operation_timeout` applies to preparation, capacity waiting, retries,
generation, and structural repair. Pinned PromptKit deliberately reserves
`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as
`ErrTransport` with the underlying deadline cause. After checking the native
operation sentinel, Audita checks a generic `context.DeadlineExceeded` before
`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason.
In PromptKit, the distinction is documented in `engine.go` and implemented in
`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`.
[Timeout settings](../../migrations/v2.md)
· [Native runtime contract](../../integrations/openai-compatible-llm.md)
· [Classifier](../../../internal/framework/llm/adapter.go#L228)
· [Pinned PromptKit](../../../go.mod#L11)
A loopback provider withheld its response beyond a one-second profile timeout.
With a ten-second operation budget and zero retries, the CLI finished after
about one second. It returned `0` with a completed transcript and reported
the failed proposal as `operation_timeout`. Continuing the run is consistent
with both timeout classes being recoverable. The reason directs an operator
toward the logical operation budget when the attempt deadline expired. Use
the native sentinel for whole-operation timeouts, preserving parent cancellation
precedence. Classify an HTTP-only timeout as transport failure or adopt a
distinct safe reason for that cause if consumers need the distinction.
[Report message mapping](../../../internal/framework/processreport/report.go#L224)
· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
### 3. Missing required credentials produce a generic setup message
The pinned PromptKit use case identifies a missing required environment
credential through `ErrAPIKeyEnvMissing` and retains its configured variable
name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a
generic safe message before delivering the operation result to the domain. A
synthetic profile with a required unset variable stopped before HTTP and
exited `1`. Stderr ended with `structured model setup failed`, the YAML report said
`Processing setup failed.`, and the interaction error artifact repeated the
generic adapter message. The cause is available to Go callers via `Unwrap`,
but the operator-facing process path does not report the missing requirement.
[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243)
· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17)
· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231)
· [Final CLI failure](../../../internal/cli/process_output.go#L61)
The fatal class and exclusion of raw provider text are appropriate. The safe
diagnostic can still identify an unset required credential and the affected role
or profile. Map the native credential sentinel to an actionable safe message,
and include the configured variable name if obtained from safe profile or
backend metadata. Carry the typed cause internally and retain preparation at
the existing execution boundary, where role selection and credential validation
already occur.
[Credential ownership](../../internal/llm-runtime.md)
· [Recovery workflow](../../operations.md#recovery)
### 4. Fatal model errors lose their chunk and operation coordinates
Proposal and validation code records an operation reference even on fatal
returns. It can identify the section, role, validator, and batch. The runner
stores these operations per work unit, but its first-error path retains only
the error value. `executePipeline` reconstructs the fatal report origin from
the failed module without identifying which section or operation stopped it.
The report builder supports these coordinates for run-scope records. In the
missing-credential probe the run record included `grammar_1` and the stage
index, with no `chunk_index` or `operation`, even though the pre-dispatch
proposal invocation had both. Fatal validator batches lose their available
validator and batch coordinates at the same boundary.
[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116)
· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176)
· [First error and cancellation](../../../internal/framework/runner/runner.go#L477)
· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60)
· [Run origin serialization](../../../internal/framework/processreport/report.go#L155)
Retain the primary error and its available `outcomes.Reference` together when
the runner selects the failure that stops the module. This lets the CLI identify
the failed operation without guessing from canceled peers or counting the fatal
error again as a recoverable work-unit error. Use module-level context for
setup and application errors with no narrower origin.
[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records)
### 5. Invalid processing selections use the runtime failure exit status
The CLI contract gives exit `2` to invalid input or usage and exit `1` to
runtime or output failure. `loadProcessInputs` maps malformed input and
configuration to usage errors. Processing readiness is checked later inside
`preflight.Prepare`, where a missing selected profile, a profile reference
that cannot be resolved, or an empty selected glossary for a required glossary
module can fail. `runProcess` treats every error at this boundary as a runtime
failure, together with output and cancellation errors. A probe selecting an
unknown proposal profile returned `1` with a specific profile diagnostic, no
provider calls, and no reports or transcript. The existing integration test
for a similarly invalid profile inheritance reference expects `2`.
[Exit status contract](../../cli.md#inspection-and-outcomes)
· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41)
· [Selection validation](../../../internal/framework/preflight/preflight.go#L96)
· [CLI status selection](../../../internal/cli/process.go#L24)
· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122)
Distinguish rejected processing selections from context and destination failures
with a typed readiness error or a separate input-check step. Return usage for
invalid operator selections and preserve failure for cancellation, I/O, and
internal defects. Retain the existing early check and the ability to inspect
configuration without requiring a processing-ready pipeline.
[Inspection contract](../../cli.md#inspection-and-outcomes)
### 6. An infeasible chunk target is recorded as an internal processing failure
A positive `chunking.target_sections` can be valid configuration and infeasible
for the normalized input. The chunker rejects that combination and the runner
wraps the error with `chunk normalized transcript`. The CLI's run record defaults
to `phase: processing`, `reason: internal_failure`, and a generic message for
errors outside the structured model and cancellation classes. With one usable
segment and an explicit target of two, a CLI probe produced that failed report,
exited `1`, and made no model calls. This failure should remain visible when
reports are enabled because it occurs after initialization. The reported phase
and reason currently misidentify an operator-correctable constraint as an
internal defect in processing.
[Chunking settings](../../config.md#settings)
· [Feasibility check](../../../internal/core/chunking/sections.go#L143)
· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126)
· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60)
· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244)
Carry a typed chunking failure through the runner and map infeasible parameters
to `phase: chunking` and an input reason. Keep a separate classification for an
unexpected failure in the chunker. Align the exit mapping with the input
contract if the run distinguishes these classes at that boundary. The earlier
correctness finding about feasible targets being rejected concerns a separate
chunking decision.
[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records)
· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets)
### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty
The release job checks `test -z "$(git status --porcelain --untracked-files=all)"`
under `set -eu`. The `test` command's exit status controls whether the shell
continues. If Git emits a diagnostic on stderr and fails with empty stdout,
the test succeeds. A shell probe substituted a status command that exited
`128` with no stdout and confirmed that execution continued after the check.
The expected bootstrap behavior is to stop before dependency preparation if
cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared`
independently checks the status command's error before delivery, limiting this
finding to the earlier bootstrap gate and diagnosis.
[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17)
· [Release procedure](../../release.md#binary-delivery)
· [Pinned ReleaseKit](../../../go.mod#L12)
Assign the status output in a separate command and check the command result
before testing the content. Preserve the existing verification in ReleaseKit.
## Related failure paths already recorded
- Cancellation during FIFO or stdout delivery can be missed after the last
context check. A blocked FIFO input can also defer cancellation before
output preflight. The runner has a separate cancellation window before
committing module changes.
[Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
· [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
· [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary)
- The failure path can drop completed proposal evidence behind an earlier
section. The internal preflight seam can also overwrite initialized reports
with empty files when a failed renderer is absent.
[Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
· [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence)
- Output cleanup can discard removal errors or forget a staging path before a
later cleanup attempt. This differs from intentionally retaining requested
debug evidence or keeping a transcript already published successfully.
[Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
## Verification
The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a
temporary CLI build, a synthetic loopback provider, a disposable output-session
caller, and a shell simulation. They verified the failure conditions described
above without persisting tests or program changes. The release check was
reviewed locally against the pinned ReleaseKit implementation and exercised at
the shell expression, without running publication.