Files
audita/docs/roadmap/audit/06-error-handling.md

16 KiB

Error-handling audit

Reviewed 2026-11-15 at 4ecb71c. The review followed application and prompt configuration, transcript parsing and chunking, model and validator execution, correction repair, output publication, and release bootstrap. It used the checked-in contracts, the pinned dependency source, existing tests, and disposable failure probes.

Architecture and intended behavior

Audita is a single-process Go CLI. CommandKit routes commands and presents outcomes. ConfigKit loads and validates application settings. On process, the CLI parses the transcript and prepares the selected outputs before normalizing and running the pipeline. PromptKit owns preparation, backend admission, HTTP, transient retry, schema validation, and structural repair. Audita's role adapter maps those errors to a class and a safe reason while preserving the typed cause. The runner isolates recoverable operations, cancels and joins peers after a fatal error, and returns partial facts for reports. The CLI checks for cancellation, delivers the transcript and reports, and selects the exit status. CLI outcomes · Pipeline · Model integration · Reporting and output ownership

The audit treated recoverable model errors, rejected corrections, and unavailable validator assessments according to their separate contracts. Existing tests confirm that one unavailable validation batch can leave other batches and later modules usable, and that parent cancellation can stop active and waiting work without publishing a transcript. Optional debug warnings can accompany exit 0, and a transcript can remain published if a later report write fails. These are selected behaviors. The report boundary intentionally omits raw provider causes, and input failures before initialization can leave no reports. Validation contract · Partial validation test · Cancellation test · Optional debug test · Output lifecycle · Early failure contract

Findings

1. Finalization publishes completed reports after it knows delivery failed

The CLI prepares a completed bundle for a successful pipeline and passes it to OutputSession.Finish. Finalize tries the transcript and closes the output session before writing reports. Even if a transcript write or cleanup has already failed, it writes the initial completed bundle and only then asks for and writes a failed-status bundle. A disposable probe supplied a short stdout writer. The failed-bundle callback received a known transcript write failure and could already read completed in all three report files. Finish returned the short-write error and the second pass rewrote the reports to failed. CLI bundle selection · Finalization order · Short-write check

Reports should reflect a failure that is known before the first report write. The existing nontransactional contract allows a later report failure to leave mixed evidence. In this case the initial success claim is avoidable, and a reader can observe it or it can persist if the failure rewrite cannot finish. Select the failed bundle before publishing reports when transcript delivery or cleanup has already failed. Keep a best-effort rewrite for errors discovered during report publication, retaining the original error and any later failures. If a failed bundle cannot be rendered, surface that error and preserve the last written evidence. The initialization callback currently discards its rendering error, and the finalization callback falls back to the previous bundle, which can still say completed. The missing phase for delivery errors is a separate, previously recorded issue. Output lifecycle contract · Initialization rendering fallback · Finalization rendering fallback · Failure retention test · Earlier output-phase finding

2. A provider request timeout is reported as the logical operation deadline

The profile's timeout_seconds bounds one provider attempt. The distinct llm.operation_timeout applies to preparation, capacity waiting, retries, generation, and structural repair. Pinned PromptKit deliberately reserves ErrOperationTimeout for the latter and exposes its HTTP attempt failure as ErrTransport with the underlying deadline cause. After checking the native operation sentinel, Audita checks a generic context.DeadlineExceeded before ErrTransport, so the HTTP timeout takes the operation_timeout reason. In PromptKit, the distinction is documented in engine.go and implemented in internal/llm/openai_compatible_client.go and internal/usecase/operation_scope.go. Timeout settings · Native runtime contract · Classifier · Pinned PromptKit

A loopback provider withheld its response beyond a one-second profile timeout. With a ten-second operation budget and zero retries, the CLI finished after about one second. It returned 0 with a completed transcript and reported the failed proposal as operation_timeout. Continuing the run is consistent with both timeout classes being recoverable. The reason directs an operator toward the logical operation budget when the attempt deadline expired. Use the native sentinel for whole-operation timeouts, preserving parent cancellation precedence. Classify an HTTP-only timeout as transport failure or adopt a distinct safe reason for that cause if consumers need the distinction. Report message mapping · Existing operation deadline and parent cancellation test

3. Missing required credentials produce a generic setup message

The pinned PromptKit use case identifies a missing required environment credential through ErrAPIKeyEnvMissing and retains its configured variable name in internal/usecase/runner.go. Audita maps it to setup_failure and a generic safe message before delivering the operation result to the domain. A synthetic profile with a required unset variable stopped before HTTP and exited 1. Stderr ended with structured model setup failed, the YAML report said Processing setup failed., and the interaction error artifact repeated the generic adapter message. The cause is available to Go callers via Unwrap, but the operator-facing process path does not report the missing requirement. Adapter setup classification · Typed cause · Proposal error artifact · Final CLI failure

The fatal class and exclusion of raw provider text are appropriate. The safe diagnostic can still identify an unset required credential and the affected role or profile. Map the native credential sentinel to an actionable safe message, and include the configured variable name if obtained from safe profile or backend metadata. Carry the typed cause internally and retain preparation at the existing execution boundary, where role selection and credential validation already occur. Credential ownership · Recovery workflow

4. Fatal model errors lose their chunk and operation coordinates

Proposal and validation code records an operation reference even on fatal returns. It can identify the section, role, validator, and batch. The runner stores these operations per work unit, but its first-error path retains only the error value. executePipeline reconstructs the fatal report origin from the failed module without identifying which section or operation stopped it. The report builder supports these coordinates for run-scope records. In the missing-credential probe the run record included grammar_1 and the stage index, with no chunk_index or operation, even though the pre-dispatch proposal invocation had both. Fatal validator batches lose their available validator and batch coordinates at the same boundary. Proposal failure operation · Validator failure operation · First error and cancellation · CLI origin construction · Run origin serialization

Retain the primary error and its available outcomes.Reference together when the runner selects the failure that stops the module. This lets the CLI identify the failed operation without guessing from canceled peers or counting the fatal error again as a recoverable work-unit error. Use module-level context for setup and application errors with no narrower origin. Run and work-unit record contract

5. Invalid processing selections use the runtime failure exit status

The CLI contract gives exit 2 to invalid input or usage and exit 1 to runtime or output failure. loadProcessInputs maps malformed input and configuration to usage errors. Processing readiness is checked later inside preflight.Prepare, where a missing selected profile, a profile reference that cannot be resolved, or an empty selected glossary for a required glossary module can fail. runProcess treats every error at this boundary as a runtime failure, together with output and cancellation errors. A probe selecting an unknown proposal profile returned 1 with a specific profile diagnostic, no provider calls, and no reports or transcript. The existing integration test for a similarly invalid profile inheritance reference expects 2. Exit status contract · Readiness and output sequencing · Selection validation · CLI status selection · Invalid profile inheritance test

Distinguish rejected processing selections from context and destination failures with a typed readiness error or a separate input-check step. Return usage for invalid operator selections and preserve failure for cancellation, I/O, and internal defects. Retain the existing early check and the ability to inspect configuration without requiring a processing-ready pipeline. Inspection contract

6. An infeasible chunk target is recorded as an internal processing failure

A positive chunking.target_sections can be valid configuration and infeasible for the normalized input. The chunker rejects that combination and the runner wraps the error with chunk normalized transcript. The CLI's run record defaults to phase: processing, reason: internal_failure, and a generic message for errors outside the structured model and cancellation classes. With one usable segment and an explicit target of two, a CLI probe produced that failed report, exited 1, and made no model calls. This failure should remain visible when reports are enabled because it occurs after initialization. The reported phase and reason currently misidentify an operator-correctable constraint as an internal defect in processing. Chunking settings · Feasibility check · Runner phase wrapper · CLI fallback classification · Existing input reason and phase ordering

Carry a typed chunking failure through the runner and map infeasible parameters to phase: chunking and an input reason. Keep a separate classification for an unexpected failure in the chunker. Align the exit mapping with the input contract if the run distinguishes these classes at that boundary. The earlier correctness finding about feasible targets being rejected concerns a separate chunking decision. Run error phase contract · Earlier chunking finding

7. CI bootstrap treats a failed Git status command as clean if stdout is empty

The release job checks test -z "$(git status --porcelain --untracked-files=all)" under set -eu. The test command's exit status controls whether the shell continues. If Git emits a diagnostic on stderr and fails with empty stdout, the test succeeds. A shell probe substituted a status command that exited 128 with no stdout and confirmed that execution continued after the check. The expected bootstrap behavior is to stop before dependency preparation if cleanliness could not be established. Pinned ReleaseKit ci-deliver --prepared independently checks the status command's error before delivery, limiting this finding to the earlier bootstrap gate and diagnosis. CI bootstrap and delivery invocation · Release procedure · Pinned ReleaseKit

Assign the status output in a separate command and check the command result before testing the content. Preserve the existing verification in ReleaseKit.

  • Cancellation during FIFO or stdout delivery can be missed after the last context check. A blocked FIFO input can also defer cancellation before output preflight. The runner has a separate cancellation window before committing module changes. Delivery · Input · Runner commit
  • The failure path can drop completed proposal evidence behind an earlier section. The internal preflight seam can also overwrite initialized reports with empty files when a failed renderer is absent. Lost proposal evidence · Missing renderer
  • Output cleanup can discard removal errors or forget a staging path before a later cleanup attempt. This differs from intentionally retaining requested debug evidence or keeping a transcript already published successfully. Cleanup finding

Verification

The full go test -count=1 ./... suite passed on Go 1.27.3. The probes used a temporary CLI build, a synthetic loopback provider, a disposable output-session caller, and a shell simulation. They verified the failure conditions described above without persisting tests or program changes. The release check was reviewed locally against the pinned ReleaseKit implementation and exercised at the shell expression, without running publication.