From 27e58fabf0efab863b748aeedced4651a3d08d34 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 15 Nov 2026 14:17:12 +0000 Subject: [PATCH] Record error handling audit findings --- docs/roadmap/audit/06-error-handling.md | 249 ++++++++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 docs/roadmap/audit/06-error-handling.md diff --git a/docs/roadmap/audit/06-error-handling.md b/docs/roadmap/audit/06-error-handling.md new file mode 100644 index 0000000..72d9b46 --- /dev/null +++ b/docs/roadmap/audit/06-error-handling.md @@ -0,0 +1,249 @@ +# Error-handling audit + +Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt +configuration, transcript parsing and chunking, model and validator execution, +correction repair, output publication, and release bootstrap. It used the +checked-in contracts, the pinned dependency source, existing tests, and +disposable failure probes. + +## Architecture and intended behavior + +Audita is a single-process Go CLI. CommandKit routes commands and presents +outcomes. ConfigKit loads and validates application settings. On `process`, the +CLI parses the transcript and prepares the selected outputs before normalizing +and running the pipeline. PromptKit owns preparation, backend admission, HTTP, +transient retry, schema validation, and structural repair. Audita's role adapter +maps those errors to a class and a safe reason while preserving the typed cause. +The runner isolates recoverable operations, cancels and joins peers after a +fatal error, and returns partial facts for reports. The CLI checks for +cancellation, delivers the transcript and reports, and selects the exit status. +[CLI outcomes](../../cli.md#inspection-and-outcomes) +· [Pipeline](../../internal/pipeline.md) +· [Model integration](../../integrations/openai-compatible-llm.md) +· [Reporting and output ownership](../../internal/diagnostics-reporting.md) + +The audit treated recoverable model errors, rejected corrections, and unavailable +validator assessments according to their separate contracts. Existing tests +confirm that one unavailable validation batch can leave other batches and later +modules usable, and that parent cancellation can stop active and waiting work +without publishing a transcript. Optional debug warnings can accompany exit +`0`, and a transcript can remain published if a later report write fails. These +are selected behaviors. The report boundary intentionally omits raw provider +causes, and input failures before initialization can leave no reports. +[Validation contract](../../internal/validators.md#llm-validator-batching) +· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091) +· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48) +· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179) +· [Output lifecycle](../../operations.md#outputs-and-lifecycle) +· [Early failure contract](../../integrations/subprocess.md#result-handling) + +## Findings + +### 1. Finalization publishes completed reports after it knows delivery failed + +The CLI prepares a `completed` bundle for a successful pipeline and passes it +to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output +session before writing reports. Even if a transcript write or cleanup has +already failed, it writes the initial `completed` bundle and only then asks for +and writes a failed-status bundle. A disposable probe supplied a short stdout +writer. The failed-bundle callback received a known transcript write failure +and could already read `completed` in all three report files. `Finish` returned +the short-write error and the second pass rewrote the reports to `failed`. +[CLI bundle selection](../../../internal/cli/process_output.go#L27) +· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273) +· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318) + +Reports should reflect a failure that is known before the first report write. +The existing nontransactional contract allows a later report failure to leave +mixed evidence. In this case the initial success claim is avoidable, and a +reader can observe it or it can persist if the failure rewrite cannot finish. +Select the failed bundle before publishing reports when transcript delivery or +cleanup has already failed. Keep a best-effort rewrite for errors discovered +during report publication, retaining the original error and any later failures. +If a failed bundle cannot be rendered, surface that error and preserve the +last written evidence. The initialization callback currently discards its +rendering error, and the finalization callback falls back to the previous +bundle, which can still say `completed`. +The missing phase for delivery errors is a separate, previously recorded issue. +[Output lifecycle contract](../../operations.md#outputs-and-lifecycle) +· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105) +· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46) +· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322) +· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) + +### 2. A provider request timeout is reported as the logical operation deadline + +The profile's `timeout_seconds` bounds one provider attempt. The distinct +`llm.operation_timeout` applies to preparation, capacity waiting, retries, +generation, and structural repair. Pinned PromptKit deliberately reserves +`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as +`ErrTransport` with the underlying deadline cause. After checking the native +operation sentinel, Audita checks a generic `context.DeadlineExceeded` before +`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason. +In PromptKit, the distinction is documented in `engine.go` and implemented in +`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`. +[Timeout settings](../../migrations/v2.md) +· [Native runtime contract](../../integrations/openai-compatible-llm.md) +· [Classifier](../../../internal/framework/llm/adapter.go#L228) +· [Pinned PromptKit](../../../go.mod#L11) + +A loopback provider withheld its response beyond a one-second profile timeout. +With a ten-second operation budget and zero retries, the CLI finished after +about one second. It returned `0` with a completed transcript and reported +the failed proposal as `operation_timeout`. Continuing the run is consistent +with both timeout classes being recoverable. The reason directs an operator +toward the logical operation budget when the attempt deadline expired. Use +the native sentinel for whole-operation timeouts, preserving parent cancellation +precedence. Classify an HTTP-only timeout as transport failure or adopt a +distinct safe reason for that cause if consumers need the distinction. +[Report message mapping](../../../internal/framework/processreport/report.go#L224) +· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406) + +### 3. Missing required credentials produce a generic setup message + +The pinned PromptKit use case identifies a missing required environment +credential through `ErrAPIKeyEnvMissing` and retains its configured variable +name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a +generic safe message before delivering the operation result to the domain. A +synthetic profile with a required unset variable stopped before HTTP and +exited `1`. Stderr ended with `structured model setup failed`, the YAML report said +`Processing setup failed.`, and the interaction error artifact repeated the +generic adapter message. The cause is available to Go callers via `Unwrap`, +but the operator-facing process path does not report the missing requirement. +[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243) +· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17) +· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231) +· [Final CLI failure](../../../internal/cli/process_output.go#L61) + +The fatal class and exclusion of raw provider text are appropriate. The safe +diagnostic can still identify an unset required credential and the affected role +or profile. Map the native credential sentinel to an actionable safe message, +and include the configured variable name if obtained from safe profile or +backend metadata. Carry the typed cause internally and retain preparation at +the existing execution boundary, where role selection and credential validation +already occur. +[Credential ownership](../../internal/llm-runtime.md) +· [Recovery workflow](../../operations.md#recovery) + +### 4. Fatal model errors lose their chunk and operation coordinates + +Proposal and validation code records an operation reference even on fatal +returns. It can identify the section, role, validator, and batch. The runner +stores these operations per work unit, but its first-error path retains only +the error value. `executePipeline` reconstructs the fatal report origin from +the failed module without identifying which section or operation stopped it. +The report builder supports these coordinates for run-scope records. In the +missing-credential probe the run record included `grammar_1` and the stage +index, with no `chunk_index` or `operation`, even though the pre-dispatch +proposal invocation had both. Fatal validator batches lose their available +validator and batch coordinates at the same boundary. +[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116) +· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176) +· [First error and cancellation](../../../internal/framework/runner/runner.go#L477) +· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60) +· [Run origin serialization](../../../internal/framework/processreport/report.go#L155) + +Retain the primary error and its available `outcomes.Reference` together when +the runner selects the failure that stops the module. This lets the CLI identify +the failed operation without guessing from canceled peers or counting the fatal +error again as a recoverable work-unit error. Use module-level context for +setup and application errors with no narrower origin. +[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records) + +### 5. Invalid processing selections use the runtime failure exit status + +The CLI contract gives exit `2` to invalid input or usage and exit `1` to +runtime or output failure. `loadProcessInputs` maps malformed input and +configuration to usage errors. Processing readiness is checked later inside +`preflight.Prepare`, where a missing selected profile, a profile reference +that cannot be resolved, or an empty selected glossary for a required glossary +module can fail. `runProcess` treats every error at this boundary as a runtime +failure, together with output and cancellation errors. A probe selecting an +unknown proposal profile returned `1` with a specific profile diagnostic, no +provider calls, and no reports or transcript. The existing integration test +for a similarly invalid profile inheritance reference expects `2`. +[Exit status contract](../../cli.md#inspection-and-outcomes) +· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41) +· [Selection validation](../../../internal/framework/preflight/preflight.go#L96) +· [CLI status selection](../../../internal/cli/process.go#L24) +· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122) + +Distinguish rejected processing selections from context and destination failures +with a typed readiness error or a separate input-check step. Return usage for +invalid operator selections and preserve failure for cancellation, I/O, and +internal defects. Retain the existing early check and the ability to inspect +configuration without requiring a processing-ready pipeline. +[Inspection contract](../../cli.md#inspection-and-outcomes) + +### 6. An infeasible chunk target is recorded as an internal processing failure + +A positive `chunking.target_sections` can be valid configuration and infeasible +for the normalized input. The chunker rejects that combination and the runner +wraps the error with `chunk normalized transcript`. The CLI's run record defaults +to `phase: processing`, `reason: internal_failure`, and a generic message for +errors outside the structured model and cancellation classes. With one usable +segment and an explicit target of two, a CLI probe produced that failed report, +exited `1`, and made no model calls. This failure should remain visible when +reports are enabled because it occurs after initialization. The reported phase +and reason currently misidentify an operator-correctable constraint as an +internal defect in processing. +[Chunking settings](../../config.md#settings) +· [Feasibility check](../../../internal/core/chunking/sections.go#L143) +· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126) +· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60) +· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244) + +Carry a typed chunking failure through the runner and map infeasible parameters +to `phase: chunking` and an input reason. Keep a separate classification for an +unexpected failure in the chunker. Align the exit mapping with the input +contract if the run distinguishes these classes at that boundary. The earlier +correctness finding about feasible targets being rejected concerns a separate +chunking decision. +[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records) +· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) + +### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty + +The release job checks `test -z "$(git status --porcelain --untracked-files=all)"` +under `set -eu`. The `test` command's exit status controls whether the shell +continues. If Git emits a diagnostic on stderr and fails with empty stdout, +the test succeeds. A shell probe substituted a status command that exited +`128` with no stdout and confirmed that execution continued after the check. +The expected bootstrap behavior is to stop before dependency preparation if +cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared` +independently checks the status command's error before delivery, limiting this +finding to the earlier bootstrap gate and diagnosis. +[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17) +· [Release procedure](../../release.md#binary-delivery) +· [Pinned ReleaseKit](../../../go.mod#L12) + +Assign the status output in a separate command and check the command result +before testing the content. Preserve the existing verification in ReleaseKit. + +## Related failure paths already recorded + +- Cancellation during FIFO or stdout delivery can be missed after the last + context check. A blocked FIFO input can also defer cancellation before + output preflight. The runner has a separate cancellation window before + committing module changes. + [Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) + · [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) + · [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary) +- The failure path can drop completed proposal evidence behind an earlier + section. The internal preflight seam can also overwrite initialized reports + with empty files when a failed renderer is absent. + [Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) + · [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence) +- Output cleanup can discard removal errors or forget a staging path before a + later cleanup attempt. This differs from intentionally retaining requested + debug evidence or keeping a transcript already published successfully. + [Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) + +## Verification + +The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a +temporary CLI build, a synthetic loopback provider, a disposable output-session +caller, and a shell simulation. They verified the failure conditions described +above without persisting tests or program changes. The release check was +reviewed locally against the pinned ReleaseKit implementation and exercised at +the shell expression, without running publication.