250 lines
16 KiB
Markdown
250 lines
16 KiB
Markdown
|
|
# Error-handling audit
|
||
|
|
|
||
|
|
Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt
|
||
|
|
configuration, transcript parsing and chunking, model and validator execution,
|
||
|
|
correction repair, output publication, and release bootstrap. It used the
|
||
|
|
checked-in contracts, the pinned dependency source, existing tests, and
|
||
|
|
disposable failure probes.
|
||
|
|
|
||
|
|
## Architecture and intended behavior
|
||
|
|
|
||
|
|
Audita is a single-process Go CLI. CommandKit routes commands and presents
|
||
|
|
outcomes. ConfigKit loads and validates application settings. On `process`, the
|
||
|
|
CLI parses the transcript and prepares the selected outputs before normalizing
|
||
|
|
and running the pipeline. PromptKit owns preparation, backend admission, HTTP,
|
||
|
|
transient retry, schema validation, and structural repair. Audita's role adapter
|
||
|
|
maps those errors to a class and a safe reason while preserving the typed cause.
|
||
|
|
The runner isolates recoverable operations, cancels and joins peers after a
|
||
|
|
fatal error, and returns partial facts for reports. The CLI checks for
|
||
|
|
cancellation, delivers the transcript and reports, and selects the exit status.
|
||
|
|
[CLI outcomes](../../cli.md#inspection-and-outcomes)
|
||
|
|
· [Pipeline](../../internal/pipeline.md)
|
||
|
|
· [Model integration](../../integrations/openai-compatible-llm.md)
|
||
|
|
· [Reporting and output ownership](../../internal/diagnostics-reporting.md)
|
||
|
|
|
||
|
|
The audit treated recoverable model errors, rejected corrections, and unavailable
|
||
|
|
validator assessments according to their separate contracts. Existing tests
|
||
|
|
confirm that one unavailable validation batch can leave other batches and later
|
||
|
|
modules usable, and that parent cancellation can stop active and waiting work
|
||
|
|
without publishing a transcript. Optional debug warnings can accompany exit
|
||
|
|
`0`, and a transcript can remain published if a later report write fails. These
|
||
|
|
are selected behaviors. The report boundary intentionally omits raw provider
|
||
|
|
causes, and input failures before initialization can leave no reports.
|
||
|
|
[Validation contract](../../internal/validators.md#llm-validator-batching)
|
||
|
|
· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091)
|
||
|
|
· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
|
||
|
|
· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179)
|
||
|
|
· [Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||
|
|
· [Early failure contract](../../integrations/subprocess.md#result-handling)
|
||
|
|
|
||
|
|
## Findings
|
||
|
|
|
||
|
|
### 1. Finalization publishes completed reports after it knows delivery failed
|
||
|
|
|
||
|
|
The CLI prepares a `completed` bundle for a successful pipeline and passes it
|
||
|
|
to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output
|
||
|
|
session before writing reports. Even if a transcript write or cleanup has
|
||
|
|
already failed, it writes the initial `completed` bundle and only then asks for
|
||
|
|
and writes a failed-status bundle. A disposable probe supplied a short stdout
|
||
|
|
writer. The failed-bundle callback received a known transcript write failure
|
||
|
|
and could already read `completed` in all three report files. `Finish` returned
|
||
|
|
the short-write error and the second pass rewrote the reports to `failed`.
|
||
|
|
[CLI bundle selection](../../../internal/cli/process_output.go#L27)
|
||
|
|
· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273)
|
||
|
|
· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318)
|
||
|
|
|
||
|
|
Reports should reflect a failure that is known before the first report write.
|
||
|
|
The existing nontransactional contract allows a later report failure to leave
|
||
|
|
mixed evidence. In this case the initial success claim is avoidable, and a
|
||
|
|
reader can observe it or it can persist if the failure rewrite cannot finish.
|
||
|
|
Select the failed bundle before publishing reports when transcript delivery or
|
||
|
|
cleanup has already failed. Keep a best-effort rewrite for errors discovered
|
||
|
|
during report publication, retaining the original error and any later failures.
|
||
|
|
If a failed bundle cannot be rendered, surface that error and preserve the
|
||
|
|
last written evidence. The initialization callback currently discards its
|
||
|
|
rendering error, and the finalization callback falls back to the previous
|
||
|
|
bundle, which can still say `completed`.
|
||
|
|
The missing phase for delivery errors is a separate, previously recorded issue.
|
||
|
|
[Output lifecycle contract](../../operations.md#outputs-and-lifecycle)
|
||
|
|
· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105)
|
||
|
|
· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46)
|
||
|
|
· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322)
|
||
|
|
· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records)
|
||
|
|
|
||
|
|
### 2. A provider request timeout is reported as the logical operation deadline
|
||
|
|
|
||
|
|
The profile's `timeout_seconds` bounds one provider attempt. The distinct
|
||
|
|
`llm.operation_timeout` applies to preparation, capacity waiting, retries,
|
||
|
|
generation, and structural repair. Pinned PromptKit deliberately reserves
|
||
|
|
`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as
|
||
|
|
`ErrTransport` with the underlying deadline cause. After checking the native
|
||
|
|
operation sentinel, Audita checks a generic `context.DeadlineExceeded` before
|
||
|
|
`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason.
|
||
|
|
In PromptKit, the distinction is documented in `engine.go` and implemented in
|
||
|
|
`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`.
|
||
|
|
[Timeout settings](../../migrations/v2.md)
|
||
|
|
· [Native runtime contract](../../integrations/openai-compatible-llm.md)
|
||
|
|
· [Classifier](../../../internal/framework/llm/adapter.go#L228)
|
||
|
|
· [Pinned PromptKit](../../../go.mod#L11)
|
||
|
|
|
||
|
|
A loopback provider withheld its response beyond a one-second profile timeout.
|
||
|
|
With a ten-second operation budget and zero retries, the CLI finished after
|
||
|
|
about one second. It returned `0` with a completed transcript and reported
|
||
|
|
the failed proposal as `operation_timeout`. Continuing the run is consistent
|
||
|
|
with both timeout classes being recoverable. The reason directs an operator
|
||
|
|
toward the logical operation budget when the attempt deadline expired. Use
|
||
|
|
the native sentinel for whole-operation timeouts, preserving parent cancellation
|
||
|
|
precedence. Classify an HTTP-only timeout as transport failure or adopt a
|
||
|
|
distinct safe reason for that cause if consumers need the distinction.
|
||
|
|
[Report message mapping](../../../internal/framework/processreport/report.go#L224)
|
||
|
|
· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
|
||
|
|
|
||
|
|
### 3. Missing required credentials produce a generic setup message
|
||
|
|
|
||
|
|
The pinned PromptKit use case identifies a missing required environment
|
||
|
|
credential through `ErrAPIKeyEnvMissing` and retains its configured variable
|
||
|
|
name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a
|
||
|
|
generic safe message before delivering the operation result to the domain. A
|
||
|
|
synthetic profile with a required unset variable stopped before HTTP and
|
||
|
|
exited `1`. Stderr ended with `structured model setup failed`, the YAML report said
|
||
|
|
`Processing setup failed.`, and the interaction error artifact repeated the
|
||
|
|
generic adapter message. The cause is available to Go callers via `Unwrap`,
|
||
|
|
but the operator-facing process path does not report the missing requirement.
|
||
|
|
[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243)
|
||
|
|
· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17)
|
||
|
|
· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231)
|
||
|
|
· [Final CLI failure](../../../internal/cli/process_output.go#L61)
|
||
|
|
|
||
|
|
The fatal class and exclusion of raw provider text are appropriate. The safe
|
||
|
|
diagnostic can still identify an unset required credential and the affected role
|
||
|
|
or profile. Map the native credential sentinel to an actionable safe message,
|
||
|
|
and include the configured variable name if obtained from safe profile or
|
||
|
|
backend metadata. Carry the typed cause internally and retain preparation at
|
||
|
|
the existing execution boundary, where role selection and credential validation
|
||
|
|
already occur.
|
||
|
|
[Credential ownership](../../internal/llm-runtime.md)
|
||
|
|
· [Recovery workflow](../../operations.md#recovery)
|
||
|
|
|
||
|
|
### 4. Fatal model errors lose their chunk and operation coordinates
|
||
|
|
|
||
|
|
Proposal and validation code records an operation reference even on fatal
|
||
|
|
returns. It can identify the section, role, validator, and batch. The runner
|
||
|
|
stores these operations per work unit, but its first-error path retains only
|
||
|
|
the error value. `executePipeline` reconstructs the fatal report origin from
|
||
|
|
the failed module without identifying which section or operation stopped it.
|
||
|
|
The report builder supports these coordinates for run-scope records. In the
|
||
|
|
missing-credential probe the run record included `grammar_1` and the stage
|
||
|
|
index, with no `chunk_index` or `operation`, even though the pre-dispatch
|
||
|
|
proposal invocation had both. Fatal validator batches lose their available
|
||
|
|
validator and batch coordinates at the same boundary.
|
||
|
|
[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116)
|
||
|
|
· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176)
|
||
|
|
· [First error and cancellation](../../../internal/framework/runner/runner.go#L477)
|
||
|
|
· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60)
|
||
|
|
· [Run origin serialization](../../../internal/framework/processreport/report.go#L155)
|
||
|
|
|
||
|
|
Retain the primary error and its available `outcomes.Reference` together when
|
||
|
|
the runner selects the failure that stops the module. This lets the CLI identify
|
||
|
|
the failed operation without guessing from canceled peers or counting the fatal
|
||
|
|
error again as a recoverable work-unit error. Use module-level context for
|
||
|
|
setup and application errors with no narrower origin.
|
||
|
|
[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records)
|
||
|
|
|
||
|
|
### 5. Invalid processing selections use the runtime failure exit status
|
||
|
|
|
||
|
|
The CLI contract gives exit `2` to invalid input or usage and exit `1` to
|
||
|
|
runtime or output failure. `loadProcessInputs` maps malformed input and
|
||
|
|
configuration to usage errors. Processing readiness is checked later inside
|
||
|
|
`preflight.Prepare`, where a missing selected profile, a profile reference
|
||
|
|
that cannot be resolved, or an empty selected glossary for a required glossary
|
||
|
|
module can fail. `runProcess` treats every error at this boundary as a runtime
|
||
|
|
failure, together with output and cancellation errors. A probe selecting an
|
||
|
|
unknown proposal profile returned `1` with a specific profile diagnostic, no
|
||
|
|
provider calls, and no reports or transcript. The existing integration test
|
||
|
|
for a similarly invalid profile inheritance reference expects `2`.
|
||
|
|
[Exit status contract](../../cli.md#inspection-and-outcomes)
|
||
|
|
· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41)
|
||
|
|
· [Selection validation](../../../internal/framework/preflight/preflight.go#L96)
|
||
|
|
· [CLI status selection](../../../internal/cli/process.go#L24)
|
||
|
|
· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122)
|
||
|
|
|
||
|
|
Distinguish rejected processing selections from context and destination failures
|
||
|
|
with a typed readiness error or a separate input-check step. Return usage for
|
||
|
|
invalid operator selections and preserve failure for cancellation, I/O, and
|
||
|
|
internal defects. Retain the existing early check and the ability to inspect
|
||
|
|
configuration without requiring a processing-ready pipeline.
|
||
|
|
[Inspection contract](../../cli.md#inspection-and-outcomes)
|
||
|
|
|
||
|
|
### 6. An infeasible chunk target is recorded as an internal processing failure
|
||
|
|
|
||
|
|
A positive `chunking.target_sections` can be valid configuration and infeasible
|
||
|
|
for the normalized input. The chunker rejects that combination and the runner
|
||
|
|
wraps the error with `chunk normalized transcript`. The CLI's run record defaults
|
||
|
|
to `phase: processing`, `reason: internal_failure`, and a generic message for
|
||
|
|
errors outside the structured model and cancellation classes. With one usable
|
||
|
|
segment and an explicit target of two, a CLI probe produced that failed report,
|
||
|
|
exited `1`, and made no model calls. This failure should remain visible when
|
||
|
|
reports are enabled because it occurs after initialization. The reported phase
|
||
|
|
and reason currently misidentify an operator-correctable constraint as an
|
||
|
|
internal defect in processing.
|
||
|
|
[Chunking settings](../../config.md#settings)
|
||
|
|
· [Feasibility check](../../../internal/core/chunking/sections.go#L143)
|
||
|
|
· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126)
|
||
|
|
· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60)
|
||
|
|
· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244)
|
||
|
|
|
||
|
|
Carry a typed chunking failure through the runner and map infeasible parameters
|
||
|
|
to `phase: chunking` and an input reason. Keep a separate classification for an
|
||
|
|
unexpected failure in the chunker. Align the exit mapping with the input
|
||
|
|
contract if the run distinguishes these classes at that boundary. The earlier
|
||
|
|
correctness finding about feasible targets being rejected concerns a separate
|
||
|
|
chunking decision.
|
||
|
|
[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records)
|
||
|
|
· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets)
|
||
|
|
|
||
|
|
### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty
|
||
|
|
|
||
|
|
The release job checks `test -z "$(git status --porcelain --untracked-files=all)"`
|
||
|
|
under `set -eu`. The `test` command's exit status controls whether the shell
|
||
|
|
continues. If Git emits a diagnostic on stderr and fails with empty stdout,
|
||
|
|
the test succeeds. A shell probe substituted a status command that exited
|
||
|
|
`128` with no stdout and confirmed that execution continued after the check.
|
||
|
|
The expected bootstrap behavior is to stop before dependency preparation if
|
||
|
|
cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared`
|
||
|
|
independently checks the status command's error before delivery, limiting this
|
||
|
|
finding to the earlier bootstrap gate and diagnosis.
|
||
|
|
[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17)
|
||
|
|
· [Release procedure](../../release.md#binary-delivery)
|
||
|
|
· [Pinned ReleaseKit](../../../go.mod#L12)
|
||
|
|
|
||
|
|
Assign the status output in a separate command and check the command result
|
||
|
|
before testing the content. Preserve the existing verification in ReleaseKit.
|
||
|
|
|
||
|
|
## Related failure paths already recorded
|
||
|
|
|
||
|
|
- Cancellation during FIFO or stdout delivery can be missed after the last
|
||
|
|
context check. A blocked FIFO input can also defer cancellation before
|
||
|
|
output preflight. The runner has a separate cancellation window before
|
||
|
|
committing module changes.
|
||
|
|
[Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
|
||
|
|
· [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
|
||
|
|
· [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary)
|
||
|
|
- The failure path can drop completed proposal evidence behind an earlier
|
||
|
|
section. The internal preflight seam can also overwrite initialized reports
|
||
|
|
with empty files when a failed renderer is absent.
|
||
|
|
[Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
|
||
|
|
· [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence)
|
||
|
|
- Output cleanup can discard removal errors or forget a staging path before a
|
||
|
|
later cleanup attempt. This differs from intentionally retaining requested
|
||
|
|
debug evidence or keeping a transcript already published successfully.
|
||
|
|
[Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
|
||
|
|
|
||
|
|
## Verification
|
||
|
|
|
||
|
|
The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a
|
||
|
|
temporary CLI build, a synthetic loopback provider, a disposable output-session
|
||
|
|
caller, and a shell simulation. They verified the failure conditions described
|
||
|
|
above without persisting tests or program changes. The release check was
|
||
|
|
reviewed locally against the pinned ReleaseKit implementation and exercised at
|
||
|
|
the shell expression, without running publication.
|