From 4ecb71c741b0d477d5bef25d36e6e54d48b5886e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 15 Nov 2026 04:01:50 +0000 Subject: [PATCH] Record resource management audit findings --- docs/roadmap/audit/05-resources.md | 276 +++++++++++++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 docs/roadmap/audit/05-resources.md diff --git a/docs/roadmap/audit/05-resources.md b/docs/roadmap/audit/05-resources.md new file mode 100644 index 0000000..4c8cdcb --- /dev/null +++ b/docs/roadmap/audit/05-resources.md @@ -0,0 +1,276 @@ +# Resource management audit + +Reviewed 2026-11-15 at `e4e5dd7`. This audit covers the Audita CLI, file and stream +I/O, configuration and prompt capture, transcript processing, the pinned +PromptKit execution path, and required and optional artifact generation. The +review also checked local release entry points for resource ownership. The +findings below identify cleanup defects and avoidable resource growth. The +remediation options are proposals for follow-up work. + +## Architecture and lifecycle + +Audita is a single-process Go CLI. The executable owns the signal context and +passes caller-owned streams through CommandKit. The CLI loads configuration and +captured prompt sources, reads the transcript, and preflights destinations. An +output session owns the temporary file and directory used to publish a regular +transcript by rename. Report replacements use their own short-lived staging +directories. Named pipes and stdout are delivered directly. Debug files are +written to a run-specific directory for later inspection. +[Signal setup](../../../cmd/audita/main.go#L16) · [Package map](../../internal/overview.md) +· [Preflight](../../../internal/framework/preflight/preflight.go#L41) +· [Output session](../../../internal/core/diagnostics/output_session.go#L45) +· [Artifact lifecycle](../../operations.md#outputs-and-lifecycle) + +The CLI normalizes the transcript before passing it to the runner. The runner +holds the working transcript and outcome records in memory, creates fixed +chunks, and joins proposal and validator workers at each module boundary. +Role-bound clients share an invocation-local PromptKit engine for HTTP +requests, backend admission, retries, structural repair, and correction-repair +exchanges. The CLI assembles report and debug artifacts from the completed or +interrupted run. +[Normalization and invocation](../../../internal/cli/process.go#L55) +· [Runner ownership](../../../internal/framework/runner/runner.go#L121) +· [Worker joins](../../../internal/framework/runner/runner.go#L435) +· [Validation drain](../../../internal/framework/runner/runner.go#L592) +· [LLM runtime](../../internal/llm-runtime.md) +· [Report assembly](../../../internal/cli/process_pipeline.go#L50) + +Resource release on the reviewed cooperative paths includes the executable's +signal subscription, proposal and validator worker joins, and PromptKit's +operation and request contexts, retry timers, admission reservations, and active +permits. PromptKit removes canceled waiters or transfers ownership of a permit +to the caller that won the race with cancellation. Its built-in HTTP path closes +response bodies and reuses the standard Go transport in the CLI. The pinned +source bounds successful response reads at 16 MiB and provider error reads at +64 KiB, applying reader limits even without a known content length. These are +per-response byte bounds. In the pinned module, the relevant ownership is in +`engine.go`, `internal/usecase/runner.go`, `internal/generation/coordinator.go`, +`internal/capacity/manager.go`, and the HTTP code in `internal/llm/`. +[Pinned version](../../../go.mod#L11) · [CLI transport selection](../../../internal/cli/process_inputs.go#L42) +· [Engine construction](../../../internal/framework/llm/runtime.go#L103) +· [Capacity cancellation test](../../../internal/framework/llm/adapter_capacity_test.go#L42) +· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L48) + +## Findings + +### 1. Failed output cleanup loses ownership of temporary paths + +`OutputSession.Close` closes the transcript descriptor, then tries to remove the +staged payload and its private directory. It clears both path fields even when +removal returns an error. A temporary permission or filesystem failure can leave +a directory or an unpublished payload behind. Finalization surfaces the first +cleanup failure, but the CLI's deferred `Close` has lost the paths it needs to +retry. The preparation and early preflight cancellation branches also discard +cleanup errors. The separate `atomicReplace` report helper ignores the results +of its deferred removes, so it can silently leave staging files or directories +on a cleanup failure. Recurring failures can accumulate disk blocks and inodes. +[Transcript preparation and cleanup](../../../internal/core/diagnostics/output_session.go#L163) +· [Finalization](../../../internal/core/diagnostics/output_session.go#L273) +· [Deferred close](../../../internal/cli/process.go#L41) +· [Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61) +· [Report replacement](../../../internal/core/diagnostics/output_session.go#L410) + +A disposable probe against the unmodified `OutputSession` forced a nonempty +staging directory. The first `Close` removed the payload and returned an error. +After the probe removed the extra entry, the second `Close` returned success +while the empty directory remained. The probe cleaned up its own files. + +Keep each path until removal succeeds or the filesystem says it is absent. Join +cleanup errors with preparation or publication failures and surface report +staging cleanup failures through the output owner. Distinguish a published file +with leftover staging from a failure to publish it. + +### 2. Blocked stream I/O extends resource lifetime past cancellation + +The input path uses `os.ReadFile` without a context. A FIFO can block before open +completes or hold an open descriptor while waiting for more input. Final +delivery similarly opens or writes a FIFO, or writes to the supplied stdout, +after the last cancellation check. With no peer or a peer that stops consuming, +the invocation retains its memory and any acquired stream descriptor. During +output it also retains the encoded transcript and rendered report buffers, and +the session cannot finish report finalization. A supervisor can therefore end up +waiting on canceled processes indefinitely. Prior subprocess probes reproduced +the blocked input, FIFO output, and stdout paths with SIGTERM and released them +by supplying or draining the streams. +[Transcript read](../../../internal/core/io/files.go#L9) +· [Final output check](../../../internal/cli/process_output.go#L54) +· [Output write and close](../../../internal/core/diagnostics/output_session.go#L318) +· [Output probe](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) +· [Input probe](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) + +Make open and readiness waits cancellable for streams Audita owns on supported +platforms. For caller-owned writers, define how the caller supplies cancellation +or drains the stream. Ensure canceled delivery reaches session finalization, +and account for any bytes already delivered. A detached write goroutine alone +would still retain the writer and buffer if the peer stays blocked. + +### 3. Backend admission leaves chunk fan-out and result retention unbounded + +Each module starts one proposal goroutine per chunk. A worker builds a section +transcript and the domain and adapter prepare request inputs before PromptKit +admission. Backend capacity limits admitted runs and active provider attempts +when configured. The pinned `internal/capacity/manager.go` imposes no limit for +backend IDs without a configured pool. With many chunks, local goroutines and +preparation grow with chunk count even when a backend has a small configured +queue. Without a pool, concurrent HTTP requests can also grow with chunk count, +bringing sockets, request timers, encoded prompts, and response buffers with +them. Timeouts and byte limits apply individually. The `examples/service` profile +illustrates explicit backend limits. +[Proposal workers](../../../internal/framework/runner/runner.go#L385) +· [Adapter request preparation](../../../internal/framework/llm/adapter.go#L113) +· [Backend capacity example](../../../examples/service/prompts.yml#L1) + +The proposal and validation channels and their companion maps are sized to the +chunk count. A slow leading chunk can keep many later proposal results, including +captured exchanges when correction repair is active, in the ordered pending map. +Validation can also start a goroutine for each ready chunk and holds completed +results in its channel until the join. PromptKit has already released those +calls' permits while Audita still holds their data. A long, finely chunked +transcript can raise client memory and descriptor demand even when each response +fits the per-call limit. Earlier probes substantiate the ordered backlog and the +interaction with a finite queue. +[Ordered buffering](../../../internal/framework/runner/runner.go#L460) +· [Validation workers](../../../internal/framework/runner/runner.go#L547) +· [Result collection](../../../internal/framework/runner/runner.go#L592) +· [Ordered backlog probe](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention) +· [Admission probe](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling) + +Consider a bounded dispatch window tied to explicit scheduling or operator +settings. Drain validation results while workers run before reducing channel +capacity. A bounded result window or earlier use of section-local identities +could reduce retention while preserving deterministic correction identifiers. +Preserve the module barrier and PromptKit's permit ownership. Decide explicitly +when backend saturation should wait or remain a recoverable capacity rejection. + +### 4. Whole-transcript copies amplify memory at input and publication + +`loadProcessInputs` reads the entire file into a byte slice. The parser then +validates the bytes, decodes the whole JSON document into generic maps and +slices, re-encodes the selected segment array, and decodes that array into typed +segments. Extra fields can occupy memory in the generic tree even if the typed +transcript discards them. The CLI keeps the raw input and typed source reachable +through its `processInputs` struct, which it passes into finalization. The raw +input's last consumer is the optional source debug artifact. At the other end, +both output encoders build one map per segment and a complete indented JSON +payload before delivery. A large transcript can run out of memory before the +first provider call or after all correction work has completed. +[Input load and ownership](../../../internal/cli/process_inputs.go#L25) +· [Parser](../../../internal/core/schema/transcript.go#L35) +· [Segment re-encoding](../../../internal/core/schema/transcript.go#L115) +· [Last source use and finalization](../../../internal/cli/process.go#L51) +· [Bare-array encoder](../../../internal/core/schema/transcript.go#L224) +· [Envelope encoder](../../../internal/core/outputschema/registry.go#L62) +· [Related parse fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) + +Decode the supported input shapes into typed segments without the generic +round trip and release source representations after their last consumers. Typed +output projections would remove the per-segment maps. For regular files, an +incremental array encoder can write into the prepared temporary file and publish +only after encoding succeeds. Preserve both schemas and decide how a writer-based +encoder would handle partial output on a stream. + +### 5. Summary-only runs retain a full grid of work and operation records + +`Runner.Run` eagerly allocates and initializes one `WorkUnitOutcome` per +configured module and chunk before the first module starts. The work-unit grid +retains nested operations and repair summaries until `processreport.Build`. +Completed modules also flatten the same operations into `ModuleResult.Operations`. +The CLI selects `SummaryOnly` only after the runner returns when reports are +disabled. The builder scans the operation and work-unit records to count them, +then skips detailed rows. On many-module, many-chunk runs, this incurs memory +proportional to the module-by-chunk product and all retained operation records, +including entries for work never started after early failure. This is in +addition to the debug decision history identified in the efficiency audit. +[Grid allocation](../../../internal/framework/runner/runner.go#L127) +· [Operation copies](../../../internal/framework/runner/runner.go#L264) +· [CLI report selection](../../../internal/cli/process_pipeline.go#L78) +· [Summary aggregation](../../../internal/framework/processreport/report.go#L41) +· [Decision retention](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history) + +When neither detailed reports nor the debug ledger is requested, fold finalized +work into invocation counters and derive not-started counts from the total +inventory. Retain the failed module identity for run-level errors. With details +enabled, consider one ownership path for operation records and deriving +not-started rows on demand, preserving the complete reporting grid and ordered +evidence. + +### 6. Debug capture adds memory and disk pressure + +Debug runs retain all logical calls and execution events in memory while also +writing several files per proposal, validation, and repair interaction. At the +end, the observation collector copies its records into a sorted snapshot. +`BuildLLMExecution` groups events by call and builds a second representation +for serialization. The JSON redactor creates encoded bytes, a decoded generic +tree, and final formatted bytes. Large event histories can produce a memory +peak while the transcript and the evidence needed for the correction ledger +are still live. Each debug invocation adds a run directory. Large or repeated +debug runs can consume space or inodes needed by the transcript and reports +when they share the filesystem. Once an artifact write fails, the CLI can +continue to collect and retain a warning for each later failure until +finalization, even if the destination stays full. +[Observer retention and snapshot](../../../internal/framework/llm/observations.go#L81) +· [Execution artifact assembly](../../../internal/framework/processreport/llm.go#L61) +· [JSON redaction](../../../internal/core/redaction/redaction.go#L53) +· [Run directories](../../../internal/core/diagnostics/run_dir.go#L34) +· [Interaction files](../../../internal/framework/llm/diagnostics.go#L35) +· [Warning accumulation](../../../internal/cli/process.go#L42) +· [Warning delivery](../../../internal/cli/process_output.go#L17) + +Reuse a single ordered set of observations for final aggregation, or stream +redacted records through a disk-backed artifact while preserving the final +summary. Coalesce repeated debug write failures and report which evidence was +lost. Give operators visibility into debug footprint and an explicit retention +or quota decision for runs that need it. Keep optional evidence failure distinct +from required output delivery. + +### 7. Correction repair replays growing history under per-correction budgets + +Each actionable rejection starts a branch from the original section exchange. +PromptKit's pinned `internal/domain/exchange.go` shares the immutable history +through parent references. Its `internal/usecase/runner.go` flattens that history +with the next turn during preparation, and the HTTP client encodes it into the +request body. Audita revalidates returned revisions and restarts the configured +turn allowance for the next rejected correction. A large section prompt is sent +again on every branch. If a branch uses `A` turns, starts with `S` bytes of +history, and each turn adds roughly `H` bytes, the cumulative logical repair +request payload is on the order of `A*S + A²*H` for that branch. Retries and +revalidation add work. With debug enabled, Audita writes the prepared history +again for each logical repair turn. Across many actionable rejections, this can +increase provider context usage, outbound traffic, and disk use even when every +call fits the configured attempt and time budgets. +[Per-correction configuration](../../config.md#settings) +· [Branch start](../../../internal/framework/runner/repair.go#L85) +· [Coordination and per-turn artifacts](../../../internal/framework/runner/repair.go#L146) +· [Revalidation](../../../internal/framework/runner/repair.go#L227) +· [Prepared snapshot](../../../internal/framework/llm/adapter.go#L152) +· [Pinned version](../../../go.mod#L11) + +Expose aggregate repair work and request size so operators can size runs. If a +work envelope is needed, consider an opt-in section or invocation budget that +accounts for revalidation as well as repair turns and records why any branch +stopped. For debug storage, consider recording a shared history once with +references sufficient to reconstruct the redacted request. Preserve the +conversation context used for editorial decisions. + +## Verification + +- The repository's complete `go test -count=1 ./...` suite passed on Go 1.27.3. + It includes output write and rename failure cleanup and release of backend + capacity after cancellation. The disposable probe in finding 1 exercises a + removal failure beyond those existing tests. + [Rename failure test](../../../internal/core/diagnostics/output_session_test.go#L98) + · [Transcript failure test](../../../internal/core/diagnostics/output_session_test.go#L394) + · [Capacity release test](../../../internal/framework/llm/adapter_capacity_test.go#L42) +- The pinned PromptKit `internal/llm`, `internal/capacity`, and + `internal/generation` packages passed their existing tests. These exercise + response byte limits and body closure, capacity contention and cancellation, + and cancellation during retry delays. + [Pinned version](../../../go.mod#L11) +- Local release entry points were reviewed without executing a release. The + wrapper replaces its shell with ReleaseKit, and the CI bootstrap registers an + exit trap to remove its temporary directory. + [Release wrapper](../../../scripts/release.sh#L10) + · [CI bootstrap](../../../.woodpecker/release.yml#L29) +- The scaling findings describe allocation and ownership from source. Peak RSS, + throughput, provider cost, and filesystem-specific failure frequency were + outside the review's measurement scope.