Record resource management audit findings
This commit is contained in:
276
docs/roadmap/audit/05-resources.md
Normal file
276
docs/roadmap/audit/05-resources.md
Normal file
@@ -0,0 +1,276 @@
|
|||||||
|
# Resource management audit
|
||||||
|
|
||||||
|
Reviewed 2026-11-15 at `e4e5dd7`. This audit covers the Audita CLI, file and stream
|
||||||
|
I/O, configuration and prompt capture, transcript processing, the pinned
|
||||||
|
PromptKit execution path, and required and optional artifact generation. The
|
||||||
|
review also checked local release entry points for resource ownership. The
|
||||||
|
findings below identify cleanup defects and avoidable resource growth. The
|
||||||
|
remediation options are proposals for follow-up work.
|
||||||
|
|
||||||
|
## Architecture and lifecycle
|
||||||
|
|
||||||
|
Audita is a single-process Go CLI. The executable owns the signal context and
|
||||||
|
passes caller-owned streams through CommandKit. The CLI loads configuration and
|
||||||
|
captured prompt sources, reads the transcript, and preflights destinations. An
|
||||||
|
output session owns the temporary file and directory used to publish a regular
|
||||||
|
transcript by rename. Report replacements use their own short-lived staging
|
||||||
|
directories. Named pipes and stdout are delivered directly. Debug files are
|
||||||
|
written to a run-specific directory for later inspection.
|
||||||
|
[Signal setup](../../../cmd/audita/main.go#L16) · [Package map](../../internal/overview.md)
|
||||||
|
· [Preflight](../../../internal/framework/preflight/preflight.go#L41)
|
||||||
|
· [Output session](../../../internal/core/diagnostics/output_session.go#L45)
|
||||||
|
· [Artifact lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||||
|
|
||||||
|
The CLI normalizes the transcript before passing it to the runner. The runner
|
||||||
|
holds the working transcript and outcome records in memory, creates fixed
|
||||||
|
chunks, and joins proposal and validator workers at each module boundary.
|
||||||
|
Role-bound clients share an invocation-local PromptKit engine for HTTP
|
||||||
|
requests, backend admission, retries, structural repair, and correction-repair
|
||||||
|
exchanges. The CLI assembles report and debug artifacts from the completed or
|
||||||
|
interrupted run.
|
||||||
|
[Normalization and invocation](../../../internal/cli/process.go#L55)
|
||||||
|
· [Runner ownership](../../../internal/framework/runner/runner.go#L121)
|
||||||
|
· [Worker joins](../../../internal/framework/runner/runner.go#L435)
|
||||||
|
· [Validation drain](../../../internal/framework/runner/runner.go#L592)
|
||||||
|
· [LLM runtime](../../internal/llm-runtime.md)
|
||||||
|
· [Report assembly](../../../internal/cli/process_pipeline.go#L50)
|
||||||
|
|
||||||
|
Resource release on the reviewed cooperative paths includes the executable's
|
||||||
|
signal subscription, proposal and validator worker joins, and PromptKit's
|
||||||
|
operation and request contexts, retry timers, admission reservations, and active
|
||||||
|
permits. PromptKit removes canceled waiters or transfers ownership of a permit
|
||||||
|
to the caller that won the race with cancellation. Its built-in HTTP path closes
|
||||||
|
response bodies and reuses the standard Go transport in the CLI. The pinned
|
||||||
|
source bounds successful response reads at 16 MiB and provider error reads at
|
||||||
|
64 KiB, applying reader limits even without a known content length. These are
|
||||||
|
per-response byte bounds. In the pinned module, the relevant ownership is in
|
||||||
|
`engine.go`, `internal/usecase/runner.go`, `internal/generation/coordinator.go`,
|
||||||
|
`internal/capacity/manager.go`, and the HTTP code in `internal/llm/`.
|
||||||
|
[Pinned version](../../../go.mod#L11) · [CLI transport selection](../../../internal/cli/process_inputs.go#L42)
|
||||||
|
· [Engine construction](../../../internal/framework/llm/runtime.go#L103)
|
||||||
|
· [Capacity cancellation test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
|
||||||
|
· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### 1. Failed output cleanup loses ownership of temporary paths
|
||||||
|
|
||||||
|
`OutputSession.Close` closes the transcript descriptor, then tries to remove the
|
||||||
|
staged payload and its private directory. It clears both path fields even when
|
||||||
|
removal returns an error. A temporary permission or filesystem failure can leave
|
||||||
|
a directory or an unpublished payload behind. Finalization surfaces the first
|
||||||
|
cleanup failure, but the CLI's deferred `Close` has lost the paths it needs to
|
||||||
|
retry. The preparation and early preflight cancellation branches also discard
|
||||||
|
cleanup errors. The separate `atomicReplace` report helper ignores the results
|
||||||
|
of its deferred removes, so it can silently leave staging files or directories
|
||||||
|
on a cleanup failure. Recurring failures can accumulate disk blocks and inodes.
|
||||||
|
[Transcript preparation and cleanup](../../../internal/core/diagnostics/output_session.go#L163)
|
||||||
|
· [Finalization](../../../internal/core/diagnostics/output_session.go#L273)
|
||||||
|
· [Deferred close](../../../internal/cli/process.go#L41)
|
||||||
|
· [Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61)
|
||||||
|
· [Report replacement](../../../internal/core/diagnostics/output_session.go#L410)
|
||||||
|
|
||||||
|
A disposable probe against the unmodified `OutputSession` forced a nonempty
|
||||||
|
staging directory. The first `Close` removed the payload and returned an error.
|
||||||
|
After the probe removed the extra entry, the second `Close` returned success
|
||||||
|
while the empty directory remained. The probe cleaned up its own files.
|
||||||
|
|
||||||
|
Keep each path until removal succeeds or the filesystem says it is absent. Join
|
||||||
|
cleanup errors with preparation or publication failures and surface report
|
||||||
|
staging cleanup failures through the output owner. Distinguish a published file
|
||||||
|
with leftover staging from a failure to publish it.
|
||||||
|
|
||||||
|
### 2. Blocked stream I/O extends resource lifetime past cancellation
|
||||||
|
|
||||||
|
The input path uses `os.ReadFile` without a context. A FIFO can block before open
|
||||||
|
completes or hold an open descriptor while waiting for more input. Final
|
||||||
|
delivery similarly opens or writes a FIFO, or writes to the supplied stdout,
|
||||||
|
after the last cancellation check. With no peer or a peer that stops consuming,
|
||||||
|
the invocation retains its memory and any acquired stream descriptor. During
|
||||||
|
output it also retains the encoded transcript and rendered report buffers, and
|
||||||
|
the session cannot finish report finalization. A supervisor can therefore end up
|
||||||
|
waiting on canceled processes indefinitely. Prior subprocess probes reproduced
|
||||||
|
the blocked input, FIFO output, and stdout paths with SIGTERM and released them
|
||||||
|
by supplying or draining the streams.
|
||||||
|
[Transcript read](../../../internal/core/io/files.go#L9)
|
||||||
|
· [Final output check](../../../internal/cli/process_output.go#L54)
|
||||||
|
· [Output write and close](../../../internal/core/diagnostics/output_session.go#L318)
|
||||||
|
· [Output probe](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
|
||||||
|
· [Input probe](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
|
||||||
|
|
||||||
|
Make open and readiness waits cancellable for streams Audita owns on supported
|
||||||
|
platforms. For caller-owned writers, define how the caller supplies cancellation
|
||||||
|
or drains the stream. Ensure canceled delivery reaches session finalization,
|
||||||
|
and account for any bytes already delivered. A detached write goroutine alone
|
||||||
|
would still retain the writer and buffer if the peer stays blocked.
|
||||||
|
|
||||||
|
### 3. Backend admission leaves chunk fan-out and result retention unbounded
|
||||||
|
|
||||||
|
Each module starts one proposal goroutine per chunk. A worker builds a section
|
||||||
|
transcript and the domain and adapter prepare request inputs before PromptKit
|
||||||
|
admission. Backend capacity limits admitted runs and active provider attempts
|
||||||
|
when configured. The pinned `internal/capacity/manager.go` imposes no limit for
|
||||||
|
backend IDs without a configured pool. With many chunks, local goroutines and
|
||||||
|
preparation grow with chunk count even when a backend has a small configured
|
||||||
|
queue. Without a pool, concurrent HTTP requests can also grow with chunk count,
|
||||||
|
bringing sockets, request timers, encoded prompts, and response buffers with
|
||||||
|
them. Timeouts and byte limits apply individually. The `examples/service` profile
|
||||||
|
illustrates explicit backend limits.
|
||||||
|
[Proposal workers](../../../internal/framework/runner/runner.go#L385)
|
||||||
|
· [Adapter request preparation](../../../internal/framework/llm/adapter.go#L113)
|
||||||
|
· [Backend capacity example](../../../examples/service/prompts.yml#L1)
|
||||||
|
|
||||||
|
The proposal and validation channels and their companion maps are sized to the
|
||||||
|
chunk count. A slow leading chunk can keep many later proposal results, including
|
||||||
|
captured exchanges when correction repair is active, in the ordered pending map.
|
||||||
|
Validation can also start a goroutine for each ready chunk and holds completed
|
||||||
|
results in its channel until the join. PromptKit has already released those
|
||||||
|
calls' permits while Audita still holds their data. A long, finely chunked
|
||||||
|
transcript can raise client memory and descriptor demand even when each response
|
||||||
|
fits the per-call limit. Earlier probes substantiate the ordered backlog and the
|
||||||
|
interaction with a finite queue.
|
||||||
|
[Ordered buffering](../../../internal/framework/runner/runner.go#L460)
|
||||||
|
· [Validation workers](../../../internal/framework/runner/runner.go#L547)
|
||||||
|
· [Result collection](../../../internal/framework/runner/runner.go#L592)
|
||||||
|
· [Ordered backlog probe](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention)
|
||||||
|
· [Admission probe](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling)
|
||||||
|
|
||||||
|
Consider a bounded dispatch window tied to explicit scheduling or operator
|
||||||
|
settings. Drain validation results while workers run before reducing channel
|
||||||
|
capacity. A bounded result window or earlier use of section-local identities
|
||||||
|
could reduce retention while preserving deterministic correction identifiers.
|
||||||
|
Preserve the module barrier and PromptKit's permit ownership. Decide explicitly
|
||||||
|
when backend saturation should wait or remain a recoverable capacity rejection.
|
||||||
|
|
||||||
|
### 4. Whole-transcript copies amplify memory at input and publication
|
||||||
|
|
||||||
|
`loadProcessInputs` reads the entire file into a byte slice. The parser then
|
||||||
|
validates the bytes, decodes the whole JSON document into generic maps and
|
||||||
|
slices, re-encodes the selected segment array, and decodes that array into typed
|
||||||
|
segments. Extra fields can occupy memory in the generic tree even if the typed
|
||||||
|
transcript discards them. The CLI keeps the raw input and typed source reachable
|
||||||
|
through its `processInputs` struct, which it passes into finalization. The raw
|
||||||
|
input's last consumer is the optional source debug artifact. At the other end,
|
||||||
|
both output encoders build one map per segment and a complete indented JSON
|
||||||
|
payload before delivery. A large transcript can run out of memory before the
|
||||||
|
first provider call or after all correction work has completed.
|
||||||
|
[Input load and ownership](../../../internal/cli/process_inputs.go#L25)
|
||||||
|
· [Parser](../../../internal/core/schema/transcript.go#L35)
|
||||||
|
· [Segment re-encoding](../../../internal/core/schema/transcript.go#L115)
|
||||||
|
· [Last source use and finalization](../../../internal/cli/process.go#L51)
|
||||||
|
· [Bare-array encoder](../../../internal/core/schema/transcript.go#L224)
|
||||||
|
· [Envelope encoder](../../../internal/core/outputschema/registry.go#L62)
|
||||||
|
· [Related parse fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity)
|
||||||
|
|
||||||
|
Decode the supported input shapes into typed segments without the generic
|
||||||
|
round trip and release source representations after their last consumers. Typed
|
||||||
|
output projections would remove the per-segment maps. For regular files, an
|
||||||
|
incremental array encoder can write into the prepared temporary file and publish
|
||||||
|
only after encoding succeeds. Preserve both schemas and decide how a writer-based
|
||||||
|
encoder would handle partial output on a stream.
|
||||||
|
|
||||||
|
### 5. Summary-only runs retain a full grid of work and operation records
|
||||||
|
|
||||||
|
`Runner.Run` eagerly allocates and initializes one `WorkUnitOutcome` per
|
||||||
|
configured module and chunk before the first module starts. The work-unit grid
|
||||||
|
retains nested operations and repair summaries until `processreport.Build`.
|
||||||
|
Completed modules also flatten the same operations into `ModuleResult.Operations`.
|
||||||
|
The CLI selects `SummaryOnly` only after the runner returns when reports are
|
||||||
|
disabled. The builder scans the operation and work-unit records to count them,
|
||||||
|
then skips detailed rows. On many-module, many-chunk runs, this incurs memory
|
||||||
|
proportional to the module-by-chunk product and all retained operation records,
|
||||||
|
including entries for work never started after early failure. This is in
|
||||||
|
addition to the debug decision history identified in the efficiency audit.
|
||||||
|
[Grid allocation](../../../internal/framework/runner/runner.go#L127)
|
||||||
|
· [Operation copies](../../../internal/framework/runner/runner.go#L264)
|
||||||
|
· [CLI report selection](../../../internal/cli/process_pipeline.go#L78)
|
||||||
|
· [Summary aggregation](../../../internal/framework/processreport/report.go#L41)
|
||||||
|
· [Decision retention](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history)
|
||||||
|
|
||||||
|
When neither detailed reports nor the debug ledger is requested, fold finalized
|
||||||
|
work into invocation counters and derive not-started counts from the total
|
||||||
|
inventory. Retain the failed module identity for run-level errors. With details
|
||||||
|
enabled, consider one ownership path for operation records and deriving
|
||||||
|
not-started rows on demand, preserving the complete reporting grid and ordered
|
||||||
|
evidence.
|
||||||
|
|
||||||
|
### 6. Debug capture adds memory and disk pressure
|
||||||
|
|
||||||
|
Debug runs retain all logical calls and execution events in memory while also
|
||||||
|
writing several files per proposal, validation, and repair interaction. At the
|
||||||
|
end, the observation collector copies its records into a sorted snapshot.
|
||||||
|
`BuildLLMExecution` groups events by call and builds a second representation
|
||||||
|
for serialization. The JSON redactor creates encoded bytes, a decoded generic
|
||||||
|
tree, and final formatted bytes. Large event histories can produce a memory
|
||||||
|
peak while the transcript and the evidence needed for the correction ledger
|
||||||
|
are still live. Each debug invocation adds a run directory. Large or repeated
|
||||||
|
debug runs can consume space or inodes needed by the transcript and reports
|
||||||
|
when they share the filesystem. Once an artifact write fails, the CLI can
|
||||||
|
continue to collect and retain a warning for each later failure until
|
||||||
|
finalization, even if the destination stays full.
|
||||||
|
[Observer retention and snapshot](../../../internal/framework/llm/observations.go#L81)
|
||||||
|
· [Execution artifact assembly](../../../internal/framework/processreport/llm.go#L61)
|
||||||
|
· [JSON redaction](../../../internal/core/redaction/redaction.go#L53)
|
||||||
|
· [Run directories](../../../internal/core/diagnostics/run_dir.go#L34)
|
||||||
|
· [Interaction files](../../../internal/framework/llm/diagnostics.go#L35)
|
||||||
|
· [Warning accumulation](../../../internal/cli/process.go#L42)
|
||||||
|
· [Warning delivery](../../../internal/cli/process_output.go#L17)
|
||||||
|
|
||||||
|
Reuse a single ordered set of observations for final aggregation, or stream
|
||||||
|
redacted records through a disk-backed artifact while preserving the final
|
||||||
|
summary. Coalesce repeated debug write failures and report which evidence was
|
||||||
|
lost. Give operators visibility into debug footprint and an explicit retention
|
||||||
|
or quota decision for runs that need it. Keep optional evidence failure distinct
|
||||||
|
from required output delivery.
|
||||||
|
|
||||||
|
### 7. Correction repair replays growing history under per-correction budgets
|
||||||
|
|
||||||
|
Each actionable rejection starts a branch from the original section exchange.
|
||||||
|
PromptKit's pinned `internal/domain/exchange.go` shares the immutable history
|
||||||
|
through parent references. Its `internal/usecase/runner.go` flattens that history
|
||||||
|
with the next turn during preparation, and the HTTP client encodes it into the
|
||||||
|
request body. Audita revalidates returned revisions and restarts the configured
|
||||||
|
turn allowance for the next rejected correction. A large section prompt is sent
|
||||||
|
again on every branch. If a branch uses `A` turns, starts with `S` bytes of
|
||||||
|
history, and each turn adds roughly `H` bytes, the cumulative logical repair
|
||||||
|
request payload is on the order of `A*S + A²*H` for that branch. Retries and
|
||||||
|
revalidation add work. With debug enabled, Audita writes the prepared history
|
||||||
|
again for each logical repair turn. Across many actionable rejections, this can
|
||||||
|
increase provider context usage, outbound traffic, and disk use even when every
|
||||||
|
call fits the configured attempt and time budgets.
|
||||||
|
[Per-correction configuration](../../config.md#settings)
|
||||||
|
· [Branch start](../../../internal/framework/runner/repair.go#L85)
|
||||||
|
· [Coordination and per-turn artifacts](../../../internal/framework/runner/repair.go#L146)
|
||||||
|
· [Revalidation](../../../internal/framework/runner/repair.go#L227)
|
||||||
|
· [Prepared snapshot](../../../internal/framework/llm/adapter.go#L152)
|
||||||
|
· [Pinned version](../../../go.mod#L11)
|
||||||
|
|
||||||
|
Expose aggregate repair work and request size so operators can size runs. If a
|
||||||
|
work envelope is needed, consider an opt-in section or invocation budget that
|
||||||
|
accounts for revalidation as well as repair turns and records why any branch
|
||||||
|
stopped. For debug storage, consider recording a shared history once with
|
||||||
|
references sufficient to reconstruct the redacted request. Preserve the
|
||||||
|
conversation context used for editorial decisions.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- The repository's complete `go test -count=1 ./...` suite passed on Go 1.27.3.
|
||||||
|
It includes output write and rename failure cleanup and release of backend
|
||||||
|
capacity after cancellation. The disposable probe in finding 1 exercises a
|
||||||
|
removal failure beyond those existing tests.
|
||||||
|
[Rename failure test](../../../internal/core/diagnostics/output_session_test.go#L98)
|
||||||
|
· [Transcript failure test](../../../internal/core/diagnostics/output_session_test.go#L394)
|
||||||
|
· [Capacity release test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
|
||||||
|
- The pinned PromptKit `internal/llm`, `internal/capacity`, and
|
||||||
|
`internal/generation` packages passed their existing tests. These exercise
|
||||||
|
response byte limits and body closure, capacity contention and cancellation,
|
||||||
|
and cancellation during retry delays.
|
||||||
|
[Pinned version](../../../go.mod#L11)
|
||||||
|
- Local release entry points were reviewed without executing a release. The
|
||||||
|
wrapper replaces its shell with ReleaseKit, and the CI bootstrap registers an
|
||||||
|
exit trap to remove its temporary directory.
|
||||||
|
[Release wrapper](../../../scripts/release.sh#L10)
|
||||||
|
· [CI bootstrap](../../../.woodpecker/release.yml#L29)
|
||||||
|
- The scaling findings describe allocation and ownership from source. Peak RSS,
|
||||||
|
throughput, provider cost, and filesystem-specific failure frequency were
|
||||||
|
outside the review's measurement scope.
|
||||||
Reference in New Issue
Block a user