Record concurrency audit findings
This commit is contained in:
227
docs/roadmap/audit/04-concurrency.md
Normal file
227
docs/roadmap/audit/04-concurrency.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# Concurrency audit
|
||||
|
||||
Reviewed source at `c0193df`. The full offline suite passed under
|
||||
`go test -race -count=1 ./...` with Go 1.27.3. Selected capacity,
|
||||
cancellation, coordination, and observation tests also passed eight runs
|
||||
under the race detector. The timing findings below use Linux subprocesses,
|
||||
synthetic loopback responses, and a temporary runner probe. The audit used
|
||||
no live provider calls.
|
||||
|
||||
## Architecture and synchronization reviewed
|
||||
|
||||
Audita is a single-process Go CLI. The executable translates SIGINT and
|
||||
SIGTERM into context cancellation. CommandKit handles commands and streams,
|
||||
ConfigKit resolves settings, and the deterministic core normalizes and chunks
|
||||
the transcript. The runner owns one stage barrier across fixed chunks.
|
||||
Proposal and validation goroutines operate within a stage, and the runner
|
||||
collects their results in a stable order before applying approved corrections.
|
||||
The invocation shares a PromptKit engine across roles, with backend-local
|
||||
admission, synchronous execution observations, and coordinated repair
|
||||
branches. The CLI joins processing before assembling reports and delivering
|
||||
output. [Signal setup](../../../cmd/audita/main.go#L16)
|
||||
· [Package map](../../internal/overview.md)
|
||||
· [Pipeline](../../internal/pipeline.md)
|
||||
· [LLM runtime](../../internal/llm-runtime.md)
|
||||
|
||||
- Proposal workers send one result per chunk. Their owner waits for all
|
||||
senders before closing the channel. The validation channel is drained after
|
||||
the worker join, which works because it has one slot per chunk and at most
|
||||
one send per validation worker. Reducing that buffer would require changing
|
||||
the drain order. A mutex protects the first fatal error, and capture of
|
||||
warnings and execution evidence completes before the CLI reads it.
|
||||
[Proposal worker lifecycle](../../../internal/framework/runner/runner.go#L385)
|
||||
· [Error and result ownership](../../../internal/framework/runner/runner.go#L463)
|
||||
· [Validation join](../../../internal/framework/runner/runner.go#L592)
|
||||
· [Warning capture](../../../internal/cli/process.go#L42)
|
||||
· [Evidence snapshot](../../../internal/cli/process_pipeline.go#L51)
|
||||
- Existing tests cover cancellation while a peer call is active and another
|
||||
waits for backend capacity, release after cancellation, independent backends,
|
||||
repair response withholding, and nested validation on the repair backend.
|
||||
Observations use synchronized collections, and a snapshot is taken after
|
||||
workers join. Config and prompt capture happen before worker startup.
|
||||
[Runner cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
|
||||
· [Backend capacity tests](../../../internal/framework/llm/adapter_capacity_test.go#L42)
|
||||
· [Repair coordination tests](../../../internal/framework/llm/coordinate_test.go#L77)
|
||||
· [Concurrent observation test](../../../internal/framework/llm/observations_test.go#L59)
|
||||
· [Snapshot contract](../../../internal/framework/llm/observations.go#L81)
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Blocked output delivery can ignore cancellation and report success
|
||||
|
||||
The CLI checks the context immediately before `OutputSession.Finish`.
|
||||
Delivery then opens a named pipe or writes stdout without observing the
|
||||
context. With no FIFO reader, SIGTERM left the child blocked with
|
||||
`in_progress` reports. When a reader attached, the child delivered the
|
||||
transcript, finalized the reports as `completed`, and exited 0. The same
|
||||
outcome occurred with a full stdout pipe and a 240 KB transcript. Draining
|
||||
the pipe after SIGTERM allowed the write to finish and the child to report
|
||||
success. The FIFO case was first recorded in the correctness audit.
|
||||
[Last context check](../../../internal/cli/process_output.go#L54)
|
||||
· [Stream open and write](../../../internal/core/diagnostics/output_session.go#L318)
|
||||
· [Earlier FIFO finding](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success)
|
||||
|
||||
Pass cancellation through delivery. For cancellable file descriptors,
|
||||
make both reader availability and write readiness interruptible. Keep the
|
||||
guidance for subprocess clients to drain both streams, and define how cancellation
|
||||
after a partial stream write is reflected in status and reports. Check
|
||||
cancellation before committing a regular-file rename and use the existing
|
||||
failed-report path when delivery is interrupted.
|
||||
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
|
||||
|
||||
### 2. Waiting for transcript input can prevent shutdown before preflight
|
||||
|
||||
`loadProcessInputs` calls a context-free `os.ReadFile` for the transcript.
|
||||
When the path names a FIFO, it blocks on a writer before the application
|
||||
can check for cancellation again. In a subprocess probe with no writer,
|
||||
SIGTERM left the child blocked without starting report initialization.
|
||||
Supplying the input afterward allowed the child to exit 1. The process
|
||||
therefore relies on a producer or forced termination to unblock that
|
||||
read. The input parser accepts the JSON once it arrives.
|
||||
[Input acquisition](../../../internal/cli/process_inputs.go#L46)
|
||||
· [File read](../../../internal/core/io/files.go#L9)
|
||||
|
||||
If pipe input remains supported, use a context-aware open and read on
|
||||
supported platforms and stop reading after cancellation. If support is
|
||||
intentionally limited to regular input files, make that contract explicit
|
||||
and validate the path before a blocking open.
|
||||
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
||||
|
||||
### 3. Cancellation can cross the runner's module commit boundary
|
||||
|
||||
`Runner.Run` checks the context after the module's workers have joined,
|
||||
then applies corrections, classifies work units, and commits the new sections
|
||||
without another cancellation check. A temporary runner probe canceled the
|
||||
parent immediately after that check sampled a live context. The final
|
||||
module returned success, an applied correction, and a completed work unit.
|
||||
An already-canceled invocation with no modules also returned success, as
|
||||
the check is inside the module loop. The CLI has later context checks that
|
||||
prevent publication in these cases, but can render canceled-run evidence
|
||||
counting edits committed after the cancellation. Normalization and encoding
|
||||
also run synchronously between CLI cancellation checks, which can delay
|
||||
shutdown for large workloads.
|
||||
[Before application](../../../internal/framework/runner/runner.go#L190)
|
||||
· [Commit and return](../../../internal/framework/runner/runner.go#L264)
|
||||
· [Local processing boundaries](../../../internal/cli/process.go#L55)
|
||||
· [Earlier empty-pipeline finding](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline)
|
||||
|
||||
Check cancellation at runner entry, before starting another module,
|
||||
and just before committing computed updates. Keep the previous working
|
||||
transcript until the commit check passes. If cancellation during large local
|
||||
passes needs a shorter response time, add cooperative checks at natural
|
||||
iteration boundaries.
|
||||
[Cancellation and partial-result contract](../../internal/pipeline.md)
|
||||
|
||||
### 4. A slow earlier proposal holds back later validation and retention
|
||||
|
||||
The runner buffers completed proposals until the next chunk in order is
|
||||
available. It assigns stage-wide indices, starts validators, and captures
|
||||
ledger candidates only as that prefix advances. With two chunks and the
|
||||
first response deliberately held open, the second response reached its
|
||||
debug artifact while no validator call started. Releasing the first
|
||||
response let both validators check the later correction. In a separate
|
||||
run, canceling at the same point retained the later raw response and
|
||||
counted one usable proposal response, but the correction ledger contained
|
||||
only the module failure row. A slow earlier retry or timeout can therefore
|
||||
stall independent validator work, and interruption can omit a completed
|
||||
candidate from the ledger.
|
||||
[Ordered proposal buffer](../../../internal/framework/runner/runner.go#L494)
|
||||
· [Candidate capture and dispatch](../../../internal/framework/runner/runner.go#L520)
|
||||
· [Earlier ledger finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
|
||||
|
||||
Retain completed responses independently of scheduling and assign
|
||||
deterministic indices to the retained evidence on a fatal return. For
|
||||
validation latency, consider section-local identities during checking and
|
||||
repair, with stable stage-wide identities assigned during collection.
|
||||
Account for diagnostic filenames and report references in that change.
|
||||
This can preserve the existing stage barrier.
|
||||
[Stage barrier decision](../future.md#independent-whole-pipeline-traversal-by-chunk)
|
||||
|
||||
### 5. Proposal fan-out makes admission under finite capacity depend on scheduling
|
||||
|
||||
Each chunk immediately attempts a proposal in its own goroutine. PromptKit
|
||||
limits admitted calls, while Audita has no ordering at the admission boundary.
|
||||
With 12 chunks, one backend slot, and zero queue capacity, six otherwise
|
||||
identical runs each reached the synthetic provider once and recorded 11
|
||||
recoverable capacity errors. Chunk 12 entered in five runs and chunk 2
|
||||
entered in the other. Which chunks can produce corrections can therefore
|
||||
vary before the provider sees any prompt. The backend enforced its configured
|
||||
limit and the error reports reflected the rejections. The weakness comes
|
||||
from submitting a complete transcript's worth of proposals at once to a
|
||||
finite queue.
|
||||
[Proposal fan-out](../../../internal/framework/runner/runner.go#L385)
|
||||
· [Capacity classification](../../../internal/framework/llm/adapter.go#L254)
|
||||
· [Engine-local admission](../../policy/architecture.md#correction-and-concurrency-invariants)
|
||||
|
||||
Make the interaction between chunk fan-out and finite queue capacity
|
||||
explicit in operator guidance. If the desired processing policy is to wait
|
||||
for more chunks to receive a turn, introduce bounded or ordered dispatch for
|
||||
that policy while preserving PromptKit's ownership of backend permits and
|
||||
the existing recoverable-rejection behavior for configurations that select
|
||||
it.
|
||||
|
||||
### 6. Report paths can change referent between initialization and finalization
|
||||
|
||||
Preflight checks report destinations once. Each subsequent `atomicReplace`
|
||||
resolves the original path again, including during the final report pass.
|
||||
In a loopback probe, a report-directory symlink pointed to one directory
|
||||
during initialization and was switched to another while the proposal
|
||||
request was held. The CLI exited 0. The original directory still held
|
||||
`in_progress` reports, and the replacement directory received
|
||||
`completed` reports for the same run ID. Destination collision checks only
|
||||
covered the referents inspected during preflight.
|
||||
[Destination selection](../../../internal/core/diagnostics/output_session.go#L61)
|
||||
· [Report initialization](../../../internal/core/diagnostics/output_session.go#L222)
|
||||
· [Per-write path resolution](../../../internal/core/diagnostics/output_session.go#L410)
|
||||
|
||||
Bind the checked report destinations for the session, or detect a changed
|
||||
referent and fail before writing to it. Use the same target identity for
|
||||
initialization, finalization, and the failed-report retry. This complements
|
||||
the earlier path-resolution finding, which covers a path whose meaning is
|
||||
wrong even without a concurrent change.
|
||||
[Earlier path resolution finding](01-correctness.md#1-output-path-resolution-can-replace-a-different-file)
|
||||
|
||||
### 7. Concurrent invocations sharing a report directory replace each other's evidence
|
||||
|
||||
Different transcript output paths can select the same report directory,
|
||||
including by default when the outputs have the same parent. Each invocation
|
||||
owns temporary files and has its own run ID, but all replace the same three
|
||||
report basenames. In a probe, run A initialized reports and then waited for
|
||||
its proposal response. Run B completed with a different transcript output in
|
||||
the shared directory. Releasing A let it complete and replace B's completed
|
||||
reports. Both commands exited 0. Because report writes are per file, a
|
||||
consumer reading during overlap can also observe files from different runs.
|
||||
[Report-directory selection](../../../internal/cli/process_inputs.go#L72)
|
||||
· [Fixed basenames](../../../internal/core/diagnostics/output_session.go#L12)
|
||||
· [Final report writes](../../../internal/core/diagnostics/output_session.go#L286)
|
||||
|
||||
Document per-invocation report destinations for concurrent subprocess
|
||||
consumers. If supporting shared destinations directly, consider
|
||||
invocation-specific publication or explicit serialization for that
|
||||
destination. Per-file atomic replacement should remain in place.
|
||||
[Report placement and lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
|
||||
### 8. Shared module and validator ownership is implicit at their interfaces
|
||||
|
||||
The runner shares one module instance across proposal workers and one
|
||||
validator instance per chain entry across validation workers. Requests
|
||||
also carry shared stage-level glossary data and transcript segment copies
|
||||
whose category slices still alias the stage's data. The interfaces accept a
|
||||
context but do not say that methods can run concurrently, borrowed data must
|
||||
be read-only, or a canceled call must return for the stage to join. A
|
||||
stateful implementation that mutates cached state or retains and mutates
|
||||
borrowed data can race with peers. An implementation that blocks through
|
||||
cancellation can prevent the worker join. Bundled modules and validators
|
||||
use read-only shared configuration and per-call working state in the paths
|
||||
reviewed. This is an internal ownership gap.
|
||||
[Module and validator contracts](../../../internal/framework/contracts/contracts.go#L21)
|
||||
· [Shared proposal instance](../../../internal/framework/runner/runner.go#L403)
|
||||
· [Shared validation chain](../../../internal/framework/runner/runner.go#L548)
|
||||
· [Shallow section copy](../../../internal/framework/runner/runner.go#L966)
|
||||
|
||||
Document concurrent invocation, read-only inputs, and cooperative cancellation
|
||||
at the contract boundary. Keep state local to each call or synchronize shared
|
||||
mutable state explicitly when a future implementation needs it. If an
|
||||
implementation needs to mutate section data, give it an owned copy of that
|
||||
data.
|
||||
Reference in New Issue
Block a user