Record concurrency audit findings
This commit is contained in:
227
docs/roadmap/audit/04-concurrency.md
Normal file
227
docs/roadmap/audit/04-concurrency.md
Normal file
@@ -0,0 +1,227 @@
|
|||||||
|
# Concurrency audit
|
||||||
|
|
||||||
|
Reviewed source at `c0193df`. The full offline suite passed under
|
||||||
|
`go test -race -count=1 ./...` with Go 1.27.3. Selected capacity,
|
||||||
|
cancellation, coordination, and observation tests also passed eight runs
|
||||||
|
under the race detector. The timing findings below use Linux subprocesses,
|
||||||
|
synthetic loopback responses, and a temporary runner probe. The audit used
|
||||||
|
no live provider calls.
|
||||||
|
|
||||||
|
## Architecture and synchronization reviewed
|
||||||
|
|
||||||
|
Audita is a single-process Go CLI. The executable translates SIGINT and
|
||||||
|
SIGTERM into context cancellation. CommandKit handles commands and streams,
|
||||||
|
ConfigKit resolves settings, and the deterministic core normalizes and chunks
|
||||||
|
the transcript. The runner owns one stage barrier across fixed chunks.
|
||||||
|
Proposal and validation goroutines operate within a stage, and the runner
|
||||||
|
collects their results in a stable order before applying approved corrections.
|
||||||
|
The invocation shares a PromptKit engine across roles, with backend-local
|
||||||
|
admission, synchronous execution observations, and coordinated repair
|
||||||
|
branches. The CLI joins processing before assembling reports and delivering
|
||||||
|
output. [Signal setup](../../../cmd/audita/main.go#L16)
|
||||||
|
· [Package map](../../internal/overview.md)
|
||||||
|
· [Pipeline](../../internal/pipeline.md)
|
||||||
|
· [LLM runtime](../../internal/llm-runtime.md)
|
||||||
|
|
||||||
|
- Proposal workers send one result per chunk. Their owner waits for all
|
||||||
|
senders before closing the channel. The validation channel is drained after
|
||||||
|
the worker join, which works because it has one slot per chunk and at most
|
||||||
|
one send per validation worker. Reducing that buffer would require changing
|
||||||
|
the drain order. A mutex protects the first fatal error, and capture of
|
||||||
|
warnings and execution evidence completes before the CLI reads it.
|
||||||
|
[Proposal worker lifecycle](../../../internal/framework/runner/runner.go#L385)
|
||||||
|
· [Error and result ownership](../../../internal/framework/runner/runner.go#L463)
|
||||||
|
· [Validation join](../../../internal/framework/runner/runner.go#L592)
|
||||||
|
· [Warning capture](../../../internal/cli/process.go#L42)
|
||||||
|
· [Evidence snapshot](../../../internal/cli/process_pipeline.go#L51)
|
||||||
|
- Existing tests cover cancellation while a peer call is active and another
|
||||||
|
waits for backend capacity, release after cancellation, independent backends,
|
||||||
|
repair response withholding, and nested validation on the repair backend.
|
||||||
|
Observations use synchronized collections, and a snapshot is taken after
|
||||||
|
workers join. Config and prompt capture happen before worker startup.
|
||||||
|
[Runner cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
|
||||||
|
· [Backend capacity tests](../../../internal/framework/llm/adapter_capacity_test.go#L42)
|
||||||
|
· [Repair coordination tests](../../../internal/framework/llm/coordinate_test.go#L77)
|
||||||
|
· [Concurrent observation test](../../../internal/framework/llm/observations_test.go#L59)
|
||||||
|
· [Snapshot contract](../../../internal/framework/llm/observations.go#L81)
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### 1. Blocked output delivery can ignore cancellation and report success
|
||||||
|
|
||||||
|
The CLI checks the context immediately before `OutputSession.Finish`.
|
||||||
|
Delivery then opens a named pipe or writes stdout without observing the
|
||||||
|
context. With no FIFO reader, SIGTERM left the child blocked with
|
||||||
|
`in_progress` reports. When a reader attached, the child delivered the
|
||||||
|
transcript, finalized the reports as `completed`, and exited 0. The same
|
||||||
|
outcome occurred with a full stdout pipe and a 240 KB transcript. Draining
|
||||||
|
the pipe after SIGTERM allowed the write to finish and the child to report
|
||||||
|
success. The FIFO case was first recorded in the correctness audit.
|
||||||
|
[Last context check](../../../internal/cli/process_output.go#L54)
|
||||||
|
· [Stream open and write](../../../internal/core/diagnostics/output_session.go#L318)
|
||||||
|
· [Earlier FIFO finding](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success)
|
||||||
|
|
||||||
|
Pass cancellation through delivery. For cancellable file descriptors,
|
||||||
|
make both reader availability and write readiness interruptible. Keep the
|
||||||
|
guidance for subprocess clients to drain both streams, and define how cancellation
|
||||||
|
after a partial stream write is reflected in status and reports. Check
|
||||||
|
cancellation before committing a regular-file rename and use the existing
|
||||||
|
failed-report path when delivery is interrupted.
|
||||||
|
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||||
|
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
|
||||||
|
|
||||||
|
### 2. Waiting for transcript input can prevent shutdown before preflight
|
||||||
|
|
||||||
|
`loadProcessInputs` calls a context-free `os.ReadFile` for the transcript.
|
||||||
|
When the path names a FIFO, it blocks on a writer before the application
|
||||||
|
can check for cancellation again. In a subprocess probe with no writer,
|
||||||
|
SIGTERM left the child blocked without starting report initialization.
|
||||||
|
Supplying the input afterward allowed the child to exit 1. The process
|
||||||
|
therefore relies on a producer or forced termination to unblock that
|
||||||
|
read. The input parser accepts the JSON once it arrives.
|
||||||
|
[Input acquisition](../../../internal/cli/process_inputs.go#L46)
|
||||||
|
· [File read](../../../internal/core/io/files.go#L9)
|
||||||
|
|
||||||
|
If pipe input remains supported, use a context-aware open and read on
|
||||||
|
supported platforms and stop reading after cancellation. If support is
|
||||||
|
intentionally limited to regular input files, make that contract explicit
|
||||||
|
and validate the path before a blocking open.
|
||||||
|
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
||||||
|
|
||||||
|
### 3. Cancellation can cross the runner's module commit boundary
|
||||||
|
|
||||||
|
`Runner.Run` checks the context after the module's workers have joined,
|
||||||
|
then applies corrections, classifies work units, and commits the new sections
|
||||||
|
without another cancellation check. A temporary runner probe canceled the
|
||||||
|
parent immediately after that check sampled a live context. The final
|
||||||
|
module returned success, an applied correction, and a completed work unit.
|
||||||
|
An already-canceled invocation with no modules also returned success, as
|
||||||
|
the check is inside the module loop. The CLI has later context checks that
|
||||||
|
prevent publication in these cases, but can render canceled-run evidence
|
||||||
|
counting edits committed after the cancellation. Normalization and encoding
|
||||||
|
also run synchronously between CLI cancellation checks, which can delay
|
||||||
|
shutdown for large workloads.
|
||||||
|
[Before application](../../../internal/framework/runner/runner.go#L190)
|
||||||
|
· [Commit and return](../../../internal/framework/runner/runner.go#L264)
|
||||||
|
· [Local processing boundaries](../../../internal/cli/process.go#L55)
|
||||||
|
· [Earlier empty-pipeline finding](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline)
|
||||||
|
|
||||||
|
Check cancellation at runner entry, before starting another module,
|
||||||
|
and just before committing computed updates. Keep the previous working
|
||||||
|
transcript until the commit check passes. If cancellation during large local
|
||||||
|
passes needs a shorter response time, add cooperative checks at natural
|
||||||
|
iteration boundaries.
|
||||||
|
[Cancellation and partial-result contract](../../internal/pipeline.md)
|
||||||
|
|
||||||
|
### 4. A slow earlier proposal holds back later validation and retention
|
||||||
|
|
||||||
|
The runner buffers completed proposals until the next chunk in order is
|
||||||
|
available. It assigns stage-wide indices, starts validators, and captures
|
||||||
|
ledger candidates only as that prefix advances. With two chunks and the
|
||||||
|
first response deliberately held open, the second response reached its
|
||||||
|
debug artifact while no validator call started. Releasing the first
|
||||||
|
response let both validators check the later correction. In a separate
|
||||||
|
run, canceling at the same point retained the later raw response and
|
||||||
|
counted one usable proposal response, but the correction ledger contained
|
||||||
|
only the module failure row. A slow earlier retry or timeout can therefore
|
||||||
|
stall independent validator work, and interruption can omit a completed
|
||||||
|
candidate from the ledger.
|
||||||
|
[Ordered proposal buffer](../../../internal/framework/runner/runner.go#L494)
|
||||||
|
· [Candidate capture and dispatch](../../../internal/framework/runner/runner.go#L520)
|
||||||
|
· [Earlier ledger finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
|
||||||
|
|
||||||
|
Retain completed responses independently of scheduling and assign
|
||||||
|
deterministic indices to the retained evidence on a fatal return. For
|
||||||
|
validation latency, consider section-local identities during checking and
|
||||||
|
repair, with stable stage-wide identities assigned during collection.
|
||||||
|
Account for diagnostic filenames and report references in that change.
|
||||||
|
This can preserve the existing stage barrier.
|
||||||
|
[Stage barrier decision](../future.md#independent-whole-pipeline-traversal-by-chunk)
|
||||||
|
|
||||||
|
### 5. Proposal fan-out makes admission under finite capacity depend on scheduling
|
||||||
|
|
||||||
|
Each chunk immediately attempts a proposal in its own goroutine. PromptKit
|
||||||
|
limits admitted calls, while Audita has no ordering at the admission boundary.
|
||||||
|
With 12 chunks, one backend slot, and zero queue capacity, six otherwise
|
||||||
|
identical runs each reached the synthetic provider once and recorded 11
|
||||||
|
recoverable capacity errors. Chunk 12 entered in five runs and chunk 2
|
||||||
|
entered in the other. Which chunks can produce corrections can therefore
|
||||||
|
vary before the provider sees any prompt. The backend enforced its configured
|
||||||
|
limit and the error reports reflected the rejections. The weakness comes
|
||||||
|
from submitting a complete transcript's worth of proposals at once to a
|
||||||
|
finite queue.
|
||||||
|
[Proposal fan-out](../../../internal/framework/runner/runner.go#L385)
|
||||||
|
· [Capacity classification](../../../internal/framework/llm/adapter.go#L254)
|
||||||
|
· [Engine-local admission](../../policy/architecture.md#correction-and-concurrency-invariants)
|
||||||
|
|
||||||
|
Make the interaction between chunk fan-out and finite queue capacity
|
||||||
|
explicit in operator guidance. If the desired processing policy is to wait
|
||||||
|
for more chunks to receive a turn, introduce bounded or ordered dispatch for
|
||||||
|
that policy while preserving PromptKit's ownership of backend permits and
|
||||||
|
the existing recoverable-rejection behavior for configurations that select
|
||||||
|
it.
|
||||||
|
|
||||||
|
### 6. Report paths can change referent between initialization and finalization
|
||||||
|
|
||||||
|
Preflight checks report destinations once. Each subsequent `atomicReplace`
|
||||||
|
resolves the original path again, including during the final report pass.
|
||||||
|
In a loopback probe, a report-directory symlink pointed to one directory
|
||||||
|
during initialization and was switched to another while the proposal
|
||||||
|
request was held. The CLI exited 0. The original directory still held
|
||||||
|
`in_progress` reports, and the replacement directory received
|
||||||
|
`completed` reports for the same run ID. Destination collision checks only
|
||||||
|
covered the referents inspected during preflight.
|
||||||
|
[Destination selection](../../../internal/core/diagnostics/output_session.go#L61)
|
||||||
|
· [Report initialization](../../../internal/core/diagnostics/output_session.go#L222)
|
||||||
|
· [Per-write path resolution](../../../internal/core/diagnostics/output_session.go#L410)
|
||||||
|
|
||||||
|
Bind the checked report destinations for the session, or detect a changed
|
||||||
|
referent and fail before writing to it. Use the same target identity for
|
||||||
|
initialization, finalization, and the failed-report retry. This complements
|
||||||
|
the earlier path-resolution finding, which covers a path whose meaning is
|
||||||
|
wrong even without a concurrent change.
|
||||||
|
[Earlier path resolution finding](01-correctness.md#1-output-path-resolution-can-replace-a-different-file)
|
||||||
|
|
||||||
|
### 7. Concurrent invocations sharing a report directory replace each other's evidence
|
||||||
|
|
||||||
|
Different transcript output paths can select the same report directory,
|
||||||
|
including by default when the outputs have the same parent. Each invocation
|
||||||
|
owns temporary files and has its own run ID, but all replace the same three
|
||||||
|
report basenames. In a probe, run A initialized reports and then waited for
|
||||||
|
its proposal response. Run B completed with a different transcript output in
|
||||||
|
the shared directory. Releasing A let it complete and replace B's completed
|
||||||
|
reports. Both commands exited 0. Because report writes are per file, a
|
||||||
|
consumer reading during overlap can also observe files from different runs.
|
||||||
|
[Report-directory selection](../../../internal/cli/process_inputs.go#L72)
|
||||||
|
· [Fixed basenames](../../../internal/core/diagnostics/output_session.go#L12)
|
||||||
|
· [Final report writes](../../../internal/core/diagnostics/output_session.go#L286)
|
||||||
|
|
||||||
|
Document per-invocation report destinations for concurrent subprocess
|
||||||
|
consumers. If supporting shared destinations directly, consider
|
||||||
|
invocation-specific publication or explicit serialization for that
|
||||||
|
destination. Per-file atomic replacement should remain in place.
|
||||||
|
[Report placement and lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||||
|
|
||||||
|
### 8. Shared module and validator ownership is implicit at their interfaces
|
||||||
|
|
||||||
|
The runner shares one module instance across proposal workers and one
|
||||||
|
validator instance per chain entry across validation workers. Requests
|
||||||
|
also carry shared stage-level glossary data and transcript segment copies
|
||||||
|
whose category slices still alias the stage's data. The interfaces accept a
|
||||||
|
context but do not say that methods can run concurrently, borrowed data must
|
||||||
|
be read-only, or a canceled call must return for the stage to join. A
|
||||||
|
stateful implementation that mutates cached state or retains and mutates
|
||||||
|
borrowed data can race with peers. An implementation that blocks through
|
||||||
|
cancellation can prevent the worker join. Bundled modules and validators
|
||||||
|
use read-only shared configuration and per-call working state in the paths
|
||||||
|
reviewed. This is an internal ownership gap.
|
||||||
|
[Module and validator contracts](../../../internal/framework/contracts/contracts.go#L21)
|
||||||
|
· [Shared proposal instance](../../../internal/framework/runner/runner.go#L403)
|
||||||
|
· [Shared validation chain](../../../internal/framework/runner/runner.go#L548)
|
||||||
|
· [Shallow section copy](../../../internal/framework/runner/runner.go#L966)
|
||||||
|
|
||||||
|
Document concurrent invocation, read-only inputs, and cooperative cancellation
|
||||||
|
at the contract boundary. Keep state local to each call or synchronize shared
|
||||||
|
mutable state explicitly when a future implementation needs it. If an
|
||||||
|
implementation needs to mutate section data, give it an owned copy of that
|
||||||
|
data.
|
||||||
Reference in New Issue
Block a user