Prepare Audita v4 release notes and retire completed plans

This commit is contained in:
2026-11-17 19:51:50 +00:00
parent 788f830b19
commit 8551121ef6
20 changed files with 92 additions and 6009 deletions

View File

@@ -1,261 +0,0 @@
# Functional correctness audit
Reviewed source at `c4ccd78`. All findings below were reproduced through the
built CLI with temporary files or a synthetic loopback provider. The existing
offline suite also passed with Go 1.27.3 using
`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`.
The provider fixtures exercised Audita's handling of scripted decisions.
Live-model editorial quality was outside the review.
## Architecture and review scope
Audita is a single-process Go CLI. CommandKit handles commands and streams,
ConfigKit composes configuration, and processing preflight checks role profiles,
glossary selections, and output destinations. Core packages parse and normalize
the transcript, derive glossary views, and partition the normalized segments into
fixed chunks. The framework runs configured correction modules in order while
sections can generate proposals concurrently. Deterministic checks and LLM
validators gate application. PromptKit handles the provider and continuation
history for bounded repair branches, and Audita revalidates revisions. The output
session publishes the transcript and reports, with opt-in diagnostics capturing
request, decision, and execution evidence. [Package map](../../internal/overview.md)
· [Processing sequence](../../internal/pipeline.md)
The review traced configuration and readiness into file handling, transcript
decoding and normalization, chunk feasibility, glossary matching, proposal
preview and application, validator batching and response reconciliation,
repair outcomes, cancellation, and ordinary and debug reporting. It checked
representative success paths alongside malformed inputs, boundaries, recoverable
failures, and interrupted work.
## 1. Output path resolution can replace a different file
**Expected.** A regular transcript should be published at the selected output
destination. Collision checks and publication should agree on which file that
path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves
`logical/link/../out.json` to `physical/out.json`. With two distinct existing
files and reporting disabled, Audita exited 0, left `physical/out.json` untouched,
and overwrote `logical/out.json`. `Prepare` stats the supplied path, but
`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving
symlinks. The cleaned path is then used for publication. The same helper feeds
collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141)
· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377)
**Remediation.** Resolve directory components with filesystem traversal
semantics before collapsing `..`. Use the resulting destination for collision
checks, regular-file preparation, and publication. Derive the default report
directory from the same destination when file output is selected.
## 2. Glossary matching misses names with accented letters or punctuation at their edges
**Expected.** Selected protected names and aliases should participate in both
deterministic protection checks. Matching names should also be available to
validators when building context. The glossary accepts ordinary strings for
names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections)
· [Validator context](../../internal/validators.md#supplemental-validation-context)
**Observed.** With `Élodie` or `C++` in the selected context and protection
collection, scripted grammar proposals replaced the names in `Élodie arrived.`
and `Use C++ for this.`. Both edits reached model validation and were applied
after scripted approvals. The validator requests also omitted the corresponding
glossary entries. The shared matcher surrounds every alternative with Go
regular-expression `\b`, whose word character class is ASCII. At the start of
`Élodie` and after the final `+` in `C++`, the required boundary fails.
[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30)
· [Context matching](../../../internal/framework/validators/llm_context.go#L40)
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
**Remediation.** Match configured spellings with boundary checks against
surrounding runes, including spellings that begin or end with punctuation.
Preserve longest-match precedence and exemption behavior, and use consistent
case matching for both the scan and identity lookup.
## 3. Merging nested segments truncates the earlier timestamp range
**Expected.** The merged interval should cover both source intervals. The same
interval should determine whether the merge fits the configured duration limit
and the gap to the next segment. Overlapping source intervals pass input
validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
· [Processing sequence](../../internal/pipeline.md)
**Observed.** With the default normalization settings and an empty pipeline,
same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]`
containing the first two utterances and a separate `[20,21]` segment.
`mergeSegments` takes `next.End` even when it is earlier than the current end.
The next gap is then measured from 6, and duration checks likewise use
`next.End - current.Start`, which can let an over-limit merged interval pass.
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153)
**Remediation.** Use the later of the two end times for both the prospective
duration and the resulting segment. Carry that end forward when calculating
subsequent gaps.
## 4. Balanced chunking rejects feasible explicit targets
**Expected.** A configured section count should succeed when the input can be
partitioned into that many contiguous sections within the token limit.
Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64)
**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers
to prevent normalization merges, fit in two chunks with a maximum of 6.
The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts
the target, but the builder stops the second chunk at 5 because it exceeds the
average target of 4. It leaves the last segment unassigned and the CLI exits 1
with “target_sections (2) is impossible under current constraints (got 2 sections).”
[Feasibility check](../../../internal/core/chunking/sections.go#L143)
· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249)
**Remediation.** Choose balanced cuts only when the remaining suffix can still
fit in the remaining count. A simpler fallback is to build the minimum number
of max-bounded chunks and split them deterministically to reach a feasible target.
## 5. Cancellation during FIFO delivery can hang and later report success
**Expected.** A FIFO can wait for a reader. A catchable cancellation while it
waits should end processing and attempt failed-status report finalization.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
**Observed.** With an empty pipeline and a named pipe as output, the child
stayed blocked after SIGTERM and left reports at `in_progress`. When a reader
attached after the signal, it received the transcript and the child exited 0
with `completed` reports. The main signal handler cancels a context, and the
CLI checks it before calling `Finish`. The direct-output branch then blocks
in `os.OpenFile` without observing that context. Delivery and final report
generation have no check for cancellation during that wait.
[Signal handling](../../../cmd/audita/main.go#L16)
· [Output finalization](../../../internal/cli/process_output.go#L54)
· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318)
**Remediation.** Pass cancellation into delivery. For supported FIFO platforms,
use a context-aware wait for reader availability and for writable capacity,
handling cancellation through the same failed-report path. Define how a
cancellation that arrives after stream bytes have already been written is reported.
## 6. Transcript shape detection loses integer and unknown-field fidelity
**Expected.** Distinct supplied IDs in the supported integer range should
remain distinct through validation. Unknown fields should be ignored.
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
**Observed.** On Linux amd64, valid IDs `9007199254740992` and
`9007199254740993` collapsed to the former value and the CLI rejected them as
duplicates. An envelope with a valid `segments` array and an unknown field
`"ignored":1e400` also failed. The first decode targets `any`, so it converts
numbers throughout the payload to `float64` before the segment array is
marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35)
· [Array extraction](../../../internal/core/schema/transcript.go#L115)
**Remediation.** Select the top-level shape while retaining the raw segment
array, then decode it directly into the typed source segment fields. Let the
typed decode handle range and type validation.
## 7. An extra numeric validator reference can discard valid sibling decisions
**Expected.** If a validator returns one usable decision per requested
reference, unexpected reference numbers should be ignored and the successful
batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching)
**Observed.** A fixture returned two valid approvals and an extra decision
with index `34`. Both corrections applied. Keeping the approvals and changing
the extra index to `9223372036854775808` or `34.0` made both decisions
unavailable with `invalid_output`, so neither correction applied. The schema
accepts those nonnegative integer values. The adapter decodes all indices into
Go `int` before reconciliation, so one schema-valid but incompatible number
discards the batch before the out-of-range index can be ignored.
[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json)
· [Adapter decode](../../../internal/framework/llm/adapter.go#L191)
· [Decision index type](../../../internal/framework/validators/llm_models.go#L46)
· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299)
**Remediation.** Preserve validator references as JSON numbers until matching
them against the known local range. Normalize integral numeric spellings and
classify out-of-range values as extras before converting requested references
to Go indices.
## 8. Span-only protection can reject a valid edit or miss a repair opportunity
**Expected.** The protection validator should check whether a correction
changes protected occurrences in the target segment. A known protection
rejection can feed repair, and application should still recheck against the
text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution)
· [Processing sequence](../../internal/pipeline.md)
**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in
`The certAIN result follows.` was rejected because the isolated span looks
like a protected occurrence. The full segment has no occurrence of `AIN` as a
name. In the other direction, a grammar change from `Silver` to `Golden` in
`The Silver Shield is equipped.` passed deterministic protection with
`Silver Shield` protected. It received two scripted model approvals and was
skipped only at application. With a positive repair budget, the provider
received no repair request. The validator normally compares the spans alone,
while application compares the whole segment.
[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100)
· [Repair eligibility](../../../internal/framework/runner/repair.go#L40)
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
**Remediation.** Use the shared segment preview for applicable candidates in
all modules and preserve the spoken-word repetition exception. Keep the
application check against the current working text for conflicts introduced by
earlier edits.
## 9. Credential redaction can change the ordinary YAML schema
**Expected.** The ordinary reports should retain their fixed schema, including
the required `summary` key and documented status values, while sanitizing
credential content. [Report schema](../../integrations/output-files.md#corrected-transcript)
**Observed.** With the synthetic credential value `summary`, a successful
processing run produced YAML with `"[REDACTED]"` in place of its required
`summary` key. After serializing the typed report, redaction applies a
substitution to every string scalar, including field names and fixed status
values. The CLI exited 0 despite the schema change.
[Report serialization](../../../internal/core/reporting/report.go#L182)
· [YAML redaction](../../../internal/core/redaction/redaction.go#L104)
**Remediation.** Sanitize input-derived report values before serialization.
Keep Audita-owned field names, status enums, and reason codes under the
control of the typed report serializer.
## 10. A completed proposal can disappear from the interruption ledger
**Expected.** Debug capture should retain proposed candidates that reached a
usable response and were subsequently interrupted. Their ledger rows should
record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files)
· [Candidate capture](../../internal/diagnostics-reporting.md)
**Observed.** In a two-chunk run, the first chunk's proposal request was held
open while the second returned a correction. After the second chunk's debug
response artifact appeared, SIGTERM stopped the first request. The run
published no transcript and correctly reported one usable proposal response
and two interrupted work units. The correction ledger contained only the
module failure row. The runner appends candidates only as it drains the
ordered validation queue. A fatal return or cancellation can leave a complete
later response in that queue or skip queuing it after recording its operation.
[Ordered collection](../../../internal/framework/runner/runner.go#L491)
· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
**Remediation.** Retain successful section results independently of validation
scheduling. After joining workers on a fatal path, assign indices to
unprocessed successful responses in section and response order and record
their candidates as interrupted.
## 11. Required delivery failures omit the phase in run error records
**Expected.** A run-scoped error should identify its phase, including failed
required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records)
**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1
and wrote a failed `errors.yml` record with `reason: output_failure` and the
destination path. The record omitted `phase`. The output session distinguishes
transcript writes, report writes, and cleanup through `FileFailure.Kind`.
The CLI mapper fills `Phase` only for report initialization.
[Failure classification](../../../internal/core/diagnostics/output_session.go#L273)
· [Run error mapping](../../../internal/cli/process_output.go#L92)
**Remediation.** Map each required output failure kind to the corresponding
run error phase, preserving the specific write or cleanup operation through
finalization and any failed-report retry.

View File

@@ -1,223 +0,0 @@
# Algorithmic correctness and efficiency audit
Reviewed the implementation at `0eca7b8`. Complexity estimates come from
source and call paths. Small synthetic probes checked match expansion,
ambiguous spans, timestamp ties, and strict parsing. The audit used no
production workloads or provider measurements.
The full offline suite passed with `go test -count=1 ./...` on Go 1.27.3.
## Architecture and review scope
Audita is a single-process Go CLI. CommandKit routes commands and streams,
ConfigKit resolves configuration, and the core packages parse, normalize, and
chunk transcripts and project glossary selections. The runner executes
correction modules in order over fixed chunks. Deterministic and LLM-backed
validators check proposals and each repair revision. PromptKit owns provider
admission, transport, and conversation coordination. The output session
publishes corrected transcripts, ordinary reports, and optional debug evidence.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
The review followed work from configuration and normalization through proposal
application and evidence assembly. It preserves the selected forward chunker,
stage barriers, replacement policies, PromptKit admission and replay, and
validator budgeting rules. Costs required by emitted payloads or full report
matrices are treated as output costs. The earlier correctness findings are
recorded separately. [Pipeline](../../internal/pipeline.md)
· [Validator budgeting](../../internal/validators.md#supplemental-validation-context)
· [Deferred traversal design](../future.md#independent-whole-pipeline-traversal-by-chunk)
· [Correctness audit](01-correctness.md)
## Efficiency and scalability findings
### 1. Normalization repeatedly copies and scans growing merge runs
`shouldMergeWithReason` constructs the merged text and estimates its tokens
from scratch. On success, `mergeSegments` constructs the text again and
allocates a new category slice that copies the entire prefix. For a run of
`k` merged segments with fixed-size nonempty text and category lists, the
merge pass reads and copies `Θ(k²)` text bytes and category elements.
The exact bound is proportional to the sum of the merge-prefix sizes.
The token cap usually limits the run, but
nonempty whitespace-only fragments with short gaps can merge at zero estimated
tokens. The cost is therefore possible under the default token cap.
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
· [Merge check and construction](../../../internal/core/normalization/normalize.go#L153)
· [Token estimator](../../../internal/core/normalization/tokens.go#L14)
· [Accepted segment text](../../integrations/transcript-glossary-files.md#transcript-json-contract)
Keep the forward merge pass. Accumulate text in an amortized builder and append
categories to an owned buffer. Maintain unrounded word and punctuation counts
per segment and combine them with the joiner before testing the token cap.
The ellipsis joiner can start a word when the previous text ends in whitespace,
and punctuation must be rounded only after adding all components. Check time
limits before processing text. This makes the merge pass linear in text bytes
and category elements, in addition to the existing chronological sort.
### 2. Application reporting searches the source chunk for each skipped edit
Application builds a segment-ID index inside `ApplyProposals`. The subsequent
`attributeApplyFiltering` pass indexes the approved proposals and sections,
but searches the original section's segment slice again for each skipped
proposal. With `s` segments and `k` skips in a section, this is
`O(s × k)` segment comparisons. Two identical, approved corrections per
segment make the second skip after the first changes its span. With one skip
per segment, the scans visit exactly `s(s+1)/2` entries. These are ordinary
stale conflicts, so the work can produce zero filtering records.
[Indexed application](../../../internal/framework/proposals/apply.go#L42)
· [Skip attribution](../../../internal/framework/runner/runner.go#L877)
Return immediately when there are no skips. For affected sections, index the
original segments once or carry original applicability forward from validation.
This reduces target lookup to expected `O(s + k)` work per affected section.
If defensive apply-time filtering adds actual defects, merge them once per
originating operation. The current singleton merge can rebuild the growing
deduplication map and sort the list after each addition. Preserve the
distinction between a response defect and a stale conflict.
[Filtering aggregation](../../../internal/framework/runner/runner.go#L837)
· [Recovery contract](../../integrations/output-files.md#error-and-recovery-records)
### 3. Validator glossary context expands every occurrence into every contributor
Context uses `ProtectedVocabulary.occurrences`, which emits one record per
matched occurrence per glossary identity. The context builder immediately
deduplicates those records into a set. For a spelling that occurs `m` times
and belongs to `g` selected entries, this costs `Θ(m × g)` records and
identity insertions after matching. A synthetic probe with 128 occurrences
of an alias shared by 96 entries produced 12,288 records to select 96
definitions. The deterministic protection check needs occurrence multiplicity.
The context builder only needs membership. [Occurrence expansion](../../../internal/framework/validators/protected_terms.go#L76)
· [Context selection](../../../internal/framework/validators/llm_context.go#L40)
Share the same scanner and match precedence. Let context deduplicate matched
spellings across the source, result, and neighbors before expanding their
identity lists. This case then takes expected `O(m + g)` membership work
before ordering the selected entries. Keep occurrence-level evidence for the
protection check and preserve glossary selection order.
### 4. Validation measures optional context that cannot fit
The production estimator correctly composes encoded fragment counts. Its
fitter nevertheless serializes all optional glossary entries and neighbors,
builds both prefix tables, and then removes entries from the end. It does
that even when the required fields alone exceed the budget. For `p` items
sharing `b` bytes of optional context across `g` entries, this can repeat
`Θ(p × b)` serialization and allocate `Θ(g)` prefix counters per fit even
when no optional entry is retained.
[Prefix measurement and fitting](../../../internal/framework/validators/llm_budget.go#L42)
· [Production batching caller](../../../internal/framework/validators/llm_batching.go#L17)
Return the required-only result as soon as its cost exceeds the budget.
Otherwise measure glossary entries in order and stop at the first one that
makes the prefix exceed the budget. Measure neighbors only if the entire
glossary prefix fits. Costs are monotone, so this preserves the existing policy
of dropping neighbors before definitions and keeping the longest affordable
ordered prefixes. An incremental fit uses constant auxiliary space for
counters and meters only retained entries plus the first exclusion in each
eligible field. Reusable glossary entry costs can also be stored with the
immutable glossary view if repeated encodings remain material. The cost of
matching candidates to glossary entries is separate.
[Context priority](../../internal/validators.md#supplemental-validation-context)
### 5. Ordinary runs retain debug-only decision history
The runner only adds candidates to the returned candidate list when debug
capture is active. It still builds a per-module candidate slice on every run
and aggregates, sorts, and stores initial validator decisions and rejections
in every `ModuleResult`. Rejected records carry original and proposed text
references. With `d_i` decision and rejection records in module `i`, finished
modules retain `Θ(Σ d_i)` records through invocation finalization, and the
runner pays for ordering them even when their production consumer, the
correction ledger, is disabled. Ordinary reports read the work-unit states and
compact operation outcomes. The current section's decisions and rejections
still serve candidate validation and repair eligibility.
[Candidate collection](../../../internal/framework/runner/runner.go#L537)
· [History sorting](../../../internal/framework/runner/runner.go#L617)
· [Module retention](../../../internal/framework/runner/runner.go#L271)
· [Ordinary reporting](../../../internal/framework/processreport/report.go#L51)
· [Debug ledger wiring](../../../internal/cli/process_pipeline.go#L54)
Separate detailed evidence retention from section-local decision state. In
the compact CLI path, skip copying candidates into aggregate slices and release
decision and rejection detail once the section finishes. Keep the operation
outcomes, counts, and failure origin used by ordinary reporting. Debug runs can
continue to retain and order the full evidence, including partially completed
work after a fatal return.
### 6. Repeated pipeline selections duplicate setup work
Module specifications select and copy the rich entries and project the
protected vocabulary independently for each pipeline entry, before the runner
starts. For `m` entries with the same selections of size `g`, setup time and
retained projections grow as `Θ(m × g)` with bounded-size entry fields. The
same data is used to prepare stage prompt variables and compile matchers again
on every repeated stage. Separately, readiness inspects a profile and its
backend for every role, even when they share IDs. PromptKit returns detached
inspection data, including copies of extra parameters. Selecting the same
resources `m` times with `b` extra parameter values can add `Θ(m × b)`
inspection work before processing.
[Module projection](../../../internal/framework/contracts/contracts.go#L107)
· [Entry copies](../../../internal/core/glossary/selection.go#L20)
· [Glossary preparation](../../../internal/framework/runner/runner.go#L169)
· [Role readiness](../../../internal/framework/preflight/preflight.go#L129)
· [Profile and backend inspection](../../../internal/cli/process_inputs.go#L125)
Reuse read-only projections and prepared data for identical effective ordered
selections, including the glossary correction exemption in the key.
Alternatively, build the projection at the module boundary and release it
before the next stage to bound retained memory when selections differ. Cache
successful profile and backend inspections for the captured engine while
keeping role-by-role checks and first-failure attribution. Shared data keeps
identical views and inspections proportional to the number of distinct inputs
plus the number of stage entries. Request messages still carry the selected
context for each call.
## Additional correctness findings
### 7. The unique-span check accepts overlapping matches
The grammar prompt requires an original span to appear exactly once.
`PreviewProposalForSegment` uses `strings.Count`, which counts nonoverlapping
matches. A probe proposed changing `ha ha` to `heh` in `ha ha ha`. The span
starts at byte offsets 0 and 3, but the preview returned applicable with
`heh ha` as the result. The same preview governs validation and application.
[Unique-span instruction](../../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml#L19)
· [Preview](../../../internal/framework/proposals/preview.go#L35)
· [Application](../../../internal/framework/proposals/apply.go#L59)
For `require_unique`, locate the first occurrence and check for another
starting after its first byte, including an overlapping start. Stop as soon
as ambiguity is established. A successful check can use the recorded offset
to assemble the replacement. This retains a linear scan in the worst case for
a fixed literal and can exit early on ambiguity. Preserve the existing
nonoverlapping replacement behavior for `replace_all`.
### 8. Normalization can reorder utterances with exactly tied timestamps
`Normalize` compares only start times in an unstable sort. With 14 segments
whose starts alternate between 1 and 0 and whose ends are start plus one, a
normalization-only CLI run emitted the start-zero group in source-index order
`9,1,13,3,7,5,11`. Those segments have identical start and end times. With
distinct speakers, the ordering change occurs without any merge. The earlier
Python implementation at `0b1b670` ordered exact ties by source position, and
the Go rewrite recorded preservation of existing behavior as its baseline.
[Current sort](../../../internal/core/normalization/normalize.go#L58)
Preserve the input order for exact time ties. A stable sort or an explicit
source-position tiebreaker retains `O(n log n)` comparisons. Specify the
timestamp tiebreaker so later refactoring preserves utterance order.
### 9. The strict transcript parser panics on a missing ID
`ParseTranscriptJSON` calls the source parser, which allows omitted and null
IDs, then dereferences every ID before checking that IDs are sequential. A
package probe with an otherwise valid segment and no ID reproduced the nil
pointer panic. This affects the strict package helper, which currently has
test callers. The processing command uses the tolerant source parser and
assigns canonical IDs separately.
[Strict parser](../../../internal/core/schema/transcript.go#L65)
· [Sequential ID validation](../../../internal/core/schema/transcript.go#L204)
· [CLI input acquisition](../../../internal/cli/process_inputs.go#L46)
Check ID presence and return the normal field-validation error before
dereferencing. This adds a constant-time check per segment and preserves the
linear typed conversion pass.

View File

@@ -1,322 +0,0 @@
# Maintainability and complexity audit
Reviewed source at `717d59c`. The codebase graph identified complex control
flow and cross-package call paths. A Go AST scan counted branches and
declaration spans for 353 functions and methods across 43 tracked Go
directories, excluding test files. The score below starts at one and adds
one per conditional, loop, non-default switch or select case, and short-circuit
operator, including those inside callbacks. The ranking guided source review.
Findings are based on static analysis.
## Architecture
Audita is a single-process Go CLI. CommandKit routes commands and streams.
ConfigKit resolves application settings, and the core packages parse,
normalize, and chunk transcripts and derive glossary selections. The
runner executes modules in order over fixed chunks. Deterministic and
LLM-backed validators gate proposals and revalidate repairs. PromptKit owns
model execution, backend admission, and bounded conversation history. Audita
applies approved edits and derives reports and debug artifacts from processing
outcomes. [Package map](../../internal/overview.md)
· [Pipeline](../../internal/pipeline.md)
· [Ownership rules](../../policy/architecture.md#ownership-and-dependency-direction)
The runner and validator runtime had the largest non-test Go source totals,
at 1,373 and 1,351 physical lines respectively. The highest-scoring functions
concentrated in orchestration, model integration, and report projection.
| Function | Branch score | Declaration lines |
| --- | ---: | ---: |
| `runner.repairCandidates` | 46 | 225 |
| `runner.runModulePipeline` | 36 | 212 |
| `processreport.Build` | 36 | 140 |
| `(*LLMBackedValidator).Validate` | 31 | 170 |
| `(*Runner).Run` | 30 | 168 |
| `runner.validateSectionCandidates` | 30 | 154 |
| `(*RoleClient).prepareCall` | 28 | 153 |
| `processreport.BuildCorrectionLedger` | 27 | 176 |
| `proposal_generation.GenerateCandidates` | 25 | 148 |
| `diagnostics.SelectOutput` | 21 | 66 |
## Refactoring opportunities
### 1. Give module execution one place to commit its result
`Run` owns the stage barrier, working transcript, inventory, module setup,
application, and partial-return assembly. The pipeline-error, application-error,
and success paths each construct a module result and copy work units into the
invocation inventory. Repair approvals are marked interrupted in both `Run`
and `runModulePipeline`. The latter also coordinates two worker lifetimes,
ordered proposal indexing, cancellation, and evidence aggregation. Changing
what an interrupted unit retains requires following both functions and their
callbacks. The earlier missing-ledger-candidate finding follows from indexing
and retention occurring only while scheduling validation. [Module commit and failure paths](../../../internal/framework/runner/runner.go#L187)
· [Section orchestration](../../../internal/framework/runner/runner.go#L441)
· [Interrupted evidence finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
Extract the section validation-and-repair worker from the collector. Let the
collector own ordered indexing and the evidence returned by workers, with
one path to register a fatal cause and cancel peers. Let one module-level
completion path resolve applied, skipped, and interrupted repair outcomes
and commit updated sections after successful application. Preserve validation
starting as ordered proposals become available, deterministic proposal IDs,
joined workers on fatal paths, and the stage barrier.
### 2. Separate one repair branch from the section's repair loop
`repairCandidates` selects candidates, builds each request, interprets
delivered turns, revalidates revisions, attributes outcomes, and writes debug
artifacts. Nested closures share mutable `current`, `localRef`, `findings`,
attempt counts, and section-level slices. For example, a successful revision
updates the branch, original proposal filtering, revalidation operations, and
approval list inside the coordinator callback. Error, withdrawal, exhaustion,
and accepted final-turn paths finish that work at different points.
[Branch setup](../../../internal/framework/runner/repair.go#L56)
· [Turn handling](../../../internal/framework/runner/repair.go#L146)
· [Repair behavior](../../internal/modules.md#failure-and-outcome-behavior)
Keep the outer loop responsible for selecting and collecting branches. Have
one branch-local operation return the terminal outcome, optional approved
revision, and operation evidence. Separate construction of the next request,
assessment of a delivered turn, and optional attempt recording. Define the
branch's internal states once so application and reports can project their
own stable labels. Keep PromptKit responsible for history and turn limits.
Retain final-turn revalidation, target changes within the section, and
independence of sibling approvals.
### 3. Put validator applicability and decision meaning at their owning boundary
`feedbackCandidates` switches on validator names and module keys to decide
whether each validator has usable evidence. The validator implementations
also check previews and module policy. The runner then rewrites an
`original_text_presence` approval if its preview is ambiguous, so the same
validator's returned verdict can mean something different outside the runner.
`validateSectionCandidates` interleaves those checks with two eligibility
modes, cardinality checks, filtering, and evidence recording. The framework
also declares identical validator interfaces under `contracts` and
`validators`. [Feedback selection](../../../internal/framework/runner/feedback.go#L13)
· [Runner verdict handling](../../../internal/framework/runner/runner.go#L680)
· [Original-text validator](../../../internal/framework/validators/deterministic.go#L37)
· [Validator contract](../../../internal/framework/contracts/contracts.go#L29)
· [Duplicate interface](../../../internal/framework/validators/models.go#L95)
Put each check's applicability and verdict together in its validator or its
built-in registration. Use one validator interface and one internal decision
disposition, deriving serialized approval from it. Keep the runner responsible
for submitting the shortlist or collecting independent findings, checking
returned identities, and applying only fully approved candidates. Preserve
ordered chains, zero-budget shortlisting, unavailable decisions, and separate
revalidation of revisions. The earlier audit records the span-only
protection mismatch. [Protection correctness](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity)
### 4. Separate validation-batch assessment from the diagnostic writer
`LLMBackedValidator.Validate` interleaves request preparation, prompt selection,
batch dispatch, reconciliation, partial-result assembly, and debug payload
construction. Its nested writer consumes the original request, local
reference mapping, raw response, reconciliation result, and failure. Proposal
generation and validation declare writer interfaces with matching signatures
and duplicate four-field artifact types. Their adapter wrappers copy each
path from `llm.InteractionArtifacts`. Repair writes through the same concrete
writer directly. A change to the common artifact envelope or safe error
serialization therefore touches several packages.
[Batch loop](../../../internal/framework/validators/llm_validators.go#L112)
· [Proposal writer contract](../../../internal/framework/proposal_generation/generate.go#L23)
· [Validation writer contract](../../../internal/framework/validators/models.go#L84)
· [Writer adapter](../../../internal/framework/runner/runner.go#L994)
· [Repair artifact capture](../../../internal/framework/runner/repair.go#L165)
Give one batch operation responsibility for dispatch and assessment. Return
reconciled decisions, its compact outcome, and only the evidence needed by the
diagnostic path. Reuse one internal artifact value type and writer seam for
common metadata, redaction, and safe write failures. Keep the per-role
payload construction with its domain owner. Preserve local-reference
reconciliation, missing and duplicate decision handling, partial results on
fatal return, and nonfatal debug-write warnings.
### 5. Make the structured client's call lifecycle explicit
`prepareCall` allocates a logical call, validates the output target, builds
the native request, installs two observer paths, captures prepared evidence,
and returns a closure for decoding and completion. That closure assigns
captured `err` from a named result and executes a reverse-order stack of
two finalizers before returning the mutated response. `Coordinate` also
finalizes a pending call on its separate undelivered-result path. Following an
error across both entry points requires keeping the prepared snapshot, native
terminal event, raw output, typed result, and logical call status in mind.
`Failure.Kind` and reason-based logical-call classification both describe
malformed-output failures. The ordinary request also retains a continuation
field whose current callers are adapter tests. [Call preparation and finalization](../../../internal/framework/llm/adapter.go#L54)
· [Coordinated call](../../../internal/framework/llm/coordinate.go#L15)
· [Request contract](../../../internal/framework/structuredoutput/call.go#L28)
· [Failure contract](../../../internal/framework/structuredoutput/failure.go#L17)
Keep the two native entry points and make their shared finish operation
explicit. Move native request construction and result decoding out of the
bookkeeping closure. Have a concrete per-call record finalize observation,
execution counts, and response evidence exactly once on both success and
failure. Derive the existing diagnostic failure label from one classification
and assess whether the test-only continuation path still needs to be exposed.
Preserve terminal counts, preparation failures without a snapshot, and the
rule that an undelivered response supplies diagnostic evidence only.
[LLM runtime](../../internal/llm-runtime.md)
### 6. Use a single proposal handoff and wire repair capability explicitly
`GenerateCandidates` returns a correction slice, parallel segment references,
and a second enriched proposal slice with its own indexing inputs.
`ExecuteModuleProposal` forwards only the first two slices and the
exchange. The runner later builds its own enriched candidates, which is where
stage-wide index assignment takes place. Repair recovers the request-local
segment reference by the proposal's slice position or falls back to looking
up the resolved target. Separately, an enabled repair budget, proposal-exchange
capture, the module's repair prompt method, and the repair client's coordinator
capability are established at different boundaries.
[Proposal result and mapping](../../../internal/framework/proposal_generation/generate.go#L72)
· [Module handoff](../../../internal/framework/proposal_generation/module_proposal.go#L75)
· [Ordered index assignment](../../../internal/framework/runner/runner.go#L521)
· [Repair readiness at use](../../../internal/framework/runner/repair.go#L98)
Carry a single proposal record with the resolved target and original local
reference, then assign the stage-wide index at the ordered runner boundary.
Assess retirement of the generator's separate enriched return once its
callers are migrated. Resolve repair availability with the concrete
module and role client at module setup. Use that capability to determine
exchange capture, and keep the disabled path simple. Preserve original
response order, invalid reference evidence, required profile readiness,
and zero-budget behavior.
### 7. Give reporting one source for operation classification and counts
`processreport.Build` has eleven work-unit and operation counters, nested
aggregation and projection, and a separate pass for run failures.
`SummaryOnly` bypasses selected detail paths inside that aggregation.
The LLM adapter decides recoverability through `structuredoutput.Failure`,
while `Build` uses a separate allowlist of recoverable reason codes after
the class has been discarded from the operation. Adding a recoverable
reason can therefore require coordinated edits to the adapter, domain
operation, and report counting. Safe report messages are a presentation
concern and can stay at that boundary.
[Snapshot aggregation](../../../internal/framework/processreport/report.go#L36)
· [Report-side classification](../../../internal/framework/processreport/report.go#L212)
· [Processing operation](../../../internal/framework/outcomes/outcomes.go#L84)
· [Adapter classification](../../../internal/framework/llm/adapter.go#L228)
Give operation outcomes a single source for success and recoverability.
Accumulate work-unit facts and summary counts in one pass, then project
rows and detail records when requested. Keep presentation ordering and safe
messages explicit. Preserve unknown versus zero counts, a single recovery
per successful operation, and agreement among YAML, Markdown, and CLI counts.
[Ordinary report contract](../../integrations/output-files.md#summary-counts)
### 8. Join correction evidence before building ledger rows
`BuildCorrectionLedger` indexes decisions, repairs, and skips, then emits
applied, skipped, rejected, repaired, interrupted, and module-failure rows
from separate loops. Each candidate path repeats identity, replacement,
and decision projections. Precedence is implicit in iteration order,
the `seen` map, and checks for repair evidence. It is easy to add a field to
one row shape and miss the other cases. The ledger also looks up validator
class by name after execution, while the runner can use metadata supplied
by the validator itself. [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
· [Repair and interruption rows](../../../internal/framework/processreport/correction_ledger.go#L197)
· [Decision grouping](../../../internal/framework/processreport/correction_ledger.go#L247)
· [Runtime class selection](../../../internal/validators/metadata/metadata.go#L44)
Gather the existing evidence into one per-proposal record keyed by module
instance and proposal index. Determine final disposition in one place, then
project common fields once and add the application and repair fields that
belong to that disposition. Consider recording the execution class used
when the decision is made, preserving built-in metadata. Keep module-only
failures distinct and retain the existing ordering, empty replacement fields,
partial decisions, and repaired skip reasons.
[Debug ledger contract](../../integrations/output-files.md#debug-only-files)
### 9. Clarify output preparation, publication, and failure ownership
The CLI builds initial and final report bundles and two failed-report
renderers. Preflight wraps the initial renderer to cover setup cancellation.
`OutputSession.Initialize` and `Finalize` each iterate required reports and
run a second write pass for failed status. File failure purposes are strings
interpreted again in the CLI. Destination selection also infers its role by
comparing the path string to the transcript destination, and preparation and
delivery each re-derive aspects of the path and file type.
[CLI preparation](../../../internal/cli/process_inputs.go#L70)
· [Preflight handoff](../../../internal/framework/preflight/preflight.go#L41)
· [Output selection](../../../internal/core/diagnostics/output_session.go#L61)
· [Publication and retry](../../../internal/core/diagnostics/output_session.go#L222)
· [Failure projection](../../../internal/cli/process_output.go#L92)
Keep a checked destination with its explicit role and publication mode.
Reuse a report-write helper for both passes and make write or cleanup phases
typed facts of the output session. Keep report rendering and status in one CLI
handoff that can distinguish a valid failed snapshot from an unavailable one.
Preserve readiness before writes, intentional in-place transcript replacement,
required-output ordering, cleanup, and the bounded best-effort failed-report
pass. Related destination, cancellation, and phase defects are in the
correctness audit. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Correctness audit](01-correctness.md)
### 10. Share root and per-module configuration rules
`resolveSettings` repeats the same presence, null, inheritance, and fallback
logic for the validation and repair profile fields. Each root field also
has a separate fallback path after the stage loop. `validate` repeats the
LLM value checks at the root and at every stage, with a separately built
ConfigKit path for each. Repair and validation currently agree, but changing
one profile rule requires reading multiple versions of it.
[Profile resolution](../../../internal/core/config/resolve.go#L24)
· [Configuration validation](../../../internal/core/config/config.go#L143)
Use a small typed helper for profile fallback that receives the correct
path, value, and dependency. Capture explicitly supplied root values before
recording root fallbacks, so a stage that selects a different proposal
profile still resolves correctly. Validate the common LLM settings with one
function taking a path prefix. Keep root-only constraints and readiness
separate, and preserve recorded origins, explicit null, and zero values.
[Configuration contract](../../config.md#settings)
## Additional correctness findings
### 11. A contradictory validator decision can advance an unavailable candidate
The internal `Decision` accepts `Approved: true` together with
`DispositionUnavailable`. Cardinality checks validate only indices. With
one such decision and no other rejection for that candidate, the runner
treats it as approved in the shortlist path and leaves it unblocked in the
independent-feedback path. This violates the invariant that an unavailable
assessment cannot advance as approval. Bundled producers set matching
values. The defect is at the internal validator result boundary. Make the
disposition authoritative or reject contradictions before processing the
decision. [Decision and cardinality](../../../internal/framework/validators/models.go#L56)
· [Approval and blocking](../../../internal/framework/runner/runner.go#L769)
· [Validator guarantees](../../internal/validators.md#llm-validator-batching)
### 12. The runner accepts an already canceled empty pipeline
With valid inputs and no module specifications, `Run` constructs the chunk
inventory and returns success even when the parent context was canceled
before entry. Its cancellation check sits inside the module loop. The
processing CLI checks cancellation around the runner, so the gap is in the
runner contract. Check the context at entry and at the run's successful
completion boundary, including normalization-only invocations.
[Runner entry](../../../internal/framework/runner/runner.go#L121)
· [Runner success return](../../../internal/framework/runner/runner.go#L286)
· [CLI cancellation check](../../../internal/cli/process.go#L65)
· [Pipeline cancellation contract](../../internal/pipeline.md)
### 13. Missing failed-report rendering can erase initialized evidence
`preflight.Prepare` converts a nil `FailedReports` callback into a callable
function returning a zero-value `ReportBundle`. If reporting is enabled and
cancellation arrives after initialization, preflight passes that empty
bundle to `Finish`. `Finalize` writes its three nil payloads as empty files,
replacing the existing `in_progress` reports. The initialization retry can
similarly replace reports that were written successfully in its first pass.
The CLI supplies a failed renderer for its ordinary reporting path. At the
internal preflight seam, carry the absence of a failed bundle through to
publication or require a valid failed bundle before acquiring resources,
so the existing evidence survives when a failed snapshot cannot be rendered.
[Callback wrapping](../../../internal/framework/preflight/preflight.go#L68)
· [Initialization write retry](../../../internal/core/diagnostics/output_session.go#L222)
· [Final report write](../../../internal/core/diagnostics/output_session.go#L273)

View File

@@ -1,227 +0,0 @@
# Concurrency audit
Reviewed source at `c0193df`. The full offline suite passed under
`go test -race -count=1 ./...` with Go 1.27.3. Selected capacity,
cancellation, coordination, and observation tests also passed eight runs
under the race detector. The timing findings below use Linux subprocesses,
synthetic loopback responses, and a temporary runner probe. The audit used
no live provider calls.
## Architecture and synchronization reviewed
Audita is a single-process Go CLI. The executable translates SIGINT and
SIGTERM into context cancellation. CommandKit handles commands and streams,
ConfigKit resolves settings, and the deterministic core normalizes and chunks
the transcript. The runner owns one stage barrier across fixed chunks.
Proposal and validation goroutines operate within a stage, and the runner
collects their results in a stable order before applying approved corrections.
The invocation shares a PromptKit engine across roles, with backend-local
admission, synchronous execution observations, and coordinated repair
branches. The CLI joins processing before assembling reports and delivering
output. [Signal setup](../../../cmd/audita/main.go#L16)
· [Package map](../../internal/overview.md)
· [Pipeline](../../internal/pipeline.md)
· [LLM runtime](../../internal/llm-runtime.md)
- Proposal workers send one result per chunk. Their owner waits for all
senders before closing the channel. The validation channel is drained after
the worker join, which works because it has one slot per chunk and at most
one send per validation worker. Reducing that buffer would require changing
the drain order. A mutex protects the first fatal error, and capture of
warnings and execution evidence completes before the CLI reads it.
[Proposal worker lifecycle](../../../internal/framework/runner/runner.go#L385)
· [Error and result ownership](../../../internal/framework/runner/runner.go#L463)
· [Validation join](../../../internal/framework/runner/runner.go#L592)
· [Warning capture](../../../internal/cli/process.go#L42)
· [Evidence snapshot](../../../internal/cli/process_pipeline.go#L51)
- Existing tests cover cancellation while a peer call is active and another
waits for backend capacity, release after cancellation, independent backends,
repair response withholding, and nested validation on the repair backend.
Observations use synchronized collections, and a snapshot is taken after
workers join. Config and prompt capture happen before worker startup.
[Runner cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
· [Backend capacity tests](../../../internal/framework/llm/adapter_capacity_test.go#L42)
· [Repair coordination tests](../../../internal/framework/llm/coordinate_test.go#L77)
· [Concurrent observation test](../../../internal/framework/llm/observations_test.go#L59)
· [Snapshot contract](../../../internal/framework/llm/observations.go#L81)
## Findings
### 1. Blocked output delivery can ignore cancellation and report success
The CLI checks the context immediately before `OutputSession.Finish`.
Delivery then opens a named pipe or writes stdout without observing the
context. With no FIFO reader, SIGTERM left the child blocked with
`in_progress` reports. When a reader attached, the child delivered the
transcript, finalized the reports as `completed`, and exited 0. The same
outcome occurred with a full stdout pipe and a 240 KB transcript. Draining
the pipe after SIGTERM allowed the write to finish and the child to report
success. The FIFO case was first recorded in the correctness audit.
[Last context check](../../../internal/cli/process_output.go#L54)
· [Stream open and write](../../../internal/core/diagnostics/output_session.go#L318)
· [Earlier FIFO finding](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success)
Pass cancellation through delivery. For cancellable file descriptors,
make both reader availability and write readiness interruptible. Keep the
guidance for subprocess clients to drain both streams, and define how cancellation
after a partial stream write is reflected in status and reports. Check
cancellation before committing a regular-file rename and use the existing
failed-report path when delivery is interrupted.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
### 2. Waiting for transcript input can prevent shutdown before preflight
`loadProcessInputs` calls a context-free `os.ReadFile` for the transcript.
When the path names a FIFO, it blocks on a writer before the application
can check for cancellation again. In a subprocess probe with no writer,
SIGTERM left the child blocked without starting report initialization.
Supplying the input afterward allowed the child to exit 1. The process
therefore relies on a producer or forced termination to unblock that
read. The input parser accepts the JSON once it arrives.
[Input acquisition](../../../internal/cli/process_inputs.go#L46)
· [File read](../../../internal/core/io/files.go#L9)
If pipe input remains supported, use a context-aware open and read on
supported platforms and stop reading after cancellation. If support is
intentionally limited to regular input files, make that contract explicit
and validate the path before a blocking open.
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
### 3. Cancellation can cross the runner's module commit boundary
`Runner.Run` checks the context after the module's workers have joined,
then applies corrections, classifies work units, and commits the new sections
without another cancellation check. A temporary runner probe canceled the
parent immediately after that check sampled a live context. The final
module returned success, an applied correction, and a completed work unit.
An already-canceled invocation with no modules also returned success, as
the check is inside the module loop. The CLI has later context checks that
prevent publication in these cases, but can render canceled-run evidence
counting edits committed after the cancellation. Normalization and encoding
also run synchronously between CLI cancellation checks, which can delay
shutdown for large workloads.
[Before application](../../../internal/framework/runner/runner.go#L190)
· [Commit and return](../../../internal/framework/runner/runner.go#L264)
· [Local processing boundaries](../../../internal/cli/process.go#L55)
· [Earlier empty-pipeline finding](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline)
Check cancellation at runner entry, before starting another module,
and just before committing computed updates. Keep the previous working
transcript until the commit check passes. If cancellation during large local
passes needs a shorter response time, add cooperative checks at natural
iteration boundaries.
[Cancellation and partial-result contract](../../internal/pipeline.md)
### 4. A slow earlier proposal holds back later validation and retention
The runner buffers completed proposals until the next chunk in order is
available. It assigns stage-wide indices, starts validators, and captures
ledger candidates only as that prefix advances. With two chunks and the
first response deliberately held open, the second response reached its
debug artifact while no validator call started. Releasing the first
response let both validators check the later correction. In a separate
run, canceling at the same point retained the later raw response and
counted one usable proposal response, but the correction ledger contained
only the module failure row. A slow earlier retry or timeout can therefore
stall independent validator work, and interruption can omit a completed
candidate from the ledger.
[Ordered proposal buffer](../../../internal/framework/runner/runner.go#L494)
· [Candidate capture and dispatch](../../../internal/framework/runner/runner.go#L520)
· [Earlier ledger finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
Retain completed responses independently of scheduling and assign
deterministic indices to the retained evidence on a fatal return. For
validation latency, consider section-local identities during checking and
repair, with stable stage-wide identities assigned during collection.
Account for diagnostic filenames and report references in that change.
This can preserve the existing stage barrier.
[Stage barrier decision](../future.md#independent-whole-pipeline-traversal-by-chunk)
### 5. Proposal fan-out makes admission under finite capacity depend on scheduling
Each chunk immediately attempts a proposal in its own goroutine. PromptKit
limits admitted calls, while Audita has no ordering at the admission boundary.
With 12 chunks, one backend slot, and zero queue capacity, six otherwise
identical runs each reached the synthetic provider once and recorded 11
recoverable capacity errors. Chunk 12 entered in five runs and chunk 2
entered in the other. Which chunks can produce corrections can therefore
vary before the provider sees any prompt. The backend enforced its configured
limit and the error reports reflected the rejections. The weakness comes
from submitting a complete transcript's worth of proposals at once to a
finite queue.
[Proposal fan-out](../../../internal/framework/runner/runner.go#L385)
· [Capacity classification](../../../internal/framework/llm/adapter.go#L254)
· [Engine-local admission](../../policy/architecture.md#correction-and-concurrency-invariants)
Make the interaction between chunk fan-out and finite queue capacity
explicit in operator guidance. If the desired processing policy is to wait
for more chunks to receive a turn, introduce bounded or ordered dispatch for
that policy while preserving PromptKit's ownership of backend permits and
the existing recoverable-rejection behavior for configurations that select
it.
### 6. Report paths can change referent between initialization and finalization
Preflight checks report destinations once. Each subsequent `atomicReplace`
resolves the original path again, including during the final report pass.
In a loopback probe, a report-directory symlink pointed to one directory
during initialization and was switched to another while the proposal
request was held. The CLI exited 0. The original directory still held
`in_progress` reports, and the replacement directory received
`completed` reports for the same run ID. Destination collision checks only
covered the referents inspected during preflight.
[Destination selection](../../../internal/core/diagnostics/output_session.go#L61)
· [Report initialization](../../../internal/core/diagnostics/output_session.go#L222)
· [Per-write path resolution](../../../internal/core/diagnostics/output_session.go#L410)
Bind the checked report destinations for the session, or detect a changed
referent and fail before writing to it. Use the same target identity for
initialization, finalization, and the failed-report retry. This complements
the earlier path-resolution finding, which covers a path whose meaning is
wrong even without a concurrent change.
[Earlier path resolution finding](01-correctness.md#1-output-path-resolution-can-replace-a-different-file)
### 7. Concurrent invocations sharing a report directory replace each other's evidence
Different transcript output paths can select the same report directory,
including by default when the outputs have the same parent. Each invocation
owns temporary files and has its own run ID, but all replace the same three
report basenames. In a probe, run A initialized reports and then waited for
its proposal response. Run B completed with a different transcript output in
the shared directory. Releasing A let it complete and replace B's completed
reports. Both commands exited 0. Because report writes are per file, a
consumer reading during overlap can also observe files from different runs.
[Report-directory selection](../../../internal/cli/process_inputs.go#L72)
· [Fixed basenames](../../../internal/core/diagnostics/output_session.go#L12)
· [Final report writes](../../../internal/core/diagnostics/output_session.go#L286)
Document per-invocation report destinations for concurrent subprocess
consumers. If supporting shared destinations directly, consider
invocation-specific publication or explicit serialization for that
destination. Per-file atomic replacement should remain in place.
[Report placement and lifecycle](../../operations.md#outputs-and-lifecycle)
### 8. Shared module and validator ownership is implicit at their interfaces
The runner shares one module instance across proposal workers and one
validator instance per chain entry across validation workers. Requests
also carry shared stage-level glossary data and transcript segment copies
whose category slices still alias the stage's data. The interfaces accept a
context but do not say that methods can run concurrently, borrowed data must
be read-only, or a canceled call must return for the stage to join. A
stateful implementation that mutates cached state or retains and mutates
borrowed data can race with peers. An implementation that blocks through
cancellation can prevent the worker join. Bundled modules and validators
use read-only shared configuration and per-call working state in the paths
reviewed. This is an internal ownership gap.
[Module and validator contracts](../../../internal/framework/contracts/contracts.go#L21)
· [Shared proposal instance](../../../internal/framework/runner/runner.go#L403)
· [Shared validation chain](../../../internal/framework/runner/runner.go#L548)
· [Shallow section copy](../../../internal/framework/runner/runner.go#L966)
Document concurrent invocation, read-only inputs, and cooperative cancellation
at the contract boundary. Keep state local to each call or synchronize shared
mutable state explicitly when a future implementation needs it. If an
implementation needs to mutate section data, give it an owned copy of that
data.

View File

@@ -1,276 +0,0 @@
# Resource management audit
Reviewed 2026-11-15 at `e4e5dd7`. This audit covers the Audita CLI, file and stream
I/O, configuration and prompt capture, transcript processing, the pinned
PromptKit execution path, and required and optional artifact generation. The
review also checked local release entry points for resource ownership. The
findings below identify cleanup defects and avoidable resource growth. The
remediation options are proposals for follow-up work.
## Architecture and lifecycle
Audita is a single-process Go CLI. The executable owns the signal context and
passes caller-owned streams through CommandKit. The CLI loads configuration and
captured prompt sources, reads the transcript, and preflights destinations. An
output session owns the temporary file and directory used to publish a regular
transcript by rename. Report replacements use their own short-lived staging
directories. Named pipes and stdout are delivered directly. Debug files are
written to a run-specific directory for later inspection.
[Signal setup](../../../cmd/audita/main.go#L16) · [Package map](../../internal/overview.md)
· [Preflight](../../../internal/framework/preflight/preflight.go#L41)
· [Output session](../../../internal/core/diagnostics/output_session.go#L45)
· [Artifact lifecycle](../../operations.md#outputs-and-lifecycle)
The CLI normalizes the transcript before passing it to the runner. The runner
holds the working transcript and outcome records in memory, creates fixed
chunks, and joins proposal and validator workers at each module boundary.
Role-bound clients share an invocation-local PromptKit engine for HTTP
requests, backend admission, retries, structural repair, and correction-repair
exchanges. The CLI assembles report and debug artifacts from the completed or
interrupted run.
[Normalization and invocation](../../../internal/cli/process.go#L55)
· [Runner ownership](../../../internal/framework/runner/runner.go#L121)
· [Worker joins](../../../internal/framework/runner/runner.go#L435)
· [Validation drain](../../../internal/framework/runner/runner.go#L592)
· [LLM runtime](../../internal/llm-runtime.md)
· [Report assembly](../../../internal/cli/process_pipeline.go#L50)
Resource release on the reviewed cooperative paths includes the executable's
signal subscription, proposal and validator worker joins, and PromptKit's
operation and request contexts, retry timers, admission reservations, and active
permits. PromptKit removes canceled waiters or transfers ownership of a permit
to the caller that won the race with cancellation. Its built-in HTTP path closes
response bodies and reuses the standard Go transport in the CLI. The pinned
source bounds successful response reads at 16 MiB and provider error reads at
64 KiB, applying reader limits even without a known content length. These are
per-response byte bounds. In the pinned module, the relevant ownership is in
`engine.go`, `internal/usecase/runner.go`, `internal/generation/coordinator.go`,
`internal/capacity/manager.go`, and the HTTP code in `internal/llm/`.
[Pinned version](../../../go.mod#L11) · [CLI transport selection](../../../internal/cli/process_inputs.go#L42)
· [Engine construction](../../../internal/framework/llm/runtime.go#L103)
· [Capacity cancellation test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
## Findings
### 1. Failed output cleanup loses ownership of temporary paths
`OutputSession.Close` closes the transcript descriptor, then tries to remove the
staged payload and its private directory. It clears both path fields even when
removal returns an error. A temporary permission or filesystem failure can leave
a directory or an unpublished payload behind. Finalization surfaces the first
cleanup failure, but the CLI's deferred `Close` has lost the paths it needs to
retry. The preparation and early preflight cancellation branches also discard
cleanup errors. The separate `atomicReplace` report helper ignores the results
of its deferred removes, so it can silently leave staging files or directories
on a cleanup failure. Recurring failures can accumulate disk blocks and inodes.
[Transcript preparation and cleanup](../../../internal/core/diagnostics/output_session.go#L163)
· [Finalization](../../../internal/core/diagnostics/output_session.go#L273)
· [Deferred close](../../../internal/cli/process.go#L41)
· [Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61)
· [Report replacement](../../../internal/core/diagnostics/output_session.go#L410)
A disposable probe against the unmodified `OutputSession` forced a nonempty
staging directory. The first `Close` removed the payload and returned an error.
After the probe removed the extra entry, the second `Close` returned success
while the empty directory remained. The probe cleaned up its own files.
Keep each path until removal succeeds or the filesystem says it is absent. Join
cleanup errors with preparation or publication failures and surface report
staging cleanup failures through the output owner. Distinguish a published file
with leftover staging from a failure to publish it.
### 2. Blocked stream I/O extends resource lifetime past cancellation
The input path uses `os.ReadFile` without a context. A FIFO can block before open
completes or hold an open descriptor while waiting for more input. Final
delivery similarly opens or writes a FIFO, or writes to the supplied stdout,
after the last cancellation check. With no peer or a peer that stops consuming,
the invocation retains its memory and any acquired stream descriptor. During
output it also retains the encoded transcript and rendered report buffers, and
the session cannot finish report finalization. A supervisor can therefore end up
waiting on canceled processes indefinitely. Prior subprocess probes reproduced
the blocked input, FIFO output, and stdout paths with SIGTERM and released them
by supplying or draining the streams.
[Transcript read](../../../internal/core/io/files.go#L9)
· [Final output check](../../../internal/cli/process_output.go#L54)
· [Output write and close](../../../internal/core/diagnostics/output_session.go#L318)
· [Output probe](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
· [Input probe](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
Make open and readiness waits cancellable for streams Audita owns on supported
platforms. For caller-owned writers, define how the caller supplies cancellation
or drains the stream. Ensure canceled delivery reaches session finalization,
and account for any bytes already delivered. A detached write goroutine alone
would still retain the writer and buffer if the peer stays blocked.
### 3. Backend admission leaves chunk fan-out and result retention unbounded
Each module starts one proposal goroutine per chunk. A worker builds a section
transcript and the domain and adapter prepare request inputs before PromptKit
admission. Backend capacity limits admitted runs and active provider attempts
when configured. The pinned `internal/capacity/manager.go` imposes no limit for
backend IDs without a configured pool. With many chunks, local goroutines and
preparation grow with chunk count even when a backend has a small configured
queue. Without a pool, concurrent HTTP requests can also grow with chunk count,
bringing sockets, request timers, encoded prompts, and response buffers with
them. Timeouts and byte limits apply individually. The `examples/service` profile
illustrates explicit backend limits.
[Proposal workers](../../../internal/framework/runner/runner.go#L385)
· [Adapter request preparation](../../../internal/framework/llm/adapter.go#L113)
· [Backend capacity example](../../../examples/service/prompts.yml#L1)
The proposal and validation channels and their companion maps are sized to the
chunk count. A slow leading chunk can keep many later proposal results, including
captured exchanges when correction repair is active, in the ordered pending map.
Validation can also start a goroutine for each ready chunk and holds completed
results in its channel until the join. PromptKit has already released those
calls' permits while Audita still holds their data. A long, finely chunked
transcript can raise client memory and descriptor demand even when each response
fits the per-call limit. Earlier probes substantiate the ordered backlog and the
interaction with a finite queue.
[Ordered buffering](../../../internal/framework/runner/runner.go#L460)
· [Validation workers](../../../internal/framework/runner/runner.go#L547)
· [Result collection](../../../internal/framework/runner/runner.go#L592)
· [Ordered backlog probe](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention)
· [Admission probe](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling)
Consider a bounded dispatch window tied to explicit scheduling or operator
settings. Drain validation results while workers run before reducing channel
capacity. A bounded result window or earlier use of section-local identities
could reduce retention while preserving deterministic correction identifiers.
Preserve the module barrier and PromptKit's permit ownership. Decide explicitly
when backend saturation should wait or remain a recoverable capacity rejection.
### 4. Whole-transcript copies amplify memory at input and publication
`loadProcessInputs` reads the entire file into a byte slice. The parser then
validates the bytes, decodes the whole JSON document into generic maps and
slices, re-encodes the selected segment array, and decodes that array into typed
segments. Extra fields can occupy memory in the generic tree even if the typed
transcript discards them. The CLI keeps the raw input and typed source reachable
through its `processInputs` struct, which it passes into finalization. The raw
input's last consumer is the optional source debug artifact. At the other end,
both output encoders build one map per segment and a complete indented JSON
payload before delivery. A large transcript can run out of memory before the
first provider call or after all correction work has completed.
[Input load and ownership](../../../internal/cli/process_inputs.go#L25)
· [Parser](../../../internal/core/schema/transcript.go#L35)
· [Segment re-encoding](../../../internal/core/schema/transcript.go#L115)
· [Last source use and finalization](../../../internal/cli/process.go#L51)
· [Bare-array encoder](../../../internal/core/schema/transcript.go#L224)
· [Envelope encoder](../../../internal/core/outputschema/registry.go#L62)
· [Related parse fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity)
Decode the supported input shapes into typed segments without the generic
round trip and release source representations after their last consumers. Typed
output projections would remove the per-segment maps. For regular files, an
incremental array encoder can write into the prepared temporary file and publish
only after encoding succeeds. Preserve both schemas and decide how a writer-based
encoder would handle partial output on a stream.
### 5. Summary-only runs retain a full grid of work and operation records
`Runner.Run` eagerly allocates and initializes one `WorkUnitOutcome` per
configured module and chunk before the first module starts. The work-unit grid
retains nested operations and repair summaries until `processreport.Build`.
Completed modules also flatten the same operations into `ModuleResult.Operations`.
The CLI selects `SummaryOnly` only after the runner returns when reports are
disabled. The builder scans the operation and work-unit records to count them,
then skips detailed rows. On many-module, many-chunk runs, this incurs memory
proportional to the module-by-chunk product and all retained operation records,
including entries for work never started after early failure. This is in
addition to the debug decision history identified in the efficiency audit.
[Grid allocation](../../../internal/framework/runner/runner.go#L127)
· [Operation copies](../../../internal/framework/runner/runner.go#L264)
· [CLI report selection](../../../internal/cli/process_pipeline.go#L78)
· [Summary aggregation](../../../internal/framework/processreport/report.go#L41)
· [Decision retention](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history)
When neither detailed reports nor the debug ledger is requested, fold finalized
work into invocation counters and derive not-started counts from the total
inventory. Retain the failed module identity for run-level errors. With details
enabled, consider one ownership path for operation records and deriving
not-started rows on demand, preserving the complete reporting grid and ordered
evidence.
### 6. Debug capture adds memory and disk pressure
Debug runs retain all logical calls and execution events in memory while also
writing several files per proposal, validation, and repair interaction. At the
end, the observation collector copies its records into a sorted snapshot.
`BuildLLMExecution` groups events by call and builds a second representation
for serialization. The JSON redactor creates encoded bytes, a decoded generic
tree, and final formatted bytes. Large event histories can produce a memory
peak while the transcript and the evidence needed for the correction ledger
are still live. Each debug invocation adds a run directory. Large or repeated
debug runs can consume space or inodes needed by the transcript and reports
when they share the filesystem. Once an artifact write fails, the CLI can
continue to collect and retain a warning for each later failure until
finalization, even if the destination stays full.
[Observer retention and snapshot](../../../internal/framework/llm/observations.go#L81)
· [Execution artifact assembly](../../../internal/framework/processreport/llm.go#L61)
· [JSON redaction](../../../internal/core/redaction/redaction.go#L53)
· [Run directories](../../../internal/core/diagnostics/run_dir.go#L34)
· [Interaction files](../../../internal/framework/llm/diagnostics.go#L35)
· [Warning accumulation](../../../internal/cli/process.go#L42)
· [Warning delivery](../../../internal/cli/process_output.go#L17)
Reuse a single ordered set of observations for final aggregation, or stream
redacted records through a disk-backed artifact while preserving the final
summary. Coalesce repeated debug write failures and report which evidence was
lost. Give operators visibility into debug footprint and an explicit retention
or quota decision for runs that need it. Keep optional evidence failure distinct
from required output delivery.
### 7. Correction repair replays growing history under per-correction budgets
Each actionable rejection starts a branch from the original section exchange.
PromptKit's pinned `internal/domain/exchange.go` shares the immutable history
through parent references. Its `internal/usecase/runner.go` flattens that history
with the next turn during preparation, and the HTTP client encodes it into the
request body. Audita revalidates returned revisions and restarts the configured
turn allowance for the next rejected correction. A large section prompt is sent
again on every branch. If a branch uses `A` turns, starts with `S` bytes of
history, and each turn adds roughly `H` bytes, the cumulative logical repair
request payload is on the order of `A*S + A²*H` for that branch. Retries and
revalidation add work. With debug enabled, Audita writes the prepared history
again for each logical repair turn. Across many actionable rejections, this can
increase provider context usage, outbound traffic, and disk use even when every
call fits the configured attempt and time budgets.
[Per-correction configuration](../../config.md#settings)
· [Branch start](../../../internal/framework/runner/repair.go#L85)
· [Coordination and per-turn artifacts](../../../internal/framework/runner/repair.go#L146)
· [Revalidation](../../../internal/framework/runner/repair.go#L227)
· [Prepared snapshot](../../../internal/framework/llm/adapter.go#L152)
· [Pinned version](../../../go.mod#L11)
Expose aggregate repair work and request size so operators can size runs. If a
work envelope is needed, consider an opt-in section or invocation budget that
accounts for revalidation as well as repair turns and records why any branch
stopped. For debug storage, consider recording a shared history once with
references sufficient to reconstruct the redacted request. Preserve the
conversation context used for editorial decisions.
## Verification
- The repository's complete `go test -count=1 ./...` suite passed on Go 1.27.3.
It includes output write and rename failure cleanup and release of backend
capacity after cancellation. The disposable probe in finding 1 exercises a
removal failure beyond those existing tests.
[Rename failure test](../../../internal/core/diagnostics/output_session_test.go#L98)
· [Transcript failure test](../../../internal/core/diagnostics/output_session_test.go#L394)
· [Capacity release test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
- The pinned PromptKit `internal/llm`, `internal/capacity`, and
`internal/generation` packages passed their existing tests. These exercise
response byte limits and body closure, capacity contention and cancellation,
and cancellation during retry delays.
[Pinned version](../../../go.mod#L11)
- Local release entry points were reviewed without executing a release. The
wrapper replaces its shell with ReleaseKit, and the CI bootstrap registers an
exit trap to remove its temporary directory.
[Release wrapper](../../../scripts/release.sh#L10)
· [CI bootstrap](../../../.woodpecker/release.yml#L29)
- The scaling findings describe allocation and ownership from source. Peak RSS,
throughput, provider cost, and filesystem-specific failure frequency were
outside the review's measurement scope.

View File

@@ -1,249 +0,0 @@
# Error-handling audit
Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt
configuration, transcript parsing and chunking, model and validator execution,
correction repair, output publication, and release bootstrap. It used the
checked-in contracts, the pinned dependency source, existing tests, and
disposable failure probes.
## Architecture and intended behavior
Audita is a single-process Go CLI. CommandKit routes commands and presents
outcomes. ConfigKit loads and validates application settings. On `process`, the
CLI parses the transcript and prepares the selected outputs before normalizing
and running the pipeline. PromptKit owns preparation, backend admission, HTTP,
transient retry, schema validation, and structural repair. Audita's role adapter
maps those errors to a class and a safe reason while preserving the typed cause.
The runner isolates recoverable operations, cancels and joins peers after a
fatal error, and returns partial facts for reports. The CLI checks for
cancellation, delivers the transcript and reports, and selects the exit status.
[CLI outcomes](../../cli.md#inspection-and-outcomes)
· [Pipeline](../../internal/pipeline.md)
· [Model integration](../../integrations/openai-compatible-llm.md)
· [Reporting and output ownership](../../internal/diagnostics-reporting.md)
The audit treated recoverable model errors, rejected corrections, and unavailable
validator assessments according to their separate contracts. Existing tests
confirm that one unavailable validation batch can leave other batches and later
modules usable, and that parent cancellation can stop active and waiting work
without publishing a transcript. Optional debug warnings can accompany exit
`0`, and a transcript can remain published if a later report write fails. These
are selected behaviors. The report boundary intentionally omits raw provider
causes, and input failures before initialization can leave no reports.
[Validation contract](../../internal/validators.md#llm-validator-batching)
· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091)
· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48)
· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179)
· [Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Early failure contract](../../integrations/subprocess.md#result-handling)
## Findings
### 1. Finalization publishes completed reports after it knows delivery failed
The CLI prepares a `completed` bundle for a successful pipeline and passes it
to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output
session before writing reports. Even if a transcript write or cleanup has
already failed, it writes the initial `completed` bundle and only then asks for
and writes a failed-status bundle. A disposable probe supplied a short stdout
writer. The failed-bundle callback received a known transcript write failure
and could already read `completed` in all three report files. `Finish` returned
the short-write error and the second pass rewrote the reports to `failed`.
[CLI bundle selection](../../../internal/cli/process_output.go#L27)
· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273)
· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318)
Reports should reflect a failure that is known before the first report write.
The existing nontransactional contract allows a later report failure to leave
mixed evidence. In this case the initial success claim is avoidable, and a
reader can observe it or it can persist if the failure rewrite cannot finish.
Select the failed bundle before publishing reports when transcript delivery or
cleanup has already failed. Keep a best-effort rewrite for errors discovered
during report publication, retaining the original error and any later failures.
If a failed bundle cannot be rendered, surface that error and preserve the
last written evidence. The initialization callback currently discards its
rendering error, and the finalization callback falls back to the previous
bundle, which can still say `completed`.
The missing phase for delivery errors is a separate, previously recorded issue.
[Output lifecycle contract](../../operations.md#outputs-and-lifecycle)
· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105)
· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46)
· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322)
· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records)
### 2. A provider request timeout is reported as the logical operation deadline
The profile's `timeout_seconds` bounds one provider attempt. The distinct
`llm.operation_timeout` applies to preparation, capacity waiting, retries,
generation, and structural repair. Pinned PromptKit deliberately reserves
`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as
`ErrTransport` with the underlying deadline cause. After checking the native
operation sentinel, Audita checks a generic `context.DeadlineExceeded` before
`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason.
In PromptKit, the distinction is documented in `engine.go` and implemented in
`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`.
[Timeout settings](../../migrations/v2.md)
· [Native runtime contract](../../integrations/openai-compatible-llm.md)
· [Classifier](../../../internal/framework/llm/adapter.go#L228)
· [Pinned PromptKit](../../../go.mod#L11)
A loopback provider withheld its response beyond a one-second profile timeout.
With a ten-second operation budget and zero retries, the CLI finished after
about one second. It returned `0` with a completed transcript and reported
the failed proposal as `operation_timeout`. Continuing the run is consistent
with both timeout classes being recoverable. The reason directs an operator
toward the logical operation budget when the attempt deadline expired. Use
the native sentinel for whole-operation timeouts, preserving parent cancellation
precedence. Classify an HTTP-only timeout as transport failure or adopt a
distinct safe reason for that cause if consumers need the distinction.
[Report message mapping](../../../internal/framework/processreport/report.go#L224)
· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
### 3. Missing required credentials produce a generic setup message
The pinned PromptKit use case identifies a missing required environment
credential through `ErrAPIKeyEnvMissing` and retains its configured variable
name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a
generic safe message before delivering the operation result to the domain. A
synthetic profile with a required unset variable stopped before HTTP and
exited `1`. Stderr ended with `structured model setup failed`, the YAML report said
`Processing setup failed.`, and the interaction error artifact repeated the
generic adapter message. The cause is available to Go callers via `Unwrap`,
but the operator-facing process path does not report the missing requirement.
[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243)
· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17)
· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231)
· [Final CLI failure](../../../internal/cli/process_output.go#L61)
The fatal class and exclusion of raw provider text are appropriate. The safe
diagnostic can still identify an unset required credential and the affected role
or profile. Map the native credential sentinel to an actionable safe message,
and include the configured variable name if obtained from safe profile or
backend metadata. Carry the typed cause internally and retain preparation at
the existing execution boundary, where role selection and credential validation
already occur.
[Credential ownership](../../internal/llm-runtime.md)
· [Recovery workflow](../../operations.md#recovery)
### 4. Fatal model errors lose their chunk and operation coordinates
Proposal and validation code records an operation reference even on fatal
returns. It can identify the section, role, validator, and batch. The runner
stores these operations per work unit, but its first-error path retains only
the error value. `executePipeline` reconstructs the fatal report origin from
the failed module without identifying which section or operation stopped it.
The report builder supports these coordinates for run-scope records. In the
missing-credential probe the run record included `grammar_1` and the stage
index, with no `chunk_index` or `operation`, even though the pre-dispatch
proposal invocation had both. Fatal validator batches lose their available
validator and batch coordinates at the same boundary.
[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116)
· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176)
· [First error and cancellation](../../../internal/framework/runner/runner.go#L477)
· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60)
· [Run origin serialization](../../../internal/framework/processreport/report.go#L155)
Retain the primary error and its available `outcomes.Reference` together when
the runner selects the failure that stops the module. This lets the CLI identify
the failed operation without guessing from canceled peers or counting the fatal
error again as a recoverable work-unit error. Use module-level context for
setup and application errors with no narrower origin.
[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records)
### 5. Invalid processing selections use the runtime failure exit status
The CLI contract gives exit `2` to invalid input or usage and exit `1` to
runtime or output failure. `loadProcessInputs` maps malformed input and
configuration to usage errors. Processing readiness is checked later inside
`preflight.Prepare`, where a missing selected profile, a profile reference
that cannot be resolved, or an empty selected glossary for a required glossary
module can fail. `runProcess` treats every error at this boundary as a runtime
failure, together with output and cancellation errors. A probe selecting an
unknown proposal profile returned `1` with a specific profile diagnostic, no
provider calls, and no reports or transcript. The existing integration test
for a similarly invalid profile inheritance reference expects `2`.
[Exit status contract](../../cli.md#inspection-and-outcomes)
· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41)
· [Selection validation](../../../internal/framework/preflight/preflight.go#L96)
· [CLI status selection](../../../internal/cli/process.go#L24)
· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122)
Distinguish rejected processing selections from context and destination failures
with a typed readiness error or a separate input-check step. Return usage for
invalid operator selections and preserve failure for cancellation, I/O, and
internal defects. Retain the existing early check and the ability to inspect
configuration without requiring a processing-ready pipeline.
[Inspection contract](../../cli.md#inspection-and-outcomes)
### 6. An infeasible chunk target is recorded as an internal processing failure
A positive `chunking.target_sections` can be valid configuration and infeasible
for the normalized input. The chunker rejects that combination and the runner
wraps the error with `chunk normalized transcript`. The CLI's run record defaults
to `phase: processing`, `reason: internal_failure`, and a generic message for
errors outside the structured model and cancellation classes. With one usable
segment and an explicit target of two, a CLI probe produced that failed report,
exited `1`, and made no model calls. This failure should remain visible when
reports are enabled because it occurs after initialization. The reported phase
and reason currently misidentify an operator-correctable constraint as an
internal defect in processing.
[Chunking settings](../../config.md#settings)
· [Feasibility check](../../../internal/core/chunking/sections.go#L143)
· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126)
· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60)
· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244)
Carry a typed chunking failure through the runner and map infeasible parameters
to `phase: chunking` and an input reason. Keep a separate classification for an
unexpected failure in the chunker. Align the exit mapping with the input
contract if the run distinguishes these classes at that boundary. The earlier
correctness finding about feasible targets being rejected concerns a separate
chunking decision.
[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records)
· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets)
### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty
The release job checks `test -z "$(git status --porcelain --untracked-files=all)"`
under `set -eu`. The `test` command's exit status controls whether the shell
continues. If Git emits a diagnostic on stderr and fails with empty stdout,
the test succeeds. A shell probe substituted a status command that exited
`128` with no stdout and confirmed that execution continued after the check.
The expected bootstrap behavior is to stop before dependency preparation if
cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared`
independently checks the status command's error before delivery, limiting this
finding to the earlier bootstrap gate and diagnosis.
[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17)
· [Release procedure](../../release.md#binary-delivery)
· [Pinned ReleaseKit](../../../go.mod#L12)
Assign the status output in a separate command and check the command result
before testing the content. Preserve the existing verification in ReleaseKit.
## Related failure paths already recorded
- Cancellation during FIFO or stdout delivery can be missed after the last
context check. A blocked FIFO input can also defer cancellation before
output preflight. The runner has a separate cancellation window before
committing module changes.
[Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
· [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
· [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary)
- The failure path can drop completed proposal evidence behind an earlier
section. The internal preflight seam can also overwrite initialized reports
with empty files when a failed renderer is absent.
[Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
· [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence)
- Output cleanup can discard removal errors or forget a staging path before a
later cleanup attempt. This differs from intentionally retaining requested
debug evidence or keeping a transcript already published successfully.
[Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
## Verification
The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a
temporary CLI build, a synthetic loopback provider, a disposable output-session
caller, and a shell simulation. They verified the failure conditions described
above without persisting tests or program changes. The release check was
reviewed locally against the pinned ReleaseKit implementation and exercised at
the shell expression, without running publication.

View File

@@ -1,254 +0,0 @@
# Security audit
Reviewed 2026-11-15 at `27e58fa`. The review covered application and prompt
configuration, transcript and glossary input, model transport and responses,
credentials, output publication, diagnostic serialization, and release
delivery. Findings identify the deployment conditions that make each issue
relevant.
## Architecture and trust boundaries
Audita is a single-process CLI. The invoking account supplies filesystem
authority and chooses the configuration and destinations. ConfigKit captures
application sources and PromptKit captures prompt and provider configuration.
The transcript, glossary text, and provider responses are data. Audita renders
model requests for selected roles, validates model responses structurally and
through domain checks, and publishes transcript files and reports. The OS or
subprocess parent controls who may invoke the CLI and under which account.
Provider authentication uses configured environment variables, with bearer
tokens added by the transport. Repair may intentionally send the original
proposal exchange and branch history to a separately selected provider.
[Application structure](../../internal/overview.md)
· [Trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope)
· [Configuration sources](../../config.md#loading-and-composition)
· [Model integration](../../integrations/openai-compatible-llm.md)
· [Repair data flow](../../operations.md#processing)
The audited input-to-model path reads transcript text as data, uses structured
schemas, checks candidate identity and applicable validators, and rechecks
protected terms when applying an approved edit. The embedded prompts tell
models to treat transcript text and repair evidence as untrusted data. Those
instructions guide editorial behavior. The runner maps model responses to
domain proposals and applies only approved edits. PromptKit validates the
initial endpoint, bounds per-response reads, and closes response bodies. For
host-backed prompt content files, PromptKit resolves the path relative to
captured provenance and the pinned ConfigKit opens it through a rooted
filesystem.
[Prompt data instructions](../../../internal/prompts/assets/shared/prompt_hardening.md)
· [Repair instructions](../../../internal/prompts/assets/shared/repair_contract.md)
· [Proposal application checks](../../../internal/framework/proposals/apply.go#L59)
· [Model and configuration versions](../../../go.mod#L9)
· [Transport limits reviewed previously](05-resources.md#architecture-and-lifecycle)
## Findings
### 1. Debug capture uses a shared path and can expose private artifacts
Debug is opt-in, and its default parent is `/tmp/audita`. `NewRunDirectory`
accepts an existing parent and creates the per-run directory with mode `0755`
before applying the process umask. Snapshot and interaction files use mode
`0644`. A normal run using a synthetic transcript and umask `022` produced
exactly those directory and file modes inside a disposable private parent.
Under a shared, traversable parent, those permissions let other local users
read the transcript snapshots, prompts, and responses. Even when credential
redaction succeeds, these artifacts contain confidential data for some runs.
[Debug default](../../../internal/core/config/config.go#L106)
· [Run directory creation](../../../internal/core/diagnostics/run_dir.go#L35)
· [Source snapshot](../../../internal/core/diagnostics/run_dir.go#L97)
· [Interaction permissions](../../../internal/framework/llm/diagnostics.go#L84)
· [Documented debug contents](../../operations.md#debug-evidence)
The predictable default parent is also a trust boundary. If it already
resolves to a directory writable by a different account, Audita accepts that
directory and later opens artifacts by path. Another account with directory
mutation rights can interfere with where evidence is written or with its
integrity. The random run ID helps avoid collisions. Directory ownership and
permissions need their own checks.
[Directory acquisition](../../../internal/core/diagnostics/run_dir.go#L43)
· [Run ID generation](../../../internal/cli/process_inputs.go#L142)
Use a per-user location for the implicit debug root and create private run
directories and files by default. If a caller explicitly selects a shared
location, document the ownership and permission assumptions. For directories
that can change concurrently, keep writes scoped to the acquired directory.
### 2. Provider redirects can cross the configured transport boundary
The process command gives PromptKit no custom HTTP client. PromptKit v0.23.0
constructs a Go client without a redirect policy and creates its generation
POST from a replayable byte reader. The Go client's default policy can follow
body-preserving redirects across origins. Its rules for forwarding authorization
headers operate independently of the body and use host trust rules rather than
the complete origin. A redirect can therefore replay transcript-bearing requests
to a different service. A downgrade on the same host can also retain the
authorization header while losing TLS protection.
[CLI transport selection](../../../internal/cli/process_inputs.go#L42)
· [PromptKit client construction](../../../internal/framework/llm/runtime.go#L103)
· [Pinned version](../../../go.mod#L11)
This matters when a provider gateway responds with a redirect across a routing
or TLS boundary that the operator has not authorized for transcript data. The
configured destination can be trusted while the destination of the redirect
has a different audience or transport security. The dependency paths are
`internal/llm/openai_compatible_client.go` and `internal/llm/http_evidence.go`,
with redirect and header rules in Go's `net/http/client.go`.
Consider a provider redirect contract that keeps generation on the selected
origin by default and rejects HTTPS downgrades. Where redirects are needed,
support explicit trusted destinations and define when credentials may be
forwarded. Keep the documented ability to deliberately select a local HTTP
provider separate from protocol changes during a request.
[Provider contract](../../integrations/openai-compatible-llm.md)
· [Local HTTP example](../../../examples/local/prompts.yml#L3)
### 3. Report destinations can escape the identity checked by preflight
Preflight checks report paths for collisions with input, configuration, and
other destinations. Each report replacement resolves the original path again
at publication. The concurrency audit verified a change of report directory
referent between initialization and finalization. If a subprocess processes
work in a directory whose entries can be changed by a lower-trust account, the
same behavior can direct a report write to an unintended file writable by the
Audita account, including a protected input or configuration file whose earlier
collision check passed. A private staging file protects incomplete bytes. The
final rename still uses the later resolution.
[Destination selection](../../../internal/core/diagnostics/output_session.go#L60)
· [Report finalization](../../../internal/core/diagnostics/output_session.go#L273)
· [Path resolution and rename](../../../internal/core/diagnostics/output_session.go#L410)
· [Earlier referent-change evidence](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization)
Bind publication to the directory and destination semantics checked at
preflight. For shared output directories, use directory-relative operations or
reject an unsafe change before writing. Preserve intentional in-place
replacement and the selected support for output symlinks. Use an
invocation-owned output directory when a subprocess parent handles
lower-trust files.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
### 4. Raw structured debug responses can retain an encoded credential
PromptKit returns the model message as `RawOutput`. The role adapter retains
that string as `FinalContent`, and debug writers store it beside the decoded
domain result. The artifact redactor parses the outer JSON artifact and
matches each string against the exact secret value and the escaped spelling
from Go's JSON encoder. It treats `final_content` as one string, even when that
field contains another JSON document. A credential present in a provider
response can therefore be masked in decoded fields but remain recoverable from
the raw response field when the provider used a different valid representation
of the same JSON string. The existing nested-content test covers the spelling
produced by Go's encoder.
[Raw output retention](../../../internal/framework/llm/adapter.go#L171)
· [Proposal artifact](../../../internal/framework/proposal_generation/generate.go#L158)
· [Validator artifact](../../../internal/framework/validators/llm_validators.go#L148)
· [Repair artifact](../../../internal/framework/runner/repair.go#L173)
· [Redactor](../../../internal/core/redaction/redaction.go#L18)
· [Nested JSON test](../../../internal/core/redaction/redaction_test.go#L43)
The condition is a provider returning credential-bearing content and an
operator retaining debug data. A gateway that echoes authentication material
into diagnostic model content can cause that exposure even if its JSON
encoder and Audita's differ only in escaping. Sharing the affected artifact
can disclose a reusable credential. Redact structured response content after
decoding it as JSON, or retain only the sanitized decoded view. If malformed
raw response text is useful for diagnosis, define how to retain it without
claiming decoded credential redaction for content that could not be interpreted.
[Credential handling requirement](../../policy/architecture.md#trust-boundaries-and-threat-scope)
### 5. Configuration discovery extends trust to the main file and its neighbors
Without `--config`, the working directory is searched before user and system
configuration. Once a main source is selected, conventional prompt and
glossary companions beside it are loaded as configuration. An explicit main
file still enables its companions. This is useful for an operator-managed
project, but it changes the trust boundary when a subprocess parent uses an
upload-controlled working directory or places trusted configuration beside
untrusted project files. Configuration discovered there can choose the model
destination and a credential variable present in the child process. In that
deployment, both text and a credential can be sent outside the destination
the integrator intended. A transcript passed as a data file to a trusted
working directory with a trusted configuration bundle does not carry those
configuration privileges.
[Application discovery](../../../internal/cli/app_wiring.go#L30)
· [Prompt imports](../../../internal/framework/llm/runtime.go#L36)
· [Captured main filesystem context](../../../internal/framework/llm/runtime.go#L75)
· [Subprocess guidance](../../integrations/subprocess.md#invocation)
· [Configuration composition](../../config.md#loading-and-composition)
Document that the selected main file, its automatic companions, and each
searched working directory are part of the trusted operator configuration.
For an integration accepting uploads, use a trusted bundle and pass the
transcript independently. Give the child process only credentials appropriate
to the provider choices authorized for that integration. An explicit discovery
option could support deployments that need a single configuration file.
### 6. Untrusted transcript filenames can become active Markdown
The CLI puts the input path in the report snapshot. The Markdown renderer
escapes HTML, table separators, backticks, and line breaks for path fields.
Markdown link and image punctuation still reaches the renderer as syntax. The
stage-label renderer already escapes those constructs. If a subprocess keeps
an uploader-controlled filename and displays rendered `report.md`, the name
can render as a link with a label that obscures the actual path. That changes
what the operator sees and can direct navigation to an unexpected destination.
The concern is the integrity of the rendered report.
[Path in the snapshot](../../../internal/cli/process_output.go#L24)
· [Path rendering](../../../internal/core/reporting/markdown.go#L41)
· [Path escaping](../../../internal/core/reporting/markdown.go#L191)
· [Stage label escaping](../../../internal/core/reporting/markdown.go#L132)
· [Existing stage-label test](../../../internal/core/reporting/markdown_test.go#L101)
Render paths as literal text with a context-appropriate Markdown encoder.
Keep exact path values in structured artifacts when the output contract calls
for them.
### 7. Untrusted input size and blocking I/O can exhaust subprocess workers
The transcript is read whole with `os.ReadFile`, then decoded and re-encoded
before chunking. Backend admission and per-response byte limits apply later.
For a parent accepting untrusted uploads without an input or process memory
budget, a sufficiently large valid transcript can exhaust the child or a
shared worker host before model calls are reached. A local integration that
allows input paths to resolve to a FIFO can also lose a worker to a read that
outlives cooperative cancellation. The size and cancellation paths were traced
in the resource audit, and the FIFO behavior was verified in the concurrency
audit.
[Input acquisition](../../../internal/cli/process_inputs.go#L46)
· [Whole-file reader](../../../internal/core/io/files.go#L9)
· [JSON acquisition and conversion](../../../internal/core/schema/transcript.go#L35)
· [Memory finding](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication)
· [FIFO finding](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight)
Keep the CLI suitable for trusted large jobs by avoiding redundant whole-input
copies. Integrations accepting untrusted input can enforce their own upload and
process budgets and require input from controlled storage. An application-level
input budget should be an explicit deployment choice. Where stream inputs are
supported, make their acquisition and reads cancellable.
[Subprocess timeout guidance](../../integrations/subprocess.md#invocation)
## Release boundary and verification
Release CI verifies the event commit and runs the tagged ReleaseKit version with
a delivery credential. The checked-in procedure relies on the forge and CI
operator to authorize who can create release tags and receive the delivery
secret. The local review covered the workflow, tag/commit checks, pinned
executable selection, and shell quoting. Forge permissions and running CI
secret settings were outside the observed state. The earlier bootstrap status
error is covered by ReleaseKit's independent prepared-checkout validation
before delivery.
[CI workflow](../../../.woodpecker/release.yml#L5)
· [Publication guards](../../policy/release.md#publication-guards)
· [Delivery procedure](../../release.md#binary-delivery)
· [Earlier bootstrap finding](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty)
- The complete offline `go test -count=1 ./...` suite passed on Go 1.27.3.
Existing tests exercise credential handling, redaction, filesystem collision
checks, cancellation, and model-validation boundaries.
[Credential redaction test](../../../cmd/audita/main_integration_test.go#L965)
· [Collision check test](../../../internal/core/diagnostics/output_session_test.go#L14)
· [Validation and transport test](../../../internal/framework/validators/llm_integration_test.go#L25)
- `go mod verify` verified cached module content. The pinned dependency source
was reviewed at the boundaries above. The checksum check does not establish
advisory status.
- A normal synthetic debug run substantiated the artifact modes. Other findings
are grounded in the cited source and contracts or in evidence already recorded
by the earlier audits. The review used local code and files throughout.

View File

@@ -1,355 +0,0 @@
# Test coverage and effectiveness audit
Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to
assess protection against consequential failures, test ownership, fixture
fidelity, and maintenance cost. The findings are follow-up options.
[Testing policy](../../policy/testing.md)
## Architecture and measured coverage
Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit
loads application settings, and core packages parse, normalize, and chunk the
transcript. The runner coordinates correction proposals, validators, repairs,
and edit application. The role adapter uses a shared PromptKit engine, and the
CLI publishes the transcript and reports through an output session. Tests use
pure domain inputs, temporary files, an injected HTTP transport or loopback
provider, and subprocesses. Release tests in this repository inspect the local
profile, while ReleaseKit owns the underlying delivery implementation.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
· [LLM runtime](../../internal/llm-runtime.md)
· [Release procedure](../../release.md#binary-delivery)
On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...`
run passed with `-coverprofile` and `-json` output directed to temporary files.
Go's cover tool reported **89.3%**
statement coverage. The JSON run log recorded 441 passing top-level tests and
210 passing named subtests. The package percentages below merge repeated
profile rows by source block before counting statements.
| Package or boundary | Statement coverage | Audit observation |
| --- | ---: | --- |
| `cmd/audita` | 0.0% | The test executable handles signals in a helper. |
| `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. |
| `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. |
| `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. |
| `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. |
| `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. |
| `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. |
| `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. |
| `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. |
The percentages describe statements reached in this run. The audit assigned
test value by the assertions and failure scenario. In particular, low coverage
in test support or thin forwarding wrappers does not justify standalone tests.
Existing higher-value checks include partial validation with surviving edits,
repair revalidation, typed failure causes, stable user-visible identities, and
the presence of a masked credential echo in an actual retained artifact.
[Coverage policy](../../policy/testing.md#coverage)
· [Partial validation](../../../cmd/audita/main_integration_test.go#L1091)
· [Repair integration](../../../cmd/audita/correction_repair_integration_test.go)
· [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10)
· [Credential echo](../../../cmd/audita/main_integration_test.go#L965)
## Behavior that needs stronger verification
### 1. Subprocess tests recreate the production executable boundary
The subprocess suite launches the test executable. Its helper registers
SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with
the returned code. The production `run` and `main` contain their own copy of
that wiring and recorded zero hits. A regression in production-only signal
registration, argument slicing, or exit propagation can pass the subprocess
suite, including the cancellation case. The existing tests still exercise
real process boundaries and the assembled CLI.
[Production entry](../../../cmd/audita/main.go#L12)
· [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28)
· [Signal test](../../../cmd/audita/main_integration_test.go#L1059)
Use the actual built executable for one representative startup and signal or
exit test, or have the helper invoke the same production startup function.
Keep the smaller existing seams for the rest of the CLI suite.
### 2. Inspection failure tests accept the wrong exit classification
`TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and
`TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero.
Both can pass if the integration maps invalid input and cancellation to the
same nonzero status. A temporary build given the same malformed operator
prompt file returned `1` via `promptkit list prompts` and `2` via `process`.
The CLI contract distinguishes input failure (`2`) from runtime failure (`1`).
The input test also sets a synthetic credential that its malformed templates,
profiles, and paths never reference, so the absence assertion provides no
evidence that a configured secret was found and redacted.
[Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24)
· [CLI exit contract](../../cli.md#inspection-and-outcomes)
· [CLI wiring](../../../internal/cli/app_wiring.go#L80)
Assert the selected status and a stable diagnostic fragment or typed cause.
Clarify the intended delegated-command classification if it differs from the
CLI contract. Test redaction where a known value reaches the relevant channel,
as the existing credential echo test does, and remove the unused secret
assertion from these cases.
[Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965)
### 3. Output tests primarily check the final artifact state
The short-write CLI test asserts final YAML status, counts, and a reason code.
The replacement test checks the result after a failed-report pass. Both can
pass when finalization briefly writes `completed` reports after it already
knows transcript delivery failed. The short-write test also passes if the
required run-error phase is missing. Collision tests cover aliases present
during selection, while the finalization path resolves report destinations
again. Existing tests that change the transcript's file type exercise a
different path. Prior audits documented both the transient report state and
the changed report referent.
[CLI short-write test](../../../internal/cli/process_output_test.go#L79)
· [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207)
· [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14)
· [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435)
· [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed)
· [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization)
At `OutputSession`, observe the first report state after a known delivery
failure and verify that a changed report destination cannot defeat the checked
identity. At the CLI boundary, assert the exit and serialized run phase. The
failure-injection seam is useful because it observes publication state at an
externally consequential boundary. Preserve the documented nontransactional
behavior when a report error occurs after a transcript has been published.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Run error fields](../../integrations/output-files.md#error-and-recovery-records)
The stream tests open a FIFO peer before final delivery and use a payload
that fits in the pipe buffer. The cancellation tests exercise a caller
canceled before delivery or while the provider is active. A supported stream
blocked in delivery therefore falls outside their assertions. Verify bounded
subprocess cancellation and failed-status finalization after the supported
stream semantics are decided. Keep the existing small delivery tests for the
successful stream contract.
[FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13)
· [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131)
· [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring
the transcript staging file or initializing the reports, and no hits in `Close`
after a temporary path removal error. The existing cleanup tests cover a write
or rename failure followed by successful cleanup. A well-placed preflight test
can observe output and report state when cancellation follows initialization.
At the output-session boundary, verify which cleanup errors are reported and
whether a temporary removal failure can be retried under the selected contract.
[Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61)
· [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394)
· [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
### 4. The timeout tests cover two of the three relevant deadline owners
The adapter test distinguishes a shorter Audita operation budget from an
already canceled parent. Other adapter cases check provider statuses and
transport failures. They do not exercise a profile's HTTP attempt deadline
expiring under a still-live parent and a longer operation budget. The earlier
CLI audit observed that case labeled `operation_timeout`, even though
PromptKit reserves its operation sentinel for the logical budget.
[Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
· [Provider status test](../../../internal/framework/llm/adapter_test.go#L462)
· [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline)
Put the deadline distinction at the adapter boundary using a controlled
transport and deliberately separated deadlines. Assert both recoverability
and the selected cause or reason. The CLI already has report mapping tests
and only needs representative coverage of the adapter-to-report connection.
[Timeout contract](../../integrations/openai-compatible-llm.md)
### 5. The validator batch fixture misses the real numeric decoding boundary
The subprocess mixed-response test and the validator reconciliation tests
protect sibling decisions when ordinary integer references are missing,
duplicated, or unexpected. The adapter has a separate overflow test with a
single out-of-range reference and no usable sibling. The gap is a response
with both a usable requested decision and an unexpected integer too large for
the Go index type. Native schema validation can accept that integer, but the
typed adapter decode can reject the whole batch before per-entry reconciliation.
The existing overflow test provides evidence for failure classification and
cannot establish sibling preservation.
[Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567)
· [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429)
· [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions)
Exercise a mixed response through the actual provider envelope, schema, and
adapter decode. Assert the usable sibling's decision and the appropriate
disposition for the extra entry. A stub returning a pretyped integer cannot
cover this case. Keep smaller reconciliation tests for the ordinary identity
rules.
[Validator reconciliation contract](../../internal/validators.md#llm-validator-batching)
### 6. Concurrency fixtures leave some schedules unproved
The runner's proposal-order test waits for all proposals to start, then
releases three gates back-to-back in reverse section order. That establishes a
release order, while workers may deliver their results in a different order.
The test checks applied-change order by target segment and final text, without
asserting the proposal indices promised by its name. Order by segment can
survive a change to proposal index assignment because corrections are applied
within their sections. The runner has other fixed-chunk and application-order
checks, so the missing evidence is deterministic indexing under a demonstrably
different completion order.
[Parallel runner test](../../../internal/framework/runner/runner_test.go#L423)
· [Result assertions](../../../internal/framework/runner/runner_test.go#L473)
· [Application and recombination](../../../internal/framework/runner/runner.go#L300)
· [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants)
The shared-backend test holds the first HTTP call, then gives the second call
a 30 ms deadline. It does not verify that the second call reached admission
or the wait before it timed out. On a delayed execution, an earlier timeout
can satisfy the same assertions. The runner cancellation test observes native
`ExecutionStarted` for both peers and one dispatch. That event precedes
admission. Twenty race-enabled repetitions of the selected capacity,
cancellation, and proposal-order tests passed. The repetitions exercise the
present fixtures, while the state-observation gaps remain.
[Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67)
· [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96)
Hold the earlier result until later completion is observable and assert the
actual assigned indices and section identities. At the adapter boundary, prove
that work reached the contested state before canceling it. Observe peak active
dispatches relative to a test-controlled limit and check that waiting work can
proceed or time out according to the configured contract. Keep wall-clock
deadlines as bounds for the test itself where possible.
[Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy)
### 7. Representative transcript fixtures leave data-integrity boundaries open
| Behavior | Current protection and gap | Follow-up |
| --- | --- | --- |
| Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) |
| A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) |
| An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) |
### 8. Debug redaction assertions mostly inspect the outer serialization
The subprocess test confirms that it actually captured a provider echo, then
checks command streams and retained files for the clear credential value. The
redactor unit test has a useful nested `final_content` case built with Go's
own JSON encoder. Neither asserts the decoded content of that nested field
when the provider uses another legal JSON representation. A literal-byte
absence check can pass while the credential survives there. This matters to
the documented diagnostic redaction boundary, even when a private debug
directory limits access on disk.
[Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965)
· [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43)
· [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential)
Use a synthetic credential and verify the semantic content after parsing the
retained artifact and any structured response field it contains. Own detailed
string representation cases at the diagnostic writer. Keep one real provider
test to prove that credential discovery and artifact redaction connect.
[Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope)
## Candidates to remove or consolidate
### 9. Delete proposal-order assertions on the test-only composition harness
`TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness`
twice and checks indices that the harness itself assigns with `nextIndex++`.
It cannot detect an indexing regression in `Runner.Run`. The harness also
passes the full transcript into a fake module and delegates range filtering
to that fake. Production builds section-local requests. Related harness cases
verify a composition that the test constructs itself. Remove the index test
once the real runner test checks indices under controlled completion. Migrate
any remaining distinct input-preservation or stale-application assertion to
the existing runner or proposal API coverage before removing its harness case.
[Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14)
· [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143)
· [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356)
· [Stable application test](../../../internal/framework/proposals/apply_test.go#L36)
### 10. Consolidate validator registry and asset metadata assertions
`TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the
class comparison in `TestRegistryBuildsClassifiedValidators`. The second test
also requires the concrete validator to implement `ClassifiedValidator` even
though `ClassOf` supports a stable-key fallback. A behavior-preserving change
to that representation would fail the type assertion. Keep one table that
builds registered keys and checks effective classes against documented roles.
Preserve protective checks of concrete domain behavior and stable selector keys.
[Duplicate registry checks](../../../internal/validators/registry_test.go#L80)
· [Explicit class examples](../../../internal/validators/registry_test.go#L48)
· [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54)
`TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already
enforced at registry construction. The native preparation tests give stronger
evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions`
checks an exact count of 11 prompts. That count detects additions while
allowing a same-size substitution. Prefer a check for each required prompt ID
and its preparability. Retain the tests for canonical schema asset agreement,
stable diagnostics data, and protective copy behavior.
[Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71)
· [Production construction](../../../internal/framework/responseschema/registry.go#L129)
· [Prompt metadata count](../../../internal/prompts/registry_test.go#L79)
· [Native preparation](../../../internal/prompts/registry_test.go#L13)
### 11. Remove the tests of the release test's own set helper
`TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined
in the same test file. It cannot detect a malformed release profile or
bootstrap behavior. A standard comparison over the decoded profile can retain
the documented target set, product and destination checks, and the credential
exclusion without a test of the comparison's implementation. If adding release
coverage, give it to the actual bootstrap and its exit handling. The known
Git-status failure is a representative offline shell boundary, and
ReleaseKit already checks the prepared checkout on its own boundary.
[Test of the helper](../../../scripts/release_test.go#L63)
· [Real profile assertions](../../../scripts/release_test.go#L10)
· [Delivery contract](../../release.md#binary-delivery)
· [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty)
### 12. Consolidate normalization fixtures at the domain and parser owners
`TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers`
exercise the same rejection with different gap values that remain below the
configured maximum. One case can assert the unchanged transcript and the
meaningful merge counts together. Seven normalization golden tests also
repeat the conversion from `SourceTranscript` to `Transcript` in test code.
That conversion mirrors the CLI and cannot catch a regression in the CLI's
copy. The fixtures compare decoded values and contain useful domain assertions,
including exact text and categories. Keep the distinct domain cases, remove
duplicate fixtures after comparing assertions, and let the parser and one
assembled CLI case own source shapes and wiring. No automatic golden update
was found in these tests.
[Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11)
· [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101)
· [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329)
· [Production conversion](../../../internal/cli/run.go#L12)
### 13. Narrow integration assertions to their distinct contracts
`TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling
of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the
same decoded ID with different escaping would fail the test. Decode the
artifacts and assert stage, role, and evidence association. Keep end-to-end
checks for per-entry profiles, successive transcript state, and output
selection, while the reporting package owns Markdown layout and escaping.
[Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145)
· [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101)
`TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure
scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure`
uses the same model script and also checks the distinct attribution in both
YAML documents. The selection matrix took about 6.9 seconds in the
instrumented run. Keep one detailed scenario with the attribution assertions
and use a smaller fixture for any duplicate combination once it still proves
that reporting and debug selections leave decisions and counts intact. The
shared fixture also writes a legacy glossary file and stores its path without
using it. Remove that unused fixture input separately from behavioral checks.
[Selection matrix](../../../cmd/audita/main_integration_test.go#L879)
· [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013)
· [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283)
## Verification scope
The full race-and-coverage run, 20 repetitions of the selected concurrency
tests, and the offline CLI classification probe completed successfully. The
audit used source inspection to identify false passes and refactor-sensitive
assertions. The full suite uses synthetic model responses, consistent with the
offline policy. Live-model editorial quality is a separate, deferred evaluation.
[Execution policy](../../policy/testing.md#test-execution-requirements)
· [Deferred evaluation](../future.md#live-model-editorial-evaluation)

View File

@@ -1,289 +0,0 @@
# Behavioral equivalence and shared abstraction audit
Reviewed the application at `7d064a6`. Audita is a single-process Go CLI for
transcript polishing. The CLI loads configuration and owns invocation and
output coordination. Core packages handle transcript data, normalization,
chunking, and serialization. The framework runner coordinates proposal
generation, validation, repair, and application through explicit contracts.
The PromptKit adapter executes model requests, and reports are derived from
framework results. [Package map](../../internal/overview.md) ·
[Pipeline](../../internal/pipeline.md) ·
[Dependency direction](../../policy/architecture.md#ownership-and-dependency-direction)
The review traced direct comparisons and separate tests with common expected
behavior to their production implementations. Each finding states what the
assertions cover and where a shared implementation could remove duplicated
policy or mechanics. The full existing suite passed with `go test -count=1 ./...`
on Go 1.27.3. [Testing policy](../../policy/testing.md)
## 1. Share validation budget policy across measurement paths
**Evidence.** `TestComposedValidationBudgetMatchesSerializedPayload` runs the
same estimator through the optimized and generic paths on a 12-item fixture.
For every positive budget up to the full payload cost, it compares complete
batches and oversized decisions, including retained context, batch boundaries,
and token counts. Its separate payload assertion checks one-based wire
references and preserved run-level proposal IDs. The fixture includes Unicode,
JSON escapes, neighboring segments, and glossary entries. [Budget comparison](../../../internal/framework/validators/llm_budget_test.go#L17) ·
[Batching contract](../../internal/validators.md#llm-validator-batching)
**Implementation.** `prepareValidationBatches` picks the composed path for the
concrete simple estimator. It premeasures serialized items and context prefixes,
then uses additive costs. The generic estimator path serializes proposed
payloads when fitting and partitioning. Both implementations trim neighbor
entries from the end before trimming glossary entries, reject items whose
required payload still exceeds the budget, and form contiguous batches in input
order. [Dispatch and partitioning](../../../internal/framework/validators/llm_batching.go#L17) ·
[Composed fitting](../../../internal/framework/validators/llm_budget.go#L59) ·
[Serialized fitting](../../../internal/framework/validators/llm_context.go#L74)
**Opportunity and tradeoffs.** Give the context trim order one owner in
`framework/validators`. Share the greedy batch boundary rule if it can consume
measurements without obscuring the different cost models. Keep the composed
path additive and the generic path able to measure the actual serialized bytes
with any supplied estimator. The serialized measurement remains an independent
oracle for the optimized token calculation. Once policy is shared, an explicit
expected sequence of retained items and batch boundaries can catch policy
regressions that path-to-path equality alone would miss.
## 2. Share regular-file staging between transcript and report publication
**Evidence.** `TestSubprocessOutputPermissionsHonorUmask` checks the transcript
and all three reports against the same expected modes. It exercises creation
under two umasks and replacement of existing mode-`0600` files under each.
`TestFinalizationCollectsFailuresAndRetainsPublishedTranscript` separately
checks that successfully publishing the transcript survives a later report
failure. [Shared mode assertions](../../../cmd/audita/main_integration_test.go#L1202) ·
[Partial finalization](../../../internal/core/diagnostics/output_session_test.go#L207)
**Implementation.** `OutputSession.Prepare` opens a payload in a private
temporary directory next to the transcript destination. It creates the payload
with mode `0644` subject to umask, then copies the existing file's permission
bits when replacing a transcript. `writeTranscript` later writes, closes, and
renames it. Reports use `atomicReplace`, which implements the same regular-file
staging within one call and has its own cleanup and error handling.
[Transcript preparation](../../../internal/core/diagnostics/output_session.go#L141) ·
[Transcript publication](../../../internal/core/diagnostics/output_session.go#L347) ·
[Report replacement](../../../internal/core/diagnostics/output_session.go#L410)
**Opportunity and tradeoffs.** Factor temporary-file creation, permission
handling, close, rename, and cleanup into a small private type in diagnostics.
Let the transcript session hold it across processing and let each report
replacement own it for one write. The caller still owns destination selection,
the transcript's final type check, streaming, publication order, and reporting
of partial failures. Keep subprocess assertions on the final files and the
failure-order test at the session boundary. [Output lifecycle](../../operations.md#outputs-and-lifecycle) ·
[Related output ownership finding](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership)
## 3. Resolve model segment references in one place
**Evidence.** `TestModuleProposalMapsLocalReferencesWithoutGuessing` sends IDs
`1` and `2` for segments with transcript IDs `71` and `903`. It verifies that
returned positions map to canonical IDs, while returning `71` as a position
produces an invalid target. `TestRepairRequestRefreshesPreviewAfterRetarget`
uses canonical IDs `81` and `104` and checks that a repair response targeting
position `2` correctly supplies the second segment's text to the next attempt.
The separate retarget-and-approve test checks preservation of the proposal
index and an unrelated approval. [Generation mapping](../../../internal/framework/proposal_generation/module_proposal_test.go#L102) ·
[Repair mapping](../../../internal/framework/runner/repair_test.go#L131) ·
[Repair identity](../../../internal/framework/runner/repair_test.go#L52)
**Implementation.** `GenerateCandidates` and the repair callback both check
that a correction's `id` is a positive in-range position and index the working
section at `id - 1`. Both leave the target ID at zero for an unresolved
position so proposal validation can handle it. [Generation translation](../../../internal/framework/proposal_generation/generate.go#L189) ·
[Repair translation](../../../internal/framework/runner/repair.go#L216)
**Opportunity and tradeoffs.** Put the conversion next to the correction
response type in `proposal_generation`, which repair already imports. A small
lookup can express this protocol without making the core transcript type aware
of model references. Preserve the raw reference for the conversation, the
existing invalid-target handling, and revalidation after retargeting. Repair's
`proposal_ref` and validation batch indices have their own namespaces. Keep
nonsequential canonical IDs in both caller tests so a positional identity bug
fails even after the callers share the lookup. [Module proposal contract](../../internal/modules.md#proposal-generation-ownership)
## 4. Share structured failure facts across model operations
**Evidence.** The generation and validation tests establish that recoverable
structured failures produce an operation with a stable reason while the caller
can continue processing. The validation case checks approvals on either side
of the failed batch, and the repair terminal-outcome table checks that a
recoverable provider failure ends that branch as unavailable. Fatal setup
fixtures check call facts at the generation boundary and previous decisions,
attempted batches, and preservation of the underlying cause at the validation
boundary. [Recoverable generation](../../../internal/framework/proposal_generation/generate_test.go#L201) ·
[Recoverable validation](../../../internal/framework/validators/llm_partial_test.go#L45) ·
[Repair outcomes](../../../internal/framework/runner/repair_test.go#L166) ·
[Fatal generation](../../../internal/framework/proposal_generation/generate_test.go#L218) ·
[Fatal validation](../../../internal/framework/validators/llm_partial_test.go#L137)
**Implementation.** `structuredoutput.Failure` already owns the typed class,
safe reason and message, and wrapped cause. Generation and validation each
unwrap the error, copy its reason or substitute `internal_failure`, and check
recoverability separately. Repair repeats the extraction with a different
edge case. A typed failure with an empty reason retains that empty reason in
repair, and repair checks recoverability by class alone. [Failure type](../../../internal/framework/structuredoutput/failure.go#L17) ·
[Generation projection](../../../internal/framework/proposal_generation/generate.go#L172) ·
[Validation projection](../../../internal/framework/validators/llm_validators.go#L176) ·
[Repair projection](../../../internal/framework/runner/repair.go#L187)
**Opportunity and tradeoffs.** Put extraction of reason and class in a small
`structuredoutput` helper and keep each caller's decision about what to do with
the facts. Generation can return a failed response, validation can mark the
affected batch unavailable, and repair can end a single branch. Preserve the
wrapped cause, execution counts, and operation coordinates. Retain current
blank-reason handling until its desired semantics are decided explicitly. Keep
role tests with literal reason codes or constants and independently constructed
errors, including the tests for surviving work and fatal propagation.
## 5. Give optional interaction artifacts one writer contract
**Evidence.** Separate proposal, validation, and repair tests inject a write
failure. They require a diagnostic warning and preserve the successful proposal
operation, approved validation decision, or withdrawn repair branch. The
proposal and validation schema-metadata tests also check the same metadata
fields against each operation's registered response schema.
[Proposal write failure](../../../internal/framework/proposal_generation/generate_test.go#L391) ·
[Validation write failure](../../../internal/framework/validators/llm_validators_test.go#L398) ·
[Repair write failure](../../../internal/framework/runner/repair_test.go#L324) ·
[Proposal schema metadata](../../../internal/framework/proposal_generation/generate_test.go#L143) ·
[Validation schema metadata](../../../internal/framework/validators/llm_validators_test.go#L335)
**Implementation.** Proposal generation and validators define equivalent
writer interfaces and four-path artifact values. Both convert the same fields
from `llm.DiagnosticsWriter` through adapters. Repair writes through that
concrete writer directly. File naming, serialization, and redaction already
live in the writer, while each caller separately converts writer failure to a
warning. [Proposal contract](../../../internal/framework/proposal_generation/generate.go#L23) ·
[Validator contract](../../../internal/framework/validators/models.go#L84) ·
[Proposal adapter](../../../internal/framework/proposal_generation/generate.go#L246) ·
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
[Shared writer](../../../internal/framework/llm/diagnostics.go#L17) ·
[Repair diagnostic write](../../../internal/framework/runner/repair.go#L178)
**Opportunity and tradeoffs.** Define the writer interface and artifact path
value once at the diagnostics boundary so the concrete writer can satisfy it
without field translation. A common best-effort write helper could also report
the warning and return paths according to one rule. Keep the role-specific
payloads, coordinates, and credential lookups at their current boundaries.
In particular, the validation adapter resolves secrets lazily, and repair can
record the revision and revalidation decisions. Keep role-specific assertions
for warning delivery and processing outcomes alongside the real writer's tests.
[Writer tests](../../../internal/framework/llm/diagnostics_test.go)
## 6. Use one set of checks for inherited configuration fields
**Evidence.** `TestResolvedSettingsValidation` rejects an empty validation
profile at root and in a module entry. `TestCorrectionRepairSettingsResolution`
rejects a negative correction-repair budget at both locations and exercises
inheritance, explicit overrides, and clearing. Those invalid-case assertions
require a load error and do not inspect its diagnostic path. A separate test
checks that explicit null and the corresponding dependency survive in
resolution metadata. [Profile validation](../../../internal/core/config/resolve_test.go#L97) ·
[Repair settings](../../../internal/core/config/resolve_test.go#L163) ·
[Resolution metadata](../../../internal/core/config/resolve_test.go#L119)
**Implementation.** `config.validate` repeats the same LLM checks for root
and per-module `LLM` values with different `ck.Path` prefixes. The duplicated
checks also include retry bounds, operation timeout, and optional repair
profiles. The validation prompt budget and description length checks repeat
at both levels as well. [Root checks](../../../internal/core/config/config.go#L161) ·
[Per-module checks](../../../internal/core/config/config.go#L175)
**Opportunity and tradeoffs.** Extract typed validation functions for the
shared settings and pass the path of each value. This keeps changes to one
field constraint from drifting between the two locations. Leave resolution,
explicit null handling, provenance, and role-specific profile readiness with
their respective owners. Preserve tests that load root and local documents
independently. Path assertions would catch a helper that applies a correct rule
at the wrong location. [Configuration contract](../../config.md)
## 7. Use the same simple token counter for text estimates
**Evidence.** `TestSimpleTokenEstimator` checks concrete counts for words,
punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization
merge-limit test independently expects that joining two single words exceeds a
one-token limit and records the corresponding skipped merge. This supports the
common word-count behavior. The normalization test does not independently fix
the punctuation weighting. [Counter examples](../../../internal/core/chunking/tokens_test.go#L7) ·
[Merge limit](../../../internal/core/normalization/normalize_test.go#L155)
**Implementation.** Normalization counts `strings.Fields` and then scans for
punctuation. Chunking counts word boundaries and punctuation in one pass. Both
count Unicode punctuation except ASCII apostrophes, hyphens, and underscores,
then add half the punctuation count using integer division. The chunking
counter also exposes those components for validation budget composition.
[Normalization counter](../../../internal/core/normalization/tokens.go#L14) ·
[Chunking counter](../../../internal/core/chunking/tokens.go#L25) ·
[Validation cost calculation](../../../internal/framework/validators/llm_budget.go#L27)
**Opportunity and tradeoffs.** Reuse the single-pass counter and its component
count, either directly or through a small core package if neither caller
should own the dependency. Retain the consumer-specific budgets and tests for
merge behavior, chunk behavior, and serialized validation payloads. Keep
literal counts in the counter tests to detect a shared regression that would
otherwise affect all consumers together. This is a shared heuristic, so
altering its estimates has a wider effect than consolidating its implementation.
## 8. Centralize transcript copies without moving the copy boundaries
**Evidence.** `TestRunnerDoesNotMutateInputTranscript` checks the caller's
original text after a successful run. `TestApplyProposalsNoMutationOfInputTranscript`
compares the caller's transcript with a pre-edit snapshot after applying a
proposal. A separate application test asserts preservation of IDs, speakers,
timing, and categories on the edited output. The snapshot test uses the
production clone, and the runner test checks text only. Neither establishes
category slice independence or a convention for nil versus empty categories.
[Runner ownership](../../../internal/framework/runner/runner_test.go#L708) ·
[Application ownership](../../../internal/framework/proposals/apply_test.go#L199) ·
[Application metadata](../../../internal/framework/proposals/apply_test.go#L221)
**Implementation.** The runner copies the input for a run, and `ApplyProposals`
copies the input it received before any edit. Both independently rebuild every
segment and copy its categories. The proposal copy preserves an explicitly
empty category slice, while the runner's append to a nil slice collapses it.
[Runner copy](../../../internal/framework/runner/runner.go#L972) ·
[Application copy](../../../internal/framework/proposals/apply.go#L107)
**Opportunity and tradeoffs.** Let the schema package own a transcript clone
that copies segment values and then copies mutable member data. Keep both
boundaries so callers retain the existing ownership guarantees. Choose and
check the treatment of empty slices explicitly when moving the code. Use
independent fixture values to verify metadata and ownership, since taking the
expected snapshot with the shared implementation can hide a shallow-copy
regression.
## 9. Reconcile the protection rule at validation and application
**Evidence.** The validator tests assert rejection for removing protected
terms and permit spoken-word repetition cleanup only when the identity
survives. `TestApplicationChecksCurrentTextWithSelectedProtection` demonstrates
the same preservation requirement at apply time. It applies one proposal and
skips the second, whose span is a fragment of a protected alias in the full
segment. `TestFeedbackReportsProtectionAndBlankResult`
also checks that a protection finding and a blank-result finding can both be
reported for one candidate. [Protected-term rejection](../../../internal/framework/validators/protected_terms_test.go#L31) ·
[Repetition policy](../../../internal/framework/validators/protected_terms_test.go#L84) ·
[Protection at application](../../../internal/framework/validators/vocabulary_test.go#L47) ·
[Independent feedback](../../../internal/framework/runner/feedback_test.go#L126)
**Implementation.** Matching and identity comparison already live in
`ProtectedVocabulary`. `ValidateProtectedTerms` starts with the proposed spans
and uses full-segment evidence for the glossary path and eligible spoken-word
repetition cleanup. The runner passes `AllowsChange` to application, which uses
the entire current segment and its replacement preview. Both assemble their
own decision about the before/after text and when to allow repetition. The
earlier correctness audit found that a span can hide the context needed to
recognize a protected name. [Validator check](../../../internal/framework/validators/deterministic.go#L100) ·
[Application predicate](../../../internal/framework/validators/protected_terms.go#L47) ·
[Runner wiring](../../../internal/framework/runner/runner.go#L328) ·
[Span-only correctness finding](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity)
**Opportunity and tradeoffs.** Use a single protection assessment that produces
a reason from the applicable before-and-after segment text and the effective
glossary view. Decide explicitly what evidence can support an independent
finding when a proposal has no usable preview. Moving ordinary validation to
segment scope is a behavior change for edits whose span would give a different
answer. Keep application-time reevaluation against current text, and keep
validation and application tests with their own expected outcomes and fixtures.

View File

@@ -1,263 +0,0 @@
# Package cohesion and architectural boundaries audit
Reviewed at `0f34231`. Audita is a single-process Go CLI. `internal/cli`
assembles configuration, inputs, model clients, and output destinations. Core
packages provide deterministic transcript, glossary, and configuration
operations. The framework runner coordinates module proposals, validators,
repair, and application. PromptKit integration lives in `framework/llm`.
Processing facts feed ordinary reports and opt-in debug artifacts.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
The audit used the code graph for discovery and call tracing, the `go list`
import graph to check dependency direction, and source and tests to check what
crosses each boundary. The import inventory covered 42 `internal` packages,
including test support. No core package directly imports a repository package
outside `internal/core`. The framework imports concrete correction modules
through its production factory and built-in validator metadata through its
runner and ledger. The findings focus on the responsibilities behind those
dependencies and on implicit coupling that the import graph cannot show.
Third-party implementations were outside the review. The existing suite
passed with `go test -count=1 ./...` on Go 1.27.3.
[Ownership policy](../../policy/architecture.md#ownership-and-dependency-direction)
## 1. Give the validator runtime interface one owner
`framework/contracts` declares the interface modules expose to the runner, but
aliases its request to `framework/validators.Request` and returns that package's
result. `framework/validators` declares a second interface with the same method
set. Its request ties together candidate evidence, resolved application
settings, the LLM client, the glossary matching cache, and diagnostic writer.
The implementation package also contains the built-in deterministic rule bodies
and a switch selecting built-in validator prompts. The named validator packages
mostly construct or wrap those implementations. As a result, a module that
only needs the validator port depends on the package that implements budgeting,
model reconciliation, and concrete rule policy.
[Module and validator ports](../../../internal/framework/contracts/contracts.go#L21) ·
[Validator request and result](../../../internal/framework/validators/models.go#L30) ·
[Duplicate interface](../../../internal/framework/validators/models.go#L95) ·
[Built-in check implementation](../../../internal/framework/validators/deterministic.go#L15) ·
[Built-in prompt selection](../../../internal/framework/validators/llm_validators.go#L222) ·
[Editorial constructor](../../../internal/validators/editorial_review/validator.go#L11)
Choose one shared owner for the port, decision types, and common cardinality
invariant. Give validators the evidence and collaborators they need through
that boundary. Keep reusable batch construction and reconciliation in the
framework, with concrete prompt selection and editorial rules owned by the
built-in validators. A small first change would establish one interface and
remove redundant aliases before narrowing the request. Preserve deterministic
ordering and the separate full and partial cardinality rules exercised by the
runner and validator tests. [Interface test](../../../internal/framework/contracts/contracts_test.go#L64) ·
[Cardinality checks](../../../internal/framework/validators/models.go#L101) ·
[Validator ownership](../../internal/validators.md#ownership)
## 2. Preserve the execution class assigned to each validator decision
The metadata package provides both a registry keyed by validator name and an
instance-level execution class. The runner uses `ClassOf`, which prefers a
valid explicit class, to order calls and decide which calls receive LLM
interaction diagnostics. The decision record carries the name but omits the
class. The correction ledger calls `ClassForKey` and buckets the record into
deterministic or LLM results. An injected validator with a valid class that
differs from its name's default can therefore execute under one classification
and appear under another. The built-in registry test checks that the two agree
for built-ins. The instance-override and ledger tests exercise their respective
inputs independently. [Class resolution](../../../internal/validators/metadata/metadata.go#L39) ·
[Runtime ordering](../../../internal/framework/runner/runner.go#L913) ·
[Decision recording](../../../internal/framework/runner/runner.go#L775) ·
[Ledger bucketing](../../../internal/framework/processreport/correction_ledger.go#L247) ·
[Built-in agreement](../../../internal/validators/registry_test.go#L97) ·
[Instance override](../../../internal/validators/metadata/metadata_test.go#L35) ·
[Ledger coverage](../../../internal/framework/processreport/correction_ledger_test.go#L12)
Treat execution class as a runtime fact. Resolve it at the runner boundary and
carry it with initial and repair-revalidation decision records. The ledger can
then project the class used for processing. Let the built-in registry attach
default class metadata during construction. The shared validation port can own
the class type and expose instance overrides. Preserve the current ledger
field names and default behavior for unclassified validators.
## 3. Put production module construction at the composition boundary
`framework/runner` already takes a `ModuleFactory` interface. The concrete
`framework/modules` package imports every built-in module and installs the four
constructors in `NewFactory`, which the CLI selects. All four production
constructor adapters ignore their context and run spec. Configuration validates
selectable identities through the dependency-light `core/modulecatalog`.
Keeping the registry populated with all built-in implementations in the
framework gives a composition concern the appearance and import direction of
a reusable execution package. [Runner factory port](../../../internal/framework/runner/runner.go#L30) ·
[Production registration](../../../internal/framework/modules/registry.go#L46) ·
[Constructor adapters](../../../internal/framework/modules/registry.go#L74) ·
[CLI composition](../../../internal/cli/process_pipeline.go#L50) ·
[Module key catalog](../../../internal/core/modulecatalog/catalog.go#L5)
Place the production factory alongside the built-in modules or in the CLI
composition code, and keep the injected factory interface in runner. A small
registration table can still support the existing constructor override seam.
Keep the config-facing catalog free of implementation imports, and use the
factory's tests to check that the catalog keys remain constructible. Module
packages can continue choosing their own proposal prompts and resolving their
explicit validator chains. [Factory and override tests](../../../internal/framework/modules/registry_test.go#L81) ·
[Module contract](../../internal/modules.md#module-contract)
## 4. Carry one checked processing plan from preflight into execution
`preflight.CheckReadiness` reads configured glossary selections and lists the
required proposal and validation profiles, adding a repair profile when the
budget is enabled. The CLI supplies an inspection callback that reaches into
the concrete runtime engine to inspect the profile and its backend. Later,
`executePipeline` traverses the settings again and binds the same roles,
directly dereferencing profile pointers based on the earlier check. In between,
`contracts.ResolveModuleRunSpecs` derives instance names and the effective
glossary protection view, including the glossary module's exemptions. The
readiness and execution steps therefore agree by convention on which roles
and selections will run. Adding a role or changing when it is active requires
coordinated edits in both places.
[Readiness checks](../../../internal/framework/preflight/preflight.go#L96) ·
[CLI inspection](../../../internal/cli/process_inputs.go#L125) ·
[Run-spec construction](../../../internal/framework/contracts/contracts.go#L106) ·
[Role binding](../../../internal/cli/process_pipeline.go#L20)
Have processing preparation derive the effective inputs and active role
selections once. Preflight can check that plan, and composition can bind the
role clients from the checked values. Keep model-facing profile inspection in
the runtime adapter behind the existing narrow shape, and let preflight own
the whole-pipeline readiness decision. The plan can preserve the typed resolved
configuration as the provenance source. This keeps inspection independent of
run readiness and preserves the requirement to reject a later unusable
selection before acquiring outputs or dispatching any model work.
[Later-stage and repair-budget tests](../../../internal/framework/preflight/readiness_test.go#L16) ·
[Glossary and readiness ADR](../../adr/0002-glossary-configuration-and-preflight.md#decision)
## 5. Give glossary matching and protection a domain owner
`core/glossary` derives identity-aware terms and exemptions. The actual
occurrence matcher and comparisons for identity and capitalization live in
`framework/validators.ProtectedVocabulary`. The LLM validation context builder
uses the same type to find mentioned glossary entries. The runner also builds
the combined validator `GlossaryView` and extracts its protection field for
apply-time checks. Matching, change protection, and prompt-context selection
have distinct callers, but the generic matcher and the apply-time policy are
contained in the validator implementation package. [Term derivation](../../../internal/core/glossary/vocabulary.go#L20) ·
[Matcher and change comparison](../../../internal/framework/validators/protected_terms.go#L23) ·
[Context matching](../../../internal/framework/validators/llm_context.go#L11) ·
[Runner preparation](../../../internal/framework/runner/runner.go#L168) ·
[Application call site](../../../internal/framework/runner/runner.go#L221)
Make glossary occurrence matching and text-to-text protection assessment
reusable by placing them next to glossary identity and vocabulary. Keep
validator-specific evidence selection, decision reasons, prompt neighbors,
and budgeting with validation. The application path should receive the
protection predicate it needs and reassess against the current text. Preserve
longest-match exemptions and the explicit spoken-word repetition allowance.
Aligning ordinary validation and application on full-segment evidence would
be a separate behavior decision. [Application and exemption tests](../../../internal/framework/validators/vocabulary_test.go#L47) ·
[Repetition contract](../../internal/validators.md#protected-name-repetition) ·
[Evidence-scope finding](10-equivalence.md#9-reconcile-the-protection-rule-at-validation-and-application)
## 6. Inject interaction diagnostics at the adapter boundary
Proposal generation advertises an injectable interaction writer, yet its
production path passes a directory and constructs `llm.DiagnosticsWriter`
itself. It discovers redaction values by type-asserting an extra method on
its structured client. Runner validation uses a second adapter around that
writer and resolves secrets lazily. Repair constructs the concrete writer and
collects values from both repair and validation clients because its retained
payload includes validation evidence. These paths make proposal generation
and the runner know about artifact destinations and credential discovery even
though structured completion already has an explicit port.
[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L129) ·
[Production proposal wiring](../../../internal/framework/proposal_generation/module_proposal.go#L55) ·
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
[Repair write](../../../internal/framework/runner/repair.go#L173) ·
[Repair credential sources](../../../internal/framework/runner/repair.go#L282)
Give interaction capture one writer contract and let composition supply the
sink when debug is enabled. Put file naming, writes, and redaction in the
diagnostics adapter, with lazy credential sources supplied by the runtime.
Give proposal, validation, and repair callers responsibility for their own
semantic coordinates and evidence. Preserve repair's combined redaction scope
and the nonfatal result of optional write errors. The same boundary could
host debug execution projection, which currently makes `processreport` import
the concrete LLM observation model for a debug artifact.
[Artifact writer](../../../internal/framework/llm/diagnostics.go#L24) ·
[Execution debug projection](../../../internal/framework/processreport/llm.go#L61) ·
[Proposal write-failure test](../../../internal/framework/proposal_generation/generate_test.go#L391) ·
[Repair artifact tests](../../../internal/framework/runner/repair_test.go#L291)
## 7. Separate required publication from retained debug artifacts
`core/diagnostics.OutputSession` owns required transcript publication and
enabled report initialization and delivery. The same type also exposes
`PrepareDebug`, which creates a retained run directory and writes invocation
and effective-config artifacts. That method uses none of the output session
state. Preflight invokes it through the session and returns the two resources
together. This couples APIs with different lifetimes and failure semantics.
The CLI integration test exercises independently selected transcript, report,
and debug destinations. [Output session](../../../internal/core/diagnostics/output_session.go#L45) ·
[Debug method](../../../internal/core/diagnostics/output_session.go#L252) ·
[Preflight handoff](../../../internal/framework/preflight/preflight.go#L76) ·
[Independent destinations](../../../cmd/audita/main_integration_test.go#L844)
First expose debug setup directly through the run-directory API. Where package
separation pays for itself, put the required output session in an output
adapter and keep retained evidence in diagnostics. Preflight or CLI composition
can still coordinate the sequence. Preserve the existing early required-output
checks, independent debug selection and warnings, transcript-before-report
publication, and cleanup of unpublished output. This split addresses which
package owns delivery. It is compatible with sharing the regular-file staging
mechanics inside that owner. [Operational lifecycle](../../operations.md#outputs-and-lifecycle) ·
[Regular-file staging finding](10-equivalence.md#2-share-regular-file-staging-between-transcript-and-report-publication)
## 8. Give run-level report facts one assembly path
`processreport.Build` maps typed processing failures to report records and
computes counts. The CLI independently assembles the initial snapshot, derives
a failure reason and origin from runner state, then mutates status, phase,
error records, and counts during finalization. The filesystem session supplies
a formatted `Kind` string, so the CLI recognizes report initialization by
matching that string. For other required-output failures the report record
currently has an empty phase. Report meaning consequently depends on how the
failure reached the renderer, even though the YAML and Markdown share the
same presentation snapshot. [Report assembly](../../../internal/framework/processreport/report.go#L13) ·
[Processing classification in CLI](../../../internal/cli/process_pipeline.go#L60) ·
[Initial snapshot in CLI](../../../internal/cli/process_inputs.go#L94) ·
[Final status and error mapping](../../../internal/cli/process_output.go#L75) ·
[File failure type](../../../internal/core/diagnostics/output_session.go#L36)
Have the runner and publication adapter supply structured failure facts with
the phase and source coordinates they know, and have `processreport` own the
initial and final mappings to the report model. The CLI can still choose when
to finalize the run and how to present its outcome. Keep the artifact renderer
and file delivery separate so failed-output handling can render an updated
snapshot with the same record and count rules. Preserve the distinction between
unknown pre-run counts and known counts, and keep work-unit and run-level
failures independent. [Delivery failure test](../../../internal/cli/process_output_test.go#L79) ·
[Existing phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) ·
[Report count contract](../../integrations/output-files.md#summary-counts)
## 9. Put transcript conversion and output shape at their owning boundaries
`schema.ParseSourceTranscriptJSON` owns source decoding and preserves optional
IDs. The processing CLI converts that object to the canonical segment model,
using one-based positions for missing IDs. `schema.ParseTranscriptJSONLenient`
has its own copy of the conversion. On output, `outputschema` chooses the
format but implements only the envelope. `schema.TranscriptToJSON` implements
the bare form and projects the same segment fields independently. This splits
the input model transition between CLI and schema, and the public segment
encoding between schema and the output encoder registry.
[CLI conversion](../../../internal/cli/run.go#L12) ·
[Lenient conversion](../../../internal/core/schema/transcript.go#L90) ·
[Bare encoder](../../../internal/core/schema/transcript.go#L224) ·
[Envelope encoder](../../../internal/core/outputschema/registry.go#L62)
Give the source-to-canonical transition a schema-level operation usable after
source acquisition, preserving the separate original source held for debug.
Let `outputschema` own both public envelopes and the common segment projection.
Keep parsing and normalization behavior independent of output-format choices.
Maintain separate fixtures for optional source IDs and the two public output
shapes. The current envelope test covers schema, version, and segment count,
so a compatibility refactor should verify payload fields independently as well.
[Lenient ID test](../../../internal/core/schema/transcript_test.go#L180) ·
[Encoder shape tests](../../../internal/core/outputschema/registry_test.go#L18) ·
[Encoding ownership](../../internal/output-schemas.md#scope-and-ownership)

View File

@@ -1,263 +0,0 @@
# Dead code and obsolete functionality audit
Reviewed at `9110aed`. Audita is a single-process Go CLI. CommandKit dispatches
the command and inspection surfaces. The application loads version 3 settings,
parses and normalizes the transcript, and passes it to a runner that chunks it
and coordinates proposals, validation, repair, and application. The PromptKit
adapter performs model calls. The CLI publishes the selected transcript format
and reports, with separate opt-in debug evidence.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
## Scope and verification
The review used Go's `deadcode` analyzer from `x/tools` v0.52.0 on `./...`,
with and without test entry points, and Staticcheck 2026.2.2 for unused
declarations and ineffectual operations. Production reachability returned the
same 19 candidates for both shipped Linux targets, amd64 and arm64. The native
test-inclusive scan returned 16 candidates, including test-only declarations.
Typed reference counts helped locate write-only fields. The review traced
whole-value serialization, command registration, embedded assets, interface
assertions, and build selection before selecting findings. The FIFO test's
Linux and Darwin build constraint is intentional. `go vet -unreachable ./...`
produced no diagnostics, and `go test -count=1 ./...` passed on Go 1.27.3.
[Release targets](../../../releasekit.json#L7) · [Platform test](../../../internal/core/diagnostics/output_stream_test.go#L1)
## 1. Retire the test-only direct continuation path
`structuredoutput.Request.ContinueFrom` is read in `RoleClient.prepareCall`,
where it has its own opaque exchange check and passes the exchange to an
ordinary `Engine.Run`. The only request literals that set it are in
`conversation_test.go`. The supported correction-repair path passes the
captured proposal exchange to `Coordinator.Coordinate`. Its callback builds
requests without `ContinueFrom`, and the PromptKit coordinator supplies
history and owns the turn budget. A function-level scan sees the shared
request preparer as live, even though its continuation branch has no
production caller. [Request fields](../../../internal/framework/structuredoutput/call.go#L26) ·
[Direct continuation](../../../internal/framework/llm/adapter.go#L127) ·
[Direct callers in tests](../../../internal/framework/llm/conversation_test.go#L73) ·
[Runner repair](../../../internal/framework/runner/repair.go#L143) ·
[Coordinator adapter](../../../internal/framework/llm/coordinate.go#L13)
Move meaningful coverage of sibling isolation, carried context, and structural
repair onto coordinated calls, then remove the request field and direct
continuation branch. Keep exchange capture for proposals and keep the live
coordinator's cancellation, final-turn, and accounting checks.
[Second continuation fixture](../../../internal/framework/llm/conversation_test.go#L113) ·
[Coordinator tests](../../../internal/framework/llm/coordinate_test.go#L15) ·
[Subprocess repair test](../../../cmd/audita/correction_repair_integration_test.go#L173)
## 2. Remove proposal enrichment that the live handoff discards
`GenerateCandidates` appends both a plain proposal and an enriched proposal for
each decoded correction. The only production wrapper, `ExecuteModuleProposal`,
passes `StartIndex: 0` and forwards the plain proposals, segment references,
exchange, and operations. The runner creates its own enriched proposals and
assigns indices in ordered section collection. Only tests consume generation's
`Result.Enriched` or supply a nonzero `StartIndex`. Those tests can still pass
when the runner assigns incorrect run-level indices.
[Duplicate construction](../../../internal/framework/proposal_generation/generate.go#L189) ·
[Production wrapper](../../../internal/framework/proposal_generation/module_proposal.go#L55) ·
[Runner assignment](../../../internal/framework/runner/runner.go#L522) ·
[Generator index assertions](../../../internal/framework/proposal_generation/generate_test.go#L238)
Return one proposal representation from generation and use the runner tests
for global ordering. The adapter integration fixture can prepare validation
input explicitly or exercise the runner handoff. Review the debug metadata
before dropping `StartIndex`, since its `start_index` field is emitted with a
constant zero through the current module path. Keep the actual local segment
references and operation facts.
[Adapter fixture](../../../internal/framework/proposal_generation/adapter_integration_test.go#L76) ·
[Runtime ordering test](../../../internal/framework/runner/runner_test.go#L423) ·
[Debug metadata field](../../../internal/framework/proposal_generation/generate.go#L148)
## 3. Remove duplicate runner bookkeeping for unused module totals
`runModulePipeline` increments a module proposal count and flattens work-unit
operations into a second slice. `Run` copies that count into each `ModuleResult`.
It uses the flattened operations on failures and, on success, flattens the
enriched work-unit operations again after apply-time attribution. No current
consumer reads `ModuleResult.ProposalCount`. Only tests read
`ModuleResult.Operations`. Ordinary reports consume `RunOutput.WorkUnits`,
and the correction ledger uses the module's decisions and applied and skipped
changes. The CLI serializes those projections, so the JSON tags on the unused
module fields never turn them into output fields.
[Count accumulation](../../../internal/framework/runner/runner.go#L536) ·
[First operation copy](../../../internal/framework/runner/runner.go#L620) ·
[Second operation copy](../../../internal/framework/runner/runner.go#L264) ·
[Report work-unit input](../../../internal/framework/processreport/report.go#L51) ·
[CLI artifact projections](../../../internal/cli/process_pipeline.go#L56)
Remove the unused count and the module-level copies once the partial-failure
and accounting tests assert the corresponding work units. Retain the
per-module status and edit evidence consumed by reporting and the ledger.
[Ledger input](../../../internal/framework/processreport/correction_ledger.go#L70) ·
[Partial-progress tests](../../../internal/framework/runner/runner_test.go#L685)
## 4. Drop the unconsumed validation request envelope
`BuildLLMValidationRequest` constructs a `LLMValidationRequest` holding the
validator name, type, module key, module instance, replacement policy, and a
slice of items. The caller takes only `.Items`. Batch encoding serializes
local-reference items, and diagnostics source their labels directly from the
validator and runtime request. The envelope is never passed to JSON encoding.
Each item also stores `SegmentID`, which is assigned after the segment lookup,
never read, and explicitly excluded from JSON. The result's duplicate
`ValidatorName` likewise has no production reader. The runner reads the
validator's `Name()` when recording decisions.
[Envelope construction](../../../internal/framework/validators/llm_validators.go#L242) ·
[Runtime consumer](../../../internal/framework/validators/llm_validators.go#L61) ·
[Fields and wire exclusions](../../../internal/framework/validators/llm_models.go#L20) ·
[Payload construction](../../../internal/framework/validators/llm_prompt_variables.go) ·
[Decision recorder](../../../internal/framework/runner/runner.go#L775)
Have the builder return items and immediate decisions. Remove the write-only
fields after checking fixtures. Keep proposal indices for reconciliation and
the active diagnostic labels. The unused duplicate `framework/validators.Validator`
interface can also go. The module and runner use `contracts.Validator`.
[Unused interface](../../../internal/framework/validators/models.go#L95) ·
[Active interface](../../../internal/framework/contracts/contracts.go#L29) ·
[Payload parity test](../../../internal/framework/validators/llm_budget_test.go#L17)
## 5. Remove structured-call fields superseded by execution observations
The adapter still sets `Response.Provider` to a fixed string and copies prompt,
completion, and total token counts out of the run result. No production consumer
reads those response fields or serializes the whole response. The debug
execution projection gets usage from terminal events and the execution group.
`CallRecord.Prepared` and `CallRecord.FinalContent` are also populated even
though the observation collector copies only the call metadata. Interaction
artifacts get those payloads from `Response`, and a test currently asserts the
duplicated final content on the callback record.
[Response model](../../../internal/framework/structuredoutput/call.go#L52) ·
[Adapter assignments](../../../internal/framework/llm/adapter.go#L153) ·
[Collector projection](../../../internal/framework/llm/observations.go#L126) ·
[Debug execution rendering](../../../internal/framework/processreport/llm.go#L61) ·
[Test-only response and callback assertions](../../../internal/framework/llm/adapter_test.go#L167)
Remove the unconsumed response summaries and callback payload copies. Move
tests to the native usage observation and actual interaction payload paths.
Preserve `Response.Execution` for ordinary report counts and retain prepared
and final response evidence in the opt-in interaction artifacts.
[Observation mapping](../../../internal/framework/llm/observations.go#L139) ·
[Interaction payload](../../../internal/framework/proposal_generation/generate.go#L159)
`Failure.Kind` is a related, explicitly transitional field. The adapter sets
it in tandem with `ReasonInvalidOutputExhausted` and production uses the reason
and class. The remaining reads are from `IsMalformedError` in tests. The debug
call record independently derives its `failure` value from the reason. Move
the tests to the typed reason and remove the compatibility field and helper.
Keep the diagnostic `FailureKind` values, the safe failure message, and the
wrapped cause. [Transitional field and helper](../../../internal/framework/structuredoutput/failure.go#L17) ·
[Paired assignments](../../../internal/framework/llm/adapter.go#L279) ·
[Debug failure mapping](../../../internal/framework/llm/adapter.go#L86) ·
[Cause and classification tests](../../../internal/framework/structuredoutput/failure_test.go#L10)
## 6. Resolve shared prompt variables once per model call
`prepareCall` invokes `WithSharedVariables`, which loads all four embedded
instruction values and fills missing keys. The following loop reloads those
same assets and overwrites every one of those keys. For processing calls,
the first pass over shared instruction values is redundant. The helper is
still useful in CLI request inspection, where caller values take precedence.
[Processing merge](../../../internal/framework/llm/adapter.go#L123) ·
[Shared values and merge semantics](../../../internal/prompts/registry.go#L82) ·
[Inspection caller](../../../internal/cli/app_wiring.go#L100)
Use a single merge in the processing path that preserves the embedded-values
precedence and the missing `ProtectedTermsJSON` default. Keep the inspection
merge semantics and retain the existing prompt rendering coverage.
[Operator prompt test](../../../internal/framework/llm/adapter_test.go#L291)
## 7. Remove orphaned normalization and file helpers
The analyzer and repository references agree that `normalization.shouldMerge`,
`core/io.ValidateWellFormedJSON`, and `core/io.WriteFile` are unreachable in
production and tests. The normalizer calls `shouldMergeWithReason` so it can
count skipped merge reasons. The CLI reads the transcript and calls the schema
parser, which already validates JSON. Current output delivery goes through
`OutputSession` and has a different lifecycle from `os.WriteFile`.
[Unused merge predicate](../../../internal/core/normalization/normalize.go#L115) ·
[Live merge caller](../../../internal/core/normalization/normalize.go#L73) ·
[File helpers](../../../internal/core/io/files.go#L17) ·
[Live input boundary](../../../internal/cli/process_inputs.go#L46) ·
[Live delivery boundary](../../../internal/core/diagnostics/output_session.go#L273)
Delete the three declarations and retain the live merge checks, labeled input
read, and publication paths. Existing normalization and output tests cover
the behavior these helpers formerly duplicated or bypassed.
[Normalization limit test](../../../internal/core/normalization/normalize_test.go#L155) ·
[Publication tests](../../../internal/core/diagnostics/output_session_test.go#L207)
## 8. Remove the transcript parser variants that only tests call
The CLI calls `ParseSourceTranscriptJSON`, keeps the source for debug, then
converts it to the processing model before normalization. Neither
`ParseTranscriptJSON` nor `ParseTranscriptJSONLenient` is reached from the CLI.
The strict wrapper and its sequential-ID helper are used by schema tests. The
strict wrapper dereferences source IDs even though source input allows them to
be omitted. The lenient wrapper duplicates the live conversion and has a
test-only caller. Neither wrapper is an external Go API because the package is
internal. [Live load](../../../internal/cli/process_inputs.go#L46) ·
[Live conversion](../../../internal/cli/run.go#L12) ·
[Strict and lenient variants](../../../internal/core/schema/transcript.go#L65) ·
[Parser and round-trip tests](../../../internal/core/schema/transcript_test.go#L166)
Move the live conversion to the schema package if a shared seam is useful.
Use the supported source decoder and explicit field expectations to test
serialized transcripts, and exercise missing-ID conversion through the live
entry point. That leaves the optional source IDs, normalization's sequential
output, and both output formats intact. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) ·
[Boundary finding](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries)
## 9. Clean up abandoned test support
`internal/testsupport` has no imports from Go code anywhere in the repository,
including tests. All four filesystem and redaction helpers are flagged by both
reachability runs. Staticcheck also reports the old subprocess configuration
view helper, the runner entry wait helper, and two runner structured-client
fixtures with no callers. Current configuration and redaction tests run with
their own fixtures. The unused test doubles differ from the explicit interface
conformance fixtures in `contracts_test.go`, whose methods exist to satisfy
compile-time type assertions.
[Orphan helper package](../../../internal/testsupport/files.go) ·
[Unused config view helper](../../../cmd/audita/main_integration_test.go#L319) ·
[Unused wait helper](../../../internal/framework/runner/runner_test.go#L503) ·
[Unused test clients](../../../internal/framework/runner/runner_test.go#L615) ·
[Compile-time fixture](../../../internal/framework/contracts/contracts_test.go#L64)
Delete the orphan package and unused fixtures. Keep the active assertions in
the configuration inspection, artifact, and runner suites.
## 10. Resolve the inherited progress flag for `process`
The built executable advertises `audita process --progress` as showing
progress messages. The process command registers no progress producer. The
pipeline attaches model observations only for debug capture, and the CLI's
processing summary and path messages are emitted at completion. In a local
normalization-only invocation, `--progress` and the default invocation returned
identical transcript bytes and empty stderr. The source trace is the evidence
for the model path too, since its observation hook is connected to the debug
artifact and no process progress callback is supplied.
[Command registration](../../../internal/cli/app_wiring.go#L84) ·
[Processing observation wiring](../../../internal/cli/process_pipeline.go#L20) ·
[Completion reporting](../../../internal/cli/process_output.go#L61)
Treat this as an exposed option with no Audita process behavior. Either attach
it to a real process event source or clarify its availability in process help
while retaining acceptance for existing scripts. The delegated inspection and
smoke commands have their own reporting behavior.
## Retained behavior
The version gate rejects older explicit configuration versions. The flags
marked for removal in the migration guide are absent from command help and
registration. `chunking.min_section_tokens` is documented, accepted and
validated, and the chunker explicitly
retains it for compatibility even though partitioning does not use it as a
target. Keep that setting. Also keep the generic validator estimator path,
which supplies the independent serialized reference in the existing budget
comparison, and the debug-only metadata reached through serialization.
[Migration guide](../../migrations/v3.md) ·
[Version gate](../../../internal/core/config/config.go#L131) ·
[Chunk compatibility field](../../../internal/core/chunking/sections.go#L27) ·
[Budget reference test](../../../internal/framework/validators/llm_budget_test.go#L14)

View File

@@ -1,280 +0,0 @@
# Credential-redaction removal and simplification audit
**Reviewed:** 2026-11-15 at `a68a951`; application code remains at `c4ccd78`.
**Status:** audit complete; implementation pending.
This review follows the maintainer's [resolved D1 decision](assessment.md#resolved-d1-withdraw-application-wide-credential-redaction).
It identifies machinery to remove or simplify after withdrawal of the
application-wide credential-redaction requirement. Native library behavior,
authentication, diagnostic evidence, and ordinary output correctness remain
independently useful. The architecture policy has already been updated.
## Scope and verification
The review traced credential discovery into CLI, reports, debug writers,
proposal generation, validation, and repair. Knowledge-graph searches and caller
traces were checked against source, including interface assertions and closures
that the graph does not resolve completely. Tests were read to distinguish
redaction-only expectations from authentication, serialization, artifact
association, and failure behavior. Documentation and previous audit remedies
were reviewed for assumptions that would reintroduce the withdrawn requirement.
This was a static review: no code changes, live provider calls, credential-value
inspection, test execution, or performance measurements. Removed passes and
allocations below are visible in the code; their production cost is unmeasured.
## 1. Remove credential discovery and tracking used only for substitution
**Remove.** `Runtime.ConfiguredSecrets` enumerates every captured profile and
backend, caches environment-variable names with `sync.Once`, and reads their
values whenever called. It includes raw and trimmed forms and deduplicates
additional names. Discovery uses `context.WithoutCancel`. `RoleClient` separately
collects prepared credential names under a mutex, solely to supply this method.
None of this selects or supplies authentication to a provider.
[Discovery and reads](../../../internal/framework/llm/secrets.go#L11) ·
[Runtime cache](../../../internal/framework/llm/runtime.go#L22) ·
[Prepared-name tracking](../../../internal/framework/llm/adapter.go#L153) ·
[Role accessor](../../../internal/framework/llm/adapter.go#L314)
Consumers are CLI report/debug preparation and finalization, the error redactor,
proposal and validation writer setup, and combined repair/validation secret
collection. Remove `secrets.go`, the runtime cache fields, role credential fields
and locking, accessor methods, and secret-source assertions after updating those
consumers. Keep `OnPrepared` request capture, terminal-event synchronization,
call IDs, source-path tracking, and native engine execution; those have separate
consumers. Do not leave empty secret-provider methods or no-op redactors.
[CLI consumers](../../../internal/cli/process_inputs.go#L90) ·
[Finalization](../../../internal/cli/process_output.go#L17) ·
[Repair collection](../../../internal/framework/runner/repair.go#L282)
Delete `TestConfiguredSecretsIncludesUnselectedBackends`. In
`TestRoleClientUsesNativeTargetAndRedactsCredential`, retain the assertion that
PromptKit sends the selected, trimmed Authorization value; remove the independent
redactor assertion. Keep native credential selection/optional-absence and
provider-free inspection tests. Update the credential-tracking description in
the [runtime guide](../../internal/llm-runtime.md) when removal lands.
**Catalog:** #63; prerequisite consumers are covered by §§2, 4–6.
## 2. Replace sanitizing serialization with direct encoding
**Remove / simplify.** The entire `core/redaction` package serves substitution:
raw/escaped secret expansion, longest-first matching, generic JSON decoding and
tree reconstruction, key-collision rejection, and YAML node/comment traversal.
Its JSON path marshals, decodes with `UseNumber`, walks, then marshals again;
its YAML path parses and re-encodes already serialized data. These passes run
even when the secret list is empty. Matching keys and fixed strings also causes
the schema corruption already recorded in catalog #9.
[Redactor](../../../internal/core/redaction/redaction.go#L18) ·
[Report YAML](../../../internal/core/reporting/report.go#L181)
Use direct JSON serialization in `RunDirectory` and `DiagnosticsWriter`, and
direct typed YAML serialization for ordinary reports. Write already-produced
YAML bytes without another parse/walk/encode pass. Remove secret parameters and
stored redactors. In Markdown, remove substitution and the redactor parameter
to `writeMatrix`, retaining the independent text/HTML/table escaping.
[Run artifacts](../../../internal/core/diagnostics/run_dir.go#L70) ·
[Interaction artifacts](../../../internal/framework/llm/diagnostics.go#L84) ·
[Markdown](../../../internal/core/reporting/markdown.go#L27)
Retain serialization errors and contextual write errors, valid artifact
envelopes, numeric and boolean types, optional fields, and raw `final_content`
as a string even when its contents are malformed model JSON. No nested-content
sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective
configuration rendering; only Audita's subsequent YAML transformation goes away.
Move useful scalar/schema and failed-write assertions from redactor tests to
the owning report/artifact tests before deleting the package. Update
[diagnostics/reporting internals](../../internal/diagnostics-reporting.md).
**Catalog:** #3 and #9; preserve #26's independent Markdown correction and
coordinate with #33's redundant-serialization work without resolving D5.
## 3. Save the acquired source snapshot without reconstructing it
**Simplify.** `WriteSourceTranscript` converts the acquired bytes to a string,
decodes a generic JSON tree, then sends it through the redactor's second
serialization/decode/walk/serialization sequence. The comment explicitly ties
this reconstruction to avoiding unredacted bytes. The CLI already has both the
acquired source bytes and parsed source, and calls this writer only for debug.
[Snapshot writer](../../../internal/core/diagnostics/run_dir.go#L96) ·
[Caller](../../../internal/cli/process.go#L54)
Write the acquired bytes directly to the source artifact, preserving the input
that was actually read. Keep parsed-source and normalized artifacts as separately
serialized typed views. Do not reopen the input file, which could have changed,
or preserve a second parser as a validation gate. Broader input-buffer lifetime
work remains catalog #31's concern.
Move source/normalized snapshot coverage out of the deleted redactor package.
Check source-byte fidelity at this boundary and decoded fields for the typed
views; preserve nonfatal debug-write behavior. Describe the source/parsed
distinction in the diagnostics guide when implemented.
**Catalog:** #31; depends on the direct writer changes in §2.
## 4. Remove secret plumbing from writer and preflight contracts
**Simplify.** Proposal generation discovers secrets through an optional client
interface and constructs an adapter-specific writer. Validation uses a lazy
`func() []string` closure and recreates a writer per interaction. Repair combines
two roles' secret lists for its writer. These are extra dependencies on model
clients for a filesystem concern.
[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L128) ·
[Validation setup](../../../internal/framework/runner/runner.go#L703) ·
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
[Repair writer](../../../internal/framework/runner/repair.go#L178)
Remove the discovery assertions, closure, combined-role helper, and secret
constructor arguments. Complete the already-proposed common interaction writer
contract without preserving these seams. Role owners still supply their payloads
and stage/section/proposal/attempt/validator coordinates; the writer returns
artifact paths and write errors. Retain the optional diagnostic-error callback
and its synchronization, which report nonfatal write failures.
Also remove `DebugInput.Secrets`, the corresponding `PrepareDebug` and
`NewRunDirectory` parameters, and secret captures in report-rendering closures.
Preflight readiness, initial/failed report construction, output-session cleanup,
and debug setup remain useful without any credential values.
[Preflight handoff](../../../internal/framework/preflight/preflight.go#L26) ·
[Debug preparation](../../../internal/core/diagnostics/output_session.go#L253) ·
[Report closures](../../../internal/cli/process_inputs.go#L95)
Adapt proposal, validation, repair, and preflight fixtures to the smaller
contracts. Preserve role attribution, distinct repair artifact paths, and
nonfatal write-failure coverage. Update the diagnostics guide with the resulting
ownership. **Catalog:** #43, coordinated with #42 and #48; no retention or
execution-policy change is implied.
## 5. Stop rewriting validator explanations sent to repair
**Remove / simplify.** Inside `repairCandidates`, `safeFindings` first constructs
domain feedback, then replaces substrings matching credentials from both repair
and validation clients. This runs for the initial rejection and subsequent
revalidation findings, independently of debug capture. It changes the next model
request, not just a saved diagnostic. A matching substring in otherwise useful
validator text therefore changes the corrective instruction.
[Feedback transformation](../../../internal/framework/runner/repair.go#L111) ·
[Repair request](../../../internal/framework/runner/repair.go#L124)
Use `repairFindings` directly at both points and remove the wrapper and its
secret dependency. Retain per-proposal filtering, validator/reason labels, and
the explicit unavailable-assessment representation: operational unavailability
is not an actionable editorial explanation. Preserve bounded coordination,
revalidation, withdrawal, and sibling independence.
[Domain feedback](../../../internal/framework/runner/feedback.go#L53)
Replace `TestRepairFeedbackRedactsValidationCredentials` with focused coverage
of actual repair feedback if existing feedback/request tests do not already
cover it. Assert explanation and proposal association through decoded request
data, without introducing an environment-driven passthrough contract or prompt
wording checks. Update the repair-flow description if needed.
**Catalog:** #64; coordinate removal of `repairSecrets` with §4.
## 6. Simplify CLI presentation while retaining typed failure ownership
**Remove / simplify.** `runProcess` invokes `Runtime.RedactError` for preparation
failures. `finishProcess` unconditionally discovers credentials and builds a
redactor, even without reports or debug; it rewrites warnings, final errors, and
announced report/debug paths. Removing these calls eliminates environment reads
and permits paths to identify their actual destinations.
[Preparation failure](../../../internal/cli/process.go#L37) ·
[Final presentation](../../../internal/cli/process_output.go#L17)
Present those existing errors and paths directly through CommandKit, preserving
typed causes, reporting modes, completion behavior, and exit classification.
Drop redaction-only parameters such as `finishProcess`'s `loaded` argument once
its only use disappears. Retain the required-output/nonfatal-debug distinction.
Simplify the path-announcement subprocess fixture; retain useful destination,
output-mode, and error assertions at their existing test owners.
There is also a distinct information-loss boundary: `classifiedFailure` reduces
native setup errors to a generic message, and `Failure.Error()` returns only
that message. Deleting substring replacement does not itself make missing
credential diagnostics actionable. Address catalog #25 by carrying the native
setup detail into the surfaced error, while retaining `Class`, `Reason`, cause
unwrapping, cancellation precedence, and operation attribution.
[Classification](../../../internal/framework/llm/adapter.go#L228) ·
[Failure wrapper](../../../internal/framework/structuredoutput/failure.go#L18)
Do not indiscriminately unwrap every cause into every report. The compact
reason/message projection in `processreport` serves the ordinary report schema,
and structured failure types drive recoverability and peer cancellation. Retain
those purposes; revise blanket comments/tests prohibiting cause presentation
where they enforce only the withdrawn guarantee. Test setup diagnostics for
actionable context and typed classification, not exact incidental wording.
[Report projection](../../../internal/framework/processreport/report.go#L224) ·
[Failure tests](../../../internal/framework/structuredoutput/failure_test.go#L10)
**Catalog:** #65 and #25; coordinate with #22–24 and #41 without changing their
classification or report-schema scope. Update operations and internal runtime
documentation alongside the implementation.
## 7. Retire obsolete assertions and documentation without losing coverage
**Remove / simplify.** Redaction assertions are mixed into tests with independent
value, and current documentation still describes the implemented sanitizer.
Remove pure substitution tests and synthetic credential-echo fixture modes used
only for them. In mixed tests, preserve native authentication, decoded request
and response evidence, schema metadata, repair path uniqueness, scalar types,
failure counts, and optional-artifact failure behavior.
| Location | Treatment when implementation lands |
| --- | --- |
| `core/redaction/redaction_test.go`, `llm/secrets_test.go` | Delete removed-mechanism tests; relocate useful writer assertions as described above. |
| `llm/diagnostics_test.go`, `proposal_generation/generate_test.go`, `runner/runner_test.go` | Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations. |
| `runner/repair_test.go` | Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary. |
| `core/reporting/report_test.go`, `llm/native_adapter_test.go` | Retain typed report encoding and actual native authentication; remove attached substitution assertions. |
| `cmd/audita/main_integration_test.go` | Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements. |
| `internal/testsupport/files.go` | Remove the already-unused secret scanners with catalog #61, rather than repurposing them. |
Review [operations](../../operations.md),
[diagnostics/reporting](../../internal/diagnostics-reporting.md),
[runtime](../../internal/llm-runtime.md),
[contributor recipes](../../internal/cli-config.md),
[component overview](../../internal/overview.md), and the
[migration guide](../../migrations/v2.md) when code lands. Remove broad redaction
claims and obsolete implementation instructions. Preserve native configuration
inspection/redaction documentation and the effective-config artifact name;
neither requires a naming migration merely because local postprocessing goes
away. Historical audit evidence stays intact, with revised remedies in the
catalog and assessment. Do not add a negative redaction policy.
**Catalog:** #3, #60, and #61, and supporting work for all findings above.
No new broad secret scanner or prompt-language change detector should replace
the removed tests.
## Retained responsibilities and boundaries
- **Native authentication and inspection:** PromptKit still owns selected
credentials and transport; ConfigKit/ckconfig/ckprompt own their normal
rendering and inspection. Keep Audita's `WriteEffective` call and standard
command wiring. Prepared snapshots contain messages and prompt metadata;
removing substitution does not require adding headers or environment dumps.
[Configuration rendering](../../../internal/cli/process_inputs.go#L118) ·
[Command wiring](../../../internal/cli/app_wiring.go#L81) ·
[Prepared snapshot](../../../internal/framework/llm/adapter.go#L309)
- **Ordinary filesystem behavior:** retain unique run-directory creation,
run-ID validation, stage-name-to-filename conversion, required-output staging,
destination checks, cleanup, and useful write errors. These prevent collisions
or incorrect writes. No extra private-root or hostile-filesystem mechanism
was found to remove in the debug writer. Catalog #4's previously proposed
private defaults lack a requirement under D1; leave current location/modes
unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain.
- **Domain and report behavior:** retain transcript parsing, validation,
feedback classification, typed operation failures, stable report identities,
Markdown escaping, and nonfatal optional diagnostics. Removing credential
substitution does not imply dropping these boundaries.
## Suggested implementation order
1. Replace serialization and source-snapshot reconstruction, removing writer and
report secret arguments together with their callers (§§2–4).
2. Remove repair-feedback and CLI substitution, then delete now-unreferenced
discovery state and the redaction package (§§1, 5–6). These are a coordinated
cleanup, not an interim design with empty secret lists or no-op methods.
3. Complete common interaction-writer ownership and actionable setup diagnostics
with catalog #43/#25. Update affected tests and canonical documentation in
each implementing change (§7).
Use targeted boundary tests and then the full suite for the cross-package code
change. Check decoded artifact fidelity, raw source preservation, native auth,
repair feedback, report types, and failure behavior. No new product decision is
needed for these D1 removals; D2–D7 remain separate and unresolved.

View File

@@ -1,338 +0,0 @@
# Audit assessment and decision register
**Reviewed baseline:** `faba8ab` (2026-11-15). **Status:** assessment and policy
decisions complete. D1–D7 were resolved on 2026-11-15 and are recorded below.
The D1 architecture-policy update and focused cleanup audit are complete;
application remediation and the associated operator-documentation updates
remain pending.
This document owns classification, recommended
scope, and pending decisions for the [catalog](catalog.md). The source audits
retain their evidence; their suggested remedies are not approved requirements.
The catalog covers 110 source findings in 65 entries. Every catalog entry is
assessed below. The application implementation remains at the audited baseline
`c4ccd78`; later documentation includes the D1 policy revision and the focused
[removal audit](14-redaction.md), reviewed at `a68a951`. This review used
the source audits, current contracts and ADRs, and targeted implementation
inspection. It did not independently rerun all audit probes, benchmarks, or the
test suite. Reproductions reported by the audits remain attributed to them;
allocation/call-path observations establish opportunities, not measured
production performance.
## Evaluation rules
The maintainer approved the taxonomy and evaluation process, not every proposed
remedy. Correctness restores established or reasonably expected behavior;
observable change alone does not make a fix a new policy. Refactor size is not
an approval criterion. Efficiency and ownership improvements should proceed even
when substantial restructuring is appropriate.
| Code | Category | Treatment |
| --- | --- | --- |
| C | Correctness | Fix substantiated departures from intended behavior. |
| E | Efficiency and resource use | Improve work, allocation, and retention while preserving domain and operator policy. |
| A | Architecture and ownership | Consolidate responsibilities and improve boundaries; retain distinct domain behavior. |
| T | Tests and verification | Protect consequential behavior at real boundaries; remove redundant and implementation-bound assertions. |
| M | Maintenance and documentation | Remove obsolete machinery and clarify implemented behavior. |
| P | Substantive policy or product change | Requires express maintainer approval before implementation. |
**Ready** means suitable for implementation planning within the stated scope,
not implemented or validated by this document. **Investigate** denotes technical
work the agent can undertake without a product decision. D1–D7 are resolved;
there are no remaining open decisions in this register. **Closed**
denotes no remediation for the specified scope. Mixed entries deliberately
separate ready work from decisions and exclusions.
Evidence is judged against the [architecture policy](../../policy/architecture.md),
[operations contract](../../operations.md),
[CLI contract](../../cli.md),
[output contract](../../integrations/output-files.md),
[validator contract](../../internal/validators.md), and relevant catalog sources.
An audit author's expectation alone does not establish a requirement.
Internal-only exposure reduces urgency, not the value of a valid invariant fix.
Tests and acceptance criteria must follow the [testing policy](../../policy/testing.md).
The maintainer's three WONTFIX decisions remain controlling: #2 in full,
#6 for interruption of already-blocked I/O, and #18 for Unicode behavior of Go
regexp word boundaries. Using the appropriate standard primitive for Audita's
own requirement is legitimate; replacing or compensating for accepted Go
semantics is not selected. The
[agent policy](../../policy/agents.md#implementation-discipline) also excludes
invented hostile-filesystem and power-loss guarantees. Source recommendations
that conflict with these dispositions must not re-enter an implementation plan.
## Entry assessments
Each entry links to the catalog's original evidence. Category codes can combine;
the final column bounds the recommended remediation and distinguishes confirmed
behavior from proposed extensions.
| Entry | Categories | Disposition | Assessment and scope |
| --- | --- | --- | --- |
| [1. Bind output writes to the checked destinations](catalog.md#1-bind-output-writes-to-the-checked-destinations) | C, A | Ready | The wrong-file reproduction and repeated report-path resolution substantiate application defects. Resolve paths with filesystem semantics, carry checked parent/destination information through publication, and share staging ownership. Preserve symlinks and intentional in-place replacement. No hostile-directory race-proof guarantee, cross-file transaction, or custom filesystem machinery is selected. |
| [2. Keep model requests within the configured transport boundary](catalog.md#2-keep-model-requests-within-the-configured-transport-boundary) | — | Closed: WONTFIX | Accepted Go HTTP redirect and credential-forwarding behavior. The source audit's proposed redirect restrictions do not authorize implementation. |
| [3. Protect credentials inside nested debug responses](catalog.md#3-protect-credentials-inside-nested-debug-responses) | A, M, T | Ready under resolved D1; original remedy superseded | The encoded-secret observation tested the withdrawn guarantee. The focused audit confirms removal of the local substitution package in favor of direct serialization, plus retirement of masking-only tests/docs. Keep malformed response evidence and native behavior; no nested sanitizer or response-omission policy. Discovery and repair/CLI consumers are separately tracked in #63–65. |
| [4. Make debug artifact permissions private by default](catalog.md#4-make-debug-artifact-permissions-private-by-default) | M | No remediation under D1 scope | The permission observation is factual, but hostile local filesystem access is outside the selected threat model. The focused audit found ordinary directory creation and filename handling to retain, not a redundant private-root mechanism. Leave defaults unchanged in D1 cleanup; any future location or permission proposal needs an independent justification. This does not close #1’s wrong-file correctness work. |
| [5. Define the trust boundary for discovered configuration](catalog.md#5-define-the-trust-boundary-for-discovered-configuration) | M; P | Ready for documentation; extension excluded | Automatic companions are intentional configuration composition, not a discovery bug. Explain that the searched locations, selected bundle, imports, and companions are trusted configuration, and keep uploaded transcripts separate. Disabling discovery or adding a new isolation mode is outside this remediation; no ConfigKit replacement loader. |
| [6. Make blocking transcript input and delivery cancellation-aware](catalog.md#6-make-blocking-transcript-input-and-delivery-cancellation-aware) | C; — | Ready in part; WONTFIX in part | Honor cancellation before regular-file publication and when blocked delivery returns; produce failure status even if bytes have escaped. Do not promise interruption of already-blocked standard I/O. Do not ban FIFOs, add detached writer goroutines, or revive the source audits' context-aware stream machinery. |
| [7. Publish truthful failure reports and preserve initialized evidence](catalog.md#7-publish-truthful-failure-reports-and-preserve-initialized-evidence) | C, A, T | Ready | Publishing completed reports after a known delivery failure contradicts their meaning. Select failed status before the first report write; preserve initialized evidence if rendering fails. Carry rendering errors explicitly and retain bounded best-effort finalization. Cross-file atomicity remains outside the contract. |
| [8. Prevent concurrent runs from clobbering ordinary reports](catalog.md#8-prevent-concurrent-runs-from-clobbering-ordinary-reports) | M | Ready for documentation under resolved D2 | The maintainer selected caller-owned destination isolation. Retain existing placement and document distinct report directories for concurrent jobs or retained run histories. Do not add automatic placement, locking, waiting, or competing-writer rejection. Ordinary checked-destination correctness remains #1’s independent scope. |
| [9. Preserve the ordinary YAML schema during redaction](catalog.md#9-preserve-the-ordinary-yaml-schema-during-redaction) | C, A, M | Ready through D1 cleanup | Schema corruption is real, but preserving the redactor is no longer the remedy. Remove application credential substitution from ordinary report serialization and preserve the typed schema. Retain meaningful output-schema tests; retire tests whose sole purpose is satisfying the withdrawn redaction promise. |
| [10. Reject contradictory validation decisions](catalog.md#10-reject-contradictory-validation-decisions) | C, A | Ready | Approved-plus-unavailable contradicts the existing acceptance invariant, although bundled producers are consistent. Establish an authoritative internal disposition and reject invalid internal results before use; derive redundant serialized approval as needed. This is an internal contract failure, distinct from an unavailable model assessment. |
| [11. Preserve usable validator decisions when extra references cannot be decoded](catalog.md#11-preserve-usable-validator-decisions-when-extra-references-cannot-be-decoded) | C, T | Ready | The mixed-response probe violates existing sibling-preservation rules. Retain exact numeric references until range reconciliation, accepting integral JSON number spellings without float rounding. Ignore extras before conversion to local indices. Preserve whole-response structural validation; do not add semantic retries or general malformed-response salvage. |
| [12. Use full-segment evidence for protected-name checks](catalog.md#12-use-full-segment-evidence-for-protected-name-checks) | C, A, T | Ready | The full-segment examples establish a mismatch with existing protection intent. Share glossary-owned assessment using usable before/after segment evidence; recheck current text at application. Preserve active-glossary exemptions and spoken-word repetition rules. Missing or ambiguous previews cannot justify a fabricated protection verdict; retain independently supported applicability findings. Empty-result previews can still provide evidence. |
| [13. Preserve transcript input fidelity during JSON shape detection](catalog.md#13-preserve-transcript-input-fidelity-during-json-shape-detection) | C, E, T | Ready | Generic numeric decoding demonstrably changes accepted IDs and processes ignored fields unnecessarily. Keep the selected array as raw JSON and decode typed fields directly. Preserve both input shapes, optional IDs, accepted ranges, and unknown-field handling; no new input restrictions. |
| [14. Honor cancellation at the runner commit boundary](catalog.md#14-honor-cancellation-at-the-runner-commit-boundary) | C, A, T | Ready | Canceled entry and precommit paths violate cooperative cancellation and stage ownership. Check natural boundaries and assemble results once, preserving earlier committed stages. Do not claim atomic synchronization between cancellation and filesystem or memory operations, or introduce forcible interruption of standard calls. |
| [15. Retain completed proposal evidence behind slow sections](catalog.md#15-retain-completed-proposal-evidence-behind-slow-sections) | C, E, A | Ready | Completed proposal evidence must survive independently of ordered dispatch. Retain it on interruption and assign deterministic identities after joining workers. Reducing within-stage head-of-line waiting is a valid efficiency change if identities, validation requirements, application order, and the module barrier survive; it does not authorize independent whole-pipeline traversal. |
| [16. Keep ownership of failed staging cleanup](catalog.md#16-keep-ownership-of-failed-staging-cleanup) | C, A, T | Ready | The cleanup probe demonstrates lost resource ownership. Keep temporary paths until removal succeeds or absence is established, and surface cleanup errors without losing the primary failure. Share cleanup mechanics with publication. No persistent scavenger or indefinite retry service is required. |
| [17. Preserve the complete time interval when merging segments](catalog.md#17-preserve-the-complete-time-interval-when-merging-segments) | C, T | Ready | Nested intervals demonstrably truncate speech coverage. Use the full interval for merge eligibility, output, and subsequent gaps. Preserve accepted overlap and merge rules; rejecting overlapping input would be a separate restriction. |
| [18. Match protected names at Unicode and punctuation boundaries](catalog.md#18-match-protected-names-at-unicode-and-punctuation-boundaries) | C; — | Ready in part; WONTFIX in part | Keep the accepted ASCII semantics of Go regexp word boundaries. Recognizing configured punctuation-edged spellings is an application matching defect: express appropriate literal-edge checks with standard primitives, preserving longest-match selection, exemptions, and case behavior. Do not turn this into Unicode boundary emulation or a replacement regexp engine. |
| [19. Detect overlapping occurrences when requiring a unique edit span](catalog.md#19-detect-overlapping-occurrences-when-requiring-a-unique-edit-span) | C, T | Ready | Require-unique means one possible literal location; the overlapping-span probe has two. Use standard search primitives to detect a second starting offset. This corrects Audita's choice of counting operation, not Go's nonoverlapping Count semantics. Preserve replace-all's existing nonoverlapping behavior. |
| [20. Satisfy feasible explicit chunk targets](catalog.md#20-satisfy-feasible-explicit-chunk-targets) | C, E, T | Ready | The feasible partition counterexample establishes a chunker defect. Guarantee count and maximum constraints before optimizing balance. Choose a proportionate deterministic algorithm and test feasibility independently; no approval is needed for the algorithm or a different balanced cut. Do not change configured budgets or add a new balancing objective. |
| [21. Keep input order for exactly tied timestamps](catalog.md#21-keep-input-order-for-exactly-tied-timestamps) | C, T | Ready | Arbitrary reorder of tied utterances has no domain justification. Preserve source order for equal sort keys with a stable sort. No new end-time ordering criterion is implied; retain the current primary chronological key. |
| [22. Preserve failure phase and origin in run reports](catalog.md#22-preserve-failure-phase-and-origin-in-run-reports) | C, A, T | Ready | Required phase and available operation origin are lost between owners. Carry structured facts and assemble run records once, keeping primary failure separate from canceled peers and secondary delivery failures. Preserve documented counts and serialized identities. |
| [23. Classify provider attempt timeouts separately from operation deadlines](catalog.md#23-classify-provider-attempt-timeouts-separately-from-operation-deadlines) | C, T | Ready | The attempt-timeout probe and classifier order establish misclassification. Prefer native PromptKit classifications and parent cancellation precedence. Use the existing transport-failure reason for an HTTP-only timeout; a new public reason code is unnecessary for this fix. |
| [24. Use input diagnostics for operator-correctable failures](catalog.md#24-use-input-diagnostics-for-operator-correctable-failures) | C, T | Ready; technical ownership check | Processing readiness and chunk infeasibility should follow the existing input-error contract while I/O and cancellation remain runtime failures. Delegated inspection has a demonstrated classification mismatch, but locate responsibility in Audita's loader versus ckprompt/CommandKit before editing. Fix or request a fix at that owner; do not create a local replacement command surface. |
| [25. Identify missing required credentials in safe diagnostics](catalog.md#25-identify-missing-required-credentials-in-safe-diagnostics) | C, M | Ready | The missing-credential cause is available but obscured by generic setup presentation. Surface actionable native setup detail with available role/profile context, preserving typed causes and classification. D1 removes the blanket restriction on cause presentation; compact ordinary report projections remain useful. Keep PromptKit responsible for credential validation; do not introduce local secret inspection or a replacement preparation subsystem. |
| [26. Render input and output paths as plain Markdown text](catalog.md#26-render-input-and-output-paths-as-plain-markdown-text) | C, T | Ready | Input paths are data; rendered Markdown currently gives some filenames unintended syntax. Escape them as literal text without altering structured path values. This restores report presentation rather than imposing filename restrictions. |
| [27. Keep the validator execution class used by the runner](catalog.md#27-keep-the-validator-execution-class-used-by-the-runner) | C, A | Ready | Runtime and ledger can disagree for supported injected validator classes. Record the effective execution class with decisions and project it into the ledger. Preserve existing output fields and fallback behavior; no new validator registration policy. |
| [28. Verify signal and exit handling in the production executable](catalog.md#28-verify-signal-and-exit-handling-in-the-production-executable) | T | Ready | The startup helper duplicates the production boundary, so coverage does not prove actual executable signal wiring. Exercise representative startup and cancellation through the real executable or shared production entry point; keep narrower tests for other CLI behavior. |
| [29. Fail the release bootstrap when cleanliness cannot be established](catalog.md#29-fail-the-release-bootstrap-when-cleanliness-cannot-be-established) | C, T | Ready | The shell probe establishes that a failed status command can appear clean. Check its exit result explicitly and test that boundary. ReleaseKit already owns the independent delivery validation; do not add another local release validator or publication gate. |
| [30. Bound pending proposal work before backend admission](catalog.md#30-bound-pending-proposal-work-before-backend-admission) | E, A, T | Investigate implementation under resolved D3 | Bound local preparation/dispatch and drain results while preserving PromptKit admission ownership, configured waiting/rejection, stable identities, module barriers, and cancellation joins. Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual-admission promise. The current scheduling-dependent selection of rejected chunks is not a compatibility requirement. Measure representative memory/contention to guide the implementation; any future admission-policy gap belongs upstream. |
| [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A | Ready under resolved D4 | Remove generic JSON round trips and release whole-input representations after their last use. The D1 audit additionally identifies source-debug reconstruction: write the acquired bytes directly and retain parsed/normalized views separately, without reopening the input. Share typed output projection; preserve encoding-failure and stream-delivery semantics. Do not introduce an application-level input-size limit; calling integrations own their upload/process budgets. Preserve accepted input formats and stream support. |
| [32. Retain only the run evidence required by enabled outputs](catalog.md#32-retain-only-the-run-evidence-required-by-enabled-outputs) | E, A | Ready | The production consumers establish that much completed evidence is unused without its corresponding output. Retain only required live repair state, compact counts, and selected report/debug detail. Derive not-started counts without a full eager grid where possible; keep complete matrices when enabled and preserve fatal-origin evidence. |
| [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A | Ready under resolved D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected-artifact information. Preserve requested evidence, self-contained interactions, nonfatal debug failures, and continued best-effort capture. Add no quotas, automatic deletion, or automatic stopping after write failures; operators own retention and cleanup. |
| [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E | Investigate costs under resolved D6 | Retain full conversation history and existing per-correction limits. Measure cumulative requests, tokens/bytes, revalidation, and debug costs using existing observations or development instrumentation; remove unnecessary local duplication. Do not add aggregate stopping budgets or reduce model context at this time. Revisit only if production evidence demonstrates a need; no new ordinary-report fields are required for this analysis. |
| [35. Avoid rebuilding growing normalization prefixes](catalog.md#35-avoid-rebuilding-growing-normalization-prefixes) | E | Ready | Growing merge prefixes are repeatedly scanned and copied. Use incremental text/category accumulation and exact estimator components while retaining joiner behavior, punctuation rounding, limits, and category order. Benchmarks should verify the improvement; large refactoring is not a reason to retain quadratic work. |
| [36. Skip validation context work beyond the usable budget](catalog.md#36-skip-validation-context-work-beyond-the-usable-budget) | E, A, T | Ready | Unusable optional context is serialized needlessly. Stop measurement once required-only or ordered-prefix costs settle the fit. Share trimming/batching policy without losing the independent serialized estimator oracle or changing context priority and oversized-item handling. |
| [37. Reuse checked processing selections and matching views](catalog.md#37-reuse-checked-processing-selections-and-matching-views) | E, A | Ready | Readiness and execution derive the same selections separately. Carry one checked processing plan and reuse immutable effective views and inspections per captured runtime. Preserve provenance and first-failure attribution; cache keys must include ordered selections and exemptions. Do not add cross-invocation caches or reload configuration. |
| [38. Select glossary context by identity membership](catalog.md#38-select-glossary-context-by-identity-membership) | E, A | Ready | Context needs identity membership, not a record for every occurrence/contributor pair. Share scanning but avoid expanding multiplicity for context. Preserve glossary order and keep occurrence evidence for protection. |
| [39. Avoid rescanning sections for skipped edits](catalog.md#39-avoid-rescanning-sections-for-skipped-edits) | E | Ready | Skipped-edit reporting repeats segment searches and filtering merges. Reuse a lookup and aggregate once. Preserve the distinction between stale conflicts and actual response filtering; do not broaden rejection or recovery rules. |
| [40. Use one validator contract and one owner for decision semantics](catalog.md#40-use-one-validator-contract-and-one-owner-for-decision-semantics) | A, C, M | Ready | Duplicate ports and runner-specific verdict rewriting obscure one existing validation contract. Consolidate types, applicability, and decision meaning with their owner; remove unused envelope fields. Preserve zero-budget shortlisting, independent repair feedback, cardinality, and unavailable outcomes. No broad plugin framework or new validator policy is needed. |
| [41. Carry structured failure facts through model operations and reporting](catalog.md#41-carry-structured-failure-facts-through-model-operations-and-reporting) | A, C | Ready | Failure class and reason already exist but are discarded and reconstructed. Carry typed facts through role-specific handling and report counts. Use a consistent internal-failure reason fallback for missing reasons while retaining the original failure class and cause. Avoid reclassifying an operation from its display reason. |
| [42. Keep required output publication separate from optional debug retention](catalog.md#42-keep-required-output-publication-separate-from-optional-debug-retention) | A, C | Ready | Required publication and optional retained evidence have distinct lifecycles. Separate their APIs and share regular-file staging where behavior is common. Preserve transcript-before-report ordering, nontransactional finalization, private staging, and nonfatal debug errors. Coordinate with #1, #7, #16, and #22. |
| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready under resolved D1 | The focused audit identifies removable secret-source assertions, lazy discovery closures, combined-role lists, and preflight/writer credential arguments. Complete common interaction writing without those dependencies. Preserve role-owned payloads, artifact association, and nonfatal write callbacks and synchronization; coordinate with #42/#48. Resolved D5 preserves best-effort capture and operator-owned retention. |
| [44. Simplify structured-call completion and remove obsolete record fields](catalog.md#44-simplify-structured-call-completion-and-remove-obsolete-record-fields) | A, M, T | Ready | Ordinary calls and coordinator turns share completion responsibilities; obsolete payload copies have no production reader. Use explicit once-only completion and remove superseded state after moving meaningful assertions. Preserve native terminal counts, typed causes, and undelivered diagnostic evidence. |
| [45. Retire the test-only direct continuation branch](catalog.md#45-retire-the-test-only-direct-continuation-branch) | M, T | Ready | Direct continuation is test-only; production uses coordination. Move sibling-isolation and structural-repair coverage onto the coordinator before removing the branch. Removing the redundant adapter surface does not remove PromptKit capability or production repair. |
| [46. Use a single proposal handoff and test real runner indexing](catalog.md#46-use-a-single-proposal-handoff-and-test-real-runner-indexing) | A, E, M, T | Ready | Generator enrichment is discarded and real indices are assigned by the runner. Keep one proposal handoff with original local references, and assert actual deterministic runner indices under observed out-of-order completion. Review emitted diagnostic start_index separately before dropping it; move useful tests before removing their harness. |
| [47. Centralize request-local segment reference resolution](catalog.md#47-centralize-request-local-segment-reference-resolution) | A, M | Ready | Initial and repaired proposals duplicate positional reference translation. Share a small protocol lookup while retaining raw references, invalid-target evidence, distinct reference namespaces, and retargeted revalidation. Independent caller expectations should survive consolidation. |
| [48. Give one repair branch a clear execution and result boundary](catalog.md#48-give-one-repair-branch-a-clear-execution-and-result-boundary) | A | Ready | Repair branch state is spread through closures and section aggregation. Give one branch its own execution/result boundary and make capabilities explicit at setup. Keep PromptKit's conversation and turn-limit ownership, independent siblings, final-turn validation, and current repair policy. |
| [49. Define concurrency and ownership in module and validator contracts](catalog.md#49-define-concurrency-and-ownership-in-module-and-validator-contracts) | M, A | Ready | Shared instances and borrowed data already impose concurrent-call and ownership obligations. Document those obligations at the port, permit synchronized stateful implementations, and use owned copies where mutation is needed. This clarifies the existing runtime, not a new ban on state or a forced-cancellation guarantee. |
| [50. Consolidate root and per-module configuration rules](catalog.md#50-consolidate-root-and-per-module-configuration-rules) | A, M | Ready | Root and stage checks duplicate application fallback rules. Consolidate typed checks and resolution helpers while preserving null, zero, inheritance, provenance, diagnostic paths, and stage-dependent defaults. ConfigKit continues to own all composition before handoff; do not reproduce its machinery. |
| [51. Unify transcript conversion and retire redundant parser wrappers](catalog.md#51-unify-transcript-conversion-and-retire-redundant-parser-wrappers) | A, M, C, T | Ready | The strict parser's panic is real but limited to an unused production path. Remove obsolete wrappers after moving useful tests and share the live source-to-canonical conversion. Fixing and retaining the obsolete parser would preserve unnecessary machinery. Keep optional source IDs and normalized output behavior. |
| [52. Give public transcript encoders one segment projection](catalog.md#52-give-public-transcript-encoders-one-segment-projection) | A, E, T | Ready | Two public encoders duplicate segment projection. Give the output boundary one projection and test both envelopes with independent expected fields. Preserve omission, category, numeric, and schema/version behavior. |
| [53. Use one simple token estimate across text-processing packages](catalog.md#53-use-one-simple-token-estimate-across-text-processing-packages) | A, E, T | Ready | Both counters implement the same heuristic. Share its implementation/components and preserve exact estimates, consumer budgets, and independent expected-count tests. A new tokenizer or changed heuristic is outside this consolidation. |
| [54. Centralize transcript cloning](catalog.md#54-centralize-transcript-cloning) | A, M, T | Ready | Deep copying belongs with the transcript model. Preserve nil versus explicitly empty mutable slices in the shared clone, both caller copy boundaries, and current external encoding. Verify independence using fixtures not constructed by the same clone. This internal fidelity choice needs no product decision. |
| [55. Assemble built-in modules at the composition boundary](catalog.md#55-assemble-built-in-modules-at-the-composition-boundary) | A | Ready | Production built-in assembly belongs at composition; the runner already accepts a factory. Move construction while keeping the configuration catalog dependency-light and retaining override seams. Package layout is an implementation choice, not a product decision. |
| [56. Merge shared prompt variables once per model call](catalog.md#56-merge-shared-prompt-variables-once-per-model-call) | E, M | Ready | Processing loads the same shared values twice. Merge once while preserving processing and inspection's different precedence and the protected-term default. Do not accidentally weaken operator prompt-definition replacement support or invent new shared-variable precedence. |
| [57. Clarify what the inherited progress flag does in processing](catalog.md#57-clarify-what-the-inherited-progress-flag-does-in-processing) | M, A, T; P approved | Ready under resolved D7 | Add processing progress through CommandKit’s reporter. An operator selecting --progress wants relatively detailed updates: emit meaningful stage/chunk, proposal, validation, and repair activity as available, rather than only coarse stage summaries. CommandKit owns reporting-mode precedence and stream routing; preserve transcript stdout. Exact event granularity and wording are implementation choices. Request upstream support for any standard reporter gap rather than building a parallel reporting system. |
| [58. Test registered validator roles and embedded assets through their behavior](catalog.md#58-test-registered-validator-roles-and-embedded-assets-through-their-behavior) | T, M | Ready | Registry tests duplicate representation assertions and prompt-count checks can miss substitutions. Check required identities, effective roles, preparability, schema identity, and protective copies at their actual owners. Do not replace them with prompt wording or private implementation assertions. |
| [59. Consolidate overlapping normalization fixtures](catalog.md#59-consolidate-overlapping-normalization-fixtures) | T, M | Ready | Two fixtures protect the same nonmerge outcome. Consolidate their meaningful assertions, keeping distinct normalization behavior and independent input expectations. Coordinate with #51 so copied parser conversion is not retained in the tests. |
| [60. Focus integration tests on decoded artifacts and independent output choices](catalog.md#60-focus-integration-tests-on-decoded-artifacts-and-independent-output-choices) | T | Ready | Decoded artifact identities and evidence associations are the contract; escaping and repeated large fixtures are incidental. Keep one full repair/filter/later-failure attribution scenario and use smaller independent-output fixtures. Remove masking assertions identified in the D1 audit without losing role selection, successive text, output-choice, and failure coverage. |
| [61. Remove abandoned test support and unused fixture inputs](catalog.md#61-remove-abandoned-test-support-and-unused-fixture-inputs) | M, T | Ready | The reachability audit found abandoned helpers and unused fixture inputs. Delete them, including unused secret scanners; the D1 audit also identifies credential-echo fixture machinery to retire when its masking-only consumers disappear. Retain compile-time interface fixtures and useful behavioral assertions, rechecking callers when implementing. |
| [62. Remove unreachable merge and file helpers](catalog.md#62-remove-unreachable-merge-and-file-helpers) | M | Ready | Unreachable merge and file helpers duplicate or bypass live owners. Remove them after a current reference check, retaining live normalization, input diagnosis, and output-session behavior. |
| [63. Remove redaction-only credential discovery and tracking](catalog.md#63-remove-redaction-only-credential-discovery-and-tracking) | A, E, M, T | Ready under resolved D1 | Profile/backend enumeration, environment reads, name caches, and role tracking exist solely for substitution. Remove them after their consumers, preserving native authentication, inspection, prepared snapshots, call IDs, and terminal synchronization. |
| [64. Preserve validator explanations in repair feedback](catalog.md#64-preserve-validator-explanations-in-repair-feedback) | A, M, T | Ready under resolved D1 | Secret substitution affects live repair feedback even without debug. Use domain repair findings directly, retaining proposal association and unavailable-assessment semantics. Keep repair budgets, withdrawal, revalidation, and sibling independence unchanged. |
| [65. Remove credential substitution from CLI presentation](catalog.md#65-remove-credential-substitution-from-cli-presentation) | A, E, M, T | Ready under resolved D1 | Remove preparation/finalization sanitizers and obsolete arguments; present existing warnings, errors, and actual paths through CommandKit. Preserve typed causes, reporting modes, exit behavior, and output failure ownership. Coordinate with #25 for separately obscured setup detail. |
## Implementation planning implications
Group related work by owner rather than implementing catalog numbers as 65
independent tasks. Fixes and their supporting ownership changes belong together:
- Output publication and diagnostic integrity: #1, #3–9, #16, #22, #26,
#42–43, #63, #65, incorporating resolved D1 cleanup, D2 caller ownership, and D5
best-effort capture with operator-owned retention.
- Transcript and glossary correctness: #11–13, #17–21, #35–39, #51–54.
- Runner and model-operation lifecycle: #10, #14–15, #23–25, #27, #30,
#32, #40–41, #44–49, #55–56, #64.
- Verification and obsolete machinery: #28–29, #58–62 and the tests attached
to each other change.
These are dependency groups, not a commitment to one large rewrite. Preserve
independent fixture expectations when consolidating implementations. Move useful
coverage to the live boundary before deleting test-only execution paths. Verify
resource improvements with representative measurements; do not require production
benchmarks before removing plainly redundant work.
Technical follow-up does not require another approval round: locate delegated
error ownership for #24; design and measure local dispatch/result draining for
#30; measure aggregate repair costs for #34. Escalate only if the concrete remedy
crosses a policy boundary below. Native ConfigKit, PromptKit, CommandKit, and
ReleaseKit responsibilities remain upstream-owned. Submitting a feature request
is preferable to maintaining a local workaround when an upstream API is the gap.
Earlier review questions about full-segment protection, cloning details, internal
decision representation, and ordinary publication ownership are resolved here as
correctness/engineering work. They are not additional open product decisions.
Independent whole-pipeline chunk traversal and semantic recovery of unavailable
validator responses remain [separate deferred features](../future.md).
## Resolved D1: withdraw application-wide credential redaction
**Accepted:** 2026-11-15. **Policy update and focused audit:** complete.
**Implementation:** pending.
The maintainer withdrew the application-wide credential-redaction requirement
and approved removing machinery whose sole purpose is serving it, plus
simplifying shared machinery made more complex by it. The architecture policy
was updated by deleting the requirement, without adding a negative assertion.
Upstream libraries' native credential/configuration behavior remains in place.
The earlier proposed response-omission policy is superseded; debug evidence,
including malformed responses, remains useful to retain.
The maintainer's stated scope places artifact access and sharing with the
operator and excludes hostile actors with local filesystem access. This changes
the premises of #3 and #4 and the remedies for #9 and #43. Ordinary publication
correctness, authentication, and required output behavior still have independent
purposes. Subsequent decisions retain their IDs.
### Focused follow-up audit: complete
The [removal and simplification audit](14-redaction.md) records seven findings,
traced consumers, retained responsibilities, tests/docs, and implementation
sequencing. It adds #63–65 and refines #3–4, #9, #25, #31, #43, and the related
test cleanup. It confirms that credential discovery, generic substitution,
serialization round trips, cross-role secret plumbing, repair-feedback rewriting,
and CLI substitution can be removed without replacing native library behavior.
Use that inventory to plan coherent removal/refactoring units. Preserve native
authentication and inspection, evidence fidelity, typed failure classification,
Markdown escaping, ordinary output correctness, and nonfatal debug failures.
No new product decision is required for these removals; subsequent decisions
are independent of D1.
The static audit did not run providers or tests or inspect real credentials.
Update implemented-behavior documentation alongside code changes. The source
audits remain historical evidence; this assessment controls remediation scope.
## Resolved D2: callers own report destination isolation
**Accepted:** 2026-11-15. **Applies to:** #8.
**Status:** decision recorded; operator-documentation update pending.
Report destination isolation remains the caller’s responsibility. Keep the
current placement contract and fixed report basenames. Document in the operations
guide that concurrent jobs select distinct report directories, and that retaining
reports from successive runs likewise requires separate destinations or caller
archiving. Different transcript filenames in one directory do not isolate reports.
Automatic per-run directories, destination locking, waiting, and competing-writer
rejection are not selected. They would add placement or contention semantics to
a workflow the caller already controls. The audit demonstrates overwriting when
callers share destinations; it does not establish an application promise of
per-run isolation. Correct writes to checked destinations in #1 remain independent.
## Resolved D3: retain native backend admission semantics
**Accepted:** 2026-11-15. **Applies to:** #30, with scheduling interactions in #15.
**Status:** decision recorded; local scheduling/resource improvements await
implementation planning.
PromptKit retains ownership of backend admission, including configured waiting
and recoverable capacity rejection. Use native settings where operators want
more waiting capacity. Audita will not add an outer admission-retry loop, a
separate backend limiter, or a guarantee that every chunk eventually receives
admission.
Improve Audita's local scheduling and memory use as engineering work: bound
pending preparation/dispatch, drain completed results, and avoid unnecessary
whole-transcript allocation. Preserve module barriers, stable identities,
ordered application, cancellation/join behavior, and native admission semantics.
The scheduling-dependent choice of chunks rejected by the current fan-out is
not a compatibility requirement; a different admission outcome caused by better
local scheduling does not itself establish a new admission policy.
Choosing to wait until every chunk can receive a turn would be a separate
processing policy. If reconsidered, first assess native PromptKit configuration
and seek upstream support for any gap. Native rejection is not a PromptKit defect.
Representative memory, contention, and cancellation checks should guide the
local implementation without creating a second admission system.
## Resolved D4: improve memory use without an input-size limit
**Accepted:** 2026-11-15. **Applies to:** #31.
**Status:** decision recorded; efficiency improvements await implementation.
Do not add an application-level transcript input-size limit in this remediation.
Remove unnecessary copies, generic serialization round trips, and retained
representations after their last use while preserving accepted input formats,
stream support, and output/error behavior. Calling applications remain
responsible for their upload and process budgets.
The audit establishes memory amplification, but provides no measured threshold
that justifies rejecting otherwise valid CLI workloads. An optional input-byte
budget could help integrations without their own enforcement; it is not selected
because it adds configuration and rejection behavior without a demonstrated need.
## Resolved D5: preserve best-effort debug capture
**Accepted:** 2026-11-15. **Applies to:** #33.
**Status:** decision recorded; efficiency improvements await implementation.
Preserve requested diagnostic evidence and continue best-effort capture. Debug
write failures remain nonfatal. Remove redundant in-memory representations and
serialization, and consolidate equivalent warnings while retaining counts and
affected-artifact information. Keep interaction artifacts self-contained.
Do not add capture quotas, automatic deletion, or automatic shutdown of later
capture after write failures in this remediation. An individual failure need
not prevent later writes from succeeding; continued capture can retain useful
evidence for an unsuccessful run. Operators own retention and cleanup.
Optional budgets or persistent-failure stopping rules could reduce disk use and
repeated failed work, but deliberately omit evidence and add configuration and
reporting obligations. They are not selected. Internal efficiency improvements
should preserve the evidence operators requested.
## Resolved D6: retain repair context and per-correction limits
**Accepted:** 2026-11-15. **Applies to:** #34.
**Status:** decision recorded; cost analysis and efficiency work remain pending.
Retain full conversation history and the existing per-correction limits. Do not
add aggregate section/run repair budgets or reduce conversation context at this
time. Measure cumulative requests, tokens/bytes, revalidation, and debug costs,
and remove unnecessary local copies without changing the model's evidence or
which corrections receive repair attempts. Use existing observations or focused
development instrumentation; no new ordinary-report fields are needed merely
to perform the analysis.
Resource growth partly reflects the deliberately selected repair behavior.
Aggregate budgets could leave later corrections unrepaired because earlier
ones consumed the allowance, while context reduction could reduce repair
quality. Neither change follows from the audit finding alone. Revisit these
options only if production evidence demonstrates a need, through a separate
policy decision.
## Resolved D7: emit detailed processing progress through CommandKit
**Accepted:** 2026-11-15. **Applies to:** #57.
**Status:** feature approved; implementation pending.
Emit processing progress through CommandKit's existing reporter, driven by real
processing lifecycle events. An operator who expressly selects `--progress` is
requesting relatively detailed information. Provide useful updates throughout
processing, including stage and chunk activity and proposal, validation, and
repair work where available; do not limit the feature to coarse stage summaries
merely to minimize output. Include work identities and observed completion counts
where they help the operator follow the run. Exact wording and event granularity
remain implementation choices.
CommandKit continues to own reporting-mode precedence and stream routing.
Preserve stdout for transcript output and respect the effective mode when
`--progress` is combined with other reporting flags. Audita supplies processing
events rather than implementing a separate output-mode or progress system.
Use available upstream execution events where appropriate; request upstream
support for a material reporter gap instead of adding a local workaround.
Retaining an accepted flag with no processing updates was considered and is not
selected: progress is useful for long-running jobs, and the maintainer expressly
approved adding it. Update CLI/operations documentation when implementation
lands. Verify representative real lifecycle updates, native reporting-mode
behavior, and transcript-output integrity without testing incidental wording.
## Decision completion
All seven decisions are resolved. Implementation planning can proceed using the
entry assessments and accepted scopes above. Technical investigations and
measurements remain where identified; they are not unresolved product decisions.

View File

@@ -1,611 +0,0 @@
# Consolidated audit catalog
Audita is a single-process Go CLI. CommandKit dispatches commands and ConfigKit
resolves application settings. Core packages parse, normalize, and chunk
transcripts. The runner coordinates proposals, validation, repair, and application
through a PromptKit adapter. The CLI publishes the corrected transcript and
ordinary reports, with separate opt-in debug artifacts.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
The 12 source audits contain 110 numbered findings. This catalog consolidates them
into 65 entries. Priority reflects potential impact, exposure, and the strength of
the reported evidence. A source can appear under several entries when it covers
distinct issues. Each link leads to the original finding and its supporting
evidence.
The [assessment and decision register](assessment.md) evaluates every entry using
the maintainer-approved taxonomy and separates ready remediation from technical
investigation and pending policy decisions. Consult it before planning changes;
source-audit recommendations are evidence and proposals, not authorization.
The three WONTFIX scopes recorded here remain controlling.
The focused [redaction-removal audit](14-redaction.md) applies resolved D1:
the earlier redaction remedies below are superseded, and #63–65 add distinct
cleanup work. Entry numbers and existing anchors are retained for traceability.
## Data protection and run integrity
### 1. Bind output writes to the checked destinations
Transcript path resolution can select the wrong target, and report writers can
change referents after preflight. Use filesystem-consistent resolution and carry
the checked destination into publication. Cover changed report referents as well
as initially ambiguous paths.
[Correctness §1](01-correctness.md#1-output-path-resolution-can-replace-a-different-file) ·
[Concurrency §6](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) ·
[Security §3](07-security.md#3-report-destinations-can-escape-the-identity-checked-by-preflight) ·
[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) ·
[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership)
### 2. Keep model requests within the configured transport boundary
Provider redirects can take transcript-bearing requests outside the selected
origin or downgrade the transport.
**WONTFIX.** PromptKit inherits the standard `net/http` client's default redirect
and credential-forwarding behavior. Audita accepts those defaults for the
built-in transport.
[Security §2](07-security.md#2-provider-redirects-can-cross-the-configured-transport-boundary)
### 3. Protect credentials inside nested debug responses
**Superseded by D1.** The original observation tested the withdrawn redaction
requirement. Remove Audita's substitution package and use direct artifact
serialization; retain useful responses, including malformed content. Retire
masking-only tests and documentation while preserving evidence-fidelity and
write-failure coverage. Credential discovery and repair/CLI consumers are covered
by #63–65; no nested sanitizer or response-omission policy is selected.
[Security §4](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) ·
[Testing §8](08-testing.md#8-debug-redaction-assertions-mostly-inspect-the-outer-serialization) ·
[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding) ·
[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage)
### 4. Make debug artifact permissions private by default
**Prior remedy withdrawn under D1.** Operators own artifact access and sharing;
hostile local filesystem access is outside the selected threat model. The focused
audit found ordinary creation and filename handling to retain, not a redundant
private-root mechanism. Leave current locations and modes unchanged in this
cleanup. Any future default change needs an independent justification.
[Security §1](07-security.md#1-debug-capture-uses-a-shared-path-and-can-expose-private-artifacts)
### 5. Define the trust boundary for discovered configuration
The selected main file and its automatic companions can influence provider and
credential selection. Document trusted configuration locations and keep
upload-controlled paths separate from operator configuration.
[Security §5](07-security.md#5-configuration-discovery-extends-trust-to-the-main-file-and-its-neighbors)
### 6. Make blocking transcript input and delivery cancellation-aware
Input can block before preflight, and FIFO or stdout delivery can stay blocked
after cancellation.
**WONTFIX (interrupting already-blocked I/O).** The standard file operations used
here take no context, and `io.Writer` offers no general interruption mechanism.
Audita accepts that those calls can remain blocked until I/O can proceed.
The application still owns the outcome when control returns. Check cancellation
before publishing a regular file and after stream delivery returns so a canceled
run exits nonzero and attempts to write failed-status reports.
[Correctness §5](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success) ·
[Concurrency §1](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) ·
[Concurrency §2](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) ·
[Resources §2](05-resources.md#2-blocked-stream-io-extends-resource-lifetime-past-cancellation) ·
[Security §7](07-security.md#7-untrusted-input-size-and-blocking-io-can-exhaust-subprocess-workers) ·
[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state)
### 7. Publish truthful failure reports and preserve initialized evidence
A delivery failure can briefly publish `completed` reports. The internal preflight
seam can also replace initialized reports with empty files when failed-report
rendering is unavailable. Verify the first published state and preserve valid
existing evidence if a failed bundle cannot be rendered.
[Error handling §1](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed) ·
[Complexity §13](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence) ·
[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state)
### 8. Prevent concurrent runs from clobbering ordinary reports
Invocations sharing a report directory replace the same basenames and can leave a
mixed set. **Resolved D2:** destination isolation remains the caller’s
responsibility. Retain current report placement and document distinct report
directories for concurrent jobs or retained run histories. Automatic per-run
placement, locking, waiting, and competing-writer rejection are outside this
remediation. Correct publication to checked destinations remains #1’s concern.
[Concurrency §7](04-concurrency.md#7-concurrent-invocations-sharing-a-report-directory-replace-each-others-evidence)
### 9. Preserve the ordinary YAML schema during redaction
A credential value can match a required key or fixed enum and alter an otherwise
successful report. Under resolved D1, remove the post-serialization redactor and
serialize the typed report directly. Keep schema/scalar fidelity coverage; do
not build a more selective sanitizer.
[Correctness §9](01-correctness.md#9-credential-redaction-can-change-the-ordinary-yaml-schema) ·
[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding)
### 10. Reject contradictory validation decisions
The internal decision boundary can advance a candidate marked both approved and
unavailable. Validate the disposition and approval invariant before filtering or
repair. The bundled validators produce consistent decisions in the audited paths.
[Complexity §11](03-complexity.md#11-a-contradictory-validator-decision-can-advance-an-unavailable-candidate)
### 11. Preserve usable validator decisions when extra references cannot be decoded
A schema-valid unexpected numeric reference can fail whole-batch Go decoding
before per-entry reconciliation. Preserve valid requested siblings and test the
mixed response at the actual adapter boundary.
[Correctness §7](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions) ·
[Testing §5](08-testing.md#5-the-validator-batch-fixture-misses-the-real-numeric-decoding-boundary)
### 12. Use full-segment evidence for protected-name checks
Span-only validation can reject a valid edit or miss a repair opportunity. Share a
glossary-owned matching and protection assessment across validation and
application, and recheck the current segment when applying an edit.
[Correctness §8](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity) ·
[Equivalence §9](10-equivalence.md#9-reconcile-the-protection-rule-at-validation-and-application) ·
[Boundaries §5](11-boundaries.md#5-give-glossary-matching-and-protection-a-domain-owner)
### 13. Preserve transcript input fidelity during JSON shape detection
Generic decoding can round large integer IDs and fail on values in fields the
typed reader ignores. Select the segment array as raw JSON and test exact IDs
across both supported input shapes.
[Correctness §6](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) ·
[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open) ·
[Resources §4](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication)
### 14. Honor cancellation at the runner commit boundary
An already canceled empty pipeline can return success, and cancellation during
module application can still allow the runner to commit edits. Check entry and
final commit boundaries and give successful and interrupted results one assembly
path.
[Complexity §1](03-complexity.md#1-give-module-execution-one-place-to-commit-its-result) ·
[Complexity §12](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline) ·
[Concurrency §3](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary)
### 15. Retain completed proposal evidence behind slow sections
The ordered prefix gate can delay validation for ready sections and omit completed
proposal evidence on interruption. Capture usable responses independently of
validation dispatch and preserve deterministic identities.
[Correctness §10](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) ·
[Concurrency §4](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention) ·
[Complexity §1](03-complexity.md#1-give-module-execution-one-place-to-commit-its-result)
### 16. Keep ownership of failed staging cleanup
Removal errors can discard temporary paths or go unreported, preventing a retry.
Keep paths until removal succeeds or the path is absent and verify cleanup
failures and cancellation after output acquisition.
[Resources §1](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) ·
[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state)
## Correctness and failure reporting
### 17. Preserve the complete time interval when merging segments
A nested segment can shorten the merged end time and change later gap and duration
decisions. Use the full interval for both eligibility and output, and cover nested
intervals in normalization tests.
[Correctness §3](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) ·
[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open)
### 18. Match protected names at Unicode and punctuation boundaries
The matcher misses some names with accented letters or punctuation at their edges.
**WONTFIX (Unicode behavior of `\b`).** Go's `regexp` defines `\b` as an ASCII
word boundary. Audita accepts that definition for the standard regexp engine.
Configured aliases with punctuation at their edges still need a glossary matching
rule for protection and context selection, even when their letters are ASCII.
[Correctness §2](01-correctness.md#2-glossary-matching-misses-names-with-accented-letters-or-punctuation-at-their-edges)
### 19. Detect overlapping occurrences when requiring a unique edit span
Nonoverlapping counting can treat a repeated overlapping span as unique. Use one
literal occurrence rule in preview, validation, and application.
[Efficiency §7](02-efficiency.md#7-the-unique-span-check-accepts-overlapping-matches)
### 20. Satisfy feasible explicit chunk targets
A balancing preference can reject inputs that have a valid contiguous partition.
Make the configured count and limits prerequisites, then optimize balance within
feasible choices. Verify skewed inputs at the feasibility boundary.
[Correctness §4](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) ·
[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open)
### 21. Keep input order for exactly tied timestamps
An unstable sort can reorder utterances with identical times. Use source position
as a stable tie-breaker.
[Efficiency §8](02-efficiency.md#8-normalization-can-reorder-utterances-with-exactly-tied-timestamps)
### 22. Preserve failure phase and origin in run reports
Required output errors can omit their phase, and fatal model errors can lose
chunk, role, validator, or batch coordinates. Carry typed failure facts from their
source and assemble run records and counts in one place. Assert the serialized
fields.
[Correctness §11](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) ·
[Error handling §4](06-error-handling.md#4-fatal-model-errors-lose-their-chunk-and-operation-coordinates) ·
[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) ·
[Boundaries §8](11-boundaries.md#8-give-run-level-report-facts-one-assembly-path) ·
[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership)
### 23. Classify provider attempt timeouts separately from operation deadlines
The adapter can report an HTTP attempt timeout as a logical operation deadline.
Use the native classifications and test attempt, operation, and parent
cancellation under distinct live budgets.
[Error handling §2](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline) ·
[Testing §4](08-testing.md#4-the-timeout-tests-cover-two-of-the-three-relevant-deadline-owners)
### 24. Use input diagnostics for operator-correctable failures
Invalid processing selections and malformed inspection configuration can return
the runtime exit status. A genuinely infeasible chunk target can also be reported
as an internal defect. Preserve the distinction between readiness, chunking, and
runtime failures and assert exact statuses in CLI tests. Preserve native
inspection behavior; retired application-wide masking expectations are not an
additional acceptance criterion.
[Error handling §5](06-error-handling.md#5-invalid-processing-selections-use-the-runtime-failure-exit-status) ·
[Error handling §6](06-error-handling.md#6-an-infeasible-chunk-target-is-recorded-as-an-internal-processing-failure) ·
[Testing §2](08-testing.md#2-inspection-failure-tests-accept-the-wrong-exit-classification)
### 25. Identify missing required credentials in safe diagnostics
The process path reduces a known missing credential requirement to a generic setup
failure. Surface actionable native setup detail and preserve the underlying
typed cause and failure classification. D1 removes the blanket restriction on
cause presentation; ordinary compact report projections remain useful.
[Error handling §3](06-error-handling.md#3-missing-required-credentials-produce-a-generic-setup-message) ·
[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership)
### 26. Render input and output paths as plain Markdown text
Untrusted filenames can alter the rendered report or its navigation. Escape path
text consistently with other untrusted labels and verify the rendered meaning.
[Security §6](07-security.md#6-untrusted-transcript-filenames-can-become-active-markdown)
### 27. Keep the validator execution class used by the runner
The ledger can infer a different class from an injected validator's name. Carry
the resolved class in decision evidence and join the evidence once before emitting
rows, preserving disposition and repair lineage.
[Complexity §8](03-complexity.md#8-join-correction-evidence-before-building-ledger-rows) ·
[Boundaries §2](11-boundaries.md#2-preserve-the-execution-class-assigned-to-each-validator-decision)
### 28. Verify signal and exit handling in the production executable
Subprocess fixtures recreate startup wiring and can miss regressions in the actual
binary. Exercise the built executable or share the production entry point for
representative startup and cancellation cases.
[Testing §1](08-testing.md#1-subprocess-tests-recreate-the-production-executable-boundary)
### 29. Fail the release bootstrap when cleanliness cannot be established
A failed Git status command can be mistaken for a clean checkout. Check its status
explicitly and exercise that boundary in place of testing the release test's own
comparison helper. The delivery tool has a separate prepared-checkout check.
[Error handling §7](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) ·
[Testing §11](08-testing.md#11-remove-the-tests-of-the-release-tests-own-set-helper)
## Scalability and resource use
### 30. Bound pending proposal work before backend admission
One goroutine per chunk makes finite-capacity rejection depend on scheduling and
can retain requests and results across the whole run. Bound dispatch and result
retention without changing the module barrier or backend admission ownership.
**Resolved D3:** preserve PromptKit's configured waiting and recoverable rejection.
Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual-
admission promise. Current scheduling-dependent rejection order is not a
compatibility requirement. Verify memory use, actual contention, and cancellation
while improving local scheduling; any future admission-policy gap belongs upstream.
[Concurrency §5](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling) ·
[Resources §3](05-resources.md#3-backend-admission-leaves-chunk-fan-out-and-result-retention-unbounded) ·
[Testing §6](08-testing.md#6-concurrency-fixtures-leave-some-schedules-unproved)
### 31. Reduce peak memory for large transcript input and output
Whole-file reads, a generic JSON tree, retained source bytes, and full output
buffers amplify memory before and after model work. Reduce live copies and
write the acquired source bytes directly to its debug snapshot instead of
reconstructing them for redaction. Keep parsed/normalized views separate.
**Resolved D4:** improve memory use without adding an application-level
transcript-size limit. Calling integrations own their upload/process budgets;
preserve accepted input formats and stream support.
[Redaction removal §3](14-redaction.md#3-save-the-acquired-source-snapshot-without-reconstructing-it) ·
[Resources §4](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication) ·
[Security §7](07-security.md#7-untrusted-input-size-and-blocking-io-can-exhaust-subprocess-workers)
### 32. Retain only the run evidence required by enabled outputs
Ordinary runs retain debug-only candidates and decision history. Summary-only runs
still allocate the full work-unit grid, and module results duplicate operation
records and unused proposal totals. Preserve live validation and repair state, and
collect compact counts when detailed evidence is disabled.
[Efficiency §5](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history) ·
[Resources §5](05-resources.md#5-summary-only-runs-retain-a-full-grid-of-work-and-operation-records) ·
[Dead code §3](12-deadcode.md#3-remove-duplicate-runner-bookkeeping-for-unused-module-totals)
### 33. Limit debug retention costs within a run
Execution histories and repeated serialization increase peak memory, and retained
interaction files and warnings can accumulate after write failures. Reduce
duplicate representations and serialization, and consolidate equivalent warnings
with counts and affected-artifact information. **Resolved D5:** preserve requested
evidence, self-contained interactions, nonfatal failures, and continued best-effort
capture. Add no quotas, automatic deletion, or automatic stopping after write
failures; operators own retention and cleanup.
[Resources §6](05-resources.md#6-debug-capture-adds-memory-and-disk-pressure)
### 34. Account for cumulative correction-repair history and work
Each rejected candidate can start its own bounded conversation and repeat the
original prompt history, with additional revalidation. **Resolved D6:** retain
full conversation history and existing per-correction limits; add no aggregate
budgets or context reduction at this time. Measure cumulative work and remove
unnecessary local duplication. Revisit behavior changes only if production
evidence demonstrates a need.
[Resources §7](05-resources.md#7-correction-repair-replays-growing-history-under-per-correction-budgets)
### 35. Avoid rebuilding growing normalization prefixes
Merge estimation and category accumulation repeatedly copy and scan the same
growing text and category prefixes. Reuse incremental work while preserving exact
text, category order, and the configured limit.
[Efficiency §1](02-efficiency.md#1-normalization-repeatedly-copies-and-scans-growing-merge-runs)
### 36. Skip validation context work beyond the usable budget
Optional context is serialized even when required fields already exceed budget.
Share trimming and batching policy, stop measuring context that cannot fit, and
keep serialized measurement as an independent reference for the optimized path.
[Efficiency §4](02-efficiency.md#4-validation-measures-optional-context-that-cannot-fit) ·
[Equivalence §1](10-equivalence.md#1-share-validation-budget-policy-across-measurement-paths)
### 37. Reuse checked processing selections and matching views
Preflight and execution separately decide active roles, and repeated profile or
glossary selections repeat inspection, copying, and preparation. Carry one checked
processing plan and reuse immutable results for equivalent selections.
[Efficiency §6](02-efficiency.md#6-repeated-pipeline-selections-duplicate-setup-work) ·
[Boundaries §4](11-boundaries.md#4-carry-one-checked-processing-plan-from-preflight-into-execution)
### 38. Select glossary context by identity membership
The validation context builder materializes each matched occurrence for every
contributor and then deduplicates them. Use a membership path for context and
preserve occurrence counts for protection.
[Efficiency §3](02-efficiency.md#3-validator-glossary-context-expands-every-occurrence-into-every-contributor)
### 39. Avoid rescanning sections for skipped edits
Apply-time filtering searches the original chunk for every skipped proposal. Reuse
a segment lookup and preserve stale-conflict and filtering distinctions.
[Efficiency §2](02-efficiency.md#2-application-reporting-searches-the-source-chunk-for-each-skipped-edit)
## Ownership and maintainability
### 40. Use one validator contract and one owner for decision semantics
Runner-side applicability and verdict handling overlap with validator
implementations. Consolidate the interface and acceptance rules, and remove the
unused validation request envelope, segment ID field, and result name copy.
[Complexity §3](03-complexity.md#3-put-validator-applicability-and-decision-meaning-at-their-owning-boundary) ·
[Boundaries §1](11-boundaries.md#1-give-the-validator-runtime-interface-one-owner) ·
[Dead code §4](12-deadcode.md#4-drop-the-unconsumed-validation-request-envelope)
### 41. Carry structured failure facts through model operations and reporting
Proposal, validation, and repair repeat class and reason extraction, and report
counts reclassify failures from reason codes. Carry typed outcome facts forward
while preserving role-specific recovery and fatal handling.
[Complexity §7](03-complexity.md#7-give-reporting-one-source-for-operation-classification-and-counts) ·
[Equivalence §4](10-equivalence.md#4-share-structured-failure-facts-across-model-operations)
### 42. Keep required output publication separate from optional debug retention
The output session combines unrelated lifecycles. Give required-output preparation
and publication a clear owner, share regular-file staging mechanics for transcript
and reports, and expose debug setup through its own API.
[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) ·
[Equivalence §2](10-equivalence.md#2-share-regular-file-staging-between-transcript-and-report-publication) ·
[Boundaries §7](11-boundaries.md#7-separate-required-publication-from-retained-debug-artifacts)
### 43. Inject one interaction diagnostics contract
Proposal, validation, and repair duplicate path types and writer wiring. Supply a
diagnostic sink from the adapter boundary, with role-specific payloads and
nonfatal write warnings. Remove secret-provider assertions, lazy discovery
closures, combined-role secret lists, and preflight/writer credential arguments.
Keep artifact association and optional write-error reporting.
[Complexity §4](03-complexity.md#4-separate-validation-batch-assessment-from-the-diagnostic-writer) ·
[Equivalence §5](10-equivalence.md#5-give-optional-interaction-artifacts-one-writer-contract) ·
[Boundaries §6](11-boundaries.md#6-inject-interaction-diagnostics-at-the-adapter-boundary) ·
[Redaction removal §4](14-redaction.md#4-remove-secret-plumbing-from-writer-and-preflight-contracts)
### 44. Simplify structured-call completion and remove obsolete record fields
Share finalization across ordinary calls and coordinated turns, and remove
obsolete error-kind state, response usage copies, and duplicate callback payloads.
Keep typed causes and live execution and artifact evidence.
[Complexity §5](03-complexity.md#5-make-the-structured-clients-call-lifecycle-explicit) ·
[Dead code §5](12-deadcode.md#5-remove-structured-call-fields-superseded-by-execution-observations)
### 45. Retire the test-only direct continuation branch
The supported repair path uses the coordinator. Move direct-continuation
assertions about sibling isolation and structural repair to coordinated calls,
then remove the unused request branch.
[Complexity §5](03-complexity.md#5-make-the-structured-clients-call-lifecycle-explicit) ·
[Dead code §1](12-deadcode.md#1-retire-the-test-only-direct-continuation-path)
### 46. Use a single proposal handoff and test real runner indexing
The generator builds enriched proposals the production wrapper discards. Give the
runner index ownership, remove the parallel enrichment, and assert indices under
observed out-of-order completion instead of in the test-only composition harness.
[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly) ·
[Testing §6](08-testing.md#6-concurrency-fixtures-leave-some-schedules-unproved) ·
[Testing §9](08-testing.md#9-delete-proposal-order-assertions-on-the-test-only-composition-harness) ·
[Dead code §2](12-deadcode.md#2-remove-proposal-enrichment-that-the-live-handoff-discards)
### 47. Centralize request-local segment reference resolution
Initial proposals and repair revisions duplicate the same local-position lookup.
Preserve raw references, stable proposal identity, and revalidation for retargeted
edits.
[Equivalence §3](10-equivalence.md#3-resolve-model-segment-references-in-one-place) ·
[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly)
### 48. Give one repair branch a clear execution and result boundary
Branch coordination, previews, revision validation, accounting, and diagnostics
share state inside the section loop. Separate branch state and make repair prompt,
exchange capture, and coordinator capability explicit.
[Complexity §2](03-complexity.md#2-separate-one-repair-branch-from-the-sections-repair-loop) ·
[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly)
### 49. Define concurrency and ownership in module and validator contracts
Workers share module and validator instances and borrow section and glossary data.
Specify concurrent calls, read-only borrowed inputs, and cooperative cancellation
at the interface boundary.
[Concurrency §8](04-concurrency.md#8-shared-module-and-validator-ownership-is-implicit-at-their-interfaces)
### 50. Consolidate root and per-module configuration rules
Resolution and validation repeat role defaults, null and inheritance rules, and
field checks. Share the rules while keeping path-specific diagnostics and
configuration provenance.
[Complexity §10](03-complexity.md#10-share-root-and-per-module-configuration-rules) ·
[Equivalence §6](10-equivalence.md#6-use-one-set-of-checks-for-inherited-configuration-fields)
### 51. Unify transcript conversion and retire redundant parser wrappers
Source-to-canonical conversion is duplicated across the CLI, a lenient wrapper,
and fixtures. The strict test-only wrapper panics on omitted IDs. Use one
conversion and the supported decoder, and preserve independent fixture
expectations.
[Efficiency §9](02-efficiency.md#9-the-strict-transcript-parser-panics-on-a-missing-id) ·
[Testing §12](08-testing.md#12-consolidate-normalization-fixtures-at-the-domain-and-parser-owners) ·
[Boundaries §9](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries) ·
[Dead code §8](12-deadcode.md#8-remove-the-transcript-parser-variants-that-only-tests-call)
### 52. Give public transcript encoders one segment projection
The output registry implements the envelope and delegates the bare form to the
schema package. Share the field projection at the output boundary and verify both
wire shapes independently.
[Boundaries §9](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries)
### 53. Use one simple token estimate across text-processing packages
Normalization and chunking implement the same word and punctuation heuristic.
Share the counter and preserve consumer-specific budgets and independent
expected-count tests.
[Equivalence §7](10-equivalence.md#7-use-the-same-simple-token-counter-for-text-estimates)
### 54. Centralize transcript cloning
Runner and proposal application duplicate deep-copy code with different
empty-category handling. Share the copy operation while keeping both caller
ownership boundaries and verifying independence from fixture data.
[Equivalence §8](10-equivalence.md#8-centralize-transcript-copies-without-moving-the-copy-boundaries)
### 55. Assemble built-in modules at the composition boundary
The framework registry imports all concrete modules even though the runner
consumes a factory interface. Keep config keys independent of implementations and
place production construction with the CLI or built-ins.
[Boundaries §3](11-boundaries.md#3-put-production-module-construction-at-the-composition-boundary)
### 56. Merge shared prompt variables once per model call
Processing loads the embedded instruction values twice. Keep processing precedence
and the missing protected-term default in one pass, and preserve caller precedence
for inspection.
[Dead code §6](12-deadcode.md#6-resolve-shared-prompt-variables-once-per-model-call)
### 57. Clarify what the inherited progress flag does in processing
`process` advertises progress messages but supplies no producer. **Resolved D7:**
add processing progress through CommandKit's reporter. Operators selecting
`--progress` want relatively detailed updates; expose meaningful stage/chunk,
proposal, validation, and repair activity as available, rather than only coarse
stage summaries. CommandKit owns mode precedence and stream routing; preserve
transcript stdout. Exact wording and event granularity are implementation choices.
[Dead code §10](12-deadcode.md#10-resolve-the-inherited-progress-flag-for-process)
## Test maintenance and dead code
### 58. Test registered validator roles and embedded assets through their behavior
Consolidate redundant execution-class checks and assert required prompt identities
and preparability. Keep schema identity, diagnostic metadata, and protective-copy
coverage.
[Testing §10](08-testing.md#10-consolidate-validator-registry-and-asset-metadata-assertions)
### 59. Consolidate overlapping normalization fixtures
Two different-speaker fixtures protect the same rejection. Keep the distinct
domain cases and meaningful merge, text, and category assertions.
[Testing §12](08-testing.md#12-consolidate-normalization-fixtures-at-the-domain-and-parser-owners)
### 60. Focus integration tests on decoded artifacts and independent output choices
Assert decoded IDs and evidence associations and use smaller fixtures for
duplicate reporting and debug combinations. Keep a full scenario for attribution
through repair and later failure. Remove masking expectations without discarding
the independent artifact and outcome assertions mixed into those fixtures.
[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) ·
[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage)
### 61. Remove abandoned test support and unused fixture inputs
The orphan test-support package, unused CLI and runner helpers, and legacy
glossary fixture input have no current callers. Preserve compile-time interface
fixtures and active behavior assertions.
[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) ·
[Dead code §9](12-deadcode.md#9-clean-up-abandoned-test-support) ·
[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts)
### 62. Remove unreachable merge and file helpers
The uncalled merge predicate and standalone JSON and file-write helpers duplicate
or bypass the live normalization, schema, and output paths.
[Dead code §7](12-deadcode.md#7-remove-orphaned-normalization-and-file-helpers)
## Additional cleanup after resolved D1
### 63. Remove redaction-only credential discovery and tracking
Audita enumerates captured profiles/backends, reads credential environment values,
caches names, and tracks prepared role names under a mutex solely for substitution.
Remove that state and the secret-source methods after updating their consumers.
Preserve PromptKit authentication, prepared evidence, execution synchronization,
and native inspection behavior.
[Redaction removal §1](14-redaction.md#1-remove-credential-discovery-and-tracking-used-only-for-substitution)
### 64. Preserve validator explanations in repair feedback
Repair rewrites validator explanations using both roles' credential values even
with debug off. Remove this transformation and use domain feedback directly.
Retain proposal association, unavailable-assessment semantics, and the existing
bounded repair/revalidation behavior.
[Redaction removal §5](14-redaction.md#5-stop-rewriting-validator-explanations-sent-to-repair)
### 65. Remove credential substitution from CLI presentation
Preparation and completion paths scrub warnings, errors, and announced paths;
completion discovers credentials even with reports/debug disabled. Use direct
CommandKit presentation, remove obsolete arguments, and retain typed causes,
reporting modes, exit status, and required-output failure handling. Coordinate
with #25 for setup detail still hidden by the separate failure wrapper.
[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership)

View File

@@ -1,829 +0,0 @@
# Reliability and Maintainability Implementation Plan
Implement the [target roadmap](reliability-and-maintainability.md) in one
end-to-end sprint, starting from its application baseline `c4ccd78`. The
[assessment](audit/assessment.md) governs the accepted scope and decisions
D1–D7. Numbers under each stage refer to the [audit catalog](audit/catalog.md).
Use the original audits for reproductions and source references. Where an old
remedy differs from a later decision, follow the assessment and target roadmap.
Work through the numbered stages in order. Each stage is sized for one Luna
implementation prompt and contributes directly to the final design. Intermediate
source may be incomplete or fail to build while adjacent changes are in flight.
Use the next stage to finish those connections. A stage needs neither a
standalone commit nor a passing full suite. Replace obsolete paths directly;
compatibility layers and temporary production adapters are unnecessary.
Use the [development guide](../development.md) and applicable adopted policies,
especially [agents](../policy/agents.md), [architecture](../policy/architecture.md),
[testing](../policy/testing.md), and [documentation](../policy/documentation.md).
Add or adapt focused tests at the behavior owner and update canonical docs when
their implemented behavior changes. Run affected tests when the relevant path
builds, then close the sprint with Stage 20. Stage checks establish behavior
without fixing private names, scheduler sizes, event order, or prompt wording.
The target roadmap defines retained contracts, selected changes, and exclusions.
## Stage 1 — Establish the delegated inspection dependency
Audit #24, delegated part.
1. Use a temporary CLI build to reproduce invalid operator configuration with
`promptkit list prompts` and `process`. Reuse the malformed companion fixture
in `internal/cli/runtime_errors_test.go` and include a separate cancellation
case. The baseline reported status 1 for the malformed companion via
inspection and status 2 via processing. Both commands must remain offline.
2. Trace `internal/cli/app_wiring.go` into the pinned `ckprompt` and `ckconfig`
commands. For the known catalog case, `ckprompt.runCatalogList` converts the
loader error to `definitionFailure` even when a typed input cause is
available. Check relevant configuration-scope acquisition paths separately.
Preserve typed causes in Audita's loader. Classify only input failures at
the responsible boundary, leaving cancellation and runtime failures distinct.
3. Check for a released upstream fix for each demonstrated delegated case. If a
suitable release exists, adopt the smallest compatible update using the
[OSS policy](../policy/oss.md) and verify classification at the Audita CLI.
Otherwise record the affected command, reproduction, owner, and needed
behavior for an upstream handoff. Continue the independent stages and use
Stage 20 to integrate and verify a supporting release if available. Keep the
native declarations and routing throughout.
A fix is demonstrated by status 2 and actionable diagnostics for the invalid
input, and status 1 for cancellation. While the supporting release is pending,
record the mismatch explicitly. A test that asserts only nonzero status does
not verify this fix.
**Complete.** The upstream handoff below records the original behavior.
The [delegated inspection follow-up](#delegated-inspection-fix) records the
adopted fix.
Build a temporary executable with `go build -o /tmp/audita-inspection ./cmd/audita`.
In a scratch directory, put `version: 3` in `config.yml`, `profiles: [` in the
adjacent `prompts.yml`, and `[]` in `transcript.json`. End each file with a newline.
| Invocation with `/tmp/audita-inspection` | Status | Diagnostic |
| --- | --- | --- |
| `promptkit list prompts --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` |
| `process transcript.json --config config.yml` | 2 | `yaml.invalid` in `prompts.yml` |
| `config show --scope promptkit --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` |
`config sources`, `config validate`, `config explain --list`, and
`config diff --against-config config.yml` also returned 1 for the `promptkit`
scope with the same diagnostic. All five `app` scope operations succeeded
with the malformed companion and returned 1 with `yaml.invalid` for a
separately selected main file containing `version: [`. These failures occur
during configuration loading, before provider work. The cancellation test
passes an already canceled context through the application entry point and
verifies status 1, `context.Canceled`, and empty stdout for catalog listing,
configuration show, and processing.
Both native adapters receive the typed ConfigKit load cause from Audita's
existing loaders. In `ckprompt`, `runCatalogList` assigns the operational
outcome through `definitionFailure`. In `ckconfig`, the scope handlers
assign it through `configFailure`, with `show` calling it through
`configAcquisitionFailure`. The upstream owners are the respective CommandKit
integrations. They need to assign status 2 for typed operator-input failures
at those boundaries, preserving diagnostics, available inspection evidence,
and the cause. Cancellation and other operational failures need status 1.
At the time of the handoff, the latest release of both integrations was
`v0.7.0`, the version Audita used for the reproduction.
## Stage 2 — Remove application credential substitution
Audit #3, #9, #31 (source bytes), #63–65. Use the
[focused removal inventory](audit/14-redaction.md) for affected sites.
1. Switch ordinary reports and debug artifacts to direct typed JSON/YAML
encoding. Keep encoded effective configuration from ConfigKit. In Markdown,
retain escaping and remove substitution. Keep malformed `final_content` as
the captured string. Write the acquired transcript bytes to the debug source
artifact and keep the parsed and normalized views separate.
2. Send domain `repairFindings` directly for initial feedback and subsequent
findings. Preserve the proposal association and typed unavailable assessment.
Present native errors, warnings, and actual paths through CommandKit. This
step removes substitution. Stage 12 exposes the native setup detail that is
currently hidden by generic failure presentation.
3. Remove writer/renderer secret arguments, `DebugInput.Secrets`, discovery
assertions and closures, repair role-list combination, cache and role
tracking, runtime secret discovery, and `internal/core/redaction`. Keep the
prepared and terminal callbacks, source-path tracking, native effective
rendering, and selected provider authentication.
4. Delete masking-only assertions, scanners, fixtures, and stale redaction
descriptions identified in the inventory. Move independent assertions to
writer or caller tests. Check decoded schema and scalar types, literal source
bytes, malformed response evidence, actual paths, and native auth using
synthetic data. Check feedback by decoding a production repair request.
**Complete.** Reports and debug artifacts encode typed data directly, source
snapshots preserve the acquired bytes, and repair requests carry the original
domain findings. Removed application credential discovery and substitution.
The CLI build and full offline suite passed with Go 1.27.3, including checks
for native authentication, malformed response capture, and destination paths.
## Stage 3 — Unify optional interaction capture
Audit #33 and #43.
1. Define one small optional interaction sink at the shared execution port.
Compose the existing filesystem writer from the run directory in CLI wiring.
Pass the sink through proposal, validation, and repair. Each role supplies
its own payload and stage/chunk/operation coordinates, including proposal,
batch, validator, and repair turn where applicable. Keep returned paths and
write errors available for ledger association and warnings.
2. Reuse the write path and artifact envelope in
`internal/framework/llm/diagnostics.go`. Remove private writer adapters and
repeated construction. Keep artifacts self-contained, including revalidation
and a final unusable or undelivered response when available. Keep debug
directory placement, naming safety, and creation modes.
3. Collect optional write warnings safely across workers, grouping equivalent
failures by operation and cause with counts and affected artifact information.
Continue later capture after a write error. Keep the warning path outside
correction decisions and required file delivery.
Verify role, chunk, batch, proposal, and attempt associations with decoded
artifacts. Exercise a failed optional write followed by later capture. Update
the [diagnostics internals](../internal/diagnostics-reporting.md) for the sink.
**Complete.** The CLI supplies one optional sink for proposal, validation,
repair, and revalidation artifacts. The writer records successful paths even
after a partial write and groups failures by role, artifact operation, and
cause. Decoded-artifact and recovery checks passed, along with the full Go
suite and targeted race checks.
## Stage 4 — Bind publication to checked destinations
Audit #1, #6 (cooperative boundaries), #7, #8, #16, #26, #42.
1. In `internal/core/diagnostics/output_session.go` and its CLI callers, resolve
checked transcript and report destinations with filesystem semantics.
Carry those destination and parent choices through staging and publication.
Keep protected-source checks, intentional in-place replacement, ordinary
symlink and stream behavior, and disabled reporting's inactive destination.
Add no directory creation or locking for required destinations.
2. Give the required-output owner a shared regular-file staging primitive for
transcript and reports. Keep optional debug setup separate from that owner.
Keep temporary paths until removal succeeds or absence is established.
Surface cleanup errors alongside the initiating error, including after
partial preparation, publication failure, or cancellation.
3. Make report rendering return an explicit error through initialization and
finalization. Publish the transcript first when processing succeeded.
Select a failed bundle before the first final report write when a required
delivery or cleanup failure is already known. If a report write first
establishes failure, switch the remaining writes to failed status and make
one best-effort pass over reports that still need failed evidence. If the
failed bundle cannot be rendered, keep initialized or published evidence
and return the rendering error.
4. Check context when transcript acquisition returns, at natural preparation
and publication boundaries, and after stream delivery returns. A cancellation
observed after bytes escape still fails the invocation. Join cancellation
and I/O/cleanup errors where available and keep the initiating failure
distinguishable. Use ordinary blocking I/O and retain completed transcript
publication if a later report write fails.
5. Escape input/output paths as literal Markdown text in `core/reporting`,
preserving raw values in structured artifacts. Update
[operations](../operations.md) for caller-owned destination isolation,
retention, and the resulting publication lifecycle.
Exercise symlink/path resolution, in-place and stream output, cancellation
after I/O, cleanup errors, initialized evidence when rendering fails, and
write behavior after known failure. Observe publication attempts as well as
final files. Leave processing fact and ledger projection to Stage 17.
**Complete.** Required outputs use the destinations resolved during selection
and share regular-file staging. Publication checks cancellation, retains and
reports cleanup failures, and uses failed-status reports when required delivery
cannot complete. Tests verified publication attempts, retained evidence when
rendering failed, symlink destinations, stream cancellation, and Markdown path
escaping. The CLI build and full offline Go suite passed.
## Stage 5 — Consolidate typed transcript data
Audit #13, #31, #51, #52, #54, #62 (file helper).
1. In `core/schema`, detect the two accepted root shapes while keeping the
segment array as raw JSON. Decode typed segments directly. Preserve source
ID presence, native integer range, duplicate checks, ignored unknown fields,
and existing format and error behavior. Keep one live source-to-canonical
conversion and retire obsolete parser wrappers after moving meaningful tests.
2. Centralize transcript cloning in the model. Use it at the runner and other
copy boundary, preserving independence and nil versus explicit empty
mutable slices. Share one typed segment projection across both encoders in
`core/schema` and `core/outputschema`. Preserve the envelopes, version,
numeric fidelity, category and omission rules, and encoding error behavior.
3. Release raw input after its final parse or source-snapshot write. Drop the
parsed source after normalization and its debug consumers. Avoid retaining
it via CLI finalization. Remove the audited unused file helper after checking
callers. Keep both file and stream inputs and current output delivery
semantics. Use a representative large fixture to check that the removed
representations no longer overlap needlessly. A fully streaming encoder is
optional if supported by the measurements and existing error semantics.
Test both envelopes, adjacent accepted IDs beyond exact float precision where
supported by native `int`, clone independence with independent expected data,
and the distinction between raw and typed debug snapshots. Update the
[input contract](../integrations/transcript-glossary-files.md) and
[encoder internals](../internal/output-schemas.md) where affected.
**Complete.** Source segments decode directly from both accepted JSON shapes.
Exact IDs, clone independence, encoder values, and debug snapshot fidelity were
verified. Processing releases raw bytes after parsing or source capture and
releases the parsed view before normalization. On a 100,000-segment object
fixture (Go 1.27.3, five iterations), parser allocations fell from 240.6 MB to
84.1 MB per operation. The CLI build and full offline suite passed.
## Stage 6 — Correct and simplify normalization
Audit #17, #21, #35, #53, #59, #62 (merge helper).
1. In `core/normalization`, sort stably by the existing time key and retain
input order for ties. Compute the prospective merge end as the later end
and use that interval for the duration limit, merged segment, and next gap.
Keep overlap acceptance and the existing speaker, gap, and token rules.
2. Use one implementation of the existing token estimate across normalization,
chunking, and its consumers. Maintain the components needed for that exact
estimate as a merge grows. Accumulate text and category membership/order
incrementally with the existing joiners. Avoid rescanning or copying a
growing prefix on each append. Retire the unreachable merge helper.
3. Consolidate overlapping nonmerge tests. Keep independent fixtures for
nested intervals, ties, joiners, category ordering, and exact estimates.
Compare allocations or growth on a representative run of same-speaker merges.
**Complete.** Verified stable start-time ties, nested intervals, exact joiner
and token behavior, and category order. The merge uses incremental text,
categories, and the shared token estimator. On synthetic same-speaker runs
(Go 1.27.3, three iterations each), allocations fell from 238.1 MB to 0.42 MB
per operation for 2,000 segments and from 960.1 MB to 0.85 MB for 4,000. The
CLI build and full offline suite passed.
## Stage 7 — Make feasible explicit section targets succeed
Audit #20 and the infeasibility portion of #24.
1. In `core/chunking/sections.go`, use indivisible, ordered segments as the
partition units. Preserve the empty-transcript behavior and the singleton
exception for segments individually above the token maximum. Keep
`min_section_tokens` validation and compatibility without adding a new
partition constraint.
2. For an explicit target, compute the farthest legal end for each start
with a rolling token sum, honoring the oversized singleton exception. Work
backward to compute each suffix's minimum section count as one plus the
count at its farthest end. In the forward builder, reserve enough segments
for the remaining nonempty sections and use the suffix count to reject cuts
that leave too little capacity. Prefer the current average-token target
among the feasible cuts. Count sections by segment presence, including
zero-token segments. Preserve implicit-target behavior.
3. Return a typed infeasibility error for impossible requested targets. Carry
it through the runner and CLI so the existing input status applies while
internal and cancellation failures retain their classes.
Verify the audited weights `[2,5,1]` with target 2 and maximum 6, impossible
targets, and the oversized singleton case. Check contiguity, order, coverage,
and feasibility independently of the chosen cut. Keep a single chunking pass.
**Complete.** Explicit targets use rolling legal ends and suffix feasibility to
admit exactly the requested count when possible. An independent enumeration
checked 1,024 weight sequences for every possible count, including zero-token
and oversized segments. The real CLI returned status 2 for infeasibility
without dispatching model work. The CLI build and full offline suite passed.
## Stage 8 — Share edit location and protection evidence
Audit #12, #18 (punctuation portion), #19, #38, #47.
1. In `framework/proposals`, search for a second literal starting offset for
require-unique replacements, including overlaps. Use the same applicability
rule in previews and apply. Preserve replace-all's nonoverlapping behavior
and the existing blank-result check.
2. Share the positional segment lookup for initial proposals and repair
revisions. Carry raw invalid model references to evidence, keep segment
positions distinct from candidate and validator response indices, and
revalidate a repair that retargets to another segment.
3. Move protected-name comparison to the glossary owner. Compare usable
full-segment before/after text and repeat the check against the current
segment at apply time. A missing or ambiguous preview cannot support a
protection verdict. An available empty preview may support the verdict
independently of blank-result rejection. Preserve active-glossary
exemptions, case and longest-match rules, and spoken-word repetition cleanup.
Use standard literal-edge checks for punctuation-edged spellings while
retaining Go regexp word-boundary semantics.
4. Reuse the glossary scan for context identity membership without expanding
each occurrence/contributor pair. Keep occurrence evidence for protection
and preserve the order of selected context.
Test overlapping occurrences, full-segment protection, punctuation edges,
ambiguous previews, spoken-word cleanup, and stale apply-time changes at the
owning boundaries. Check the initial and repair lookup against independent
expected references, and compare work on a repeated-glossary fixture.
**Complete.** Proposal preview and application agree on overlapping spans.
Initial and repair responses share positional lookup and retain original model
references in diagnostic evidence. Protection runs on usable full-segment
previews and checks the current text at application. The 2,000-mention
fixture with 128 shared contributors measured 81.0 MB to 0.27 MB allocated
and 77.8 ms to 2.1 ms per operation (Go 1.27.3, ten iterations). The CLI build
and full offline suite passed.
## Stage 9 — Consolidate validator decisions and reconciliation
Audit #10, #11, #27 (runtime class), #40, #49 (validator port).
1. Use one shared owner for the validator interface, applicability, decisions,
and cardinality helpers. Remove duplicate ports and unused validation-only
envelopes after tracing live consumers. Keep concrete validator chains,
built-in prompt/rule ownership, and reusable LLM batching.
2. Resolve a single authoritative disposition before a candidate advances.
Reject contradictory internal results as fatal contract failures, and derive
redundant serialized approval from the validated disposition. Preserve full
normal cardinality and checked partial results on fatal returns. Preserve
zero-budget shortlisting, independent repair feedback, unavailable
assessments, and deterministic validator ordering.
3. Record the effective execution class selected by the runner on both initial
and revalidation decisions, including supported instance overrides. Keep
the existing default for validators without an explicit class. Document
concurrent-call and borrowed-input behavior at the port and allow
synchronized stateful implementations.
4. Preserve exact numeric references through structured-response decoding
using `json.Number` or raw numeric tokens. Reconcile with the requested
batch-local range before converting to local `int`. Recognize equivalent
integral decimal and exponent forms exactly and ignore extras that cannot
name a requested candidate, including values beyond native integer range.
Keep supplied numeric references in diagnostic evidence. Continue whole
response schema validation and structural repair upstream. One usable
decision keeps its outcome. Omission or duplication affects only its
requested candidate, and ignored extras on a successful batch retain their
existing recovery classification.
Test with actual response decoding on a mixed batch and one legitimate decision
beside an out-of-range reference. Cover equivalent numeric spellings,
contradictory internal results, and class override from execution through the
record supplied to Stage 17. Update [validator internals](../internal/validators.md).
**Complete.** Validators share one port and applicability owner. The runner
rejects contradictory decisions and records the effective class on initial and
repair validation evidence. Controlled provider responses verified exact
numeric decoding, equivalent spellings, duplicate and missing decisions, and
a usable decision alongside a reference above the native integer range. The
CLI build and full offline suite passed.
## Stage 10 — Reduce validation context work
Audit #36.
1. In `framework/validators`, consolidate budget and optional-context policy
across the live validation batch builder and trimmer. Measure the required
payload first and stop testing optional material beyond the usable
ordered prefix. Reuse computations where the exact serialized estimate
permits it. Keep context priority, numbering, effective budget, oversized
single-input behavior, and independently measured wire size.
2. Retain a test oracle built from serialized payloads rather than the shared
estimator. Use a representative long optional-context fixture to demonstrate
reduced measurements or allocation. Retain request-shape and available
sibling expectations under over-budget input.
**Complete.** Validation budgets measure required data first and stop at the
first optional entry that does not fit, with glossary entries preceding
neighbors. An independent serialized-payload oracle checked fit and batching.
With 5,000 optional entries and room for one, allocations fell from 3.12 MB to
2.13 KB per operation (Go 1.27.3, twenty iterations). The CLI build and full
offline suite passed.
## Stage 11 — Hand checked selections to execution
Audit #24 (readiness), #37, #49 (module port), #50, #55, #56.
1. Consolidate Audita-owned root and stage validation/fallback helpers in
`core/config` after ConfigKit composition. Preserve presence/null,
explicit zero, inherited values, diagnostic paths and provenance, and
stage/role-dependent defaults. ConfigKit retains its load and override paths.
2. Replace duplicate derivation in `framework/preflight` and
`internal/cli/process_pipeline.go` with one checked processing plan. Store
resolved specs, validated role choices, and effective glossary/protection
views. Cache repeated profile/backend inspections and immutable glossary
views only within the captured runtime. Include ordered selection and
exemptions in reuse keys. Preserve readiness check order, first failure
attribution, protected source paths, no model calls during preflight, and
preflight-before-publication. Return typed status-2 failures for invalid
selections; keep runtime/canceled outcomes distinct.
3. Move built-in construction to composition beside the concrete modules or
in CLI wiring. Keep the injected runner factory and dependency-light key
catalog. Document concurrent calls and borrowed data at the module port.
Copy owned inputs where mutation is required.
4. In processing and inspection request construction, merge shared variables
once per call while preserving each path's precedence, the protected-term
default, and operator replacements for bundled prompts.
Check representative effective selections and provenance, first-failure
attribution, provider-free inspection, constructible catalog entries, and both
request precedence paths. Update [CLI/config internals](../internal/cli-config.md),
[module internals](../internal/modules.md), and configuration docs where affected.
**Complete.** Configuration uses common validation and role fallback after
composition. Preflight passes checked role and glossary selections to execution
and reuses inspection and glossary work within the invocation. The built-in
factory lives beside the modules. Processing and native inspection preserve
their respective variable precedence with one merge per request. The CLI build
and full offline suite passed, including status 2 for an unknown processing
profile before provider dispatch or output initialization.
## Stage 12 — Unify structured-call completion and failure facts
Audit #23, #25, #41, #44.
1. In `framework/llm` and `framework/structuredoutput`, share once-only
completion between ordinary execution and coordinated turns. Keep
prepared snapshots, call IDs, terminal synchronization, native group and
call counts, coverage and unknown usage, and diagnostic-only content from
undelivered responses. Remove obsolete response/usage copies and callback
payloads after checking report and debug consumers.
2. Carry typed class, reason and cause from adapter through operation records.
Use the internal-failure reason when a reason is absent, retaining the
original class and cause. Apply parent cancellation precedence, identify
native logical-operation deadline expiry, then native transport/attempt
timeout before treating a generic deadline as logical timeout. Use the
existing transport-failure reason for HTTP-only timeouts.
3. Make setup diagnostics actionable at the CLI. Surface the native missing
credential requirement and available role/profile context with the typed
cause intact. Keep ordinary reports on their compact typed projections.
Exercise distinct parent, operation, and attempt timeouts with independently
controlled budgets. Check setup class/cause and useful native information,
plus ordinary/coordinator terminal and undelivered-evidence paths. Update
[LLM runtime internals](../internal/llm-runtime.md).
**Complete.** Ordinary and coordinated calls share once-only completion,
including prepared and undelivered evidence and native terminal counts.
Proposal, validation, and repair retain typed failure facts. Independent
parent, operation, and attempt deadlines produced the expected classifications.
The CLI identifies a missing required credential by role, profile, and
environment variable while the ordinary report retains the setup reason.
The CLI build, full offline Go suite, and targeted race checks passed.
## Stage 13 — Use one proposal handoff
Audit #46.
1. In `framework/proposal_generation` and `framework/runner`, return one
set of decoded proposal results with raw local references, filtering
evidence, and optional native exchange. Remove the generator's enriched
projection discarded by the live wrapper. Keep runner ownership of
sequential proposal indices within a stage, ordered by chunk and response.
2. Check emitted interaction metadata such as `start_index` separately from
private references. Preserve its meaningful serialized association and
retire fields confirmed to have no surviving purpose.
3. Move proposal-order assertions from the test-only composition harness to
real runner results. Observe out-of-order completion in the fixture and
verify indices and original references. Keep model response and reference
coverage in the generator's owning tests.
**Complete.** The generator and runner share one decoded proposal result with
raw local references, operation facts, and the optional native exchange. The
runner assigns indices in section and response order. A production-module
fixture observed the later response first and verified applied indices,
operation associations, and raw references. Proposal diagnostics retain
the response-local `start_index`. The full offline suite and repeated
ordering checks under the race detector passed.
## Stage 14 — Own each correction repair branch explicitly
Audit #34, #45, #48, #64 (branch integration).
1. In `framework/runner/repair.go`, give one proposal branch a clear
execution input and result. Make repair capability explicit at setup. Keep
branch state and terminal outcome together. Merge outcomes into the section
after each branch, without mutating unrelated approvals or decisions.
2. Run repairs through PromptKit coordination with full exchange/history and
configured per-correction limits. Preserve initial and subsequent direct
domain feedback, sibling independence, withdrawal, unavailable assessment,
retargeting, and revalidation of the final allowed revision. Move useful
structural repair and isolation tests onto this path before deleting the
direct-continuation-only test branch.
3. Use controlled observations or temporary instrumentation to measure
cumulative turns, available token/byte costs, revalidation calls, and debug
work on a representative multi-correction fixture. Remove unnecessary local
copies. Keep those measurements out of the ordinary report contract.
Check independent approved siblings, attempts/turn counts, actual branch
history and feedback, final-turn revalidation, and associated artifacts in
production coordination.
**Complete.** The runner checks repair capability before proposal dispatch.
Each proposal branch owns its history reference, attempt facts, and terminal
outcome. The section merges the returned evidence and approvals. Tests exercise
PromptKit coordination for independent histories, replacement prompts,
structural repair, final-turn revalidation, and retained siblings and
artifacts. The CLI build, full offline suite, and targeted race checks passed.
The three-correction CLI fixture used two repair branches and three repair
turns, with four revalidation requests and ten provider requests overall.
The default prompt sent 9,487, 13,385, and 9,483 repair request bytes in both
debug modes. Debug produced 39 files totaling 135,659 bytes in the observed
run, compared with zero files when disabled. The replacement prompt sent
6,303, 7,017, and 6,299 repair request bytes and produced 122,943 debug bytes
across 39 files. The fixture supplied no provider token usage. Temporary
measurement instrumentation was removed.
## Stage 15 — Give stage execution one commit boundary
Audit #14, #15 (evidence), #22 and #41 (failure facts).
1. In `framework/runner`, check context at entry, including empty pipelines,
and before committing a stage. Use one stage completion path to record
outcomes and evidence. Only a successful active stage advances the working
transcript. Preserve previously committed text for partial results.
2. Carry initiating cause, phase and available stage/chunk/operation reference
from the owner. Cancel and join peers on fatal failure or parent cancellation;
keep peer interruption and secondary errors attributable without promoting
them over the initiating failure. Continue recoverable work under existing
partial-success semantics.
3. Collect completed proposal evidence independently of dispatch to validators.
After workers join, include usable out-of-order completions in interrupted
diagnostics with deterministic indices and associations. Keep raw evidence
from undelivered responses separate from usable proposals. Retain the ordered
indexing/validation boundary until Stage 16 evaluates it.
Test canceled entry and precommit, a later completed proposal behind a slow
earlier chunk, and a fatal stage after earlier committed text. Assert causal
origin and preserved evidence as well as returned failure. Leave report/ledger
projection to Stage 17.
**Complete.** A single module completion path advances the transcript only
after a successful application and context check. The runner returns the
initiating error with its phase and available origin, and keeps peer failures
on their work units. After joining, it assigns deterministic indices to usable
proposal responses that never reached validation. Tests exercised canceled
entry, cancellation after approval, a later response behind an earlier
failure, and preservation of earlier committed text. The CLI build, full
offline suite, and targeted race checks passed.
## Stage 16 — Bound local preparation and actively drain results
Audit #15 (dispatch) and #30, under D3.
1. Replace the per-chunk goroutines and chunk-count-sized pending channels in
`collectSectionProposals` and `runModulePipeline` with a private rolling
stage window. Admit chunks to the window in input order. Charge a slot for
each chunk still in preparation, model work, or awaiting downstream work.
Release it when the chunk is finished and its final facts have been drained.
Start with a small fixed local window and keep it private. Drain outcomes
while workers run and on cancellation, then join and finish. Keep compact
or requested completed evidence apart from remaining work. Collect results
while the workers can still send them.
2. Measure staged overlap and peak pending work with many sections, a
controlled slow first proposal, and shared PromptKit capacity. Where active
draining and ordered indexing yield adequate progress, keep that simpler
boundary. If ordered indexing still leaves repeatable idle capacity with
validation-ready chunks, use a chunk index and response position to identify
candidates during work. At ordered collection, assign the existing stage
indices and translate every projected proposal reference, including failure
indices, repair lineage, and interaction-to-ledger mappings. Keep the
identity-sensitive artifact data in the charged window until the index is
assigned; use the Stage 3 writer for the final artifacts. Keep batch-local
model references and raw response evidence intact. Check the same mappings
after a cancellation joins the workers.
3. Keep the module barrier and deterministic application. PromptKit retains
admission, queueing, limits, retries and recoverable rejection. Include
nested validation/repair in the local resource and cancellation checks
without duplicating backend admission or holding its permits across a
nested call. Tune the window using the measured workload and contention;
no public window, throughput minimum, or eventual-admission guarantee follows.
Test actual observed overlap, finite native contention, cancellation with work
pending, and identical index ordering for the same usable results under
different completion orders. Report representative pending-work measurements
without pinning a timing or worker-count threshold in the default suite.
**Complete.** The runner admits sections through a bounded rolling window,
drains active outcomes on cancellation, and lets validation and repair proceed
while earlier proposals are pending. It assigns final indices in section order
and publishes identity-sensitive artifacts through the interaction writer.
Tests checked native contention and overlap, stable ordering, and repair and
revalidation evidence after completion and cancellation. The full offline Go
suite and targeted race checks passed.
In three runs of a 36-section fixture with two shared provider slots and the
first proposal held for 250 ms, ordered dispatch completed seven later proposals
and no validations during the hold. With section-local identities, seven later
validations also reached the provider before release, with eight sections
charged to the window in both variants. Both variants completed 36 proposals
and 36 validations overall. The window of eight was retained for this workload.
## Stage 17 — Project typed report and ledger facts
Audit #22, #27 (ledger), #39, #41 (projection).
1. In `framework/processreport`, build the snapshot from the Stage 12 and 15
facts and Stage 4 delivery outcomes. Preserve phase and available origin for
setup, processing and publication errors. Keep recovery and work-unit counts
and known versus unknown values. Distinguish the primary run error,
interrupted peers, and secondary output failures without counting a fatal
run failure again as a recoverable unit failure.
2. Join proposals, validator decisions, repairs, application and skip evidence
once by stable identities in `correction_ledger.go`. Use the recorded
execution class from Stage 9. Project one candidate row with correct
disposition, lineage and literal empty replacement values. Retain
module-only failures. Preserve serialized fields and logical ordering.
3. In runner application attribution, reuse section/segment lookups and
aggregate skipped evidence once. Preserve the distinction between stale
apply-time conflicts and response filtering. Keep exact correction,
filtered-proposal and validation-decision identities across projection.
Verify decoded YAML, Markdown counts, and ledger facts against an independently
specified mixed run, including partial decisions, repair/filter interaction,
later failure, and an overridden validator class. Update the
[output contract](../integrations/output-files.md) and diagnostics internals.
**Complete.** Reports project typed operation, processing, and delivery
facts, with separate attribution for the primary failure, interrupted peers,
and file failures. The ledger joins each candidate's evidence once, uses
recorded validator classes, and preserves literal empty replacements and
repair lineage. Apply-time filtering reuses original section lookups and
groups skip evidence by its originating operation. The full offline Go suite
passed, including mixed YAML and Markdown counts, ledger class overrides,
and the repair/filter/later-failure subprocess fixture.
## Stage 18 — Retain evidence according to selected outputs
Audit #32 and supporting work for #30, #33, and #44.
1. Derive evidence needs once from enabled ordinary reports and effective debug
capture. Keep compact CLI totals and available failure facts in summary-only
mode. Allocate work-unit detail only when a consumer needs it, and avoid
eager whole-run grids and debug-only candidate/decision histories in
summary-only paths. Keep complete enabled report matrices and debug evidence.
2. Discard active validation, exchange and repair state after their last
consumer, including data copied solely for obsolete runner projections.
Remove duplicate operation lists and unused module totals after checking
their projections. Keep native observations and requested self-contained
artifacts when capture is selected.
3. Compare live/retained state for representative long runs with summary only,
ordinary reports, debug, and their existing combinations. Check shared
totals and logical results across modes, with detail present where selected.
Resource measurements support reduction of waste without setting a
constant-memory or disk-use requirement.
**Complete.** The CLI selects ordinary report and debug evidence before
processing. Section outcomes feed shared totals as they drain. The runner
retains the full work grid only for reports, and retains candidate and
validator history only for debug in the CLI. Duplicate module operations and
unused proposal totals were removed. Prepared messages are captured only for
interaction diagnostics. Mode comparison tests and the full offline suite
passed.
A synthetic 4,000-section run with recoverable failures, proposal recoveries,
and apply-time filtering retained about 0.46 MB with summary only, 2.73 MB
with ordinary reports, 1.56 MB with debug evidence, and 3.83 MB with both
after garbage collection. Allocations per run were approximately 20.9,
22.0, 24.0, and 25.0 MB respectively. These runner measurements exclude
provider calls and disk artifacts. All four modes agreed on the transcript
and counters. Temporary measurement instrumentation was removed.
## Stage 19 — Emit native processing progress and update operator guidance
Audit #5, #57, and the operator-facing parts of #8, #24 and #25.
1. Use a small event callback from the live lifecycle owners. Supply stage and
chunk activity, proposal/validation/repair phases and useful completions
with available identities and observed counts. Reuse suitable PromptKit
events where they convey useful activity. Keep event delivery within the
invocation and finish workers before the reporter closes.
2. Wire that callback in CLI to `Invocation.Reporter().Progress` in pinned
CommandKit. Let CommandKit determine visibility, routing, and presentation
failure status, independently of transcript/report processing outcomes.
Preserve stdout transcript data. Respect ordinary mode, explicit progress,
debug defaults, explicit `--progress=false`, and quiet suppression.
3. Update the [CLI reference](../cli.md), [operations](../operations.md) and
[subprocess contract](../integrations/subprocess.md) at their owned
boundaries. Cover meaningful progress, caller-owned report destinations and
artifact retention, and the trusted configuration bundle, including
discovery, imports and automatic companions. Keep the distinction between
native configuration inspection and processing readiness.
Use representative command tests for live activity and combined modes,
asserting useful identities and stream separation without exact wording or
concurrent ordering. Keep a progress-specific check separate from delegated
inspection availability.
**Complete.** Runner lifecycle and native model events flow through
CommandKit progress reporting. Subprocess tests observed proposal progress
while a provider was blocked, checked repair activity, verified stdout
separation and reporting-mode combinations, and confirmed that a failed
progress destination leaves completed output intact. The CLI, operations,
and subprocess guides cover live progress, destination ownership, and
configuration trust. The full offline Go suite passed.
## Stage 20 — Verify the integrated sprint and release bootstrap
Audit #28, #29, #58, #60–62 and sprint closure.
1. In `.woodpecker/release.yml`, capture `git status --porcelain --untracked-files=all`
in an assignment that checks the command's exit status separately from its
content. Then require empty output. Preserve ReleaseKit's independent
prepared-checkout validation and tag delivery. In `scripts/release_test.go`,
cover clean, dirty and failed-status-with-empty-output behavior at the
command boundary. Remove redundant tests of the comparison helper while
keeping real release target and wiring checks.
2. Verify signal, cancellation and exit behavior through the real executable
or a shared production entry point. Keep narrower checks where a process
adds no evidence. Verify effective validator roles, required prompt
identities, embedded asset preparability, schema identity, and protective
copies through use. Retain a representative repair/filter/later-failure
integration scenario and independent output-choice cases. Remove the
audited abandoned fixture inputs and helpers after checking callers.
3. Recheck Stage 1. Adopt and verify a released upstream fix that became
available using the same scope. Otherwise report the exact outstanding
upstream-dependent behavior and its disposition. Treat that item as
outstanding, while verifying the independent Audita work. Keep the
invalid-input and cancellation cases distinct.
4. Account for the [catalog](audit/catalog.md) against the landed changes and
stage evidence. Entries #2 and #4 need no implementation, and only the
accepted portions of #6 and #18 were selected. Check canonical docs,
examples, and affected links and remove stale implemented-behavior claims.
Keep historical audit observations intact.
5. From the integrated tree, build the CLI and run the full offline
`go test ./...` with the supported Go toolchain. Run targeted race checks
for changed shared-state paths and review the representative measurements
from Stages 6, 8, 10, 14, 16 and 18. Verify formatting and whitespace,
and inspect the final diff. Reuse successful checks for unchanged content.
Existing [release validation](../policy/release.md#candidate-validation)
applies when a release is prepared.
The completed sprint should leave the target architecture reachable through
`audita process` and the standard delegated commands. Report checks and
measurements actually performed, including any outstanding external dependency.
**Complete.** The integrated Audita work and release bootstrap passed checks.
The [delegated inspection follow-up](#delegated-inspection-fix) closes the
upstream item that was open at the end of the sprint.
The bootstrap rejects a dirty checkout and a failed status command with no
output. A clean fixture reaches ReleaseKit with the prepared checkout and
expected commit arguments. Process fixtures share the production entry point
and check interrupt and termination exit status. Registry coverage checks
effective roles, required prompt identities and preparation, schema identity,
and independent copies. The ordered processing fixture decodes debug artifact
coordinates. The full repair/filter/later-failure case remains separate from
the smaller output-choice matrix. Unused fixture helpers and the legacy
glossary input were removed.
### Delegated inspection status
At sprint close, the latest CommandKit integration releases were `ckprompt`
and `ckconfig` v0.7.0. The built Audita executable reproduced the Stage 1
mismatch:
| Invalid input | Path | Observed status and diagnostic |
| --- | --- | --- |
| Malformed automatic `prompts.yml` | `process` | 2, `yaml.invalid` |
| Malformed automatic `prompts.yml` | `promptkit list prompts` | 1, `yaml.invalid` |
| Malformed automatic `prompts.yml` | `config show/sources/validate/explain/diff --scope promptkit` | 1, `yaml.invalid` |
| Malformed explicit app main file | `config show/sources/validate/explain/diff --scope app` | 1, `yaml.invalid` |
Caller cancellation returned status 1 in the shared CLI and subprocess tests.
### Delegated inspection fix
Audita adopted `ckprompt` v0.9.0, `ckconfig` v0.8.0, and PromptKit v0.23.1.
PromptKit preserves `ErrInvalidConfig` through `errors.Is` and unwraps to the
original cause. Both integrations can now classify the typed ConfigKit load
error as invalid input. Audita uses the existing native command wiring.
[ckprompt v0.9.0 release notes](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/29694ea874b53151d832c141b7c5913779852e0e/integrations/ckprompt/docs/releases/v0.9.0.md) ·
[PromptKit v0.23.1 release notes](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/74066167942fcf6f70c112359995739ec9789fa6/docs/releases/v0.23.1.md)
The Stage 1 fixture now returns status 2 with `yaml.invalid` identifying
`prompts.yml` for `promptkit list prompts`, `process`, and all five PromptKit
configuration commands. `config sources` and `config validate` still emit
their reports. All five app-scope commands likewise return status 2 for a
malformed main file. The CLI cancellation test still verifies status 1,
`context.Canceled` in the cause chain, and empty stdout.
### Catalog closure
The table identifies the implementing stages and exceptions without changing
the historical audit observations. Each stage records its verification above.
| Catalog findings | Implementing stages or disposition |
| --- | --- |
| #1, #6–8, #16, #26, #42 | Stage 4. Only cooperative cancellation boundaries from #6 were selected. |
| #2, #4 | No implementation. The existing catalog dispositions stand. |
| #3, #9, #63–65 | Stages 2 and 14. |
| #5, #57 | Stage 19. |
| #10–11, #27, #40, #49 | Stages 9, 11, and 17. |
| #12, #18–19, #38, #47 | Stage 8. Only punctuation-edge handling from #18 was selected. |
| #13, #31, #51–52, #54 | Stages 2 and 5. |
| #14–15, #22, #30–32, #39, #41 | Stages 5, 12, and 15–18. |
| #17, #21, #35, #53, #59, #62 | Stages 5 and 6, with final caller checks in Stage 20. |
| #20, #24 | Stages 7 and 11 for processing, and the [delegated inspection fix](#delegated-inspection-fix). |
| #23, #25, #44 | Stages 12, 18, and 19. |
| #28–29, #58, #60–61 | Stage 20. |
| #33, #43 | Stages 3 and 18. |
| #34, #45, #48 | Stage 14. |
| #36 | Stage 10. |
| #37, #50, #55–56 | Stage 11. |
| #46 | Stage 13. |
Verified with Go 1.27.3 by building the CLI, running `go test -count=1 ./...`
offline, and running targeted race checks for the runner, LLM adapter,
validators, reports, diagnostics, glossary, and CLI. The final review
checked Go formatting, whitespace, and 278 relative documentation links
and exercised the example configurations. Measurements above cover merge
and glossary allocation reductions, validation context, repair request and
artifact costs, bounded section contention, and retained evidence by output
mode. These are synthetic measurements. No release was published.

View File

@@ -1,444 +0,0 @@
# Audita Reliability and Maintainability Roadmap
**Date:** 2026-11-15. **Status:** [Implementation landed](implementation.md).
[Post-implementation findings and their status](remediation.md) track follow-up work.
**Planning baseline:** `730c1e4`; its application code matched the audited `c4ccd78`.
## Purpose and scope
Audita should produce faithful transcripts and useful evidence, handle partial
success and failures consistently, and use resources proportionately to the work
and outputs selected. Its components should have clear owners, with less local
machinery where standard Go facilities or the adopted libraries already supply
the behavior. Operators requesting progress should be able to follow a run in
detail through CommandKit.
This roadmap organizes the accepted audit remediation around that target state.
The [assessment](audit/assessment.md) owns the disposition of individual findings
and the rationale for decisions D1–D7; the [catalog](audit/catalog.md) links to
their evidence. The coverage appendix below maps that evidence to the target areas.
Earlier source-audit remedies remain proposals where the assessment supersedes
them. This document specifies the target state and accepted scope. The
[implementation plan](implementation.md) owns the sequence and verification.
The baseline already had native library integration, preflight,
ordered module execution, partial-success validation, bounded correction repair,
ordinary reports, and opt-in debug artifacts. The sprint improved those paths
and added processing progress. It preserved editorial policy and the public
configuration schema. The D1 architecture-policy update preceded the related
code cleanup.
### Design boundaries
Follow the [architecture policy](../policy/architecture.md) and use the libraries'
supported paths. ConfigKit owns composition, presence, provenance, and resolution
before handoff. PromptKit owns credentials, model execution, admission, retries,
and conversation coordination. CommandKit owns standard commands, reporting
modes, and stream routing. ReleaseKit retains its release-validation ownership.
Audita owns transcript meaning, role selection, domain validation and repair
decisions, processing coordination, and presentation of its results.
The accepted decisions establish these boundaries for this remediation:
| Area | Selected scope |
| --- | --- |
| Diagnostic content | Remove Audita's credential substitution and the discovery/plumbing supporting it; retain native library behavior and useful evidence. |
| Report destinations | Keep existing placement; callers isolate concurrent jobs and retained run histories. |
| Admission | Keep native waiting and recoverable rejection. Improve local scheduling without a second backend limiter, admission-retry loop, or eventual-admission promise. |
| Input and capture resources | Remove waste; add no transcript-size cap, capture quota, automatic artifact deletion, or automatic stop-capture policy. Operators own retention and integrations own their upload/process budgets. |
| Correction repair | Retain full conversation history and existing per-correction limits. Aggregate budgets and reduced context are not selected at this time. |
| Progress | Supply relatively detailed processing activity through CommandKit when progress is requested, subject to its effective reporting mode. |
Respect the three accepted Go limitations: standard HTTP redirect behavior,
standard regexp word-boundary semantics, and the inability to interrupt an
already-blocked ordinary file/stream call. Fix Audita's own choice of operation
and its handling when control returns. Output correctness here does not promise
hostile-filesystem race resistance, cross-file transactions, power-loss recovery,
or atomic synchronization with cancellation. Keep ordinary symlink, in-place
output, and stream support. Debug root locations and creation modes stay as they
are for this cleanup.
The separate features in [deferred work](future.md), including semantic recovery
of unavailable validator decisions and independent whole-pipeline traversal by
chunk, remain outside this roadmap. Internal refactoring should preserve current
contracts except for the deliberate changes and correctness fixes described here;
it need not preserve incidental implementation details or known defects.
## 1. Truthful output and useful diagnostic evidence
### Target state
Required output writes use the destinations that were actually checked. Reports
reflect known processing and delivery outcomes, retain available failure origin,
and remain useful when finalization itself encounters an error. Optional debug
capture has a separate lifecycle and does not determine processing success.
Diagnostic representations preserve the information supplied to them through
ordinary serialization.
The baseline resolves some output paths differently at checking and writing,
can publish a completed report after learning of delivery failure, and can lose
initialized evidence or staging-cleanup ownership. Credential substitution adds
serialization passes and can alter schema keys, messages, and path announcements.
The [output contract](../integrations/output-files.md) and
[operations guide](../operations.md) describe the existing output surfaces.
### Work to reach the target
- Give required-output preparation, staging, publication, and cleanup one clear
owner. Resolve paths using filesystem semantics and carry checked destination
information into publication. Share regular-file staging mechanics between
transcript and reports, retaining private staging and intentional in-place
replacement. Keep ownership of temporary paths until removal succeeds or
absence is established; report cleanup failures alongside the primary error.
- Keep transcript-before-report publication and nontransactional finalization.
Select failed report status when a delivery failure is already known, carry
report-rendering errors explicitly, and preserve initialized reports when a
replacement failure bundle cannot be rendered. Retain bounded best-effort
attempts to publish enabled reports; a later report failure does not erase an
already-published transcript.
- Check cancellation at natural publication boundaries and after stream delivery
returns. A canceled run should report failure even if bytes have already
escaped. Do not implement detached I/O workers to simulate interruption of
standard blocking calls.
- Build ordinary reports from processing-owned facts, preserving phase, available
stage/chunk/operation origin, documented counts, and existing serialized
identities. Distinguish the initiating failure, peer cancellation, and secondary
output errors. Join correction evidence once per proposal before projecting
ledger rows; carry the validator execution class actually used rather than
reconstructing it from a name. Preserve disposition, repair lineage, empty
replacement values, partial decisions, and module-only failures.
- Complete the [redaction-removal inventory](audit/14-redaction.md): delete
redaction-only credential enumeration, environment reads, name caches, role
tracking, substitution utilities, and secret-only parameters/interfaces. Encode typed
JSON/YAML directly. Save acquired source bytes without reconstructing or
reopening the source; keep parsed and normalized views separately. Preserve
ConfigKit rendering and native authentication/inspection.
- Supply one interaction-writer contract from the adapter/composition boundary.
Proposal, validation, and repair owners supply their own payloads and
association coordinates. Remove lazy secret callbacks and cross-role secret
lists. Retain self-contained interaction artifacts, malformed final response
content as evidence, nonfatal write callbacks, and their synchronization.
Consolidate equivalent write warnings with counts and affected-artifact
information while continuing best-effort capture.
- Keep Markdown escaping separate from substitution: input/output paths should
render as literal text without changing their structured values. Document
caller-owned report-directory isolation and artifact retention when updating
the operator guide.
### Completion evidence
Focused filesystem tests cover checked destinations, known delivery failure,
failed report rendering, cleanup errors, and cancellation once I/O returns.
Assertions inspect publication behavior as well as final files. Decoded reports
retain their schema and types, and ledger evidence agrees with execution.
Artifact tests establish source fidelity, role/attempt association, and nonfatal
debug failures. Redaction-only code and assertions are removed without replacing
them with no-op seams or losing native authentication coverage.
## 2. Bounded processing with clear operation ownership
### Target state
Local preparation, dispatch, and completed-result handling avoid allocating a
worker and prepared request for every chunk at once. Completed evidence is owned
independently of ordered application. Each stage and model operation has a clear
completion path, and typed outcomes reach reports without being reconstructed
from display strings.
The baseline fans out work across all chunks, retains out-of-order results behind
slow earlier work, and duplicates operation state across callbacks and projections.
Backend admission already belongs to PromptKit; it does not bound all of this
local preparation and retention. The [pipeline](../internal/pipeline.md) remains
ordered by stage, with fixed chunk membership and deterministic proposal/application
identity.
### Work to reach the target
- Give stage execution a single result/commit boundary. Honor cancellation on
entry, including an empty pipeline, and before committing new stage work.
Cancel and join peers on fatal failure or parent cancellation, retaining earlier
committed stage text and available completed evidence. Recoverable operation
failures remain local under the existing partial-success rules.
- Design bounded local preparation and dispatch together with concurrent draining
of results. Preserve deterministic identities, validator requirements, ordered
application, and the module barrier. Reduce avoidable within-stage waiting
without introducing independent whole-pipeline chunk traversal. Retain completed
proposal evidence on interruption even when an earlier section is unfinished.
Preserve section- and response-ordered serialized proposal indices. If
measured waiting warrants earlier validation, use local candidate references
during processing and assign serialized indices at ordered collection.
PromptKit's configured rejection remains valid; the old scheduling-dependent
choice of rejected chunks is not a compatibility requirement.
- Retain only the state needed by active processing and selected outputs. Keep
live validation/repair state, compact summary counts, and required failure
evidence. Avoid full debug histories or work-unit matrices in summary-only
paths; preserve complete detail when that output is enabled. Remove duplicate
operation arrays, unused module totals, and obsolete response/callback copies.
This is not a constant-memory promise: selected outputs and useful conversation
history can still grow with the workload.
- Carry typed class, reason, and cause through proposal, validation, repair, and
reporting. Preserve the original class/cause if a missing reason uses the
internal-failure fallback. Distinguish parent cancellation, logical-operation
timeout, and HTTP-attempt timeout using native classifications; an HTTP-only
timeout uses the existing transport-failure reason.
- Share ordinary-call and coordinator-turn completion, retaining native terminal
counts and diagnostic evidence from undelivered responses. Remove the test-only
direct-continuation branch after moving useful coverage onto production
coordination. Keep one proposal handoff with raw local references; the runner
owns global indexing. Review emitted metadata before deleting fields that only
appear unused to a function-level reference search.
- Give each repair branch an explicit execution/result boundary. Preserve native
conversation coordination, full history, per-correction limits, sibling
independence, withdrawal, and validation of the final permitted revision.
Pass domain validator explanations directly instead of applying credential
substitution; retain the distinction between editorial rejection and unavailable
assessment. Measure aggregate work to guide efficiency work, without adding
new stopping rules or ordinary-report fields for that analysis.
### Completion evidence
Controlled overlap and cancellation tests exercise actual dispatch, admission,
result draining, and stage commit paths. They establish native shared backend
capacity, useful evidence retention, and deterministic identities under observed
out-of-order completion. Distinct live timeout budgets exercise the correct
failure owner. Representative measurements demonstrate reduced pending work and
retained representations; they do not impose a new throughput target or require
every chunk to obtain admission. Repair tests run through the production
coordinator and preserve accepted feedback and revalidation behavior.
## 3. Faithful transcript and correction semantics
### Target state
Transcript parsing preserves accepted values, normalization preserves time and
ordering meaning, and chunking satisfies feasible configured constraints.
Validation and application use consistent domain evidence. Efficient shared
operations reduce repeated work without changing editorial acceptance rules.
The baseline's generic JSON shape conversion can round IDs; normalization and
chunking have concrete interval, ordering, and feasibility counterexamples.
Some protected-term checks use only edit spans, and validation response conversion
can discard usable decisions because of irrelevant extra references.
### Work to reach the target
- Detect the supported transcript envelopes while keeping their segment array as
raw JSON for typed decoding. Preserve optional IDs, accepted numeric ranges,
unknown-field behavior, and both input shapes. Share live source-to-canonical
conversion, retire obsolete strict/lenient wrappers, and release whole-input
representations after their last use. Keep accepted formats and stream inputs.
- Use the full merged time interval for merge eligibility, output, and subsequent
gaps. Preserve source order for tied timestamps without introducing a new
end-time sorting criterion. Build growing text/category state incrementally,
preserving exact joins, category order, and the current token estimate.
- Satisfy feasible explicit chunk count and maximum-size constraints before
optimizing balance. Preserve the single-segment exception above the token
maximum. Use a proportionate deterministic algorithm and retain the
configured budgets and current balancing objective. Keep accepted
compatibility settings, including `chunking.min_section_tokens`, even where
they are not active partition targets.
- Give validator applicability and decision meaning one owner and one runtime
contract. Reject contradictory internal outcomes before they advance a
candidate; treat them as internal contract failures, not unavailable model
assessments. Derive redundant approval representation where needed. Preserve
decision cardinality, zero-budget shortlisting, independent repair feedback,
and unavailable outcomes. Reconcile exact numeric response references before
local conversion, accepting integral JSON number spellings without float
rounding and ignoring irrelevant extras. Keep whole-response structural
validation and existing partial-success behavior.
- Share glossary-owned protection assessment using usable before/after segment
evidence, and recheck current text at application. Preserve active-glossary
exemptions, case/longest-match behavior, and spoken-word repetition rules.
Missing or ambiguous previews cannot support a fabricated protection verdict;
independently supported applicability findings remain valid. Empty-result
previews can still supply protection evidence without authorizing whole-segment
removal. Handle configured punctuation-edged spellings with standard primitives
while accepting Go's existing word-boundary semantics.
- For require-unique replacement, detect a second possible literal starting
offset even when matches overlap. Preserve replace-all's existing nonoverlapping
behavior. Share request-local segment lookup across initial and repaired
proposals, retaining distinct reference namespaces, raw invalid references,
and retargeted revalidation.
- Reuse glossary identity membership for context selection rather than expanding
every occurrence/contributor pair; keep occurrence evidence for protection.
Avoid validation-context work once the usable budget is settled, sharing
trimming/batching policy while preserving context priority, oversized-input
handling, and an independent serialized measurement reference. Reuse section
lookups and aggregate skipped-edit evidence once, distinguishing stale
application conflicts from response filtering.
### Completion evidence
Boundary regressions reproduce the audited numeric, nested-interval, tied-order,
feasible-chunk, overlapping-match, and full-segment-protection cases. Validator
tests pass actual structured responses through decoding and reconciliation, with
usable siblings and irrelevant extra references. Shared optimized operations
are checked against independently constructed expectations, including exact
estimates and representative allocation/work measurements. Existing editorial
acceptance, repair, and apply-time rules continue to govern results.
## 4. Smaller components with explicit ownership
### Target state
Preflight and execution share resolved application decisions. Domain models own
their conversions and copies; adapter boundaries own external effects; the
composition boundary assembles built-ins. Modules and validators remain distinct
components with documented concurrent-call and borrowed-data responsibilities.
The baseline repeats root/stage configuration rules, readiness selections,
transcript projections/clones, token estimates, and shared prompt-variable work.
Some interfaces and helpers survive only through tests or unused wrappers.
### Work to reach the target
- Consolidate Audita-owned root/stage validation and fallback rules after
ConfigKit handoff, retaining null, explicit zero, inheritance, provenance,
stage-dependent defaults, and diagnostic paths. Carry checked processing
selections from preflight into execution. Reuse effective profile/glossary
views within the captured runtime, with selection order and exemptions in
reuse keys; retain first-failure attribution.
- Share transcript cloning at the model boundary, preserving nil versus explicit
empty mutable slices and the separate caller ownership boundaries. Give public
encoders one segment projection while retaining both envelopes, schema/version
behavior, categories, omission rules, and numeric fidelity. Use one implementation
of the existing token estimate across consumers, retaining its exact results.
- Assemble built-in modules at composition, preserving the runner factory seam
and a dependency-light configuration catalog. Document concurrent-call and
borrowed-input obligations at module/validator ports; permit synchronized
stateful implementations and use owned copies where mutation is needed.
- Merge shared prompt variables once per model call, preserving the different
processing/inspection precedence, protected-term default, and operator prompt
replacement support. Remove dead helpers, unused envelopes, redundant adapters,
and abandoned test support after checking live registrations and serialization.
Move useful tests to the surviving behavior before removing test-only paths.
These changes complement the writer, validator, operation, and repair ownership
work above. Use small contracts and helpers where they serve an actual shared
responsibility; package layout and private type names are implementation choices.
### Completion evidence
Real consumers use the shared owners and obsolete paths are removed. Behavioral
tests establish copy independence, both output envelopes, effective configuration
semantics, native integration, and processing/inspection precedence. Refactoring
does not depend on a new configuration loader, plugin framework, cross-invocation
cache, or a generic replacement for the existing domain components.
## 5. Actionable errors and detailed processing progress
### Target state
Operators can identify the failed phase and relevant setup requirement, and
follow long jobs with meaningful progress. Standard inspection and processing
remain integrated with CommandKit rather than acquiring parallel command or
reporting systems.
The baseline hides some native setup detail, classifies some operator-correctable
failures as runtime failures, and exposes progress without producing processing
events. The current [CLI contract](../cli.md) supplies the stream and exit-status
boundary that the fixes should follow.
### Work to reach the target
- Classify invalid processing selections and genuinely infeasible chunk requests
according to the existing input-error contract; retain runtime classification
for I/O and cancellation. Trace delegated inspection failures to Audita's
loader or the upstream integration. The pinned ckprompt catalog handler
classifies malformed prompt configuration as an operational failure. Fix
that classification upstream while keeping native command routing.
Processing progress can use CommandKit's existing reporter independently.
- Surface actionable native setup detail, including the configured missing
credential requirement and available role/profile context, while preserving
typed causes and failure classification. Remove CLI substitution of errors,
warnings, and announced paths. Keep compact ordinary-report projections where
they serve the report schema; deleting substitution is not a request to expand
every report with raw provider causes.
- Emit relatively detailed stage/chunk, proposal, validation, and repair activity
through CommandKit's reporter, using actual lifecycle state and available
upstream execution events. Include useful work identities and observed
completion counts. Do not restrict `--progress` to coarse stage summaries just
to minimize output. CommandKit determines effective reporting-mode precedence
and routes output, including when flags are combined; stdout remains available
for the transcript. Event granularity and wording are implementation choices.
- Document the trusted configuration bundle, imports, and automatic companions
for operators and subprocess integrations. Explain how caller-owned report
destinations and artifact retention fit the workflow. Preserve native discovery
and inspection; these clarifications do not add an isolation mode.
### Completion evidence
CLI tests check the appropriate exit classifications, useful diagnostic context,
native mode combinations, and unchanged transcript stdout. Progress tests exercise
real processing activity and meaningful updates without fixing incidental wording
or concurrent event order. Update user-facing documentation alongside the feature.
## 6. Verification and delivery that support change
### Target state and work
Tests protect consequential behavior at its owning boundary and make valid
refactoring and prompt iteration inexpensive. Apply the
[testing policy](../policy/testing.md) throughout the work, rather than adding a
second set of implementation-shaped assertions at the end.
- Exercise representative signal, cancellation, and exit behavior through the
real executable or a shared production entry point. Keep smaller CLI fixtures
for cases that do not need a process boundary. Use observed overlap and
independently controlled deadlines for concurrency/timeout tests.
- Verify registered identities, effective validator roles, embedded-asset
preparability, schema identity, and protective copies through behavior. Replace
incidental registry counts, duplicate normalization fixtures, and tests of test
helpers with coverage of their actual consumer risks. Assert decoded artifact
identities and association; keep a representative repair/filter/later-failure
integration scenario and smaller independent-output cases.
- Remove redaction-only tests and fixture modes while retaining authentication,
serialization, evidence, and failure coverage. Avoid prompt-wording change
detectors, broad incidental snapshots, and new checks that recreate withdrawn
guarantees. The default suite remains offline and uses synthetic credentials.
- Fix the CI bootstrap so failure to run its Git cleanliness check cannot be
interpreted as a clean checkout. Test that command-status boundary and remove
redundant tests of the release test's comparison helper. Keep ReleaseKit's
selected checks and the existing [release policy](../policy/release.md); this
roadmap introduces no new release gates.
- Update canonical CLI, configuration, operations, integration, and internal
documentation with the corresponding changes. Remove stale redaction promises
and recipes without adding a negative architecture assertion. Keep native
configuration-rendering documentation and historical audit evidence. Use ADRs
only where the [documentation policy](../policy/documentation.md) calls for a
significant architectural record, not for every helper or audit entry.
### Completion evidence
Run focused regression and boundary tests with each change, the full offline
suite for substantial integration changes, and applicable existing candidate
checks at release preparation. Use targeted race checks for changed shared-state
paths and representative measurements for resource changes. Reuse successful
checks for unchanged content. Link/example/whitespace checks accompany relevant
documentation changes. Do not substitute coverage percentages or a collection
of test counts for evidence of the selected outcomes.
## Completion and audit coverage
The [implementation plan](implementation.md) records the landed changes and
verification. The coverage map accounts for every catalog entry, including
closed and superseded remedies. The [assessment](audit/assessment.md) records
which remedies were selected.
| Target area or disposition | Catalog entries |
| --- | --- |
| §1: checked output, truthful finalization, caller destinations | 1, 6, 7, 8, 16, 42 |
| §1: serialization, diagnostic ownership, and useful evidence | 3, 9, 22, 26, 27, 33, 43, 63 |
| §2: stage/operation lifecycle, scheduling, and retention | 14, 15, 23, 30, 32, 34, 41, 44, 45, 46, 48, 64 |
| §3: transcript fidelity, normalization, and chunking | 13, 17, 20, 21, 31, 35, 51 |
| §3: decision semantics, protection, references, and context | 10, 11, 12, 18 (punctuation portion), 19, 36, 38, 39, 40, 47 |
| §4: shared selections, models, configuration, and composition | 37, 49, 50, 52, 53, 54, 55, 56 |
| §5: operator guidance, errors, and progress | 5, 24, 25, 57, 65 |
| §6: verification, delivery, and dead machinery | 28, 29, 58, 59, 60, 61, 62 |
| Accepted Go limitations; no workaround | 2; 6 (interrupting blocked I/O); 18 (Unicode word-boundary emulation) |
| Withdrawn private-debug-default remedy; current defaults retained | 4 |
Cross-cutting test and documentation work belongs with each implementing change.
The [assessment](audit/assessment.md) remains the detailed boundary for mixed
entries. All seven findings in the focused [removal audit](audit/14-redaction.md)
are covered by the evidence, repair, CLI, and verification sections above.

View File

@@ -1,523 +0,0 @@
# Reliability and Maintainability Review Remediation
**Reviewed revision:** `e8c0708`
**Status:** All fourteen findings addressed after review of the 20
[implementation stages](implementation.md) against the
[target roadmap](reliability-and-maintainability.md), including follow-up
audits of component interactions and algorithmic costs.
## Review Result
The sprint established the target architecture. ConfigKit and native
CommandKit integrations handle configuration and inspection, and preflight
hands checked processing selections to the runner. Audita decodes and
normalizes transcripts directly and partitions them once. A bounded section
window allows proposal, validation, and repair to overlap within a stage.
The runner assigns final proposal indices in section order and commits edits
at the stage boundary. PromptKit owns model execution, backend admission,
retries, and repair history. Required publication uses checked destinations.
Ordinary reports use typed processing facts, while optional debug capture
retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks
for the runner, LLM adapter, validators, report projection, diagnostics,
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
under the race detector. A built CLI returned status 2 with source diagnostics
for malformed PromptKit and application configuration across the delegated
commands and processing cases in the plan. Focused review probes reproduced
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in
validator chains, model execution, publication, and report and debug readers.
Additional probes reproduced the case-variant input behavior in R4 and the
cross-component failures in R6–R8.
The algorithmic audit kept the selected stage window, full repair history,
and chunk-partitioning algorithm. The normalization merge builds text and
categories incrementally after sorting, and the explicit partitioner advances
through the weights in linear passes. The remaining opportunities are
primarily in repeated validation and glossary preparation, queued debug
snapshots, and evidence retained after it has served its purpose. R9 was
reproduced through the PromptKit role adapter with a synthetic provider.
R10–R14 follow from source analysis. Focused offline tests passed for
normalization, chunking, preflight, validators, the runner, and reports.
## Findings
### R1 — Include Later Delivery Failures in Writable Reports (P2)
**Complete.** Later report writes include newly observed delivery failures.
One best-effort pass updates earlier reports with stale evidence. Diagnostics
tests cover both initially failed runs and failures that arise during
publication of a completed run.
The follow-up fix counts repeated write failures for one report as one
delivery failure and retains both causes. The regression makes both writes to
`recoveries.yml` fail and verifies that the surviving reports agree on the count.
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
renders a failed bundle using the failures accumulated at that point. Later
writes append to the returned error list, but the remaining reports and the
best-effort pass reuse the same bytes. The
[report projection](../../internal/framework/processreport/failures.go#L41)
can include secondary file failures only if publication supplies them.
**Reproduced behavior:** Start finalization with a processing failure. Make
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
**Required change:** Use the freshest observed failures at the report
publication boundary and make a bounded best-effort update of already-written
reports when their failure evidence becomes stale. Retain existing evidence
if rendering the updated failed bundle fails. Keep transcript-first,
nontransactional publication.
**Acceptance:** Extend the output-session regression to inspect the facts
rendered into a surviving report when an earlier failure is followed by a
second write failure. Check failure attribution and counts as well as status.
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
**Complete.** Version 1 retains the `work_unit` and `run` scopes. The
initiating run record carries available peer details in an optional
`interruptions` sequence. The mixed-run YAML and Markdown test verifies
peer attribution and unchanged counts.
**Affected plan:** Stage 17, item 1.
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
adds a top-level `scope: interrupted` record to the
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
The baseline contract described `work_unit` and `run` records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
also preserves the documented output contract for minor and patch releases.
**Required change:** Keep the existing top-level variants. Carry available
peer interruption details as optional evidence on an existing record, for
example an optional sequence on the initiating run failure. Keep the primary
origin and peer origins separately identifiable and preserve the work-unit and
run-error counts. Treat a new top-level variant as an explicit contract change
if that representation is preferred.
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
Check that a consumer of the existing record variants can still read the
report, and that the peer's available phase, reason, and origin remain present.
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
**Complete.** The operation and run report use the same typed reason fallback.
The regression covers a missing reason with a wrapped deadline, retained
operation origin and cause, and distinct parent cancellation.
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
uses `internal_failure` when a typed model failure has no reason and retains
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
**Reproduced behavior:** A typed fatal failure with an empty reason and a
wrapped `context.DeadlineExceeded` is recorded on the operation as
`fatal` / `internal_failure` and serialized in the run report as
`canceled`. The cause is available, but it does not make this a parent
cancellation.
**Required change:** Share the typed reason projection and its empty-reason
fallback. Derive a reason from the raw cause only when a typed failure is absent.
**Acceptance:** Verify matching fallbacks for the operation and the run record
and retain the phase, origin, and original cause. Keep genuine parent
cancellation distinct.
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
**Complete.** Each segment resolves duplicate keys before field conversion
and processes case variants in the baseline key order. The decoder keeps
numeric tokens intact. Schema tests cover superseded and effective type
mismatches, both case-variant orders, and exact IDs in both envelopes.
The follow-up restores direct typed decoding for ordinary segment arrays.
Arrays with conflicting or escaped keys use compatibility decoding where
needed. Both envelopes were verified with mixed segments. On the same
100,000-segment fixture, median parse time fell from about 0.83 to 0.30
seconds and allocation fell from 162 to 78 MB per parse.
**Affected plan:** Stage 5, item 1.
**Evidence:** The baseline selected the final value of an exact duplicate JSON
member when it decoded the segment into a generic object, before applying
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
now applies the field type to every occurrence. An earlier type error survives
even when the final value is valid. The refactor was meant to retain accepted
inputs while improving numeric fidelity.
**Reproduced behavior:** The baseline at `d226bdd` accepts
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
speaker `A`. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing `"text":"expected utterance"` followed
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
normalization. The new decoder passes `other metadata`. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
**Required change:** Preserve the effective-member behavior for existing
inputs without reviving whole-transcript generic decoding or float conversion
for IDs. Keep the compatibility handling focused on this changed boundary.
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
in the effective member, plus the case-variant example in both member orders.
Keep the existing exact-ID and unknown-field tests for both input envelopes.
### R5 — Align Roadmap Status and Runtime Guidance (P3)
**Complete.** The roadmap points to the landed implementation and follow-up
review. The pipeline guide describes available peer evidence in reporting mode.
The CLI guide renders the preflight sentence normally.
**Affected plan:** Stages 15, 19, and 20.
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
describes the implementation as pending despite the completed sprint.
The [pipeline guide](../internal/pipeline.md) says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
literal `\n` between “selected” and “profiles.”
**Required change:** Mark the roadmap with the landed sprint and this review,
while retaining its historical baseline context. Describe interruption detail
in terms of outcomes actually available and selected for retention. Fix the
CLI sentence and keep public guidance focused on the operator-visible behavior.
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
reflects the evidence modes, and the CLI paragraph renders normally.
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
**Complete.** Progress is emitted before and after each timed model call,
including coordinated repair turns. An integration regression delays stderr
beyond the model deadline and verifies the same provider calls and correction.
Live progress, reporting modes, and presentation failure tests also passed.
The follow-up checks the parent context before each progress notification.
Adapter tests cover calls canceled before progress starts and calls canceled
during the start notification. The CLI progress and cancellation tests passed.
**Affected plan:** Stages 12 and 19.
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
calls [progress rendering](../../internal/cli/process_progress.go#L34)
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
PromptKit v0.23.1 starts the operation deadline before emitting the start
event, so a blocked or slow write to CommandKit's diagnostic stream spends
that operation's budget before the provider request starts.
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported `operation_timeout`, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
**Required change:** Deliver progress outside PromptKit's timed model-call
observation path. Keep native observations for execution facts and use
application lifecycle boundaries for synchronous progress delivery, or another
delivery mechanism that keeps stderr waits out of the per-operation budget.
Preserve CommandKit's routing and presentation status, and keep the parent
context authoritative for cancellation of the invocation.
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
Verify that it still dispatches and applies the same correction within the
model's own budget, and retain live progress and presentation-failure coverage.
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
**Complete.** Processing uses the native configuration error classifier and
retains source diagnostics and typed causes. Tests compare processing,
catalog listing, and configuration inspection for malformed content,
permission failure, and caller cancellation.
**Affected plan:** Stages 1, 11, and 20.
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing [process entry point](../../internal/cli/process.go#L27)
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
unconditionally returns a usage outcome for those errors.
**Reproduced behavior:** A readable main file beside an unreadable automatic
`prompts.yml` produces `source.permission` in all three commands.
`promptkit list prompts` and `config show --scope promptkit` return status 1.
`process` returns status 2 and prints usage guidance for the same loading
failure.
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
**Acceptance:** Compare `process` and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
### R8 — Use the Same Span Applicability With and Without Repair (P3)
**Complete.** The original-text validator rejects ambiguous require-unique
spans at either budget. A built CLI regression verifies local attribution,
no protection approval or model validation for the span, and repair only
at the positive budget. Whitespace and blank-result tests also passed.
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
rejects an ambiguous require-unique span only when independent repair evidence
is enabled. Without repair, the existing shortlist forwards the candidate to
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
which records approval even when it cannot obtain a usable preview. The
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
later rejects that same local ambiguity before dispatching a provider call.
**Reproduced behavior:** In the built-in grammar stage, offer to replace
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
protected. With a zero repair budget, the recorded decisions show protection
approval and `editorial_review` rejection for `ambiguous_original_text`.
With repair enabled, `original_text_presence` records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
**Required change:** Give the local applicability check ownership of
require-unique ambiguity in both paths. Keep independent evidence and repair
attempts conditional on the configured budget. Retain the ability to assess
protection when a located edit produces an empty preview.
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
Verify consistent local attribution, no unsupported protection approval,
and no model validation call for the inapplicable candidate. Keep repair
available in the positive-budget case.
### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
**Complete.** Proposal and repair references use the same exact integer
decoder as transcript IDs. The CLI conversation test covers a usable sibling
and a repair revision with decimal or exponent references. The repair test
continues to reject a mismatched reference.
**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9
already handles the corresponding validator references.
**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20)
and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9)
specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21)
and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24)
use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184)
classifies a conversion error after schema validation as recoverable invalid
output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123)
then returns an operation failure with no candidates.
**Reproduced behavior:** A synthetic provider returned two proposals with
references `1` and `2.0`. PromptKit accepted the response in one request with
zero structural repairs. Audita returned `invalid_output` at typed decode.
The control case with `1` and `2` decoded both proposals. The repair decoder
has the same conversion for `proposal_ref` and the revision's `id`.
**Required change:** Decode wire references as exact numeric tokens and
reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9)
before local position or correspondence checks. Preserve the distinction
between segment and proposal references, existing out-of-range handling, and
the original response in debug capture.
**Acceptance:** Exercise the actual structured paths with an integer and an
equivalent decimal or exponent form. A valid sibling survives proposal
decoding, and the repair revision reaches the same revalidation as its integer
form. Retain existing handling for a mismatched repair reference.
### R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
**Complete.** Fitting checks required data before loading glossary and
neighbor context. Context selection reuses source matches within a validator
request and matches each revision independently. Tests verify skipped loading,
candidate-specific context, and equivalence with the serialized budget oracle.
**Affected plan:** Stages 8 and 10.
**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251)
builds optional context for every previewable candidate before
[fitting the payload](../../internal/framework/validators/llm_budget.go#L50).
[Context construction](../../internal/framework/validators/llm_context.go#L39)
rescans the original and adjacent segments, finds matches in the proposed
text, sorts matched entry positions, and builds the entry slice. The fitter
can discard all optional context while retaining the original slice's
backing array on a fitted item.
**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B`
bytes of shared source and neighboring text, the unchanged text is scanned
`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate
add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work
happens even when required fields consume the budget. Distinct proposed texts
still need their own match evidence when optional context is used.
**Recommended change:** Fit the required fields before collecting optional
context for a candidate. Reuse identity membership for unchanged text within
the section and combine it with the corrected-text matches only when needed.
Keep existing glossary and neighbor order, exact serialized budget accounting,
and oversized-input behavior. Drop unused optional storage before dispatch.
**Acceptance:** Compare the emitted payload to the existing serialized-budget
oracle for usable and oversized candidates. A controlled no-room case should
skip optional matching. Sibling candidates should share unchanged-text lookup
while still receiving definitions specific to their revisions.
### R11 — Reuse Glossary Selections Before Building Their Views (P3)
**Complete.** Stage resolution shares read-only selected entries and
protection terms before preparing the views. Both use the same ordered
selection key. Preflight and runner tests cover repeated selections,
ordering, glossary-stage exemptions, and stage-specific execution.
**Affected plan:** Stage 11, item 2.
**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115)
clones selected glossary entries and aliases and derives protection terms for
every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152)
can reuse prompt variables and validator views only after all those copies
have been built. The retained specs still hold separate entry and term slices
for identical selections.
**Cost:** With `S` stages sharing a selection of `G` entries and spelling
contributions, materializing and retaining those selection data costs
`Θ(S × G)`. Sharing immutable selection data brings this portion to
`Θ(G + S)` for repeated fixed-length selection keys.
**Recommended change:** Resolve and prepare the shared glossary data on the
first occurrence of an ordered context/protection selection and exemption.
Reuse the complete read-only selection in later specs. Keep stage identity,
first-failure attribution, and copy-on-mutation at consumer boundaries.
**Acceptance:** Exercise repeated selections, a reordered selection, and a
glossary-stage exemption. Reused selections should supply the same data to
proposal, validation, and application as the existing independent views.
### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
**Complete.** Pending repair requests share their immutable history
prefixes. Each write retains the exact request and final proposal index.
An ordered collector test compares two real coordinator snapshots with the
written requests, checks ledger paths, and verifies that a later request
still writes after a capture failure. Repeated race checks passed.
**Affected plan:** Stages 3, 14, 16, and 18.
**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151)
converts every turn's full rendered message list to a new slice. Each
[repair builder](../../internal/framework/runner/repair.go#L194) captures that
snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22)
holds the builders until ordered proposal indices are assigned, including
during earlier sections' model work.
**Cost:** A branch with `H` starting messages and `T` successful turns that
add a constant number of messages per turn can retain `Θ(T × H + T²)` message
descriptors at once. Content strings may share backing storage. The window
bounds the number of sections held, and a single section can still contain
many turns and branches. Full-history requests and self-contained debug files
require repeated serialization by design. They can be serialized one at a
time without keeping every prepared list live.
**Recommended change:** Keep compact evidence pending ordered identity
assignment. For example, stage the request payload under section-local
coordinates while it is live and use its final name when indices are assigned.
An in-memory approach can share immutable history prefixes and expand each
snapshot at write time if it can reproduce the exact rendered messages.
Retain full model context, nonfatal capture, and self-contained final files.
**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section.
Compare its decoded request bodies, final artifact names, and ledger paths
with the current representation, including a capture failure followed by a
successful write.
### R13 — Retain Only Operations Used by Ordinary Reports (P3)
**Complete.** After totals and application attribution are settled, the
runner keeps only reportable errors and recoveries. Successful proposal
index lists and per-operation causes no longer persist through the run.
Runner, report, and CLI tests cover totals, filtering, interrupted origin,
and evidence across output modes.
**Affected plan:** Stages 12, 17, and 18.
**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632)
counts each work unit and retains all its operations when reports are enabled.
It also [stores every candidate index](../../internal/framework/runner/runner.go#L548)
on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79)
uses only recoverable failures and successful recoveries. Failure projection
uses affected indices and typed facts. Recovery projection uses counts and
filtered evidence. Neither uses the cause stored on each operation, which is
needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22).
**Cost:** The retained operation records, unused successful proposal index
lists, and causes account for `Θ(M + P + B)` storage, for `M` operation
records, `P` successful proposal indices, and `B` bytes of retained causes.
PromptKit's non-success HTTP error can include a bounded
provider diagnostic in each cause. Even an all-success run with no recoveries
keeps and later sorts operation rows that produce no error or recovery entry.
The report matrix itself still needs a row per work unit.
**Recommended change:** After counting and resolving application filtering and
failure attribution, retain compact facts only for operations that feed the
reports. Keep typed initiating and peer causes where they belong on execution
failures. Debug execution observations and correction-ledger inputs have
their own consumers. Avoid retaining proposal index lists on operations that
never project those lists.
**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals,
and debug association across existing evidence modes. Include clean successes,
both initial and repair filtering, and an interrupted run with available origin.
### R14 — Reuse Original Applicability When Attributing Application Skips (P3)
**Complete.** Application still checks current text. Skip attribution
reuses known original applicability, and repair filtering indexes approved
operations once per affected unit. Tests distinguish stale conflicts from
original defects and verify attribution across multiple repairs.
**Affected plan:** Stages 8 and 17, item 3.
**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59)
checks each proposal against evolving segment text. On a defective-looking
skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953)
previews the proposal again against the original section to distinguish a
stale conflict from a defective response. The built-in validation path has
already determined the latter preview's applicability. Its result is discarded
before the attribution pass.
**Cost:** For `K` conflicting proposals against one original segment of `B`
bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of
original-text previews, even if application quickly rejected those proposals
against the changed text. This is separate from the checks required against
current text. When a validator chain can approve originally inapplicable
revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46)
can also scan `O(F × M)` operations for `F` such filtered revisions and `M`
operations in the same unit. Built-in validators normally reject those revisions
before application.
**Recommended change:** Carry a compact original-applicability fact for
approved candidates when the validator already computed it. Recompute only
when no such fact is available. If the repair attribution branch needs changes,
index or group originating repair operations once per affected unit. Preserve
ordered application, current-text protection checks, and the distinction
between response filtering and staleness.
**Acceptance:** Keep both a stale-conflict case with initially valid proposals
and an originally inapplicable approval from a controlled validator. Verify
the skip, correction counts, and filtering or recovery attribution.