diff --git a/docs/migrations/v3.md b/docs/migrations/v3.md index cf4c77a..4b59d27 100644 --- a/docs/migrations/v3.md +++ b/docs/migrations/v3.md @@ -1,7 +1,7 @@ # Migrating to application configuration v3 -Audita 3.1.x uses application configuration schema v2. For the cutover to schema -v3, set `version: 3` in the application file and move glossary entries into +Audita v4.0.0 adopts application configuration schema v3. Audita v3.1.x uses +schema v2. Set `version: 3` in the application file and move glossary entries into configuration. Use the [configuration reference](../config.md) and [maintained examples](../../examples/) for the new formats. diff --git a/docs/releases/README.md b/docs/releases/README.md index 72520c6..b7c8752 100644 --- a/docs/releases/README.md +++ b/docs/releases/README.md @@ -1,5 +1,6 @@ # Release Notes +- [Audita v4.0.0](v4.0.0.md) - [Audita v3.1.1](v3.1.1.md) - [Audita v3.1.0](v3.1.0.md) - [Audita v3.0.1](v3.0.1.md) diff --git a/docs/releases/v4.0.0.md b/docs/releases/v4.0.0.md new file mode 100644 index 0000000..164236d --- /dev/null +++ b/docs/releases/v4.0.0.md @@ -0,0 +1,89 @@ +# Audita v4.0.0 + +Audita v4.0.0 adopts application configuration schema v3 and native library +commands, adds repair of rejected corrections, and improves processing +reliability and resource use. This major release requires migration from +Audita v3.1.x configurations and command invocations. + +## Configuration and command migration + +- Replace `pipeline.modules` with ordered `pipeline.stages` records. Each entry + can select its own profiles, glossary context, protected terms, validation + budget, and transcript description. Optional stage IDs identify repeated + modules in reports and debug evidence. +- Move standalone glossaries into named `glossary_collections`, composed through + ConfigKit imports and companion files. Select glossary context and protected + terms independently. Replace `--glossary` and `--modules` invocations with the + new configuration and collection-selection flags. +- Move profiles and backends into native PromptKit resources. Application + configuration discovery, overlays, imports, and prompt companions now follow + the adopted library conventions. Processing preflight checks the entire + selected pipeline before preparing outputs or invoking a model. +- Replace `config print-effective` and the curated profile summary with native + configuration and PromptKit inspection commands. Help lists the available + inspection, comparison, preview, validation, and explicit provider smoke + commands. +- Remove confidence thresholds and confidence fields from custom prompts and + response examples. Audita now decides whether to apply corrections through + validation without model-supplied confidence scores. +- Update automation for CommandKit reporting modes. Completion counts require + `--verbose` or `--debug`. Quiet processing requires a transcript file + destination or `/dev/null`. `--debug` enables retained diagnostic artifacts; + `--progress` shows stage, chunk, proposal, validation, and repair activity on + stderr. + +Follow the [configuration migration guide](../migrations/v3.md) and +[maintained examples](../../examples/), then inspect the resolved configuration +and try a small transcript. The migration guide's v3 refers to the application +configuration schema. The [CLI reference](../cli.md) owns current commands and +exit semantics. + +## Correction quality and recovery + +Rejected proposals can be revised using validator feedback and the original +proposal conversation. Correction repair defaults to one attempt per proposal, +can use a separate profile, and can be disabled. Every revision passes through +validation again. Withdrawn, exhausted, and unavailable branches finish +independently while other approved corrections can proceed. + +Prompts clarify module focus, incidental editorial cleanup, protected names, +span deletion, and repair outcomes. Missing required validator decisions continue +to affect only their candidates. Recoverable model failures can coexist with a +successful run and applied corrections. See [configuration](../config.md#settings) +for budgets and profile selection. + +## Reliability and reports + +Processing uses a bounded window of sections within each stage, preserves +deterministic correction identities, and commits stage text after successful +execution. Fatal failures and cancellation stop and join active workers. +Transcript decoding, normalization, chunking, glossary preparation, and evidence +retention use less repeated work and fewer unnecessary copies. An explicit +target section count now produces exactly that many nonempty sections when +feasible and reports infeasible requests as input errors. + +Output publication uses checked destinations and retains processing and delivery +failure evidence. Ordinary reports retain schema version 1 and add stage IDs, +correction repair outcomes, and available peer interruption details. See the +[output contract](../integrations/output-files.md) and +[operations guide](../operations.md) for report interpretation and output +lifecycle. + +Audita no longer performs application-wide credential substitution in diagnostic +messages or artifacts. Review diagnostic handling before deployment and protect +retained debug data according to the sensitivity of its transcript, prompt, and +response content. Native library inspection retains its own rendering behavior. +Debug artifacts have changed, including removal of confidence fields and the +Audita-owned utilization diagnostics file. + +## Dependencies and source builds + +Source builds require Go 1.27.0 or newer. Release builds and Woodpecker use +Go 1.27.4. Distributed Linux binaries require no Go installation. + +This release selects ConfigKit v1.2.1, PromptKit v0.23.2, CommandKit v0.7.0, +ckconfig v0.9.0, ckprompt v0.10.0, ReleaseKit v0.10.0, and stable go-yaml v4.0.0. +Audita's direct YAML imports now use v4. Linux amd64 and arm64 remain the +production targets; macOS arm64 receives best-effort development support. + +[COPYRIGHT](https://gitea.maximumdirect.net/eric/audita/src/tag/v4.0.0/COPYRIGHT) | [LICENSE](https://gitea.maximumdirect.net/eric/audita/src/tag/v4.0.0/LICENSE) | [ATTRIBUTION](https://gitea.maximumdirect.net/eric/audita/src/tag/v4.0.0/docs/oss/attribution.md) diff --git a/docs/roadmap/audit/01-correctness.md b/docs/roadmap/audit/01-correctness.md deleted file mode 100644 index c4b46bd..0000000 --- a/docs/roadmap/audit/01-correctness.md +++ /dev/null @@ -1,261 +0,0 @@ -# Functional correctness audit - -Reviewed source at `c4ccd78`. All findings below were reproduced through the -built CLI with temporary files or a synthetic loopback provider. The existing -offline suite also passed with Go 1.27.3 using -`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`. -The provider fixtures exercised Audita's handling of scripted decisions. -Live-model editorial quality was outside the review. - -## Architecture and review scope - -Audita is a single-process Go CLI. CommandKit handles commands and streams, -ConfigKit composes configuration, and processing preflight checks role profiles, -glossary selections, and output destinations. Core packages parse and normalize -the transcript, derive glossary views, and partition the normalized segments into -fixed chunks. The framework runs configured correction modules in order while -sections can generate proposals concurrently. Deterministic checks and LLM -validators gate application. PromptKit handles the provider and continuation -history for bounded repair branches, and Audita revalidates revisions. The output -session publishes the transcript and reports, with opt-in diagnostics capturing -request, decision, and execution evidence. [Package map](../../internal/overview.md) -· [Processing sequence](../../internal/pipeline.md) - -The review traced configuration and readiness into file handling, transcript -decoding and normalization, chunk feasibility, glossary matching, proposal -preview and application, validator batching and response reconciliation, -repair outcomes, cancellation, and ordinary and debug reporting. It checked -representative success paths alongside malformed inputs, boundaries, recoverable -failures, and interrupted work. - -## 1. Output path resolution can replace a different file - -**Expected.** A regular transcript should be published at the selected output -destination. Collision checks and publication should agree on which file that -path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle) - -**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves -`logical/link/../out.json` to `physical/out.json`. With two distinct existing -files and reporting disabled, Audita exited 0, left `physical/out.json` untouched, -and overwrote `logical/out.json`. `Prepare` stats the supplied path, but -`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving -symlinks. The cleaned path is then used for publication. The same helper feeds -collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141) -· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377) - -**Remediation.** Resolve directory components with filesystem traversal -semantics before collapsing `..`. Use the resulting destination for collision -checks, regular-file preparation, and publication. Derive the default report -directory from the same destination when file output is selected. - -## 2. Glossary matching misses names with accented letters or punctuation at their edges - -**Expected.** Selected protected names and aliases should participate in both -deterministic protection checks. Matching names should also be available to -validators when building context. The glossary accepts ordinary strings for -names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections) -· [Validator context](../../internal/validators.md#supplemental-validation-context) - -**Observed.** With `Élodie` or `C++` in the selected context and protection -collection, scripted grammar proposals replaced the names in `Élodie arrived.` -and `Use C++ for this.`. Both edits reached model validation and were applied -after scripted approvals. The validator requests also omitted the corresponding -glossary entries. The shared matcher surrounds every alternative with Go -regular-expression `\b`, whose word character class is ASCII. At the start of -`Élodie` and after the final `+` in `C++`, the required boundary fails. -[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30) -· [Context matching](../../../internal/framework/validators/llm_context.go#L40) -· [Apply-time protection](../../../internal/framework/runner/runner.go#L300) - -**Remediation.** Match configured spellings with boundary checks against -surrounding runes, including spellings that begin or end with punctuation. -Preserve longest-match precedence and exemption behavior, and use consistent -case matching for both the scan and identity lookup. - -## 3. Merging nested segments truncates the earlier timestamp range - -**Expected.** The merged interval should cover both source intervals. The same -interval should determine whether the merge fits the configured duration limit -and the gap to the next segment. Overlapping source intervals pass input -validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) -· [Processing sequence](../../internal/pipeline.md) - -**Observed.** With the default normalization settings and an empty pipeline, -same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]` -containing the first two utterances and a separate `[20,21]` segment. -`mergeSegments` takes `next.End` even when it is earlier than the current end. -The next gap is then measured from 6, and duration checks likewise use -`next.End - current.Start`, which can let an over-limit merged interval pass. -[Merge loop](../../../internal/core/normalization/normalize.go#L46) -· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153) - -**Remediation.** Use the later of the two end times for both the prospective -duration and the resulting segment. Carry that end forward when calculating -subsequent gaps. - -## 4. Balanced chunking rejects feasible explicit targets - -**Expected.** A configured section count should succeed when the input can be -partitioned into that many contiguous sections within the token limit. -Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64) - -**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers -to prevent normalization merges, fit in two chunks with a maximum of 6. -The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts -the target, but the builder stops the second chunk at 5 because it exceeds the -average target of 4. It leaves the last segment unassigned and the CLI exits 1 -with “target_sections (2) is impossible under current constraints (got 2 sections).” -[Feasibility check](../../../internal/core/chunking/sections.go#L143) -· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249) - -**Remediation.** Choose balanced cuts only when the remaining suffix can still -fit in the remaining count. A simpler fallback is to build the minimum number -of max-bounded chunks and split them deterministically to reach a feasible target. - -## 5. Cancellation during FIFO delivery can hang and later report success - -**Expected.** A FIFO can wait for a reader. A catchable cancellation while it -waits should end processing and attempt failed-status report finalization. -[Output lifecycle](../../operations.md#outputs-and-lifecycle) -· [Subprocess cancellation](../../integrations/subprocess.md#result-handling) - -**Observed.** With an empty pipeline and a named pipe as output, the child -stayed blocked after SIGTERM and left reports at `in_progress`. When a reader -attached after the signal, it received the transcript and the child exited 0 -with `completed` reports. The main signal handler cancels a context, and the -CLI checks it before calling `Finish`. The direct-output branch then blocks -in `os.OpenFile` without observing that context. Delivery and final report -generation have no check for cancellation during that wait. -[Signal handling](../../../cmd/audita/main.go#L16) -· [Output finalization](../../../internal/cli/process_output.go#L54) -· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318) - -**Remediation.** Pass cancellation into delivery. For supported FIFO platforms, -use a context-aware wait for reader availability and for writable capacity, -handling cancellation through the same failed-report path. Define how a -cancellation that arrives after stream bytes have already been written is reported. - -## 6. Transcript shape detection loses integer and unknown-field fidelity - -**Expected.** Distinct supplied IDs in the supported integer range should -remain distinct through validation. Unknown fields should be ignored. -[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) - -**Observed.** On Linux amd64, valid IDs `9007199254740992` and -`9007199254740993` collapsed to the former value and the CLI rejected them as -duplicates. An envelope with a valid `segments` array and an unknown field -`"ignored":1e400` also failed. The first decode targets `any`, so it converts -numbers throughout the payload to `float64` before the segment array is -marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35) -· [Array extraction](../../../internal/core/schema/transcript.go#L115) - -**Remediation.** Select the top-level shape while retaining the raw segment -array, then decode it directly into the typed source segment fields. Let the -typed decode handle range and type validation. - -## 7. An extra numeric validator reference can discard valid sibling decisions - -**Expected.** If a validator returns one usable decision per requested -reference, unexpected reference numbers should be ignored and the successful -batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching) - -**Observed.** A fixture returned two valid approvals and an extra decision -with index `34`. Both corrections applied. Keeping the approvals and changing -the extra index to `9223372036854775808` or `34.0` made both decisions -unavailable with `invalid_output`, so neither correction applied. The schema -accepts those nonnegative integer values. The adapter decodes all indices into -Go `int` before reconciliation, so one schema-valid but incompatible number -discards the batch before the out-of-range index can be ignored. -[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json) -· [Adapter decode](../../../internal/framework/llm/adapter.go#L191) -· [Decision index type](../../../internal/framework/validators/llm_models.go#L46) -· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299) - -**Remediation.** Preserve validator references as JSON numbers until matching -them against the known local range. Normalize integral numeric spellings and -classify out-of-range values as extras before converting requested references -to Go indices. - -## 8. Span-only protection can reject a valid edit or miss a repair opportunity - -**Expected.** The protection validator should check whether a correction -changes protected occurrences in the target segment. A known protection -rejection can feed repair, and application should still recheck against the -text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution) -· [Processing sequence](../../internal/pipeline.md) - -**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in -`The certAIN result follows.` was rejected because the isolated span looks -like a protected occurrence. The full segment has no occurrence of `AIN` as a -name. In the other direction, a grammar change from `Silver` to `Golden` in -`The Silver Shield is equipped.` passed deterministic protection with -`Silver Shield` protected. It received two scripted model approvals and was -skipped only at application. With a positive repair budget, the provider -received no repair request. The validator normally compares the spans alone, -while application compares the whole segment. -[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100) -· [Repair eligibility](../../../internal/framework/runner/repair.go#L40) -· [Apply-time protection](../../../internal/framework/runner/runner.go#L300) - -**Remediation.** Use the shared segment preview for applicable candidates in -all modules and preserve the spoken-word repetition exception. Keep the -application check against the current working text for conflicts introduced by -earlier edits. - -## 9. Credential redaction can change the ordinary YAML schema - -**Expected.** The ordinary reports should retain their fixed schema, including -the required `summary` key and documented status values, while sanitizing -credential content. [Report schema](../../integrations/output-files.md#corrected-transcript) - -**Observed.** With the synthetic credential value `summary`, a successful -processing run produced YAML with `"[REDACTED]"` in place of its required -`summary` key. After serializing the typed report, redaction applies a -substitution to every string scalar, including field names and fixed status -values. The CLI exited 0 despite the schema change. -[Report serialization](../../../internal/core/reporting/report.go#L182) -· [YAML redaction](../../../internal/core/redaction/redaction.go#L104) - -**Remediation.** Sanitize input-derived report values before serialization. -Keep Audita-owned field names, status enums, and reason codes under the -control of the typed report serializer. - -## 10. A completed proposal can disappear from the interruption ledger - -**Expected.** Debug capture should retain proposed candidates that reached a -usable response and were subsequently interrupted. Their ledger rows should -record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files) -· [Candidate capture](../../internal/diagnostics-reporting.md) - -**Observed.** In a two-chunk run, the first chunk's proposal request was held -open while the second returned a correction. After the second chunk's debug -response artifact appeared, SIGTERM stopped the first request. The run -published no transcript and correctly reported one usable proposal response -and two interrupted work units. The correction ledger contained only the -module failure row. The runner appends candidates only as it drains the -ordered validation queue. A fatal return or cancellation can leave a complete -later response in that queue or skip queuing it after recording its operation. -[Ordered collection](../../../internal/framework/runner/runner.go#L491) -· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70) - -**Remediation.** Retain successful section results independently of validation -scheduling. After joining workers on a fatal path, assign indices to -unprocessed successful responses in section and response order and record -their candidates as interrupted. - -## 11. Required delivery failures omit the phase in run error records - -**Expected.** A run-scoped error should identify its phase, including failed -required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records) - -**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1 -and wrote a failed `errors.yml` record with `reason: output_failure` and the -destination path. The record omitted `phase`. The output session distinguishes -transcript writes, report writes, and cleanup through `FileFailure.Kind`. -The CLI mapper fills `Phase` only for report initialization. -[Failure classification](../../../internal/core/diagnostics/output_session.go#L273) -· [Run error mapping](../../../internal/cli/process_output.go#L92) - -**Remediation.** Map each required output failure kind to the corresponding -run error phase, preserving the specific write or cleanup operation through -finalization and any failed-report retry. diff --git a/docs/roadmap/audit/02-efficiency.md b/docs/roadmap/audit/02-efficiency.md deleted file mode 100644 index 7a94f4d..0000000 --- a/docs/roadmap/audit/02-efficiency.md +++ /dev/null @@ -1,223 +0,0 @@ -# Algorithmic correctness and efficiency audit - -Reviewed the implementation at `0eca7b8`. Complexity estimates come from -source and call paths. Small synthetic probes checked match expansion, -ambiguous spans, timestamp ties, and strict parsing. The audit used no -production workloads or provider measurements. -The full offline suite passed with `go test -count=1 ./...` on Go 1.27.3. - -## Architecture and review scope - -Audita is a single-process Go CLI. CommandKit routes commands and streams, -ConfigKit resolves configuration, and the core packages parse, normalize, and -chunk transcripts and project glossary selections. The runner executes -correction modules in order over fixed chunks. Deterministic and LLM-backed -validators check proposals and each repair revision. PromptKit owns provider -admission, transport, and conversation coordination. The output session -publishes corrected transcripts, ordinary reports, and optional debug evidence. -[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) - -The review followed work from configuration and normalization through proposal -application and evidence assembly. It preserves the selected forward chunker, -stage barriers, replacement policies, PromptKit admission and replay, and -validator budgeting rules. Costs required by emitted payloads or full report -matrices are treated as output costs. The earlier correctness findings are -recorded separately. [Pipeline](../../internal/pipeline.md) -· [Validator budgeting](../../internal/validators.md#supplemental-validation-context) -· [Deferred traversal design](../future.md#independent-whole-pipeline-traversal-by-chunk) -· [Correctness audit](01-correctness.md) - -## Efficiency and scalability findings - -### 1. Normalization repeatedly copies and scans growing merge runs - -`shouldMergeWithReason` constructs the merged text and estimates its tokens -from scratch. On success, `mergeSegments` constructs the text again and -allocates a new category slice that copies the entire prefix. For a run of -`k` merged segments with fixed-size nonempty text and category lists, the -merge pass reads and copies `Θ(k²)` text bytes and category elements. -The exact bound is proportional to the sum of the merge-prefix sizes. -The token cap usually limits the run, but -nonempty whitespace-only fragments with short gaps can merge at zero estimated -tokens. The cost is therefore possible under the default token cap. -[Merge loop](../../../internal/core/normalization/normalize.go#L46) -· [Merge check and construction](../../../internal/core/normalization/normalize.go#L153) -· [Token estimator](../../../internal/core/normalization/tokens.go#L14) -· [Accepted segment text](../../integrations/transcript-glossary-files.md#transcript-json-contract) - -Keep the forward merge pass. Accumulate text in an amortized builder and append -categories to an owned buffer. Maintain unrounded word and punctuation counts -per segment and combine them with the joiner before testing the token cap. -The ellipsis joiner can start a word when the previous text ends in whitespace, -and punctuation must be rounded only after adding all components. Check time -limits before processing text. This makes the merge pass linear in text bytes -and category elements, in addition to the existing chronological sort. - -### 2. Application reporting searches the source chunk for each skipped edit - -Application builds a segment-ID index inside `ApplyProposals`. The subsequent -`attributeApplyFiltering` pass indexes the approved proposals and sections, -but searches the original section's segment slice again for each skipped -proposal. With `s` segments and `k` skips in a section, this is -`O(s × k)` segment comparisons. Two identical, approved corrections per -segment make the second skip after the first changes its span. With one skip -per segment, the scans visit exactly `s(s+1)/2` entries. These are ordinary -stale conflicts, so the work can produce zero filtering records. -[Indexed application](../../../internal/framework/proposals/apply.go#L42) -· [Skip attribution](../../../internal/framework/runner/runner.go#L877) - -Return immediately when there are no skips. For affected sections, index the -original segments once or carry original applicability forward from validation. -This reduces target lookup to expected `O(s + k)` work per affected section. -If defensive apply-time filtering adds actual defects, merge them once per -originating operation. The current singleton merge can rebuild the growing -deduplication map and sort the list after each addition. Preserve the -distinction between a response defect and a stale conflict. -[Filtering aggregation](../../../internal/framework/runner/runner.go#L837) -· [Recovery contract](../../integrations/output-files.md#error-and-recovery-records) - -### 3. Validator glossary context expands every occurrence into every contributor - -Context uses `ProtectedVocabulary.occurrences`, which emits one record per -matched occurrence per glossary identity. The context builder immediately -deduplicates those records into a set. For a spelling that occurs `m` times -and belongs to `g` selected entries, this costs `Θ(m × g)` records and -identity insertions after matching. A synthetic probe with 128 occurrences -of an alias shared by 96 entries produced 12,288 records to select 96 -definitions. The deterministic protection check needs occurrence multiplicity. -The context builder only needs membership. [Occurrence expansion](../../../internal/framework/validators/protected_terms.go#L76) -· [Context selection](../../../internal/framework/validators/llm_context.go#L40) - -Share the same scanner and match precedence. Let context deduplicate matched -spellings across the source, result, and neighbors before expanding their -identity lists. This case then takes expected `O(m + g)` membership work -before ordering the selected entries. Keep occurrence-level evidence for the -protection check and preserve glossary selection order. - -### 4. Validation measures optional context that cannot fit - -The production estimator correctly composes encoded fragment counts. Its -fitter nevertheless serializes all optional glossary entries and neighbors, -builds both prefix tables, and then removes entries from the end. It does -that even when the required fields alone exceed the budget. For `p` items -sharing `b` bytes of optional context across `g` entries, this can repeat -`Θ(p × b)` serialization and allocate `Θ(g)` prefix counters per fit even -when no optional entry is retained. -[Prefix measurement and fitting](../../../internal/framework/validators/llm_budget.go#L42) -· [Production batching caller](../../../internal/framework/validators/llm_batching.go#L17) - -Return the required-only result as soon as its cost exceeds the budget. -Otherwise measure glossary entries in order and stop at the first one that -makes the prefix exceed the budget. Measure neighbors only if the entire -glossary prefix fits. Costs are monotone, so this preserves the existing policy -of dropping neighbors before definitions and keeping the longest affordable -ordered prefixes. An incremental fit uses constant auxiliary space for -counters and meters only retained entries plus the first exclusion in each -eligible field. Reusable glossary entry costs can also be stored with the -immutable glossary view if repeated encodings remain material. The cost of -matching candidates to glossary entries is separate. -[Context priority](../../internal/validators.md#supplemental-validation-context) - -### 5. Ordinary runs retain debug-only decision history - -The runner only adds candidates to the returned candidate list when debug -capture is active. It still builds a per-module candidate slice on every run -and aggregates, sorts, and stores initial validator decisions and rejections -in every `ModuleResult`. Rejected records carry original and proposed text -references. With `d_i` decision and rejection records in module `i`, finished -modules retain `Θ(Σ d_i)` records through invocation finalization, and the -runner pays for ordering them even when their production consumer, the -correction ledger, is disabled. Ordinary reports read the work-unit states and -compact operation outcomes. The current section's decisions and rejections -still serve candidate validation and repair eligibility. -[Candidate collection](../../../internal/framework/runner/runner.go#L537) -· [History sorting](../../../internal/framework/runner/runner.go#L617) -· [Module retention](../../../internal/framework/runner/runner.go#L271) -· [Ordinary reporting](../../../internal/framework/processreport/report.go#L51) -· [Debug ledger wiring](../../../internal/cli/process_pipeline.go#L54) - -Separate detailed evidence retention from section-local decision state. In -the compact CLI path, skip copying candidates into aggregate slices and release -decision and rejection detail once the section finishes. Keep the operation -outcomes, counts, and failure origin used by ordinary reporting. Debug runs can -continue to retain and order the full evidence, including partially completed -work after a fatal return. - -### 6. Repeated pipeline selections duplicate setup work - -Module specifications select and copy the rich entries and project the -protected vocabulary independently for each pipeline entry, before the runner -starts. For `m` entries with the same selections of size `g`, setup time and -retained projections grow as `Θ(m × g)` with bounded-size entry fields. The -same data is used to prepare stage prompt variables and compile matchers again -on every repeated stage. Separately, readiness inspects a profile and its -backend for every role, even when they share IDs. PromptKit returns detached -inspection data, including copies of extra parameters. Selecting the same -resources `m` times with `b` extra parameter values can add `Θ(m × b)` -inspection work before processing. -[Module projection](../../../internal/framework/contracts/contracts.go#L107) -· [Entry copies](../../../internal/core/glossary/selection.go#L20) -· [Glossary preparation](../../../internal/framework/runner/runner.go#L169) -· [Role readiness](../../../internal/framework/preflight/preflight.go#L129) -· [Profile and backend inspection](../../../internal/cli/process_inputs.go#L125) - -Reuse read-only projections and prepared data for identical effective ordered -selections, including the glossary correction exemption in the key. -Alternatively, build the projection at the module boundary and release it -before the next stage to bound retained memory when selections differ. Cache -successful profile and backend inspections for the captured engine while -keeping role-by-role checks and first-failure attribution. Shared data keeps -identical views and inspections proportional to the number of distinct inputs -plus the number of stage entries. Request messages still carry the selected -context for each call. - -## Additional correctness findings - -### 7. The unique-span check accepts overlapping matches - -The grammar prompt requires an original span to appear exactly once. -`PreviewProposalForSegment` uses `strings.Count`, which counts nonoverlapping -matches. A probe proposed changing `ha ha` to `heh` in `ha ha ha`. The span -starts at byte offsets 0 and 3, but the preview returned applicable with -`heh ha` as the result. The same preview governs validation and application. -[Unique-span instruction](../../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml#L19) -· [Preview](../../../internal/framework/proposals/preview.go#L35) -· [Application](../../../internal/framework/proposals/apply.go#L59) - -For `require_unique`, locate the first occurrence and check for another -starting after its first byte, including an overlapping start. Stop as soon -as ambiguity is established. A successful check can use the recorded offset -to assemble the replacement. This retains a linear scan in the worst case for -a fixed literal and can exit early on ambiguity. Preserve the existing -nonoverlapping replacement behavior for `replace_all`. - -### 8. Normalization can reorder utterances with exactly tied timestamps - -`Normalize` compares only start times in an unstable sort. With 14 segments -whose starts alternate between 1 and 0 and whose ends are start plus one, a -normalization-only CLI run emitted the start-zero group in source-index order -`9,1,13,3,7,5,11`. Those segments have identical start and end times. With -distinct speakers, the ordering change occurs without any merge. The earlier -Python implementation at `0b1b670` ordered exact ties by source position, and -the Go rewrite recorded preservation of existing behavior as its baseline. -[Current sort](../../../internal/core/normalization/normalize.go#L58) - -Preserve the input order for exact time ties. A stable sort or an explicit -source-position tiebreaker retains `O(n log n)` comparisons. Specify the -timestamp tiebreaker so later refactoring preserves utterance order. - -### 9. The strict transcript parser panics on a missing ID - -`ParseTranscriptJSON` calls the source parser, which allows omitted and null -IDs, then dereferences every ID before checking that IDs are sequential. A -package probe with an otherwise valid segment and no ID reproduced the nil -pointer panic. This affects the strict package helper, which currently has -test callers. The processing command uses the tolerant source parser and -assigns canonical IDs separately. -[Strict parser](../../../internal/core/schema/transcript.go#L65) -· [Sequential ID validation](../../../internal/core/schema/transcript.go#L204) -· [CLI input acquisition](../../../internal/cli/process_inputs.go#L46) - -Check ID presence and return the normal field-validation error before -dereferencing. This adds a constant-time check per segment and preserves the -linear typed conversion pass. diff --git a/docs/roadmap/audit/03-complexity.md b/docs/roadmap/audit/03-complexity.md deleted file mode 100644 index 9d45456..0000000 --- a/docs/roadmap/audit/03-complexity.md +++ /dev/null @@ -1,322 +0,0 @@ -# Maintainability and complexity audit - -Reviewed source at `717d59c`. The codebase graph identified complex control -flow and cross-package call paths. A Go AST scan counted branches and -declaration spans for 353 functions and methods across 43 tracked Go -directories, excluding test files. The score below starts at one and adds -one per conditional, loop, non-default switch or select case, and short-circuit -operator, including those inside callbacks. The ranking guided source review. -Findings are based on static analysis. - -## Architecture - -Audita is a single-process Go CLI. CommandKit routes commands and streams. -ConfigKit resolves application settings, and the core packages parse, -normalize, and chunk transcripts and derive glossary selections. The -runner executes modules in order over fixed chunks. Deterministic and -LLM-backed validators gate proposals and revalidate repairs. PromptKit owns -model execution, backend admission, and bounded conversation history. Audita -applies approved edits and derives reports and debug artifacts from processing -outcomes. [Package map](../../internal/overview.md) -· [Pipeline](../../internal/pipeline.md) -· [Ownership rules](../../policy/architecture.md#ownership-and-dependency-direction) - -The runner and validator runtime had the largest non-test Go source totals, -at 1,373 and 1,351 physical lines respectively. The highest-scoring functions -concentrated in orchestration, model integration, and report projection. - -| Function | Branch score | Declaration lines | -| --- | ---: | ---: | -| `runner.repairCandidates` | 46 | 225 | -| `runner.runModulePipeline` | 36 | 212 | -| `processreport.Build` | 36 | 140 | -| `(*LLMBackedValidator).Validate` | 31 | 170 | -| `(*Runner).Run` | 30 | 168 | -| `runner.validateSectionCandidates` | 30 | 154 | -| `(*RoleClient).prepareCall` | 28 | 153 | -| `processreport.BuildCorrectionLedger` | 27 | 176 | -| `proposal_generation.GenerateCandidates` | 25 | 148 | -| `diagnostics.SelectOutput` | 21 | 66 | - -## Refactoring opportunities - -### 1. Give module execution one place to commit its result - -`Run` owns the stage barrier, working transcript, inventory, module setup, -application, and partial-return assembly. The pipeline-error, application-error, -and success paths each construct a module result and copy work units into the -invocation inventory. Repair approvals are marked interrupted in both `Run` -and `runModulePipeline`. The latter also coordinates two worker lifetimes, -ordered proposal indexing, cancellation, and evidence aggregation. Changing -what an interrupted unit retains requires following both functions and their -callbacks. The earlier missing-ledger-candidate finding follows from indexing -and retention occurring only while scheduling validation. [Module commit and failure paths](../../../internal/framework/runner/runner.go#L187) -· [Section orchestration](../../../internal/framework/runner/runner.go#L441) -· [Interrupted evidence finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) - -Extract the section validation-and-repair worker from the collector. Let the -collector own ordered indexing and the evidence returned by workers, with -one path to register a fatal cause and cancel peers. Let one module-level -completion path resolve applied, skipped, and interrupted repair outcomes -and commit updated sections after successful application. Preserve validation -starting as ordered proposals become available, deterministic proposal IDs, -joined workers on fatal paths, and the stage barrier. - -### 2. Separate one repair branch from the section's repair loop - -`repairCandidates` selects candidates, builds each request, interprets -delivered turns, revalidates revisions, attributes outcomes, and writes debug -artifacts. Nested closures share mutable `current`, `localRef`, `findings`, -attempt counts, and section-level slices. For example, a successful revision -updates the branch, original proposal filtering, revalidation operations, and -approval list inside the coordinator callback. Error, withdrawal, exhaustion, -and accepted final-turn paths finish that work at different points. -[Branch setup](../../../internal/framework/runner/repair.go#L56) -· [Turn handling](../../../internal/framework/runner/repair.go#L146) -· [Repair behavior](../../internal/modules.md#failure-and-outcome-behavior) - -Keep the outer loop responsible for selecting and collecting branches. Have -one branch-local operation return the terminal outcome, optional approved -revision, and operation evidence. Separate construction of the next request, -assessment of a delivered turn, and optional attempt recording. Define the -branch's internal states once so application and reports can project their -own stable labels. Keep PromptKit responsible for history and turn limits. -Retain final-turn revalidation, target changes within the section, and -independence of sibling approvals. - -### 3. Put validator applicability and decision meaning at their owning boundary - -`feedbackCandidates` switches on validator names and module keys to decide -whether each validator has usable evidence. The validator implementations -also check previews and module policy. The runner then rewrites an -`original_text_presence` approval if its preview is ambiguous, so the same -validator's returned verdict can mean something different outside the runner. -`validateSectionCandidates` interleaves those checks with two eligibility -modes, cardinality checks, filtering, and evidence recording. The framework -also declares identical validator interfaces under `contracts` and -`validators`. [Feedback selection](../../../internal/framework/runner/feedback.go#L13) -· [Runner verdict handling](../../../internal/framework/runner/runner.go#L680) -· [Original-text validator](../../../internal/framework/validators/deterministic.go#L37) -· [Validator contract](../../../internal/framework/contracts/contracts.go#L29) -· [Duplicate interface](../../../internal/framework/validators/models.go#L95) - -Put each check's applicability and verdict together in its validator or its -built-in registration. Use one validator interface and one internal decision -disposition, deriving serialized approval from it. Keep the runner responsible -for submitting the shortlist or collecting independent findings, checking -returned identities, and applying only fully approved candidates. Preserve -ordered chains, zero-budget shortlisting, unavailable decisions, and separate -revalidation of revisions. The earlier audit records the span-only -protection mismatch. [Protection correctness](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity) - -### 4. Separate validation-batch assessment from the diagnostic writer - -`LLMBackedValidator.Validate` interleaves request preparation, prompt selection, -batch dispatch, reconciliation, partial-result assembly, and debug payload -construction. Its nested writer consumes the original request, local -reference mapping, raw response, reconciliation result, and failure. Proposal -generation and validation declare writer interfaces with matching signatures -and duplicate four-field artifact types. Their adapter wrappers copy each -path from `llm.InteractionArtifacts`. Repair writes through the same concrete -writer directly. A change to the common artifact envelope or safe error -serialization therefore touches several packages. -[Batch loop](../../../internal/framework/validators/llm_validators.go#L112) -· [Proposal writer contract](../../../internal/framework/proposal_generation/generate.go#L23) -· [Validation writer contract](../../../internal/framework/validators/models.go#L84) -· [Writer adapter](../../../internal/framework/runner/runner.go#L994) -· [Repair artifact capture](../../../internal/framework/runner/repair.go#L165) - -Give one batch operation responsibility for dispatch and assessment. Return -reconciled decisions, its compact outcome, and only the evidence needed by the -diagnostic path. Reuse one internal artifact value type and writer seam for -common metadata, redaction, and safe write failures. Keep the per-role -payload construction with its domain owner. Preserve local-reference -reconciliation, missing and duplicate decision handling, partial results on -fatal return, and nonfatal debug-write warnings. - -### 5. Make the structured client's call lifecycle explicit - -`prepareCall` allocates a logical call, validates the output target, builds -the native request, installs two observer paths, captures prepared evidence, -and returns a closure for decoding and completion. That closure assigns -captured `err` from a named result and executes a reverse-order stack of -two finalizers before returning the mutated response. `Coordinate` also -finalizes a pending call on its separate undelivered-result path. Following an -error across both entry points requires keeping the prepared snapshot, native -terminal event, raw output, typed result, and logical call status in mind. -`Failure.Kind` and reason-based logical-call classification both describe -malformed-output failures. The ordinary request also retains a continuation -field whose current callers are adapter tests. [Call preparation and finalization](../../../internal/framework/llm/adapter.go#L54) -· [Coordinated call](../../../internal/framework/llm/coordinate.go#L15) -· [Request contract](../../../internal/framework/structuredoutput/call.go#L28) -· [Failure contract](../../../internal/framework/structuredoutput/failure.go#L17) - -Keep the two native entry points and make their shared finish operation -explicit. Move native request construction and result decoding out of the -bookkeeping closure. Have a concrete per-call record finalize observation, -execution counts, and response evidence exactly once on both success and -failure. Derive the existing diagnostic failure label from one classification -and assess whether the test-only continuation path still needs to be exposed. -Preserve terminal counts, preparation failures without a snapshot, and the -rule that an undelivered response supplies diagnostic evidence only. -[LLM runtime](../../internal/llm-runtime.md) - -### 6. Use a single proposal handoff and wire repair capability explicitly - -`GenerateCandidates` returns a correction slice, parallel segment references, -and a second enriched proposal slice with its own indexing inputs. -`ExecuteModuleProposal` forwards only the first two slices and the -exchange. The runner later builds its own enriched candidates, which is where -stage-wide index assignment takes place. Repair recovers the request-local -segment reference by the proposal's slice position or falls back to looking -up the resolved target. Separately, an enabled repair budget, proposal-exchange -capture, the module's repair prompt method, and the repair client's coordinator -capability are established at different boundaries. -[Proposal result and mapping](../../../internal/framework/proposal_generation/generate.go#L72) -· [Module handoff](../../../internal/framework/proposal_generation/module_proposal.go#L75) -· [Ordered index assignment](../../../internal/framework/runner/runner.go#L521) -· [Repair readiness at use](../../../internal/framework/runner/repair.go#L98) - -Carry a single proposal record with the resolved target and original local -reference, then assign the stage-wide index at the ordered runner boundary. -Assess retirement of the generator's separate enriched return once its -callers are migrated. Resolve repair availability with the concrete -module and role client at module setup. Use that capability to determine -exchange capture, and keep the disabled path simple. Preserve original -response order, invalid reference evidence, required profile readiness, -and zero-budget behavior. - -### 7. Give reporting one source for operation classification and counts - -`processreport.Build` has eleven work-unit and operation counters, nested -aggregation and projection, and a separate pass for run failures. -`SummaryOnly` bypasses selected detail paths inside that aggregation. -The LLM adapter decides recoverability through `structuredoutput.Failure`, -while `Build` uses a separate allowlist of recoverable reason codes after -the class has been discarded from the operation. Adding a recoverable -reason can therefore require coordinated edits to the adapter, domain -operation, and report counting. Safe report messages are a presentation -concern and can stay at that boundary. -[Snapshot aggregation](../../../internal/framework/processreport/report.go#L36) -· [Report-side classification](../../../internal/framework/processreport/report.go#L212) -· [Processing operation](../../../internal/framework/outcomes/outcomes.go#L84) -· [Adapter classification](../../../internal/framework/llm/adapter.go#L228) - -Give operation outcomes a single source for success and recoverability. -Accumulate work-unit facts and summary counts in one pass, then project -rows and detail records when requested. Keep presentation ordering and safe -messages explicit. Preserve unknown versus zero counts, a single recovery -per successful operation, and agreement among YAML, Markdown, and CLI counts. -[Ordinary report contract](../../integrations/output-files.md#summary-counts) - -### 8. Join correction evidence before building ledger rows - -`BuildCorrectionLedger` indexes decisions, repairs, and skips, then emits -applied, skipped, rejected, repaired, interrupted, and module-failure rows -from separate loops. Each candidate path repeats identity, replacement, -and decision projections. Precedence is implicit in iteration order, -the `seen` map, and checks for repair evidence. It is easy to add a field to -one row shape and miss the other cases. The ledger also looks up validator -class by name after execution, while the runner can use metadata supplied -by the validator itself. [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70) -· [Repair and interruption rows](../../../internal/framework/processreport/correction_ledger.go#L197) -· [Decision grouping](../../../internal/framework/processreport/correction_ledger.go#L247) -· [Runtime class selection](../../../internal/validators/metadata/metadata.go#L44) - -Gather the existing evidence into one per-proposal record keyed by module -instance and proposal index. Determine final disposition in one place, then -project common fields once and add the application and repair fields that -belong to that disposition. Consider recording the execution class used -when the decision is made, preserving built-in metadata. Keep module-only -failures distinct and retain the existing ordering, empty replacement fields, -partial decisions, and repaired skip reasons. -[Debug ledger contract](../../integrations/output-files.md#debug-only-files) - -### 9. Clarify output preparation, publication, and failure ownership - -The CLI builds initial and final report bundles and two failed-report -renderers. Preflight wraps the initial renderer to cover setup cancellation. -`OutputSession.Initialize` and `Finalize` each iterate required reports and -run a second write pass for failed status. File failure purposes are strings -interpreted again in the CLI. Destination selection also infers its role by -comparing the path string to the transcript destination, and preparation and -delivery each re-derive aspects of the path and file type. -[CLI preparation](../../../internal/cli/process_inputs.go#L70) -· [Preflight handoff](../../../internal/framework/preflight/preflight.go#L41) -· [Output selection](../../../internal/core/diagnostics/output_session.go#L61) -· [Publication and retry](../../../internal/core/diagnostics/output_session.go#L222) -· [Failure projection](../../../internal/cli/process_output.go#L92) - -Keep a checked destination with its explicit role and publication mode. -Reuse a report-write helper for both passes and make write or cleanup phases -typed facts of the output session. Keep report rendering and status in one CLI -handoff that can distinguish a valid failed snapshot from an unavailable one. -Preserve readiness before writes, intentional in-place transcript replacement, -required-output ordering, cleanup, and the bounded best-effort failed-report -pass. Related destination, cancellation, and phase defects are in the -correctness audit. [Output lifecycle](../../operations.md#outputs-and-lifecycle) -· [Correctness audit](01-correctness.md) - -### 10. Share root and per-module configuration rules - -`resolveSettings` repeats the same presence, null, inheritance, and fallback -logic for the validation and repair profile fields. Each root field also -has a separate fallback path after the stage loop. `validate` repeats the -LLM value checks at the root and at every stage, with a separately built -ConfigKit path for each. Repair and validation currently agree, but changing -one profile rule requires reading multiple versions of it. -[Profile resolution](../../../internal/core/config/resolve.go#L24) -· [Configuration validation](../../../internal/core/config/config.go#L143) - -Use a small typed helper for profile fallback that receives the correct -path, value, and dependency. Capture explicitly supplied root values before -recording root fallbacks, so a stage that selects a different proposal -profile still resolves correctly. Validate the common LLM settings with one -function taking a path prefix. Keep root-only constraints and readiness -separate, and preserve recorded origins, explicit null, and zero values. -[Configuration contract](../../config.md#settings) - -## Additional correctness findings - -### 11. A contradictory validator decision can advance an unavailable candidate - -The internal `Decision` accepts `Approved: true` together with -`DispositionUnavailable`. Cardinality checks validate only indices. With -one such decision and no other rejection for that candidate, the runner -treats it as approved in the shortlist path and leaves it unblocked in the -independent-feedback path. This violates the invariant that an unavailable -assessment cannot advance as approval. Bundled producers set matching -values. The defect is at the internal validator result boundary. Make the -disposition authoritative or reject contradictions before processing the -decision. [Decision and cardinality](../../../internal/framework/validators/models.go#L56) -· [Approval and blocking](../../../internal/framework/runner/runner.go#L769) -· [Validator guarantees](../../internal/validators.md#llm-validator-batching) - -### 12. The runner accepts an already canceled empty pipeline - -With valid inputs and no module specifications, `Run` constructs the chunk -inventory and returns success even when the parent context was canceled -before entry. Its cancellation check sits inside the module loop. The -processing CLI checks cancellation around the runner, so the gap is in the -runner contract. Check the context at entry and at the run's successful -completion boundary, including normalization-only invocations. -[Runner entry](../../../internal/framework/runner/runner.go#L121) -· [Runner success return](../../../internal/framework/runner/runner.go#L286) -· [CLI cancellation check](../../../internal/cli/process.go#L65) -· [Pipeline cancellation contract](../../internal/pipeline.md) - -### 13. Missing failed-report rendering can erase initialized evidence - -`preflight.Prepare` converts a nil `FailedReports` callback into a callable -function returning a zero-value `ReportBundle`. If reporting is enabled and -cancellation arrives after initialization, preflight passes that empty -bundle to `Finish`. `Finalize` writes its three nil payloads as empty files, -replacing the existing `in_progress` reports. The initialization retry can -similarly replace reports that were written successfully in its first pass. -The CLI supplies a failed renderer for its ordinary reporting path. At the -internal preflight seam, carry the absence of a failed bundle through to -publication or require a valid failed bundle before acquiring resources, -so the existing evidence survives when a failed snapshot cannot be rendered. -[Callback wrapping](../../../internal/framework/preflight/preflight.go#L68) -· [Initialization write retry](../../../internal/core/diagnostics/output_session.go#L222) -· [Final report write](../../../internal/core/diagnostics/output_session.go#L273) diff --git a/docs/roadmap/audit/04-concurrency.md b/docs/roadmap/audit/04-concurrency.md deleted file mode 100644 index d93cf34..0000000 --- a/docs/roadmap/audit/04-concurrency.md +++ /dev/null @@ -1,227 +0,0 @@ -# Concurrency audit - -Reviewed source at `c0193df`. The full offline suite passed under -`go test -race -count=1 ./...` with Go 1.27.3. Selected capacity, -cancellation, coordination, and observation tests also passed eight runs -under the race detector. The timing findings below use Linux subprocesses, -synthetic loopback responses, and a temporary runner probe. The audit used -no live provider calls. - -## Architecture and synchronization reviewed - -Audita is a single-process Go CLI. The executable translates SIGINT and -SIGTERM into context cancellation. CommandKit handles commands and streams, -ConfigKit resolves settings, and the deterministic core normalizes and chunks -the transcript. The runner owns one stage barrier across fixed chunks. -Proposal and validation goroutines operate within a stage, and the runner -collects their results in a stable order before applying approved corrections. -The invocation shares a PromptKit engine across roles, with backend-local -admission, synchronous execution observations, and coordinated repair -branches. The CLI joins processing before assembling reports and delivering -output. [Signal setup](../../../cmd/audita/main.go#L16) -· [Package map](../../internal/overview.md) -· [Pipeline](../../internal/pipeline.md) -· [LLM runtime](../../internal/llm-runtime.md) - -- Proposal workers send one result per chunk. Their owner waits for all - senders before closing the channel. The validation channel is drained after - the worker join, which works because it has one slot per chunk and at most - one send per validation worker. Reducing that buffer would require changing - the drain order. A mutex protects the first fatal error, and capture of - warnings and execution evidence completes before the CLI reads it. - [Proposal worker lifecycle](../../../internal/framework/runner/runner.go#L385) - · [Error and result ownership](../../../internal/framework/runner/runner.go#L463) - · [Validation join](../../../internal/framework/runner/runner.go#L592) - · [Warning capture](../../../internal/cli/process.go#L42) - · [Evidence snapshot](../../../internal/cli/process_pipeline.go#L51) -- Existing tests cover cancellation while a peer call is active and another - waits for backend capacity, release after cancellation, independent backends, - repair response withholding, and nested validation on the repair backend. - Observations use synchronized collections, and a snapshot is taken after - workers join. Config and prompt capture happen before worker startup. - [Runner cancellation test](../../../internal/framework/runner/cancellation_test.go#L48) - · [Backend capacity tests](../../../internal/framework/llm/adapter_capacity_test.go#L42) - · [Repair coordination tests](../../../internal/framework/llm/coordinate_test.go#L77) - · [Concurrent observation test](../../../internal/framework/llm/observations_test.go#L59) - · [Snapshot contract](../../../internal/framework/llm/observations.go#L81) - -## Findings - -### 1. Blocked output delivery can ignore cancellation and report success - -The CLI checks the context immediately before `OutputSession.Finish`. -Delivery then opens a named pipe or writes stdout without observing the -context. With no FIFO reader, SIGTERM left the child blocked with -`in_progress` reports. When a reader attached, the child delivered the -transcript, finalized the reports as `completed`, and exited 0. The same -outcome occurred with a full stdout pipe and a 240 KB transcript. Draining -the pipe after SIGTERM allowed the write to finish and the child to report -success. The FIFO case was first recorded in the correctness audit. -[Last context check](../../../internal/cli/process_output.go#L54) -· [Stream open and write](../../../internal/core/diagnostics/output_session.go#L318) -· [Earlier FIFO finding](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success) - -Pass cancellation through delivery. For cancellable file descriptors, -make both reader availability and write readiness interruptible. Keep the -guidance for subprocess clients to drain both streams, and define how cancellation -after a partial stream write is reflected in status and reports. Check -cancellation before committing a regular-file rename and use the existing -failed-report path when delivery is interrupted. -[Output lifecycle](../../operations.md#outputs-and-lifecycle) -· [Subprocess cancellation](../../integrations/subprocess.md#result-handling) - -### 2. Waiting for transcript input can prevent shutdown before preflight - -`loadProcessInputs` calls a context-free `os.ReadFile` for the transcript. -When the path names a FIFO, it blocks on a writer before the application -can check for cancellation again. In a subprocess probe with no writer, -SIGTERM left the child blocked without starting report initialization. -Supplying the input afterward allowed the child to exit 1. The process -therefore relies on a producer or forced termination to unblock that -read. The input parser accepts the JSON once it arrives. -[Input acquisition](../../../internal/cli/process_inputs.go#L46) -· [File read](../../../internal/core/io/files.go#L9) - -If pipe input remains supported, use a context-aware open and read on -supported platforms and stop reading after cancellation. If support is -intentionally limited to regular input files, make that contract explicit -and validate the path before a blocking open. -[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) - -### 3. Cancellation can cross the runner's module commit boundary - -`Runner.Run` checks the context after the module's workers have joined, -then applies corrections, classifies work units, and commits the new sections -without another cancellation check. A temporary runner probe canceled the -parent immediately after that check sampled a live context. The final -module returned success, an applied correction, and a completed work unit. -An already-canceled invocation with no modules also returned success, as -the check is inside the module loop. The CLI has later context checks that -prevent publication in these cases, but can render canceled-run evidence -counting edits committed after the cancellation. Normalization and encoding -also run synchronously between CLI cancellation checks, which can delay -shutdown for large workloads. -[Before application](../../../internal/framework/runner/runner.go#L190) -· [Commit and return](../../../internal/framework/runner/runner.go#L264) -· [Local processing boundaries](../../../internal/cli/process.go#L55) -· [Earlier empty-pipeline finding](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline) - -Check cancellation at runner entry, before starting another module, -and just before committing computed updates. Keep the previous working -transcript until the commit check passes. If cancellation during large local -passes needs a shorter response time, add cooperative checks at natural -iteration boundaries. -[Cancellation and partial-result contract](../../internal/pipeline.md) - -### 4. A slow earlier proposal holds back later validation and retention - -The runner buffers completed proposals until the next chunk in order is -available. It assigns stage-wide indices, starts validators, and captures -ledger candidates only as that prefix advances. With two chunks and the -first response deliberately held open, the second response reached its -debug artifact while no validator call started. Releasing the first -response let both validators check the later correction. In a separate -run, canceling at the same point retained the later raw response and -counted one usable proposal response, but the correction ledger contained -only the module failure row. A slow earlier retry or timeout can therefore -stall independent validator work, and interruption can omit a completed -candidate from the ledger. -[Ordered proposal buffer](../../../internal/framework/runner/runner.go#L494) -· [Candidate capture and dispatch](../../../internal/framework/runner/runner.go#L520) -· [Earlier ledger finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) - -Retain completed responses independently of scheduling and assign -deterministic indices to the retained evidence on a fatal return. For -validation latency, consider section-local identities during checking and -repair, with stable stage-wide identities assigned during collection. -Account for diagnostic filenames and report references in that change. -This can preserve the existing stage barrier. -[Stage barrier decision](../future.md#independent-whole-pipeline-traversal-by-chunk) - -### 5. Proposal fan-out makes admission under finite capacity depend on scheduling - -Each chunk immediately attempts a proposal in its own goroutine. PromptKit -limits admitted calls, while Audita has no ordering at the admission boundary. -With 12 chunks, one backend slot, and zero queue capacity, six otherwise -identical runs each reached the synthetic provider once and recorded 11 -recoverable capacity errors. Chunk 12 entered in five runs and chunk 2 -entered in the other. Which chunks can produce corrections can therefore -vary before the provider sees any prompt. The backend enforced its configured -limit and the error reports reflected the rejections. The weakness comes -from submitting a complete transcript's worth of proposals at once to a -finite queue. -[Proposal fan-out](../../../internal/framework/runner/runner.go#L385) -· [Capacity classification](../../../internal/framework/llm/adapter.go#L254) -· [Engine-local admission](../../policy/architecture.md#correction-and-concurrency-invariants) - -Make the interaction between chunk fan-out and finite queue capacity -explicit in operator guidance. If the desired processing policy is to wait -for more chunks to receive a turn, introduce bounded or ordered dispatch for -that policy while preserving PromptKit's ownership of backend permits and -the existing recoverable-rejection behavior for configurations that select -it. - -### 6. Report paths can change referent between initialization and finalization - -Preflight checks report destinations once. Each subsequent `atomicReplace` -resolves the original path again, including during the final report pass. -In a loopback probe, a report-directory symlink pointed to one directory -during initialization and was switched to another while the proposal -request was held. The CLI exited 0. The original directory still held -`in_progress` reports, and the replacement directory received -`completed` reports for the same run ID. Destination collision checks only -covered the referents inspected during preflight. -[Destination selection](../../../internal/core/diagnostics/output_session.go#L61) -· [Report initialization](../../../internal/core/diagnostics/output_session.go#L222) -· [Per-write path resolution](../../../internal/core/diagnostics/output_session.go#L410) - -Bind the checked report destinations for the session, or detect a changed -referent and fail before writing to it. Use the same target identity for -initialization, finalization, and the failed-report retry. This complements -the earlier path-resolution finding, which covers a path whose meaning is -wrong even without a concurrent change. -[Earlier path resolution finding](01-correctness.md#1-output-path-resolution-can-replace-a-different-file) - -### 7. Concurrent invocations sharing a report directory replace each other's evidence - -Different transcript output paths can select the same report directory, -including by default when the outputs have the same parent. Each invocation -owns temporary files and has its own run ID, but all replace the same three -report basenames. In a probe, run A initialized reports and then waited for -its proposal response. Run B completed with a different transcript output in -the shared directory. Releasing A let it complete and replace B's completed -reports. Both commands exited 0. Because report writes are per file, a -consumer reading during overlap can also observe files from different runs. -[Report-directory selection](../../../internal/cli/process_inputs.go#L72) -· [Fixed basenames](../../../internal/core/diagnostics/output_session.go#L12) -· [Final report writes](../../../internal/core/diagnostics/output_session.go#L286) - -Document per-invocation report destinations for concurrent subprocess -consumers. If supporting shared destinations directly, consider -invocation-specific publication or explicit serialization for that -destination. Per-file atomic replacement should remain in place. -[Report placement and lifecycle](../../operations.md#outputs-and-lifecycle) - -### 8. Shared module and validator ownership is implicit at their interfaces - -The runner shares one module instance across proposal workers and one -validator instance per chain entry across validation workers. Requests -also carry shared stage-level glossary data and transcript segment copies -whose category slices still alias the stage's data. The interfaces accept a -context but do not say that methods can run concurrently, borrowed data must -be read-only, or a canceled call must return for the stage to join. A -stateful implementation that mutates cached state or retains and mutates -borrowed data can race with peers. An implementation that blocks through -cancellation can prevent the worker join. Bundled modules and validators -use read-only shared configuration and per-call working state in the paths -reviewed. This is an internal ownership gap. -[Module and validator contracts](../../../internal/framework/contracts/contracts.go#L21) -· [Shared proposal instance](../../../internal/framework/runner/runner.go#L403) -· [Shared validation chain](../../../internal/framework/runner/runner.go#L548) -· [Shallow section copy](../../../internal/framework/runner/runner.go#L966) - -Document concurrent invocation, read-only inputs, and cooperative cancellation -at the contract boundary. Keep state local to each call or synchronize shared -mutable state explicitly when a future implementation needs it. If an -implementation needs to mutate section data, give it an owned copy of that -data. diff --git a/docs/roadmap/audit/05-resources.md b/docs/roadmap/audit/05-resources.md deleted file mode 100644 index 4c8cdcb..0000000 --- a/docs/roadmap/audit/05-resources.md +++ /dev/null @@ -1,276 +0,0 @@ -# Resource management audit - -Reviewed 2026-11-15 at `e4e5dd7`. This audit covers the Audita CLI, file and stream -I/O, configuration and prompt capture, transcript processing, the pinned -PromptKit execution path, and required and optional artifact generation. The -review also checked local release entry points for resource ownership. The -findings below identify cleanup defects and avoidable resource growth. The -remediation options are proposals for follow-up work. - -## Architecture and lifecycle - -Audita is a single-process Go CLI. The executable owns the signal context and -passes caller-owned streams through CommandKit. The CLI loads configuration and -captured prompt sources, reads the transcript, and preflights destinations. An -output session owns the temporary file and directory used to publish a regular -transcript by rename. Report replacements use their own short-lived staging -directories. Named pipes and stdout are delivered directly. Debug files are -written to a run-specific directory for later inspection. -[Signal setup](../../../cmd/audita/main.go#L16) · [Package map](../../internal/overview.md) -· [Preflight](../../../internal/framework/preflight/preflight.go#L41) -· [Output session](../../../internal/core/diagnostics/output_session.go#L45) -· [Artifact lifecycle](../../operations.md#outputs-and-lifecycle) - -The CLI normalizes the transcript before passing it to the runner. The runner -holds the working transcript and outcome records in memory, creates fixed -chunks, and joins proposal and validator workers at each module boundary. -Role-bound clients share an invocation-local PromptKit engine for HTTP -requests, backend admission, retries, structural repair, and correction-repair -exchanges. The CLI assembles report and debug artifacts from the completed or -interrupted run. -[Normalization and invocation](../../../internal/cli/process.go#L55) -· [Runner ownership](../../../internal/framework/runner/runner.go#L121) -· [Worker joins](../../../internal/framework/runner/runner.go#L435) -· [Validation drain](../../../internal/framework/runner/runner.go#L592) -· [LLM runtime](../../internal/llm-runtime.md) -· [Report assembly](../../../internal/cli/process_pipeline.go#L50) - -Resource release on the reviewed cooperative paths includes the executable's -signal subscription, proposal and validator worker joins, and PromptKit's -operation and request contexts, retry timers, admission reservations, and active -permits. PromptKit removes canceled waiters or transfers ownership of a permit -to the caller that won the race with cancellation. Its built-in HTTP path closes -response bodies and reuses the standard Go transport in the CLI. The pinned -source bounds successful response reads at 16 MiB and provider error reads at -64 KiB, applying reader limits even without a known content length. These are -per-response byte bounds. In the pinned module, the relevant ownership is in -`engine.go`, `internal/usecase/runner.go`, `internal/generation/coordinator.go`, -`internal/capacity/manager.go`, and the HTTP code in `internal/llm/`. -[Pinned version](../../../go.mod#L11) · [CLI transport selection](../../../internal/cli/process_inputs.go#L42) -· [Engine construction](../../../internal/framework/llm/runtime.go#L103) -· [Capacity cancellation test](../../../internal/framework/llm/adapter_capacity_test.go#L42) -· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L48) - -## Findings - -### 1. Failed output cleanup loses ownership of temporary paths - -`OutputSession.Close` closes the transcript descriptor, then tries to remove the -staged payload and its private directory. It clears both path fields even when -removal returns an error. A temporary permission or filesystem failure can leave -a directory or an unpublished payload behind. Finalization surfaces the first -cleanup failure, but the CLI's deferred `Close` has lost the paths it needs to -retry. The preparation and early preflight cancellation branches also discard -cleanup errors. The separate `atomicReplace` report helper ignores the results -of its deferred removes, so it can silently leave staging files or directories -on a cleanup failure. Recurring failures can accumulate disk blocks and inodes. -[Transcript preparation and cleanup](../../../internal/core/diagnostics/output_session.go#L163) -· [Finalization](../../../internal/core/diagnostics/output_session.go#L273) -· [Deferred close](../../../internal/cli/process.go#L41) -· [Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61) -· [Report replacement](../../../internal/core/diagnostics/output_session.go#L410) - -A disposable probe against the unmodified `OutputSession` forced a nonempty -staging directory. The first `Close` removed the payload and returned an error. -After the probe removed the extra entry, the second `Close` returned success -while the empty directory remained. The probe cleaned up its own files. - -Keep each path until removal succeeds or the filesystem says it is absent. Join -cleanup errors with preparation or publication failures and surface report -staging cleanup failures through the output owner. Distinguish a published file -with leftover staging from a failure to publish it. - -### 2. Blocked stream I/O extends resource lifetime past cancellation - -The input path uses `os.ReadFile` without a context. A FIFO can block before open -completes or hold an open descriptor while waiting for more input. Final -delivery similarly opens or writes a FIFO, or writes to the supplied stdout, -after the last cancellation check. With no peer or a peer that stops consuming, -the invocation retains its memory and any acquired stream descriptor. During -output it also retains the encoded transcript and rendered report buffers, and -the session cannot finish report finalization. A supervisor can therefore end up -waiting on canceled processes indefinitely. Prior subprocess probes reproduced -the blocked input, FIFO output, and stdout paths with SIGTERM and released them -by supplying or draining the streams. -[Transcript read](../../../internal/core/io/files.go#L9) -· [Final output check](../../../internal/cli/process_output.go#L54) -· [Output write and close](../../../internal/core/diagnostics/output_session.go#L318) -· [Output probe](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) -· [Input probe](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) - -Make open and readiness waits cancellable for streams Audita owns on supported -platforms. For caller-owned writers, define how the caller supplies cancellation -or drains the stream. Ensure canceled delivery reaches session finalization, -and account for any bytes already delivered. A detached write goroutine alone -would still retain the writer and buffer if the peer stays blocked. - -### 3. Backend admission leaves chunk fan-out and result retention unbounded - -Each module starts one proposal goroutine per chunk. A worker builds a section -transcript and the domain and adapter prepare request inputs before PromptKit -admission. Backend capacity limits admitted runs and active provider attempts -when configured. The pinned `internal/capacity/manager.go` imposes no limit for -backend IDs without a configured pool. With many chunks, local goroutines and -preparation grow with chunk count even when a backend has a small configured -queue. Without a pool, concurrent HTTP requests can also grow with chunk count, -bringing sockets, request timers, encoded prompts, and response buffers with -them. Timeouts and byte limits apply individually. The `examples/service` profile -illustrates explicit backend limits. -[Proposal workers](../../../internal/framework/runner/runner.go#L385) -· [Adapter request preparation](../../../internal/framework/llm/adapter.go#L113) -· [Backend capacity example](../../../examples/service/prompts.yml#L1) - -The proposal and validation channels and their companion maps are sized to the -chunk count. A slow leading chunk can keep many later proposal results, including -captured exchanges when correction repair is active, in the ordered pending map. -Validation can also start a goroutine for each ready chunk and holds completed -results in its channel until the join. PromptKit has already released those -calls' permits while Audita still holds their data. A long, finely chunked -transcript can raise client memory and descriptor demand even when each response -fits the per-call limit. Earlier probes substantiate the ordered backlog and the -interaction with a finite queue. -[Ordered buffering](../../../internal/framework/runner/runner.go#L460) -· [Validation workers](../../../internal/framework/runner/runner.go#L547) -· [Result collection](../../../internal/framework/runner/runner.go#L592) -· [Ordered backlog probe](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention) -· [Admission probe](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling) - -Consider a bounded dispatch window tied to explicit scheduling or operator -settings. Drain validation results while workers run before reducing channel -capacity. A bounded result window or earlier use of section-local identities -could reduce retention while preserving deterministic correction identifiers. -Preserve the module barrier and PromptKit's permit ownership. Decide explicitly -when backend saturation should wait or remain a recoverable capacity rejection. - -### 4. Whole-transcript copies amplify memory at input and publication - -`loadProcessInputs` reads the entire file into a byte slice. The parser then -validates the bytes, decodes the whole JSON document into generic maps and -slices, re-encodes the selected segment array, and decodes that array into typed -segments. Extra fields can occupy memory in the generic tree even if the typed -transcript discards them. The CLI keeps the raw input and typed source reachable -through its `processInputs` struct, which it passes into finalization. The raw -input's last consumer is the optional source debug artifact. At the other end, -both output encoders build one map per segment and a complete indented JSON -payload before delivery. A large transcript can run out of memory before the -first provider call or after all correction work has completed. -[Input load and ownership](../../../internal/cli/process_inputs.go#L25) -· [Parser](../../../internal/core/schema/transcript.go#L35) -· [Segment re-encoding](../../../internal/core/schema/transcript.go#L115) -· [Last source use and finalization](../../../internal/cli/process.go#L51) -· [Bare-array encoder](../../../internal/core/schema/transcript.go#L224) -· [Envelope encoder](../../../internal/core/outputschema/registry.go#L62) -· [Related parse fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) - -Decode the supported input shapes into typed segments without the generic -round trip and release source representations after their last consumers. Typed -output projections would remove the per-segment maps. For regular files, an -incremental array encoder can write into the prepared temporary file and publish -only after encoding succeeds. Preserve both schemas and decide how a writer-based -encoder would handle partial output on a stream. - -### 5. Summary-only runs retain a full grid of work and operation records - -`Runner.Run` eagerly allocates and initializes one `WorkUnitOutcome` per -configured module and chunk before the first module starts. The work-unit grid -retains nested operations and repair summaries until `processreport.Build`. -Completed modules also flatten the same operations into `ModuleResult.Operations`. -The CLI selects `SummaryOnly` only after the runner returns when reports are -disabled. The builder scans the operation and work-unit records to count them, -then skips detailed rows. On many-module, many-chunk runs, this incurs memory -proportional to the module-by-chunk product and all retained operation records, -including entries for work never started after early failure. This is in -addition to the debug decision history identified in the efficiency audit. -[Grid allocation](../../../internal/framework/runner/runner.go#L127) -· [Operation copies](../../../internal/framework/runner/runner.go#L264) -· [CLI report selection](../../../internal/cli/process_pipeline.go#L78) -· [Summary aggregation](../../../internal/framework/processreport/report.go#L41) -· [Decision retention](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history) - -When neither detailed reports nor the debug ledger is requested, fold finalized -work into invocation counters and derive not-started counts from the total -inventory. Retain the failed module identity for run-level errors. With details -enabled, consider one ownership path for operation records and deriving -not-started rows on demand, preserving the complete reporting grid and ordered -evidence. - -### 6. Debug capture adds memory and disk pressure - -Debug runs retain all logical calls and execution events in memory while also -writing several files per proposal, validation, and repair interaction. At the -end, the observation collector copies its records into a sorted snapshot. -`BuildLLMExecution` groups events by call and builds a second representation -for serialization. The JSON redactor creates encoded bytes, a decoded generic -tree, and final formatted bytes. Large event histories can produce a memory -peak while the transcript and the evidence needed for the correction ledger -are still live. Each debug invocation adds a run directory. Large or repeated -debug runs can consume space or inodes needed by the transcript and reports -when they share the filesystem. Once an artifact write fails, the CLI can -continue to collect and retain a warning for each later failure until -finalization, even if the destination stays full. -[Observer retention and snapshot](../../../internal/framework/llm/observations.go#L81) -· [Execution artifact assembly](../../../internal/framework/processreport/llm.go#L61) -· [JSON redaction](../../../internal/core/redaction/redaction.go#L53) -· [Run directories](../../../internal/core/diagnostics/run_dir.go#L34) -· [Interaction files](../../../internal/framework/llm/diagnostics.go#L35) -· [Warning accumulation](../../../internal/cli/process.go#L42) -· [Warning delivery](../../../internal/cli/process_output.go#L17) - -Reuse a single ordered set of observations for final aggregation, or stream -redacted records through a disk-backed artifact while preserving the final -summary. Coalesce repeated debug write failures and report which evidence was -lost. Give operators visibility into debug footprint and an explicit retention -or quota decision for runs that need it. Keep optional evidence failure distinct -from required output delivery. - -### 7. Correction repair replays growing history under per-correction budgets - -Each actionable rejection starts a branch from the original section exchange. -PromptKit's pinned `internal/domain/exchange.go` shares the immutable history -through parent references. Its `internal/usecase/runner.go` flattens that history -with the next turn during preparation, and the HTTP client encodes it into the -request body. Audita revalidates returned revisions and restarts the configured -turn allowance for the next rejected correction. A large section prompt is sent -again on every branch. If a branch uses `A` turns, starts with `S` bytes of -history, and each turn adds roughly `H` bytes, the cumulative logical repair -request payload is on the order of `A*S + A²*H` for that branch. Retries and -revalidation add work. With debug enabled, Audita writes the prepared history -again for each logical repair turn. Across many actionable rejections, this can -increase provider context usage, outbound traffic, and disk use even when every -call fits the configured attempt and time budgets. -[Per-correction configuration](../../config.md#settings) -· [Branch start](../../../internal/framework/runner/repair.go#L85) -· [Coordination and per-turn artifacts](../../../internal/framework/runner/repair.go#L146) -· [Revalidation](../../../internal/framework/runner/repair.go#L227) -· [Prepared snapshot](../../../internal/framework/llm/adapter.go#L152) -· [Pinned version](../../../go.mod#L11) - -Expose aggregate repair work and request size so operators can size runs. If a -work envelope is needed, consider an opt-in section or invocation budget that -accounts for revalidation as well as repair turns and records why any branch -stopped. For debug storage, consider recording a shared history once with -references sufficient to reconstruct the redacted request. Preserve the -conversation context used for editorial decisions. - -## Verification - -- The repository's complete `go test -count=1 ./...` suite passed on Go 1.27.3. - It includes output write and rename failure cleanup and release of backend - capacity after cancellation. The disposable probe in finding 1 exercises a - removal failure beyond those existing tests. - [Rename failure test](../../../internal/core/diagnostics/output_session_test.go#L98) - · [Transcript failure test](../../../internal/core/diagnostics/output_session_test.go#L394) - · [Capacity release test](../../../internal/framework/llm/adapter_capacity_test.go#L42) -- The pinned PromptKit `internal/llm`, `internal/capacity`, and - `internal/generation` packages passed their existing tests. These exercise - response byte limits and body closure, capacity contention and cancellation, - and cancellation during retry delays. - [Pinned version](../../../go.mod#L11) -- Local release entry points were reviewed without executing a release. The - wrapper replaces its shell with ReleaseKit, and the CI bootstrap registers an - exit trap to remove its temporary directory. - [Release wrapper](../../../scripts/release.sh#L10) - · [CI bootstrap](../../../.woodpecker/release.yml#L29) -- The scaling findings describe allocation and ownership from source. Peak RSS, - throughput, provider cost, and filesystem-specific failure frequency were - outside the review's measurement scope. diff --git a/docs/roadmap/audit/06-error-handling.md b/docs/roadmap/audit/06-error-handling.md deleted file mode 100644 index 72d9b46..0000000 --- a/docs/roadmap/audit/06-error-handling.md +++ /dev/null @@ -1,249 +0,0 @@ -# Error-handling audit - -Reviewed 2026-11-15 at `4ecb71c`. The review followed application and prompt -configuration, transcript parsing and chunking, model and validator execution, -correction repair, output publication, and release bootstrap. It used the -checked-in contracts, the pinned dependency source, existing tests, and -disposable failure probes. - -## Architecture and intended behavior - -Audita is a single-process Go CLI. CommandKit routes commands and presents -outcomes. ConfigKit loads and validates application settings. On `process`, the -CLI parses the transcript and prepares the selected outputs before normalizing -and running the pipeline. PromptKit owns preparation, backend admission, HTTP, -transient retry, schema validation, and structural repair. Audita's role adapter -maps those errors to a class and a safe reason while preserving the typed cause. -The runner isolates recoverable operations, cancels and joins peers after a -fatal error, and returns partial facts for reports. The CLI checks for -cancellation, delivers the transcript and reports, and selects the exit status. -[CLI outcomes](../../cli.md#inspection-and-outcomes) -· [Pipeline](../../internal/pipeline.md) -· [Model integration](../../integrations/openai-compatible-llm.md) -· [Reporting and output ownership](../../internal/diagnostics-reporting.md) - -The audit treated recoverable model errors, rejected corrections, and unavailable -validator assessments according to their separate contracts. Existing tests -confirm that one unavailable validation batch can leave other batches and later -modules usable, and that parent cancellation can stop active and waiting work -without publishing a transcript. Optional debug warnings can accompany exit -`0`, and a transcript can remain published if a later report write fails. These -are selected behaviors. The report boundary intentionally omits raw provider -causes, and input failures before initialization can leave no reports. -[Validation contract](../../internal/validators.md#llm-validator-batching) -· [Partial validation test](../../../cmd/audita/main_integration_test.go#L1091) -· [Cancellation test](../../../internal/framework/runner/cancellation_test.go#L48) -· [Optional debug test](../../../cmd/audita/main_integration_test.go#L1179) -· [Output lifecycle](../../operations.md#outputs-and-lifecycle) -· [Early failure contract](../../integrations/subprocess.md#result-handling) - -## Findings - -### 1. Finalization publishes completed reports after it knows delivery failed - -The CLI prepares a `completed` bundle for a successful pipeline and passes it -to `OutputSession.Finish`. `Finalize` tries the transcript and closes the output -session before writing reports. Even if a transcript write or cleanup has -already failed, it writes the initial `completed` bundle and only then asks for -and writes a failed-status bundle. A disposable probe supplied a short stdout -writer. The failed-bundle callback received a known transcript write failure -and could already read `completed` in all three report files. `Finish` returned -the short-write error and the second pass rewrote the reports to `failed`. -[CLI bundle selection](../../../internal/cli/process_output.go#L27) -· [Finalization order](../../../internal/core/diagnostics/output_session.go#L273) -· [Short-write check](../../../internal/core/diagnostics/output_session.go#L318) - -Reports should reflect a failure that is known before the first report write. -The existing nontransactional contract allows a later report failure to leave -mixed evidence. In this case the initial success claim is avoidable, and a -reader can observe it or it can persist if the failure rewrite cannot finish. -Select the failed bundle before publishing reports when transcript delivery or -cleanup has already failed. Keep a best-effort rewrite for errors discovered -during report publication, retaining the original error and any later failures. -If a failed bundle cannot be rendered, surface that error and preserve the -last written evidence. The initialization callback currently discards its -rendering error, and the finalization callback falls back to the previous -bundle, which can still say `completed`. -The missing phase for delivery errors is a separate, previously recorded issue. -[Output lifecycle contract](../../operations.md#outputs-and-lifecycle) -· [Initialization rendering fallback](../../../internal/cli/process_inputs.go#L105) -· [Finalization rendering fallback](../../../internal/cli/process_output.go#L46) -· [Failure retention test](../../../internal/core/diagnostics/output_session_test.go#L322) -· [Earlier output-phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) - -### 2. A provider request timeout is reported as the logical operation deadline - -The profile's `timeout_seconds` bounds one provider attempt. The distinct -`llm.operation_timeout` applies to preparation, capacity waiting, retries, -generation, and structural repair. Pinned PromptKit deliberately reserves -`ErrOperationTimeout` for the latter and exposes its HTTP attempt failure as -`ErrTransport` with the underlying deadline cause. After checking the native -operation sentinel, Audita checks a generic `context.DeadlineExceeded` before -`ErrTransport`, so the HTTP timeout takes the `operation_timeout` reason. -In PromptKit, the distinction is documented in `engine.go` and implemented in -`internal/llm/openai_compatible_client.go` and `internal/usecase/operation_scope.go`. -[Timeout settings](../../migrations/v2.md) -· [Native runtime contract](../../integrations/openai-compatible-llm.md) -· [Classifier](../../../internal/framework/llm/adapter.go#L228) -· [Pinned PromptKit](../../../go.mod#L11) - -A loopback provider withheld its response beyond a one-second profile timeout. -With a ten-second operation budget and zero retries, the CLI finished after -about one second. It returned `0` with a completed transcript and reported -the failed proposal as `operation_timeout`. Continuing the run is consistent -with both timeout classes being recoverable. The reason directs an operator -toward the logical operation budget when the attempt deadline expired. Use -the native sentinel for whole-operation timeouts, preserving parent cancellation -precedence. Classify an HTTP-only timeout as transport failure or adopt a -distinct safe reason for that cause if consumers need the distinction. -[Report message mapping](../../../internal/framework/processreport/report.go#L224) -· [Existing operation deadline and parent cancellation test](../../../internal/framework/llm/adapter_test.go#L406) - -### 3. Missing required credentials produce a generic setup message - -The pinned PromptKit use case identifies a missing required environment -credential through `ErrAPIKeyEnvMissing` and retains its configured variable -name in `internal/usecase/runner.go`. Audita maps it to `setup_failure` and a -generic safe message before delivering the operation result to the domain. A -synthetic profile with a required unset variable stopped before HTTP and -exited `1`. Stderr ended with `structured model setup failed`, the YAML report said -`Processing setup failed.`, and the interaction error artifact repeated the -generic adapter message. The cause is available to Go callers via `Unwrap`, -but the operator-facing process path does not report the missing requirement. -[Adapter setup classification](../../../internal/framework/llm/adapter.go#L243) -· [Typed cause](../../../internal/framework/structuredoutput/failure.go#L17) -· [Proposal error artifact](../../../internal/framework/proposal_generation/generate.go#L231) -· [Final CLI failure](../../../internal/cli/process_output.go#L61) - -The fatal class and exclusion of raw provider text are appropriate. The safe -diagnostic can still identify an unset required credential and the affected role -or profile. Map the native credential sentinel to an actionable safe message, -and include the configured variable name if obtained from safe profile or -backend metadata. Carry the typed cause internally and retain preparation at -the existing execution boundary, where role selection and credential validation -already occur. -[Credential ownership](../../internal/llm-runtime.md) -· [Recovery workflow](../../operations.md#recovery) - -### 4. Fatal model errors lose their chunk and operation coordinates - -Proposal and validation code records an operation reference even on fatal -returns. It can identify the section, role, validator, and batch. The runner -stores these operations per work unit, but its first-error path retains only -the error value. `executePipeline` reconstructs the fatal report origin from -the failed module without identifying which section or operation stopped it. -The report builder supports these coordinates for run-scope records. In the -missing-credential probe the run record included `grammar_1` and the stage -index, with no `chunk_index` or `operation`, even though the pre-dispatch -proposal invocation had both. Fatal validator batches lose their available -validator and batch coordinates at the same boundary. -[Proposal failure operation](../../../internal/framework/proposal_generation/generate.go#L116) -· [Validator failure operation](../../../internal/framework/validators/llm_validators.go#L176) -· [First error and cancellation](../../../internal/framework/runner/runner.go#L477) -· [CLI origin construction](../../../internal/cli/process_pipeline.go#L60) -· [Run origin serialization](../../../internal/framework/processreport/report.go#L155) - -Retain the primary error and its available `outcomes.Reference` together when -the runner selects the failure that stops the module. This lets the CLI identify -the failed operation without guessing from canceled peers or counting the fatal -error again as a recoverable work-unit error. Use module-level context for -setup and application errors with no narrower origin. -[Run and work-unit record contract](../../integrations/output-files.md#error-and-recovery-records) - -### 5. Invalid processing selections use the runtime failure exit status - -The CLI contract gives exit `2` to invalid input or usage and exit `1` to -runtime or output failure. `loadProcessInputs` maps malformed input and -configuration to usage errors. Processing readiness is checked later inside -`preflight.Prepare`, where a missing selected profile, a profile reference -that cannot be resolved, or an empty selected glossary for a required glossary -module can fail. `runProcess` treats every error at this boundary as a runtime -failure, together with output and cancellation errors. A probe selecting an -unknown proposal profile returned `1` with a specific profile diagnostic, no -provider calls, and no reports or transcript. The existing integration test -for a similarly invalid profile inheritance reference expects `2`. -[Exit status contract](../../cli.md#inspection-and-outcomes) -· [Readiness and output sequencing](../../../internal/framework/preflight/preflight.go#L41) -· [Selection validation](../../../internal/framework/preflight/preflight.go#L96) -· [CLI status selection](../../../internal/cli/process.go#L24) -· [Invalid profile inheritance test](../../../cmd/audita/main_integration_test.go#L1122) - -Distinguish rejected processing selections from context and destination failures -with a typed readiness error or a separate input-check step. Return usage for -invalid operator selections and preserve failure for cancellation, I/O, and -internal defects. Retain the existing early check and the ability to inspect -configuration without requiring a processing-ready pipeline. -[Inspection contract](../../cli.md#inspection-and-outcomes) - -### 6. An infeasible chunk target is recorded as an internal processing failure - -A positive `chunking.target_sections` can be valid configuration and infeasible -for the normalized input. The chunker rejects that combination and the runner -wraps the error with `chunk normalized transcript`. The CLI's run record defaults -to `phase: processing`, `reason: internal_failure`, and a generic message for -errors outside the structured model and cancellation classes. With one usable -segment and an explicit target of two, a CLI probe produced that failed report, -exited `1`, and made no model calls. This failure should remain visible when -reports are enabled because it occurs after initialization. The reported phase -and reason currently misidentify an operator-correctable constraint as an -internal defect in processing. -[Chunking settings](../../config.md#settings) -· [Feasibility check](../../../internal/core/chunking/sections.go#L143) -· [Runner phase wrapper](../../../internal/framework/runner/runner.go#L126) -· [CLI fallback classification](../../../internal/cli/process_pipeline.go#L60) -· [Existing input reason and phase ordering](../../../internal/framework/processreport/report.go#L244) - -Carry a typed chunking failure through the runner and map infeasible parameters -to `phase: chunking` and an input reason. Keep a separate classification for an -unexpected failure in the chunker. Align the exit mapping with the input -contract if the run distinguishes these classes at that boundary. The earlier -correctness finding about feasible targets being rejected concerns a separate -chunking decision. -[Run error phase contract](../../integrations/output-files.md#error-and-recovery-records) -· [Earlier chunking finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) - -### 7. CI bootstrap treats a failed Git status command as clean if stdout is empty - -The release job checks `test -z "$(git status --porcelain --untracked-files=all)"` -under `set -eu`. The `test` command's exit status controls whether the shell -continues. If Git emits a diagnostic on stderr and fails with empty stdout, -the test succeeds. A shell probe substituted a status command that exited -`128` with no stdout and confirmed that execution continued after the check. -The expected bootstrap behavior is to stop before dependency preparation if -cleanliness could not be established. Pinned ReleaseKit `ci-deliver --prepared` -independently checks the status command's error before delivery, limiting this -finding to the earlier bootstrap gate and diagnosis. -[CI bootstrap and delivery invocation](../../../.woodpecker/release.yml#L17) -· [Release procedure](../../release.md#binary-delivery) -· [Pinned ReleaseKit](../../../go.mod#L12) - -Assign the status output in a separate command and check the command result -before testing the content. Preserve the existing verification in ReleaseKit. - -## Related failure paths already recorded - -- Cancellation during FIFO or stdout delivery can be missed after the last - context check. A blocked FIFO input can also defer cancellation before - output preflight. The runner has a separate cancellation window before - committing module changes. - [Delivery](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) - · [Input](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) - · [Runner commit](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary) -- The failure path can drop completed proposal evidence behind an earlier - section. The internal preflight seam can also overwrite initialized reports - with empty files when a failed renderer is absent. - [Lost proposal evidence](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) - · [Missing renderer](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence) -- Output cleanup can discard removal errors or forget a staging path before a - later cleanup attempt. This differs from intentionally retaining requested - debug evidence or keeping a transcript already published successfully. - [Cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) - -## Verification - -The full `go test -count=1 ./...` suite passed on Go 1.27.3. The probes used a -temporary CLI build, a synthetic loopback provider, a disposable output-session -caller, and a shell simulation. They verified the failure conditions described -above without persisting tests or program changes. The release check was -reviewed locally against the pinned ReleaseKit implementation and exercised at -the shell expression, without running publication. diff --git a/docs/roadmap/audit/07-security.md b/docs/roadmap/audit/07-security.md deleted file mode 100644 index ec642fa..0000000 --- a/docs/roadmap/audit/07-security.md +++ /dev/null @@ -1,254 +0,0 @@ -# Security audit - -Reviewed 2026-11-15 at `27e58fa`. The review covered application and prompt -configuration, transcript and glossary input, model transport and responses, -credentials, output publication, diagnostic serialization, and release -delivery. Findings identify the deployment conditions that make each issue -relevant. - -## Architecture and trust boundaries - -Audita is a single-process CLI. The invoking account supplies filesystem -authority and chooses the configuration and destinations. ConfigKit captures -application sources and PromptKit captures prompt and provider configuration. -The transcript, glossary text, and provider responses are data. Audita renders -model requests for selected roles, validates model responses structurally and -through domain checks, and publishes transcript files and reports. The OS or -subprocess parent controls who may invoke the CLI and under which account. -Provider authentication uses configured environment variables, with bearer -tokens added by the transport. Repair may intentionally send the original -proposal exchange and branch history to a separately selected provider. -[Application structure](../../internal/overview.md) -· [Trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope) -· [Configuration sources](../../config.md#loading-and-composition) -· [Model integration](../../integrations/openai-compatible-llm.md) -· [Repair data flow](../../operations.md#processing) - -The audited input-to-model path reads transcript text as data, uses structured -schemas, checks candidate identity and applicable validators, and rechecks -protected terms when applying an approved edit. The embedded prompts tell -models to treat transcript text and repair evidence as untrusted data. Those -instructions guide editorial behavior. The runner maps model responses to -domain proposals and applies only approved edits. PromptKit validates the -initial endpoint, bounds per-response reads, and closes response bodies. For -host-backed prompt content files, PromptKit resolves the path relative to -captured provenance and the pinned ConfigKit opens it through a rooted -filesystem. -[Prompt data instructions](../../../internal/prompts/assets/shared/prompt_hardening.md) -· [Repair instructions](../../../internal/prompts/assets/shared/repair_contract.md) -· [Proposal application checks](../../../internal/framework/proposals/apply.go#L59) -· [Model and configuration versions](../../../go.mod#L9) -· [Transport limits reviewed previously](05-resources.md#architecture-and-lifecycle) - -## Findings - -### 1. Debug capture uses a shared path and can expose private artifacts - -Debug is opt-in, and its default parent is `/tmp/audita`. `NewRunDirectory` -accepts an existing parent and creates the per-run directory with mode `0755` -before applying the process umask. Snapshot and interaction files use mode -`0644`. A normal run using a synthetic transcript and umask `022` produced -exactly those directory and file modes inside a disposable private parent. -Under a shared, traversable parent, those permissions let other local users -read the transcript snapshots, prompts, and responses. Even when credential -redaction succeeds, these artifacts contain confidential data for some runs. -[Debug default](../../../internal/core/config/config.go#L106) -· [Run directory creation](../../../internal/core/diagnostics/run_dir.go#L35) -· [Source snapshot](../../../internal/core/diagnostics/run_dir.go#L97) -· [Interaction permissions](../../../internal/framework/llm/diagnostics.go#L84) -· [Documented debug contents](../../operations.md#debug-evidence) - -The predictable default parent is also a trust boundary. If it already -resolves to a directory writable by a different account, Audita accepts that -directory and later opens artifacts by path. Another account with directory -mutation rights can interfere with where evidence is written or with its -integrity. The random run ID helps avoid collisions. Directory ownership and -permissions need their own checks. -[Directory acquisition](../../../internal/core/diagnostics/run_dir.go#L43) -· [Run ID generation](../../../internal/cli/process_inputs.go#L142) - -Use a per-user location for the implicit debug root and create private run -directories and files by default. If a caller explicitly selects a shared -location, document the ownership and permission assumptions. For directories -that can change concurrently, keep writes scoped to the acquired directory. - -### 2. Provider redirects can cross the configured transport boundary - -The process command gives PromptKit no custom HTTP client. PromptKit v0.23.0 -constructs a Go client without a redirect policy and creates its generation -POST from a replayable byte reader. The Go client's default policy can follow -body-preserving redirects across origins. Its rules for forwarding authorization -headers operate independently of the body and use host trust rules rather than -the complete origin. A redirect can therefore replay transcript-bearing requests -to a different service. A downgrade on the same host can also retain the -authorization header while losing TLS protection. -[CLI transport selection](../../../internal/cli/process_inputs.go#L42) -· [PromptKit client construction](../../../internal/framework/llm/runtime.go#L103) -· [Pinned version](../../../go.mod#L11) - -This matters when a provider gateway responds with a redirect across a routing -or TLS boundary that the operator has not authorized for transcript data. The -configured destination can be trusted while the destination of the redirect -has a different audience or transport security. The dependency paths are -`internal/llm/openai_compatible_client.go` and `internal/llm/http_evidence.go`, -with redirect and header rules in Go's `net/http/client.go`. - -Consider a provider redirect contract that keeps generation on the selected -origin by default and rejects HTTPS downgrades. Where redirects are needed, -support explicit trusted destinations and define when credentials may be -forwarded. Keep the documented ability to deliberately select a local HTTP -provider separate from protocol changes during a request. -[Provider contract](../../integrations/openai-compatible-llm.md) -· [Local HTTP example](../../../examples/local/prompts.yml#L3) - -### 3. Report destinations can escape the identity checked by preflight - -Preflight checks report paths for collisions with input, configuration, and -other destinations. Each report replacement resolves the original path again -at publication. The concurrency audit verified a change of report directory -referent between initialization and finalization. If a subprocess processes -work in a directory whose entries can be changed by a lower-trust account, the -same behavior can direct a report write to an unintended file writable by the -Audita account, including a protected input or configuration file whose earlier -collision check passed. A private staging file protects incomplete bytes. The -final rename still uses the later resolution. -[Destination selection](../../../internal/core/diagnostics/output_session.go#L60) -· [Report finalization](../../../internal/core/diagnostics/output_session.go#L273) -· [Path resolution and rename](../../../internal/core/diagnostics/output_session.go#L410) -· [Earlier referent-change evidence](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) - -Bind publication to the directory and destination semantics checked at -preflight. For shared output directories, use directory-relative operations or -reject an unsafe change before writing. Preserve intentional in-place -replacement and the selected support for output symlinks. Use an -invocation-owned output directory when a subprocess parent handles -lower-trust files. -[Output lifecycle](../../operations.md#outputs-and-lifecycle) - -### 4. Raw structured debug responses can retain an encoded credential - -PromptKit returns the model message as `RawOutput`. The role adapter retains -that string as `FinalContent`, and debug writers store it beside the decoded -domain result. The artifact redactor parses the outer JSON artifact and -matches each string against the exact secret value and the escaped spelling -from Go's JSON encoder. It treats `final_content` as one string, even when that -field contains another JSON document. A credential present in a provider -response can therefore be masked in decoded fields but remain recoverable from -the raw response field when the provider used a different valid representation -of the same JSON string. The existing nested-content test covers the spelling -produced by Go's encoder. -[Raw output retention](../../../internal/framework/llm/adapter.go#L171) -· [Proposal artifact](../../../internal/framework/proposal_generation/generate.go#L158) -· [Validator artifact](../../../internal/framework/validators/llm_validators.go#L148) -· [Repair artifact](../../../internal/framework/runner/repair.go#L173) -· [Redactor](../../../internal/core/redaction/redaction.go#L18) -· [Nested JSON test](../../../internal/core/redaction/redaction_test.go#L43) - -The condition is a provider returning credential-bearing content and an -operator retaining debug data. A gateway that echoes authentication material -into diagnostic model content can cause that exposure even if its JSON -encoder and Audita's differ only in escaping. Sharing the affected artifact -can disclose a reusable credential. Redact structured response content after -decoding it as JSON, or retain only the sanitized decoded view. If malformed -raw response text is useful for diagnosis, define how to retain it without -claiming decoded credential redaction for content that could not be interpreted. -[Credential handling requirement](../../policy/architecture.md#trust-boundaries-and-threat-scope) - -### 5. Configuration discovery extends trust to the main file and its neighbors - -Without `--config`, the working directory is searched before user and system -configuration. Once a main source is selected, conventional prompt and -glossary companions beside it are loaded as configuration. An explicit main -file still enables its companions. This is useful for an operator-managed -project, but it changes the trust boundary when a subprocess parent uses an -upload-controlled working directory or places trusted configuration beside -untrusted project files. Configuration discovered there can choose the model -destination and a credential variable present in the child process. In that -deployment, both text and a credential can be sent outside the destination -the integrator intended. A transcript passed as a data file to a trusted -working directory with a trusted configuration bundle does not carry those -configuration privileges. -[Application discovery](../../../internal/cli/app_wiring.go#L30) -· [Prompt imports](../../../internal/framework/llm/runtime.go#L36) -· [Captured main filesystem context](../../../internal/framework/llm/runtime.go#L75) -· [Subprocess guidance](../../integrations/subprocess.md#invocation) -· [Configuration composition](../../config.md#loading-and-composition) - -Document that the selected main file, its automatic companions, and each -searched working directory are part of the trusted operator configuration. -For an integration accepting uploads, use a trusted bundle and pass the -transcript independently. Give the child process only credentials appropriate -to the provider choices authorized for that integration. An explicit discovery -option could support deployments that need a single configuration file. - -### 6. Untrusted transcript filenames can become active Markdown - -The CLI puts the input path in the report snapshot. The Markdown renderer -escapes HTML, table separators, backticks, and line breaks for path fields. -Markdown link and image punctuation still reaches the renderer as syntax. The -stage-label renderer already escapes those constructs. If a subprocess keeps -an uploader-controlled filename and displays rendered `report.md`, the name -can render as a link with a label that obscures the actual path. That changes -what the operator sees and can direct navigation to an unexpected destination. -The concern is the integrity of the rendered report. -[Path in the snapshot](../../../internal/cli/process_output.go#L24) -· [Path rendering](../../../internal/core/reporting/markdown.go#L41) -· [Path escaping](../../../internal/core/reporting/markdown.go#L191) -· [Stage label escaping](../../../internal/core/reporting/markdown.go#L132) -· [Existing stage-label test](../../../internal/core/reporting/markdown_test.go#L101) - -Render paths as literal text with a context-appropriate Markdown encoder. -Keep exact path values in structured artifacts when the output contract calls -for them. - -### 7. Untrusted input size and blocking I/O can exhaust subprocess workers - -The transcript is read whole with `os.ReadFile`, then decoded and re-encoded -before chunking. Backend admission and per-response byte limits apply later. -For a parent accepting untrusted uploads without an input or process memory -budget, a sufficiently large valid transcript can exhaust the child or a -shared worker host before model calls are reached. A local integration that -allows input paths to resolve to a FIFO can also lose a worker to a read that -outlives cooperative cancellation. The size and cancellation paths were traced -in the resource audit, and the FIFO behavior was verified in the concurrency -audit. -[Input acquisition](../../../internal/cli/process_inputs.go#L46) -· [Whole-file reader](../../../internal/core/io/files.go#L9) -· [JSON acquisition and conversion](../../../internal/core/schema/transcript.go#L35) -· [Memory finding](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication) -· [FIFO finding](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) - -Keep the CLI suitable for trusted large jobs by avoiding redundant whole-input -copies. Integrations accepting untrusted input can enforce their own upload and -process budgets and require input from controlled storage. An application-level -input budget should be an explicit deployment choice. Where stream inputs are -supported, make their acquisition and reads cancellable. -[Subprocess timeout guidance](../../integrations/subprocess.md#invocation) - -## Release boundary and verification - -Release CI verifies the event commit and runs the tagged ReleaseKit version with -a delivery credential. The checked-in procedure relies on the forge and CI -operator to authorize who can create release tags and receive the delivery -secret. The local review covered the workflow, tag/commit checks, pinned -executable selection, and shell quoting. Forge permissions and running CI -secret settings were outside the observed state. The earlier bootstrap status -error is covered by ReleaseKit's independent prepared-checkout validation -before delivery. -[CI workflow](../../../.woodpecker/release.yml#L5) -· [Publication guards](../../policy/release.md#publication-guards) -· [Delivery procedure](../../release.md#binary-delivery) -· [Earlier bootstrap finding](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) - -- The complete offline `go test -count=1 ./...` suite passed on Go 1.27.3. - Existing tests exercise credential handling, redaction, filesystem collision - checks, cancellation, and model-validation boundaries. - [Credential redaction test](../../../cmd/audita/main_integration_test.go#L965) - · [Collision check test](../../../internal/core/diagnostics/output_session_test.go#L14) - · [Validation and transport test](../../../internal/framework/validators/llm_integration_test.go#L25) -- `go mod verify` verified cached module content. The pinned dependency source - was reviewed at the boundaries above. The checksum check does not establish - advisory status. -- A normal synthetic debug run substantiated the artifact modes. Other findings - are grounded in the cited source and contracts or in evidence already recorded - by the earlier audits. The review used local code and files throughout. diff --git a/docs/roadmap/audit/08-testing.md b/docs/roadmap/audit/08-testing.md deleted file mode 100644 index 3426693..0000000 --- a/docs/roadmap/audit/08-testing.md +++ /dev/null @@ -1,355 +0,0 @@ -# Test coverage and effectiveness audit - -Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to -assess protection against consequential failures, test ownership, fixture -fidelity, and maintenance cost. The findings are follow-up options. -[Testing policy](../../policy/testing.md) - -## Architecture and measured coverage - -Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit -loads application settings, and core packages parse, normalize, and chunk the -transcript. The runner coordinates correction proposals, validators, repairs, -and edit application. The role adapter uses a shared PromptKit engine, and the -CLI publishes the transcript and reports through an output session. Tests use -pure domain inputs, temporary files, an injected HTTP transport or loopback -provider, and subprocesses. Release tests in this repository inspect the local -profile, while ReleaseKit owns the underlying delivery implementation. -[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) -· [LLM runtime](../../internal/llm-runtime.md) -· [Release procedure](../../release.md#binary-delivery) - -On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...` -run passed with `-coverprofile` and `-json` output directed to temporary files. -Go's cover tool reported **89.3%** -statement coverage. The JSON run log recorded 441 passing top-level tests and -210 passing named subtests. The package percentages below merge repeated -profile rows by source block before counting statements. - -| Package or boundary | Statement coverage | Audit observation | -| --- | ---: | --- | -| `cmd/audita` | 0.0% | The test executable handles signals in a helper. | -| `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. | -| `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. | -| `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. | -| `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. | -| `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. | -| `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. | -| `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. | -| `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. | - -The percentages describe statements reached in this run. The audit assigned -test value by the assertions and failure scenario. In particular, low coverage -in test support or thin forwarding wrappers does not justify standalone tests. -Existing higher-value checks include partial validation with surviving edits, -repair revalidation, typed failure causes, stable user-visible identities, and -the presence of a masked credential echo in an actual retained artifact. -[Coverage policy](../../policy/testing.md#coverage) -· [Partial validation](../../../cmd/audita/main_integration_test.go#L1091) -· [Repair integration](../../../cmd/audita/correction_repair_integration_test.go) -· [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10) -· [Credential echo](../../../cmd/audita/main_integration_test.go#L965) - -## Behavior that needs stronger verification - -### 1. Subprocess tests recreate the production executable boundary - -The subprocess suite launches the test executable. Its helper registers -SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with -the returned code. The production `run` and `main` contain their own copy of -that wiring and recorded zero hits. A regression in production-only signal -registration, argument slicing, or exit propagation can pass the subprocess -suite, including the cancellation case. The existing tests still exercise -real process boundaries and the assembled CLI. -[Production entry](../../../cmd/audita/main.go#L12) -· [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28) -· [Signal test](../../../cmd/audita/main_integration_test.go#L1059) - -Use the actual built executable for one representative startup and signal or -exit test, or have the helper invoke the same production startup function. -Keep the smaller existing seams for the rest of the CLI suite. - -### 2. Inspection failure tests accept the wrong exit classification - -`TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and -`TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero. -Both can pass if the integration maps invalid input and cancellation to the -same nonzero status. A temporary build given the same malformed operator -prompt file returned `1` via `promptkit list prompts` and `2` via `process`. -The CLI contract distinguishes input failure (`2`) from runtime failure (`1`). -The input test also sets a synthetic credential that its malformed templates, -profiles, and paths never reference, so the absence assertion provides no -evidence that a configured secret was found and redacted. -[Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24) -· [CLI exit contract](../../cli.md#inspection-and-outcomes) -· [CLI wiring](../../../internal/cli/app_wiring.go#L80) - -Assert the selected status and a stable diagnostic fragment or typed cause. -Clarify the intended delegated-command classification if it differs from the -CLI contract. Test redaction where a known value reaches the relevant channel, -as the existing credential echo test does, and remove the unused secret -assertion from these cases. -[Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965) - -### 3. Output tests primarily check the final artifact state - -The short-write CLI test asserts final YAML status, counts, and a reason code. -The replacement test checks the result after a failed-report pass. Both can -pass when finalization briefly writes `completed` reports after it already -knows transcript delivery failed. The short-write test also passes if the -required run-error phase is missing. Collision tests cover aliases present -during selection, while the finalization path resolves report destinations -again. Existing tests that change the transcript's file type exercise a -different path. Prior audits documented both the transient report state and -the changed report referent. -[CLI short-write test](../../../internal/cli/process_output_test.go#L79) -· [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207) -· [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14) -· [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435) -· [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed) -· [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) - -At `OutputSession`, observe the first report state after a known delivery -failure and verify that a changed report destination cannot defeat the checked -identity. At the CLI boundary, assert the exit and serialized run phase. The -failure-injection seam is useful because it observes publication state at an -externally consequential boundary. Preserve the documented nontransactional -behavior when a report error occurs after a transcript has been published. -[Output lifecycle](../../operations.md#outputs-and-lifecycle) -· [Run error fields](../../integrations/output-files.md#error-and-recovery-records) - -The stream tests open a FIFO peer before final delivery and use a payload -that fits in the pipe buffer. The cancellation tests exercise a caller -canceled before delivery or while the provider is active. A supported stream -blocked in delivery therefore falls outside their assertions. Verify bounded -subprocess cancellation and failed-status finalization after the supported -stream semantics are decided. Keep the existing small delivery tests for the -successful stream contract. -[FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13) -· [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131) -· [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) - -Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring -the transcript staging file or initializing the reports, and no hits in `Close` -after a temporary path removal error. The existing cleanup tests cover a write -or rename failure followed by successful cleanup. A well-placed preflight test -can observe output and report state when cancellation follows initialization. -At the output-session boundary, verify which cleanup errors are reported and -whether a temporary removal failure can be retried under the selected contract. -[Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61) -· [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394) -· [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) - -### 4. The timeout tests cover two of the three relevant deadline owners - -The adapter test distinguishes a shorter Audita operation budget from an -already canceled parent. Other adapter cases check provider statuses and -transport failures. They do not exercise a profile's HTTP attempt deadline -expiring under a still-live parent and a longer operation budget. The earlier -CLI audit observed that case labeled `operation_timeout`, even though -PromptKit reserves its operation sentinel for the logical budget. -[Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406) -· [Provider status test](../../../internal/framework/llm/adapter_test.go#L462) -· [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline) - -Put the deadline distinction at the adapter boundary using a controlled -transport and deliberately separated deadlines. Assert both recoverability -and the selected cause or reason. The CLI already has report mapping tests -and only needs representative coverage of the adapter-to-report connection. -[Timeout contract](../../integrations/openai-compatible-llm.md) - -### 5. The validator batch fixture misses the real numeric decoding boundary - -The subprocess mixed-response test and the validator reconciliation tests -protect sibling decisions when ordinary integer references are missing, -duplicated, or unexpected. The adapter has a separate overflow test with a -single out-of-range reference and no usable sibling. The gap is a response -with both a usable requested decision and an unexpected integer too large for -the Go index type. Native schema validation can accept that integer, but the -typed adapter decode can reject the whole batch before per-entry reconciliation. -The existing overflow test provides evidence for failure classification and -cannot establish sibling preservation. -[Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567) -· [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429) -· [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions) - -Exercise a mixed response through the actual provider envelope, schema, and -adapter decode. Assert the usable sibling's decision and the appropriate -disposition for the extra entry. A stub returning a pretyped integer cannot -cover this case. Keep smaller reconciliation tests for the ordinary identity -rules. -[Validator reconciliation contract](../../internal/validators.md#llm-validator-batching) - -### 6. Concurrency fixtures leave some schedules unproved - -The runner's proposal-order test waits for all proposals to start, then -releases three gates back-to-back in reverse section order. That establishes a -release order, while workers may deliver their results in a different order. -The test checks applied-change order by target segment and final text, without -asserting the proposal indices promised by its name. Order by segment can -survive a change to proposal index assignment because corrections are applied -within their sections. The runner has other fixed-chunk and application-order -checks, so the missing evidence is deterministic indexing under a demonstrably -different completion order. -[Parallel runner test](../../../internal/framework/runner/runner_test.go#L423) -· [Result assertions](../../../internal/framework/runner/runner_test.go#L473) -· [Application and recombination](../../../internal/framework/runner/runner.go#L300) -· [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants) - -The shared-backend test holds the first HTTP call, then gives the second call -a 30 ms deadline. It does not verify that the second call reached admission -or the wait before it timed out. On a delayed execution, an earlier timeout -can satisfy the same assertions. The runner cancellation test observes native -`ExecutionStarted` for both peers and one dispatch. That event precedes -admission. Twenty race-enabled repetitions of the selected capacity, -cancellation, and proposal-order tests passed. The repetitions exercise the -present fixtures, while the state-observation gaps remain. -[Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42) -· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67) -· [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96) - -Hold the earlier result until later completion is observable and assert the -actual assigned indices and section identities. At the adapter boundary, prove -that work reached the contested state before canceling it. Observe peak active -dispatches relative to a test-controlled limit and check that waiting work can -proceed or time out according to the configured contract. Keep wall-clock -deadlines as bounds for the test itself where possible. -[Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy) - -### 7. Representative transcript fixtures leave data-integrity boundaries open - -| Behavior | Current protection and gap | Follow-up | -| --- | --- | --- | -| Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) | -| A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) | -| An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) | - -### 8. Debug redaction assertions mostly inspect the outer serialization - -The subprocess test confirms that it actually captured a provider echo, then -checks command streams and retained files for the clear credential value. The -redactor unit test has a useful nested `final_content` case built with Go's -own JSON encoder. Neither asserts the decoded content of that nested field -when the provider uses another legal JSON representation. A literal-byte -absence check can pass while the credential survives there. This matters to -the documented diagnostic redaction boundary, even when a private debug -directory limits access on disk. -[Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965) -· [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43) -· [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) - -Use a synthetic credential and verify the semantic content after parsing the -retained artifact and any structured response field it contains. Own detailed -string representation cases at the diagnostic writer. Keep one real provider -test to prove that credential discovery and artifact redaction connect. -[Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope) - -## Candidates to remove or consolidate - -### 9. Delete proposal-order assertions on the test-only composition harness - -`TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness` -twice and checks indices that the harness itself assigns with `nextIndex++`. -It cannot detect an indexing regression in `Runner.Run`. The harness also -passes the full transcript into a fake module and delegates range filtering -to that fake. Production builds section-local requests. Related harness cases -verify a composition that the test constructs itself. Remove the index test -once the real runner test checks indices under controlled completion. Migrate -any remaining distinct input-preservation or stale-application assertion to -the existing runner or proposal API coverage before removing its harness case. -[Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14) -· [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143) -· [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356) -· [Stable application test](../../../internal/framework/proposals/apply_test.go#L36) - -### 10. Consolidate validator registry and asset metadata assertions - -`TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the -class comparison in `TestRegistryBuildsClassifiedValidators`. The second test -also requires the concrete validator to implement `ClassifiedValidator` even -though `ClassOf` supports a stable-key fallback. A behavior-preserving change -to that representation would fail the type assertion. Keep one table that -builds registered keys and checks effective classes against documented roles. -Preserve protective checks of concrete domain behavior and stable selector keys. -[Duplicate registry checks](../../../internal/validators/registry_test.go#L80) -· [Explicit class examples](../../../internal/validators/registry_test.go#L48) -· [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54) - -`TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already -enforced at registry construction. The native preparation tests give stronger -evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions` -checks an exact count of 11 prompts. That count detects additions while -allowing a same-size substitution. Prefer a check for each required prompt ID -and its preparability. Retain the tests for canonical schema asset agreement, -stable diagnostics data, and protective copy behavior. -[Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71) -· [Production construction](../../../internal/framework/responseschema/registry.go#L129) -· [Prompt metadata count](../../../internal/prompts/registry_test.go#L79) -· [Native preparation](../../../internal/prompts/registry_test.go#L13) - -### 11. Remove the tests of the release test's own set helper - -`TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined -in the same test file. It cannot detect a malformed release profile or -bootstrap behavior. A standard comparison over the decoded profile can retain -the documented target set, product and destination checks, and the credential -exclusion without a test of the comparison's implementation. If adding release -coverage, give it to the actual bootstrap and its exit handling. The known -Git-status failure is a representative offline shell boundary, and -ReleaseKit already checks the prepared checkout on its own boundary. -[Test of the helper](../../../scripts/release_test.go#L63) -· [Real profile assertions](../../../scripts/release_test.go#L10) -· [Delivery contract](../../release.md#binary-delivery) -· [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) - -### 12. Consolidate normalization fixtures at the domain and parser owners - -`TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers` -exercise the same rejection with different gap values that remain below the -configured maximum. One case can assert the unchanged transcript and the -meaningful merge counts together. Seven normalization golden tests also -repeat the conversion from `SourceTranscript` to `Transcript` in test code. -That conversion mirrors the CLI and cannot catch a regression in the CLI's -copy. The fixtures compare decoded values and contain useful domain assertions, -including exact text and categories. Keep the distinct domain cases, remove -duplicate fixtures after comparing assertions, and let the parser and one -assembled CLI case own source shapes and wiring. No automatic golden update -was found in these tests. -[Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11) -· [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101) -· [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329) -· [Production conversion](../../../internal/cli/run.go#L12) - -### 13. Narrow integration assertions to their distinct contracts - -`TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling -of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the -same decoded ID with different escaping would fail the test. Decode the -artifacts and assert stage, role, and evidence association. Keep end-to-end -checks for per-entry profiles, successive transcript state, and output -selection, while the reporting package owns Markdown layout and escaping. -[Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145) -· [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101) - -`TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure -scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure` -uses the same model script and also checks the distinct attribution in both -YAML documents. The selection matrix took about 6.9 seconds in the -instrumented run. Keep one detailed scenario with the attribution assertions -and use a smaller fixture for any duplicate combination once it still proves -that reporting and debug selections leave decisions and counts intact. The -shared fixture also writes a legacy glossary file and stores its path without -using it. Remove that unused fixture input separately from behavioral checks. -[Selection matrix](../../../cmd/audita/main_integration_test.go#L879) -· [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013) -· [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283) - -## Verification scope - -The full race-and-coverage run, 20 repetitions of the selected concurrency -tests, and the offline CLI classification probe completed successfully. The -audit used source inspection to identify false passes and refactor-sensitive -assertions. The full suite uses synthetic model responses, consistent with the -offline policy. Live-model editorial quality is a separate, deferred evaluation. -[Execution policy](../../policy/testing.md#test-execution-requirements) -· [Deferred evaluation](../future.md#live-model-editorial-evaluation) diff --git a/docs/roadmap/audit/10-equivalence.md b/docs/roadmap/audit/10-equivalence.md deleted file mode 100644 index 1aabef3..0000000 --- a/docs/roadmap/audit/10-equivalence.md +++ /dev/null @@ -1,289 +0,0 @@ -# Behavioral equivalence and shared abstraction audit - -Reviewed the application at `7d064a6`. Audita is a single-process Go CLI for -transcript polishing. The CLI loads configuration and owns invocation and -output coordination. Core packages handle transcript data, normalization, -chunking, and serialization. The framework runner coordinates proposal -generation, validation, repair, and application through explicit contracts. -The PromptKit adapter executes model requests, and reports are derived from -framework results. [Package map](../../internal/overview.md) · -[Pipeline](../../internal/pipeline.md) · -[Dependency direction](../../policy/architecture.md#ownership-and-dependency-direction) - -The review traced direct comparisons and separate tests with common expected -behavior to their production implementations. Each finding states what the -assertions cover and where a shared implementation could remove duplicated -policy or mechanics. The full existing suite passed with `go test -count=1 ./...` -on Go 1.27.3. [Testing policy](../../policy/testing.md) - -## 1. Share validation budget policy across measurement paths - -**Evidence.** `TestComposedValidationBudgetMatchesSerializedPayload` runs the -same estimator through the optimized and generic paths on a 12-item fixture. -For every positive budget up to the full payload cost, it compares complete -batches and oversized decisions, including retained context, batch boundaries, -and token counts. Its separate payload assertion checks one-based wire -references and preserved run-level proposal IDs. The fixture includes Unicode, -JSON escapes, neighboring segments, and glossary entries. [Budget comparison](../../../internal/framework/validators/llm_budget_test.go#L17) · -[Batching contract](../../internal/validators.md#llm-validator-batching) - -**Implementation.** `prepareValidationBatches` picks the composed path for the -concrete simple estimator. It premeasures serialized items and context prefixes, -then uses additive costs. The generic estimator path serializes proposed -payloads when fitting and partitioning. Both implementations trim neighbor -entries from the end before trimming glossary entries, reject items whose -required payload still exceeds the budget, and form contiguous batches in input -order. [Dispatch and partitioning](../../../internal/framework/validators/llm_batching.go#L17) · -[Composed fitting](../../../internal/framework/validators/llm_budget.go#L59) · -[Serialized fitting](../../../internal/framework/validators/llm_context.go#L74) - -**Opportunity and tradeoffs.** Give the context trim order one owner in -`framework/validators`. Share the greedy batch boundary rule if it can consume -measurements without obscuring the different cost models. Keep the composed -path additive and the generic path able to measure the actual serialized bytes -with any supplied estimator. The serialized measurement remains an independent -oracle for the optimized token calculation. Once policy is shared, an explicit -expected sequence of retained items and batch boundaries can catch policy -regressions that path-to-path equality alone would miss. - -## 2. Share regular-file staging between transcript and report publication - -**Evidence.** `TestSubprocessOutputPermissionsHonorUmask` checks the transcript -and all three reports against the same expected modes. It exercises creation -under two umasks and replacement of existing mode-`0600` files under each. -`TestFinalizationCollectsFailuresAndRetainsPublishedTranscript` separately -checks that successfully publishing the transcript survives a later report -failure. [Shared mode assertions](../../../cmd/audita/main_integration_test.go#L1202) · -[Partial finalization](../../../internal/core/diagnostics/output_session_test.go#L207) - -**Implementation.** `OutputSession.Prepare` opens a payload in a private -temporary directory next to the transcript destination. It creates the payload -with mode `0644` subject to umask, then copies the existing file's permission -bits when replacing a transcript. `writeTranscript` later writes, closes, and -renames it. Reports use `atomicReplace`, which implements the same regular-file -staging within one call and has its own cleanup and error handling. -[Transcript preparation](../../../internal/core/diagnostics/output_session.go#L141) · -[Transcript publication](../../../internal/core/diagnostics/output_session.go#L347) · -[Report replacement](../../../internal/core/diagnostics/output_session.go#L410) - -**Opportunity and tradeoffs.** Factor temporary-file creation, permission -handling, close, rename, and cleanup into a small private type in diagnostics. -Let the transcript session hold it across processing and let each report -replacement own it for one write. The caller still owns destination selection, -the transcript's final type check, streaming, publication order, and reporting -of partial failures. Keep subprocess assertions on the final files and the -failure-order test at the session boundary. [Output lifecycle](../../operations.md#outputs-and-lifecycle) · -[Related output ownership finding](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) - -## 3. Resolve model segment references in one place - -**Evidence.** `TestModuleProposalMapsLocalReferencesWithoutGuessing` sends IDs -`1` and `2` for segments with transcript IDs `71` and `903`. It verifies that -returned positions map to canonical IDs, while returning `71` as a position -produces an invalid target. `TestRepairRequestRefreshesPreviewAfterRetarget` -uses canonical IDs `81` and `104` and checks that a repair response targeting -position `2` correctly supplies the second segment's text to the next attempt. -The separate retarget-and-approve test checks preservation of the proposal -index and an unrelated approval. [Generation mapping](../../../internal/framework/proposal_generation/module_proposal_test.go#L102) · -[Repair mapping](../../../internal/framework/runner/repair_test.go#L131) · -[Repair identity](../../../internal/framework/runner/repair_test.go#L52) - -**Implementation.** `GenerateCandidates` and the repair callback both check -that a correction's `id` is a positive in-range position and index the working -section at `id - 1`. Both leave the target ID at zero for an unresolved -position so proposal validation can handle it. [Generation translation](../../../internal/framework/proposal_generation/generate.go#L189) · -[Repair translation](../../../internal/framework/runner/repair.go#L216) - -**Opportunity and tradeoffs.** Put the conversion next to the correction -response type in `proposal_generation`, which repair already imports. A small -lookup can express this protocol without making the core transcript type aware -of model references. Preserve the raw reference for the conversation, the -existing invalid-target handling, and revalidation after retargeting. Repair's -`proposal_ref` and validation batch indices have their own namespaces. Keep -nonsequential canonical IDs in both caller tests so a positional identity bug -fails even after the callers share the lookup. [Module proposal contract](../../internal/modules.md#proposal-generation-ownership) - -## 4. Share structured failure facts across model operations - -**Evidence.** The generation and validation tests establish that recoverable -structured failures produce an operation with a stable reason while the caller -can continue processing. The validation case checks approvals on either side -of the failed batch, and the repair terminal-outcome table checks that a -recoverable provider failure ends that branch as unavailable. Fatal setup -fixtures check call facts at the generation boundary and previous decisions, -attempted batches, and preservation of the underlying cause at the validation -boundary. [Recoverable generation](../../../internal/framework/proposal_generation/generate_test.go#L201) · -[Recoverable validation](../../../internal/framework/validators/llm_partial_test.go#L45) · -[Repair outcomes](../../../internal/framework/runner/repair_test.go#L166) · -[Fatal generation](../../../internal/framework/proposal_generation/generate_test.go#L218) · -[Fatal validation](../../../internal/framework/validators/llm_partial_test.go#L137) - -**Implementation.** `structuredoutput.Failure` already owns the typed class, -safe reason and message, and wrapped cause. Generation and validation each -unwrap the error, copy its reason or substitute `internal_failure`, and check -recoverability separately. Repair repeats the extraction with a different -edge case. A typed failure with an empty reason retains that empty reason in -repair, and repair checks recoverability by class alone. [Failure type](../../../internal/framework/structuredoutput/failure.go#L17) · -[Generation projection](../../../internal/framework/proposal_generation/generate.go#L172) · -[Validation projection](../../../internal/framework/validators/llm_validators.go#L176) · -[Repair projection](../../../internal/framework/runner/repair.go#L187) - -**Opportunity and tradeoffs.** Put extraction of reason and class in a small -`structuredoutput` helper and keep each caller's decision about what to do with -the facts. Generation can return a failed response, validation can mark the -affected batch unavailable, and repair can end a single branch. Preserve the -wrapped cause, execution counts, and operation coordinates. Retain current -blank-reason handling until its desired semantics are decided explicitly. Keep -role tests with literal reason codes or constants and independently constructed -errors, including the tests for surviving work and fatal propagation. - -## 5. Give optional interaction artifacts one writer contract - -**Evidence.** Separate proposal, validation, and repair tests inject a write -failure. They require a diagnostic warning and preserve the successful proposal -operation, approved validation decision, or withdrawn repair branch. The -proposal and validation schema-metadata tests also check the same metadata -fields against each operation's registered response schema. -[Proposal write failure](../../../internal/framework/proposal_generation/generate_test.go#L391) · -[Validation write failure](../../../internal/framework/validators/llm_validators_test.go#L398) · -[Repair write failure](../../../internal/framework/runner/repair_test.go#L324) · -[Proposal schema metadata](../../../internal/framework/proposal_generation/generate_test.go#L143) · -[Validation schema metadata](../../../internal/framework/validators/llm_validators_test.go#L335) - -**Implementation.** Proposal generation and validators define equivalent -writer interfaces and four-path artifact values. Both convert the same fields -from `llm.DiagnosticsWriter` through adapters. Repair writes through that -concrete writer directly. File naming, serialization, and redaction already -live in the writer, while each caller separately converts writer failure to a -warning. [Proposal contract](../../../internal/framework/proposal_generation/generate.go#L23) · -[Validator contract](../../../internal/framework/validators/models.go#L84) · -[Proposal adapter](../../../internal/framework/proposal_generation/generate.go#L246) · -[Validation adapter](../../../internal/framework/runner/runner.go#L994) · -[Shared writer](../../../internal/framework/llm/diagnostics.go#L17) · -[Repair diagnostic write](../../../internal/framework/runner/repair.go#L178) - -**Opportunity and tradeoffs.** Define the writer interface and artifact path -value once at the diagnostics boundary so the concrete writer can satisfy it -without field translation. A common best-effort write helper could also report -the warning and return paths according to one rule. Keep the role-specific -payloads, coordinates, and credential lookups at their current boundaries. -In particular, the validation adapter resolves secrets lazily, and repair can -record the revision and revalidation decisions. Keep role-specific assertions -for warning delivery and processing outcomes alongside the real writer's tests. -[Writer tests](../../../internal/framework/llm/diagnostics_test.go) - -## 6. Use one set of checks for inherited configuration fields - -**Evidence.** `TestResolvedSettingsValidation` rejects an empty validation -profile at root and in a module entry. `TestCorrectionRepairSettingsResolution` -rejects a negative correction-repair budget at both locations and exercises -inheritance, explicit overrides, and clearing. Those invalid-case assertions -require a load error and do not inspect its diagnostic path. A separate test -checks that explicit null and the corresponding dependency survive in -resolution metadata. [Profile validation](../../../internal/core/config/resolve_test.go#L97) · -[Repair settings](../../../internal/core/config/resolve_test.go#L163) · -[Resolution metadata](../../../internal/core/config/resolve_test.go#L119) - -**Implementation.** `config.validate` repeats the same LLM checks for root -and per-module `LLM` values with different `ck.Path` prefixes. The duplicated -checks also include retry bounds, operation timeout, and optional repair -profiles. The validation prompt budget and description length checks repeat -at both levels as well. [Root checks](../../../internal/core/config/config.go#L161) · -[Per-module checks](../../../internal/core/config/config.go#L175) - -**Opportunity and tradeoffs.** Extract typed validation functions for the -shared settings and pass the path of each value. This keeps changes to one -field constraint from drifting between the two locations. Leave resolution, -explicit null handling, provenance, and role-specific profile readiness with -their respective owners. Preserve tests that load root and local documents -independently. Path assertions would catch a helper that applies a correct rule -at the wrong location. [Configuration contract](../../config.md) - -## 7. Use the same simple token counter for text estimates - -**Evidence.** `TestSimpleTokenEstimator` checks concrete counts for words, -punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization -merge-limit test independently expects that joining two single words exceeds a -one-token limit and records the corresponding skipped merge. This supports the -common word-count behavior. The normalization test does not independently fix -the punctuation weighting. [Counter examples](../../../internal/core/chunking/tokens_test.go#L7) · -[Merge limit](../../../internal/core/normalization/normalize_test.go#L155) - -**Implementation.** Normalization counts `strings.Fields` and then scans for -punctuation. Chunking counts word boundaries and punctuation in one pass. Both -count Unicode punctuation except ASCII apostrophes, hyphens, and underscores, -then add half the punctuation count using integer division. The chunking -counter also exposes those components for validation budget composition. -[Normalization counter](../../../internal/core/normalization/tokens.go#L14) · -[Chunking counter](../../../internal/core/chunking/tokens.go#L25) · -[Validation cost calculation](../../../internal/framework/validators/llm_budget.go#L27) - -**Opportunity and tradeoffs.** Reuse the single-pass counter and its component -count, either directly or through a small core package if neither caller -should own the dependency. Retain the consumer-specific budgets and tests for -merge behavior, chunk behavior, and serialized validation payloads. Keep -literal counts in the counter tests to detect a shared regression that would -otherwise affect all consumers together. This is a shared heuristic, so -altering its estimates has a wider effect than consolidating its implementation. - -## 8. Centralize transcript copies without moving the copy boundaries - -**Evidence.** `TestRunnerDoesNotMutateInputTranscript` checks the caller's -original text after a successful run. `TestApplyProposalsNoMutationOfInputTranscript` -compares the caller's transcript with a pre-edit snapshot after applying a -proposal. A separate application test asserts preservation of IDs, speakers, -timing, and categories on the edited output. The snapshot test uses the -production clone, and the runner test checks text only. Neither establishes -category slice independence or a convention for nil versus empty categories. -[Runner ownership](../../../internal/framework/runner/runner_test.go#L708) · -[Application ownership](../../../internal/framework/proposals/apply_test.go#L199) · -[Application metadata](../../../internal/framework/proposals/apply_test.go#L221) - -**Implementation.** The runner copies the input for a run, and `ApplyProposals` -copies the input it received before any edit. Both independently rebuild every -segment and copy its categories. The proposal copy preserves an explicitly -empty category slice, while the runner's append to a nil slice collapses it. -[Runner copy](../../../internal/framework/runner/runner.go#L972) · -[Application copy](../../../internal/framework/proposals/apply.go#L107) - -**Opportunity and tradeoffs.** Let the schema package own a transcript clone -that copies segment values and then copies mutable member data. Keep both -boundaries so callers retain the existing ownership guarantees. Choose and -check the treatment of empty slices explicitly when moving the code. Use -independent fixture values to verify metadata and ownership, since taking the -expected snapshot with the shared implementation can hide a shallow-copy -regression. - -## 9. Reconcile the protection rule at validation and application - -**Evidence.** The validator tests assert rejection for removing protected -terms and permit spoken-word repetition cleanup only when the identity -survives. `TestApplicationChecksCurrentTextWithSelectedProtection` demonstrates -the same preservation requirement at apply time. It applies one proposal and -skips the second, whose span is a fragment of a protected alias in the full -segment. `TestFeedbackReportsProtectionAndBlankResult` -also checks that a protection finding and a blank-result finding can both be -reported for one candidate. [Protected-term rejection](../../../internal/framework/validators/protected_terms_test.go#L31) · -[Repetition policy](../../../internal/framework/validators/protected_terms_test.go#L84) · -[Protection at application](../../../internal/framework/validators/vocabulary_test.go#L47) · -[Independent feedback](../../../internal/framework/runner/feedback_test.go#L126) - -**Implementation.** Matching and identity comparison already live in -`ProtectedVocabulary`. `ValidateProtectedTerms` starts with the proposed spans -and uses full-segment evidence for the glossary path and eligible spoken-word -repetition cleanup. The runner passes `AllowsChange` to application, which uses -the entire current segment and its replacement preview. Both assemble their -own decision about the before/after text and when to allow repetition. The -earlier correctness audit found that a span can hide the context needed to -recognize a protected name. [Validator check](../../../internal/framework/validators/deterministic.go#L100) · -[Application predicate](../../../internal/framework/validators/protected_terms.go#L47) · -[Runner wiring](../../../internal/framework/runner/runner.go#L328) · -[Span-only correctness finding](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity) - -**Opportunity and tradeoffs.** Use a single protection assessment that produces -a reason from the applicable before-and-after segment text and the effective -glossary view. Decide explicitly what evidence can support an independent -finding when a proposal has no usable preview. Moving ordinary validation to -segment scope is a behavior change for edits whose span would give a different -answer. Keep application-time reevaluation against current text, and keep -validation and application tests with their own expected outcomes and fixtures. diff --git a/docs/roadmap/audit/11-boundaries.md b/docs/roadmap/audit/11-boundaries.md deleted file mode 100644 index 13125d4..0000000 --- a/docs/roadmap/audit/11-boundaries.md +++ /dev/null @@ -1,263 +0,0 @@ -# Package cohesion and architectural boundaries audit - -Reviewed at `0f34231`. Audita is a single-process Go CLI. `internal/cli` -assembles configuration, inputs, model clients, and output destinations. Core -packages provide deterministic transcript, glossary, and configuration -operations. The framework runner coordinates module proposals, validators, -repair, and application. PromptKit integration lives in `framework/llm`. -Processing facts feed ordinary reports and opt-in debug artifacts. -[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) - -The audit used the code graph for discovery and call tracing, the `go list` -import graph to check dependency direction, and source and tests to check what -crosses each boundary. The import inventory covered 42 `internal` packages, -including test support. No core package directly imports a repository package -outside `internal/core`. The framework imports concrete correction modules -through its production factory and built-in validator metadata through its -runner and ledger. The findings focus on the responsibilities behind those -dependencies and on implicit coupling that the import graph cannot show. -Third-party implementations were outside the review. The existing suite -passed with `go test -count=1 ./...` on Go 1.27.3. -[Ownership policy](../../policy/architecture.md#ownership-and-dependency-direction) - -## 1. Give the validator runtime interface one owner - -`framework/contracts` declares the interface modules expose to the runner, but -aliases its request to `framework/validators.Request` and returns that package's -result. `framework/validators` declares a second interface with the same method -set. Its request ties together candidate evidence, resolved application -settings, the LLM client, the glossary matching cache, and diagnostic writer. -The implementation package also contains the built-in deterministic rule bodies -and a switch selecting built-in validator prompts. The named validator packages -mostly construct or wrap those implementations. As a result, a module that -only needs the validator port depends on the package that implements budgeting, -model reconciliation, and concrete rule policy. -[Module and validator ports](../../../internal/framework/contracts/contracts.go#L21) · -[Validator request and result](../../../internal/framework/validators/models.go#L30) · -[Duplicate interface](../../../internal/framework/validators/models.go#L95) · -[Built-in check implementation](../../../internal/framework/validators/deterministic.go#L15) · -[Built-in prompt selection](../../../internal/framework/validators/llm_validators.go#L222) · -[Editorial constructor](../../../internal/validators/editorial_review/validator.go#L11) - -Choose one shared owner for the port, decision types, and common cardinality -invariant. Give validators the evidence and collaborators they need through -that boundary. Keep reusable batch construction and reconciliation in the -framework, with concrete prompt selection and editorial rules owned by the -built-in validators. A small first change would establish one interface and -remove redundant aliases before narrowing the request. Preserve deterministic -ordering and the separate full and partial cardinality rules exercised by the -runner and validator tests. [Interface test](../../../internal/framework/contracts/contracts_test.go#L64) · -[Cardinality checks](../../../internal/framework/validators/models.go#L101) · -[Validator ownership](../../internal/validators.md#ownership) - -## 2. Preserve the execution class assigned to each validator decision - -The metadata package provides both a registry keyed by validator name and an -instance-level execution class. The runner uses `ClassOf`, which prefers a -valid explicit class, to order calls and decide which calls receive LLM -interaction diagnostics. The decision record carries the name but omits the -class. The correction ledger calls `ClassForKey` and buckets the record into -deterministic or LLM results. An injected validator with a valid class that -differs from its name's default can therefore execute under one classification -and appear under another. The built-in registry test checks that the two agree -for built-ins. The instance-override and ledger tests exercise their respective -inputs independently. [Class resolution](../../../internal/validators/metadata/metadata.go#L39) · -[Runtime ordering](../../../internal/framework/runner/runner.go#L913) · -[Decision recording](../../../internal/framework/runner/runner.go#L775) · -[Ledger bucketing](../../../internal/framework/processreport/correction_ledger.go#L247) · -[Built-in agreement](../../../internal/validators/registry_test.go#L97) · -[Instance override](../../../internal/validators/metadata/metadata_test.go#L35) · -[Ledger coverage](../../../internal/framework/processreport/correction_ledger_test.go#L12) - -Treat execution class as a runtime fact. Resolve it at the runner boundary and -carry it with initial and repair-revalidation decision records. The ledger can -then project the class used for processing. Let the built-in registry attach -default class metadata during construction. The shared validation port can own -the class type and expose instance overrides. Preserve the current ledger -field names and default behavior for unclassified validators. - -## 3. Put production module construction at the composition boundary - -`framework/runner` already takes a `ModuleFactory` interface. The concrete -`framework/modules` package imports every built-in module and installs the four -constructors in `NewFactory`, which the CLI selects. All four production -constructor adapters ignore their context and run spec. Configuration validates -selectable identities through the dependency-light `core/modulecatalog`. -Keeping the registry populated with all built-in implementations in the -framework gives a composition concern the appearance and import direction of -a reusable execution package. [Runner factory port](../../../internal/framework/runner/runner.go#L30) · -[Production registration](../../../internal/framework/modules/registry.go#L46) · -[Constructor adapters](../../../internal/framework/modules/registry.go#L74) · -[CLI composition](../../../internal/cli/process_pipeline.go#L50) · -[Module key catalog](../../../internal/core/modulecatalog/catalog.go#L5) - -Place the production factory alongside the built-in modules or in the CLI -composition code, and keep the injected factory interface in runner. A small -registration table can still support the existing constructor override seam. -Keep the config-facing catalog free of implementation imports, and use the -factory's tests to check that the catalog keys remain constructible. Module -packages can continue choosing their own proposal prompts and resolving their -explicit validator chains. [Factory and override tests](../../../internal/framework/modules/registry_test.go#L81) · -[Module contract](../../internal/modules.md#module-contract) - -## 4. Carry one checked processing plan from preflight into execution - -`preflight.CheckReadiness` reads configured glossary selections and lists the -required proposal and validation profiles, adding a repair profile when the -budget is enabled. The CLI supplies an inspection callback that reaches into -the concrete runtime engine to inspect the profile and its backend. Later, -`executePipeline` traverses the settings again and binds the same roles, -directly dereferencing profile pointers based on the earlier check. In between, -`contracts.ResolveModuleRunSpecs` derives instance names and the effective -glossary protection view, including the glossary module's exemptions. The -readiness and execution steps therefore agree by convention on which roles -and selections will run. Adding a role or changing when it is active requires -coordinated edits in both places. -[Readiness checks](../../../internal/framework/preflight/preflight.go#L96) · -[CLI inspection](../../../internal/cli/process_inputs.go#L125) · -[Run-spec construction](../../../internal/framework/contracts/contracts.go#L106) · -[Role binding](../../../internal/cli/process_pipeline.go#L20) - -Have processing preparation derive the effective inputs and active role -selections once. Preflight can check that plan, and composition can bind the -role clients from the checked values. Keep model-facing profile inspection in -the runtime adapter behind the existing narrow shape, and let preflight own -the whole-pipeline readiness decision. The plan can preserve the typed resolved -configuration as the provenance source. This keeps inspection independent of -run readiness and preserves the requirement to reject a later unusable -selection before acquiring outputs or dispatching any model work. -[Later-stage and repair-budget tests](../../../internal/framework/preflight/readiness_test.go#L16) · -[Glossary and readiness ADR](../../adr/0002-glossary-configuration-and-preflight.md#decision) - -## 5. Give glossary matching and protection a domain owner - -`core/glossary` derives identity-aware terms and exemptions. The actual -occurrence matcher and comparisons for identity and capitalization live in -`framework/validators.ProtectedVocabulary`. The LLM validation context builder -uses the same type to find mentioned glossary entries. The runner also builds -the combined validator `GlossaryView` and extracts its protection field for -apply-time checks. Matching, change protection, and prompt-context selection -have distinct callers, but the generic matcher and the apply-time policy are -contained in the validator implementation package. [Term derivation](../../../internal/core/glossary/vocabulary.go#L20) · -[Matcher and change comparison](../../../internal/framework/validators/protected_terms.go#L23) · -[Context matching](../../../internal/framework/validators/llm_context.go#L11) · -[Runner preparation](../../../internal/framework/runner/runner.go#L168) · -[Application call site](../../../internal/framework/runner/runner.go#L221) - -Make glossary occurrence matching and text-to-text protection assessment -reusable by placing them next to glossary identity and vocabulary. Keep -validator-specific evidence selection, decision reasons, prompt neighbors, -and budgeting with validation. The application path should receive the -protection predicate it needs and reassess against the current text. Preserve -longest-match exemptions and the explicit spoken-word repetition allowance. -Aligning ordinary validation and application on full-segment evidence would -be a separate behavior decision. [Application and exemption tests](../../../internal/framework/validators/vocabulary_test.go#L47) · -[Repetition contract](../../internal/validators.md#protected-name-repetition) · -[Evidence-scope finding](10-equivalence.md#9-reconcile-the-protection-rule-at-validation-and-application) - -## 6. Inject interaction diagnostics at the adapter boundary - -Proposal generation advertises an injectable interaction writer, yet its -production path passes a directory and constructs `llm.DiagnosticsWriter` -itself. It discovers redaction values by type-asserting an extra method on -its structured client. Runner validation uses a second adapter around that -writer and resolves secrets lazily. Repair constructs the concrete writer and -collects values from both repair and validation clients because its retained -payload includes validation evidence. These paths make proposal generation -and the runner know about artifact destinations and credential discovery even -though structured completion already has an explicit port. -[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L129) · -[Production proposal wiring](../../../internal/framework/proposal_generation/module_proposal.go#L55) · -[Validation adapter](../../../internal/framework/runner/runner.go#L994) · -[Repair write](../../../internal/framework/runner/repair.go#L173) · -[Repair credential sources](../../../internal/framework/runner/repair.go#L282) - -Give interaction capture one writer contract and let composition supply the -sink when debug is enabled. Put file naming, writes, and redaction in the -diagnostics adapter, with lazy credential sources supplied by the runtime. -Give proposal, validation, and repair callers responsibility for their own -semantic coordinates and evidence. Preserve repair's combined redaction scope -and the nonfatal result of optional write errors. The same boundary could -host debug execution projection, which currently makes `processreport` import -the concrete LLM observation model for a debug artifact. -[Artifact writer](../../../internal/framework/llm/diagnostics.go#L24) · -[Execution debug projection](../../../internal/framework/processreport/llm.go#L61) · -[Proposal write-failure test](../../../internal/framework/proposal_generation/generate_test.go#L391) · -[Repair artifact tests](../../../internal/framework/runner/repair_test.go#L291) - -## 7. Separate required publication from retained debug artifacts - -`core/diagnostics.OutputSession` owns required transcript publication and -enabled report initialization and delivery. The same type also exposes -`PrepareDebug`, which creates a retained run directory and writes invocation -and effective-config artifacts. That method uses none of the output session -state. Preflight invokes it through the session and returns the two resources -together. This couples APIs with different lifetimes and failure semantics. -The CLI integration test exercises independently selected transcript, report, -and debug destinations. [Output session](../../../internal/core/diagnostics/output_session.go#L45) · -[Debug method](../../../internal/core/diagnostics/output_session.go#L252) · -[Preflight handoff](../../../internal/framework/preflight/preflight.go#L76) · -[Independent destinations](../../../cmd/audita/main_integration_test.go#L844) - -First expose debug setup directly through the run-directory API. Where package -separation pays for itself, put the required output session in an output -adapter and keep retained evidence in diagnostics. Preflight or CLI composition -can still coordinate the sequence. Preserve the existing early required-output -checks, independent debug selection and warnings, transcript-before-report -publication, and cleanup of unpublished output. This split addresses which -package owns delivery. It is compatible with sharing the regular-file staging -mechanics inside that owner. [Operational lifecycle](../../operations.md#outputs-and-lifecycle) · -[Regular-file staging finding](10-equivalence.md#2-share-regular-file-staging-between-transcript-and-report-publication) - -## 8. Give run-level report facts one assembly path - -`processreport.Build` maps typed processing failures to report records and -computes counts. The CLI independently assembles the initial snapshot, derives -a failure reason and origin from runner state, then mutates status, phase, -error records, and counts during finalization. The filesystem session supplies -a formatted `Kind` string, so the CLI recognizes report initialization by -matching that string. For other required-output failures the report record -currently has an empty phase. Report meaning consequently depends on how the -failure reached the renderer, even though the YAML and Markdown share the -same presentation snapshot. [Report assembly](../../../internal/framework/processreport/report.go#L13) · -[Processing classification in CLI](../../../internal/cli/process_pipeline.go#L60) · -[Initial snapshot in CLI](../../../internal/cli/process_inputs.go#L94) · -[Final status and error mapping](../../../internal/cli/process_output.go#L75) · -[File failure type](../../../internal/core/diagnostics/output_session.go#L36) - -Have the runner and publication adapter supply structured failure facts with -the phase and source coordinates they know, and have `processreport` own the -initial and final mappings to the report model. The CLI can still choose when -to finalize the run and how to present its outcome. Keep the artifact renderer -and file delivery separate so failed-output handling can render an updated -snapshot with the same record and count rules. Preserve the distinction between -unknown pre-run counts and known counts, and keep work-unit and run-level -failures independent. [Delivery failure test](../../../internal/cli/process_output_test.go#L79) · -[Existing phase finding](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) · -[Report count contract](../../integrations/output-files.md#summary-counts) - -## 9. Put transcript conversion and output shape at their owning boundaries - -`schema.ParseSourceTranscriptJSON` owns source decoding and preserves optional -IDs. The processing CLI converts that object to the canonical segment model, -using one-based positions for missing IDs. `schema.ParseTranscriptJSONLenient` -has its own copy of the conversion. On output, `outputschema` chooses the -format but implements only the envelope. `schema.TranscriptToJSON` implements -the bare form and projects the same segment fields independently. This splits -the input model transition between CLI and schema, and the public segment -encoding between schema and the output encoder registry. -[CLI conversion](../../../internal/cli/run.go#L12) · -[Lenient conversion](../../../internal/core/schema/transcript.go#L90) · -[Bare encoder](../../../internal/core/schema/transcript.go#L224) · -[Envelope encoder](../../../internal/core/outputschema/registry.go#L62) - -Give the source-to-canonical transition a schema-level operation usable after -source acquisition, preserving the separate original source held for debug. -Let `outputschema` own both public envelopes and the common segment projection. -Keep parsing and normalization behavior independent of output-format choices. -Maintain separate fixtures for optional source IDs and the two public output -shapes. The current envelope test covers schema, version, and segment count, -so a compatibility refactor should verify payload fields independently as well. -[Lenient ID test](../../../internal/core/schema/transcript_test.go#L180) · -[Encoder shape tests](../../../internal/core/outputschema/registry_test.go#L18) · -[Encoding ownership](../../internal/output-schemas.md#scope-and-ownership) diff --git a/docs/roadmap/audit/12-deadcode.md b/docs/roadmap/audit/12-deadcode.md deleted file mode 100644 index 490b25d..0000000 --- a/docs/roadmap/audit/12-deadcode.md +++ /dev/null @@ -1,263 +0,0 @@ -# Dead code and obsolete functionality audit - -Reviewed at `9110aed`. Audita is a single-process Go CLI. CommandKit dispatches -the command and inspection surfaces. The application loads version 3 settings, -parses and normalizes the transcript, and passes it to a runner that chunks it -and coordinates proposals, validation, repair, and application. The PromptKit -adapter performs model calls. The CLI publishes the selected transcript format -and reports, with separate opt-in debug evidence. -[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) - -## Scope and verification - -The review used Go's `deadcode` analyzer from `x/tools` v0.52.0 on `./...`, -with and without test entry points, and Staticcheck 2026.2.2 for unused -declarations and ineffectual operations. Production reachability returned the -same 19 candidates for both shipped Linux targets, amd64 and arm64. The native -test-inclusive scan returned 16 candidates, including test-only declarations. -Typed reference counts helped locate write-only fields. The review traced -whole-value serialization, command registration, embedded assets, interface -assertions, and build selection before selecting findings. The FIFO test's -Linux and Darwin build constraint is intentional. `go vet -unreachable ./...` -produced no diagnostics, and `go test -count=1 ./...` passed on Go 1.27.3. -[Release targets](../../../releasekit.json#L7) · [Platform test](../../../internal/core/diagnostics/output_stream_test.go#L1) - -## 1. Retire the test-only direct continuation path - -`structuredoutput.Request.ContinueFrom` is read in `RoleClient.prepareCall`, -where it has its own opaque exchange check and passes the exchange to an -ordinary `Engine.Run`. The only request literals that set it are in -`conversation_test.go`. The supported correction-repair path passes the -captured proposal exchange to `Coordinator.Coordinate`. Its callback builds -requests without `ContinueFrom`, and the PromptKit coordinator supplies -history and owns the turn budget. A function-level scan sees the shared -request preparer as live, even though its continuation branch has no -production caller. [Request fields](../../../internal/framework/structuredoutput/call.go#L26) · -[Direct continuation](../../../internal/framework/llm/adapter.go#L127) · -[Direct callers in tests](../../../internal/framework/llm/conversation_test.go#L73) · -[Runner repair](../../../internal/framework/runner/repair.go#L143) · -[Coordinator adapter](../../../internal/framework/llm/coordinate.go#L13) - -Move meaningful coverage of sibling isolation, carried context, and structural -repair onto coordinated calls, then remove the request field and direct -continuation branch. Keep exchange capture for proposals and keep the live -coordinator's cancellation, final-turn, and accounting checks. -[Second continuation fixture](../../../internal/framework/llm/conversation_test.go#L113) · -[Coordinator tests](../../../internal/framework/llm/coordinate_test.go#L15) · -[Subprocess repair test](../../../cmd/audita/correction_repair_integration_test.go#L173) - -## 2. Remove proposal enrichment that the live handoff discards - -`GenerateCandidates` appends both a plain proposal and an enriched proposal for -each decoded correction. The only production wrapper, `ExecuteModuleProposal`, -passes `StartIndex: 0` and forwards the plain proposals, segment references, -exchange, and operations. The runner creates its own enriched proposals and -assigns indices in ordered section collection. Only tests consume generation's -`Result.Enriched` or supply a nonzero `StartIndex`. Those tests can still pass -when the runner assigns incorrect run-level indices. -[Duplicate construction](../../../internal/framework/proposal_generation/generate.go#L189) · -[Production wrapper](../../../internal/framework/proposal_generation/module_proposal.go#L55) · -[Runner assignment](../../../internal/framework/runner/runner.go#L522) · -[Generator index assertions](../../../internal/framework/proposal_generation/generate_test.go#L238) - -Return one proposal representation from generation and use the runner tests -for global ordering. The adapter integration fixture can prepare validation -input explicitly or exercise the runner handoff. Review the debug metadata -before dropping `StartIndex`, since its `start_index` field is emitted with a -constant zero through the current module path. Keep the actual local segment -references and operation facts. -[Adapter fixture](../../../internal/framework/proposal_generation/adapter_integration_test.go#L76) · -[Runtime ordering test](../../../internal/framework/runner/runner_test.go#L423) · -[Debug metadata field](../../../internal/framework/proposal_generation/generate.go#L148) - -## 3. Remove duplicate runner bookkeeping for unused module totals - -`runModulePipeline` increments a module proposal count and flattens work-unit -operations into a second slice. `Run` copies that count into each `ModuleResult`. -It uses the flattened operations on failures and, on success, flattens the -enriched work-unit operations again after apply-time attribution. No current -consumer reads `ModuleResult.ProposalCount`. Only tests read -`ModuleResult.Operations`. Ordinary reports consume `RunOutput.WorkUnits`, -and the correction ledger uses the module's decisions and applied and skipped -changes. The CLI serializes those projections, so the JSON tags on the unused -module fields never turn them into output fields. -[Count accumulation](../../../internal/framework/runner/runner.go#L536) · -[First operation copy](../../../internal/framework/runner/runner.go#L620) · -[Second operation copy](../../../internal/framework/runner/runner.go#L264) · -[Report work-unit input](../../../internal/framework/processreport/report.go#L51) · -[CLI artifact projections](../../../internal/cli/process_pipeline.go#L56) - -Remove the unused count and the module-level copies once the partial-failure -and accounting tests assert the corresponding work units. Retain the -per-module status and edit evidence consumed by reporting and the ledger. -[Ledger input](../../../internal/framework/processreport/correction_ledger.go#L70) · -[Partial-progress tests](../../../internal/framework/runner/runner_test.go#L685) - -## 4. Drop the unconsumed validation request envelope - -`BuildLLMValidationRequest` constructs a `LLMValidationRequest` holding the -validator name, type, module key, module instance, replacement policy, and a -slice of items. The caller takes only `.Items`. Batch encoding serializes -local-reference items, and diagnostics source their labels directly from the -validator and runtime request. The envelope is never passed to JSON encoding. -Each item also stores `SegmentID`, which is assigned after the segment lookup, -never read, and explicitly excluded from JSON. The result's duplicate -`ValidatorName` likewise has no production reader. The runner reads the -validator's `Name()` when recording decisions. -[Envelope construction](../../../internal/framework/validators/llm_validators.go#L242) · -[Runtime consumer](../../../internal/framework/validators/llm_validators.go#L61) · -[Fields and wire exclusions](../../../internal/framework/validators/llm_models.go#L20) · -[Payload construction](../../../internal/framework/validators/llm_prompt_variables.go) · -[Decision recorder](../../../internal/framework/runner/runner.go#L775) - -Have the builder return items and immediate decisions. Remove the write-only -fields after checking fixtures. Keep proposal indices for reconciliation and -the active diagnostic labels. The unused duplicate `framework/validators.Validator` -interface can also go. The module and runner use `contracts.Validator`. -[Unused interface](../../../internal/framework/validators/models.go#L95) · -[Active interface](../../../internal/framework/contracts/contracts.go#L29) · -[Payload parity test](../../../internal/framework/validators/llm_budget_test.go#L17) - -## 5. Remove structured-call fields superseded by execution observations - -The adapter still sets `Response.Provider` to a fixed string and copies prompt, -completion, and total token counts out of the run result. No production consumer -reads those response fields or serializes the whole response. The debug -execution projection gets usage from terminal events and the execution group. -`CallRecord.Prepared` and `CallRecord.FinalContent` are also populated even -though the observation collector copies only the call metadata. Interaction -artifacts get those payloads from `Response`, and a test currently asserts the -duplicated final content on the callback record. -[Response model](../../../internal/framework/structuredoutput/call.go#L52) · -[Adapter assignments](../../../internal/framework/llm/adapter.go#L153) · -[Collector projection](../../../internal/framework/llm/observations.go#L126) · -[Debug execution rendering](../../../internal/framework/processreport/llm.go#L61) · -[Test-only response and callback assertions](../../../internal/framework/llm/adapter_test.go#L167) - -Remove the unconsumed response summaries and callback payload copies. Move -tests to the native usage observation and actual interaction payload paths. -Preserve `Response.Execution` for ordinary report counts and retain prepared -and final response evidence in the opt-in interaction artifacts. -[Observation mapping](../../../internal/framework/llm/observations.go#L139) · -[Interaction payload](../../../internal/framework/proposal_generation/generate.go#L159) - -`Failure.Kind` is a related, explicitly transitional field. The adapter sets -it in tandem with `ReasonInvalidOutputExhausted` and production uses the reason -and class. The remaining reads are from `IsMalformedError` in tests. The debug -call record independently derives its `failure` value from the reason. Move -the tests to the typed reason and remove the compatibility field and helper. -Keep the diagnostic `FailureKind` values, the safe failure message, and the -wrapped cause. [Transitional field and helper](../../../internal/framework/structuredoutput/failure.go#L17) · -[Paired assignments](../../../internal/framework/llm/adapter.go#L279) · -[Debug failure mapping](../../../internal/framework/llm/adapter.go#L86) · -[Cause and classification tests](../../../internal/framework/structuredoutput/failure_test.go#L10) - -## 6. Resolve shared prompt variables once per model call - -`prepareCall` invokes `WithSharedVariables`, which loads all four embedded -instruction values and fills missing keys. The following loop reloads those -same assets and overwrites every one of those keys. For processing calls, -the first pass over shared instruction values is redundant. The helper is -still useful in CLI request inspection, where caller values take precedence. -[Processing merge](../../../internal/framework/llm/adapter.go#L123) · -[Shared values and merge semantics](../../../internal/prompts/registry.go#L82) · -[Inspection caller](../../../internal/cli/app_wiring.go#L100) - -Use a single merge in the processing path that preserves the embedded-values -precedence and the missing `ProtectedTermsJSON` default. Keep the inspection -merge semantics and retain the existing prompt rendering coverage. -[Operator prompt test](../../../internal/framework/llm/adapter_test.go#L291) - -## 7. Remove orphaned normalization and file helpers - -The analyzer and repository references agree that `normalization.shouldMerge`, -`core/io.ValidateWellFormedJSON`, and `core/io.WriteFile` are unreachable in -production and tests. The normalizer calls `shouldMergeWithReason` so it can -count skipped merge reasons. The CLI reads the transcript and calls the schema -parser, which already validates JSON. Current output delivery goes through -`OutputSession` and has a different lifecycle from `os.WriteFile`. -[Unused merge predicate](../../../internal/core/normalization/normalize.go#L115) · -[Live merge caller](../../../internal/core/normalization/normalize.go#L73) · -[File helpers](../../../internal/core/io/files.go#L17) · -[Live input boundary](../../../internal/cli/process_inputs.go#L46) · -[Live delivery boundary](../../../internal/core/diagnostics/output_session.go#L273) - -Delete the three declarations and retain the live merge checks, labeled input -read, and publication paths. Existing normalization and output tests cover -the behavior these helpers formerly duplicated or bypassed. -[Normalization limit test](../../../internal/core/normalization/normalize_test.go#L155) · -[Publication tests](../../../internal/core/diagnostics/output_session_test.go#L207) - -## 8. Remove the transcript parser variants that only tests call - -The CLI calls `ParseSourceTranscriptJSON`, keeps the source for debug, then -converts it to the processing model before normalization. Neither -`ParseTranscriptJSON` nor `ParseTranscriptJSONLenient` is reached from the CLI. -The strict wrapper and its sequential-ID helper are used by schema tests. The -strict wrapper dereferences source IDs even though source input allows them to -be omitted. The lenient wrapper duplicates the live conversion and has a -test-only caller. Neither wrapper is an external Go API because the package is -internal. [Live load](../../../internal/cli/process_inputs.go#L46) · -[Live conversion](../../../internal/cli/run.go#L12) · -[Strict and lenient variants](../../../internal/core/schema/transcript.go#L65) · -[Parser and round-trip tests](../../../internal/core/schema/transcript_test.go#L166) - -Move the live conversion to the schema package if a shared seam is useful. -Use the supported source decoder and explicit field expectations to test -serialized transcripts, and exercise missing-ID conversion through the live -entry point. That leaves the optional source IDs, normalization's sequential -output, and both output formats intact. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) · -[Boundary finding](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries) - -## 9. Clean up abandoned test support - -`internal/testsupport` has no imports from Go code anywhere in the repository, -including tests. All four filesystem and redaction helpers are flagged by both -reachability runs. Staticcheck also reports the old subprocess configuration -view helper, the runner entry wait helper, and two runner structured-client -fixtures with no callers. Current configuration and redaction tests run with -their own fixtures. The unused test doubles differ from the explicit interface -conformance fixtures in `contracts_test.go`, whose methods exist to satisfy -compile-time type assertions. -[Orphan helper package](../../../internal/testsupport/files.go) · -[Unused config view helper](../../../cmd/audita/main_integration_test.go#L319) · -[Unused wait helper](../../../internal/framework/runner/runner_test.go#L503) · -[Unused test clients](../../../internal/framework/runner/runner_test.go#L615) · -[Compile-time fixture](../../../internal/framework/contracts/contracts_test.go#L64) - -Delete the orphan package and unused fixtures. Keep the active assertions in -the configuration inspection, artifact, and runner suites. - -## 10. Resolve the inherited progress flag for `process` - -The built executable advertises `audita process --progress` as showing -progress messages. The process command registers no progress producer. The -pipeline attaches model observations only for debug capture, and the CLI's -processing summary and path messages are emitted at completion. In a local -normalization-only invocation, `--progress` and the default invocation returned -identical transcript bytes and empty stderr. The source trace is the evidence -for the model path too, since its observation hook is connected to the debug -artifact and no process progress callback is supplied. -[Command registration](../../../internal/cli/app_wiring.go#L84) · -[Processing observation wiring](../../../internal/cli/process_pipeline.go#L20) · -[Completion reporting](../../../internal/cli/process_output.go#L61) - -Treat this as an exposed option with no Audita process behavior. Either attach -it to a real process event source or clarify its availability in process help -while retaining acceptance for existing scripts. The delegated inspection and -smoke commands have their own reporting behavior. - -## Retained behavior - -The version gate rejects older explicit configuration versions. The flags -marked for removal in the migration guide are absent from command help and -registration. `chunking.min_section_tokens` is documented, accepted and -validated, and the chunker explicitly -retains it for compatibility even though partitioning does not use it as a -target. Keep that setting. Also keep the generic validator estimator path, -which supplies the independent serialized reference in the existing budget -comparison, and the debug-only metadata reached through serialization. -[Migration guide](../../migrations/v3.md) · -[Version gate](../../../internal/core/config/config.go#L131) · -[Chunk compatibility field](../../../internal/core/chunking/sections.go#L27) · -[Budget reference test](../../../internal/framework/validators/llm_budget_test.go#L14) diff --git a/docs/roadmap/audit/14-redaction.md b/docs/roadmap/audit/14-redaction.md deleted file mode 100644 index 172de0c..0000000 --- a/docs/roadmap/audit/14-redaction.md +++ /dev/null @@ -1,280 +0,0 @@ -# Credential-redaction removal and simplification audit - -**Reviewed:** 2026-11-15 at `a68a951`; application code remains at `c4ccd78`. -**Status:** audit complete; implementation pending. - -This review follows the maintainer's [resolved D1 decision](assessment.md#resolved-d1-withdraw-application-wide-credential-redaction). -It identifies machinery to remove or simplify after withdrawal of the -application-wide credential-redaction requirement. Native library behavior, -authentication, diagnostic evidence, and ordinary output correctness remain -independently useful. The architecture policy has already been updated. - -## Scope and verification - -The review traced credential discovery into CLI, reports, debug writers, -proposal generation, validation, and repair. Knowledge-graph searches and caller -traces were checked against source, including interface assertions and closures -that the graph does not resolve completely. Tests were read to distinguish -redaction-only expectations from authentication, serialization, artifact -association, and failure behavior. Documentation and previous audit remedies -were reviewed for assumptions that would reintroduce the withdrawn requirement. - -This was a static review: no code changes, live provider calls, credential-value -inspection, test execution, or performance measurements. Removed passes and -allocations below are visible in the code; their production cost is unmeasured. - -## 1. Remove credential discovery and tracking used only for substitution - -**Remove.** `Runtime.ConfiguredSecrets` enumerates every captured profile and -backend, caches environment-variable names with `sync.Once`, and reads their -values whenever called. It includes raw and trimmed forms and deduplicates -additional names. Discovery uses `context.WithoutCancel`. `RoleClient` separately -collects prepared credential names under a mutex, solely to supply this method. -None of this selects or supplies authentication to a provider. -[Discovery and reads](../../../internal/framework/llm/secrets.go#L11) · -[Runtime cache](../../../internal/framework/llm/runtime.go#L22) · -[Prepared-name tracking](../../../internal/framework/llm/adapter.go#L153) · -[Role accessor](../../../internal/framework/llm/adapter.go#L314) - -Consumers are CLI report/debug preparation and finalization, the error redactor, -proposal and validation writer setup, and combined repair/validation secret -collection. Remove `secrets.go`, the runtime cache fields, role credential fields -and locking, accessor methods, and secret-source assertions after updating those -consumers. Keep `OnPrepared` request capture, terminal-event synchronization, -call IDs, source-path tracking, and native engine execution; those have separate -consumers. Do not leave empty secret-provider methods or no-op redactors. -[CLI consumers](../../../internal/cli/process_inputs.go#L90) · -[Finalization](../../../internal/cli/process_output.go#L17) · -[Repair collection](../../../internal/framework/runner/repair.go#L282) - -Delete `TestConfiguredSecretsIncludesUnselectedBackends`. In -`TestRoleClientUsesNativeTargetAndRedactsCredential`, retain the assertion that -PromptKit sends the selected, trimmed Authorization value; remove the independent -redactor assertion. Keep native credential selection/optional-absence and -provider-free inspection tests. Update the credential-tracking description in -the [runtime guide](../../internal/llm-runtime.md) when removal lands. -**Catalog:** #63; prerequisite consumers are covered by §§2, 4–6. - -## 2. Replace sanitizing serialization with direct encoding - -**Remove / simplify.** The entire `core/redaction` package serves substitution: -raw/escaped secret expansion, longest-first matching, generic JSON decoding and -tree reconstruction, key-collision rejection, and YAML node/comment traversal. -Its JSON path marshals, decodes with `UseNumber`, walks, then marshals again; -its YAML path parses and re-encodes already serialized data. These passes run -even when the secret list is empty. Matching keys and fixed strings also causes -the schema corruption already recorded in catalog #9. -[Redactor](../../../internal/core/redaction/redaction.go#L18) · -[Report YAML](../../../internal/core/reporting/report.go#L181) - -Use direct JSON serialization in `RunDirectory` and `DiagnosticsWriter`, and -direct typed YAML serialization for ordinary reports. Write already-produced -YAML bytes without another parse/walk/encode pass. Remove secret parameters and -stored redactors. In Markdown, remove substitution and the redactor parameter -to `writeMatrix`, retaining the independent text/HTML/table escaping. -[Run artifacts](../../../internal/core/diagnostics/run_dir.go#L70) · -[Interaction artifacts](../../../internal/framework/llm/diagnostics.go#L84) · -[Markdown](../../../internal/core/reporting/markdown.go#L27) - -Retain serialization errors and contextual write errors, valid artifact -envelopes, numeric and boolean types, optional fields, and raw `final_content` -as a string even when its contents are malformed model JSON. No nested-content -sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective -configuration rendering; only Audita's subsequent YAML transformation goes away. - -Move useful scalar/schema and failed-write assertions from redactor tests to -the owning report/artifact tests before deleting the package. Update -[diagnostics/reporting internals](../../internal/diagnostics-reporting.md). -**Catalog:** #3 and #9; preserve #26's independent Markdown correction and -coordinate with #33's redundant-serialization work without resolving D5. - -## 3. Save the acquired source snapshot without reconstructing it - -**Simplify.** `WriteSourceTranscript` converts the acquired bytes to a string, -decodes a generic JSON tree, then sends it through the redactor's second -serialization/decode/walk/serialization sequence. The comment explicitly ties -this reconstruction to avoiding unredacted bytes. The CLI already has both the -acquired source bytes and parsed source, and calls this writer only for debug. -[Snapshot writer](../../../internal/core/diagnostics/run_dir.go#L96) · -[Caller](../../../internal/cli/process.go#L54) - -Write the acquired bytes directly to the source artifact, preserving the input -that was actually read. Keep parsed-source and normalized artifacts as separately -serialized typed views. Do not reopen the input file, which could have changed, -or preserve a second parser as a validation gate. Broader input-buffer lifetime -work remains catalog #31's concern. - -Move source/normalized snapshot coverage out of the deleted redactor package. -Check source-byte fidelity at this boundary and decoded fields for the typed -views; preserve nonfatal debug-write behavior. Describe the source/parsed -distinction in the diagnostics guide when implemented. -**Catalog:** #31; depends on the direct writer changes in §2. - -## 4. Remove secret plumbing from writer and preflight contracts - -**Simplify.** Proposal generation discovers secrets through an optional client -interface and constructs an adapter-specific writer. Validation uses a lazy -`func() []string` closure and recreates a writer per interaction. Repair combines -two roles' secret lists for its writer. These are extra dependencies on model -clients for a filesystem concern. -[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L128) · -[Validation setup](../../../internal/framework/runner/runner.go#L703) · -[Validation adapter](../../../internal/framework/runner/runner.go#L994) · -[Repair writer](../../../internal/framework/runner/repair.go#L178) - -Remove the discovery assertions, closure, combined-role helper, and secret -constructor arguments. Complete the already-proposed common interaction writer -contract without preserving these seams. Role owners still supply their payloads -and stage/section/proposal/attempt/validator coordinates; the writer returns -artifact paths and write errors. Retain the optional diagnostic-error callback -and its synchronization, which report nonfatal write failures. - -Also remove `DebugInput.Secrets`, the corresponding `PrepareDebug` and -`NewRunDirectory` parameters, and secret captures in report-rendering closures. -Preflight readiness, initial/failed report construction, output-session cleanup, -and debug setup remain useful without any credential values. -[Preflight handoff](../../../internal/framework/preflight/preflight.go#L26) · -[Debug preparation](../../../internal/core/diagnostics/output_session.go#L253) · -[Report closures](../../../internal/cli/process_inputs.go#L95) - -Adapt proposal, validation, repair, and preflight fixtures to the smaller -contracts. Preserve role attribution, distinct repair artifact paths, and -nonfatal write-failure coverage. Update the diagnostics guide with the resulting -ownership. **Catalog:** #43, coordinated with #42 and #48; no retention or -execution-policy change is implied. - -## 5. Stop rewriting validator explanations sent to repair - -**Remove / simplify.** Inside `repairCandidates`, `safeFindings` first constructs -domain feedback, then replaces substrings matching credentials from both repair -and validation clients. This runs for the initial rejection and subsequent -revalidation findings, independently of debug capture. It changes the next model -request, not just a saved diagnostic. A matching substring in otherwise useful -validator text therefore changes the corrective instruction. -[Feedback transformation](../../../internal/framework/runner/repair.go#L111) · -[Repair request](../../../internal/framework/runner/repair.go#L124) - -Use `repairFindings` directly at both points and remove the wrapper and its -secret dependency. Retain per-proposal filtering, validator/reason labels, and -the explicit unavailable-assessment representation: operational unavailability -is not an actionable editorial explanation. Preserve bounded coordination, -revalidation, withdrawal, and sibling independence. -[Domain feedback](../../../internal/framework/runner/feedback.go#L53) - -Replace `TestRepairFeedbackRedactsValidationCredentials` with focused coverage -of actual repair feedback if existing feedback/request tests do not already -cover it. Assert explanation and proposal association through decoded request -data, without introducing an environment-driven passthrough contract or prompt -wording checks. Update the repair-flow description if needed. -**Catalog:** #64; coordinate removal of `repairSecrets` with §4. - -## 6. Simplify CLI presentation while retaining typed failure ownership - -**Remove / simplify.** `runProcess` invokes `Runtime.RedactError` for preparation -failures. `finishProcess` unconditionally discovers credentials and builds a -redactor, even without reports or debug; it rewrites warnings, final errors, and -announced report/debug paths. Removing these calls eliminates environment reads -and permits paths to identify their actual destinations. -[Preparation failure](../../../internal/cli/process.go#L37) · -[Final presentation](../../../internal/cli/process_output.go#L17) - -Present those existing errors and paths directly through CommandKit, preserving -typed causes, reporting modes, completion behavior, and exit classification. -Drop redaction-only parameters such as `finishProcess`'s `loaded` argument once -its only use disappears. Retain the required-output/nonfatal-debug distinction. -Simplify the path-announcement subprocess fixture; retain useful destination, -output-mode, and error assertions at their existing test owners. - -There is also a distinct information-loss boundary: `classifiedFailure` reduces -native setup errors to a generic message, and `Failure.Error()` returns only -that message. Deleting substring replacement does not itself make missing -credential diagnostics actionable. Address catalog #25 by carrying the native -setup detail into the surfaced error, while retaining `Class`, `Reason`, cause -unwrapping, cancellation precedence, and operation attribution. -[Classification](../../../internal/framework/llm/adapter.go#L228) · -[Failure wrapper](../../../internal/framework/structuredoutput/failure.go#L18) - -Do not indiscriminately unwrap every cause into every report. The compact -reason/message projection in `processreport` serves the ordinary report schema, -and structured failure types drive recoverability and peer cancellation. Retain -those purposes; revise blanket comments/tests prohibiting cause presentation -where they enforce only the withdrawn guarantee. Test setup diagnostics for -actionable context and typed classification, not exact incidental wording. -[Report projection](../../../internal/framework/processreport/report.go#L224) · -[Failure tests](../../../internal/framework/structuredoutput/failure_test.go#L10) - -**Catalog:** #65 and #25; coordinate with #22–24 and #41 without changing their -classification or report-schema scope. Update operations and internal runtime -documentation alongside the implementation. - -## 7. Retire obsolete assertions and documentation without losing coverage - -**Remove / simplify.** Redaction assertions are mixed into tests with independent -value, and current documentation still describes the implemented sanitizer. -Remove pure substitution tests and synthetic credential-echo fixture modes used -only for them. In mixed tests, preserve native authentication, decoded request -and response evidence, schema metadata, repair path uniqueness, scalar types, -failure counts, and optional-artifact failure behavior. - -| Location | Treatment when implementation lands | -| --- | --- | -| `core/redaction/redaction_test.go`, `llm/secrets_test.go` | Delete removed-mechanism tests; relocate useful writer assertions as described above. | -| `llm/diagnostics_test.go`, `proposal_generation/generate_test.go`, `runner/runner_test.go` | Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations. | -| `runner/repair_test.go` | Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary. | -| `core/reporting/report_test.go`, `llm/native_adapter_test.go` | Retain typed report encoding and actual native authentication; remove attached substitution assertions. | -| `cmd/audita/main_integration_test.go` | Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements. | -| `internal/testsupport/files.go` | Remove the already-unused secret scanners with catalog #61, rather than repurposing them. | - -Review [operations](../../operations.md), -[diagnostics/reporting](../../internal/diagnostics-reporting.md), -[runtime](../../internal/llm-runtime.md), -[contributor recipes](../../internal/cli-config.md), -[component overview](../../internal/overview.md), and the -[migration guide](../../migrations/v2.md) when code lands. Remove broad redaction -claims and obsolete implementation instructions. Preserve native configuration -inspection/redaction documentation and the effective-config artifact name; -neither requires a naming migration merely because local postprocessing goes -away. Historical audit evidence stays intact, with revised remedies in the -catalog and assessment. Do not add a negative redaction policy. - -**Catalog:** #3, #60, and #61, and supporting work for all findings above. -No new broad secret scanner or prompt-language change detector should replace -the removed tests. - -## Retained responsibilities and boundaries - -- **Native authentication and inspection:** PromptKit still owns selected - credentials and transport; ConfigKit/ckconfig/ckprompt own their normal - rendering and inspection. Keep Audita's `WriteEffective` call and standard - command wiring. Prepared snapshots contain messages and prompt metadata; - removing substitution does not require adding headers or environment dumps. - [Configuration rendering](../../../internal/cli/process_inputs.go#L118) · - [Command wiring](../../../internal/cli/app_wiring.go#L81) · - [Prepared snapshot](../../../internal/framework/llm/adapter.go#L309) -- **Ordinary filesystem behavior:** retain unique run-directory creation, - run-ID validation, stage-name-to-filename conversion, required-output staging, - destination checks, cleanup, and useful write errors. These prevent collisions - or incorrect writes. No extra private-root or hostile-filesystem mechanism - was found to remove in the debug writer. Catalog #4's previously proposed - private defaults lack a requirement under D1; leave current location/modes - unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain. -- **Domain and report behavior:** retain transcript parsing, validation, - feedback classification, typed operation failures, stable report identities, - Markdown escaping, and nonfatal optional diagnostics. Removing credential - substitution does not imply dropping these boundaries. - -## Suggested implementation order - -1. Replace serialization and source-snapshot reconstruction, removing writer and - report secret arguments together with their callers (§§2–4). -2. Remove repair-feedback and CLI substitution, then delete now-unreferenced - discovery state and the redaction package (§§1, 5–6). These are a coordinated - cleanup, not an interim design with empty secret lists or no-op methods. -3. Complete common interaction-writer ownership and actionable setup diagnostics - with catalog #43/#25. Update affected tests and canonical documentation in - each implementing change (§7). - -Use targeted boundary tests and then the full suite for the cross-package code -change. Check decoded artifact fidelity, raw source preservation, native auth, -repair feedback, report types, and failure behavior. No new product decision is -needed for these D1 removals; D2–D7 remain separate and unresolved. diff --git a/docs/roadmap/audit/assessment.md b/docs/roadmap/audit/assessment.md deleted file mode 100644 index bf2c8e6..0000000 --- a/docs/roadmap/audit/assessment.md +++ /dev/null @@ -1,338 +0,0 @@ -# Audit assessment and decision register - -**Reviewed baseline:** `faba8ab` (2026-11-15). **Status:** assessment and policy -decisions complete. D1–D7 were resolved on 2026-11-15 and are recorded below. -The D1 architecture-policy update and focused cleanup audit are complete; -application remediation and the associated operator-documentation updates -remain pending. -This document owns classification, recommended -scope, and pending decisions for the [catalog](catalog.md). The source audits -retain their evidence; their suggested remedies are not approved requirements. - -The catalog covers 110 source findings in 65 entries. Every catalog entry is -assessed below. The application implementation remains at the audited baseline -`c4ccd78`; later documentation includes the D1 policy revision and the focused -[removal audit](14-redaction.md), reviewed at `a68a951`. This review used -the source audits, current contracts and ADRs, and targeted implementation -inspection. It did not independently rerun all audit probes, benchmarks, or the -test suite. Reproductions reported by the audits remain attributed to them; -allocation/call-path observations establish opportunities, not measured -production performance. - -## Evaluation rules - -The maintainer approved the taxonomy and evaluation process, not every proposed -remedy. Correctness restores established or reasonably expected behavior; -observable change alone does not make a fix a new policy. Refactor size is not -an approval criterion. Efficiency and ownership improvements should proceed even -when substantial restructuring is appropriate. - -| Code | Category | Treatment | -| --- | --- | --- | -| C | Correctness | Fix substantiated departures from intended behavior. | -| E | Efficiency and resource use | Improve work, allocation, and retention while preserving domain and operator policy. | -| A | Architecture and ownership | Consolidate responsibilities and improve boundaries; retain distinct domain behavior. | -| T | Tests and verification | Protect consequential behavior at real boundaries; remove redundant and implementation-bound assertions. | -| M | Maintenance and documentation | Remove obsolete machinery and clarify implemented behavior. | -| P | Substantive policy or product change | Requires express maintainer approval before implementation. | - -**Ready** means suitable for implementation planning within the stated scope, -not implemented or validated by this document. **Investigate** denotes technical -work the agent can undertake without a product decision. D1–D7 are resolved; -there are no remaining open decisions in this register. **Closed** -denotes no remediation for the specified scope. Mixed entries deliberately -separate ready work from decisions and exclusions. - -Evidence is judged against the [architecture policy](../../policy/architecture.md), -[operations contract](../../operations.md), -[CLI contract](../../cli.md), -[output contract](../../integrations/output-files.md), -[validator contract](../../internal/validators.md), and relevant catalog sources. -An audit author's expectation alone does not establish a requirement. -Internal-only exposure reduces urgency, not the value of a valid invariant fix. -Tests and acceptance criteria must follow the [testing policy](../../policy/testing.md). - -The maintainer's three WONTFIX decisions remain controlling: #2 in full, -#6 for interruption of already-blocked I/O, and #18 for Unicode behavior of Go -regexp word boundaries. Using the appropriate standard primitive for Audita's -own requirement is legitimate; replacing or compensating for accepted Go -semantics is not selected. The -[agent policy](../../policy/agents.md#implementation-discipline) also excludes -invented hostile-filesystem and power-loss guarantees. Source recommendations -that conflict with these dispositions must not re-enter an implementation plan. - -## Entry assessments - -Each entry links to the catalog's original evidence. Category codes can combine; -the final column bounds the recommended remediation and distinguishes confirmed -behavior from proposed extensions. - -| Entry | Categories | Disposition | Assessment and scope | -| --- | --- | --- | --- | -| [1. Bind output writes to the checked destinations](catalog.md#1-bind-output-writes-to-the-checked-destinations) | C, A | Ready | The wrong-file reproduction and repeated report-path resolution substantiate application defects. Resolve paths with filesystem semantics, carry checked parent/destination information through publication, and share staging ownership. Preserve symlinks and intentional in-place replacement. No hostile-directory race-proof guarantee, cross-file transaction, or custom filesystem machinery is selected. | -| [2. Keep model requests within the configured transport boundary](catalog.md#2-keep-model-requests-within-the-configured-transport-boundary) | — | Closed: WONTFIX | Accepted Go HTTP redirect and credential-forwarding behavior. The source audit's proposed redirect restrictions do not authorize implementation. | -| [3. Protect credentials inside nested debug responses](catalog.md#3-protect-credentials-inside-nested-debug-responses) | A, M, T | Ready under resolved D1; original remedy superseded | The encoded-secret observation tested the withdrawn guarantee. The focused audit confirms removal of the local substitution package in favor of direct serialization, plus retirement of masking-only tests/docs. Keep malformed response evidence and native behavior; no nested sanitizer or response-omission policy. Discovery and repair/CLI consumers are separately tracked in #63–65. | -| [4. Make debug artifact permissions private by default](catalog.md#4-make-debug-artifact-permissions-private-by-default) | M | No remediation under D1 scope | The permission observation is factual, but hostile local filesystem access is outside the selected threat model. The focused audit found ordinary directory creation and filename handling to retain, not a redundant private-root mechanism. Leave defaults unchanged in D1 cleanup; any future location or permission proposal needs an independent justification. This does not close #1’s wrong-file correctness work. | -| [5. Define the trust boundary for discovered configuration](catalog.md#5-define-the-trust-boundary-for-discovered-configuration) | M; P | Ready for documentation; extension excluded | Automatic companions are intentional configuration composition, not a discovery bug. Explain that the searched locations, selected bundle, imports, and companions are trusted configuration, and keep uploaded transcripts separate. Disabling discovery or adding a new isolation mode is outside this remediation; no ConfigKit replacement loader. | -| [6. Make blocking transcript input and delivery cancellation-aware](catalog.md#6-make-blocking-transcript-input-and-delivery-cancellation-aware) | C; — | Ready in part; WONTFIX in part | Honor cancellation before regular-file publication and when blocked delivery returns; produce failure status even if bytes have escaped. Do not promise interruption of already-blocked standard I/O. Do not ban FIFOs, add detached writer goroutines, or revive the source audits' context-aware stream machinery. | -| [7. Publish truthful failure reports and preserve initialized evidence](catalog.md#7-publish-truthful-failure-reports-and-preserve-initialized-evidence) | C, A, T | Ready | Publishing completed reports after a known delivery failure contradicts their meaning. Select failed status before the first report write; preserve initialized evidence if rendering fails. Carry rendering errors explicitly and retain bounded best-effort finalization. Cross-file atomicity remains outside the contract. | -| [8. Prevent concurrent runs from clobbering ordinary reports](catalog.md#8-prevent-concurrent-runs-from-clobbering-ordinary-reports) | M | Ready for documentation under resolved D2 | The maintainer selected caller-owned destination isolation. Retain existing placement and document distinct report directories for concurrent jobs or retained run histories. Do not add automatic placement, locking, waiting, or competing-writer rejection. Ordinary checked-destination correctness remains #1’s independent scope. | -| [9. Preserve the ordinary YAML schema during redaction](catalog.md#9-preserve-the-ordinary-yaml-schema-during-redaction) | C, A, M | Ready through D1 cleanup | Schema corruption is real, but preserving the redactor is no longer the remedy. Remove application credential substitution from ordinary report serialization and preserve the typed schema. Retain meaningful output-schema tests; retire tests whose sole purpose is satisfying the withdrawn redaction promise. | -| [10. Reject contradictory validation decisions](catalog.md#10-reject-contradictory-validation-decisions) | C, A | Ready | Approved-plus-unavailable contradicts the existing acceptance invariant, although bundled producers are consistent. Establish an authoritative internal disposition and reject invalid internal results before use; derive redundant serialized approval as needed. This is an internal contract failure, distinct from an unavailable model assessment. | -| [11. Preserve usable validator decisions when extra references cannot be decoded](catalog.md#11-preserve-usable-validator-decisions-when-extra-references-cannot-be-decoded) | C, T | Ready | The mixed-response probe violates existing sibling-preservation rules. Retain exact numeric references until range reconciliation, accepting integral JSON number spellings without float rounding. Ignore extras before conversion to local indices. Preserve whole-response structural validation; do not add semantic retries or general malformed-response salvage. | -| [12. Use full-segment evidence for protected-name checks](catalog.md#12-use-full-segment-evidence-for-protected-name-checks) | C, A, T | Ready | The full-segment examples establish a mismatch with existing protection intent. Share glossary-owned assessment using usable before/after segment evidence; recheck current text at application. Preserve active-glossary exemptions and spoken-word repetition rules. Missing or ambiguous previews cannot justify a fabricated protection verdict; retain independently supported applicability findings. Empty-result previews can still provide evidence. | -| [13. Preserve transcript input fidelity during JSON shape detection](catalog.md#13-preserve-transcript-input-fidelity-during-json-shape-detection) | C, E, T | Ready | Generic numeric decoding demonstrably changes accepted IDs and processes ignored fields unnecessarily. Keep the selected array as raw JSON and decode typed fields directly. Preserve both input shapes, optional IDs, accepted ranges, and unknown-field handling; no new input restrictions. | -| [14. Honor cancellation at the runner commit boundary](catalog.md#14-honor-cancellation-at-the-runner-commit-boundary) | C, A, T | Ready | Canceled entry and precommit paths violate cooperative cancellation and stage ownership. Check natural boundaries and assemble results once, preserving earlier committed stages. Do not claim atomic synchronization between cancellation and filesystem or memory operations, or introduce forcible interruption of standard calls. | -| [15. Retain completed proposal evidence behind slow sections](catalog.md#15-retain-completed-proposal-evidence-behind-slow-sections) | C, E, A | Ready | Completed proposal evidence must survive independently of ordered dispatch. Retain it on interruption and assign deterministic identities after joining workers. Reducing within-stage head-of-line waiting is a valid efficiency change if identities, validation requirements, application order, and the module barrier survive; it does not authorize independent whole-pipeline traversal. | -| [16. Keep ownership of failed staging cleanup](catalog.md#16-keep-ownership-of-failed-staging-cleanup) | C, A, T | Ready | The cleanup probe demonstrates lost resource ownership. Keep temporary paths until removal succeeds or absence is established, and surface cleanup errors without losing the primary failure. Share cleanup mechanics with publication. No persistent scavenger or indefinite retry service is required. | -| [17. Preserve the complete time interval when merging segments](catalog.md#17-preserve-the-complete-time-interval-when-merging-segments) | C, T | Ready | Nested intervals demonstrably truncate speech coverage. Use the full interval for merge eligibility, output, and subsequent gaps. Preserve accepted overlap and merge rules; rejecting overlapping input would be a separate restriction. | -| [18. Match protected names at Unicode and punctuation boundaries](catalog.md#18-match-protected-names-at-unicode-and-punctuation-boundaries) | C; — | Ready in part; WONTFIX in part | Keep the accepted ASCII semantics of Go regexp word boundaries. Recognizing configured punctuation-edged spellings is an application matching defect: express appropriate literal-edge checks with standard primitives, preserving longest-match selection, exemptions, and case behavior. Do not turn this into Unicode boundary emulation or a replacement regexp engine. | -| [19. Detect overlapping occurrences when requiring a unique edit span](catalog.md#19-detect-overlapping-occurrences-when-requiring-a-unique-edit-span) | C, T | Ready | Require-unique means one possible literal location; the overlapping-span probe has two. Use standard search primitives to detect a second starting offset. This corrects Audita's choice of counting operation, not Go's nonoverlapping Count semantics. Preserve replace-all's existing nonoverlapping behavior. | -| [20. Satisfy feasible explicit chunk targets](catalog.md#20-satisfy-feasible-explicit-chunk-targets) | C, E, T | Ready | The feasible partition counterexample establishes a chunker defect. Guarantee count and maximum constraints before optimizing balance. Choose a proportionate deterministic algorithm and test feasibility independently; no approval is needed for the algorithm or a different balanced cut. Do not change configured budgets or add a new balancing objective. | -| [21. Keep input order for exactly tied timestamps](catalog.md#21-keep-input-order-for-exactly-tied-timestamps) | C, T | Ready | Arbitrary reorder of tied utterances has no domain justification. Preserve source order for equal sort keys with a stable sort. No new end-time ordering criterion is implied; retain the current primary chronological key. | -| [22. Preserve failure phase and origin in run reports](catalog.md#22-preserve-failure-phase-and-origin-in-run-reports) | C, A, T | Ready | Required phase and available operation origin are lost between owners. Carry structured facts and assemble run records once, keeping primary failure separate from canceled peers and secondary delivery failures. Preserve documented counts and serialized identities. | -| [23. Classify provider attempt timeouts separately from operation deadlines](catalog.md#23-classify-provider-attempt-timeouts-separately-from-operation-deadlines) | C, T | Ready | The attempt-timeout probe and classifier order establish misclassification. Prefer native PromptKit classifications and parent cancellation precedence. Use the existing transport-failure reason for an HTTP-only timeout; a new public reason code is unnecessary for this fix. | -| [24. Use input diagnostics for operator-correctable failures](catalog.md#24-use-input-diagnostics-for-operator-correctable-failures) | C, T | Ready; technical ownership check | Processing readiness and chunk infeasibility should follow the existing input-error contract while I/O and cancellation remain runtime failures. Delegated inspection has a demonstrated classification mismatch, but locate responsibility in Audita's loader versus ckprompt/CommandKit before editing. Fix or request a fix at that owner; do not create a local replacement command surface. | -| [25. Identify missing required credentials in safe diagnostics](catalog.md#25-identify-missing-required-credentials-in-safe-diagnostics) | C, M | Ready | The missing-credential cause is available but obscured by generic setup presentation. Surface actionable native setup detail with available role/profile context, preserving typed causes and classification. D1 removes the blanket restriction on cause presentation; compact ordinary report projections remain useful. Keep PromptKit responsible for credential validation; do not introduce local secret inspection or a replacement preparation subsystem. | -| [26. Render input and output paths as plain Markdown text](catalog.md#26-render-input-and-output-paths-as-plain-markdown-text) | C, T | Ready | Input paths are data; rendered Markdown currently gives some filenames unintended syntax. Escape them as literal text without altering structured path values. This restores report presentation rather than imposing filename restrictions. | -| [27. Keep the validator execution class used by the runner](catalog.md#27-keep-the-validator-execution-class-used-by-the-runner) | C, A | Ready | Runtime and ledger can disagree for supported injected validator classes. Record the effective execution class with decisions and project it into the ledger. Preserve existing output fields and fallback behavior; no new validator registration policy. | -| [28. Verify signal and exit handling in the production executable](catalog.md#28-verify-signal-and-exit-handling-in-the-production-executable) | T | Ready | The startup helper duplicates the production boundary, so coverage does not prove actual executable signal wiring. Exercise representative startup and cancellation through the real executable or shared production entry point; keep narrower tests for other CLI behavior. | -| [29. Fail the release bootstrap when cleanliness cannot be established](catalog.md#29-fail-the-release-bootstrap-when-cleanliness-cannot-be-established) | C, T | Ready | The shell probe establishes that a failed status command can appear clean. Check its exit result explicitly and test that boundary. ReleaseKit already owns the independent delivery validation; do not add another local release validator or publication gate. | -| [30. Bound pending proposal work before backend admission](catalog.md#30-bound-pending-proposal-work-before-backend-admission) | E, A, T | Investigate implementation under resolved D3 | Bound local preparation/dispatch and drain results while preserving PromptKit admission ownership, configured waiting/rejection, stable identities, module barriers, and cancellation joins. Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual-admission promise. The current scheduling-dependent selection of rejected chunks is not a compatibility requirement. Measure representative memory/contention to guide the implementation; any future admission-policy gap belongs upstream. | -| [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A | Ready under resolved D4 | Remove generic JSON round trips and release whole-input representations after their last use. The D1 audit additionally identifies source-debug reconstruction: write the acquired bytes directly and retain parsed/normalized views separately, without reopening the input. Share typed output projection; preserve encoding-failure and stream-delivery semantics. Do not introduce an application-level input-size limit; calling integrations own their upload/process budgets. Preserve accepted input formats and stream support. | -| [32. Retain only the run evidence required by enabled outputs](catalog.md#32-retain-only-the-run-evidence-required-by-enabled-outputs) | E, A | Ready | The production consumers establish that much completed evidence is unused without its corresponding output. Retain only required live repair state, compact counts, and selected report/debug detail. Derive not-started counts without a full eager grid where possible; keep complete matrices when enabled and preserve fatal-origin evidence. | -| [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A | Ready under resolved D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected-artifact information. Preserve requested evidence, self-contained interactions, nonfatal debug failures, and continued best-effort capture. Add no quotas, automatic deletion, or automatic stopping after write failures; operators own retention and cleanup. | -| [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E | Investigate costs under resolved D6 | Retain full conversation history and existing per-correction limits. Measure cumulative requests, tokens/bytes, revalidation, and debug costs using existing observations or development instrumentation; remove unnecessary local duplication. Do not add aggregate stopping budgets or reduce model context at this time. Revisit only if production evidence demonstrates a need; no new ordinary-report fields are required for this analysis. | -| [35. Avoid rebuilding growing normalization prefixes](catalog.md#35-avoid-rebuilding-growing-normalization-prefixes) | E | Ready | Growing merge prefixes are repeatedly scanned and copied. Use incremental text/category accumulation and exact estimator components while retaining joiner behavior, punctuation rounding, limits, and category order. Benchmarks should verify the improvement; large refactoring is not a reason to retain quadratic work. | -| [36. Skip validation context work beyond the usable budget](catalog.md#36-skip-validation-context-work-beyond-the-usable-budget) | E, A, T | Ready | Unusable optional context is serialized needlessly. Stop measurement once required-only or ordered-prefix costs settle the fit. Share trimming/batching policy without losing the independent serialized estimator oracle or changing context priority and oversized-item handling. | -| [37. Reuse checked processing selections and matching views](catalog.md#37-reuse-checked-processing-selections-and-matching-views) | E, A | Ready | Readiness and execution derive the same selections separately. Carry one checked processing plan and reuse immutable effective views and inspections per captured runtime. Preserve provenance and first-failure attribution; cache keys must include ordered selections and exemptions. Do not add cross-invocation caches or reload configuration. | -| [38. Select glossary context by identity membership](catalog.md#38-select-glossary-context-by-identity-membership) | E, A | Ready | Context needs identity membership, not a record for every occurrence/contributor pair. Share scanning but avoid expanding multiplicity for context. Preserve glossary order and keep occurrence evidence for protection. | -| [39. Avoid rescanning sections for skipped edits](catalog.md#39-avoid-rescanning-sections-for-skipped-edits) | E | Ready | Skipped-edit reporting repeats segment searches and filtering merges. Reuse a lookup and aggregate once. Preserve the distinction between stale conflicts and actual response filtering; do not broaden rejection or recovery rules. | -| [40. Use one validator contract and one owner for decision semantics](catalog.md#40-use-one-validator-contract-and-one-owner-for-decision-semantics) | A, C, M | Ready | Duplicate ports and runner-specific verdict rewriting obscure one existing validation contract. Consolidate types, applicability, and decision meaning with their owner; remove unused envelope fields. Preserve zero-budget shortlisting, independent repair feedback, cardinality, and unavailable outcomes. No broad plugin framework or new validator policy is needed. | -| [41. Carry structured failure facts through model operations and reporting](catalog.md#41-carry-structured-failure-facts-through-model-operations-and-reporting) | A, C | Ready | Failure class and reason already exist but are discarded and reconstructed. Carry typed facts through role-specific handling and report counts. Use a consistent internal-failure reason fallback for missing reasons while retaining the original failure class and cause. Avoid reclassifying an operation from its display reason. | -| [42. Keep required output publication separate from optional debug retention](catalog.md#42-keep-required-output-publication-separate-from-optional-debug-retention) | A, C | Ready | Required publication and optional retained evidence have distinct lifecycles. Separate their APIs and share regular-file staging where behavior is common. Preserve transcript-before-report ordering, nontransactional finalization, private staging, and nonfatal debug errors. Coordinate with #1, #7, #16, and #22. | -| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready under resolved D1 | The focused audit identifies removable secret-source assertions, lazy discovery closures, combined-role lists, and preflight/writer credential arguments. Complete common interaction writing without those dependencies. Preserve role-owned payloads, artifact association, and nonfatal write callbacks and synchronization; coordinate with #42/#48. Resolved D5 preserves best-effort capture and operator-owned retention. | -| [44. Simplify structured-call completion and remove obsolete record fields](catalog.md#44-simplify-structured-call-completion-and-remove-obsolete-record-fields) | A, M, T | Ready | Ordinary calls and coordinator turns share completion responsibilities; obsolete payload copies have no production reader. Use explicit once-only completion and remove superseded state after moving meaningful assertions. Preserve native terminal counts, typed causes, and undelivered diagnostic evidence. | -| [45. Retire the test-only direct continuation branch](catalog.md#45-retire-the-test-only-direct-continuation-branch) | M, T | Ready | Direct continuation is test-only; production uses coordination. Move sibling-isolation and structural-repair coverage onto the coordinator before removing the branch. Removing the redundant adapter surface does not remove PromptKit capability or production repair. | -| [46. Use a single proposal handoff and test real runner indexing](catalog.md#46-use-a-single-proposal-handoff-and-test-real-runner-indexing) | A, E, M, T | Ready | Generator enrichment is discarded and real indices are assigned by the runner. Keep one proposal handoff with original local references, and assert actual deterministic runner indices under observed out-of-order completion. Review emitted diagnostic start_index separately before dropping it; move useful tests before removing their harness. | -| [47. Centralize request-local segment reference resolution](catalog.md#47-centralize-request-local-segment-reference-resolution) | A, M | Ready | Initial and repaired proposals duplicate positional reference translation. Share a small protocol lookup while retaining raw references, invalid-target evidence, distinct reference namespaces, and retargeted revalidation. Independent caller expectations should survive consolidation. | -| [48. Give one repair branch a clear execution and result boundary](catalog.md#48-give-one-repair-branch-a-clear-execution-and-result-boundary) | A | Ready | Repair branch state is spread through closures and section aggregation. Give one branch its own execution/result boundary and make capabilities explicit at setup. Keep PromptKit's conversation and turn-limit ownership, independent siblings, final-turn validation, and current repair policy. | -| [49. Define concurrency and ownership in module and validator contracts](catalog.md#49-define-concurrency-and-ownership-in-module-and-validator-contracts) | M, A | Ready | Shared instances and borrowed data already impose concurrent-call and ownership obligations. Document those obligations at the port, permit synchronized stateful implementations, and use owned copies where mutation is needed. This clarifies the existing runtime, not a new ban on state or a forced-cancellation guarantee. | -| [50. Consolidate root and per-module configuration rules](catalog.md#50-consolidate-root-and-per-module-configuration-rules) | A, M | Ready | Root and stage checks duplicate application fallback rules. Consolidate typed checks and resolution helpers while preserving null, zero, inheritance, provenance, diagnostic paths, and stage-dependent defaults. ConfigKit continues to own all composition before handoff; do not reproduce its machinery. | -| [51. Unify transcript conversion and retire redundant parser wrappers](catalog.md#51-unify-transcript-conversion-and-retire-redundant-parser-wrappers) | A, M, C, T | Ready | The strict parser's panic is real but limited to an unused production path. Remove obsolete wrappers after moving useful tests and share the live source-to-canonical conversion. Fixing and retaining the obsolete parser would preserve unnecessary machinery. Keep optional source IDs and normalized output behavior. | -| [52. Give public transcript encoders one segment projection](catalog.md#52-give-public-transcript-encoders-one-segment-projection) | A, E, T | Ready | Two public encoders duplicate segment projection. Give the output boundary one projection and test both envelopes with independent expected fields. Preserve omission, category, numeric, and schema/version behavior. | -| [53. Use one simple token estimate across text-processing packages](catalog.md#53-use-one-simple-token-estimate-across-text-processing-packages) | A, E, T | Ready | Both counters implement the same heuristic. Share its implementation/components and preserve exact estimates, consumer budgets, and independent expected-count tests. A new tokenizer or changed heuristic is outside this consolidation. | -| [54. Centralize transcript cloning](catalog.md#54-centralize-transcript-cloning) | A, M, T | Ready | Deep copying belongs with the transcript model. Preserve nil versus explicitly empty mutable slices in the shared clone, both caller copy boundaries, and current external encoding. Verify independence using fixtures not constructed by the same clone. This internal fidelity choice needs no product decision. | -| [55. Assemble built-in modules at the composition boundary](catalog.md#55-assemble-built-in-modules-at-the-composition-boundary) | A | Ready | Production built-in assembly belongs at composition; the runner already accepts a factory. Move construction while keeping the configuration catalog dependency-light and retaining override seams. Package layout is an implementation choice, not a product decision. | -| [56. Merge shared prompt variables once per model call](catalog.md#56-merge-shared-prompt-variables-once-per-model-call) | E, M | Ready | Processing loads the same shared values twice. Merge once while preserving processing and inspection's different precedence and the protected-term default. Do not accidentally weaken operator prompt-definition replacement support or invent new shared-variable precedence. | -| [57. Clarify what the inherited progress flag does in processing](catalog.md#57-clarify-what-the-inherited-progress-flag-does-in-processing) | M, A, T; P approved | Ready under resolved D7 | Add processing progress through CommandKit’s reporter. An operator selecting --progress wants relatively detailed updates: emit meaningful stage/chunk, proposal, validation, and repair activity as available, rather than only coarse stage summaries. CommandKit owns reporting-mode precedence and stream routing; preserve transcript stdout. Exact event granularity and wording are implementation choices. Request upstream support for any standard reporter gap rather than building a parallel reporting system. | -| [58. Test registered validator roles and embedded assets through their behavior](catalog.md#58-test-registered-validator-roles-and-embedded-assets-through-their-behavior) | T, M | Ready | Registry tests duplicate representation assertions and prompt-count checks can miss substitutions. Check required identities, effective roles, preparability, schema identity, and protective copies at their actual owners. Do not replace them with prompt wording or private implementation assertions. | -| [59. Consolidate overlapping normalization fixtures](catalog.md#59-consolidate-overlapping-normalization-fixtures) | T, M | Ready | Two fixtures protect the same nonmerge outcome. Consolidate their meaningful assertions, keeping distinct normalization behavior and independent input expectations. Coordinate with #51 so copied parser conversion is not retained in the tests. | -| [60. Focus integration tests on decoded artifacts and independent output choices](catalog.md#60-focus-integration-tests-on-decoded-artifacts-and-independent-output-choices) | T | Ready | Decoded artifact identities and evidence associations are the contract; escaping and repeated large fixtures are incidental. Keep one full repair/filter/later-failure attribution scenario and use smaller independent-output fixtures. Remove masking assertions identified in the D1 audit without losing role selection, successive text, output-choice, and failure coverage. | -| [61. Remove abandoned test support and unused fixture inputs](catalog.md#61-remove-abandoned-test-support-and-unused-fixture-inputs) | M, T | Ready | The reachability audit found abandoned helpers and unused fixture inputs. Delete them, including unused secret scanners; the D1 audit also identifies credential-echo fixture machinery to retire when its masking-only consumers disappear. Retain compile-time interface fixtures and useful behavioral assertions, rechecking callers when implementing. | -| [62. Remove unreachable merge and file helpers](catalog.md#62-remove-unreachable-merge-and-file-helpers) | M | Ready | Unreachable merge and file helpers duplicate or bypass live owners. Remove them after a current reference check, retaining live normalization, input diagnosis, and output-session behavior. | -| [63. Remove redaction-only credential discovery and tracking](catalog.md#63-remove-redaction-only-credential-discovery-and-tracking) | A, E, M, T | Ready under resolved D1 | Profile/backend enumeration, environment reads, name caches, and role tracking exist solely for substitution. Remove them after their consumers, preserving native authentication, inspection, prepared snapshots, call IDs, and terminal synchronization. | -| [64. Preserve validator explanations in repair feedback](catalog.md#64-preserve-validator-explanations-in-repair-feedback) | A, M, T | Ready under resolved D1 | Secret substitution affects live repair feedback even without debug. Use domain repair findings directly, retaining proposal association and unavailable-assessment semantics. Keep repair budgets, withdrawal, revalidation, and sibling independence unchanged. | -| [65. Remove credential substitution from CLI presentation](catalog.md#65-remove-credential-substitution-from-cli-presentation) | A, E, M, T | Ready under resolved D1 | Remove preparation/finalization sanitizers and obsolete arguments; present existing warnings, errors, and actual paths through CommandKit. Preserve typed causes, reporting modes, exit behavior, and output failure ownership. Coordinate with #25 for separately obscured setup detail. | - -## Implementation planning implications - -Group related work by owner rather than implementing catalog numbers as 65 -independent tasks. Fixes and their supporting ownership changes belong together: - -- Output publication and diagnostic integrity: #1, #3–9, #16, #22, #26, - #42–43, #63, #65, incorporating resolved D1 cleanup, D2 caller ownership, and D5 - best-effort capture with operator-owned retention. -- Transcript and glossary correctness: #11–13, #17–21, #35–39, #51–54. -- Runner and model-operation lifecycle: #10, #14–15, #23–25, #27, #30, - #32, #40–41, #44–49, #55–56, #64. -- Verification and obsolete machinery: #28–29, #58–62 and the tests attached - to each other change. - -These are dependency groups, not a commitment to one large rewrite. Preserve -independent fixture expectations when consolidating implementations. Move useful -coverage to the live boundary before deleting test-only execution paths. Verify -resource improvements with representative measurements; do not require production -benchmarks before removing plainly redundant work. - -Technical follow-up does not require another approval round: locate delegated -error ownership for #24; design and measure local dispatch/result draining for -#30; measure aggregate repair costs for #34. Escalate only if the concrete remedy -crosses a policy boundary below. Native ConfigKit, PromptKit, CommandKit, and -ReleaseKit responsibilities remain upstream-owned. Submitting a feature request -is preferable to maintaining a local workaround when an upstream API is the gap. - -Earlier review questions about full-segment protection, cloning details, internal -decision representation, and ordinary publication ownership are resolved here as -correctness/engineering work. They are not additional open product decisions. -Independent whole-pipeline chunk traversal and semantic recovery of unavailable -validator responses remain [separate deferred features](../future.md). - -## Resolved D1: withdraw application-wide credential redaction - -**Accepted:** 2026-11-15. **Policy update and focused audit:** complete. -**Implementation:** pending. - -The maintainer withdrew the application-wide credential-redaction requirement -and approved removing machinery whose sole purpose is serving it, plus -simplifying shared machinery made more complex by it. The architecture policy -was updated by deleting the requirement, without adding a negative assertion. -Upstream libraries' native credential/configuration behavior remains in place. -The earlier proposed response-omission policy is superseded; debug evidence, -including malformed responses, remains useful to retain. - -The maintainer's stated scope places artifact access and sharing with the -operator and excludes hostile actors with local filesystem access. This changes -the premises of #3 and #4 and the remedies for #9 and #43. Ordinary publication -correctness, authentication, and required output behavior still have independent -purposes. Subsequent decisions retain their IDs. - -### Focused follow-up audit: complete - -The [removal and simplification audit](14-redaction.md) records seven findings, -traced consumers, retained responsibilities, tests/docs, and implementation -sequencing. It adds #63–65 and refines #3–4, #9, #25, #31, #43, and the related -test cleanup. It confirms that credential discovery, generic substitution, -serialization round trips, cross-role secret plumbing, repair-feedback rewriting, -and CLI substitution can be removed without replacing native library behavior. - -Use that inventory to plan coherent removal/refactoring units. Preserve native -authentication and inspection, evidence fidelity, typed failure classification, -Markdown escaping, ordinary output correctness, and nonfatal debug failures. -No new product decision is required for these removals; subsequent decisions -are independent of D1. -The static audit did not run providers or tests or inspect real credentials. -Update implemented-behavior documentation alongside code changes. The source -audits remain historical evidence; this assessment controls remediation scope. - -## Resolved D2: callers own report destination isolation - -**Accepted:** 2026-11-15. **Applies to:** #8. -**Status:** decision recorded; operator-documentation update pending. - -Report destination isolation remains the caller’s responsibility. Keep the -current placement contract and fixed report basenames. Document in the operations -guide that concurrent jobs select distinct report directories, and that retaining -reports from successive runs likewise requires separate destinations or caller -archiving. Different transcript filenames in one directory do not isolate reports. - -Automatic per-run directories, destination locking, waiting, and competing-writer -rejection are not selected. They would add placement or contention semantics to -a workflow the caller already controls. The audit demonstrates overwriting when -callers share destinations; it does not establish an application promise of -per-run isolation. Correct writes to checked destinations in #1 remain independent. - -## Resolved D3: retain native backend admission semantics - -**Accepted:** 2026-11-15. **Applies to:** #30, with scheduling interactions in #15. -**Status:** decision recorded; local scheduling/resource improvements await -implementation planning. - -PromptKit retains ownership of backend admission, including configured waiting -and recoverable capacity rejection. Use native settings where operators want -more waiting capacity. Audita will not add an outer admission-retry loop, a -separate backend limiter, or a guarantee that every chunk eventually receives -admission. - -Improve Audita's local scheduling and memory use as engineering work: bound -pending preparation/dispatch, drain completed results, and avoid unnecessary -whole-transcript allocation. Preserve module barriers, stable identities, -ordered application, cancellation/join behavior, and native admission semantics. -The scheduling-dependent choice of chunks rejected by the current fan-out is -not a compatibility requirement; a different admission outcome caused by better -local scheduling does not itself establish a new admission policy. - -Choosing to wait until every chunk can receive a turn would be a separate -processing policy. If reconsidered, first assess native PromptKit configuration -and seek upstream support for any gap. Native rejection is not a PromptKit defect. -Representative memory, contention, and cancellation checks should guide the -local implementation without creating a second admission system. - -## Resolved D4: improve memory use without an input-size limit - -**Accepted:** 2026-11-15. **Applies to:** #31. -**Status:** decision recorded; efficiency improvements await implementation. - -Do not add an application-level transcript input-size limit in this remediation. -Remove unnecessary copies, generic serialization round trips, and retained -representations after their last use while preserving accepted input formats, -stream support, and output/error behavior. Calling applications remain -responsible for their upload and process budgets. - -The audit establishes memory amplification, but provides no measured threshold -that justifies rejecting otherwise valid CLI workloads. An optional input-byte -budget could help integrations without their own enforcement; it is not selected -because it adds configuration and rejection behavior without a demonstrated need. - -## Resolved D5: preserve best-effort debug capture - -**Accepted:** 2026-11-15. **Applies to:** #33. -**Status:** decision recorded; efficiency improvements await implementation. - -Preserve requested diagnostic evidence and continue best-effort capture. Debug -write failures remain nonfatal. Remove redundant in-memory representations and -serialization, and consolidate equivalent warnings while retaining counts and -affected-artifact information. Keep interaction artifacts self-contained. - -Do not add capture quotas, automatic deletion, or automatic shutdown of later -capture after write failures in this remediation. An individual failure need -not prevent later writes from succeeding; continued capture can retain useful -evidence for an unsuccessful run. Operators own retention and cleanup. - -Optional budgets or persistent-failure stopping rules could reduce disk use and -repeated failed work, but deliberately omit evidence and add configuration and -reporting obligations. They are not selected. Internal efficiency improvements -should preserve the evidence operators requested. - -## Resolved D6: retain repair context and per-correction limits - -**Accepted:** 2026-11-15. **Applies to:** #34. -**Status:** decision recorded; cost analysis and efficiency work remain pending. - -Retain full conversation history and the existing per-correction limits. Do not -add aggregate section/run repair budgets or reduce conversation context at this -time. Measure cumulative requests, tokens/bytes, revalidation, and debug costs, -and remove unnecessary local copies without changing the model's evidence or -which corrections receive repair attempts. Use existing observations or focused -development instrumentation; no new ordinary-report fields are needed merely -to perform the analysis. - -Resource growth partly reflects the deliberately selected repair behavior. -Aggregate budgets could leave later corrections unrepaired because earlier -ones consumed the allowance, while context reduction could reduce repair -quality. Neither change follows from the audit finding alone. Revisit these -options only if production evidence demonstrates a need, through a separate -policy decision. - -## Resolved D7: emit detailed processing progress through CommandKit - -**Accepted:** 2026-11-15. **Applies to:** #57. -**Status:** feature approved; implementation pending. - -Emit processing progress through CommandKit's existing reporter, driven by real -processing lifecycle events. An operator who expressly selects `--progress` is -requesting relatively detailed information. Provide useful updates throughout -processing, including stage and chunk activity and proposal, validation, and -repair work where available; do not limit the feature to coarse stage summaries -merely to minimize output. Include work identities and observed completion counts -where they help the operator follow the run. Exact wording and event granularity -remain implementation choices. - -CommandKit continues to own reporting-mode precedence and stream routing. -Preserve stdout for transcript output and respect the effective mode when -`--progress` is combined with other reporting flags. Audita supplies processing -events rather than implementing a separate output-mode or progress system. -Use available upstream execution events where appropriate; request upstream -support for a material reporter gap instead of adding a local workaround. - -Retaining an accepted flag with no processing updates was considered and is not -selected: progress is useful for long-running jobs, and the maintainer expressly -approved adding it. Update CLI/operations documentation when implementation -lands. Verify representative real lifecycle updates, native reporting-mode -behavior, and transcript-output integrity without testing incidental wording. - -## Decision completion - -All seven decisions are resolved. Implementation planning can proceed using the -entry assessments and accepted scopes above. Technical investigations and -measurements remain where identified; they are not unresolved product decisions. diff --git a/docs/roadmap/audit/catalog.md b/docs/roadmap/audit/catalog.md deleted file mode 100644 index b4bff85..0000000 --- a/docs/roadmap/audit/catalog.md +++ /dev/null @@ -1,611 +0,0 @@ -# Consolidated audit catalog - -Audita is a single-process Go CLI. CommandKit dispatches commands and ConfigKit -resolves application settings. Core packages parse, normalize, and chunk -transcripts. The runner coordinates proposals, validation, repair, and application -through a PromptKit adapter. The CLI publishes the corrected transcript and -ordinary reports, with separate opt-in debug artifacts. -[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) - -The 12 source audits contain 110 numbered findings. This catalog consolidates them -into 65 entries. Priority reflects potential impact, exposure, and the strength of -the reported evidence. A source can appear under several entries when it covers -distinct issues. Each link leads to the original finding and its supporting -evidence. - -The [assessment and decision register](assessment.md) evaluates every entry using -the maintainer-approved taxonomy and separates ready remediation from technical -investigation and pending policy decisions. Consult it before planning changes; -source-audit recommendations are evidence and proposals, not authorization. -The three WONTFIX scopes recorded here remain controlling. -The focused [redaction-removal audit](14-redaction.md) applies resolved D1: -the earlier redaction remedies below are superseded, and #63–65 add distinct -cleanup work. Entry numbers and existing anchors are retained for traceability. - -## Data protection and run integrity - -### 1. Bind output writes to the checked destinations - -Transcript path resolution can select the wrong target, and report writers can -change referents after preflight. Use filesystem-consistent resolution and carry -the checked destination into publication. Cover changed report referents as well -as initially ambiguous paths. -[Correctness §1](01-correctness.md#1-output-path-resolution-can-replace-a-different-file) · -[Concurrency §6](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) · -[Security §3](07-security.md#3-report-destinations-can-escape-the-identity-checked-by-preflight) · -[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) · -[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) - -### 2. Keep model requests within the configured transport boundary - -Provider redirects can take transcript-bearing requests outside the selected -origin or downgrade the transport. - -**WONTFIX.** PromptKit inherits the standard `net/http` client's default redirect -and credential-forwarding behavior. Audita accepts those defaults for the -built-in transport. -[Security §2](07-security.md#2-provider-redirects-can-cross-the-configured-transport-boundary) - -### 3. Protect credentials inside nested debug responses - -**Superseded by D1.** The original observation tested the withdrawn redaction -requirement. Remove Audita's substitution package and use direct artifact -serialization; retain useful responses, including malformed content. Retire -masking-only tests and documentation while preserving evidence-fidelity and -write-failure coverage. Credential discovery and repair/CLI consumers are covered -by #63–65; no nested sanitizer or response-omission policy is selected. -[Security §4](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) · -[Testing §8](08-testing.md#8-debug-redaction-assertions-mostly-inspect-the-outer-serialization) · -[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding) · -[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) - -### 4. Make debug artifact permissions private by default - -**Prior remedy withdrawn under D1.** Operators own artifact access and sharing; -hostile local filesystem access is outside the selected threat model. The focused -audit found ordinary creation and filename handling to retain, not a redundant -private-root mechanism. Leave current locations and modes unchanged in this -cleanup. Any future default change needs an independent justification. -[Security §1](07-security.md#1-debug-capture-uses-a-shared-path-and-can-expose-private-artifacts) - -### 5. Define the trust boundary for discovered configuration - -The selected main file and its automatic companions can influence provider and -credential selection. Document trusted configuration locations and keep -upload-controlled paths separate from operator configuration. -[Security §5](07-security.md#5-configuration-discovery-extends-trust-to-the-main-file-and-its-neighbors) - -### 6. Make blocking transcript input and delivery cancellation-aware - -Input can block before preflight, and FIFO or stdout delivery can stay blocked -after cancellation. - -**WONTFIX (interrupting already-blocked I/O).** The standard file operations used -here take no context, and `io.Writer` offers no general interruption mechanism. -Audita accepts that those calls can remain blocked until I/O can proceed. - -The application still owns the outcome when control returns. Check cancellation -before publishing a regular file and after stream delivery returns so a canceled -run exits nonzero and attempts to write failed-status reports. -[Correctness §5](01-correctness.md#5-cancellation-during-fifo-delivery-can-hang-and-later-report-success) · -[Concurrency §1](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) · -[Concurrency §2](04-concurrency.md#2-waiting-for-transcript-input-can-prevent-shutdown-before-preflight) · -[Resources §2](05-resources.md#2-blocked-stream-io-extends-resource-lifetime-past-cancellation) · -[Security §7](07-security.md#7-untrusted-input-size-and-blocking-io-can-exhaust-subprocess-workers) · -[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) - -### 7. Publish truthful failure reports and preserve initialized evidence - -A delivery failure can briefly publish `completed` reports. The internal preflight -seam can also replace initialized reports with empty files when failed-report -rendering is unavailable. Verify the first published state and preserve valid -existing evidence if a failed bundle cannot be rendered. -[Error handling §1](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed) · -[Complexity §13](03-complexity.md#13-missing-failed-report-rendering-can-erase-initialized-evidence) · -[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) - -### 8. Prevent concurrent runs from clobbering ordinary reports - -Invocations sharing a report directory replace the same basenames and can leave a -mixed set. **Resolved D2:** destination isolation remains the caller’s -responsibility. Retain current report placement and document distinct report -directories for concurrent jobs or retained run histories. Automatic per-run -placement, locking, waiting, and competing-writer rejection are outside this -remediation. Correct publication to checked destinations remains #1’s concern. -[Concurrency §7](04-concurrency.md#7-concurrent-invocations-sharing-a-report-directory-replace-each-others-evidence) - -### 9. Preserve the ordinary YAML schema during redaction - -A credential value can match a required key or fixed enum and alter an otherwise -successful report. Under resolved D1, remove the post-serialization redactor and -serialize the typed report directly. Keep schema/scalar fidelity coverage; do -not build a more selective sanitizer. -[Correctness §9](01-correctness.md#9-credential-redaction-can-change-the-ordinary-yaml-schema) · -[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding) - -### 10. Reject contradictory validation decisions - -The internal decision boundary can advance a candidate marked both approved and -unavailable. Validate the disposition and approval invariant before filtering or -repair. The bundled validators produce consistent decisions in the audited paths. -[Complexity §11](03-complexity.md#11-a-contradictory-validator-decision-can-advance-an-unavailable-candidate) - -### 11. Preserve usable validator decisions when extra references cannot be decoded - -A schema-valid unexpected numeric reference can fail whole-batch Go decoding -before per-entry reconciliation. Preserve valid requested siblings and test the -mixed response at the actual adapter boundary. -[Correctness §7](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions) · -[Testing §5](08-testing.md#5-the-validator-batch-fixture-misses-the-real-numeric-decoding-boundary) - -### 12. Use full-segment evidence for protected-name checks - -Span-only validation can reject a valid edit or miss a repair opportunity. Share a -glossary-owned matching and protection assessment across validation and -application, and recheck the current segment when applying an edit. -[Correctness §8](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity) · -[Equivalence §9](10-equivalence.md#9-reconcile-the-protection-rule-at-validation-and-application) · -[Boundaries §5](11-boundaries.md#5-give-glossary-matching-and-protection-a-domain-owner) - -### 13. Preserve transcript input fidelity during JSON shape detection - -Generic decoding can round large integer IDs and fail on values in fields the -typed reader ignores. Select the segment array as raw JSON and test exact IDs -across both supported input shapes. -[Correctness §6](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) · -[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open) · -[Resources §4](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication) - -### 14. Honor cancellation at the runner commit boundary - -An already canceled empty pipeline can return success, and cancellation during -module application can still allow the runner to commit edits. Check entry and -final commit boundaries and give successful and interrupted results one assembly -path. -[Complexity §1](03-complexity.md#1-give-module-execution-one-place-to-commit-its-result) · -[Complexity §12](03-complexity.md#12-the-runner-accepts-an-already-canceled-empty-pipeline) · -[Concurrency §3](04-concurrency.md#3-cancellation-can-cross-the-runners-module-commit-boundary) - -### 15. Retain completed proposal evidence behind slow sections - -The ordered prefix gate can delay validation for ready sections and omit completed -proposal evidence on interruption. Capture usable responses independently of -validation dispatch and preserve deterministic identities. -[Correctness §10](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger) · -[Concurrency §4](04-concurrency.md#4-a-slow-earlier-proposal-holds-back-later-validation-and-retention) · -[Complexity §1](03-complexity.md#1-give-module-execution-one-place-to-commit-its-result) - -### 16. Keep ownership of failed staging cleanup - -Removal errors can discard temporary paths or go unreported, preventing a retry. -Keep paths until removal succeeds or the path is absent and verify cleanup -failures and cancellation after output acquisition. -[Resources §1](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) · -[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) - -## Correctness and failure reporting - -### 17. Preserve the complete time interval when merging segments - -A nested segment can shorten the merged end time and change later gap and duration -decisions. Use the full interval for both eligibility and output, and cover nested -intervals in normalization tests. -[Correctness §3](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) · -[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open) - -### 18. Match protected names at Unicode and punctuation boundaries - -The matcher misses some names with accented letters or punctuation at their edges. - -**WONTFIX (Unicode behavior of `\b`).** Go's `regexp` defines `\b` as an ASCII -word boundary. Audita accepts that definition for the standard regexp engine. - -Configured aliases with punctuation at their edges still need a glossary matching -rule for protection and context selection, even when their letters are ASCII. -[Correctness §2](01-correctness.md#2-glossary-matching-misses-names-with-accented-letters-or-punctuation-at-their-edges) - -### 19. Detect overlapping occurrences when requiring a unique edit span - -Nonoverlapping counting can treat a repeated overlapping span as unique. Use one -literal occurrence rule in preview, validation, and application. -[Efficiency §7](02-efficiency.md#7-the-unique-span-check-accepts-overlapping-matches) - -### 20. Satisfy feasible explicit chunk targets - -A balancing preference can reject inputs that have a valid contiguous partition. -Make the configured count and limits prerequisites, then optimize balance within -feasible choices. Verify skewed inputs at the feasibility boundary. -[Correctness §4](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) · -[Testing §7](08-testing.md#7-representative-transcript-fixtures-leave-data-integrity-boundaries-open) - -### 21. Keep input order for exactly tied timestamps - -An unstable sort can reorder utterances with identical times. Use source position -as a stable tie-breaker. -[Efficiency §8](02-efficiency.md#8-normalization-can-reorder-utterances-with-exactly-tied-timestamps) - -### 22. Preserve failure phase and origin in run reports - -Required output errors can omit their phase, and fatal model errors can lose -chunk, role, validator, or batch coordinates. Carry typed failure facts from their -source and assemble run records and counts in one place. Assert the serialized -fields. -[Correctness §11](01-correctness.md#11-required-delivery-failures-omit-the-phase-in-run-error-records) · -[Error handling §4](06-error-handling.md#4-fatal-model-errors-lose-their-chunk-and-operation-coordinates) · -[Testing §3](08-testing.md#3-output-tests-primarily-check-the-final-artifact-state) · -[Boundaries §8](11-boundaries.md#8-give-run-level-report-facts-one-assembly-path) · -[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) - -### 23. Classify provider attempt timeouts separately from operation deadlines - -The adapter can report an HTTP attempt timeout as a logical operation deadline. -Use the native classifications and test attempt, operation, and parent -cancellation under distinct live budgets. -[Error handling §2](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline) · -[Testing §4](08-testing.md#4-the-timeout-tests-cover-two-of-the-three-relevant-deadline-owners) - -### 24. Use input diagnostics for operator-correctable failures - -Invalid processing selections and malformed inspection configuration can return -the runtime exit status. A genuinely infeasible chunk target can also be reported -as an internal defect. Preserve the distinction between readiness, chunking, and -runtime failures and assert exact statuses in CLI tests. Preserve native -inspection behavior; retired application-wide masking expectations are not an -additional acceptance criterion. -[Error handling §5](06-error-handling.md#5-invalid-processing-selections-use-the-runtime-failure-exit-status) · -[Error handling §6](06-error-handling.md#6-an-infeasible-chunk-target-is-recorded-as-an-internal-processing-failure) · -[Testing §2](08-testing.md#2-inspection-failure-tests-accept-the-wrong-exit-classification) - -### 25. Identify missing required credentials in safe diagnostics - -The process path reduces a known missing credential requirement to a generic setup -failure. Surface actionable native setup detail and preserve the underlying -typed cause and failure classification. D1 removes the blanket restriction on -cause presentation; ordinary compact report projections remain useful. -[Error handling §3](06-error-handling.md#3-missing-required-credentials-produce-a-generic-setup-message) · -[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership) - -### 26. Render input and output paths as plain Markdown text - -Untrusted filenames can alter the rendered report or its navigation. Escape path -text consistently with other untrusted labels and verify the rendered meaning. -[Security §6](07-security.md#6-untrusted-transcript-filenames-can-become-active-markdown) - -### 27. Keep the validator execution class used by the runner - -The ledger can infer a different class from an injected validator's name. Carry -the resolved class in decision evidence and join the evidence once before emitting -rows, preserving disposition and repair lineage. -[Complexity §8](03-complexity.md#8-join-correction-evidence-before-building-ledger-rows) · -[Boundaries §2](11-boundaries.md#2-preserve-the-execution-class-assigned-to-each-validator-decision) - -### 28. Verify signal and exit handling in the production executable - -Subprocess fixtures recreate startup wiring and can miss regressions in the actual -binary. Exercise the built executable or share the production entry point for -representative startup and cancellation cases. -[Testing §1](08-testing.md#1-subprocess-tests-recreate-the-production-executable-boundary) - -### 29. Fail the release bootstrap when cleanliness cannot be established - -A failed Git status command can be mistaken for a clean checkout. Check its status -explicitly and exercise that boundary in place of testing the release test's own -comparison helper. The delivery tool has a separate prepared-checkout check. -[Error handling §7](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) · -[Testing §11](08-testing.md#11-remove-the-tests-of-the-release-tests-own-set-helper) - -## Scalability and resource use - -### 30. Bound pending proposal work before backend admission - -One goroutine per chunk makes finite-capacity rejection depend on scheduling and -can retain requests and results across the whole run. Bound dispatch and result -retention without changing the module barrier or backend admission ownership. -**Resolved D3:** preserve PromptKit's configured waiting and recoverable rejection. -Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual- -admission promise. Current scheduling-dependent rejection order is not a -compatibility requirement. Verify memory use, actual contention, and cancellation -while improving local scheduling; any future admission-policy gap belongs upstream. -[Concurrency §5](04-concurrency.md#5-proposal-fan-out-makes-admission-under-finite-capacity-depend-on-scheduling) · -[Resources §3](05-resources.md#3-backend-admission-leaves-chunk-fan-out-and-result-retention-unbounded) · -[Testing §6](08-testing.md#6-concurrency-fixtures-leave-some-schedules-unproved) - -### 31. Reduce peak memory for large transcript input and output - -Whole-file reads, a generic JSON tree, retained source bytes, and full output -buffers amplify memory before and after model work. Reduce live copies and -write the acquired source bytes directly to its debug snapshot instead of -reconstructing them for redaction. Keep parsed/normalized views separate. -**Resolved D4:** improve memory use without adding an application-level -transcript-size limit. Calling integrations own their upload/process budgets; -preserve accepted input formats and stream support. -[Redaction removal §3](14-redaction.md#3-save-the-acquired-source-snapshot-without-reconstructing-it) · -[Resources §4](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication) · -[Security §7](07-security.md#7-untrusted-input-size-and-blocking-io-can-exhaust-subprocess-workers) - -### 32. Retain only the run evidence required by enabled outputs - -Ordinary runs retain debug-only candidates and decision history. Summary-only runs -still allocate the full work-unit grid, and module results duplicate operation -records and unused proposal totals. Preserve live validation and repair state, and -collect compact counts when detailed evidence is disabled. -[Efficiency §5](02-efficiency.md#5-ordinary-runs-retain-debug-only-decision-history) · -[Resources §5](05-resources.md#5-summary-only-runs-retain-a-full-grid-of-work-and-operation-records) · -[Dead code §3](12-deadcode.md#3-remove-duplicate-runner-bookkeeping-for-unused-module-totals) - -### 33. Limit debug retention costs within a run - -Execution histories and repeated serialization increase peak memory, and retained -interaction files and warnings can accumulate after write failures. Reduce -duplicate representations and serialization, and consolidate equivalent warnings -with counts and affected-artifact information. **Resolved D5:** preserve requested -evidence, self-contained interactions, nonfatal failures, and continued best-effort -capture. Add no quotas, automatic deletion, or automatic stopping after write -failures; operators own retention and cleanup. -[Resources §6](05-resources.md#6-debug-capture-adds-memory-and-disk-pressure) - -### 34. Account for cumulative correction-repair history and work - -Each rejected candidate can start its own bounded conversation and repeat the -original prompt history, with additional revalidation. **Resolved D6:** retain -full conversation history and existing per-correction limits; add no aggregate -budgets or context reduction at this time. Measure cumulative work and remove -unnecessary local duplication. Revisit behavior changes only if production -evidence demonstrates a need. -[Resources §7](05-resources.md#7-correction-repair-replays-growing-history-under-per-correction-budgets) - -### 35. Avoid rebuilding growing normalization prefixes - -Merge estimation and category accumulation repeatedly copy and scan the same -growing text and category prefixes. Reuse incremental work while preserving exact -text, category order, and the configured limit. -[Efficiency §1](02-efficiency.md#1-normalization-repeatedly-copies-and-scans-growing-merge-runs) - -### 36. Skip validation context work beyond the usable budget - -Optional context is serialized even when required fields already exceed budget. -Share trimming and batching policy, stop measuring context that cannot fit, and -keep serialized measurement as an independent reference for the optimized path. -[Efficiency §4](02-efficiency.md#4-validation-measures-optional-context-that-cannot-fit) · -[Equivalence §1](10-equivalence.md#1-share-validation-budget-policy-across-measurement-paths) - -### 37. Reuse checked processing selections and matching views - -Preflight and execution separately decide active roles, and repeated profile or -glossary selections repeat inspection, copying, and preparation. Carry one checked -processing plan and reuse immutable results for equivalent selections. -[Efficiency §6](02-efficiency.md#6-repeated-pipeline-selections-duplicate-setup-work) · -[Boundaries §4](11-boundaries.md#4-carry-one-checked-processing-plan-from-preflight-into-execution) - -### 38. Select glossary context by identity membership - -The validation context builder materializes each matched occurrence for every -contributor and then deduplicates them. Use a membership path for context and -preserve occurrence counts for protection. -[Efficiency §3](02-efficiency.md#3-validator-glossary-context-expands-every-occurrence-into-every-contributor) - -### 39. Avoid rescanning sections for skipped edits - -Apply-time filtering searches the original chunk for every skipped proposal. Reuse -a segment lookup and preserve stale-conflict and filtering distinctions. -[Efficiency §2](02-efficiency.md#2-application-reporting-searches-the-source-chunk-for-each-skipped-edit) - -## Ownership and maintainability - -### 40. Use one validator contract and one owner for decision semantics - -Runner-side applicability and verdict handling overlap with validator -implementations. Consolidate the interface and acceptance rules, and remove the -unused validation request envelope, segment ID field, and result name copy. -[Complexity §3](03-complexity.md#3-put-validator-applicability-and-decision-meaning-at-their-owning-boundary) · -[Boundaries §1](11-boundaries.md#1-give-the-validator-runtime-interface-one-owner) · -[Dead code §4](12-deadcode.md#4-drop-the-unconsumed-validation-request-envelope) - -### 41. Carry structured failure facts through model operations and reporting - -Proposal, validation, and repair repeat class and reason extraction, and report -counts reclassify failures from reason codes. Carry typed outcome facts forward -while preserving role-specific recovery and fatal handling. -[Complexity §7](03-complexity.md#7-give-reporting-one-source-for-operation-classification-and-counts) · -[Equivalence §4](10-equivalence.md#4-share-structured-failure-facts-across-model-operations) - -### 42. Keep required output publication separate from optional debug retention - -The output session combines unrelated lifecycles. Give required-output preparation -and publication a clear owner, share regular-file staging mechanics for transcript -and reports, and expose debug setup through its own API. -[Complexity §9](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) · -[Equivalence §2](10-equivalence.md#2-share-regular-file-staging-between-transcript-and-report-publication) · -[Boundaries §7](11-boundaries.md#7-separate-required-publication-from-retained-debug-artifacts) - -### 43. Inject one interaction diagnostics contract - -Proposal, validation, and repair duplicate path types and writer wiring. Supply a -diagnostic sink from the adapter boundary, with role-specific payloads and -nonfatal write warnings. Remove secret-provider assertions, lazy discovery -closures, combined-role secret lists, and preflight/writer credential arguments. -Keep artifact association and optional write-error reporting. -[Complexity §4](03-complexity.md#4-separate-validation-batch-assessment-from-the-diagnostic-writer) · -[Equivalence §5](10-equivalence.md#5-give-optional-interaction-artifacts-one-writer-contract) · -[Boundaries §6](11-boundaries.md#6-inject-interaction-diagnostics-at-the-adapter-boundary) · -[Redaction removal §4](14-redaction.md#4-remove-secret-plumbing-from-writer-and-preflight-contracts) - -### 44. Simplify structured-call completion and remove obsolete record fields - -Share finalization across ordinary calls and coordinated turns, and remove -obsolete error-kind state, response usage copies, and duplicate callback payloads. -Keep typed causes and live execution and artifact evidence. -[Complexity §5](03-complexity.md#5-make-the-structured-clients-call-lifecycle-explicit) · -[Dead code §5](12-deadcode.md#5-remove-structured-call-fields-superseded-by-execution-observations) - -### 45. Retire the test-only direct continuation branch - -The supported repair path uses the coordinator. Move direct-continuation -assertions about sibling isolation and structural repair to coordinated calls, -then remove the unused request branch. -[Complexity §5](03-complexity.md#5-make-the-structured-clients-call-lifecycle-explicit) · -[Dead code §1](12-deadcode.md#1-retire-the-test-only-direct-continuation-path) - -### 46. Use a single proposal handoff and test real runner indexing - -The generator builds enriched proposals the production wrapper discards. Give the -runner index ownership, remove the parallel enrichment, and assert indices under -observed out-of-order completion instead of in the test-only composition harness. -[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly) · -[Testing §6](08-testing.md#6-concurrency-fixtures-leave-some-schedules-unproved) · -[Testing §9](08-testing.md#9-delete-proposal-order-assertions-on-the-test-only-composition-harness) · -[Dead code §2](12-deadcode.md#2-remove-proposal-enrichment-that-the-live-handoff-discards) - -### 47. Centralize request-local segment reference resolution - -Initial proposals and repair revisions duplicate the same local-position lookup. -Preserve raw references, stable proposal identity, and revalidation for retargeted -edits. -[Equivalence §3](10-equivalence.md#3-resolve-model-segment-references-in-one-place) · -[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly) - -### 48. Give one repair branch a clear execution and result boundary - -Branch coordination, previews, revision validation, accounting, and diagnostics -share state inside the section loop. Separate branch state and make repair prompt, -exchange capture, and coordinator capability explicit. -[Complexity §2](03-complexity.md#2-separate-one-repair-branch-from-the-sections-repair-loop) · -[Complexity §6](03-complexity.md#6-use-a-single-proposal-handoff-and-wire-repair-capability-explicitly) - -### 49. Define concurrency and ownership in module and validator contracts - -Workers share module and validator instances and borrow section and glossary data. -Specify concurrent calls, read-only borrowed inputs, and cooperative cancellation -at the interface boundary. -[Concurrency §8](04-concurrency.md#8-shared-module-and-validator-ownership-is-implicit-at-their-interfaces) - -### 50. Consolidate root and per-module configuration rules - -Resolution and validation repeat role defaults, null and inheritance rules, and -field checks. Share the rules while keeping path-specific diagnostics and -configuration provenance. -[Complexity §10](03-complexity.md#10-share-root-and-per-module-configuration-rules) · -[Equivalence §6](10-equivalence.md#6-use-one-set-of-checks-for-inherited-configuration-fields) - -### 51. Unify transcript conversion and retire redundant parser wrappers - -Source-to-canonical conversion is duplicated across the CLI, a lenient wrapper, -and fixtures. The strict test-only wrapper panics on omitted IDs. Use one -conversion and the supported decoder, and preserve independent fixture -expectations. -[Efficiency §9](02-efficiency.md#9-the-strict-transcript-parser-panics-on-a-missing-id) · -[Testing §12](08-testing.md#12-consolidate-normalization-fixtures-at-the-domain-and-parser-owners) · -[Boundaries §9](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries) · -[Dead code §8](12-deadcode.md#8-remove-the-transcript-parser-variants-that-only-tests-call) - -### 52. Give public transcript encoders one segment projection - -The output registry implements the envelope and delegates the bare form to the -schema package. Share the field projection at the output boundary and verify both -wire shapes independently. -[Boundaries §9](11-boundaries.md#9-put-transcript-conversion-and-output-shape-at-their-owning-boundaries) - -### 53. Use one simple token estimate across text-processing packages - -Normalization and chunking implement the same word and punctuation heuristic. -Share the counter and preserve consumer-specific budgets and independent -expected-count tests. -[Equivalence §7](10-equivalence.md#7-use-the-same-simple-token-counter-for-text-estimates) - -### 54. Centralize transcript cloning - -Runner and proposal application duplicate deep-copy code with different -empty-category handling. Share the copy operation while keeping both caller -ownership boundaries and verifying independence from fixture data. -[Equivalence §8](10-equivalence.md#8-centralize-transcript-copies-without-moving-the-copy-boundaries) - -### 55. Assemble built-in modules at the composition boundary - -The framework registry imports all concrete modules even though the runner -consumes a factory interface. Keep config keys independent of implementations and -place production construction with the CLI or built-ins. -[Boundaries §3](11-boundaries.md#3-put-production-module-construction-at-the-composition-boundary) - -### 56. Merge shared prompt variables once per model call - -Processing loads the embedded instruction values twice. Keep processing precedence -and the missing protected-term default in one pass, and preserve caller precedence -for inspection. -[Dead code §6](12-deadcode.md#6-resolve-shared-prompt-variables-once-per-model-call) - -### 57. Clarify what the inherited progress flag does in processing - -`process` advertises progress messages but supplies no producer. **Resolved D7:** -add processing progress through CommandKit's reporter. Operators selecting -`--progress` want relatively detailed updates; expose meaningful stage/chunk, -proposal, validation, and repair activity as available, rather than only coarse -stage summaries. CommandKit owns mode precedence and stream routing; preserve -transcript stdout. Exact wording and event granularity are implementation choices. -[Dead code §10](12-deadcode.md#10-resolve-the-inherited-progress-flag-for-process) - -## Test maintenance and dead code - -### 58. Test registered validator roles and embedded assets through their behavior - -Consolidate redundant execution-class checks and assert required prompt identities -and preparability. Keep schema identity, diagnostic metadata, and protective-copy -coverage. -[Testing §10](08-testing.md#10-consolidate-validator-registry-and-asset-metadata-assertions) - -### 59. Consolidate overlapping normalization fixtures - -Two different-speaker fixtures protect the same rejection. Keep the distinct -domain cases and meaningful merge, text, and category assertions. -[Testing §12](08-testing.md#12-consolidate-normalization-fixtures-at-the-domain-and-parser-owners) - -### 60. Focus integration tests on decoded artifacts and independent output choices - -Assert decoded IDs and evidence associations and use smaller fixtures for -duplicate reporting and debug combinations. Keep a full scenario for attribution -through repair and later failure. Remove masking expectations without discarding -the independent artifact and outcome assertions mixed into those fixtures. -[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) · -[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) - -### 61. Remove abandoned test support and unused fixture inputs - -The orphan test-support package, unused CLI and runner helpers, and legacy -glossary fixture input have no current callers. Preserve compile-time interface -fixtures and active behavior assertions. -[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) · -[Dead code §9](12-deadcode.md#9-clean-up-abandoned-test-support) · -[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) - -### 62. Remove unreachable merge and file helpers - -The uncalled merge predicate and standalone JSON and file-write helpers duplicate -or bypass the live normalization, schema, and output paths. -[Dead code §7](12-deadcode.md#7-remove-orphaned-normalization-and-file-helpers) - -## Additional cleanup after resolved D1 - -### 63. Remove redaction-only credential discovery and tracking - -Audita enumerates captured profiles/backends, reads credential environment values, -caches names, and tracks prepared role names under a mutex solely for substitution. -Remove that state and the secret-source methods after updating their consumers. -Preserve PromptKit authentication, prepared evidence, execution synchronization, -and native inspection behavior. -[Redaction removal §1](14-redaction.md#1-remove-credential-discovery-and-tracking-used-only-for-substitution) - -### 64. Preserve validator explanations in repair feedback - -Repair rewrites validator explanations using both roles' credential values even -with debug off. Remove this transformation and use domain feedback directly. -Retain proposal association, unavailable-assessment semantics, and the existing -bounded repair/revalidation behavior. -[Redaction removal §5](14-redaction.md#5-stop-rewriting-validator-explanations-sent-to-repair) - -### 65. Remove credential substitution from CLI presentation - -Preparation and completion paths scrub warnings, errors, and announced paths; -completion discovers credentials even with reports/debug disabled. Use direct -CommandKit presentation, remove obsolete arguments, and retain typed causes, -reporting modes, exit status, and required-output failure handling. Coordinate -with #25 for setup detail still hidden by the separate failure wrapper. -[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 23e290b..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,829 +0,0 @@ -# Reliability and Maintainability Implementation Plan - -Implement the [target roadmap](reliability-and-maintainability.md) in one -end-to-end sprint, starting from its application baseline `c4ccd78`. The -[assessment](audit/assessment.md) governs the accepted scope and decisions -D1–D7. Numbers under each stage refer to the [audit catalog](audit/catalog.md). -Use the original audits for reproductions and source references. Where an old -remedy differs from a later decision, follow the assessment and target roadmap. - -Work through the numbered stages in order. Each stage is sized for one Luna -implementation prompt and contributes directly to the final design. Intermediate -source may be incomplete or fail to build while adjacent changes are in flight. -Use the next stage to finish those connections. A stage needs neither a -standalone commit nor a passing full suite. Replace obsolete paths directly; -compatibility layers and temporary production adapters are unnecessary. - -Use the [development guide](../development.md) and applicable adopted policies, -especially [agents](../policy/agents.md), [architecture](../policy/architecture.md), -[testing](../policy/testing.md), and [documentation](../policy/documentation.md). -Add or adapt focused tests at the behavior owner and update canonical docs when -their implemented behavior changes. Run affected tests when the relevant path -builds, then close the sprint with Stage 20. Stage checks establish behavior -without fixing private names, scheduler sizes, event order, or prompt wording. -The target roadmap defines retained contracts, selected changes, and exclusions. - -## Stage 1 — Establish the delegated inspection dependency - -Audit #24, delegated part. - -1. Use a temporary CLI build to reproduce invalid operator configuration with - `promptkit list prompts` and `process`. Reuse the malformed companion fixture - in `internal/cli/runtime_errors_test.go` and include a separate cancellation - case. The baseline reported status 1 for the malformed companion via - inspection and status 2 via processing. Both commands must remain offline. -2. Trace `internal/cli/app_wiring.go` into the pinned `ckprompt` and `ckconfig` - commands. For the known catalog case, `ckprompt.runCatalogList` converts the - loader error to `definitionFailure` even when a typed input cause is - available. Check relevant configuration-scope acquisition paths separately. - Preserve typed causes in Audita's loader. Classify only input failures at - the responsible boundary, leaving cancellation and runtime failures distinct. -3. Check for a released upstream fix for each demonstrated delegated case. If a - suitable release exists, adopt the smallest compatible update using the - [OSS policy](../policy/oss.md) and verify classification at the Audita CLI. - Otherwise record the affected command, reproduction, owner, and needed - behavior for an upstream handoff. Continue the independent stages and use - Stage 20 to integrate and verify a supporting release if available. Keep the - native declarations and routing throughout. - -A fix is demonstrated by status 2 and actionable diagnostics for the invalid -input, and status 1 for cancellation. While the supporting release is pending, -record the mismatch explicitly. A test that asserts only nonzero status does -not verify this fix. - -**Complete.** The upstream handoff below records the original behavior. -The [delegated inspection follow-up](#delegated-inspection-fix) records the -adopted fix. - -Build a temporary executable with `go build -o /tmp/audita-inspection ./cmd/audita`. -In a scratch directory, put `version: 3` in `config.yml`, `profiles: [` in the -adjacent `prompts.yml`, and `[]` in `transcript.json`. End each file with a newline. - -| Invocation with `/tmp/audita-inspection` | Status | Diagnostic | -| --- | --- | --- | -| `promptkit list prompts --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` | -| `process transcript.json --config config.yml` | 2 | `yaml.invalid` in `prompts.yml` | -| `config show --scope promptkit --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` | - -`config sources`, `config validate`, `config explain --list`, and -`config diff --against-config config.yml` also returned 1 for the `promptkit` -scope with the same diagnostic. All five `app` scope operations succeeded -with the malformed companion and returned 1 with `yaml.invalid` for a -separately selected main file containing `version: [`. These failures occur -during configuration loading, before provider work. The cancellation test -passes an already canceled context through the application entry point and -verifies status 1, `context.Canceled`, and empty stdout for catalog listing, -configuration show, and processing. - -Both native adapters receive the typed ConfigKit load cause from Audita's -existing loaders. In `ckprompt`, `runCatalogList` assigns the operational -outcome through `definitionFailure`. In `ckconfig`, the scope handlers -assign it through `configFailure`, with `show` calling it through -`configAcquisitionFailure`. The upstream owners are the respective CommandKit -integrations. They need to assign status 2 for typed operator-input failures -at those boundaries, preserving diagnostics, available inspection evidence, -and the cause. Cancellation and other operational failures need status 1. -At the time of the handoff, the latest release of both integrations was -`v0.7.0`, the version Audita used for the reproduction. - -## Stage 2 — Remove application credential substitution - -Audit #3, #9, #31 (source bytes), #63–65. Use the -[focused removal inventory](audit/14-redaction.md) for affected sites. - -1. Switch ordinary reports and debug artifacts to direct typed JSON/YAML - encoding. Keep encoded effective configuration from ConfigKit. In Markdown, - retain escaping and remove substitution. Keep malformed `final_content` as - the captured string. Write the acquired transcript bytes to the debug source - artifact and keep the parsed and normalized views separate. -2. Send domain `repairFindings` directly for initial feedback and subsequent - findings. Preserve the proposal association and typed unavailable assessment. - Present native errors, warnings, and actual paths through CommandKit. This - step removes substitution. Stage 12 exposes the native setup detail that is - currently hidden by generic failure presentation. -3. Remove writer/renderer secret arguments, `DebugInput.Secrets`, discovery - assertions and closures, repair role-list combination, cache and role - tracking, runtime secret discovery, and `internal/core/redaction`. Keep the - prepared and terminal callbacks, source-path tracking, native effective - rendering, and selected provider authentication. -4. Delete masking-only assertions, scanners, fixtures, and stale redaction - descriptions identified in the inventory. Move independent assertions to - writer or caller tests. Check decoded schema and scalar types, literal source - bytes, malformed response evidence, actual paths, and native auth using - synthetic data. Check feedback by decoding a production repair request. - -**Complete.** Reports and debug artifacts encode typed data directly, source -snapshots preserve the acquired bytes, and repair requests carry the original -domain findings. Removed application credential discovery and substitution. -The CLI build and full offline suite passed with Go 1.27.3, including checks -for native authentication, malformed response capture, and destination paths. - -## Stage 3 — Unify optional interaction capture - -Audit #33 and #43. - -1. Define one small optional interaction sink at the shared execution port. - Compose the existing filesystem writer from the run directory in CLI wiring. - Pass the sink through proposal, validation, and repair. Each role supplies - its own payload and stage/chunk/operation coordinates, including proposal, - batch, validator, and repair turn where applicable. Keep returned paths and - write errors available for ledger association and warnings. -2. Reuse the write path and artifact envelope in - `internal/framework/llm/diagnostics.go`. Remove private writer adapters and - repeated construction. Keep artifacts self-contained, including revalidation - and a final unusable or undelivered response when available. Keep debug - directory placement, naming safety, and creation modes. -3. Collect optional write warnings safely across workers, grouping equivalent - failures by operation and cause with counts and affected artifact information. - Continue later capture after a write error. Keep the warning path outside - correction decisions and required file delivery. - -Verify role, chunk, batch, proposal, and attempt associations with decoded -artifacts. Exercise a failed optional write followed by later capture. Update -the [diagnostics internals](../internal/diagnostics-reporting.md) for the sink. - -**Complete.** The CLI supplies one optional sink for proposal, validation, -repair, and revalidation artifacts. The writer records successful paths even -after a partial write and groups failures by role, artifact operation, and -cause. Decoded-artifact and recovery checks passed, along with the full Go -suite and targeted race checks. - -## Stage 4 — Bind publication to checked destinations - -Audit #1, #6 (cooperative boundaries), #7, #8, #16, #26, #42. - -1. In `internal/core/diagnostics/output_session.go` and its CLI callers, resolve - checked transcript and report destinations with filesystem semantics. - Carry those destination and parent choices through staging and publication. - Keep protected-source checks, intentional in-place replacement, ordinary - symlink and stream behavior, and disabled reporting's inactive destination. - Add no directory creation or locking for required destinations. -2. Give the required-output owner a shared regular-file staging primitive for - transcript and reports. Keep optional debug setup separate from that owner. - Keep temporary paths until removal succeeds or absence is established. - Surface cleanup errors alongside the initiating error, including after - partial preparation, publication failure, or cancellation. -3. Make report rendering return an explicit error through initialization and - finalization. Publish the transcript first when processing succeeded. - Select a failed bundle before the first final report write when a required - delivery or cleanup failure is already known. If a report write first - establishes failure, switch the remaining writes to failed status and make - one best-effort pass over reports that still need failed evidence. If the - failed bundle cannot be rendered, keep initialized or published evidence - and return the rendering error. -4. Check context when transcript acquisition returns, at natural preparation - and publication boundaries, and after stream delivery returns. A cancellation - observed after bytes escape still fails the invocation. Join cancellation - and I/O/cleanup errors where available and keep the initiating failure - distinguishable. Use ordinary blocking I/O and retain completed transcript - publication if a later report write fails. -5. Escape input/output paths as literal Markdown text in `core/reporting`, - preserving raw values in structured artifacts. Update - [operations](../operations.md) for caller-owned destination isolation, - retention, and the resulting publication lifecycle. - -Exercise symlink/path resolution, in-place and stream output, cancellation -after I/O, cleanup errors, initialized evidence when rendering fails, and -write behavior after known failure. Observe publication attempts as well as -final files. Leave processing fact and ledger projection to Stage 17. - -**Complete.** Required outputs use the destinations resolved during selection -and share regular-file staging. Publication checks cancellation, retains and -reports cleanup failures, and uses failed-status reports when required delivery -cannot complete. Tests verified publication attempts, retained evidence when -rendering failed, symlink destinations, stream cancellation, and Markdown path -escaping. The CLI build and full offline Go suite passed. - -## Stage 5 — Consolidate typed transcript data - -Audit #13, #31, #51, #52, #54, #62 (file helper). - -1. In `core/schema`, detect the two accepted root shapes while keeping the - segment array as raw JSON. Decode typed segments directly. Preserve source - ID presence, native integer range, duplicate checks, ignored unknown fields, - and existing format and error behavior. Keep one live source-to-canonical - conversion and retire obsolete parser wrappers after moving meaningful tests. -2. Centralize transcript cloning in the model. Use it at the runner and other - copy boundary, preserving independence and nil versus explicit empty - mutable slices. Share one typed segment projection across both encoders in - `core/schema` and `core/outputschema`. Preserve the envelopes, version, - numeric fidelity, category and omission rules, and encoding error behavior. -3. Release raw input after its final parse or source-snapshot write. Drop the - parsed source after normalization and its debug consumers. Avoid retaining - it via CLI finalization. Remove the audited unused file helper after checking - callers. Keep both file and stream inputs and current output delivery - semantics. Use a representative large fixture to check that the removed - representations no longer overlap needlessly. A fully streaming encoder is - optional if supported by the measurements and existing error semantics. - -Test both envelopes, adjacent accepted IDs beyond exact float precision where -supported by native `int`, clone independence with independent expected data, -and the distinction between raw and typed debug snapshots. Update the -[input contract](../integrations/transcript-glossary-files.md) and -[encoder internals](../internal/output-schemas.md) where affected. - -**Complete.** Source segments decode directly from both accepted JSON shapes. -Exact IDs, clone independence, encoder values, and debug snapshot fidelity were -verified. Processing releases raw bytes after parsing or source capture and -releases the parsed view before normalization. On a 100,000-segment object -fixture (Go 1.27.3, five iterations), parser allocations fell from 240.6 MB to -84.1 MB per operation. The CLI build and full offline suite passed. - -## Stage 6 — Correct and simplify normalization - -Audit #17, #21, #35, #53, #59, #62 (merge helper). - -1. In `core/normalization`, sort stably by the existing time key and retain - input order for ties. Compute the prospective merge end as the later end - and use that interval for the duration limit, merged segment, and next gap. - Keep overlap acceptance and the existing speaker, gap, and token rules. -2. Use one implementation of the existing token estimate across normalization, - chunking, and its consumers. Maintain the components needed for that exact - estimate as a merge grows. Accumulate text and category membership/order - incrementally with the existing joiners. Avoid rescanning or copying a - growing prefix on each append. Retire the unreachable merge helper. -3. Consolidate overlapping nonmerge tests. Keep independent fixtures for - nested intervals, ties, joiners, category ordering, and exact estimates. - Compare allocations or growth on a representative run of same-speaker merges. - -**Complete.** Verified stable start-time ties, nested intervals, exact joiner -and token behavior, and category order. The merge uses incremental text, -categories, and the shared token estimator. On synthetic same-speaker runs -(Go 1.27.3, three iterations each), allocations fell from 238.1 MB to 0.42 MB -per operation for 2,000 segments and from 960.1 MB to 0.85 MB for 4,000. The -CLI build and full offline suite passed. - -## Stage 7 — Make feasible explicit section targets succeed - -Audit #20 and the infeasibility portion of #24. - -1. In `core/chunking/sections.go`, use indivisible, ordered segments as the - partition units. Preserve the empty-transcript behavior and the singleton - exception for segments individually above the token maximum. Keep - `min_section_tokens` validation and compatibility without adding a new - partition constraint. -2. For an explicit target, compute the farthest legal end for each start - with a rolling token sum, honoring the oversized singleton exception. Work - backward to compute each suffix's minimum section count as one plus the - count at its farthest end. In the forward builder, reserve enough segments - for the remaining nonempty sections and use the suffix count to reject cuts - that leave too little capacity. Prefer the current average-token target - among the feasible cuts. Count sections by segment presence, including - zero-token segments. Preserve implicit-target behavior. -3. Return a typed infeasibility error for impossible requested targets. Carry - it through the runner and CLI so the existing input status applies while - internal and cancellation failures retain their classes. - -Verify the audited weights `[2,5,1]` with target 2 and maximum 6, impossible -targets, and the oversized singleton case. Check contiguity, order, coverage, -and feasibility independently of the chosen cut. Keep a single chunking pass. - -**Complete.** Explicit targets use rolling legal ends and suffix feasibility to -admit exactly the requested count when possible. An independent enumeration -checked 1,024 weight sequences for every possible count, including zero-token -and oversized segments. The real CLI returned status 2 for infeasibility -without dispatching model work. The CLI build and full offline suite passed. - -## Stage 8 — Share edit location and protection evidence - -Audit #12, #18 (punctuation portion), #19, #38, #47. - -1. In `framework/proposals`, search for a second literal starting offset for - require-unique replacements, including overlaps. Use the same applicability - rule in previews and apply. Preserve replace-all's nonoverlapping behavior - and the existing blank-result check. -2. Share the positional segment lookup for initial proposals and repair - revisions. Carry raw invalid model references to evidence, keep segment - positions distinct from candidate and validator response indices, and - revalidate a repair that retargets to another segment. -3. Move protected-name comparison to the glossary owner. Compare usable - full-segment before/after text and repeat the check against the current - segment at apply time. A missing or ambiguous preview cannot support a - protection verdict. An available empty preview may support the verdict - independently of blank-result rejection. Preserve active-glossary - exemptions, case and longest-match rules, and spoken-word repetition cleanup. - Use standard literal-edge checks for punctuation-edged spellings while - retaining Go regexp word-boundary semantics. -4. Reuse the glossary scan for context identity membership without expanding - each occurrence/contributor pair. Keep occurrence evidence for protection - and preserve the order of selected context. - -Test overlapping occurrences, full-segment protection, punctuation edges, -ambiguous previews, spoken-word cleanup, and stale apply-time changes at the -owning boundaries. Check the initial and repair lookup against independent -expected references, and compare work on a repeated-glossary fixture. - -**Complete.** Proposal preview and application agree on overlapping spans. -Initial and repair responses share positional lookup and retain original model -references in diagnostic evidence. Protection runs on usable full-segment -previews and checks the current text at application. The 2,000-mention -fixture with 128 shared contributors measured 81.0 MB to 0.27 MB allocated -and 77.8 ms to 2.1 ms per operation (Go 1.27.3, ten iterations). The CLI build -and full offline suite passed. - -## Stage 9 — Consolidate validator decisions and reconciliation - -Audit #10, #11, #27 (runtime class), #40, #49 (validator port). - -1. Use one shared owner for the validator interface, applicability, decisions, - and cardinality helpers. Remove duplicate ports and unused validation-only - envelopes after tracing live consumers. Keep concrete validator chains, - built-in prompt/rule ownership, and reusable LLM batching. -2. Resolve a single authoritative disposition before a candidate advances. - Reject contradictory internal results as fatal contract failures, and derive - redundant serialized approval from the validated disposition. Preserve full - normal cardinality and checked partial results on fatal returns. Preserve - zero-budget shortlisting, independent repair feedback, unavailable - assessments, and deterministic validator ordering. -3. Record the effective execution class selected by the runner on both initial - and revalidation decisions, including supported instance overrides. Keep - the existing default for validators without an explicit class. Document - concurrent-call and borrowed-input behavior at the port and allow - synchronized stateful implementations. -4. Preserve exact numeric references through structured-response decoding - using `json.Number` or raw numeric tokens. Reconcile with the requested - batch-local range before converting to local `int`. Recognize equivalent - integral decimal and exponent forms exactly and ignore extras that cannot - name a requested candidate, including values beyond native integer range. - Keep supplied numeric references in diagnostic evidence. Continue whole - response schema validation and structural repair upstream. One usable - decision keeps its outcome. Omission or duplication affects only its - requested candidate, and ignored extras on a successful batch retain their - existing recovery classification. - -Test with actual response decoding on a mixed batch and one legitimate decision -beside an out-of-range reference. Cover equivalent numeric spellings, -contradictory internal results, and class override from execution through the -record supplied to Stage 17. Update [validator internals](../internal/validators.md). - -**Complete.** Validators share one port and applicability owner. The runner -rejects contradictory decisions and records the effective class on initial and -repair validation evidence. Controlled provider responses verified exact -numeric decoding, equivalent spellings, duplicate and missing decisions, and -a usable decision alongside a reference above the native integer range. The -CLI build and full offline suite passed. - -## Stage 10 — Reduce validation context work - -Audit #36. - -1. In `framework/validators`, consolidate budget and optional-context policy - across the live validation batch builder and trimmer. Measure the required - payload first and stop testing optional material beyond the usable - ordered prefix. Reuse computations where the exact serialized estimate - permits it. Keep context priority, numbering, effective budget, oversized - single-input behavior, and independently measured wire size. -2. Retain a test oracle built from serialized payloads rather than the shared - estimator. Use a representative long optional-context fixture to demonstrate - reduced measurements or allocation. Retain request-shape and available - sibling expectations under over-budget input. - -**Complete.** Validation budgets measure required data first and stop at the -first optional entry that does not fit, with glossary entries preceding -neighbors. An independent serialized-payload oracle checked fit and batching. -With 5,000 optional entries and room for one, allocations fell from 3.12 MB to -2.13 KB per operation (Go 1.27.3, twenty iterations). The CLI build and full -offline suite passed. - -## Stage 11 — Hand checked selections to execution - -Audit #24 (readiness), #37, #49 (module port), #50, #55, #56. - -1. Consolidate Audita-owned root and stage validation/fallback helpers in - `core/config` after ConfigKit composition. Preserve presence/null, - explicit zero, inherited values, diagnostic paths and provenance, and - stage/role-dependent defaults. ConfigKit retains its load and override paths. -2. Replace duplicate derivation in `framework/preflight` and - `internal/cli/process_pipeline.go` with one checked processing plan. Store - resolved specs, validated role choices, and effective glossary/protection - views. Cache repeated profile/backend inspections and immutable glossary - views only within the captured runtime. Include ordered selection and - exemptions in reuse keys. Preserve readiness check order, first failure - attribution, protected source paths, no model calls during preflight, and - preflight-before-publication. Return typed status-2 failures for invalid - selections; keep runtime/canceled outcomes distinct. -3. Move built-in construction to composition beside the concrete modules or - in CLI wiring. Keep the injected runner factory and dependency-light key - catalog. Document concurrent calls and borrowed data at the module port. - Copy owned inputs where mutation is required. -4. In processing and inspection request construction, merge shared variables - once per call while preserving each path's precedence, the protected-term - default, and operator replacements for bundled prompts. - -Check representative effective selections and provenance, first-failure -attribution, provider-free inspection, constructible catalog entries, and both -request precedence paths. Update [CLI/config internals](../internal/cli-config.md), -[module internals](../internal/modules.md), and configuration docs where affected. - -**Complete.** Configuration uses common validation and role fallback after -composition. Preflight passes checked role and glossary selections to execution -and reuses inspection and glossary work within the invocation. The built-in -factory lives beside the modules. Processing and native inspection preserve -their respective variable precedence with one merge per request. The CLI build -and full offline suite passed, including status 2 for an unknown processing -profile before provider dispatch or output initialization. - -## Stage 12 — Unify structured-call completion and failure facts - -Audit #23, #25, #41, #44. - -1. In `framework/llm` and `framework/structuredoutput`, share once-only - completion between ordinary execution and coordinated turns. Keep - prepared snapshots, call IDs, terminal synchronization, native group and - call counts, coverage and unknown usage, and diagnostic-only content from - undelivered responses. Remove obsolete response/usage copies and callback - payloads after checking report and debug consumers. -2. Carry typed class, reason and cause from adapter through operation records. - Use the internal-failure reason when a reason is absent, retaining the - original class and cause. Apply parent cancellation precedence, identify - native logical-operation deadline expiry, then native transport/attempt - timeout before treating a generic deadline as logical timeout. Use the - existing transport-failure reason for HTTP-only timeouts. -3. Make setup diagnostics actionable at the CLI. Surface the native missing - credential requirement and available role/profile context with the typed - cause intact. Keep ordinary reports on their compact typed projections. - -Exercise distinct parent, operation, and attempt timeouts with independently -controlled budgets. Check setup class/cause and useful native information, -plus ordinary/coordinator terminal and undelivered-evidence paths. Update -[LLM runtime internals](../internal/llm-runtime.md). - -**Complete.** Ordinary and coordinated calls share once-only completion, -including prepared and undelivered evidence and native terminal counts. -Proposal, validation, and repair retain typed failure facts. Independent -parent, operation, and attempt deadlines produced the expected classifications. -The CLI identifies a missing required credential by role, profile, and -environment variable while the ordinary report retains the setup reason. -The CLI build, full offline Go suite, and targeted race checks passed. - -## Stage 13 — Use one proposal handoff - -Audit #46. - -1. In `framework/proposal_generation` and `framework/runner`, return one - set of decoded proposal results with raw local references, filtering - evidence, and optional native exchange. Remove the generator's enriched - projection discarded by the live wrapper. Keep runner ownership of - sequential proposal indices within a stage, ordered by chunk and response. -2. Check emitted interaction metadata such as `start_index` separately from - private references. Preserve its meaningful serialized association and - retire fields confirmed to have no surviving purpose. -3. Move proposal-order assertions from the test-only composition harness to - real runner results. Observe out-of-order completion in the fixture and - verify indices and original references. Keep model response and reference - coverage in the generator's owning tests. - -**Complete.** The generator and runner share one decoded proposal result with -raw local references, operation facts, and the optional native exchange. The -runner assigns indices in section and response order. A production-module -fixture observed the later response first and verified applied indices, -operation associations, and raw references. Proposal diagnostics retain -the response-local `start_index`. The full offline suite and repeated -ordering checks under the race detector passed. - -## Stage 14 — Own each correction repair branch explicitly - -Audit #34, #45, #48, #64 (branch integration). - -1. In `framework/runner/repair.go`, give one proposal branch a clear - execution input and result. Make repair capability explicit at setup. Keep - branch state and terminal outcome together. Merge outcomes into the section - after each branch, without mutating unrelated approvals or decisions. -2. Run repairs through PromptKit coordination with full exchange/history and - configured per-correction limits. Preserve initial and subsequent direct - domain feedback, sibling independence, withdrawal, unavailable assessment, - retargeting, and revalidation of the final allowed revision. Move useful - structural repair and isolation tests onto this path before deleting the - direct-continuation-only test branch. -3. Use controlled observations or temporary instrumentation to measure - cumulative turns, available token/byte costs, revalidation calls, and debug - work on a representative multi-correction fixture. Remove unnecessary local - copies. Keep those measurements out of the ordinary report contract. - -Check independent approved siblings, attempts/turn counts, actual branch -history and feedback, final-turn revalidation, and associated artifacts in -production coordination. - -**Complete.** The runner checks repair capability before proposal dispatch. -Each proposal branch owns its history reference, attempt facts, and terminal -outcome. The section merges the returned evidence and approvals. Tests exercise -PromptKit coordination for independent histories, replacement prompts, -structural repair, final-turn revalidation, and retained siblings and -artifacts. The CLI build, full offline suite, and targeted race checks passed. - -The three-correction CLI fixture used two repair branches and three repair -turns, with four revalidation requests and ten provider requests overall. -The default prompt sent 9,487, 13,385, and 9,483 repair request bytes in both -debug modes. Debug produced 39 files totaling 135,659 bytes in the observed -run, compared with zero files when disabled. The replacement prompt sent -6,303, 7,017, and 6,299 repair request bytes and produced 122,943 debug bytes -across 39 files. The fixture supplied no provider token usage. Temporary -measurement instrumentation was removed. - -## Stage 15 — Give stage execution one commit boundary - -Audit #14, #15 (evidence), #22 and #41 (failure facts). - -1. In `framework/runner`, check context at entry, including empty pipelines, - and before committing a stage. Use one stage completion path to record - outcomes and evidence. Only a successful active stage advances the working - transcript. Preserve previously committed text for partial results. -2. Carry initiating cause, phase and available stage/chunk/operation reference - from the owner. Cancel and join peers on fatal failure or parent cancellation; - keep peer interruption and secondary errors attributable without promoting - them over the initiating failure. Continue recoverable work under existing - partial-success semantics. -3. Collect completed proposal evidence independently of dispatch to validators. - After workers join, include usable out-of-order completions in interrupted - diagnostics with deterministic indices and associations. Keep raw evidence - from undelivered responses separate from usable proposals. Retain the ordered - indexing/validation boundary until Stage 16 evaluates it. - -Test canceled entry and precommit, a later completed proposal behind a slow -earlier chunk, and a fatal stage after earlier committed text. Assert causal -origin and preserved evidence as well as returned failure. Leave report/ledger -projection to Stage 17. - -**Complete.** A single module completion path advances the transcript only -after a successful application and context check. The runner returns the -initiating error with its phase and available origin, and keeps peer failures -on their work units. After joining, it assigns deterministic indices to usable -proposal responses that never reached validation. Tests exercised canceled -entry, cancellation after approval, a later response behind an earlier -failure, and preservation of earlier committed text. The CLI build, full -offline suite, and targeted race checks passed. - -## Stage 16 — Bound local preparation and actively drain results - -Audit #15 (dispatch) and #30, under D3. - -1. Replace the per-chunk goroutines and chunk-count-sized pending channels in - `collectSectionProposals` and `runModulePipeline` with a private rolling - stage window. Admit chunks to the window in input order. Charge a slot for - each chunk still in preparation, model work, or awaiting downstream work. - Release it when the chunk is finished and its final facts have been drained. - Start with a small fixed local window and keep it private. Drain outcomes - while workers run and on cancellation, then join and finish. Keep compact - or requested completed evidence apart from remaining work. Collect results - while the workers can still send them. -2. Measure staged overlap and peak pending work with many sections, a - controlled slow first proposal, and shared PromptKit capacity. Where active - draining and ordered indexing yield adequate progress, keep that simpler - boundary. If ordered indexing still leaves repeatable idle capacity with - validation-ready chunks, use a chunk index and response position to identify - candidates during work. At ordered collection, assign the existing stage - indices and translate every projected proposal reference, including failure - indices, repair lineage, and interaction-to-ledger mappings. Keep the - identity-sensitive artifact data in the charged window until the index is - assigned; use the Stage 3 writer for the final artifacts. Keep batch-local - model references and raw response evidence intact. Check the same mappings - after a cancellation joins the workers. -3. Keep the module barrier and deterministic application. PromptKit retains - admission, queueing, limits, retries and recoverable rejection. Include - nested validation/repair in the local resource and cancellation checks - without duplicating backend admission or holding its permits across a - nested call. Tune the window using the measured workload and contention; - no public window, throughput minimum, or eventual-admission guarantee follows. - -Test actual observed overlap, finite native contention, cancellation with work -pending, and identical index ordering for the same usable results under -different completion orders. Report representative pending-work measurements -without pinning a timing or worker-count threshold in the default suite. - -**Complete.** The runner admits sections through a bounded rolling window, -drains active outcomes on cancellation, and lets validation and repair proceed -while earlier proposals are pending. It assigns final indices in section order -and publishes identity-sensitive artifacts through the interaction writer. -Tests checked native contention and overlap, stable ordering, and repair and -revalidation evidence after completion and cancellation. The full offline Go -suite and targeted race checks passed. - -In three runs of a 36-section fixture with two shared provider slots and the -first proposal held for 250 ms, ordered dispatch completed seven later proposals -and no validations during the hold. With section-local identities, seven later -validations also reached the provider before release, with eight sections -charged to the window in both variants. Both variants completed 36 proposals -and 36 validations overall. The window of eight was retained for this workload. - -## Stage 17 — Project typed report and ledger facts - -Audit #22, #27 (ledger), #39, #41 (projection). - -1. In `framework/processreport`, build the snapshot from the Stage 12 and 15 - facts and Stage 4 delivery outcomes. Preserve phase and available origin for - setup, processing and publication errors. Keep recovery and work-unit counts - and known versus unknown values. Distinguish the primary run error, - interrupted peers, and secondary output failures without counting a fatal - run failure again as a recoverable unit failure. -2. Join proposals, validator decisions, repairs, application and skip evidence - once by stable identities in `correction_ledger.go`. Use the recorded - execution class from Stage 9. Project one candidate row with correct - disposition, lineage and literal empty replacement values. Retain - module-only failures. Preserve serialized fields and logical ordering. -3. In runner application attribution, reuse section/segment lookups and - aggregate skipped evidence once. Preserve the distinction between stale - apply-time conflicts and response filtering. Keep exact correction, - filtered-proposal and validation-decision identities across projection. - -Verify decoded YAML, Markdown counts, and ledger facts against an independently -specified mixed run, including partial decisions, repair/filter interaction, -later failure, and an overridden validator class. Update the -[output contract](../integrations/output-files.md) and diagnostics internals. - -**Complete.** Reports project typed operation, processing, and delivery -facts, with separate attribution for the primary failure, interrupted peers, -and file failures. The ledger joins each candidate's evidence once, uses -recorded validator classes, and preserves literal empty replacements and -repair lineage. Apply-time filtering reuses original section lookups and -groups skip evidence by its originating operation. The full offline Go suite -passed, including mixed YAML and Markdown counts, ledger class overrides, -and the repair/filter/later-failure subprocess fixture. - -## Stage 18 — Retain evidence according to selected outputs - -Audit #32 and supporting work for #30, #33, and #44. - -1. Derive evidence needs once from enabled ordinary reports and effective debug - capture. Keep compact CLI totals and available failure facts in summary-only - mode. Allocate work-unit detail only when a consumer needs it, and avoid - eager whole-run grids and debug-only candidate/decision histories in - summary-only paths. Keep complete enabled report matrices and debug evidence. -2. Discard active validation, exchange and repair state after their last - consumer, including data copied solely for obsolete runner projections. - Remove duplicate operation lists and unused module totals after checking - their projections. Keep native observations and requested self-contained - artifacts when capture is selected. -3. Compare live/retained state for representative long runs with summary only, - ordinary reports, debug, and their existing combinations. Check shared - totals and logical results across modes, with detail present where selected. - Resource measurements support reduction of waste without setting a - constant-memory or disk-use requirement. - -**Complete.** The CLI selects ordinary report and debug evidence before -processing. Section outcomes feed shared totals as they drain. The runner -retains the full work grid only for reports, and retains candidate and -validator history only for debug in the CLI. Duplicate module operations and -unused proposal totals were removed. Prepared messages are captured only for -interaction diagnostics. Mode comparison tests and the full offline suite -passed. - -A synthetic 4,000-section run with recoverable failures, proposal recoveries, -and apply-time filtering retained about 0.46 MB with summary only, 2.73 MB -with ordinary reports, 1.56 MB with debug evidence, and 3.83 MB with both -after garbage collection. Allocations per run were approximately 20.9, -22.0, 24.0, and 25.0 MB respectively. These runner measurements exclude -provider calls and disk artifacts. All four modes agreed on the transcript -and counters. Temporary measurement instrumentation was removed. - -## Stage 19 — Emit native processing progress and update operator guidance - -Audit #5, #57, and the operator-facing parts of #8, #24 and #25. - -1. Use a small event callback from the live lifecycle owners. Supply stage and - chunk activity, proposal/validation/repair phases and useful completions - with available identities and observed counts. Reuse suitable PromptKit - events where they convey useful activity. Keep event delivery within the - invocation and finish workers before the reporter closes. -2. Wire that callback in CLI to `Invocation.Reporter().Progress` in pinned - CommandKit. Let CommandKit determine visibility, routing, and presentation - failure status, independently of transcript/report processing outcomes. - Preserve stdout transcript data. Respect ordinary mode, explicit progress, - debug defaults, explicit `--progress=false`, and quiet suppression. -3. Update the [CLI reference](../cli.md), [operations](../operations.md) and - [subprocess contract](../integrations/subprocess.md) at their owned - boundaries. Cover meaningful progress, caller-owned report destinations and - artifact retention, and the trusted configuration bundle, including - discovery, imports and automatic companions. Keep the distinction between - native configuration inspection and processing readiness. - -Use representative command tests for live activity and combined modes, -asserting useful identities and stream separation without exact wording or -concurrent ordering. Keep a progress-specific check separate from delegated -inspection availability. - -**Complete.** Runner lifecycle and native model events flow through -CommandKit progress reporting. Subprocess tests observed proposal progress -while a provider was blocked, checked repair activity, verified stdout -separation and reporting-mode combinations, and confirmed that a failed -progress destination leaves completed output intact. The CLI, operations, -and subprocess guides cover live progress, destination ownership, and -configuration trust. The full offline Go suite passed. - -## Stage 20 — Verify the integrated sprint and release bootstrap - -Audit #28, #29, #58, #60–62 and sprint closure. - -1. In `.woodpecker/release.yml`, capture `git status --porcelain --untracked-files=all` - in an assignment that checks the command's exit status separately from its - content. Then require empty output. Preserve ReleaseKit's independent - prepared-checkout validation and tag delivery. In `scripts/release_test.go`, - cover clean, dirty and failed-status-with-empty-output behavior at the - command boundary. Remove redundant tests of the comparison helper while - keeping real release target and wiring checks. -2. Verify signal, cancellation and exit behavior through the real executable - or a shared production entry point. Keep narrower checks where a process - adds no evidence. Verify effective validator roles, required prompt - identities, embedded asset preparability, schema identity, and protective - copies through use. Retain a representative repair/filter/later-failure - integration scenario and independent output-choice cases. Remove the - audited abandoned fixture inputs and helpers after checking callers. -3. Recheck Stage 1. Adopt and verify a released upstream fix that became - available using the same scope. Otherwise report the exact outstanding - upstream-dependent behavior and its disposition. Treat that item as - outstanding, while verifying the independent Audita work. Keep the - invalid-input and cancellation cases distinct. -4. Account for the [catalog](audit/catalog.md) against the landed changes and - stage evidence. Entries #2 and #4 need no implementation, and only the - accepted portions of #6 and #18 were selected. Check canonical docs, - examples, and affected links and remove stale implemented-behavior claims. - Keep historical audit observations intact. -5. From the integrated tree, build the CLI and run the full offline - `go test ./...` with the supported Go toolchain. Run targeted race checks - for changed shared-state paths and review the representative measurements - from Stages 6, 8, 10, 14, 16 and 18. Verify formatting and whitespace, - and inspect the final diff. Reuse successful checks for unchanged content. - Existing [release validation](../policy/release.md#candidate-validation) - applies when a release is prepared. - -The completed sprint should leave the target architecture reachable through -`audita process` and the standard delegated commands. Report checks and -measurements actually performed, including any outstanding external dependency. - -**Complete.** The integrated Audita work and release bootstrap passed checks. -The [delegated inspection follow-up](#delegated-inspection-fix) closes the -upstream item that was open at the end of the sprint. - -The bootstrap rejects a dirty checkout and a failed status command with no -output. A clean fixture reaches ReleaseKit with the prepared checkout and -expected commit arguments. Process fixtures share the production entry point -and check interrupt and termination exit status. Registry coverage checks -effective roles, required prompt identities and preparation, schema identity, -and independent copies. The ordered processing fixture decodes debug artifact -coordinates. The full repair/filter/later-failure case remains separate from -the smaller output-choice matrix. Unused fixture helpers and the legacy -glossary input were removed. - -### Delegated inspection status - -At sprint close, the latest CommandKit integration releases were `ckprompt` -and `ckconfig` v0.7.0. The built Audita executable reproduced the Stage 1 -mismatch: - -| Invalid input | Path | Observed status and diagnostic | -| --- | --- | --- | -| Malformed automatic `prompts.yml` | `process` | 2, `yaml.invalid` | -| Malformed automatic `prompts.yml` | `promptkit list prompts` | 1, `yaml.invalid` | -| Malformed automatic `prompts.yml` | `config show/sources/validate/explain/diff --scope promptkit` | 1, `yaml.invalid` | -| Malformed explicit app main file | `config show/sources/validate/explain/diff --scope app` | 1, `yaml.invalid` | - -Caller cancellation returned status 1 in the shared CLI and subprocess tests. - -### Delegated inspection fix - -Audita adopted `ckprompt` v0.9.0, `ckconfig` v0.8.0, and PromptKit v0.23.1. -PromptKit preserves `ErrInvalidConfig` through `errors.Is` and unwraps to the -original cause. Both integrations can now classify the typed ConfigKit load -error as invalid input. Audita uses the existing native command wiring. -[ckprompt v0.9.0 release notes](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/29694ea874b53151d832c141b7c5913779852e0e/integrations/ckprompt/docs/releases/v0.9.0.md) · -[PromptKit v0.23.1 release notes](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/74066167942fcf6f70c112359995739ec9789fa6/docs/releases/v0.23.1.md) - -The Stage 1 fixture now returns status 2 with `yaml.invalid` identifying -`prompts.yml` for `promptkit list prompts`, `process`, and all five PromptKit -configuration commands. `config sources` and `config validate` still emit -their reports. All five app-scope commands likewise return status 2 for a -malformed main file. The CLI cancellation test still verifies status 1, -`context.Canceled` in the cause chain, and empty stdout. - -### Catalog closure - -The table identifies the implementing stages and exceptions without changing -the historical audit observations. Each stage records its verification above. - -| Catalog findings | Implementing stages or disposition | -| --- | --- | -| #1, #6–8, #16, #26, #42 | Stage 4. Only cooperative cancellation boundaries from #6 were selected. | -| #2, #4 | No implementation. The existing catalog dispositions stand. | -| #3, #9, #63–65 | Stages 2 and 14. | -| #5, #57 | Stage 19. | -| #10–11, #27, #40, #49 | Stages 9, 11, and 17. | -| #12, #18–19, #38, #47 | Stage 8. Only punctuation-edge handling from #18 was selected. | -| #13, #31, #51–52, #54 | Stages 2 and 5. | -| #14–15, #22, #30–32, #39, #41 | Stages 5, 12, and 15–18. | -| #17, #21, #35, #53, #59, #62 | Stages 5 and 6, with final caller checks in Stage 20. | -| #20, #24 | Stages 7 and 11 for processing, and the [delegated inspection fix](#delegated-inspection-fix). | -| #23, #25, #44 | Stages 12, 18, and 19. | -| #28–29, #58, #60–61 | Stage 20. | -| #33, #43 | Stages 3 and 18. | -| #34, #45, #48 | Stage 14. | -| #36 | Stage 10. | -| #37, #50, #55–56 | Stage 11. | -| #46 | Stage 13. | - -Verified with Go 1.27.3 by building the CLI, running `go test -count=1 ./...` -offline, and running targeted race checks for the runner, LLM adapter, -validators, reports, diagnostics, glossary, and CLI. The final review -checked Go formatting, whitespace, and 278 relative documentation links -and exercised the example configurations. Measurements above cover merge -and glossary allocation reductions, validation context, repair request and -artifact costs, bounded section contention, and retained evidence by output -mode. These are synthetic measurements. No release was published. diff --git a/docs/roadmap/reliability-and-maintainability.md b/docs/roadmap/reliability-and-maintainability.md deleted file mode 100644 index a448745..0000000 --- a/docs/roadmap/reliability-and-maintainability.md +++ /dev/null @@ -1,444 +0,0 @@ -# Audita Reliability and Maintainability Roadmap - -**Date:** 2026-11-15. **Status:** [Implementation landed](implementation.md). -[Post-implementation findings and their status](remediation.md) track follow-up work. -**Planning baseline:** `730c1e4`; its application code matched the audited `c4ccd78`. - -## Purpose and scope - -Audita should produce faithful transcripts and useful evidence, handle partial -success and failures consistently, and use resources proportionately to the work -and outputs selected. Its components should have clear owners, with less local -machinery where standard Go facilities or the adopted libraries already supply -the behavior. Operators requesting progress should be able to follow a run in -detail through CommandKit. - -This roadmap organizes the accepted audit remediation around that target state. -The [assessment](audit/assessment.md) owns the disposition of individual findings -and the rationale for decisions D1–D7; the [catalog](audit/catalog.md) links to -their evidence. The coverage appendix below maps that evidence to the target areas. -Earlier source-audit remedies remain proposals where the assessment supersedes -them. This document specifies the target state and accepted scope. The -[implementation plan](implementation.md) owns the sequence and verification. - -The baseline already had native library integration, preflight, -ordered module execution, partial-success validation, bounded correction repair, -ordinary reports, and opt-in debug artifacts. The sprint improved those paths -and added processing progress. It preserved editorial policy and the public -configuration schema. The D1 architecture-policy update preceded the related -code cleanup. - -### Design boundaries - -Follow the [architecture policy](../policy/architecture.md) and use the libraries' -supported paths. ConfigKit owns composition, presence, provenance, and resolution -before handoff. PromptKit owns credentials, model execution, admission, retries, -and conversation coordination. CommandKit owns standard commands, reporting -modes, and stream routing. ReleaseKit retains its release-validation ownership. -Audita owns transcript meaning, role selection, domain validation and repair -decisions, processing coordination, and presentation of its results. - -The accepted decisions establish these boundaries for this remediation: - -| Area | Selected scope | -| --- | --- | -| Diagnostic content | Remove Audita's credential substitution and the discovery/plumbing supporting it; retain native library behavior and useful evidence. | -| Report destinations | Keep existing placement; callers isolate concurrent jobs and retained run histories. | -| Admission | Keep native waiting and recoverable rejection. Improve local scheduling without a second backend limiter, admission-retry loop, or eventual-admission promise. | -| Input and capture resources | Remove waste; add no transcript-size cap, capture quota, automatic artifact deletion, or automatic stop-capture policy. Operators own retention and integrations own their upload/process budgets. | -| Correction repair | Retain full conversation history and existing per-correction limits. Aggregate budgets and reduced context are not selected at this time. | -| Progress | Supply relatively detailed processing activity through CommandKit when progress is requested, subject to its effective reporting mode. | - -Respect the three accepted Go limitations: standard HTTP redirect behavior, -standard regexp word-boundary semantics, and the inability to interrupt an -already-blocked ordinary file/stream call. Fix Audita's own choice of operation -and its handling when control returns. Output correctness here does not promise -hostile-filesystem race resistance, cross-file transactions, power-loss recovery, -or atomic synchronization with cancellation. Keep ordinary symlink, in-place -output, and stream support. Debug root locations and creation modes stay as they -are for this cleanup. - -The separate features in [deferred work](future.md), including semantic recovery -of unavailable validator decisions and independent whole-pipeline traversal by -chunk, remain outside this roadmap. Internal refactoring should preserve current -contracts except for the deliberate changes and correctness fixes described here; -it need not preserve incidental implementation details or known defects. - -## 1. Truthful output and useful diagnostic evidence - -### Target state - -Required output writes use the destinations that were actually checked. Reports -reflect known processing and delivery outcomes, retain available failure origin, -and remain useful when finalization itself encounters an error. Optional debug -capture has a separate lifecycle and does not determine processing success. -Diagnostic representations preserve the information supplied to them through -ordinary serialization. - -The baseline resolves some output paths differently at checking and writing, -can publish a completed report after learning of delivery failure, and can lose -initialized evidence or staging-cleanup ownership. Credential substitution adds -serialization passes and can alter schema keys, messages, and path announcements. -The [output contract](../integrations/output-files.md) and -[operations guide](../operations.md) describe the existing output surfaces. - -### Work to reach the target - -- Give required-output preparation, staging, publication, and cleanup one clear - owner. Resolve paths using filesystem semantics and carry checked destination - information into publication. Share regular-file staging mechanics between - transcript and reports, retaining private staging and intentional in-place - replacement. Keep ownership of temporary paths until removal succeeds or - absence is established; report cleanup failures alongside the primary error. -- Keep transcript-before-report publication and nontransactional finalization. - Select failed report status when a delivery failure is already known, carry - report-rendering errors explicitly, and preserve initialized reports when a - replacement failure bundle cannot be rendered. Retain bounded best-effort - attempts to publish enabled reports; a later report failure does not erase an - already-published transcript. -- Check cancellation at natural publication boundaries and after stream delivery - returns. A canceled run should report failure even if bytes have already - escaped. Do not implement detached I/O workers to simulate interruption of - standard blocking calls. -- Build ordinary reports from processing-owned facts, preserving phase, available - stage/chunk/operation origin, documented counts, and existing serialized - identities. Distinguish the initiating failure, peer cancellation, and secondary - output errors. Join correction evidence once per proposal before projecting - ledger rows; carry the validator execution class actually used rather than - reconstructing it from a name. Preserve disposition, repair lineage, empty - replacement values, partial decisions, and module-only failures. -- Complete the [redaction-removal inventory](audit/14-redaction.md): delete - redaction-only credential enumeration, environment reads, name caches, role - tracking, substitution utilities, and secret-only parameters/interfaces. Encode typed - JSON/YAML directly. Save acquired source bytes without reconstructing or - reopening the source; keep parsed and normalized views separately. Preserve - ConfigKit rendering and native authentication/inspection. -- Supply one interaction-writer contract from the adapter/composition boundary. - Proposal, validation, and repair owners supply their own payloads and - association coordinates. Remove lazy secret callbacks and cross-role secret - lists. Retain self-contained interaction artifacts, malformed final response - content as evidence, nonfatal write callbacks, and their synchronization. - Consolidate equivalent write warnings with counts and affected-artifact - information while continuing best-effort capture. -- Keep Markdown escaping separate from substitution: input/output paths should - render as literal text without changing their structured values. Document - caller-owned report-directory isolation and artifact retention when updating - the operator guide. - -### Completion evidence - -Focused filesystem tests cover checked destinations, known delivery failure, -failed report rendering, cleanup errors, and cancellation once I/O returns. -Assertions inspect publication behavior as well as final files. Decoded reports -retain their schema and types, and ledger evidence agrees with execution. -Artifact tests establish source fidelity, role/attempt association, and nonfatal -debug failures. Redaction-only code and assertions are removed without replacing -them with no-op seams or losing native authentication coverage. - -## 2. Bounded processing with clear operation ownership - -### Target state - -Local preparation, dispatch, and completed-result handling avoid allocating a -worker and prepared request for every chunk at once. Completed evidence is owned -independently of ordered application. Each stage and model operation has a clear -completion path, and typed outcomes reach reports without being reconstructed -from display strings. - -The baseline fans out work across all chunks, retains out-of-order results behind -slow earlier work, and duplicates operation state across callbacks and projections. -Backend admission already belongs to PromptKit; it does not bound all of this -local preparation and retention. The [pipeline](../internal/pipeline.md) remains -ordered by stage, with fixed chunk membership and deterministic proposal/application -identity. - -### Work to reach the target - -- Give stage execution a single result/commit boundary. Honor cancellation on - entry, including an empty pipeline, and before committing new stage work. - Cancel and join peers on fatal failure or parent cancellation, retaining earlier - committed stage text and available completed evidence. Recoverable operation - failures remain local under the existing partial-success rules. -- Design bounded local preparation and dispatch together with concurrent draining - of results. Preserve deterministic identities, validator requirements, ordered - application, and the module barrier. Reduce avoidable within-stage waiting - without introducing independent whole-pipeline chunk traversal. Retain completed - proposal evidence on interruption even when an earlier section is unfinished. - Preserve section- and response-ordered serialized proposal indices. If - measured waiting warrants earlier validation, use local candidate references - during processing and assign serialized indices at ordered collection. - PromptKit's configured rejection remains valid; the old scheduling-dependent - choice of rejected chunks is not a compatibility requirement. -- Retain only the state needed by active processing and selected outputs. Keep - live validation/repair state, compact summary counts, and required failure - evidence. Avoid full debug histories or work-unit matrices in summary-only - paths; preserve complete detail when that output is enabled. Remove duplicate - operation arrays, unused module totals, and obsolete response/callback copies. - This is not a constant-memory promise: selected outputs and useful conversation - history can still grow with the workload. -- Carry typed class, reason, and cause through proposal, validation, repair, and - reporting. Preserve the original class/cause if a missing reason uses the - internal-failure fallback. Distinguish parent cancellation, logical-operation - timeout, and HTTP-attempt timeout using native classifications; an HTTP-only - timeout uses the existing transport-failure reason. -- Share ordinary-call and coordinator-turn completion, retaining native terminal - counts and diagnostic evidence from undelivered responses. Remove the test-only - direct-continuation branch after moving useful coverage onto production - coordination. Keep one proposal handoff with raw local references; the runner - owns global indexing. Review emitted metadata before deleting fields that only - appear unused to a function-level reference search. -- Give each repair branch an explicit execution/result boundary. Preserve native - conversation coordination, full history, per-correction limits, sibling - independence, withdrawal, and validation of the final permitted revision. - Pass domain validator explanations directly instead of applying credential - substitution; retain the distinction between editorial rejection and unavailable - assessment. Measure aggregate work to guide efficiency work, without adding - new stopping rules or ordinary-report fields for that analysis. - -### Completion evidence - -Controlled overlap and cancellation tests exercise actual dispatch, admission, -result draining, and stage commit paths. They establish native shared backend -capacity, useful evidence retention, and deterministic identities under observed -out-of-order completion. Distinct live timeout budgets exercise the correct -failure owner. Representative measurements demonstrate reduced pending work and -retained representations; they do not impose a new throughput target or require -every chunk to obtain admission. Repair tests run through the production -coordinator and preserve accepted feedback and revalidation behavior. - -## 3. Faithful transcript and correction semantics - -### Target state - -Transcript parsing preserves accepted values, normalization preserves time and -ordering meaning, and chunking satisfies feasible configured constraints. -Validation and application use consistent domain evidence. Efficient shared -operations reduce repeated work without changing editorial acceptance rules. - -The baseline's generic JSON shape conversion can round IDs; normalization and -chunking have concrete interval, ordering, and feasibility counterexamples. -Some protected-term checks use only edit spans, and validation response conversion -can discard usable decisions because of irrelevant extra references. - -### Work to reach the target - -- Detect the supported transcript envelopes while keeping their segment array as - raw JSON for typed decoding. Preserve optional IDs, accepted numeric ranges, - unknown-field behavior, and both input shapes. Share live source-to-canonical - conversion, retire obsolete strict/lenient wrappers, and release whole-input - representations after their last use. Keep accepted formats and stream inputs. -- Use the full merged time interval for merge eligibility, output, and subsequent - gaps. Preserve source order for tied timestamps without introducing a new - end-time sorting criterion. Build growing text/category state incrementally, - preserving exact joins, category order, and the current token estimate. -- Satisfy feasible explicit chunk count and maximum-size constraints before - optimizing balance. Preserve the single-segment exception above the token - maximum. Use a proportionate deterministic algorithm and retain the - configured budgets and current balancing objective. Keep accepted - compatibility settings, including `chunking.min_section_tokens`, even where - they are not active partition targets. -- Give validator applicability and decision meaning one owner and one runtime - contract. Reject contradictory internal outcomes before they advance a - candidate; treat them as internal contract failures, not unavailable model - assessments. Derive redundant approval representation where needed. Preserve - decision cardinality, zero-budget shortlisting, independent repair feedback, - and unavailable outcomes. Reconcile exact numeric response references before - local conversion, accepting integral JSON number spellings without float - rounding and ignoring irrelevant extras. Keep whole-response structural - validation and existing partial-success behavior. -- Share glossary-owned protection assessment using usable before/after segment - evidence, and recheck current text at application. Preserve active-glossary - exemptions, case/longest-match behavior, and spoken-word repetition rules. - Missing or ambiguous previews cannot support a fabricated protection verdict; - independently supported applicability findings remain valid. Empty-result - previews can still supply protection evidence without authorizing whole-segment - removal. Handle configured punctuation-edged spellings with standard primitives - while accepting Go's existing word-boundary semantics. -- For require-unique replacement, detect a second possible literal starting - offset even when matches overlap. Preserve replace-all's existing nonoverlapping - behavior. Share request-local segment lookup across initial and repaired - proposals, retaining distinct reference namespaces, raw invalid references, - and retargeted revalidation. -- Reuse glossary identity membership for context selection rather than expanding - every occurrence/contributor pair; keep occurrence evidence for protection. - Avoid validation-context work once the usable budget is settled, sharing - trimming/batching policy while preserving context priority, oversized-input - handling, and an independent serialized measurement reference. Reuse section - lookups and aggregate skipped-edit evidence once, distinguishing stale - application conflicts from response filtering. - -### Completion evidence - -Boundary regressions reproduce the audited numeric, nested-interval, tied-order, -feasible-chunk, overlapping-match, and full-segment-protection cases. Validator -tests pass actual structured responses through decoding and reconciliation, with -usable siblings and irrelevant extra references. Shared optimized operations -are checked against independently constructed expectations, including exact -estimates and representative allocation/work measurements. Existing editorial -acceptance, repair, and apply-time rules continue to govern results. - -## 4. Smaller components with explicit ownership - -### Target state - -Preflight and execution share resolved application decisions. Domain models own -their conversions and copies; adapter boundaries own external effects; the -composition boundary assembles built-ins. Modules and validators remain distinct -components with documented concurrent-call and borrowed-data responsibilities. - -The baseline repeats root/stage configuration rules, readiness selections, -transcript projections/clones, token estimates, and shared prompt-variable work. -Some interfaces and helpers survive only through tests or unused wrappers. - -### Work to reach the target - -- Consolidate Audita-owned root/stage validation and fallback rules after - ConfigKit handoff, retaining null, explicit zero, inheritance, provenance, - stage-dependent defaults, and diagnostic paths. Carry checked processing - selections from preflight into execution. Reuse effective profile/glossary - views within the captured runtime, with selection order and exemptions in - reuse keys; retain first-failure attribution. -- Share transcript cloning at the model boundary, preserving nil versus explicit - empty mutable slices and the separate caller ownership boundaries. Give public - encoders one segment projection while retaining both envelopes, schema/version - behavior, categories, omission rules, and numeric fidelity. Use one implementation - of the existing token estimate across consumers, retaining its exact results. -- Assemble built-in modules at composition, preserving the runner factory seam - and a dependency-light configuration catalog. Document concurrent-call and - borrowed-input obligations at module/validator ports; permit synchronized - stateful implementations and use owned copies where mutation is needed. -- Merge shared prompt variables once per model call, preserving the different - processing/inspection precedence, protected-term default, and operator prompt - replacement support. Remove dead helpers, unused envelopes, redundant adapters, - and abandoned test support after checking live registrations and serialization. - Move useful tests to the surviving behavior before removing test-only paths. - -These changes complement the writer, validator, operation, and repair ownership -work above. Use small contracts and helpers where they serve an actual shared -responsibility; package layout and private type names are implementation choices. - -### Completion evidence - -Real consumers use the shared owners and obsolete paths are removed. Behavioral -tests establish copy independence, both output envelopes, effective configuration -semantics, native integration, and processing/inspection precedence. Refactoring -does not depend on a new configuration loader, plugin framework, cross-invocation -cache, or a generic replacement for the existing domain components. - -## 5. Actionable errors and detailed processing progress - -### Target state - -Operators can identify the failed phase and relevant setup requirement, and -follow long jobs with meaningful progress. Standard inspection and processing -remain integrated with CommandKit rather than acquiring parallel command or -reporting systems. - -The baseline hides some native setup detail, classifies some operator-correctable -failures as runtime failures, and exposes progress without producing processing -events. The current [CLI contract](../cli.md) supplies the stream and exit-status -boundary that the fixes should follow. - -### Work to reach the target - -- Classify invalid processing selections and genuinely infeasible chunk requests - according to the existing input-error contract; retain runtime classification - for I/O and cancellation. Trace delegated inspection failures to Audita's - loader or the upstream integration. The pinned ckprompt catalog handler - classifies malformed prompt configuration as an operational failure. Fix - that classification upstream while keeping native command routing. - Processing progress can use CommandKit's existing reporter independently. -- Surface actionable native setup detail, including the configured missing - credential requirement and available role/profile context, while preserving - typed causes and failure classification. Remove CLI substitution of errors, - warnings, and announced paths. Keep compact ordinary-report projections where - they serve the report schema; deleting substitution is not a request to expand - every report with raw provider causes. -- Emit relatively detailed stage/chunk, proposal, validation, and repair activity - through CommandKit's reporter, using actual lifecycle state and available - upstream execution events. Include useful work identities and observed - completion counts. Do not restrict `--progress` to coarse stage summaries just - to minimize output. CommandKit determines effective reporting-mode precedence - and routes output, including when flags are combined; stdout remains available - for the transcript. Event granularity and wording are implementation choices. -- Document the trusted configuration bundle, imports, and automatic companions - for operators and subprocess integrations. Explain how caller-owned report - destinations and artifact retention fit the workflow. Preserve native discovery - and inspection; these clarifications do not add an isolation mode. - -### Completion evidence - -CLI tests check the appropriate exit classifications, useful diagnostic context, -native mode combinations, and unchanged transcript stdout. Progress tests exercise -real processing activity and meaningful updates without fixing incidental wording -or concurrent event order. Update user-facing documentation alongside the feature. - -## 6. Verification and delivery that support change - -### Target state and work - -Tests protect consequential behavior at its owning boundary and make valid -refactoring and prompt iteration inexpensive. Apply the -[testing policy](../policy/testing.md) throughout the work, rather than adding a -second set of implementation-shaped assertions at the end. - -- Exercise representative signal, cancellation, and exit behavior through the - real executable or a shared production entry point. Keep smaller CLI fixtures - for cases that do not need a process boundary. Use observed overlap and - independently controlled deadlines for concurrency/timeout tests. -- Verify registered identities, effective validator roles, embedded-asset - preparability, schema identity, and protective copies through behavior. Replace - incidental registry counts, duplicate normalization fixtures, and tests of test - helpers with coverage of their actual consumer risks. Assert decoded artifact - identities and association; keep a representative repair/filter/later-failure - integration scenario and smaller independent-output cases. -- Remove redaction-only tests and fixture modes while retaining authentication, - serialization, evidence, and failure coverage. Avoid prompt-wording change - detectors, broad incidental snapshots, and new checks that recreate withdrawn - guarantees. The default suite remains offline and uses synthetic credentials. -- Fix the CI bootstrap so failure to run its Git cleanliness check cannot be - interpreted as a clean checkout. Test that command-status boundary and remove - redundant tests of the release test's comparison helper. Keep ReleaseKit's - selected checks and the existing [release policy](../policy/release.md); this - roadmap introduces no new release gates. -- Update canonical CLI, configuration, operations, integration, and internal - documentation with the corresponding changes. Remove stale redaction promises - and recipes without adding a negative architecture assertion. Keep native - configuration-rendering documentation and historical audit evidence. Use ADRs - only where the [documentation policy](../policy/documentation.md) calls for a - significant architectural record, not for every helper or audit entry. - -### Completion evidence - -Run focused regression and boundary tests with each change, the full offline -suite for substantial integration changes, and applicable existing candidate -checks at release preparation. Use targeted race checks for changed shared-state -paths and representative measurements for resource changes. Reuse successful -checks for unchanged content. Link/example/whitespace checks accompany relevant -documentation changes. Do not substitute coverage percentages or a collection -of test counts for evidence of the selected outcomes. - -## Completion and audit coverage - -The [implementation plan](implementation.md) records the landed changes and -verification. The coverage map accounts for every catalog entry, including -closed and superseded remedies. The [assessment](audit/assessment.md) records -which remedies were selected. - -| Target area or disposition | Catalog entries | -| --- | --- | -| §1: checked output, truthful finalization, caller destinations | 1, 6, 7, 8, 16, 42 | -| §1: serialization, diagnostic ownership, and useful evidence | 3, 9, 22, 26, 27, 33, 43, 63 | -| §2: stage/operation lifecycle, scheduling, and retention | 14, 15, 23, 30, 32, 34, 41, 44, 45, 46, 48, 64 | -| §3: transcript fidelity, normalization, and chunking | 13, 17, 20, 21, 31, 35, 51 | -| §3: decision semantics, protection, references, and context | 10, 11, 12, 18 (punctuation portion), 19, 36, 38, 39, 40, 47 | -| §4: shared selections, models, configuration, and composition | 37, 49, 50, 52, 53, 54, 55, 56 | -| §5: operator guidance, errors, and progress | 5, 24, 25, 57, 65 | -| §6: verification, delivery, and dead machinery | 28, 29, 58, 59, 60, 61, 62 | -| Accepted Go limitations; no workaround | 2; 6 (interrupting blocked I/O); 18 (Unicode word-boundary emulation) | -| Withdrawn private-debug-default remedy; current defaults retained | 4 | - -Cross-cutting test and documentation work belongs with each implementing change. -The [assessment](audit/assessment.md) remains the detailed boundary for mixed -entries. All seven findings in the focused [removal audit](audit/14-redaction.md) -are covered by the evidence, repair, CLI, and verification sections above. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index 1ed72e1..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,523 +0,0 @@ -# Reliability and Maintainability Review Remediation - -**Reviewed revision:** `e8c0708` - -**Status:** All fourteen findings addressed after review of the 20 -[implementation stages](implementation.md) against the -[target roadmap](reliability-and-maintainability.md), including follow-up -audits of component interactions and algorithmic costs. - -## Review Result - -The sprint established the target architecture. ConfigKit and native -CommandKit integrations handle configuration and inspection, and preflight -hands checked processing selections to the runner. Audita decodes and -normalizes transcripts directly and partitions them once. A bounded section -window allows proposal, validation, and repair to overlap within a stage. -The runner assigns final proposal indices in section order and commits edits -at the stage boundary. PromptKit owns model execution, backend admission, -retries, and repair history. Required publication uses checked destinations. -Ordinary reports use typed processing facts, while optional debug capture -retains interactions and decision history. CommandKit renders live progress. - -The offline CLI build and full Go suite passed, as did targeted race checks -for the runner, LLM adapter, validators, report projection, diagnostics, -glossary, and CLI. The ordered repair artifact test also passed 150 repetitions -under the race detector. A built CLI returned status 2 with source diagnostics -for malformed PromptKit and application configuration across the delegated -commands and processing cases in the plan. Focused review probes reproduced -R1, R3, and R4. R2 and R5 follow from the versioned output contract and current -documentation. - -The follow-up audit traced the new paths through CLI dispatch, the built-in -validator chains, model execution, publication, and report and debug readers. -Additional probes reproduced the case-variant input behavior in R4 and the -cross-component failures in R6–R8. - -The algorithmic audit kept the selected stage window, full repair history, -and chunk-partitioning algorithm. The normalization merge builds text and -categories incrementally after sorting, and the explicit partitioner advances -through the weights in linear passes. The remaining opportunities are -primarily in repeated validation and glossary preparation, queued debug -snapshots, and evidence retained after it has served its purpose. R9 was -reproduced through the PromptKit role adapter with a synthetic provider. -R10–R14 follow from source analysis. Focused offline tests passed for -normalization, chunking, preflight, validators, the runner, and reports. - -## Findings - -### R1 — Include Later Delivery Failures in Writable Reports (P2) - -**Complete.** Later report writes include newly observed delivery failures. -One best-effort pass updates earlier reports with stale evidence. Diagnostics -tests cover both initially failed runs and failures that arise during -publication of a completed run. - -The follow-up fix counts repeated write failures for one report as one -delivery failure and retains both causes. The regression makes both writes to -`recoveries.yml` fail and verifies that the surviving reports agree on the count. - -**Affected plan:** Stage 4, item 3, and Stage 17, item 1. - -**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311) -renders a failed bundle using the failures accumulated at that point. Later -writes append to the returned error list, but the remaining reports and the -best-effort pass reuse the same bytes. The -[report projection](../../internal/framework/processreport/failures.go#L41) -can include secondary file failures only if publication supplies them. - -**Reproduced behavior:** Start finalization with a processing failure. Make -`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed. -The returned error contains the processing failure and both delivery failures. -The renderer receives only the processing failure, so the writable report -omits the two already-observed failures. The same stale-bundle problem can -follow a report error that first switches a successful run to failed status. - -**Required change:** Use the freshest observed failures at the report -publication boundary and make a bounded best-effort update of already-written -reports when their failure evidence becomes stale. Retain existing evidence -if rendering the updated failed bundle fails. Keep transcript-first, -nontransactional publication. - -**Acceptance:** Extend the output-session regression to inspect the facts -rendered into a surviving report when an earlier failure is followed by a -second write failure. Check failure attribution and counts as well as status. - -### R2 — Preserve the Version 1 Error Record Discriminator (P2) - -**Complete.** Version 1 retains the `work_unit` and `run` scopes. The -initiating run record carries available peer details in an optional -`interruptions` sequence. The mixed-run YAML and Markdown test verifies -peer attribution and unchanged counts. - -**Affected plan:** Stage 17, item 1. - -**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108) -adds a top-level `scope: interrupted` record to the -[version 1 error document](../integrations/output-files.md#error-and-recovery-records). -The baseline contract described `work_unit` and `run` records. An exhaustive -reader of those variants encounters a third variant under the same version. -The target asks to retain peer attribution and existing counts and schema. -Audita's [release policy](../policy/release.md#versioning-and-compatibility) -also preserves the documented output contract for minor and patch releases. - -**Required change:** Keep the existing top-level variants. Carry available -peer interruption details as optional evidence on an existing record, for -example an optional sequence on the initiating run failure. Keep the primary -origin and peer origins separately identifiable and preserve the work-unit and -run-error counts. Treat a new top-level variant as an explicit contract change -if that representation is preferred. - -**Acceptance:** Exercise a fatal operation and interrupted peer in one run. -Check that a consumer of the existing record variants can still read the -report, and that the peer's available phase, reason, and origin remain present. - -### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3) - -**Complete.** The operation and run report use the same typed reason fallback. -The regression covers a missing reason with a wrapped deadline, retained -operation origin and cause, and distinct parent cancellation. - -**Affected plan:** Stage 12, item 2, and Stage 17, item 1. - -**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42) -uses `internal_failure` when a typed model failure has no reason and retains -its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16) -recognizes a typed failure only when its reason is nonempty. It can then derive -a different reason from a wrapped cause or the processing phase. - -**Reproduced behavior:** A typed fatal failure with an empty reason and a -wrapped `context.DeadlineExceeded` is recorded on the operation as -`fatal` / `internal_failure` and serialized in the run report as -`canceled`. The cause is available, but it does not make this a parent -cancellation. - -**Required change:** Share the typed reason projection and its empty-reason -fallback. Derive a reason from the raw cause only when a typed failure is absent. - -**Acceptance:** Verify matching fallbacks for the operation and the run record -and retain the phase, origin, and original cause. Keep genuine parent -cancellation distinct. - -### R4 — Preserve Effective Transcript Members During Typed Decoding (P3) - -**Complete.** Each segment resolves duplicate keys before field conversion -and processes case variants in the baseline key order. The decoder keeps -numeric tokens intact. Schema tests cover superseded and effective type -mismatches, both case-variant orders, and exact IDs in both envelopes. - -The follow-up restores direct typed decoding for ordinary segment arrays. -Arrays with conflicting or escaped keys use compatibility decoding where -needed. Both envelopes were verified with mixed segments. On the same -100,000-segment fixture, median parse time fell from about 0.83 to 0.30 -seconds and allocation fell from 162 to 78 MB per parse. - -**Affected plan:** Stage 5, item 1. - -**Evidence:** The baseline selected the final value of an exact duplicate JSON -member when it decoded the segment into a generic object, before applying -field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39) -now applies the field type to every occurrence. An earlier type error survives -even when the final value is valid. The refactor was meant to retain accepted -inputs while improving numeric fidelity. - -**Reproduced behavior:** The baseline at `d226bdd` accepts -`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads -speaker `A`. The sprint revision rejects the same input because of the -superseded numeric value. - -The follow-up audit also found a silent change when known member names differ -only by case. For a segment containing `"text":"expected utterance"` followed -by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to -normalization. The new decoder passes `other metadata`. Both accept the input. -The old map round trip ordered the member names before typed decoding. Direct -decoding applies them in input order. - -**Required change:** Preserve the effective-member behavior for existing -inputs without reviving whole-transcript generic decoding or float conversion -for IDs. Keep the compatibility handling focused on this changed boundary. - -**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch -in the effective member, plus the case-variant example in both member orders. -Keep the existing exact-ID and unknown-field tests for both input envelopes. - -### R5 — Align Roadmap Status and Runtime Guidance (P3) - -**Complete.** The roadmap points to the landed implementation and follow-up -review. The pipeline guide describes available peer evidence in reporting mode. -The CLI guide renders the preflight sentence normally. - -**Affected plan:** Stages 15, 19, and 20. - -**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still -describes the implementation as pending despite the completed sprint. -The [pipeline guide](../internal/pipeline.md) says work-unit -evidence preserves each peer's interruption or error. The runner preserves -available outcomes and projects detailed work units according to output -selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a -literal `\n` between “selected” and “profiles.” - -**Required change:** Mark the roadmap with the landed sprint and this review, -while retaining its historical baseline context. Describe interruption detail -in terms of outcomes actually available and selected for retention. Fix the -CLI sentence and keep public guidance focused on the operator-visible behavior. - -**Acceptance:** The roadmap and implementation status agree, the pipeline guide -reflects the evidence modes, and the CLI paragraph renders normally. - -### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2) - -**Complete.** Progress is emitted before and after each timed model call, -including coordinated repair turns. An integration regression delays stderr -beyond the model deadline and verifies the same provider calls and correction. -Live progress, reporting modes, and presentation failure tests also passed. - -The follow-up checks the parent context before each progress notification. -Adapter tests cover calls canceled before progress starts and calls canceled -during the start notification. The CLI progress and cancellation tests passed. - -**Affected plan:** Stages 12 and 19. - -**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30) -calls [progress rendering](../../internal/cli/process_progress.go#L34) -synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139). -PromptKit v0.23.1 starts the operation deadline before emitting the start -event, so a blocked or slow write to CommandKit's diagnostic stream spends -that operation's budget before the provider request starts. - -**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate -loopback provider, the ordinary run dispatched three requests and changed -`hello world` to `Hello world`. With `--progress` and a diagnostic writer that -delayed the proposal-start message by 750 ms, Audita dispatched zero requests, -reported `operation_timeout`, and returned the uncorrected transcript with -status 0 under the existing recoverable-failure policy. The writer eventually -succeeded in delivering the message. - -**Required change:** Deliver progress outside PromptKit's timed model-call -observation path. Keep native observations for execution facts and use -application lifecycle boundaries for synchronous progress delivery, or another -delivery mechanism that keeps stderr waits out of the per-operation budget. -Preserve CommandKit's routing and presentation status, and keep the parent -context authoritative for cancellation of the invocation. - -**Acceptance:** Vary only the delay on the progress stream of a controlled run. -Verify that it still dispatches and applies the same correction within the -model's own budget, and retain live progress and presentation-failure coverage. - -### R7 — Classify Configuration Acquisition Errors Consistently (P2) - -**Complete.** Processing uses the native configuration error classifier and -retains source diagnostics and typed causes. Tests compare processing, -catalog listing, and configuration inspection for malformed content, -permission failure, and caller cancellation. - -**Affected plan:** Stages 1, 11, and 20. - -**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use -configuration diagnostics to distinguish invalid content from operational -errors. The existing [process entry point](../../internal/cli/process.go#L27) -uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then -unconditionally returns a usage outcome for those errors. - -**Reproduced behavior:** A readable main file beside an unreadable automatic -`prompts.yml` produces `source.permission` in all three commands. -`promptkit list prompts` and `config show --scope promptkit` return status 1. -`process` returns status 2 and prints usage guidance for the same loading -failure. - -**Required change:** Use the shared `ckconfig.ClassifyConfigurationError` -policy at the processing configuration boundary. Preserve typed causes and -source diagnostics for both outcome classes. - -**Acceptance:** Compare `process` and representative delegated commands for an -operational acquisition failure and malformed YAML. Check status 1 and status -2 respectively, and preserve cancellation precedence and the diagnostic source. - -### R8 — Use the Same Span Applicability With and Without Repair (P3) - -**Complete.** The original-text validator rejects ambiguous require-unique -spans at either budget. A built CLI regression verifies local attribution, -no protection approval or model validation for the span, and repair only -at the positive budget. Whitespace and blank-result tests also passed. - -**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17. - -**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38) -rejects an ambiguous require-unique span only when independent repair evidence -is enabled. Without repair, the existing shortlist forwards the candidate to -the [protected-term validator](../../internal/framework/validators/deterministic.go#L106), -which records approval even when it cannot obtain a usable preview. The -[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251) -later rejects that same local ambiguity before dispatching a provider call. - -**Reproduced behavior:** In the built-in grammar stage, offer to replace -`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor` -protected. With a zero repair budget, the recorded decisions show protection -approval and `editorial_review` rejection for `ambiguous_original_text`. -With repair enabled, `original_text_presence` records the ambiguity and -protection emits no decision. Neither run applies the ambiguous proposal. - -**Required change:** Give the local applicability check ownership of -require-unique ambiguity in both paths. Keep independent evidence and repair -attempts conditional on the configured budget. Retain the ability to assess -protection when a located edit produces an empty preview. - -**Acceptance:** Use the built-in chain with zero and positive repair budgets. -Verify consistent local attribution, no unsupported protection approval, -and no model validation call for the inapplicable candidate. Keep repair -available in the positive-budget case. - -### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2) - -**Complete.** Proposal and repair references use the same exact integer -decoder as transcript IDs. The CLI conversation test covers a usable sibling -and a repair revision with decimal or exponent references. The repair test -continues to reject a mismatched reference. - -**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9 -already handles the corresponding validator references. - -**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20) -and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9) -specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21) -and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24) -use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184) -classifies a conversion error after schema validation as recoverable invalid -output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123) -then returns an operation failure with no candidates. - -**Reproduced behavior:** A synthetic provider returned two proposals with -references `1` and `2.0`. PromptKit accepted the response in one request with -zero structural repairs. Audita returned `invalid_output` at typed decode. -The control case with `1` and `2` decoded both proposals. The repair decoder -has the same conversion for `proposal_ref` and the revision's `id`. - -**Required change:** Decode wire references as exact numeric tokens and -reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9) -before local position or correspondence checks. Preserve the distinction -between segment and proposal references, existing out-of-range handling, and -the original response in debug capture. - -**Acceptance:** Exercise the actual structured paths with an integer and an -equivalent decimal or exponent form. A valid sibling survives proposal -decoding, and the repair revision reaches the same revalidation as its integer -form. Retain existing handling for a mismatched repair reference. - -### R10 — Fit Required Validation Data Before Collecting Optional Context (P3) - -**Complete.** Fitting checks required data before loading glossary and -neighbor context. Context selection reuses source matches within a validator -request and matches each revision independently. Tests verify skipped loading, -candidate-specific context, and equivalence with the serialized budget oracle. - -**Affected plan:** Stages 8 and 10. - -**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251) -builds optional context for every previewable candidate before -[fitting the payload](../../internal/framework/validators/llm_budget.go#L50). -[Context construction](../../internal/framework/validators/llm_context.go#L39) -rescans the original and adjacent segments, finds matches in the proposed -text, sorts matched entry positions, and builds the entry slice. The fitter -can discard all optional context while retaining the original slice's -backing array on a fitted item. - -**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B` -bytes of shared source and neighboring text, the unchanged text is scanned -`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate -add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work -happens even when required fields consume the budget. Distinct proposed texts -still need their own match evidence when optional context is used. - -**Recommended change:** Fit the required fields before collecting optional -context for a candidate. Reuse identity membership for unchanged text within -the section and combine it with the corrected-text matches only when needed. -Keep existing glossary and neighbor order, exact serialized budget accounting, -and oversized-input behavior. Drop unused optional storage before dispatch. - -**Acceptance:** Compare the emitted payload to the existing serialized-budget -oracle for usable and oversized candidates. A controlled no-room case should -skip optional matching. Sibling candidates should share unchanged-text lookup -while still receiving definitions specific to their revisions. - -### R11 — Reuse Glossary Selections Before Building Their Views (P3) - -**Complete.** Stage resolution shares read-only selected entries and -protection terms before preparing the views. Both use the same ordered -selection key. Preflight and runner tests cover repeated selections, -ordering, glossary-stage exemptions, and stage-specific execution. - -**Affected plan:** Stage 11, item 2. - -**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115) -clones selected glossary entries and aliases and derives protection terms for -every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152) -can reuse prompt variables and validator views only after all those copies -have been built. The retained specs still hold separate entry and term slices -for identical selections. - -**Cost:** With `S` stages sharing a selection of `G` entries and spelling -contributions, materializing and retaining those selection data costs -`Θ(S × G)`. Sharing immutable selection data brings this portion to -`Θ(G + S)` for repeated fixed-length selection keys. - -**Recommended change:** Resolve and prepare the shared glossary data on the -first occurrence of an ordered context/protection selection and exemption. -Reuse the complete read-only selection in later specs. Keep stage identity, -first-failure attribution, and copy-on-mutation at consumer boundaries. - -**Acceptance:** Exercise repeated selections, a reordered selection, and a -glossary-stage exemption. Reused selections should supply the same data to -proposal, validation, and application as the existing independent views. - -### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3) - -**Complete.** Pending repair requests share their immutable history -prefixes. Each write retains the exact request and final proposal index. -An ordered collector test compares two real coordinator snapshots with the -written requests, checks ledger paths, and verifies that a later request -still writes after a capture failure. Repeated race checks passed. - -**Affected plan:** Stages 3, 14, 16, and 18. - -**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151) -converts every turn's full rendered message list to a new slice. Each -[repair builder](../../internal/framework/runner/repair.go#L194) captures that -snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22) -holds the builders until ordered proposal indices are assigned, including -during earlier sections' model work. - -**Cost:** A branch with `H` starting messages and `T` successful turns that -add a constant number of messages per turn can retain `Θ(T × H + T²)` message -descriptors at once. Content strings may share backing storage. The window -bounds the number of sections held, and a single section can still contain -many turns and branches. Full-history requests and self-contained debug files -require repeated serialization by design. They can be serialized one at a -time without keeping every prepared list live. - -**Recommended change:** Keep compact evidence pending ordered identity -assignment. For example, stage the request payload under section-local -coordinates while it is live and use its final name when indices are assigned. -An in-memory approach can share immutable history prefixes and expand each -snapshot at write time if it can reproduce the exact rendered messages. -Retain full model context, nonfatal capture, and self-contained final files. - -**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section. -Compare its decoded request bodies, final artifact names, and ledger paths -with the current representation, including a capture failure followed by a -successful write. - -### R13 — Retain Only Operations Used by Ordinary Reports (P3) - -**Complete.** After totals and application attribution are settled, the -runner keeps only reportable errors and recoveries. Successful proposal -index lists and per-operation causes no longer persist through the run. -Runner, report, and CLI tests cover totals, filtering, interrupted origin, -and evidence across output modes. - -**Affected plan:** Stages 12, 17, and 18. - -**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632) -counts each work unit and retains all its operations when reports are enabled. -It also [stores every candidate index](../../internal/framework/runner/runner.go#L548) -on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79) -uses only recoverable failures and successful recoveries. Failure projection -uses affected indices and typed facts. Recovery projection uses counts and -filtered evidence. Neither uses the cause stored on each operation, which is -needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22). - -**Cost:** The retained operation records, unused successful proposal index -lists, and causes account for `Θ(M + P + B)` storage, for `M` operation -records, `P` successful proposal indices, and `B` bytes of retained causes. -PromptKit's non-success HTTP error can include a bounded -provider diagnostic in each cause. Even an all-success run with no recoveries -keeps and later sorts operation rows that produce no error or recovery entry. -The report matrix itself still needs a row per work unit. - -**Recommended change:** After counting and resolving application filtering and -failure attribution, retain compact facts only for operations that feed the -reports. Keep typed initiating and peer causes where they belong on execution -failures. Debug execution observations and correction-ledger inputs have -their own consumers. Avoid retaining proposal index lists on operations that -never project those lists. - -**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals, -and debug association across existing evidence modes. Include clean successes, -both initial and repair filtering, and an interrupted run with available origin. - -### R14 — Reuse Original Applicability When Attributing Application Skips (P3) - -**Complete.** Application still checks current text. Skip attribution -reuses known original applicability, and repair filtering indexes approved -operations once per affected unit. Tests distinguish stale conflicts from -original defects and verify attribution across multiple repairs. - -**Affected plan:** Stages 8 and 17, item 3. - -**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59) -checks each proposal against evolving segment text. On a defective-looking -skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953) -previews the proposal again against the original section to distinguish a -stale conflict from a defective response. The built-in validation path has -already determined the latter preview's applicability. Its result is discarded -before the attribution pass. - -**Cost:** For `K` conflicting proposals against one original segment of `B` -bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of -original-text previews, even if application quickly rejected those proposals -against the changed text. This is separate from the checks required against -current text. When a validator chain can approve originally inapplicable -revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46) -can also scan `O(F × M)` operations for `F` such filtered revisions and `M` -operations in the same unit. Built-in validators normally reject those revisions -before application. - -**Recommended change:** Carry a compact original-applicability fact for -approved candidates when the validator already computed it. Recompute only -when no such fact is available. If the repair attribution branch needs changes, -index or group originating repair operations once per affected unit. Preserve -ordered application, current-text protection checks, and the distinction -between response filtering and staleness. - -**Acceptance:** Keep both a stale-conflict case with initially valid proposals -and an originally inapplicable approval from a controlled validator. Verify -the skip, correction counts, and filtering or recovery attribution.