# Test coverage and effectiveness audit Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to assess protection against consequential failures, test ownership, fixture fidelity, and maintenance cost. The findings are follow-up options. [Testing policy](../../policy/testing.md) ## Architecture and measured coverage Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit loads application settings, and core packages parse, normalize, and chunk the transcript. The runner coordinates correction proposals, validators, repairs, and edit application. The role adapter uses a shared PromptKit engine, and the CLI publishes the transcript and reports through an output session. Tests use pure domain inputs, temporary files, an injected HTTP transport or loopback provider, and subprocesses. Release tests in this repository inspect the local profile, while ReleaseKit owns the underlying delivery implementation. [Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) · [LLM runtime](../../internal/llm-runtime.md) · [Release procedure](../../release.md#binary-delivery) On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...` run passed with `-coverprofile` and `-json` output directed to temporary files. Go's cover tool reported **89.3%** statement coverage. The JSON run log recorded 441 passing top-level tests and 210 passing named subtests. The package percentages below merge repeated profile rows by source block before counting statements. | Package or boundary | Statement coverage | Audit observation | | --- | ---: | --- | | `cmd/audita` | 0.0% | The test executable handles signals in a helper. | | `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. | | `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. | | `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. | | `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. | | `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. | | `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. | | `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. | | `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. | The percentages describe statements reached in this run. The audit assigned test value by the assertions and failure scenario. In particular, low coverage in test support or thin forwarding wrappers does not justify standalone tests. Existing higher-value checks include partial validation with surviving edits, repair revalidation, typed failure causes, stable user-visible identities, and the presence of a masked credential echo in an actual retained artifact. [Coverage policy](../../policy/testing.md#coverage) · [Partial validation](../../../cmd/audita/main_integration_test.go#L1091) · [Repair integration](../../../cmd/audita/correction_repair_integration_test.go) · [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10) · [Credential echo](../../../cmd/audita/main_integration_test.go#L965) ## Behavior that needs stronger verification ### 1. Subprocess tests recreate the production executable boundary The subprocess suite launches the test executable. Its helper registers SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with the returned code. The production `run` and `main` contain their own copy of that wiring and recorded zero hits. A regression in production-only signal registration, argument slicing, or exit propagation can pass the subprocess suite, including the cancellation case. The existing tests still exercise real process boundaries and the assembled CLI. [Production entry](../../../cmd/audita/main.go#L12) · [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28) · [Signal test](../../../cmd/audita/main_integration_test.go#L1059) Use the actual built executable for one representative startup and signal or exit test, or have the helper invoke the same production startup function. Keep the smaller existing seams for the rest of the CLI suite. ### 2. Inspection failure tests accept the wrong exit classification `TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and `TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero. Both can pass if the integration maps invalid input and cancellation to the same nonzero status. A temporary build given the same malformed operator prompt file returned `1` via `promptkit list prompts` and `2` via `process`. The CLI contract distinguishes input failure (`2`) from runtime failure (`1`). The input test also sets a synthetic credential that its malformed templates, profiles, and paths never reference, so the absence assertion provides no evidence that a configured secret was found and redacted. [Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24) · [CLI exit contract](../../cli.md#inspection-and-outcomes) · [CLI wiring](../../../internal/cli/app_wiring.go#L80) Assert the selected status and a stable diagnostic fragment or typed cause. Clarify the intended delegated-command classification if it differs from the CLI contract. Test redaction where a known value reaches the relevant channel, as the existing credential echo test does, and remove the unused secret assertion from these cases. [Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965) ### 3. Output tests primarily check the final artifact state The short-write CLI test asserts final YAML status, counts, and a reason code. The replacement test checks the result after a failed-report pass. Both can pass when finalization briefly writes `completed` reports after it already knows transcript delivery failed. The short-write test also passes if the required run-error phase is missing. Collision tests cover aliases present during selection, while the finalization path resolves report destinations again. Existing tests that change the transcript's file type exercise a different path. Prior audits documented both the transient report state and the changed report referent. [CLI short-write test](../../../internal/cli/process_output_test.go#L79) · [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207) · [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14) · [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435) · [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed) · [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) At `OutputSession`, observe the first report state after a known delivery failure and verify that a changed report destination cannot defeat the checked identity. At the CLI boundary, assert the exit and serialized run phase. The failure-injection seam is useful because it observes publication state at an externally consequential boundary. Preserve the documented nontransactional behavior when a report error occurs after a transcript has been published. [Output lifecycle](../../operations.md#outputs-and-lifecycle) · [Run error fields](../../integrations/output-files.md#error-and-recovery-records) The stream tests open a FIFO peer before final delivery and use a payload that fits in the pipe buffer. The cancellation tests exercise a caller canceled before delivery or while the provider is active. A supported stream blocked in delivery therefore falls outside their assertions. Verify bounded subprocess cancellation and failed-status finalization after the supported stream semantics are decided. Keep the existing small delivery tests for the successful stream contract. [FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13) · [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131) · [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring the transcript staging file or initializing the reports, and no hits in `Close` after a temporary path removal error. The existing cleanup tests cover a write or rename failure followed by successful cleanup. A well-placed preflight test can observe output and report state when cancellation follows initialization. At the output-session boundary, verify which cleanup errors are reported and whether a temporary removal failure can be retried under the selected contract. [Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61) · [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394) · [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) ### 4. The timeout tests cover two of the three relevant deadline owners The adapter test distinguishes a shorter Audita operation budget from an already canceled parent. Other adapter cases check provider statuses and transport failures. They do not exercise a profile's HTTP attempt deadline expiring under a still-live parent and a longer operation budget. The earlier CLI audit observed that case labeled `operation_timeout`, even though PromptKit reserves its operation sentinel for the logical budget. [Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406) · [Provider status test](../../../internal/framework/llm/adapter_test.go#L462) · [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline) Put the deadline distinction at the adapter boundary using a controlled transport and deliberately separated deadlines. Assert both recoverability and the selected cause or reason. The CLI already has report mapping tests and only needs representative coverage of the adapter-to-report connection. [Timeout contract](../../integrations/openai-compatible-llm.md) ### 5. The validator batch fixture misses the real numeric decoding boundary The subprocess mixed-response test and the validator reconciliation tests protect sibling decisions when ordinary integer references are missing, duplicated, or unexpected. The adapter has a separate overflow test with a single out-of-range reference and no usable sibling. The gap is a response with both a usable requested decision and an unexpected integer too large for the Go index type. Native schema validation can accept that integer, but the typed adapter decode can reject the whole batch before per-entry reconciliation. The existing overflow test provides evidence for failure classification and cannot establish sibling preservation. [Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567) · [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429) · [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions) Exercise a mixed response through the actual provider envelope, schema, and adapter decode. Assert the usable sibling's decision and the appropriate disposition for the extra entry. A stub returning a pretyped integer cannot cover this case. Keep smaller reconciliation tests for the ordinary identity rules. [Validator reconciliation contract](../../internal/validators.md#llm-validator-batching) ### 6. Concurrency fixtures leave some schedules unproved The runner's proposal-order test waits for all proposals to start, then releases three gates back-to-back in reverse section order. That establishes a release order, while workers may deliver their results in a different order. The test checks applied-change order by target segment and final text, without asserting the proposal indices promised by its name. Order by segment can survive a change to proposal index assignment because corrections are applied within their sections. The runner has other fixed-chunk and application-order checks, so the missing evidence is deterministic indexing under a demonstrably different completion order. [Parallel runner test](../../../internal/framework/runner/runner_test.go#L423) · [Result assertions](../../../internal/framework/runner/runner_test.go#L473) · [Application and recombination](../../../internal/framework/runner/runner.go#L300) · [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants) The shared-backend test holds the first HTTP call, then gives the second call a 30 ms deadline. It does not verify that the second call reached admission or the wait before it timed out. On a delayed execution, an earlier timeout can satisfy the same assertions. The runner cancellation test observes native `ExecutionStarted` for both peers and one dispatch. That event precedes admission. Twenty race-enabled repetitions of the selected capacity, cancellation, and proposal-order tests passed. The repetitions exercise the present fixtures, while the state-observation gaps remain. [Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42) · [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67) · [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96) Hold the earlier result until later completion is observable and assert the actual assigned indices and section identities. At the adapter boundary, prove that work reached the contested state before canceling it. Observe peak active dispatches relative to a test-controlled limit and check that waiting work can proceed or time out according to the configured contract. Keep wall-clock deadlines as bounds for the test itself where possible. [Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy) ### 7. Representative transcript fixtures leave data-integrity boundaries open | Behavior | Current protection and gap | Follow-up | | --- | --- | --- | | Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) | | A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) | | An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) | ### 8. Debug redaction assertions mostly inspect the outer serialization The subprocess test confirms that it actually captured a provider echo, then checks command streams and retained files for the clear credential value. The redactor unit test has a useful nested `final_content` case built with Go's own JSON encoder. Neither asserts the decoded content of that nested field when the provider uses another legal JSON representation. A literal-byte absence check can pass while the credential survives there. This matters to the documented diagnostic redaction boundary, even when a private debug directory limits access on disk. [Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965) · [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43) · [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) Use a synthetic credential and verify the semantic content after parsing the retained artifact and any structured response field it contains. Own detailed string representation cases at the diagnostic writer. Keep one real provider test to prove that credential discovery and artifact redaction connect. [Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope) ## Candidates to remove or consolidate ### 9. Delete proposal-order assertions on the test-only composition harness `TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness` twice and checks indices that the harness itself assigns with `nextIndex++`. It cannot detect an indexing regression in `Runner.Run`. The harness also passes the full transcript into a fake module and delegates range filtering to that fake. Production builds section-local requests. Related harness cases verify a composition that the test constructs itself. Remove the index test once the real runner test checks indices under controlled completion. Migrate any remaining distinct input-preservation or stale-application assertion to the existing runner or proposal API coverage before removing its harness case. [Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14) · [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143) · [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356) · [Stable application test](../../../internal/framework/proposals/apply_test.go#L36) ### 10. Consolidate validator registry and asset metadata assertions `TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the class comparison in `TestRegistryBuildsClassifiedValidators`. The second test also requires the concrete validator to implement `ClassifiedValidator` even though `ClassOf` supports a stable-key fallback. A behavior-preserving change to that representation would fail the type assertion. Keep one table that builds registered keys and checks effective classes against documented roles. Preserve protective checks of concrete domain behavior and stable selector keys. [Duplicate registry checks](../../../internal/validators/registry_test.go#L80) · [Explicit class examples](../../../internal/validators/registry_test.go#L48) · [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54) `TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already enforced at registry construction. The native preparation tests give stronger evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions` checks an exact count of 11 prompts. That count detects additions while allowing a same-size substitution. Prefer a check for each required prompt ID and its preparability. Retain the tests for canonical schema asset agreement, stable diagnostics data, and protective copy behavior. [Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71) · [Production construction](../../../internal/framework/responseschema/registry.go#L129) · [Prompt metadata count](../../../internal/prompts/registry_test.go#L79) · [Native preparation](../../../internal/prompts/registry_test.go#L13) ### 11. Remove the tests of the release test's own set helper `TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined in the same test file. It cannot detect a malformed release profile or bootstrap behavior. A standard comparison over the decoded profile can retain the documented target set, product and destination checks, and the credential exclusion without a test of the comparison's implementation. If adding release coverage, give it to the actual bootstrap and its exit handling. The known Git-status failure is a representative offline shell boundary, and ReleaseKit already checks the prepared checkout on its own boundary. [Test of the helper](../../../scripts/release_test.go#L63) · [Real profile assertions](../../../scripts/release_test.go#L10) · [Delivery contract](../../release.md#binary-delivery) · [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) ### 12. Consolidate normalization fixtures at the domain and parser owners `TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers` exercise the same rejection with different gap values that remain below the configured maximum. One case can assert the unchanged transcript and the meaningful merge counts together. Seven normalization golden tests also repeat the conversion from `SourceTranscript` to `Transcript` in test code. That conversion mirrors the CLI and cannot catch a regression in the CLI's copy. The fixtures compare decoded values and contain useful domain assertions, including exact text and categories. Keep the distinct domain cases, remove duplicate fixtures after comparing assertions, and let the parser and one assembled CLI case own source shapes and wiring. No automatic golden update was found in these tests. [Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11) · [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101) · [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329) · [Production conversion](../../../internal/cli/run.go#L12) ### 13. Narrow integration assertions to their distinct contracts `TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the same decoded ID with different escaping would fail the test. Decode the artifacts and assert stage, role, and evidence association. Keep end-to-end checks for per-entry profiles, successive transcript state, and output selection, while the reporting package owns Markdown layout and escaping. [Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145) · [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101) `TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure` uses the same model script and also checks the distinct attribution in both YAML documents. The selection matrix took about 6.9 seconds in the instrumented run. Keep one detailed scenario with the attribution assertions and use a smaller fixture for any duplicate combination once it still proves that reporting and debug selections leave decisions and counts intact. The shared fixture also writes a legacy glossary file and stores its path without using it. Remove that unused fixture input separately from behavioral checks. [Selection matrix](../../../cmd/audita/main_integration_test.go#L879) · [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013) · [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283) ## Verification scope The full race-and-coverage run, 20 repetitions of the selected concurrency tests, and the offline CLI classification probe completed successfully. The audit used source inspection to identify false passes and refactor-sensitive assertions. The full suite uses synthetic model responses, consistent with the offline policy. Live-model editorial quality is a separate, deferred evaluation. [Execution policy](../../policy/testing.md#test-execution-requirements) · [Deferred evaluation](../future.md#live-model-editorial-evaluation)