diff --git a/docs/roadmap/audit/08-testing.md b/docs/roadmap/audit/08-testing.md new file mode 100644 index 0000000..3426693 --- /dev/null +++ b/docs/roadmap/audit/08-testing.md @@ -0,0 +1,355 @@ +# Test coverage and effectiveness audit + +Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to +assess protection against consequential failures, test ownership, fixture +fidelity, and maintenance cost. The findings are follow-up options. +[Testing policy](../../policy/testing.md) + +## Architecture and measured coverage + +Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit +loads application settings, and core packages parse, normalize, and chunk the +transcript. The runner coordinates correction proposals, validators, repairs, +and edit application. The role adapter uses a shared PromptKit engine, and the +CLI publishes the transcript and reports through an output session. Tests use +pure domain inputs, temporary files, an injected HTTP transport or loopback +provider, and subprocesses. Release tests in this repository inspect the local +profile, while ReleaseKit owns the underlying delivery implementation. +[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) +· [LLM runtime](../../internal/llm-runtime.md) +· [Release procedure](../../release.md#binary-delivery) + +On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...` +run passed with `-coverprofile` and `-json` output directed to temporary files. +Go's cover tool reported **89.3%** +statement coverage. The JSON run log recorded 441 passing top-level tests and +210 passing named subtests. The package percentages below merge repeated +profile rows by source block before counting statements. + +| Package or boundary | Statement coverage | Audit observation | +| --- | ---: | --- | +| `cmd/audita` | 0.0% | The test executable handles signals in a helper. | +| `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. | +| `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. | +| `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. | +| `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. | +| `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. | +| `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. | +| `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. | +| `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. | + +The percentages describe statements reached in this run. The audit assigned +test value by the assertions and failure scenario. In particular, low coverage +in test support or thin forwarding wrappers does not justify standalone tests. +Existing higher-value checks include partial validation with surviving edits, +repair revalidation, typed failure causes, stable user-visible identities, and +the presence of a masked credential echo in an actual retained artifact. +[Coverage policy](../../policy/testing.md#coverage) +· [Partial validation](../../../cmd/audita/main_integration_test.go#L1091) +· [Repair integration](../../../cmd/audita/correction_repair_integration_test.go) +· [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10) +· [Credential echo](../../../cmd/audita/main_integration_test.go#L965) + +## Behavior that needs stronger verification + +### 1. Subprocess tests recreate the production executable boundary + +The subprocess suite launches the test executable. Its helper registers +SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with +the returned code. The production `run` and `main` contain their own copy of +that wiring and recorded zero hits. A regression in production-only signal +registration, argument slicing, or exit propagation can pass the subprocess +suite, including the cancellation case. The existing tests still exercise +real process boundaries and the assembled CLI. +[Production entry](../../../cmd/audita/main.go#L12) +· [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28) +· [Signal test](../../../cmd/audita/main_integration_test.go#L1059) + +Use the actual built executable for one representative startup and signal or +exit test, or have the helper invoke the same production startup function. +Keep the smaller existing seams for the rest of the CLI suite. + +### 2. Inspection failure tests accept the wrong exit classification + +`TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and +`TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero. +Both can pass if the integration maps invalid input and cancellation to the +same nonzero status. A temporary build given the same malformed operator +prompt file returned `1` via `promptkit list prompts` and `2` via `process`. +The CLI contract distinguishes input failure (`2`) from runtime failure (`1`). +The input test also sets a synthetic credential that its malformed templates, +profiles, and paths never reference, so the absence assertion provides no +evidence that a configured secret was found and redacted. +[Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24) +· [CLI exit contract](../../cli.md#inspection-and-outcomes) +· [CLI wiring](../../../internal/cli/app_wiring.go#L80) + +Assert the selected status and a stable diagnostic fragment or typed cause. +Clarify the intended delegated-command classification if it differs from the +CLI contract. Test redaction where a known value reaches the relevant channel, +as the existing credential echo test does, and remove the unused secret +assertion from these cases. +[Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965) + +### 3. Output tests primarily check the final artifact state + +The short-write CLI test asserts final YAML status, counts, and a reason code. +The replacement test checks the result after a failed-report pass. Both can +pass when finalization briefly writes `completed` reports after it already +knows transcript delivery failed. The short-write test also passes if the +required run-error phase is missing. Collision tests cover aliases present +during selection, while the finalization path resolves report destinations +again. Existing tests that change the transcript's file type exercise a +different path. Prior audits documented both the transient report state and +the changed report referent. +[CLI short-write test](../../../internal/cli/process_output_test.go#L79) +· [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207) +· [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14) +· [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435) +· [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed) +· [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization) + +At `OutputSession`, observe the first report state after a known delivery +failure and verify that a changed report destination cannot defeat the checked +identity. At the CLI boundary, assert the exit and serialized run phase. The +failure-injection seam is useful because it observes publication state at an +externally consequential boundary. Preserve the documented nontransactional +behavior when a report error occurs after a transcript has been published. +[Output lifecycle](../../operations.md#outputs-and-lifecycle) +· [Run error fields](../../integrations/output-files.md#error-and-recovery-records) + +The stream tests open a FIFO peer before final delivery and use a payload +that fits in the pipe buffer. The cancellation tests exercise a caller +canceled before delivery or while the provider is active. A supported stream +blocked in delivery therefore falls outside their assertions. Verify bounded +subprocess cancellation and failed-status finalization after the supported +stream semantics are decided. Keep the existing small delivery tests for the +successful stream contract. +[FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13) +· [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131) +· [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success) + +Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring +the transcript staging file or initializing the reports, and no hits in `Close` +after a temporary path removal error. The existing cleanup tests cover a write +or rename failure followed by successful cleanup. A well-placed preflight test +can observe output and report state when cancellation follows initialization. +At the output-session boundary, verify which cleanup errors are reported and +whether a temporary removal failure can be retried under the selected contract. +[Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61) +· [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394) +· [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths) + +### 4. The timeout tests cover two of the three relevant deadline owners + +The adapter test distinguishes a shorter Audita operation budget from an +already canceled parent. Other adapter cases check provider statuses and +transport failures. They do not exercise a profile's HTTP attempt deadline +expiring under a still-live parent and a longer operation budget. The earlier +CLI audit observed that case labeled `operation_timeout`, even though +PromptKit reserves its operation sentinel for the logical budget. +[Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406) +· [Provider status test](../../../internal/framework/llm/adapter_test.go#L462) +· [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline) + +Put the deadline distinction at the adapter boundary using a controlled +transport and deliberately separated deadlines. Assert both recoverability +and the selected cause or reason. The CLI already has report mapping tests +and only needs representative coverage of the adapter-to-report connection. +[Timeout contract](../../integrations/openai-compatible-llm.md) + +### 5. The validator batch fixture misses the real numeric decoding boundary + +The subprocess mixed-response test and the validator reconciliation tests +protect sibling decisions when ordinary integer references are missing, +duplicated, or unexpected. The adapter has a separate overflow test with a +single out-of-range reference and no usable sibling. The gap is a response +with both a usable requested decision and an unexpected integer too large for +the Go index type. Native schema validation can accept that integer, but the +typed adapter decode can reject the whole batch before per-entry reconciliation. +The existing overflow test provides evidence for failure classification and +cannot establish sibling preservation. +[Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567) +· [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429) +· [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions) + +Exercise a mixed response through the actual provider envelope, schema, and +adapter decode. Assert the usable sibling's decision and the appropriate +disposition for the extra entry. A stub returning a pretyped integer cannot +cover this case. Keep smaller reconciliation tests for the ordinary identity +rules. +[Validator reconciliation contract](../../internal/validators.md#llm-validator-batching) + +### 6. Concurrency fixtures leave some schedules unproved + +The runner's proposal-order test waits for all proposals to start, then +releases three gates back-to-back in reverse section order. That establishes a +release order, while workers may deliver their results in a different order. +The test checks applied-change order by target segment and final text, without +asserting the proposal indices promised by its name. Order by segment can +survive a change to proposal index assignment because corrections are applied +within their sections. The runner has other fixed-chunk and application-order +checks, so the missing evidence is deterministic indexing under a demonstrably +different completion order. +[Parallel runner test](../../../internal/framework/runner/runner_test.go#L423) +· [Result assertions](../../../internal/framework/runner/runner_test.go#L473) +· [Application and recombination](../../../internal/framework/runner/runner.go#L300) +· [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants) + +The shared-backend test holds the first HTTP call, then gives the second call +a 30 ms deadline. It does not verify that the second call reached admission +or the wait before it timed out. On a delayed execution, an earlier timeout +can satisfy the same assertions. The runner cancellation test observes native +`ExecutionStarted` for both peers and one dispatch. That event precedes +admission. Twenty race-enabled repetitions of the selected capacity, +cancellation, and proposal-order tests passed. The repetitions exercise the +present fixtures, while the state-observation gaps remain. +[Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42) +· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67) +· [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96) + +Hold the earlier result until later completion is observable and assert the +actual assigned indices and section identities. At the adapter boundary, prove +that work reached the contested state before canceling it. Observe peak active +dispatches relative to a test-controlled limit and check that waiting work can +proceed or time out according to the configured contract. Keep wall-clock +deadlines as bounds for the test itself where possible. +[Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy) + +### 7. Representative transcript fixtures leave data-integrity boundaries open + +| Behavior | Current protection and gap | Follow-up | +| --- | --- | --- | +| Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) | +| A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) | +| An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) | + +### 8. Debug redaction assertions mostly inspect the outer serialization + +The subprocess test confirms that it actually captured a provider echo, then +checks command streams and retained files for the clear credential value. The +redactor unit test has a useful nested `final_content` case built with Go's +own JSON encoder. Neither asserts the decoded content of that nested field +when the provider uses another legal JSON representation. A literal-byte +absence check can pass while the credential survives there. This matters to +the documented diagnostic redaction boundary, even when a private debug +directory limits access on disk. +[Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965) +· [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43) +· [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) + +Use a synthetic credential and verify the semantic content after parsing the +retained artifact and any structured response field it contains. Own detailed +string representation cases at the diagnostic writer. Keep one real provider +test to prove that credential discovery and artifact redaction connect. +[Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope) + +## Candidates to remove or consolidate + +### 9. Delete proposal-order assertions on the test-only composition harness + +`TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness` +twice and checks indices that the harness itself assigns with `nextIndex++`. +It cannot detect an indexing regression in `Runner.Run`. The harness also +passes the full transcript into a fake module and delegates range filtering +to that fake. Production builds section-local requests. Related harness cases +verify a composition that the test constructs itself. Remove the index test +once the real runner test checks indices under controlled completion. Migrate +any remaining distinct input-preservation or stale-application assertion to +the existing runner or proposal API coverage before removing its harness case. +[Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14) +· [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143) +· [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356) +· [Stable application test](../../../internal/framework/proposals/apply_test.go#L36) + +### 10. Consolidate validator registry and asset metadata assertions + +`TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the +class comparison in `TestRegistryBuildsClassifiedValidators`. The second test +also requires the concrete validator to implement `ClassifiedValidator` even +though `ClassOf` supports a stable-key fallback. A behavior-preserving change +to that representation would fail the type assertion. Keep one table that +builds registered keys and checks effective classes against documented roles. +Preserve protective checks of concrete domain behavior and stable selector keys. +[Duplicate registry checks](../../../internal/validators/registry_test.go#L80) +· [Explicit class examples](../../../internal/validators/registry_test.go#L48) +· [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54) + +`TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already +enforced at registry construction. The native preparation tests give stronger +evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions` +checks an exact count of 11 prompts. That count detects additions while +allowing a same-size substitution. Prefer a check for each required prompt ID +and its preparability. Retain the tests for canonical schema asset agreement, +stable diagnostics data, and protective copy behavior. +[Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71) +· [Production construction](../../../internal/framework/responseschema/registry.go#L129) +· [Prompt metadata count](../../../internal/prompts/registry_test.go#L79) +· [Native preparation](../../../internal/prompts/registry_test.go#L13) + +### 11. Remove the tests of the release test's own set helper + +`TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined +in the same test file. It cannot detect a malformed release profile or +bootstrap behavior. A standard comparison over the decoded profile can retain +the documented target set, product and destination checks, and the credential +exclusion without a test of the comparison's implementation. If adding release +coverage, give it to the actual bootstrap and its exit handling. The known +Git-status failure is a representative offline shell boundary, and +ReleaseKit already checks the prepared checkout on its own boundary. +[Test of the helper](../../../scripts/release_test.go#L63) +· [Real profile assertions](../../../scripts/release_test.go#L10) +· [Delivery contract](../../release.md#binary-delivery) +· [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty) + +### 12. Consolidate normalization fixtures at the domain and parser owners + +`TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers` +exercise the same rejection with different gap values that remain below the +configured maximum. One case can assert the unchanged transcript and the +meaningful merge counts together. Seven normalization golden tests also +repeat the conversion from `SourceTranscript` to `Transcript` in test code. +That conversion mirrors the CLI and cannot catch a regression in the CLI's +copy. The fixtures compare decoded values and contain useful domain assertions, +including exact text and categories. Keep the distinct domain cases, remove +duplicate fixtures after comparing assertions, and let the parser and one +assembled CLI case own source shapes and wiring. No automatic golden update +was found in these tests. +[Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11) +· [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101) +· [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329) +· [Production conversion](../../../internal/cli/run.go#L12) + +### 13. Narrow integration assertions to their distinct contracts + +`TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling +of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the +same decoded ID with different escaping would fail the test. Decode the +artifacts and assert stage, role, and evidence association. Keep end-to-end +checks for per-entry profiles, successive transcript state, and output +selection, while the reporting package owns Markdown layout and escaping. +[Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145) +· [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101) + +`TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure +scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure` +uses the same model script and also checks the distinct attribution in both +YAML documents. The selection matrix took about 6.9 seconds in the +instrumented run. Keep one detailed scenario with the attribution assertions +and use a smaller fixture for any duplicate combination once it still proves +that reporting and debug selections leave decisions and counts intact. The +shared fixture also writes a legacy glossary file and stores its path without +using it. Remove that unused fixture input separately from behavioral checks. +[Selection matrix](../../../cmd/audita/main_integration_test.go#L879) +· [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013) +· [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283) + +## Verification scope + +The full race-and-coverage run, 20 repetitions of the selected concurrency +tests, and the offline CLI classification probe completed successfully. The +audit used source inspection to identify false passes and refactor-sensitive +assertions. The full suite uses synthetic model responses, consistent with the +offline policy. Live-model editorial quality is a separate, deferred evaluation. +[Execution policy](../../policy/testing.md#test-execution-requirements) +· [Deferred evaluation](../future.md#live-model-editorial-evaluation)