Record test coverage and effectiveness audit findings

This commit is contained in:
2026-11-15 14:40:45 +00:00
parent 416cdea625
commit 7d064a6d5a

View File

@@ -0,0 +1,355 @@
# Test coverage and effectiveness audit
Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to
assess protection against consequential failures, test ownership, fixture
fidelity, and maintenance cost. The findings are follow-up options.
[Testing policy](../../policy/testing.md)
## Architecture and measured coverage
Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit
loads application settings, and core packages parse, normalize, and chunk the
transcript. The runner coordinates correction proposals, validators, repairs,
and edit application. The role adapter uses a shared PromptKit engine, and the
CLI publishes the transcript and reports through an output session. Tests use
pure domain inputs, temporary files, an injected HTTP transport or loopback
provider, and subprocesses. Release tests in this repository inspect the local
profile, while ReleaseKit owns the underlying delivery implementation.
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
· [LLM runtime](../../internal/llm-runtime.md)
· [Release procedure](../../release.md#binary-delivery)
On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...`
run passed with `-coverprofile` and `-json` output directed to temporary files.
Go's cover tool reported **89.3%**
statement coverage. The JSON run log recorded 441 passing top-level tests and
210 passing named subtests. The package percentages below merge repeated
profile rows by source block before counting statements.
| Package or boundary | Statement coverage | Audit observation |
| --- | ---: | --- |
| `cmd/audita` | 0.0% | The test executable handles signals in a helper. |
| `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. |
| `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. |
| `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. |
| `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. |
| `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. |
| `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. |
| `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. |
| `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. |
The percentages describe statements reached in this run. The audit assigned
test value by the assertions and failure scenario. In particular, low coverage
in test support or thin forwarding wrappers does not justify standalone tests.
Existing higher-value checks include partial validation with surviving edits,
repair revalidation, typed failure causes, stable user-visible identities, and
the presence of a masked credential echo in an actual retained artifact.
[Coverage policy](../../policy/testing.md#coverage)
· [Partial validation](../../../cmd/audita/main_integration_test.go#L1091)
· [Repair integration](../../../cmd/audita/correction_repair_integration_test.go)
· [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10)
· [Credential echo](../../../cmd/audita/main_integration_test.go#L965)
## Behavior that needs stronger verification
### 1. Subprocess tests recreate the production executable boundary
The subprocess suite launches the test executable. Its helper registers
SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with
the returned code. The production `run` and `main` contain their own copy of
that wiring and recorded zero hits. A regression in production-only signal
registration, argument slicing, or exit propagation can pass the subprocess
suite, including the cancellation case. The existing tests still exercise
real process boundaries and the assembled CLI.
[Production entry](../../../cmd/audita/main.go#L12)
· [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28)
· [Signal test](../../../cmd/audita/main_integration_test.go#L1059)
Use the actual built executable for one representative startup and signal or
exit test, or have the helper invoke the same production startup function.
Keep the smaller existing seams for the rest of the CLI suite.
### 2. Inspection failure tests accept the wrong exit classification
`TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and
`TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero.
Both can pass if the integration maps invalid input and cancellation to the
same nonzero status. A temporary build given the same malformed operator
prompt file returned `1` via `promptkit list prompts` and `2` via `process`.
The CLI contract distinguishes input failure (`2`) from runtime failure (`1`).
The input test also sets a synthetic credential that its malformed templates,
profiles, and paths never reference, so the absence assertion provides no
evidence that a configured secret was found and redacted.
[Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24)
· [CLI exit contract](../../cli.md#inspection-and-outcomes)
· [CLI wiring](../../../internal/cli/app_wiring.go#L80)
Assert the selected status and a stable diagnostic fragment or typed cause.
Clarify the intended delegated-command classification if it differs from the
CLI contract. Test redaction where a known value reaches the relevant channel,
as the existing credential echo test does, and remove the unused secret
assertion from these cases.
[Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965)
### 3. Output tests primarily check the final artifact state
The short-write CLI test asserts final YAML status, counts, and a reason code.
The replacement test checks the result after a failed-report pass. Both can
pass when finalization briefly writes `completed` reports after it already
knows transcript delivery failed. The short-write test also passes if the
required run-error phase is missing. Collision tests cover aliases present
during selection, while the finalization path resolves report destinations
again. Existing tests that change the transcript's file type exercise a
different path. Prior audits documented both the transient report state and
the changed report referent.
[CLI short-write test](../../../internal/cli/process_output_test.go#L79)
· [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207)
· [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14)
· [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435)
· [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed)
· [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization)
At `OutputSession`, observe the first report state after a known delivery
failure and verify that a changed report destination cannot defeat the checked
identity. At the CLI boundary, assert the exit and serialized run phase. The
failure-injection seam is useful because it observes publication state at an
externally consequential boundary. Preserve the documented nontransactional
behavior when a report error occurs after a transcript has been published.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Run error fields](../../integrations/output-files.md#error-and-recovery-records)
The stream tests open a FIFO peer before final delivery and use a payload
that fits in the pipe buffer. The cancellation tests exercise a caller
canceled before delivery or while the provider is active. A supported stream
blocked in delivery therefore falls outside their assertions. Verify bounded
subprocess cancellation and failed-status finalization after the supported
stream semantics are decided. Keep the existing small delivery tests for the
successful stream contract.
[FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13)
· [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131)
· [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring
the transcript staging file or initializing the reports, and no hits in `Close`
after a temporary path removal error. The existing cleanup tests cover a write
or rename failure followed by successful cleanup. A well-placed preflight test
can observe output and report state when cancellation follows initialization.
At the output-session boundary, verify which cleanup errors are reported and
whether a temporary removal failure can be retried under the selected contract.
[Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61)
· [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394)
· [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
### 4. The timeout tests cover two of the three relevant deadline owners
The adapter test distinguishes a shorter Audita operation budget from an
already canceled parent. Other adapter cases check provider statuses and
transport failures. They do not exercise a profile's HTTP attempt deadline
expiring under a still-live parent and a longer operation budget. The earlier
CLI audit observed that case labeled `operation_timeout`, even though
PromptKit reserves its operation sentinel for the logical budget.
[Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
· [Provider status test](../../../internal/framework/llm/adapter_test.go#L462)
· [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline)
Put the deadline distinction at the adapter boundary using a controlled
transport and deliberately separated deadlines. Assert both recoverability
and the selected cause or reason. The CLI already has report mapping tests
and only needs representative coverage of the adapter-to-report connection.
[Timeout contract](../../integrations/openai-compatible-llm.md)
### 5. The validator batch fixture misses the real numeric decoding boundary
The subprocess mixed-response test and the validator reconciliation tests
protect sibling decisions when ordinary integer references are missing,
duplicated, or unexpected. The adapter has a separate overflow test with a
single out-of-range reference and no usable sibling. The gap is a response
with both a usable requested decision and an unexpected integer too large for
the Go index type. Native schema validation can accept that integer, but the
typed adapter decode can reject the whole batch before per-entry reconciliation.
The existing overflow test provides evidence for failure classification and
cannot establish sibling preservation.
[Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567)
· [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429)
· [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions)
Exercise a mixed response through the actual provider envelope, schema, and
adapter decode. Assert the usable sibling's decision and the appropriate
disposition for the extra entry. A stub returning a pretyped integer cannot
cover this case. Keep smaller reconciliation tests for the ordinary identity
rules.
[Validator reconciliation contract](../../internal/validators.md#llm-validator-batching)
### 6. Concurrency fixtures leave some schedules unproved
The runner's proposal-order test waits for all proposals to start, then
releases three gates back-to-back in reverse section order. That establishes a
release order, while workers may deliver their results in a different order.
The test checks applied-change order by target segment and final text, without
asserting the proposal indices promised by its name. Order by segment can
survive a change to proposal index assignment because corrections are applied
within their sections. The runner has other fixed-chunk and application-order
checks, so the missing evidence is deterministic indexing under a demonstrably
different completion order.
[Parallel runner test](../../../internal/framework/runner/runner_test.go#L423)
· [Result assertions](../../../internal/framework/runner/runner_test.go#L473)
· [Application and recombination](../../../internal/framework/runner/runner.go#L300)
· [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants)
The shared-backend test holds the first HTTP call, then gives the second call
a 30 ms deadline. It does not verify that the second call reached admission
or the wait before it timed out. On a delayed execution, an earlier timeout
can satisfy the same assertions. The runner cancellation test observes native
`ExecutionStarted` for both peers and one dispatch. That event precedes
admission. Twenty race-enabled repetitions of the selected capacity,
cancellation, and proposal-order tests passed. The repetitions exercise the
present fixtures, while the state-observation gaps remain.
[Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67)
· [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96)
Hold the earlier result until later completion is observable and assert the
actual assigned indices and section identities. At the adapter boundary, prove
that work reached the contested state before canceling it. Observe peak active
dispatches relative to a test-controlled limit and check that waiting work can
proceed or time out according to the configured contract. Keep wall-clock
deadlines as bounds for the test itself where possible.
[Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy)
### 7. Representative transcript fixtures leave data-integrity boundaries open
| Behavior | Current protection and gap | Follow-up |
| --- | --- | --- |
| Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) |
| A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) |
| An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) |
### 8. Debug redaction assertions mostly inspect the outer serialization
The subprocess test confirms that it actually captured a provider echo, then
checks command streams and retained files for the clear credential value. The
redactor unit test has a useful nested `final_content` case built with Go's
own JSON encoder. Neither asserts the decoded content of that nested field
when the provider uses another legal JSON representation. A literal-byte
absence check can pass while the credential survives there. This matters to
the documented diagnostic redaction boundary, even when a private debug
directory limits access on disk.
[Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965)
· [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43)
· [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential)
Use a synthetic credential and verify the semantic content after parsing the
retained artifact and any structured response field it contains. Own detailed
string representation cases at the diagnostic writer. Keep one real provider
test to prove that credential discovery and artifact redaction connect.
[Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope)
## Candidates to remove or consolidate
### 9. Delete proposal-order assertions on the test-only composition harness
`TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness`
twice and checks indices that the harness itself assigns with `nextIndex++`.
It cannot detect an indexing regression in `Runner.Run`. The harness also
passes the full transcript into a fake module and delegates range filtering
to that fake. Production builds section-local requests. Related harness cases
verify a composition that the test constructs itself. Remove the index test
once the real runner test checks indices under controlled completion. Migrate
any remaining distinct input-preservation or stale-application assertion to
the existing runner or proposal API coverage before removing its harness case.
[Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14)
· [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143)
· [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356)
· [Stable application test](../../../internal/framework/proposals/apply_test.go#L36)
### 10. Consolidate validator registry and asset metadata assertions
`TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the
class comparison in `TestRegistryBuildsClassifiedValidators`. The second test
also requires the concrete validator to implement `ClassifiedValidator` even
though `ClassOf` supports a stable-key fallback. A behavior-preserving change
to that representation would fail the type assertion. Keep one table that
builds registered keys and checks effective classes against documented roles.
Preserve protective checks of concrete domain behavior and stable selector keys.
[Duplicate registry checks](../../../internal/validators/registry_test.go#L80)
· [Explicit class examples](../../../internal/validators/registry_test.go#L48)
· [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54)
`TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already
enforced at registry construction. The native preparation tests give stronger
evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions`
checks an exact count of 11 prompts. That count detects additions while
allowing a same-size substitution. Prefer a check for each required prompt ID
and its preparability. Retain the tests for canonical schema asset agreement,
stable diagnostics data, and protective copy behavior.
[Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71)
· [Production construction](../../../internal/framework/responseschema/registry.go#L129)
· [Prompt metadata count](../../../internal/prompts/registry_test.go#L79)
· [Native preparation](../../../internal/prompts/registry_test.go#L13)
### 11. Remove the tests of the release test's own set helper
`TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined
in the same test file. It cannot detect a malformed release profile or
bootstrap behavior. A standard comparison over the decoded profile can retain
the documented target set, product and destination checks, and the credential
exclusion without a test of the comparison's implementation. If adding release
coverage, give it to the actual bootstrap and its exit handling. The known
Git-status failure is a representative offline shell boundary, and
ReleaseKit already checks the prepared checkout on its own boundary.
[Test of the helper](../../../scripts/release_test.go#L63)
· [Real profile assertions](../../../scripts/release_test.go#L10)
· [Delivery contract](../../release.md#binary-delivery)
· [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty)
### 12. Consolidate normalization fixtures at the domain and parser owners
`TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers`
exercise the same rejection with different gap values that remain below the
configured maximum. One case can assert the unchanged transcript and the
meaningful merge counts together. Seven normalization golden tests also
repeat the conversion from `SourceTranscript` to `Transcript` in test code.
That conversion mirrors the CLI and cannot catch a regression in the CLI's
copy. The fixtures compare decoded values and contain useful domain assertions,
including exact text and categories. Keep the distinct domain cases, remove
duplicate fixtures after comparing assertions, and let the parser and one
assembled CLI case own source shapes and wiring. No automatic golden update
was found in these tests.
[Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11)
· [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101)
· [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329)
· [Production conversion](../../../internal/cli/run.go#L12)
### 13. Narrow integration assertions to their distinct contracts
`TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling
of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the
same decoded ID with different escaping would fail the test. Decode the
artifacts and assert stage, role, and evidence association. Keep end-to-end
checks for per-entry profiles, successive transcript state, and output
selection, while the reporting package owns Markdown layout and escaping.
[Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145)
· [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101)
`TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure
scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure`
uses the same model script and also checks the distinct attribution in both
YAML documents. The selection matrix took about 6.9 seconds in the
instrumented run. Keep one detailed scenario with the attribution assertions
and use a smaller fixture for any duplicate combination once it still proves
that reporting and debug selections leave decisions and counts intact. The
shared fixture also writes a legacy glossary file and stores its path without
using it. Remove that unused fixture input separately from behavioral checks.
[Selection matrix](../../../cmd/audita/main_integration_test.go#L879)
· [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013)
· [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283)
## Verification scope
The full race-and-coverage run, 20 repetitions of the selected concurrency
tests, and the offline CLI classification probe completed successfully. The
audit used source inspection to identify false passes and refactor-sensitive
assertions. The full suite uses synthetic model responses, consistent with the
offline policy. Live-model editorial quality is a separate, deferred evaluation.
[Execution policy](../../policy/testing.md#test-execution-requirements)
· [Deferred evaluation](../future.md#live-model-editorial-evaluation)