356 lines
24 KiB
Markdown
356 lines
24 KiB
Markdown
|
|
# Test coverage and effectiveness audit
|
||
|
|
|
||
|
|
Reviewed 2026-11-15 at `416cdea`. The audit used the canonical testing policy to
|
||
|
|
assess protection against consequential failures, test ownership, fixture
|
||
|
|
fidelity, and maintenance cost. The findings are follow-up options.
|
||
|
|
[Testing policy](../../policy/testing.md)
|
||
|
|
|
||
|
|
## Architecture and measured coverage
|
||
|
|
|
||
|
|
Audita is a single-process Go CLI. CommandKit handles invocation, ConfigKit
|
||
|
|
loads application settings, and core packages parse, normalize, and chunk the
|
||
|
|
transcript. The runner coordinates correction proposals, validators, repairs,
|
||
|
|
and edit application. The role adapter uses a shared PromptKit engine, and the
|
||
|
|
CLI publishes the transcript and reports through an output session. Tests use
|
||
|
|
pure domain inputs, temporary files, an injected HTTP transport or loopback
|
||
|
|
provider, and subprocesses. Release tests in this repository inspect the local
|
||
|
|
profile, while ReleaseKit owns the underlying delivery implementation.
|
||
|
|
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
|
||
|
|
· [LLM runtime](../../internal/llm-runtime.md)
|
||
|
|
· [Release procedure](../../release.md#binary-delivery)
|
||
|
|
|
||
|
|
On Go 1.27.3, the full `go test -count=1 -race -covermode=atomic -coverpkg=./...`
|
||
|
|
run passed with `-coverprofile` and `-json` output directed to temporary files.
|
||
|
|
Go's cover tool reported **89.3%**
|
||
|
|
statement coverage. The JSON run log recorded 441 passing top-level tests and
|
||
|
|
210 passing named subtests. The package percentages below merge repeated
|
||
|
|
profile rows by source block before counting statements.
|
||
|
|
|
||
|
|
| Package or boundary | Statement coverage | Audit observation |
|
||
|
|
| --- | ---: | --- |
|
||
|
|
| `cmd/audita` | 0.0% | The test executable handles signals in a helper. |
|
||
|
|
| `internal/framework/preflight` | 73.8% | `Prepare` is at 63.9%, including gaps around cancellation and report recovery. |
|
||
|
|
| `internal/core/normalization` | 78.6% | Several merge cases exercise ordinary timestamps. |
|
||
|
|
| `internal/core/diagnostics` | 81.4% | Tests exercise publication and cleanup, with gaps in failure sequencing. |
|
||
|
|
| `internal/core/schema` | 83.3% | Source decoding has shape and validation cases, with a precision gap. |
|
||
|
|
| `internal/cli` | 87.6% | Command invocation is covered more broadly than failure classification. |
|
||
|
|
| `internal/framework/runner` | 91.3% | Partial outcomes, repair, and cancellation are exercised. |
|
||
|
|
| `internal/framework/llm` | 91.4% | Adapter failures and shared capacity are covered with conditions noted below. |
|
||
|
|
| `internal/core/chunking` | 94.5% | The explicit-target builder is at 85.2%. |
|
||
|
|
|
||
|
|
The percentages describe statements reached in this run. The audit assigned
|
||
|
|
test value by the assertions and failure scenario. In particular, low coverage
|
||
|
|
in test support or thin forwarding wrappers does not justify standalone tests.
|
||
|
|
Existing higher-value checks include partial validation with surviving edits,
|
||
|
|
repair revalidation, typed failure causes, stable user-visible identities, and
|
||
|
|
the presence of a masked credential echo in an actual retained artifact.
|
||
|
|
[Coverage policy](../../policy/testing.md#coverage)
|
||
|
|
· [Partial validation](../../../cmd/audita/main_integration_test.go#L1091)
|
||
|
|
· [Repair integration](../../../cmd/audita/correction_repair_integration_test.go)
|
||
|
|
· [Failure identity](../../../internal/framework/structuredoutput/failure_test.go#L10)
|
||
|
|
· [Credential echo](../../../cmd/audita/main_integration_test.go#L965)
|
||
|
|
|
||
|
|
## Behavior that needs stronger verification
|
||
|
|
|
||
|
|
### 1. Subprocess tests recreate the production executable boundary
|
||
|
|
|
||
|
|
The subprocess suite launches the test executable. Its helper registers
|
||
|
|
SIGINT and SIGTERM, passes arguments and streams to `cli.Run`, and exits with
|
||
|
|
the returned code. The production `run` and `main` contain their own copy of
|
||
|
|
that wiring and recorded zero hits. A regression in production-only signal
|
||
|
|
registration, argument slicing, or exit propagation can pass the subprocess
|
||
|
|
suite, including the cancellation case. The existing tests still exercise
|
||
|
|
real process boundaries and the assembled CLI.
|
||
|
|
[Production entry](../../../cmd/audita/main.go#L12)
|
||
|
|
· [Subprocess helper](../../../cmd/audita/main_integration_test.go#L28)
|
||
|
|
· [Signal test](../../../cmd/audita/main_integration_test.go#L1059)
|
||
|
|
|
||
|
|
Use the actual built executable for one representative startup and signal or
|
||
|
|
exit test, or have the helper invoke the same production startup function.
|
||
|
|
Keep the smaller existing seams for the rest of the CLI suite.
|
||
|
|
|
||
|
|
### 2. Inspection failure tests accept the wrong exit classification
|
||
|
|
|
||
|
|
`TestResourceCommandsClassifyOperatorPromptErrorsAsInputErrors` and
|
||
|
|
`TestResourceCommandKeepsCallerCancellationAsFailure` reject only exit zero.
|
||
|
|
Both can pass if the integration maps invalid input and cancellation to the
|
||
|
|
same nonzero status. A temporary build given the same malformed operator
|
||
|
|
prompt file returned `1` via `promptkit list prompts` and `2` via `process`.
|
||
|
|
The CLI contract distinguishes input failure (`2`) from runtime failure (`1`).
|
||
|
|
The input test also sets a synthetic credential that its malformed templates,
|
||
|
|
profiles, and paths never reference, so the absence assertion provides no
|
||
|
|
evidence that a configured secret was found and redacted.
|
||
|
|
[Tests and fixtures](../../../internal/cli/runtime_errors_test.go#L24)
|
||
|
|
· [CLI exit contract](../../cli.md#inspection-and-outcomes)
|
||
|
|
· [CLI wiring](../../../internal/cli/app_wiring.go#L80)
|
||
|
|
|
||
|
|
Assert the selected status and a stable diagnostic fragment or typed cause.
|
||
|
|
Clarify the intended delegated-command classification if it differs from the
|
||
|
|
CLI contract. Test redaction where a known value reaches the relevant channel,
|
||
|
|
as the existing credential echo test does, and remove the unused secret
|
||
|
|
assertion from these cases.
|
||
|
|
[Nonvacuous redaction test](../../../cmd/audita/main_integration_test.go#L965)
|
||
|
|
|
||
|
|
### 3. Output tests primarily check the final artifact state
|
||
|
|
|
||
|
|
The short-write CLI test asserts final YAML status, counts, and a reason code.
|
||
|
|
The replacement test checks the result after a failed-report pass. Both can
|
||
|
|
pass when finalization briefly writes `completed` reports after it already
|
||
|
|
knows transcript delivery failed. The short-write test also passes if the
|
||
|
|
required run-error phase is missing. Collision tests cover aliases present
|
||
|
|
during selection, while the finalization path resolves report destinations
|
||
|
|
again. Existing tests that change the transcript's file type exercise a
|
||
|
|
different path. Prior audits documented both the transient report state and
|
||
|
|
the changed report referent.
|
||
|
|
[CLI short-write test](../../../internal/cli/process_output_test.go#L79)
|
||
|
|
· [Final report pass test](../../../internal/core/diagnostics/output_session_test.go#L207)
|
||
|
|
· [Collision test](../../../internal/core/diagnostics/output_session_test.go#L14)
|
||
|
|
· [Changed transcript type test](../../../internal/core/diagnostics/output_session_test.go#L435)
|
||
|
|
· [Report publication finding](06-error-handling.md#1-finalization-publishes-completed-reports-after-it-knows-delivery-failed)
|
||
|
|
· [Report path finding](04-concurrency.md#6-report-paths-can-change-referent-between-initialization-and-finalization)
|
||
|
|
|
||
|
|
At `OutputSession`, observe the first report state after a known delivery
|
||
|
|
failure and verify that a changed report destination cannot defeat the checked
|
||
|
|
identity. At the CLI boundary, assert the exit and serialized run phase. The
|
||
|
|
failure-injection seam is useful because it observes publication state at an
|
||
|
|
externally consequential boundary. Preserve the documented nontransactional
|
||
|
|
behavior when a report error occurs after a transcript has been published.
|
||
|
|
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||
|
|
· [Run error fields](../../integrations/output-files.md#error-and-recovery-records)
|
||
|
|
|
||
|
|
The stream tests open a FIFO peer before final delivery and use a payload
|
||
|
|
that fits in the pipe buffer. The cancellation tests exercise a caller
|
||
|
|
canceled before delivery or while the provider is active. A supported stream
|
||
|
|
blocked in delivery therefore falls outside their assertions. Verify bounded
|
||
|
|
subprocess cancellation and failed-status finalization after the supported
|
||
|
|
stream semantics are decided. Keep the existing small delivery tests for the
|
||
|
|
successful stream contract.
|
||
|
|
[FIFO fixture](../../../internal/core/diagnostics/output_stream_test.go#L13)
|
||
|
|
· [Pre-delivery cancellation test](../../../internal/cli/process_output_test.go#L131)
|
||
|
|
· [Blocked-stream finding](04-concurrency.md#1-blocked-output-delivery-can-ignore-cancellation-and-report-success)
|
||
|
|
|
||
|
|
Coverage recorded no hits in `Prepare`'s cancellation handling after acquiring
|
||
|
|
the transcript staging file or initializing the reports, and no hits in `Close`
|
||
|
|
after a temporary path removal error. The existing cleanup tests cover a write
|
||
|
|
or rename failure followed by successful cleanup. A well-placed preflight test
|
||
|
|
can observe output and report state when cancellation follows initialization.
|
||
|
|
At the output-session boundary, verify which cleanup errors are reported and
|
||
|
|
whether a temporary removal failure can be retried under the selected contract.
|
||
|
|
[Preflight cancellation](../../../internal/framework/preflight/preflight.go#L61)
|
||
|
|
· [Existing cleanup test](../../../internal/core/diagnostics/output_session_test.go#L394)
|
||
|
|
· [Earlier cleanup finding](05-resources.md#1-failed-output-cleanup-loses-ownership-of-temporary-paths)
|
||
|
|
|
||
|
|
### 4. The timeout tests cover two of the three relevant deadline owners
|
||
|
|
|
||
|
|
The adapter test distinguishes a shorter Audita operation budget from an
|
||
|
|
already canceled parent. Other adapter cases check provider statuses and
|
||
|
|
transport failures. They do not exercise a profile's HTTP attempt deadline
|
||
|
|
expiring under a still-live parent and a longer operation budget. The earlier
|
||
|
|
CLI audit observed that case labeled `operation_timeout`, even though
|
||
|
|
PromptKit reserves its operation sentinel for the logical budget.
|
||
|
|
[Timeout and cancellation test](../../../internal/framework/llm/adapter_test.go#L406)
|
||
|
|
· [Provider status test](../../../internal/framework/llm/adapter_test.go#L462)
|
||
|
|
· [Earlier timeout finding](06-error-handling.md#2-a-provider-request-timeout-is-reported-as-the-logical-operation-deadline)
|
||
|
|
|
||
|
|
Put the deadline distinction at the adapter boundary using a controlled
|
||
|
|
transport and deliberately separated deadlines. Assert both recoverability
|
||
|
|
and the selected cause or reason. The CLI already has report mapping tests
|
||
|
|
and only needs representative coverage of the adapter-to-report connection.
|
||
|
|
[Timeout contract](../../integrations/openai-compatible-llm.md)
|
||
|
|
|
||
|
|
### 5. The validator batch fixture misses the real numeric decoding boundary
|
||
|
|
|
||
|
|
The subprocess mixed-response test and the validator reconciliation tests
|
||
|
|
protect sibling decisions when ordinary integer references are missing,
|
||
|
|
duplicated, or unexpected. The adapter has a separate overflow test with a
|
||
|
|
single out-of-range reference and no usable sibling. The gap is a response
|
||
|
|
with both a usable requested decision and an unexpected integer too large for
|
||
|
|
the Go index type. Native schema validation can accept that integer, but the
|
||
|
|
typed adapter decode can reject the whole batch before per-entry reconciliation.
|
||
|
|
The existing overflow test provides evidence for failure classification and
|
||
|
|
cannot establish sibling preservation.
|
||
|
|
[Mixed-response integration](../../../cmd/audita/main_integration_test.go#L567)
|
||
|
|
· [Adapter overflow case](../../../internal/framework/llm/adapter_test.go#L429)
|
||
|
|
· [Earlier reference decoding finding](01-correctness.md#7-an-extra-numeric-validator-reference-can-discard-valid-sibling-decisions)
|
||
|
|
|
||
|
|
Exercise a mixed response through the actual provider envelope, schema, and
|
||
|
|
adapter decode. Assert the usable sibling's decision and the appropriate
|
||
|
|
disposition for the extra entry. A stub returning a pretyped integer cannot
|
||
|
|
cover this case. Keep smaller reconciliation tests for the ordinary identity
|
||
|
|
rules.
|
||
|
|
[Validator reconciliation contract](../../internal/validators.md#llm-validator-batching)
|
||
|
|
|
||
|
|
### 6. Concurrency fixtures leave some schedules unproved
|
||
|
|
|
||
|
|
The runner's proposal-order test waits for all proposals to start, then
|
||
|
|
releases three gates back-to-back in reverse section order. That establishes a
|
||
|
|
release order, while workers may deliver their results in a different order.
|
||
|
|
The test checks applied-change order by target segment and final text, without
|
||
|
|
asserting the proposal indices promised by its name. Order by segment can
|
||
|
|
survive a change to proposal index assignment because corrections are applied
|
||
|
|
within their sections. The runner has other fixed-chunk and application-order
|
||
|
|
checks, so the missing evidence is deterministic indexing under a demonstrably
|
||
|
|
different completion order.
|
||
|
|
[Parallel runner test](../../../internal/framework/runner/runner_test.go#L423)
|
||
|
|
· [Result assertions](../../../internal/framework/runner/runner_test.go#L473)
|
||
|
|
· [Application and recombination](../../../internal/framework/runner/runner.go#L300)
|
||
|
|
· [Deterministic indexing policy](../../policy/architecture.md#correction-and-concurrency-invariants)
|
||
|
|
|
||
|
|
The shared-backend test holds the first HTTP call, then gives the second call
|
||
|
|
a 30 ms deadline. It does not verify that the second call reached admission
|
||
|
|
or the wait before it timed out. On a delayed execution, an earlier timeout
|
||
|
|
can satisfy the same assertions. The runner cancellation test observes native
|
||
|
|
`ExecutionStarted` for both peers and one dispatch. That event precedes
|
||
|
|
admission. Twenty race-enabled repetitions of the selected capacity,
|
||
|
|
cancellation, and proposal-order tests passed. The repetitions exercise the
|
||
|
|
present fixtures, while the state-observation gaps remain.
|
||
|
|
[Shared capacity test](../../../internal/framework/llm/adapter_capacity_test.go#L42)
|
||
|
|
· [Peer cancellation test](../../../internal/framework/runner/cancellation_test.go#L67)
|
||
|
|
· [Distinct backend control](../../../internal/framework/llm/adapter_capacity_test.go#L96)
|
||
|
|
|
||
|
|
Hold the earlier result until later completion is observable and assert the
|
||
|
|
actual assigned indices and section identities. At the adapter boundary, prove
|
||
|
|
that work reached the contested state before canceling it. Observe peak active
|
||
|
|
dispatches relative to a test-controlled limit and check that waiting work can
|
||
|
|
proceed or time out according to the configured contract. Keep wall-clock
|
||
|
|
deadlines as bounds for the test itself where possible.
|
||
|
|
[Concurrency testing policy](../../policy/testing.md#separate-mechanism-from-policy)
|
||
|
|
|
||
|
|
### 7. Representative transcript fixtures leave data-integrity boundaries open
|
||
|
|
|
||
|
|
| Behavior | Current protection and gap | Follow-up |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| Accepted source IDs remain distinct | Source parser tests cover both shapes and ordinary duplicate IDs. They miss adjacent integers within native `int` range that collapse in the intermediate generic JSON decode. | Assert exact typed IDs and duplicate detection at the precision boundary in the source parser, with a word-size-appropriate fixture. [Parser cases](../../../internal/core/schema/transcript_test.go#L133) · [Earlier fidelity finding](01-correctness.md#6-transcript-shape-detection-loses-integer-and-unknown-field-fidelity) |
|
||
|
|
| A merged interval covers all source speech | Merge tests use increasing end times. The chronological test checks the outer start and end for disjoint inputs. A nested same-speaker interval can shorten the result without failing them. | Assert the applicable merged time envelope for overlapping or nested segments, along with the text or category invariants already owned by normalization. [Existing time assertions](../../../internal/core/normalization/normalize_test.go#L182) · [Earlier timestamp finding](01-correctness.md#3-merging-nested-segments-truncates-the-earlier-timestamp-range) |
|
||
|
|
| An achievable explicit chunk target succeeds | The success case uses equal weights and ample capacity. The impossible cases check obvious upper and lower bounds. These can pass when a skewed but feasible partition is incorrectly rejected. | Use heterogeneous weights and a case near the limit. For broader confidence, a small exhaustive partition oracle can check feasibility, contiguity, order, and coverage without pinning the optimizer. [Target cases](../../../internal/core/chunking/sections_test.go#L279) · [Earlier feasibility finding](01-correctness.md#4-balanced-chunking-rejects-feasible-explicit-targets) |
|
||
|
|
|
||
|
|
### 8. Debug redaction assertions mostly inspect the outer serialization
|
||
|
|
|
||
|
|
The subprocess test confirms that it actually captured a provider echo, then
|
||
|
|
checks command streams and retained files for the clear credential value. The
|
||
|
|
redactor unit test has a useful nested `final_content` case built with Go's
|
||
|
|
own JSON encoder. Neither asserts the decoded content of that nested field
|
||
|
|
when the provider uses another legal JSON representation. A literal-byte
|
||
|
|
absence check can pass while the credential survives there. This matters to
|
||
|
|
the documented diagnostic redaction boundary, even when a private debug
|
||
|
|
directory limits access on disk.
|
||
|
|
[Subprocess echo assertions](../../../cmd/audita/main_integration_test.go#L965)
|
||
|
|
· [Nested redaction test](../../../internal/core/redaction/redaction_test.go#L43)
|
||
|
|
· [Earlier debug finding](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential)
|
||
|
|
|
||
|
|
Use a synthetic credential and verify the semantic content after parsing the
|
||
|
|
retained artifact and any structured response field it contains. Own detailed
|
||
|
|
string representation cases at the diagnostic writer. Keep one real provider
|
||
|
|
test to prove that credential discovery and artifact redaction connect.
|
||
|
|
[Diagnostic trust policy](../../policy/architecture.md#trust-boundaries-and-threat-scope)
|
||
|
|
|
||
|
|
## Candidates to remove or consolidate
|
||
|
|
|
||
|
|
### 9. Delete proposal-order assertions on the test-only composition harness
|
||
|
|
|
||
|
|
`TestDeterministicProposalIndexAssignment` runs `testChunkProposalHarness`
|
||
|
|
twice and checks indices that the harness itself assigns with `nextIndex++`.
|
||
|
|
It cannot detect an indexing regression in `Runner.Run`. The harness also
|
||
|
|
passes the full transcript into a fake module and delegates range filtering
|
||
|
|
to that fake. Production builds section-local requests. Related harness cases
|
||
|
|
verify a composition that the test constructs itself. Remove the index test
|
||
|
|
once the real runner test checks indices under controlled completion. Migrate
|
||
|
|
any remaining distinct input-preservation or stale-application assertion to
|
||
|
|
the existing runner or proposal API coverage before removing its harness case.
|
||
|
|
[Test-only orchestration](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L14)
|
||
|
|
· [Test-only index assertion](../../../internal/framework/contracts/chunk_proposal_composition_test.go#L143)
|
||
|
|
· [Real runner section contract](../../../internal/framework/runner/runner_test.go#L356)
|
||
|
|
· [Stable application test](../../../internal/framework/proposals/apply_test.go#L36)
|
||
|
|
|
||
|
|
### 10. Consolidate validator registry and asset metadata assertions
|
||
|
|
|
||
|
|
`TestExecutionClassResolvableByStableKeyAndByValidatorInstance` repeats the
|
||
|
|
class comparison in `TestRegistryBuildsClassifiedValidators`. The second test
|
||
|
|
also requires the concrete validator to implement `ClassifiedValidator` even
|
||
|
|
though `ClassOf` supports a stable-key fallback. A behavior-preserving change
|
||
|
|
to that representation would fail the type assertion. Keep one table that
|
||
|
|
builds registered keys and checks effective classes against documented roles.
|
||
|
|
Preserve protective checks of concrete domain behavior and stable selector keys.
|
||
|
|
[Duplicate registry checks](../../../internal/validators/registry_test.go#L80)
|
||
|
|
· [Explicit class examples](../../../internal/validators/registry_test.go#L48)
|
||
|
|
· [Fallback behavior](../../../internal/validators/metadata/metadata_test.go#L54)
|
||
|
|
|
||
|
|
`TestRegisteredSchemasLoadReadableEmbeddedJSON` repeats JSON validity already
|
||
|
|
enforced at registry construction. The native preparation tests give stronger
|
||
|
|
evidence that PromptKit can use the schemas. `TestRegistryMetadataMatchesEmbeddedDefinitions`
|
||
|
|
checks an exact count of 11 prompts. That count detects additions while
|
||
|
|
allowing a same-size substitution. Prefer a check for each required prompt ID
|
||
|
|
and its preparability. Retain the tests for canonical schema asset agreement,
|
||
|
|
stable diagnostics data, and protective copy behavior.
|
||
|
|
[Repeated asset validity check](../../../internal/framework/responseschema/registry_test.go#L71)
|
||
|
|
· [Production construction](../../../internal/framework/responseschema/registry.go#L129)
|
||
|
|
· [Prompt metadata count](../../../internal/prompts/registry_test.go#L79)
|
||
|
|
· [Native preparation](../../../internal/prompts/registry_test.go#L13)
|
||
|
|
|
||
|
|
### 11. Remove the tests of the release test's own set helper
|
||
|
|
|
||
|
|
`TestReleaseKitTargetMembership` calls only `sameMembers`, a helper defined
|
||
|
|
in the same test file. It cannot detect a malformed release profile or
|
||
|
|
bootstrap behavior. A standard comparison over the decoded profile can retain
|
||
|
|
the documented target set, product and destination checks, and the credential
|
||
|
|
exclusion without a test of the comparison's implementation. If adding release
|
||
|
|
coverage, give it to the actual bootstrap and its exit handling. The known
|
||
|
|
Git-status failure is a representative offline shell boundary, and
|
||
|
|
ReleaseKit already checks the prepared checkout on its own boundary.
|
||
|
|
[Test of the helper](../../../scripts/release_test.go#L63)
|
||
|
|
· [Real profile assertions](../../../scripts/release_test.go#L10)
|
||
|
|
· [Delivery contract](../../release.md#binary-delivery)
|
||
|
|
· [Bootstrap failure](06-error-handling.md#7-ci-bootstrap-treats-a-failed-git-status-command-as-clean-if-stdout-is-empty)
|
||
|
|
|
||
|
|
### 12. Consolidate normalization fixtures at the domain and parser owners
|
||
|
|
|
||
|
|
`TestNormalizationNoOp` and `TestNormalizationNoMergeDifferentSpeakers`
|
||
|
|
exercise the same rejection with different gap values that remain below the
|
||
|
|
configured maximum. One case can assert the unchanged transcript and the
|
||
|
|
meaningful merge counts together. Seven normalization golden tests also
|
||
|
|
repeat the conversion from `SourceTranscript` to `Transcript` in test code.
|
||
|
|
That conversion mirrors the CLI and cannot catch a regression in the CLI's
|
||
|
|
copy. The fixtures compare decoded values and contain useful domain assertions,
|
||
|
|
including exact text and categories. Keep the distinct domain cases, remove
|
||
|
|
duplicate fixtures after comparing assertions, and let the parser and one
|
||
|
|
assembled CLI case own source shapes and wiring. No automatic golden update
|
||
|
|
was found in these tests.
|
||
|
|
[Overlapping cases](../../../internal/core/normalization/normalize_test.go#L11)
|
||
|
|
· [Different-speaker case](../../../internal/core/normalization/normalize_test.go#L101)
|
||
|
|
· [Copied conversion and semantic comparison](../../../internal/core/normalization/normalize_test.go#L329)
|
||
|
|
· [Production conversion](../../../internal/cli/run.go#L12)
|
||
|
|
|
||
|
|
### 13. Narrow integration assertions to their distinct contracts
|
||
|
|
|
||
|
|
`TestProcessingUsesPerEntryProfilesInOrder` checks the escaped byte spelling
|
||
|
|
of stage IDs in two JSON debug artifacts. A JSON encoder that preserves the
|
||
|
|
same decoded ID with different escaping would fail the test. Decode the
|
||
|
|
artifacts and assert stage, role, and evidence association. Keep end-to-end
|
||
|
|
checks for per-entry profiles, successive transcript state, and output
|
||
|
|
selection, while the reporting package owns Markdown layout and escaping.
|
||
|
|
[Integration artifact assertions](../../../internal/cli/process_ordered_integration_test.go#L145)
|
||
|
|
· [Dedicated report coverage](../../../internal/core/reporting/markdown_test.go#L101)
|
||
|
|
|
||
|
|
`TestSubprocessReportingAndDebugSelections` runs the repair/filter/later-failure
|
||
|
|
scenario in all four flag combinations. `TestSubprocessRecoveredFilteringSurvivesLaterFailure`
|
||
|
|
uses the same model script and also checks the distinct attribution in both
|
||
|
|
YAML documents. The selection matrix took about 6.9 seconds in the
|
||
|
|
instrumented run. Keep one detailed scenario with the attribution assertions
|
||
|
|
and use a smaller fixture for any duplicate combination once it still proves
|
||
|
|
that reporting and debug selections leave decisions and counts intact. The
|
||
|
|
shared fixture also writes a legacy glossary file and stores its path without
|
||
|
|
using it. Remove that unused fixture input separately from behavioral checks.
|
||
|
|
[Selection matrix](../../../cmd/audita/main_integration_test.go#L879)
|
||
|
|
· [Detailed attribution](../../../cmd/audita/main_integration_test.go#L1013)
|
||
|
|
· [Unused fixture input](../../../cmd/audita/main_integration_test.go#L283)
|
||
|
|
|
||
|
|
## Verification scope
|
||
|
|
|
||
|
|
The full race-and-coverage run, 20 repetitions of the selected concurrency
|
||
|
|
tests, and the offline CLI classification probe completed successfully. The
|
||
|
|
audit used source inspection to identify false passes and refactor-sensitive
|
||
|
|
assertions. The full suite uses synthetic model responses, consistent with the
|
||
|
|
offline policy. Live-model editorial quality is a separate, deferred evaluation.
|
||
|
|
[Execution policy](../../policy/testing.md#test-execution-requirements)
|
||
|
|
· [Deferred evaluation](../future.md#live-model-editorial-evaluation)
|