base-layer/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Record final validation and repair audit links

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-05 20:19:06 +00:00
parent 96c619f7d4
commit 5ee602cc4b
13 changed files with 168 additions and 158 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
docs/roadmap/audit/08-testing.md
View File

@@ -62,8 +62,8 @@ rather than becoming duplicate remediation projects.
**Priority:** P2. Missing protection for a documented confidentiality boundary.
**Relevant code:** [RunRequest formatting](../../../formatting.go:5) and the
analogous existing [SmokeTestRequest privacy test](../../../smoke_test_contract_test.go:135).
**Relevant code:** [RunRequest formatting](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/formatting.go#L5) and the
analogous existing [SmokeTestRequest privacy test](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/smoke_test_contract_test.go#L135).
**Deficiency:** `RunRequest.String`, `GoString`, and their shared formatter all
have 0% combined statement coverage. The suite protects formatting of generated
@@ -86,12 +86,12 @@ Reflection-based disclosure is explicitly outside this formatter guarantee.
**Priority:** P2. An exported source-selection path is entirely unprotected.
**Relevant code:** [WithSchemaFile](../../../engine.go:212),
[SetSchemaFile](../../../internal/engine/engine.go:75), and
[fileSource](../../../internal/engine/engine.go:211).
**Relevant code:** [WithSchemaFile](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/engine.go#L212),
[SetSchemaFile](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/engine/engine.go#L75), and
[fileSource](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/engine/engine.go#L211).
**Deficiency:** All three functions have 0% combined coverage. The existing
[single-file validator test](../../../internal/validate/standard_validator_test.go:596)
[single-file validator test](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/validate/standard_validator_test.go#L596)
constructs `FSValidator` directly with `fstest.MapFS`; it does not test the public
option, host-file selection, or its engine wiring.
@@ -113,9 +113,9 @@ helper-by-helper test matrix or model-provider access.
**Priority:** P2. Conversion and ownership coverage misses known data-integrity
and deferred-error risks.
**Relevant tests:** [programmatic source ownership/order](../../../sources_contract_test.go:14)
and [programmatic construction](../../../construction_contract_test.go:226).
**Implementation:** [providerValueForMapping](../../../provider_values.go:31).
**Relevant tests:** [programmatic source ownership/order](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/sources_contract_test.go#L14)
and [programmatic construction](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/construction_contract_test.go#L226).
**Implementation:** [providerValueForMapping](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/provider_values.go#L31).
**Deficiency:** The converter has only 11.9% combined coverage. The principal
source-ownership test mutates a flat string-valued map. It does not establish
@@ -143,10 +143,10 @@ coverage for the separately identified shared-graph issue belongs with that fix.
**Priority:** P2. Covered code paths lack important boundary-case assertions.
**Relevant tests:** [public generation failures](../../../generation_error_contract_test.go:111),
[direct mapper cases](../../../errors_internal_test.go:13),
[body cancellation](../../../internal/llm/prepared_generation_test.go:161), and
[inference diagnostic cases](../../../internal/usecase/smoke_inference_test.go:14).
**Relevant tests:** [public generation failures](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/generation_error_contract_test.go#L111),
[direct mapper cases](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/errors_internal_test.go#L13),
[body cancellation](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/llm/prepared_generation_test.go#L161), and
[inference diagnostic cases](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/usecase/smoke_inference_test.go#L14).
**Deficiency and demonstrated blind spots:**
@@ -180,9 +180,9 @@ precedence, best-effort non-2xx detail parsing, and status-only retry policy.
**Priority:** P2. Missing assembled validation-to-wire assertion.
**Relevant tests:** [prepared execution source freezing](../../../prepared_execution_contract_test.go:19),
its [referenced schema fixture](../../../prepared_execution_contract_test.go:809),
and [local reference validation](../../../internal/validate/standard_validator_test.go:627).
**Relevant tests:** [prepared execution source freezing](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/prepared_execution_contract_test.go#L19),
its [referenced schema fixture](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/prepared_execution_contract_test.go#L809),
and [local reference validation](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/validate/standard_validator_test.go#L627).
**Deficiency:** Local tests establish that referenced schemas compile and
validate. The prepared-execution test checks that the injected client's schema
@@ -210,11 +210,11 @@ promise support for arbitrary provider schema dialect features.
**Priority:** P3. Remove or rewrite change-detector assertions, retaining the
substantive file-reading tests.
**Relevant assertions:** [size-hint test](../../../internal/artifact/reader_test.go:287)
and [benchmark correctness check](../../../internal/artifact/reader_benchmark_test.go:156).
**Relevant assertions:** [size-hint test](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/artifact/reader_test.go#L287)
and [benchmark correctness check](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/artifact/reader_benchmark_test.go#L156).
**Deficiency:** Both independently calculate a hex SHA-256 digest and require
that exact value. The [public Artifact.Hash contract](../../../types.go:290)
that exact value. The [public Artifact.Hash contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/types.go#L290)
expressly leaves its format and algorithm unspecified. A legitimate algorithm
or encoding change would fail tests unrelated to the requested behavior.
@@ -222,7 +222,7 @@ or encoding change would fail tests unrelated to the requested behavior.
opaque equality relationships needed for each test. For the under/overreported
size cases, comparing with the inline reader for the same complete body still
detects hashing only a reported-size prefix. The existing
[source-parity test](../../../internal/artifact/reader_test.go:27) already owns
[source-parity test](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/artifact/reader_test.go#L27) already owns
repeatability, same-content equality, and changed-content inequality. Retain
body, actual-size, metadata, and read-boundary assertions. The benchmark may
reuse an opaque expected hash prepared outside measurement or rely on the
@@ -232,10 +232,10 @@ owning behavioral test; it need not maintain its own SHA-256 contract.
**Priority:** P3. Consolidation/removal candidate for duplicated assertions.
**Relevant tests:** [internal complete catalog table](../../../internal/configuration/bundled_test.go:12),
[public option/inventory table](../../../bundled_openrouter_contract_test.go:54),
[HTTP payload matrix](../../../bundled_execution_contract_test.go:51), and
[bundled smoke defaults](../../../bundled_smoke_contract_test.go:16).
**Relevant tests:** [internal complete catalog table](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/internal/configuration/bundled_test.go#L12),
[public option/inventory table](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/bundled_openrouter_contract_test.go#L54),
[HTTP payload matrix](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/bundled_execution_contract_test.go#L51), and
[bundled smoke defaults](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/f7085d7/bundled_smoke_contract_test.go#L16).
**Deficiency:** The current count of 17, complete ID inventory, selected
ID-to-model mappings, tier-plugin data, and diagnostic default model recur