diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md new file mode 100644 index 00000000..1f767ef9 --- /dev/null +++ b/docs/roadmap/remediation.md @@ -0,0 +1,165 @@ +# Native Migration Review Remediation + +## Scope And Baseline + +Review date: 2026-11-22. + +Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`. + +This review followed the 46 stages in [the implementation plan](implementation.md) +against the [native migration target](promptkit.md), current owning code, +focused tests, maintained examples, and canonical documentation. The migration +is substantially complete. The required follow-up is limited to the two findings +below; neither calls for another compatibility layer or application execution +abstraction. This document records findings, not implemented fixes. + +## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection + +Priority: P2. Relevant stages: 10, 25, 37, and 46. + +### Evidence And Impact + +[`selectRunError`](../../internal/framework/pipeline/runner_concurrent.go), +lines 398–428 at the reviewed revision, suppresses every error matching +`context.Canceled` or `context.DeadlineExceeded` when any other failure is +available. This retained cancellation filter does not distinguish a canceled +sibling from an independently failed native operation. + +A native PromptKit operation timeout matches both +`promptkit.ErrOperationTimeout` and `context.DeadlineExceeded`, even when the +caller context remains active. Consequently, a genuine timeout in an earlier +stage/lane/chunk scope is discarded in favor of a later non-context error, +rather than participating in the runner's stable scope ordering. The returned +cause then loses that native timeout identity. + +The producer helper itself handles this distinction correctly; the gap is at +the concurrent runner's final selection boundary. A temporary review test used +the real PromptKit engine with a fake provider, obtained an actual native +operation timeout with an active caller, and supplied it as the earlier of two +scoped failures. The selector returned the later error instead. Existing helper +timeout coverage does not exercise this aggregation boundary. + +### Required Change + +Refine error selection so genuine operational deadline/timeout failures remain +eligible for the existing stable scope ordering. Distinguish them from +runner-induced sibling cancellation using the relevant context state and native +causes. Preserve authoritative caller cancellation and suppression of derivative +cancellation noise. An active caller's deadline-shaped operation failure must +not be discarded merely because it matches `context.DeadlineExceeded`. + +Keep this a small runner correction; no new timeout controls, retry loop, or +application error taxonomy is needed. + +### Acceptance + +- With an active caller, an earlier-scoped native operation timeout and a later + ordinary failure select the timeout and retain its native `errors.Is` identity. +- A non-caller provider deadline also remains a genuine operational failure. +- Caller cancellation/deadline and runner-induced sibling cancellation retain + their intended precedence/noise handling. +- Ordinary multi-failure ordering and worker cancellation/draining remain intact. +- Add focused behavioral coverage at the runner error-selection boundary, using + the existing native timeout fixture where useful. Run the pipeline tests and + their concurrency/race checks. + +## Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee + +Priority: P2. Relevant stages: 11, 38, 39, and 42. + +### Evidence And Impact + +Three current documents make source checking sound unconditional: + +- [Architecture policy](../policy/architecture.md), lines 229–232: intrinsic + codec/source correctness still applies with an empty chain. +- [Configuration reference](../config.md), lines 350–356: intrinsic typed + codec/source checks remain with an empty configured chain. +- [Pipeline internals](../internal/pipeline.md), lines 75–77: an empty chain + cannot remove intrinsic source-addressing checks. + +Intrinsic durable-codec validation does remain. However, ordinary artifact +checks that cited units exist in the current source or lie within an extraction +chunk are supplied by the configured source-reference validators; final artifact +serialization does not independently perform those checks. See +[`serializeArtifact`](../../internal/framework/pipeline/runner_typed.go) and +the configured [candidate assessor](../../internal/framework/pipeline/candidate_assessment.go). + +A temporary offline native-CLI probe configured `dnd/spells` extraction and +`noop` normalization with `validators: []`, disabled cache use, and supplied a +schema-valid spell citation to unit 999 against a transcript containing only +unit 1. The command succeeded and published that citation. This does not show a +failure of the default validator chains: the probe explicitly replaced them. +It demonstrates that the broad documented guarantee is stronger than the +implemented configurable validation boundary. + +### Required Change + +Align the three documents with the actual, selected boundary. Distinguish: + +- intrinsic durable-codec/schema validation; +- source-document validation and chunk-plan materialization; +- checks a particular module necessarily performs while interpreting, + constructing fallback, or decoding a cache hit; and +- artifact citation existence/containment checks supplied by configured + validators. + +State that an explicit empty/replacement chain can omit those ordinary artifact +source-reference checks. Link to the canonical default-chain/configuration +description rather than repeating detailed validator inventories. + +Preserve operator replacement of validator chains. Do not add a mandatory +artifact validator, prohibit empty chains, or make every codec consult the +source document merely to satisfy the stronger wording. Those would expand the +requested contract. The Stage 11 completion summary's claim of mandatory final +codec/source checks should be qualified consistently while the active plan is +retained. + +### Acceptance + +- The architecture, configuration reference, pipeline guide, and active plan + distinguish unconditional codec checks from configurable citation validation. +- They do not imply that disabling validators disables source parsing, + materialization, or necessary module/fallback/cache construction checks. +- Existing default chains, explicit chain replacement, and runtime behavior + remain unchanged. +- Review related current documentation for the same overbroad claim. No new + prose-snapshot or mandatory-source-validation test is needed for this + documentation correction. + +## Review Coverage And Verification + +Owning boundaries were inspected in implementation order: + +| Stages | Reviewed boundary | +| --- | --- | +| 1–7 | Declared dependencies/notices, native resources/schemas, one ConfigKit result and engine, CommandKit/ckconfig/ckprompt construction and supplied values. | +| 8–11 | Typed production/assessment, full ordered validator chains, native continuation and terminal disposition, runner handoffs and eligibility. | +| 12–24 | All ten D&D extractors, three semantic registry normalizers, scene planning, direct combat-semantic validation, deterministic paths and registration. | +| 25–28 | Ready-job workers, native admission, accepted-artifact cache, prepared handles/initial keys, current-source cache usability and chunk-plan materialization. | +| 29–34 | Domain-only publication, incremental requested capture, offline preflight/inspection, run composition, native reports and failure outcomes. | +| 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. | +| 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. | + +Checks completed during review: + +- `GOWORK=off GOPROXY=off go test ./...` passed. +- `GOWORK=off GOPROXY=off go vet ./...` passed. +- A fresh `GOWORK=off GOPROXY=off go build` to a temporary binary passed. +- `GOWORK=off GOPROXY=off go test -race ./internal/framework/pipeline + ./internal/cli` passed. +- Offline executable checks passed for version, minimal/complete pipeline + preflight, and native prompt requirements; native diff check returned its + expected differences status (3). +- The migration diff contains no changes to model-facing JSON schemas or D&D + intrinsic codec schema files. Historical ADR decision bodies are retained; + supersession edits are confined to status descriptions. +- Temporary Go-overlay probes reproduced both findings without modifying + repository code or tests. Their deliberately failing assertions are review + evidence, separate from the passing default suite. + +No live provider inference was performed. This review does not establish +production model quality or performance. Intentional changes such as the hard +configuration/CLI cutover, native retry ownership, content-only cache identity, +no semantic reassessment on hits, explicit permissive fallback, and native +diagnostic visibility are part of the agreed target, not regressions.