Record the native migration review findings
This commit is contained in:
165
docs/roadmap/remediation.md
Normal file
165
docs/roadmap/remediation.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# Native Migration Review Remediation
|
||||
|
||||
## Scope And Baseline
|
||||
|
||||
Review date: 2026-11-22.
|
||||
|
||||
Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`.
|
||||
|
||||
This review followed the 46 stages in [the implementation plan](implementation.md)
|
||||
against the [native migration target](promptkit.md), current owning code,
|
||||
focused tests, maintained examples, and canonical documentation. The migration
|
||||
is substantially complete. The required follow-up is limited to the two findings
|
||||
below; neither calls for another compatibility layer or application execution
|
||||
abstraction. This document records findings, not implemented fixes.
|
||||
|
||||
## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
|
||||
|
||||
Priority: P2. Relevant stages: 10, 25, 37, and 46.
|
||||
|
||||
### Evidence And Impact
|
||||
|
||||
[`selectRunError`](../../internal/framework/pipeline/runner_concurrent.go),
|
||||
lines 398–428 at the reviewed revision, suppresses every error matching
|
||||
`context.Canceled` or `context.DeadlineExceeded` when any other failure is
|
||||
available. This retained cancellation filter does not distinguish a canceled
|
||||
sibling from an independently failed native operation.
|
||||
|
||||
A native PromptKit operation timeout matches both
|
||||
`promptkit.ErrOperationTimeout` and `context.DeadlineExceeded`, even when the
|
||||
caller context remains active. Consequently, a genuine timeout in an earlier
|
||||
stage/lane/chunk scope is discarded in favor of a later non-context error,
|
||||
rather than participating in the runner's stable scope ordering. The returned
|
||||
cause then loses that native timeout identity.
|
||||
|
||||
The producer helper itself handles this distinction correctly; the gap is at
|
||||
the concurrent runner's final selection boundary. A temporary review test used
|
||||
the real PromptKit engine with a fake provider, obtained an actual native
|
||||
operation timeout with an active caller, and supplied it as the earlier of two
|
||||
scoped failures. The selector returned the later error instead. Existing helper
|
||||
timeout coverage does not exercise this aggregation boundary.
|
||||
|
||||
### Required Change
|
||||
|
||||
Refine error selection so genuine operational deadline/timeout failures remain
|
||||
eligible for the existing stable scope ordering. Distinguish them from
|
||||
runner-induced sibling cancellation using the relevant context state and native
|
||||
causes. Preserve authoritative caller cancellation and suppression of derivative
|
||||
cancellation noise. An active caller's deadline-shaped operation failure must
|
||||
not be discarded merely because it matches `context.DeadlineExceeded`.
|
||||
|
||||
Keep this a small runner correction; no new timeout controls, retry loop, or
|
||||
application error taxonomy is needed.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- With an active caller, an earlier-scoped native operation timeout and a later
|
||||
ordinary failure select the timeout and retain its native `errors.Is` identity.
|
||||
- A non-caller provider deadline also remains a genuine operational failure.
|
||||
- Caller cancellation/deadline and runner-induced sibling cancellation retain
|
||||
their intended precedence/noise handling.
|
||||
- Ordinary multi-failure ordering and worker cancellation/draining remain intact.
|
||||
- Add focused behavioral coverage at the runner error-selection boundary, using
|
||||
the existing native timeout fixture where useful. Run the pipeline tests and
|
||||
their concurrency/race checks.
|
||||
|
||||
## Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee
|
||||
|
||||
Priority: P2. Relevant stages: 11, 38, 39, and 42.
|
||||
|
||||
### Evidence And Impact
|
||||
|
||||
Three current documents make source checking sound unconditional:
|
||||
|
||||
- [Architecture policy](../policy/architecture.md), lines 229–232: intrinsic
|
||||
codec/source correctness still applies with an empty chain.
|
||||
- [Configuration reference](../config.md), lines 350–356: intrinsic typed
|
||||
codec/source checks remain with an empty configured chain.
|
||||
- [Pipeline internals](../internal/pipeline.md), lines 75–77: an empty chain
|
||||
cannot remove intrinsic source-addressing checks.
|
||||
|
||||
Intrinsic durable-codec validation does remain. However, ordinary artifact
|
||||
checks that cited units exist in the current source or lie within an extraction
|
||||
chunk are supplied by the configured source-reference validators; final artifact
|
||||
serialization does not independently perform those checks. See
|
||||
[`serializeArtifact`](../../internal/framework/pipeline/runner_typed.go) and
|
||||
the configured [candidate assessor](../../internal/framework/pipeline/candidate_assessment.go).
|
||||
|
||||
A temporary offline native-CLI probe configured `dnd/spells` extraction and
|
||||
`noop` normalization with `validators: []`, disabled cache use, and supplied a
|
||||
schema-valid spell citation to unit 999 against a transcript containing only
|
||||
unit 1. The command succeeded and published that citation. This does not show a
|
||||
failure of the default validator chains: the probe explicitly replaced them.
|
||||
It demonstrates that the broad documented guarantee is stronger than the
|
||||
implemented configurable validation boundary.
|
||||
|
||||
### Required Change
|
||||
|
||||
Align the three documents with the actual, selected boundary. Distinguish:
|
||||
|
||||
- intrinsic durable-codec/schema validation;
|
||||
- source-document validation and chunk-plan materialization;
|
||||
- checks a particular module necessarily performs while interpreting,
|
||||
constructing fallback, or decoding a cache hit; and
|
||||
- artifact citation existence/containment checks supplied by configured
|
||||
validators.
|
||||
|
||||
State that an explicit empty/replacement chain can omit those ordinary artifact
|
||||
source-reference checks. Link to the canonical default-chain/configuration
|
||||
description rather than repeating detailed validator inventories.
|
||||
|
||||
Preserve operator replacement of validator chains. Do not add a mandatory
|
||||
artifact validator, prohibit empty chains, or make every codec consult the
|
||||
source document merely to satisfy the stronger wording. Those would expand the
|
||||
requested contract. The Stage 11 completion summary's claim of mandatory final
|
||||
codec/source checks should be qualified consistently while the active plan is
|
||||
retained.
|
||||
|
||||
### Acceptance
|
||||
|
||||
- The architecture, configuration reference, pipeline guide, and active plan
|
||||
distinguish unconditional codec checks from configurable citation validation.
|
||||
- They do not imply that disabling validators disables source parsing,
|
||||
materialization, or necessary module/fallback/cache construction checks.
|
||||
- Existing default chains, explicit chain replacement, and runtime behavior
|
||||
remain unchanged.
|
||||
- Review related current documentation for the same overbroad claim. No new
|
||||
prose-snapshot or mandatory-source-validation test is needed for this
|
||||
documentation correction.
|
||||
|
||||
## Review Coverage And Verification
|
||||
|
||||
Owning boundaries were inspected in implementation order:
|
||||
|
||||
| Stages | Reviewed boundary |
|
||||
| --- | --- |
|
||||
| 1–7 | Declared dependencies/notices, native resources/schemas, one ConfigKit result and engine, CommandKit/ckconfig/ckprompt construction and supplied values. |
|
||||
| 8–11 | Typed production/assessment, full ordered validator chains, native continuation and terminal disposition, runner handoffs and eligibility. |
|
||||
| 12–24 | All ten D&D extractors, three semantic registry normalizers, scene planning, direct combat-semantic validation, deterministic paths and registration. |
|
||||
| 25–28 | Ready-job workers, native admission, accepted-artifact cache, prepared handles/initial keys, current-source cache usability and chunk-plan materialization. |
|
||||
| 29–34 | Domain-only publication, incremental requested capture, offline preflight/inspection, run composition, native reports and failure outcomes. |
|
||||
| 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. |
|
||||
| 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. |
|
||||
|
||||
Checks completed during review:
|
||||
|
||||
- `GOWORK=off GOPROXY=off go test ./...` passed.
|
||||
- `GOWORK=off GOPROXY=off go vet ./...` passed.
|
||||
- A fresh `GOWORK=off GOPROXY=off go build` to a temporary binary passed.
|
||||
- `GOWORK=off GOPROXY=off go test -race ./internal/framework/pipeline
|
||||
./internal/cli` passed.
|
||||
- Offline executable checks passed for version, minimal/complete pipeline
|
||||
preflight, and native prompt requirements; native diff check returned its
|
||||
expected differences status (3).
|
||||
- The migration diff contains no changes to model-facing JSON schemas or D&D
|
||||
intrinsic codec schema files. Historical ADR decision bodies are retained;
|
||||
supersession edits are confined to status descriptions.
|
||||
- Temporary Go-overlay probes reproduced both findings without modifying
|
||||
repository code or tests. Their deliberately failing assertions are review
|
||||
evidence, separate from the passing default suite.
|
||||
|
||||
No live provider inference was performed. This review does not establish
|
||||
production model quality or performance. Intentional changes such as the hard
|
||||
configuration/CLI cutover, native retry ownership, content-only cache identity,
|
||||
no semantic reassessment on hits, explicit permissive fallback, and native
|
||||
diagnostic visibility are part of the agreed target, not regressions.
|
||||
Reference in New Issue
Block a user