Files
notarius/docs/roadmap/remediation.md

166 lines
8.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Native Migration Review Remediation
## Scope And Baseline
Review date: 2026-11-22.
Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`.
This review followed the 46 stages in [the implementation plan](implementation.md)
against the [native migration target](promptkit.md), current owning code,
focused tests, maintained examples, and canonical documentation. The migration
is substantially complete. The required follow-up is limited to the two findings
below; neither calls for another compatibility layer or application execution
abstraction. This document records findings, not implemented fixes.
## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
Priority: P2. Relevant stages: 10, 25, 37, and 46.
### Evidence And Impact
[`selectRunError`](../../internal/framework/pipeline/runner_concurrent.go),
lines 398–428 at the reviewed revision, suppresses every error matching
`context.Canceled` or `context.DeadlineExceeded` when any other failure is
available. This retained cancellation filter does not distinguish a canceled
sibling from an independently failed native operation.
A native PromptKit operation timeout matches both
`promptkit.ErrOperationTimeout` and `context.DeadlineExceeded`, even when the
caller context remains active. Consequently, a genuine timeout in an earlier
stage/lane/chunk scope is discarded in favor of a later non-context error,
rather than participating in the runner's stable scope ordering. The returned
cause then loses that native timeout identity.
The producer helper itself handles this distinction correctly; the gap is at
the concurrent runner's final selection boundary. A temporary review test used
the real PromptKit engine with a fake provider, obtained an actual native
operation timeout with an active caller, and supplied it as the earlier of two
scoped failures. The selector returned the later error instead. Existing helper
timeout coverage does not exercise this aggregation boundary.
### Required Change
Refine error selection so genuine operational deadline/timeout failures remain
eligible for the existing stable scope ordering. Distinguish them from
runner-induced sibling cancellation using the relevant context state and native
causes. Preserve authoritative caller cancellation and suppression of derivative
cancellation noise. An active caller's deadline-shaped operation failure must
not be discarded merely because it matches `context.DeadlineExceeded`.
Keep this a small runner correction; no new timeout controls, retry loop, or
application error taxonomy is needed.
### Acceptance
- With an active caller, an earlier-scoped native operation timeout and a later
ordinary failure select the timeout and retain its native `errors.Is` identity.
- A non-caller provider deadline also remains a genuine operational failure.
- Caller cancellation/deadline and runner-induced sibling cancellation retain
their intended precedence/noise handling.
- Ordinary multi-failure ordering and worker cancellation/draining remain intact.
- Add focused behavioral coverage at the runner error-selection boundary, using
the existing native timeout fixture where useful. Run the pipeline tests and
their concurrency/race checks.
## Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee
Priority: P2. Relevant stages: 11, 38, 39, and 42.
### Evidence And Impact
Three current documents make source checking sound unconditional:
- [Architecture policy](../policy/architecture.md), lines 229–232: intrinsic
codec/source correctness still applies with an empty chain.
- [Configuration reference](../config.md), lines 350–356: intrinsic typed
codec/source checks remain with an empty configured chain.
- [Pipeline internals](../internal/pipeline.md), lines 75–77: an empty chain
cannot remove intrinsic source-addressing checks.
Intrinsic durable-codec validation does remain. However, ordinary artifact
checks that cited units exist in the current source or lie within an extraction
chunk are supplied by the configured source-reference validators; final artifact
serialization does not independently perform those checks. See
[`serializeArtifact`](../../internal/framework/pipeline/runner_typed.go) and
the configured [candidate assessor](../../internal/framework/pipeline/candidate_assessment.go).
A temporary offline native-CLI probe configured `dnd/spells` extraction and
`noop` normalization with `validators: []`, disabled cache use, and supplied a
schema-valid spell citation to unit 999 against a transcript containing only
unit 1. The command succeeded and published that citation. This does not show a
failure of the default validator chains: the probe explicitly replaced them.
It demonstrates that the broad documented guarantee is stronger than the
implemented configurable validation boundary.
### Required Change
Align the three documents with the actual, selected boundary. Distinguish:
- intrinsic durable-codec/schema validation;
- source-document validation and chunk-plan materialization;
- checks a particular module necessarily performs while interpreting,
constructing fallback, or decoding a cache hit; and
- artifact citation existence/containment checks supplied by configured
validators.
State that an explicit empty/replacement chain can omit those ordinary artifact
source-reference checks. Link to the canonical default-chain/configuration
description rather than repeating detailed validator inventories.
Preserve operator replacement of validator chains. Do not add a mandatory
artifact validator, prohibit empty chains, or make every codec consult the
source document merely to satisfy the stronger wording. Those would expand the
requested contract. The Stage 11 completion summary's claim of mandatory final
codec/source checks should be qualified consistently while the active plan is
retained.
### Acceptance
- The architecture, configuration reference, pipeline guide, and active plan
distinguish unconditional codec checks from configurable citation validation.
- They do not imply that disabling validators disables source parsing,
materialization, or necessary module/fallback/cache construction checks.
- Existing default chains, explicit chain replacement, and runtime behavior
remain unchanged.
- Review related current documentation for the same overbroad claim. No new
prose-snapshot or mandatory-source-validation test is needed for this
documentation correction.
## Review Coverage And Verification
Owning boundaries were inspected in implementation order:
| Stages | Reviewed boundary |
| --- | --- |
| 1–7 | Declared dependencies/notices, native resources/schemas, one ConfigKit result and engine, CommandKit/ckconfig/ckprompt construction and supplied values. |
| 8–11 | Typed production/assessment, full ordered validator chains, native continuation and terminal disposition, runner handoffs and eligibility. |
| 12–24 | All ten D&D extractors, three semantic registry normalizers, scene planning, direct combat-semantic validation, deterministic paths and registration. |
| 25–28 | Ready-job workers, native admission, accepted-artifact cache, prepared handles/initial keys, current-source cache usability and chunk-plan materialization. |
| 29–34 | Domain-only publication, incremental requested capture, offline preflight/inspection, run composition, native reports and failure outcomes. |
| 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. |
| 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. |
Checks completed during review:
- `GOWORK=off GOPROXY=off go test ./...` passed.
- `GOWORK=off GOPROXY=off go vet ./...` passed.
- A fresh `GOWORK=off GOPROXY=off go build` to a temporary binary passed.
- `GOWORK=off GOPROXY=off go test -race ./internal/framework/pipeline
./internal/cli` passed.
- Offline executable checks passed for version, minimal/complete pipeline
preflight, and native prompt requirements; native diff check returned its
expected differences status (3).
- The migration diff contains no changes to model-facing JSON schemas or D&D
intrinsic codec schema files. Historical ADR decision bodies are retained;
supersession edits are confined to status descriptions.
- Temporary Go-overlay probes reproduced both findings without modifying
repository code or tests. Their deliberately failing assertions are review
evidence, separate from the passing default suite.
No live provider inference was performed. This review does not establish
production model quality or performance. Intentional changes such as the hard
configuration/CLI cutover, native retry ownership, content-only cache identity,
no semantic reassessment on hits, explicit permissive fallback, and native
diagnostic visibility are part of the agreed target, not regressions.