18 KiB
Native Migration Review Remediation
Scope And Baseline
Review date: 2026-11-22.
Reviewed commit: 75444b8ef300ca8efee57d7a2271d21ce258b49c.
Integration follow-up baseline: fd134e85e54c6e0f74aab5c77df5bada7c1b1e6a.
The follow-up baseline adds only the initial review findings; application code
is unchanged between these revisions.
This review followed the 46 stages in the implementation plan against the native migration target, current owning code, focused tests, maintained examples, and canonical documentation. The migration is substantially complete. Findings 1–2 come from the initial review. Findings 3–5 come from a second, stage-ordered pass through the interaction of the native migration with existing domain, validation, and diagnostic behavior. None calls for another compatibility layer or application execution abstraction. This document records findings, not implemented fixes.
Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
Priority: P2. Relevant stages: 10, 25, 37, and 46.
Evidence And Impact
selectRunError,
lines 398–428 at the reviewed revision, suppresses every error matching
context.Canceled or context.DeadlineExceeded when any other failure is
available. This retained cancellation filter does not distinguish a canceled
sibling from an independently failed native operation.
A native PromptKit operation timeout matches both
promptkit.ErrOperationTimeout and context.DeadlineExceeded, even when the
caller context remains active. Consequently, a genuine timeout in an earlier
stage/lane/chunk scope is discarded in favor of a later non-context error,
rather than participating in the runner's stable scope ordering. The returned
cause then loses that native timeout identity.
The producer helper itself handles this distinction correctly; the gap is at the concurrent runner's final selection boundary. A temporary review test used the real PromptKit engine with a fake provider, obtained an actual native operation timeout with an active caller, and supplied it as the earlier of two scoped failures. The selector returned the later error instead. Existing helper timeout coverage does not exercise this aggregation boundary.
Required Change
Refine error selection so genuine operational deadline/timeout failures remain
eligible for the existing stable scope ordering. Distinguish them from
runner-induced sibling cancellation using the relevant context state and native
causes. Preserve authoritative caller cancellation and suppression of derivative
cancellation noise. An active caller's deadline-shaped operation failure must
not be discarded merely because it matches context.DeadlineExceeded.
Keep this a small runner correction; no new timeout controls, retry loop, or application error taxonomy is needed.
Acceptance
- With an active caller, an earlier-scoped native operation timeout and a later
ordinary failure select the timeout and retain its native
errors.Isidentity. - A non-caller provider deadline also remains a genuine operational failure.
- Caller cancellation/deadline and runner-induced sibling cancellation retain their intended precedence/noise handling.
- Ordinary multi-failure ordering and worker cancellation/draining remain intact.
- Add focused behavioral coverage at the runner error-selection boundary, using the existing native timeout fixture where useful. Run the pipeline tests and their concurrency/race checks.
Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee
Priority: P2. Relevant stages: 11, 38, 39, and 42.
Evidence And Impact
Three current documents make source checking sound unconditional:
- Architecture policy, lines 229–232: intrinsic codec/source correctness still applies with an empty chain.
- Configuration reference, lines 350–356: intrinsic typed codec/source checks remain with an empty configured chain.
- Pipeline internals, lines 75–77: an empty chain cannot remove intrinsic source-addressing checks.
Intrinsic durable-codec validation does remain. However, ordinary artifact
checks that cited units exist in the current source or lie within an extraction
chunk are supplied by the configured source-reference validators; final artifact
serialization does not independently perform those checks. See
serializeArtifact and
the configured candidate assessor.
A temporary offline native-CLI probe configured dnd/spells extraction and
noop normalization with validators: [], disabled cache use, and supplied a
schema-valid spell citation to unit 999 against a transcript containing only
unit 1. The command succeeded and published that citation. This does not show a
failure of the default validator chains: the probe explicitly replaced them.
It demonstrates that the broad documented guarantee is stronger than the
implemented configurable validation boundary.
Required Change
Align the three documents with the actual, selected boundary. Distinguish:
- intrinsic durable-codec/schema validation;
- source-document validation and chunk-plan materialization;
- checks a particular module necessarily performs while interpreting, constructing fallback, or decoding a cache hit; and
- artifact citation existence/containment checks supplied by configured validators.
State that an explicit empty/replacement chain can omit those ordinary artifact source-reference checks. Link to the canonical default-chain/configuration description rather than repeating detailed validator inventories.
Preserve operator replacement of validator chains. Do not add a mandatory artifact validator, prohibit empty chains, or make every codec consult the source document merely to satisfy the stronger wording. Those would expand the requested contract. The Stage 11 completion summary's claim of mandatory final codec/source checks should be qualified consistently while the active plan is retained.
Acceptance
- The architecture, configuration reference, pipeline guide, and active plan distinguish unconditional codec checks from configurable citation validation.
- They do not imply that disabling validators disables source parsing, materialization, or necessary module/fallback/cache construction checks.
- Existing default chains, explicit chain replacement, and runtime behavior remain unchanged.
- Review related current documentation for the same overbroad claim. No new prose-snapshot or mandatory-source-validation test is needed for this documentation correction.
Finding 3: Rebind Cached Scene Identity To The Current Chunk Plan
Priority: P1. Relevant stages: 11, 23, 27–29, and 37.
Evidence And Impact
The new message-content cache interacts incorrectly with the existing position-derived scene identity:
MaterializeChunkPlan, lines 169–179, renders only source units into chunk content, while assigningchunk-000001, etc. from the range's position in the plan.- Scene extraction
deterministically attaches the current chunk ID and range in
mapResponse. The model does not supply this identity, and the ID is not rendered into the initial scene-description messages. artifactProducerCache, lines 21–37, decodes a hit and checks cited source units, but does not rebind its scene identity. The hit bypasses extraction interpretation and assessment.
Changing an earlier scene boundary can therefore renumber an unchanged later scene without changing that later scene's prompt hash. Its old accepted artifact is a legitimate content-key hit, but its stored scene ID belongs to the old plan.
A temporary native-CLI integration probe used real scene planning, extraction,
merging, normalization, default validators, and a filesystem cache. The chunk
binding used cache: refresh; scene extraction used ordinary auto. The fake
provider returned these two valid plans for the same three-unit transcript:
[1–2], [3], followed by[1], [2], [3]: the second run failed normalization withscene ID "chunk-000002" has conflicting records.- The reverse change: the second run succeeded but published IDs
chunk-000001andchunk-000003for a two-chunk plan. The last ID should have beenchunk-000002.
Both cases pass with invocation-wide cache bypass. The unchanged suffix was reused in the failing cases, as confirmed by provider-call counts. This is not an invalid generated plan or intentionally skipped semantic revalidation.
The stale ID also conflicts with the existing scene registry's exact match: combat/enemy consumers require both the current chunk ID and range, so a stale scene can become unavailable classification and suppress downstream extraction.
Required Change
Add the missing current-operation adaptation at the scene cache-hit boundary. For a usable single-scene artifact describing the exact current range, deterministically reattach the current chunk identity while retaining its accepted kind/title/summary. If a cached artifact cannot be safely adapted, treat it as an unusable miss before returning it as accepted work.
Keep domain-specific adaptation with the scene module, using the supplied typed cache collaboration or a small extension to that boundary if needed. Preserve the content-only native cache key and no-semantic-reassessment policy. Do not render an internal scene ID into the prompt, add a partition fingerprint/key dimension, restore canonical source-only chunking, or ask the model to copy IDs. Review other current cached artifacts for equivalent operation-derived fields; do not invent a generic identity framework for families that do not need one.
Acceptance
- Repartitioning earlier units can move an unchanged later scene to a different chunk ordinal without duplicate or stale scene IDs in the published artifact.
- Current scene IDs/ranges match the chunk map and remain usable by the existing combat/enemy eligibility resolver.
- The unchanged scene can reuse its accepted semantic content without another model call or validator assessment when deterministic adaptation is safe.
- Wrong-range, multi-scene, or otherwise unusable entries are not silently rebound into valid-looking scenes; they miss normally.
- Add assembled two-run expansion/collapse coverage using the existing native fake-provider/cache setup, and retain ordinary same-plan hit coverage.
Finding 4: Bound Framework-Generated Fallback Diagnostic Messages
Priority: P2. Relevant stages: 9, 11, 24, 29, and 34.
Evidence And Impact
FinishProducerFailure
retains the complete failure text in ProducerOutcome.FallbackReason after a
supported permissive fallback is successfully constructed.
operationDiagnosticGroups,
lines 51–58, copies that text unchanged into a framework-generated warning sample.
The existing diagnostic contract
requires valid UTF-8 and a maximum sample-message size of 4096 bytes.
Native errors and joined assessment errors are not bounded diagnostic samples. Even individually bounded domain validator messages can exceed the sample limit after joining several findings and adding validator context. The new fallback bridge thus creates invalid diagnostics from otherwise valid operation outcomes.
A temporary offline native-CLI probe ran a minimal spell pipeline in permissive
mode with a fake provider returning a long ordinary error. The valid empty spell
fallback was constructed, but the run then failed with
producer diagnostic 0: diagnostic sample 0: diagnostic sample message exceeds maximum length.
The failure occurs even with ordinary visibility hiding optional warnings.
Required Change
Render the framework's fallback warning into a bounded, valid-UTF-8 sample before
promoting it. Reuse or factor the small bounding behavior already present in
producerCacheObserver
rather than adding a second diagnostic system.
Bound only the generated diagnostic sample, leaving the original error/outcome text and requested evidence unchanged. This is operator diagnostic presentation, not model correction guidance. Preserve validation of module-supplied diagnostics; do not raise/remove the existing cap or suppress genuine malformed-module contracts to work around the framework's own unbounded sample.
Acceptance
- A supported permissive fallback still publishes qualified output when its native or aggregated-assessment cause exceeds the diagnostic sample limit.
- The generated warning is nonblank, bounded, and valid UTF-8, including at a multibyte truncation boundary.
- Strict handling, caller cancellation, fallback counting/reuse exclusion, full requested evidence, and module-diagnostic validation remain unchanged.
- Add focused fallback-to-diagnostic integration coverage, including a real typed producer path with a fake long native error. No model retry is added.
Finding 5: Remove The Disconnected Legacy LLM Error Contract
Priority: P3. Relevant stages: 8, 24, 35, and 44.
Evidence And Impact
contracts/errors.go still
defines the pre-migration ErrInvalidStructuredOutput, ErrLLMCapacityExceeded,
ErrLLMGeneration, LLMGenerationError, constructors, and copied failure/attempt
metadata. Current producers and validators retain native PromptKit errors instead.
Graph/code searches find this old taxonomy only in that file and its
self-test, not in an active
application consumer.
This is leftover contract surface from the removed adapter, not a second active execution path. It nevertheless presents obsolete sentinels and metadata accessors as available framework conventions, although matching actual native failures against them no longer works. Stage 35's hard-cutover cleanup is incomplete here.
Required Change And Acceptance
Remove the unused legacy error definitions and their self-test. Keep native
errors.Is/errors.As behavior and the separate, actively used application
assessment/capture error types. Do not bridge native errors into these retired
sentinels or add aliases for compatibility. Verify no current callers remain and
run contracts, pipeline, CLI, and module integration tests after removal.
Review Coverage And Verification
Owning boundaries were inspected in implementation order:
| Stages | Reviewed boundary |
|---|---|
| 1–7 | Declared dependencies/notices, native resources/schemas, one ConfigKit result and engine, CommandKit/ckconfig/ckprompt construction and supplied values. |
| 8–11 | Typed production/assessment, full ordered validator chains, native continuation and terminal disposition, runner handoffs and eligibility. |
| 12–24 | All ten D&D extractors, three semantic registry normalizers, scene planning, direct combat-semantic validation, deterministic paths and registration. |
| 25–28 | Ready-job workers, native admission, accepted-artifact cache, prepared handles/initial keys, current-source cache usability and chunk-plan materialization. |
| 29–34 | Domain-only publication, incremental requested capture, offline preflight/inspection, run composition, native reports and failure outcomes. |
| 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. |
| 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. |
Checks completed during review:
GOWORK=off GOPROXY=off go test ./...passed.GOWORK=off GOPROXY=off go vet ./...passed.- A fresh
GOWORK=off GOPROXY=off go buildto a temporary binary passed. GOWORK=off GOPROXY=off go test -race ./internal/framework/pipeline ./internal/clipassed.- Offline executable checks passed for version, minimal/complete pipeline preflight, and native prompt requirements; native diff check returned its expected differences status (3).
- The migration diff contains no changes to model-facing JSON schemas or D&D intrinsic codec schema files. Historical ADR decision bodies are retained; supersession edits are confined to status descriptions.
- Temporary Go-overlay probes reproduced both findings without modifying repository code or tests. Their deliberately failing assertions are review evidence, separate from the passing default suite.
The integration follow-up inspected the stage boundaries above against the retained source/chunk identity, domain codecs/validators, generated references, diagnostic limits, physical publication, and subprocess contracts. It reused the earlier full-suite/vet/build/race results only after confirming that the application code and dependencies were unchanged. Additional checks were:
GOWORK=off GOPROXY=off go test ./internal/core/config ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/integration ./internal/clipassed.- Real native-CLI/fake-provider Go-overlay probes reproduced both directions of scene repartitioning in Finding 3. Both corresponding bypass controls passed.
- A native-CLI/fake-provider overlay probe reproduced the long-error permissive fallback failure in Finding 4.
- Graph/code searches confirmed that Finding 5's legacy taxonomy has no active callers outside its own definition/self-test.
These probes live only in a temporary review directory. No production code, test, configuration, schema, or existing canonical documentation was changed.
No live provider inference was performed. This review does not establish production model quality or performance. Intentional changes such as the hard configuration/CLI cutover, native retry ownership, content-only cache identity, no semantic reassessment on hits, explicit permissive fallback, and native diagnostic visibility are part of the agreed target, not regressions.