Record native migration integration gaps
This commit is contained in:
@@ -6,12 +6,18 @@ Review date: 2026-11-22.
|
|||||||
|
|
||||||
Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`.
|
Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`.
|
||||||
|
|
||||||
|
Integration follow-up baseline: `fd134e85e54c6e0f74aab5c77df5bada7c1b1e6a`.
|
||||||
|
The follow-up baseline adds only the initial review findings; application code
|
||||||
|
is unchanged between these revisions.
|
||||||
|
|
||||||
This review followed the 46 stages in [the implementation plan](implementation.md)
|
This review followed the 46 stages in [the implementation plan](implementation.md)
|
||||||
against the [native migration target](promptkit.md), current owning code,
|
against the [native migration target](promptkit.md), current owning code,
|
||||||
focused tests, maintained examples, and canonical documentation. The migration
|
focused tests, maintained examples, and canonical documentation. The migration
|
||||||
is substantially complete. The required follow-up is limited to the two findings
|
is substantially complete. Findings 1–2 come from the initial review. Findings
|
||||||
below; neither calls for another compatibility layer or application execution
|
3–5 come from a second, stage-ordered pass through the interaction of the native
|
||||||
abstraction. This document records findings, not implemented fixes.
|
migration with existing domain, validation, and diagnostic behavior. None calls
|
||||||
|
for another compatibility layer or application execution abstraction. This
|
||||||
|
document records findings, not implemented fixes.
|
||||||
|
|
||||||
## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
|
## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
|
||||||
|
|
||||||
@@ -127,6 +133,154 @@ retained.
|
|||||||
prose-snapshot or mandatory-source-validation test is needed for this
|
prose-snapshot or mandatory-source-validation test is needed for this
|
||||||
documentation correction.
|
documentation correction.
|
||||||
|
|
||||||
|
## Finding 3: Rebind Cached Scene Identity To The Current Chunk Plan
|
||||||
|
|
||||||
|
Priority: P1. Relevant stages: 11, 23, 27–29, and 37.
|
||||||
|
|
||||||
|
### Evidence And Impact
|
||||||
|
|
||||||
|
The new message-content cache interacts incorrectly with the existing
|
||||||
|
position-derived scene identity:
|
||||||
|
|
||||||
|
- [`MaterializeChunkPlan`](../../internal/core/source/chunk_plan.go), lines
|
||||||
|
169–179, renders only source units into chunk content, while assigning
|
||||||
|
`chunk-000001`, etc. from the range's position in the plan.
|
||||||
|
- [Scene extraction](../../internal/modules/dnd/extract/scenedescriptions/extractor.go)
|
||||||
|
deterministically attaches the current chunk ID and range in `mapResponse`.
|
||||||
|
The model does not supply this identity, and the ID is not rendered into the
|
||||||
|
initial scene-description messages.
|
||||||
|
- [`artifactProducerCache`](../../internal/framework/pipeline/runner_cache.go),
|
||||||
|
lines 21–37, decodes a hit and checks cited source units, but does not rebind
|
||||||
|
its scene identity. The hit bypasses extraction interpretation and assessment.
|
||||||
|
|
||||||
|
Changing an earlier scene boundary can therefore renumber an unchanged later
|
||||||
|
scene without changing that later scene's prompt hash. Its old accepted artifact
|
||||||
|
is a legitimate content-key hit, but its stored scene ID belongs to the old plan.
|
||||||
|
|
||||||
|
A temporary native-CLI integration probe used real scene planning, extraction,
|
||||||
|
merging, normalization, default validators, and a filesystem cache. The chunk
|
||||||
|
binding used `cache: refresh`; scene extraction used ordinary `auto`. The fake
|
||||||
|
provider returned these two valid plans for the same three-unit transcript:
|
||||||
|
|
||||||
|
1. `[1–2], [3]`, followed by `[1], [2], [3]`: the second run failed normalization
|
||||||
|
with `scene ID "chunk-000002" has conflicting records`.
|
||||||
|
2. The reverse change: the second run succeeded but published IDs
|
||||||
|
`chunk-000001` and `chunk-000003` for a two-chunk plan. The last ID should have
|
||||||
|
been `chunk-000002`.
|
||||||
|
|
||||||
|
Both cases pass with invocation-wide cache bypass. The unchanged suffix was
|
||||||
|
reused in the failing cases, as confirmed by provider-call counts. This is not
|
||||||
|
an invalid generated plan or intentionally skipped semantic revalidation.
|
||||||
|
|
||||||
|
The stale ID also conflicts with the existing [scene registry's exact match](../../internal/modules/dnd/scenedescriptions/registry/registry.go):
|
||||||
|
combat/enemy consumers require both the current chunk ID and range, so a stale
|
||||||
|
scene can become unavailable classification and suppress downstream extraction.
|
||||||
|
|
||||||
|
### Required Change
|
||||||
|
|
||||||
|
Add the missing current-operation adaptation at the scene cache-hit boundary.
|
||||||
|
For a usable single-scene artifact describing the exact current range,
|
||||||
|
deterministically reattach the current chunk identity while retaining its accepted
|
||||||
|
kind/title/summary. If a cached artifact cannot be safely adapted, treat it as an
|
||||||
|
unusable miss before returning it as accepted work.
|
||||||
|
|
||||||
|
Keep domain-specific adaptation with the scene module, using the supplied typed
|
||||||
|
cache collaboration or a small extension to that boundary if needed. Preserve
|
||||||
|
the content-only native cache key and no-semantic-reassessment policy. Do not
|
||||||
|
render an internal scene ID into the prompt, add a partition fingerprint/key
|
||||||
|
dimension, restore canonical source-only chunking, or ask the model to copy IDs.
|
||||||
|
Review other current cached artifacts for equivalent operation-derived fields;
|
||||||
|
do not invent a generic identity framework for families that do not need one.
|
||||||
|
|
||||||
|
### Acceptance
|
||||||
|
|
||||||
|
- Repartitioning earlier units can move an unchanged later scene to a different
|
||||||
|
chunk ordinal without duplicate or stale scene IDs in the published artifact.
|
||||||
|
- Current scene IDs/ranges match the chunk map and remain usable by the existing
|
||||||
|
combat/enemy eligibility resolver.
|
||||||
|
- The unchanged scene can reuse its accepted semantic content without another
|
||||||
|
model call or validator assessment when deterministic adaptation is safe.
|
||||||
|
- Wrong-range, multi-scene, or otherwise unusable entries are not silently
|
||||||
|
rebound into valid-looking scenes; they miss normally.
|
||||||
|
- Add assembled two-run expansion/collapse coverage using the existing native
|
||||||
|
fake-provider/cache setup, and retain ordinary same-plan hit coverage.
|
||||||
|
|
||||||
|
## Finding 4: Bound Framework-Generated Fallback Diagnostic Messages
|
||||||
|
|
||||||
|
Priority: P2. Relevant stages: 9, 11, 24, 29, and 34.
|
||||||
|
|
||||||
|
### Evidence And Impact
|
||||||
|
|
||||||
|
[`FinishProducerFailure`](../../internal/framework/pipeline/producer_outcome.go)
|
||||||
|
retains the complete failure text in `ProducerOutcome.FallbackReason` after a
|
||||||
|
supported permissive fallback is successfully constructed.
|
||||||
|
[`operationDiagnosticGroups`](../../internal/framework/pipeline/runner_operation.go),
|
||||||
|
lines 51–58, copies that text unchanged into a framework-generated warning sample.
|
||||||
|
The existing [diagnostic contract](../../internal/framework/contracts/diagnostics.go)
|
||||||
|
requires valid UTF-8 and a maximum sample-message size of 4096 bytes.
|
||||||
|
|
||||||
|
Native errors and joined assessment errors are not bounded diagnostic samples.
|
||||||
|
Even individually bounded domain validator messages can exceed the sample limit
|
||||||
|
after joining several findings and adding validator context. The new fallback
|
||||||
|
bridge thus creates invalid diagnostics from otherwise valid operation outcomes.
|
||||||
|
|
||||||
|
A temporary offline native-CLI probe ran a minimal spell pipeline in permissive
|
||||||
|
mode with a fake provider returning a long ordinary error. The valid empty spell
|
||||||
|
fallback was constructed, but the run then failed with
|
||||||
|
`producer diagnostic 0: diagnostic sample 0: diagnostic sample message exceeds maximum length`.
|
||||||
|
The failure occurs even with ordinary visibility hiding optional warnings.
|
||||||
|
|
||||||
|
### Required Change
|
||||||
|
|
||||||
|
Render the framework's fallback warning into a bounded, valid-UTF-8 sample before
|
||||||
|
promoting it. Reuse or factor the small bounding behavior already present in
|
||||||
|
[`producerCacheObserver`](../../internal/framework/pipeline/runner_cache.go)
|
||||||
|
rather than adding a second diagnostic system.
|
||||||
|
|
||||||
|
Bound only the generated diagnostic sample, leaving the original error/outcome
|
||||||
|
text and requested evidence unchanged. This is operator diagnostic presentation,
|
||||||
|
not model correction guidance. Preserve validation of module-supplied diagnostics;
|
||||||
|
do not raise/remove the existing cap or suppress genuine malformed-module
|
||||||
|
contracts to work around the framework's own unbounded sample.
|
||||||
|
|
||||||
|
### Acceptance
|
||||||
|
|
||||||
|
- A supported permissive fallback still publishes qualified output when its
|
||||||
|
native or aggregated-assessment cause exceeds the diagnostic sample limit.
|
||||||
|
- The generated warning is nonblank, bounded, and valid UTF-8, including at a
|
||||||
|
multibyte truncation boundary.
|
||||||
|
- Strict handling, caller cancellation, fallback counting/reuse exclusion,
|
||||||
|
full requested evidence, and module-diagnostic validation remain unchanged.
|
||||||
|
- Add focused fallback-to-diagnostic integration coverage, including a real
|
||||||
|
typed producer path with a fake long native error. No model retry is added.
|
||||||
|
|
||||||
|
## Finding 5: Remove The Disconnected Legacy LLM Error Contract
|
||||||
|
|
||||||
|
Priority: P3. Relevant stages: 8, 24, 35, and 44.
|
||||||
|
|
||||||
|
### Evidence And Impact
|
||||||
|
|
||||||
|
[`contracts/errors.go`](../../internal/framework/contracts/errors.go) still
|
||||||
|
defines the pre-migration `ErrInvalidStructuredOutput`, `ErrLLMCapacityExceeded`,
|
||||||
|
`ErrLLMGeneration`, `LLMGenerationError`, constructors, and copied failure/attempt
|
||||||
|
metadata. Current producers and validators retain native PromptKit errors instead.
|
||||||
|
Graph/code searches find this old taxonomy only in that file and its
|
||||||
|
[self-test](../../internal/framework/contracts/errors_test.go), not in an active
|
||||||
|
application consumer.
|
||||||
|
|
||||||
|
This is leftover contract surface from the removed adapter, not a second active
|
||||||
|
execution path. It nevertheless presents obsolete sentinels and metadata accessors
|
||||||
|
as available framework conventions, although matching actual native failures
|
||||||
|
against them no longer works. Stage 35's hard-cutover cleanup is incomplete here.
|
||||||
|
|
||||||
|
### Required Change And Acceptance
|
||||||
|
|
||||||
|
Remove the unused legacy error definitions and their self-test. Keep native
|
||||||
|
`errors.Is`/`errors.As` behavior and the separate, actively used application
|
||||||
|
assessment/capture error types. Do not bridge native errors into these retired
|
||||||
|
sentinels or add aliases for compatibility. Verify no current callers remain and
|
||||||
|
run contracts, pipeline, CLI, and module integration tests after removal.
|
||||||
|
|
||||||
## Review Coverage And Verification
|
## Review Coverage And Verification
|
||||||
|
|
||||||
Owning boundaries were inspected in implementation order:
|
Owning boundaries were inspected in implementation order:
|
||||||
@@ -158,6 +312,25 @@ Checks completed during review:
|
|||||||
repository code or tests. Their deliberately failing assertions are review
|
repository code or tests. Their deliberately failing assertions are review
|
||||||
evidence, separate from the passing default suite.
|
evidence, separate from the passing default suite.
|
||||||
|
|
||||||
|
The integration follow-up inspected the stage boundaries above against the
|
||||||
|
retained source/chunk identity, domain codecs/validators, generated references,
|
||||||
|
diagnostic limits, physical publication, and subprocess contracts. It reused
|
||||||
|
the earlier full-suite/vet/build/race results only after confirming that the
|
||||||
|
application code and dependencies were unchanged. Additional checks were:
|
||||||
|
|
||||||
|
- `GOWORK=off GOPROXY=off go test ./internal/core/config
|
||||||
|
./internal/framework/pipeline ./internal/framework/semanticreconcile
|
||||||
|
./internal/modules/integration ./internal/cli` passed.
|
||||||
|
- Real native-CLI/fake-provider Go-overlay probes reproduced both directions of
|
||||||
|
scene repartitioning in Finding 3. Both corresponding bypass controls passed.
|
||||||
|
- A native-CLI/fake-provider overlay probe reproduced the long-error permissive
|
||||||
|
fallback failure in Finding 4.
|
||||||
|
- Graph/code searches confirmed that Finding 5's legacy taxonomy has no active
|
||||||
|
callers outside its own definition/self-test.
|
||||||
|
|
||||||
|
These probes live only in a temporary review directory. No production code,
|
||||||
|
test, configuration, schema, or existing canonical documentation was changed.
|
||||||
|
|
||||||
No live provider inference was performed. This review does not establish
|
No live provider inference was performed. This review does not establish
|
||||||
production model quality or performance. Intentional changes such as the hard
|
production model quality or performance. Intentional changes such as the hard
|
||||||
configuration/CLI cutover, native retry ownership, content-only cache identity,
|
configuration/CLI cutover, native retry ownership, content-only cache identity,
|
||||||
|
|||||||
Reference in New Issue
Block a user