diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 00000000..adf26d15 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,1369 @@ +# Native Migration Implementation Plan + +## Purpose And Execution Rules + +Implement the complete target in [Native PromptKit And CommandKit Migration](promptkit.md). +That feature roadmap owns product choices, defaults, ownership, and acceptance +semantics. This document owns execution order and stage completion. All stages +are pending. The inspected application baseline is Notarius commit +`b3664d587914b1d7b0377f25fa377b57fa97c815`; its recent roadmap commits have not +implemented the migration. + +Each numbered stage is one implementation prompt for a Luna coding agent. +Execute stages in integer order as one end-to-end sprint. Intermediate stages +may leave callers, tests, or packages unbuildable. Do not spend prompts preserving +intermediate compilation with compatibility adapters, duplicate loaders, aliases +for retired inputs, or replacement completion abstractions. Final compilation, +behavior, and documentation must satisfy the entire target. The native `run` +alias is a selected new interface, not a legacy forwarding implementation. + +Before each stage, read `docs/development.md`, follow its applicable policy links, +and inspect the stage's named source and focused tests. Prefer the codebase +knowledge graph for code discovery. Use the selected upstream public APIs and +maintained examples, not old dependency APIs inferred from current Notarius code. +Keep code and tests scoped to that stage. Commit each completed nonempty unit +under the agent policy, accurately recording any checks blocked by not-yet-migrated +callers. Do not claim a blocked test passed. Later stages own the corresponding +integration; do not omit their work because an earlier package check was blocked. + +The validation commands below are focused checks, not additional permanent +release gates. Run them when their dependencies can compile; otherwise inspect +the affected code, record the concrete dependency blocker, and continue the +sprint. Stage 46 requires the complete assembled repository to pass. Test with +real kits and a fake `promptkit.LLMClient` or local fake HTTP provider, as appropriate. +No ordinary test requires real credentials, a production transcript, paid inference, +or an external service. Dependency acquisition is separate from offline testing. + +No stage authorizes a release, push, tag, deployment, rewrite of published history, +or changes to sibling repositories. Do not delete legacy cache data on disk. +Do not add security restrictions, filesystem permission policy, schema changes, +backend limits, retry caps, or public reporting guarantees beyond the feature +roadmap. Retain deterministic domain correctness and the existing intrinsic +artifact schemas; retire only machinery and contracts the roadmap replaces. + +## Settled Design And Implementation Boundaries + +- Use PromptKit `v0.24.0`, ConfigKit `v1.4.0`, CommandKit `v0.7.0`, ckconfig + `v0.10.0`, and ckprompt `v0.11.0` from their `go.maximumdirect.net` paths. + Keep the Go consumer minimum `1.27.0`; use stable go-yaml v4 and selected + transitive versions from the feature roadmap. +- Version 6 is the sole application configuration version. One application-owned + ConfigKit result supplies native PromptKit settings, engine construction, and + both inspection scopes. Composition-root loaders live in `internal/cli`. +- Use ordinary `assets` embedding, native source contexts, and + `promptkit.collections.notarius`. Prompt declarations own default output + contracts. PromptKit owns schema preparation and structural repair. +- Keep typed stage interfaces. Model-executing code receives the shared native + engine. A small producer helper in `internal/framework/pipeline` accepts ordinary + module request/interpretation/correction/fallback functions and the pipeline's + typed assessment callback. No new LLM service interface or execution-plan language. +- The pipeline assesses each initial/revised candidate once through the whole + applicable validator chain. Native coordination owns growing conversation + history and turn iteration. `max_additional_turns` resolves binding, pipeline, + then `2`; explicit zero disables semantic correction. Native transient and + structural-repair defaults remain unspecified by Notarius. +- `mode` resolves producer binding, pipeline, then `strict`. A permissive fallback + is qualified output, not ordinary acceptance. Cancellation stops without fallback. + Fallback and affected dependent work skip cache lookup and storage. +- `concurrency.workers`, default `64`, bounds one shared ready-job pool for + extraction and lane completion. Native backend admission is independent; no + injected backend limits or replacement global LLM ceiling. +- Cache only final accepted artifacts/plans under the initial prepared + `MessageContentHash`. The key has no extra dimensions. Hits get ordinary decode/ + usability checks and no new semantic assessment. `auto`, `bypass`, and `refresh` + use the roadmap's exact precedence; cache I/O is best effort. +- Publish domain artifacts at their existing run-specific paths. Report one small + native completion value, including a positive `fallback_count` when applicable. + Retire required manifests, receipts, index files, and reporting sidecars. +- Use one invocation `ExecutionGroup`, native observations, and optional convenient + debug evidence. Detailed debug is not a public serialization contract. Preserve + native credential omissions and ConfigKit secret-aware inspection; do not add + heuristic content rewriting. + +Implementation details such as private helper names and debug filenames may be +refined without changing these choices. Put application outcome/callback types +where typed stage contracts can use them without a `contracts -> pipeline` import +cycle. Do not pass pipeline collaborators through hidden process globals. Modules +still receive no physical roots or CommandKit reporters. + +### Domain Fallback Construction + +Use this construction guidance in the corresponding module stages, under U6: + +| Producer | Supported fallback construction | +| --- | --- | +| D&D scene chunker | One plan entry covering the complete source in document order; use normal materialization and source-addressing rules. | +| Spell, registry, occurrence, combat-turn, and enemy-event extraction | The family's codec-valid empty collection, retaining required top-level shape and any deterministically required context. No invented entity or event. | +| Scene-description extraction | Use the final usable candidate only when it satisfies intrinsic shape, source identity, and downstream consumption invariants. Its artifact requires at least one scene, so an empty list is not a valid fallback. If no such final candidate exists, fail fallback construction; do not fabricate kind/title/summary or change the schema. | +| Semantic registry normalization | Apply compatible safe groups from the final usable proposal afresh to fixed originals, preserving other originals separately. With no usable final proposal, use the deterministic merged input. | +| Deterministic merge/normalization | Reuse suitable incoming/merged data only when the module can construct an invariant-preserving result. An operation error or inability to construct that result fails normally; there is no generic retry. | + +Necessary fallback checks belong to the module; they do not rerun the configured +ordinary validator chain. This table does not promise that every operation can +recover from every failure. Insufficient reconciliation candidates can remain +an ordinary deterministic success; exceeding a required semantic preparation +limit is not silently presented as normally validated model work. + +## Stage 1: Select Dependencies And Update Their Notices + +**Goal:** make the reviewed upstream APIs available without a compatibility patch. + +**Inspect/change:** `go.mod`, `go.sum`, `docs/oss/attribution.md`; selected kit +`go.mod`, `COPYRIGHT`, `LICENSE`, and release notes. Read the roadmap's dependency +and evidence sections. Consult the composed CLI example under +`../commandkit/integrations/ckprompt/examples/composed/` as a reference, not a new +Notarius dependency. + +**Work:** add the five selected vanity-path dependencies, retain Go `1.27.0`, +and let Go resolve the reviewed stable YAML/jsonschema versions. Convert import +paths as their callers migrate; do not patch upstream or keep permanent `replace` +directives. Old-path dependencies may remain temporarily while old callers exist; +Stage 35 removes them. Add/update factual attribution with exact selected upstream +commit links and required license material under the OSS policy. + +**Acceptance:** all five selected modules resolve together using declared +versions; no sibling workspace substitutes for a release. Notices describe actual +incorporated dependencies. API incompatibilities are identified as migration work, +not hidden by shims. + +**Validation:** `GOWORK=off go list -m all`, inspect `git diff -- go.mod go.sum +docs/oss/attribution.md`, and `git diff --check`. Do not require the old application +to compile against the new APIs yet. + +## Stage 2: Introduce The Native Version 6 Configuration Definition + +**Goal:** preserve one typed ConfigKit load and its original result evidence. + +**Inspect/change:** `internal/core/config/document.go`, `load.go`, +`loaded_config.go`, `document_normalizer.go`, `operational_document.go`, and their +definition/loading tests; PromptKit `Configure`, `Settings`, and `NewFromConfig`. + +**Work:** replace `PromptKitDocument` with native `promptkit.Settings` at +`yaml:"promptkit"`, compose the application definition with `promptkit.Configure`, +and select application version 6 only. Retain the native `configkit.Result` for +handoff and inspection rather than reserializing settings. Preserve application +pipeline normalization, ConfigKit discovery/import/version semantics, and +source-relative application paths. Remove `profile_file`, `profile_dir`, and +`local_backend` consumption; add no separate PromptKit configuration file setting. +Runtime-only client options belong in native `RuntimeConfig`, and invocation +transient-retry overrides use ckprompt's native request override. Do not recreate +the removed configuration wrapper to hold library behavior. + +**Acceptance:** `version: 6` and its native string equivalent follow ConfigKit's +version handling; version 5 has no compatibility translation. PromptKit rules, +including resource identity and secret marking, participate in the same definition. +The whole result can be supplied to `NewFromConfig` without another load. + +**Validation:** focused `go test ./internal/core/config`; test typed `/promptkit` +handoff and unchanged source-origin semantics. Source assembly is completed in +Stage 5; no fake result or YAML round-trip is an acceptable substitute. + +## Stage 3: Resolve The New Application Controls + +**Goal:** define the selected budgets, modes, workers, and state paths once. + +**Inspect/change:** configuration document/runtime conversion and validation, +`internal/framework/pipeline/profile.go`, `options.go`, `validation_policy.go`, +and policy-resolution tests. + +**Work:** add omission-aware pipeline/binding `mode`, `max_additional_turns`, and +`cache`; root `cache_directory`; and positive `concurrency.workers`. Resolve the +roadmap's precedence/defaults, including explicit zero turns, strict normalization, +and the CLI cache override taking precedence over every binding. Remove producer/ +validator `retries`, application structural-repair fields, failure-category +policies, nested legacy cache settings, `total_llm`, and extraction-worker coupling. +Use native source origins for nonempty config paths. Preserve explicit empty debug +directory as capture-disabled; omitted debug root is `/tmp/notarius-debug`. +Omitted/empty cache root uses `os.UserCacheDir()/notarius`; a configured root is exact. + +**Acceptance:** resolved operations contain final controls rather than repeating +inheritance at execution. Tests distinguish omission, zero, and explicit values. +There are no module-declared permissive defaults, new maximum worker/turn caps, +backend settings derived from workers, or replacement non-LLM retry options. + +**Validation:** focused configuration and pipeline resolution tests, including +binding-over-pipeline mode/turn/cache selection, invocation cache selection, +worker positivity, and explicit-empty path behavior. + +## Stage 4: Convert Embedded Initial Resources And Schema Declarations + +**Goal:** native declarations become the only default prompt/schema assembly. + +**Inspect/change:** existing `assets/**/prompts/prompt.yaml`, +`assets/dnd/profiles/dnd-extraction.yaml`, `assets/package.go`, and old prompt/schema +manifest tests. Inspect all current content-file paths before changing them. + +**Work:** convert initial producer, validator, and generic reconciliation prompts +to native identified lists in the `notarius` collection, retaining local IDs and +using qualified prompt/profile references. Add one ordinary embedded native source +that imports the resources and declares the collection schema root from the assets +filesystem context. Use real relative paths to shared files, not `sharedassets` +mounts. Set each prompt's response format, validation mode, and schema selection; +remove explicit `repair_attempts`. Keep the default extraction profile's current +model, timeout, service tier, unset reasoning, and unqualified `openrouter` backend. +Remove old resource-version selectors; no new PromptKit version header is needed. + +**Acceptance:** every existing LLM-facing schema is still self-contained and +selected from its actual asset location. Intrinsic artifact schemas remain +unchanged. Shared prose/order/cache controls are preserved except for deliberate +conversation wording changes made with each correction pair later. The assets +Go package remains a content-only leaf, without a behavioral registry. + +**Validation:** native ConfigKit load plus PromptKit construction/preparation with +fake request inputs; cover real shared-file paths and the embedded schema root. +Do not add prose snapshots or fixed shared-prefix-length tests. + +## Stage 5: Assemble One Application Load And Native Engine + +**Goal:** runtime and standard commands consume the same selected resources. + +**Inspect/change:** `internal/cli/configkit_load.go`, +`internal/core/config/loaded_config.go`, new focused CLI loader/engine files, and +old `internal/framework/llm` source assembly. Read upstream shared-loading guidance. + +**Work:** select the actual application main source using the existing explicit/ +environment/native discovery policy. Reject explicitly empty `--config` rather +than discovering another file. Assemble bundled OpenRouter, embedded Notarius, +and operator companions as ordered underlays of the one application load. +Discover optional `prompts.yml` and recursive `.yml`/`.yaml` files in `prompts/` +beside that selected main source, in one strict operator peer group. Preserve +native imports/destinations/source contexts; do not fall back to the working +directory or hide malformed operator sources. Main imports, overlay, and explicit +overrides retain native precedence. Construct `NewFromConfig` from the successful +whole result, with runtime-only options, without callbacks/loading happening again. + +**Acceptance:** invocation-owned reuse distinguishes complete selections, +including overlay/overrides and the two sides of `config diff`. Changes to source +files after capture do not mutate an engine. Shared loaders create no run storage. +No duplicate asset registry, settings reparse, or persistent loader cache remains +on the new path. + +**Validation:** focused CLI/config tests using real temporary files: absent +companions, file+directory conflicts, selective replacement, native inheritance, +imports, source-relative messages, embedded/external schema roots, inline/ +source-relative schemas, and missing-file behavior after root replacement. + +## Stage 6: Install CommandKit And The Standard Configuration Suite + +**Goal:** one native application constructor replaces manual command plumbing. + +**Inspect/change:** `cmd/notarius`, `internal/cli/run.go`, `configkit_commands.go`, +`configkit_load.go`, `internal/buildinfo`, and new focused command-composition files. + +**Work:** declare application identity, native help/version, Bash/Zsh/Fish completion, +and presentation through CommandKit. Install ckconfig scopes named `app` and +`promptkit`, using the shared result and native focused-subtree adapter where +appropriate. Install native validation/show/sources/explain/diff and convenience +aliases. Preserve native causes/outcome classification and supplied-value inputs. +Construction itself performs no configuration load, engine build, or storage I/O. +Retire the old configuration command handlers rather than keeping two surfaces. + +**Acceptance:** help/version/completion work without any config or API key; +configuration commands use native report values and the same load evidence. +`config diff` does not accidentally compare a memoized selection to itself. +Release-linked version identity remains available at the real executable boundary. + +**Validation:** focused CLI tests and representative subprocess invocations for +help/version/completion, one app-scope explanation, one focused PromptKit report, +and diff status `3`. Test semantics, not entire upstream help text. + +## Stage 7: Install ckprompt And Reusable Run Overrides + +**Goal:** adopt the full selected native diagnostics and execution input suite. + +**Inspect/change:** CLI composition from Stage 6, old `promptkit_profiles.go`, and +ckprompt's maintained composed example and public integration declarations. + +**Work:** use `FromEngine`, request validation/preparation, smoke integration, and +the shared loaders to install all selected resource listings/inspection, +requirements, preview, and explicit smoke checks. Use upstream request builders +unless an application callback is genuinely needed; preserve supplied inputs. +Expose ckprompt's reusable profile/session/execution flags for pipeline run-wide +overrides, carrying omission and explicit empty reasoning distinctly. Use native +`--llm-profile` and `--llm-session-id`; remove copied profile inspectors and +effective-parameter reports. Do not build new native-command wrappers or probes. + +**Acceptance:** standard inspection needs no transcript/provider call. Concrete +request validation/preview performs native request preparation; smoke work is +explicit only and preserves supported/applicable/result distinctions. The reusable +override value can reach every producer, validator, and correction without +mutating captured configuration. Native transient retries replace the former +application-specific override machinery. + +**Validation:** focused CLI tests with a fake provider: listings, partial profile, +explicit preview inputs, no implicit smoke, supplied false/zero/empty values, and +one explicit smoke outcome. Full pipeline binding is completed in Stage 33. + +## Stage 8: Define Typed Producer Collaboration And Resource Dependencies + +**Goal:** expose only the application contracts required by native execution. + +**Inspect/change:** `internal/framework/contracts/{contracts,typed_pipeline}.go`, +pipeline construction/registries/preparation, module specifications, and typed +registration tests. + +**Work:** replace `ModuleDependencies.LLM` with the shared native engine for +model-executing modules. Add small typed operation collaboration/outcome types +for the pipeline assessment callback, accepted/fallback disposition, fallback +reason, and reuse eligibility. Pass native execution overrides/observers without +creating another completion request/result facade. Adapt typed method arguments +or call options to carry the assessment callback explicitly without an import +cycle. Add static initial/correction prompt dependencies using the same IDs as +request builders; keep dynamic selection runtime-checked. Validators remain +assessment modules, not producer-loop participants. + +**Acceptance:** artifact exact-type checks, codec/evidence compatibility, and +module option/reference validation remain intact. New dependency declarations +are small ID selections, not asset manifests, schema registries, or fingerprints. +Pure domain types and deterministic helpers acquire no PromptKit dependency merely +because surrounding producers do. + +**Validation:** focused contracts/registry/preparation tests, including incompatible +typed registrations and statically declared correction dependencies. This stage +need not migrate every module caller. + +## Stage 9: Centralize Candidate Assessment And Terminal Disposition + +**Goal:** use one complete assessment path for initial and revised candidates. + +**Inspect/change:** `validation_executor.go`, `validation_policy.go`, +`producer_attempts.go`, typed validator execution, and validation-isolation tests. + +**Work:** remove the validator invocation retry loop. Invoke each applicable +validator once per candidate through its public boundary, in configured order, +with immutable candidate views. Collect all rejections/guidance despite earlier +rejections; distinguish reject, fail, and unavailable/skip. Preserve actionable +`CorrectionGuidance` separately from reason codes and operator prose. Expose one +typed assessor to the producer helper. Apply resolved strict/permissive handling +to terminal nonacceptance; unavailable assessment is not a semantic defect. +Fallback construction checks remain module-owned, outside ordinary revalidation. + +**Acceptance:** repeated/unchanged corrected candidates still run the complete +chain. A producer does not acquire acceptance from native execution success alone. +No reason-code text becomes correction instructions, no validator rerun extends +native structural repair, and no final duplicate assessment occurs. + +**Validation:** focused pipeline tests for two independent rejections, earlier +approval followed by a changed candidate, isolated typed validator views, +unavailable assessment, guidance contract errors, and cancellation. + +## Stage 10: Implement The Small Native Producer Helper + +**Goal:** delegate turn iteration while retaining typed Notarius acceptance. + +**Inspect/change:** a focused helper alongside pipeline assessment; upstream +`Coordinate`, `RunRequest`, prepared execution, and native validation status. + +**Work:** accept the module's initial native request, interpretation, correction +builder, supported fallback functions, and the pipeline assessor. Use `Engine.Run` +for uncached work with exchange capture when correction/debug needs it. Check +native errors and `ValidationFailed` before interpretation, including nil-Go-error +failure results. Assess the initial usable candidate, then coordinate actionable +correction from its captured exchange with native `AdditionalTurns`. On the +initial coordination callback, use existing findings; do not reassess the initial +response. Assess each newly delivered result exactly once, including the final +allowed turn. Return typed acceptance/fallback state, not a second retry protocol. +If a continuation fails without a usable response, do not retain an earlier +proposal as the terminal fallback source; use the module's no-usable-response +fallback direction. + +**Acceptance:** native coordination alone counts turns and carries history. +Acceptance stops early; exhausted/stopped are not acceptance. Structural terminal +failure and unavailable assessment do not generate invented semantic guidance. +Cancellation and `Undelivered` do not produce acceptance, fallback, or cache writes. +Distinguish caller cancellation from native execution failures using caller context +state and native typed causes rather than flattening every timeout into cancellation. +No earlier-candidate recovery, cross-turn plan union, or manual assistant replay. + +**Validation:** real PromptKit with a fake provider: initial acceptance, zero turns, +correction acceptance, final-turn acceptance, exhaustion, validation failure with +nil error, operational failure, and cancellation before callback delivery. +Stage 27 adds cache selection to this same helper, not a parallel executor. + +## Stage 11: Connect Typed Runner Operations And Eligibility Propagation + +**Goal:** the runner supplies assessment once and consumes the returned outcome. + +**Inspect/change:** `runner_typed.go`, `typed_execution.go`, `runner.go`, +`runner_chunk_plan.go`, `handoff.go`, and the old state/correction hooks. + +**Work:** bind each resolved producer's assessor and controls to its operation, +route LLM producers through the shared helper, and retain a direct production/ +assessment path for deterministic stages. Remove outer assessment/retry calls +around helper-owned assessment. Carry fallback and cache-eligibility facts through +extract -> merge -> normalize and generated reference handoffs. A chunk fallback +affects all derived lane work; a lane fallback affects that lane and actual +consumers, not unrelated siblings. Keep only the small facts needed for outcomes, +fallback counting, and reuse suppression, without dependency fingerprints. + +**Acceptance:** generated references may carry valid fallback artifacts with +explicit reuse ineligibility. Stable ordering, exact codec handoff, required +references, and fixed pipeline steps remain intact. Each actual fallback operation +contributes once; affected downstream operations are not themselves counted unless +they also emit a fallback. + +**Validation:** focused runner/handoff tests using small typed modules: full-chain +assessment once, deterministic single execution, chunk/lane/generated-reference +eligibility propagation, unrelated predecessors unaffected, and cancellation. +Native production modules are migrated in Stages 12–23. + +## Stage 12: Migrate Spell Extraction As The Representative Producer + +**Goal:** prove native extraction with the real typed module and validators. + +**Inspect/change:** `internal/modules/dnd/extract/spells`, shared extraction input +helpers, spell catalog projection, spell validators, and `assets/dnd/spells`. + +**Work:** prepare transcript, catalog, references, and NPC resolution once; build +native initial requests and interpret native results through the helper. Preserve +source-reference ordering/swapping, contextual entity resolution, spell identity, +catalog semantics, and exact artifact encoding. Add `notarius/dnd.spells.repair` +using the same schema and default profile as the initial prompt. Put enduring +principles in the initial system context and only findings/new correction +assignment in the follow-up. Keep explicit profile/run overrides on both requests. +Implement the valid empty spell fallback and static dependencies. + +**Acceptance:** one generic-chunker spell pipeline exercises native production, +ordered validation, correction, strict failure, and permissive empty output. +Source/ref input bytes are not regenerated during continuation. The helper owns +the operation and the runner does not validate it again. + +**Validation:** `go test ./internal/modules/dnd/extract/spells +./internal/modules/dnd/validate/spells/...`; adapt the assembled spell contract +test to a native provider fake, including catalog rejection and final correction. + +## Stage 13: Refactor The Shared Reconciliation Core + +**Goal:** separate one-time evidence preparation from proposal interpretation. + +**Inspect/change:** `internal/framework/semanticreconcile/{engine,preparation, +proposal,application,schema}.go` and relevant tests; generic normalization prompt assets. + +**Work:** expose reusable preparation, native request construction, and proposal +decoding/assessment as ordinary functions. Keep fixed candidate-number mappings, +range declarations, namespace guidance, bounded source windows, proposal safety, +and model-facing issue prose. Remove the core's old completion client/correction +transport and response-schema registration/identity machinery. The module's +producer helper invocation owns native execution; do not create a replacement +reconciliation service/exchange wrapper. Convert the generic default prompt and +add its correction pair, reusing the same response schema and native defaults. + +**Acceptance:** each operation prepares candidates/windows once and applies every +complete proposal afresh to those originals. Unknown/overlapping handles and +unsafe groups remain rejected. No source evidence or opaque identity is copied by +the model. Pure proposal assessment/application remains independently testable. + +**Validation:** `go test ./internal/framework/semanticreconcile`; retain the +candidate-number, group-safety, deterministic ordering, limits, and source-window +behavioral tests. Native structural failures terminate at the kit boundary, not +as a request to restart the core. + +## Stage 14: Migrate Item Registry Normalization + +**Goal:** prove a complete native normalization operation and safe fallback. + +**Inspect/change:** `internal/modules/dnd/normalize/itemregistry`, its validators/ +codec/identity helpers, and `assets/dnd/item-registry/normalize`. + +**Work:** connect the core functions to the typed producer helper. Preprocess +fixed originals once, use initial/correction requests with fixed handles, and +apply each proposal afresh before the complete normalization assessment. Add the +domain correction prompt and dependencies. Treat preparation-limit inability and +unusable native output through resolved strict/permissive handling, without +automatic permissive defaults. On permissive exhaustion apply only safe groups +from the final usable proposal; otherwise retain the deterministic merged input. +Reuse already available merged data, never rerun deterministic stages to get it. + +**Acceptance:** a failed correction cannot mutate originals, union earlier groups, +or silently claim complete semantic success. Insufficient candidates avoid a model +call as before. Reuse eligibility distinguishes ordinary accepted normalization +from final safe partial/deterministic fallback. + +**Validation:** focused item-normalizer tests with a native fake provider: initial +success, repaired proposal, unsafe final groups, strict failure, permissive +preservation, unchanged-candidate reassessment, and cancellation. + +## Stage 15: Migrate NPC And Location Registry Normalization + +**Goal:** apply the proven normalizer path without domain-policy drift. + +**Inspect/change:** `internal/modules/dnd/normalize/{npcregistry,locationregistry}` +and their normalization assets, identity/grounding helpers, and focused tests. + +**Work:** migrate these two parallel consumers using Stages 13–14's shared core +and producer helper. Preserve each family's canonical naming, proper-name scope, +source grounding, identity computation, compatible-group checks, and output order. +Add each domain correction prompt with the same initial schema/default profile; +retain generic protocol/candidate presentation as ordinary shared files. Remove +their retry directives, schema/policy fingerprints, and manifest-only metadata. + +**Acceptance:** all three registry normalizers share execution and preparation +mechanics but retain their typed application rules. Every turn operates on the +same candidate namespace. Final safe-group/deterministic fallbacks are explicit, +strict remains default, and model errors do not trigger an outer restart. + +**Validation:** `go test ./internal/modules/dnd/normalize/npcregistry +./internal/modules/dnd/normalize/locationregistry`; adapt boundary tests to native +provider fakes and retain domain identity/reconciliation regression coverage. + +## Stage 16: Migrate D&D Scene Planning And Chunk Assessment + +**Goal:** one native chunk producer with deterministic materialization. + +**Inspect/change:** `internal/modules/dnd/chunk/scenes`, chunk-plan interpretation/ +materialization in the runner, chunk validation, and `assets/dnd/scenes`. + +**Work:** prepare the full transcript once, produce/interpret a complete native +plan, materialize it through existing source-addressing checks, and assess it +through the pipeline's chunk validator chain. Add the scene-plan correction +prompt using its initial schema/profile. Implement the permissive one-whole-source +plan and materialize it normally; do not fabricate scene descriptions/tags or add +deterministic slicing. Retain source order for nonconsecutive unit IDs. Remove +outer attempt/replay logic around plan generation. Stage 28 replaces storage. + +**Acceptance:** malformed or unusable plans never become ordinary acceptance. +Every revised usable plan gets complete chunk assessment, including the final +turn. A fallback preserves the full source and marks all derived work ineligible +for cache reuse. Empty/invalid input and fallback-construction errors fail normally. + +**Validation:** focused scene chunker, chunk materialization, and chunk-validator +tests with a fake native provider, including reversed/resolvable ranges, invalid +units, full-source fallback, final-turn acceptance, and cancellation. + +## Stage 17: Migrate The Combat-Semantics LLM Validator + +**Goal:** validators use native execution without another semantic retry loop. + +**Inspect/change:** `internal/modules/dnd/validate/scenedescriptions/combat_semantics`, +its prompt/schema assets, registration, and evaluator tests. + +**Work:** build a native request using the shared engine and run-wide overrides. +Use the configured validator profile/default and native output contract. Check +errors and failed structural validation before decoding. Return domain rejection +with actionable correction guidance, or an unavailable assessment when execution/ +interpretation cannot complete. Preserve scene-local contextual inputs and the +combat policy. Remove validator retry/repair fields and private schema/client +adapters; do not give the validator its own producer coordination loop. + +**Acceptance:** native structural repair may occur internally, but Notarius +invokes the validator only once for each producer candidate. Capacity/provider +failures are unavailable assessment, not semantic rejection. A successful rejection +supplies useful prose to the scene producer and no opaque diagnostic code. + +**Validation:** focused validator and ordered-chain tests for approve, reject, +malformed/failed validation, nil-error failure status, unavailable provider, and +nested same-backend admission with a limit of one. + +## Stage 18: Migrate The Three Registry Extractors + +**Goal:** canonical noun extraction follows the representative native path. + +**Inspect/change:** `internal/modules/dnd/extract/{npcregistry,itemregistry, +locationregistry}`, their extraction assets and shared citation/identity helpers. + +**Work:** migrate the three parallel list producers to native builders, +interpretation, and the shared helper. Add domain correction prompts, static +dependencies, and codec-valid empty fallbacks. Prepare transcript/reference +material once per chunk. Keep proper-name scope, item/currency identity rules, +source evidence resolution, deterministic IDs, and required validator chains. +Remove per-module prompt/schema manifests and completion metadata, without +consolidating unlike domain interpretation rules. + +**Acceptance:** each produces its existing typed durable artifact and preserves +source refs/identity across correction. Empty fallback is distinguished from a +normally accepted empty extraction. No module implements its own turn loop or +pins structural repair. + +**Validation:** focused tests for all three extractor packages and one +representative correction/fallback case each, using the shared native fake pattern +from Stage 12. Retain meaningful per-family grounding and identity regressions. + +## Stage 19: Migrate NPC And Location Occurrence Extraction + +**Goal:** registry-consuming occurrence lanes retain contextual resolution. + +**Inspect/change:** `internal/modules/dnd/extract/{npcoccurrences,locationoccurrences}`, +their assets, registry resolvers, and source/registry validators. + +**Work:** build initial native requests from fixed chunk/registry material; +interpret contextual selections and attach durable IDs deterministically. Add +the two correction pairs and empty occurrence fallbacks. Keep occurrence kinds, +unknown/ambiguous reference handling, accepted source citations, and compatible +generated-reference expectations. Pass profiles/session/overrides unchanged. + +**Acceptance:** model responses never reproduce opaque application IDs. Source +context and registry resolution remain fixed throughout each correction exchange. +Fallback effects propagate to actual later consumers without invalidating +unrelated registry/predecessor artifacts. + +**Validation:** focused NPC/location occurrence tests covering contextual matches, +ambiguous selection, invalid source refs, corrected final candidate, and fallback. +Do not relax existing domain invariants merely to make the migration tests pass. + +## Stage 20: Migrate Item Occurrence Extraction + +**Goal:** retain the smaller-model holder reliability fixes in the native path. + +**Inspect/change:** `internal/modules/dnd/extract/itemoccurrences`, associated +holder/registry/source validators and assets, and holder regression tests. + +**Work:** use native initial/correction requests and fixed item registry context. +Preserve `discovered`, `acquired`, `lost`, `consumed`, and `transferred` semantics, +the private response-to-durable holder mapping, party-member handling, and +source-grounded deterministic entity resolution. Add the same-schema correction +pair and valid empty fallback; keep module interpretation guidance separate from +operator messages. Remove only execution/asset registration machinery. + +**Acceptance:** existing acquired/from/to reliability behavior and event ordering +survive. No additional optional model-facing schema fields, synthesized holders, +opaque model IDs, or durable artifact changes are introduced. + +**Validation:** focused item occurrence extractor/validator tests with native +provider fakes, including the previously reported incompatible-holder regression, +correction acceptance, empty fallback, and final semantic rejection. + +## Stage 21: Migrate Combat-Turn Extraction + +**Goal:** native combat production preserves scene eligibility and evidence rules. + +**Inspect/change:** `internal/modules/dnd/extract/combatturns`, combat validators, +scene-description reference selection, and combat prompt assets. + +**Work:** migrate initial native requests, result interpretation, correction pair, +and empty fallback. Preserve combat-only scene eligibility, actor/turn-kind rules, +source-order endpoint normalization, and source/registry context. Skipped +ineligible chunks remain deterministic no-model outcomes, not invented native +requests or synthetic cache entries. + +**Acceptance:** no combat model call is made for an ineligible scene; correction +uses the same fixed operation inputs and schema. The prior resolvable reversed- +endpoint reliability behavior remains without new semantic retry cost. + +**Validation:** focused combat extractor/validator tests and assembled combat +eligibility coverage with a native fake provider. Check no-call cases by provider +invocations, not by obsolete completion-interface mocks. + +## Stage 22: Migrate Enemy-Event Extraction + +**Goal:** preserve multi-artifact grounding while changing execution ownership. + +**Inspect/change:** `internal/modules/dnd/extract/enemyevents`, grounding/mapping +helpers, its validators, and enemy-event assets. + +**Work:** move the module to the shared helper and native initial/correction pair. +Keep NPC/scene/combat-turn/NPC-occurrence references, evidence intersection, +engagement checks, contextual names, and durable event order. Prepare reference +projections once per operation; use the codec-valid empty event fallback. Remove +its private completion and manifest/fingerprint metadata. + +**Acceptance:** an operator override applies to initial, correction, and nested +validation without changing domain reference compatibility. Failed/fallback +predecessors affect reuse eligibility but valid fallback references remain usable +in the current run. No fabricated enemy event or expanded extraction scope. + +**Validation:** focused enemy-event extraction/validation tests plus the assembled +enemy-event codec contract, using a native provider fake and current typed codecs. + +## Stage 23: Migrate Scene-Description Extraction + +**Goal:** retain the required scene artifact while making fallback limits explicit. + +**Inspect/change:** `internal/modules/dnd/extract/scenedescriptions`, its codec, +shape/source/invariant validators, semantic validator integration, and assets. + +**Work:** use native initial/correction requests with one description for the +operation's source range as currently required. Preserve deterministic scene IDs +and source ranges; keep the kind/title/summary contract unchanged. Add the +domain correction prompt and same-schema/profile dependency pair. Implement the +fallback rule in this plan's table: only a final usable candidate meeting the +intrinsic artifact/downstream invariants can be qualified fallback output. If +the final execution yields no usable scene, fail fallback construction rather +than emitting an empty list, inventing a classification, or recovering an earlier +candidate. Ordinary semantic revalidation is not a fallback prerequisite. + +**Acceptance:** strict rejection fails; permissive handling never publishes an +invalid scene artifact or claims semantic approval. A rejected combat judgment's +guidance reaches correction, and every revised scene gets the full chain. + +**Validation:** focused scene-description/semantic-validator tests with native +fakes: correction, valid-but-semantically-rejected final fallback, structurally +unusable final response, no prior-candidate recovery, and source identity. + +## Stage 24: Finish Deterministic Stage Paths And Production Registration + +**Goal:** all non-LLM work runs directly, and production registration is asset-free. + +**Inspect/change:** generic unit chunker/append-order merger/noop normalizer, +deterministic D&D normalizers, Seriatim adapter, D&D/generic registrars, typed runner, +and module integration fixtures. + +**Work:** remove outer retries and obsolete correction/repair arguments from +deterministic operations. Preserve typed merge/normalize rules, ordering, domain +validators, generated references, evidence projectors, and raw artifact codecs. +Use resolved mode only for rejected candidates with a supported module fallback; +ordinary parse/materialization/application/encoding errors propagate normally. +Register modules, validator defaults, and static dependencies without registering +prompt files, schemas, or fallback profiles. Complete migration of every LLM role +identified in Stages 12–23, not just the examples. + +**Acceptance:** there is no synthetic PromptKit request for deterministic work +or generic retry setting. Durable artifact-schema validation remains; removing +duplicate model-response schema validation does not remove this application +boundary. Pipeline default-chain replacement, including explicit empty chains, +and exact typed registration still work. + +**Validation:** focused generic/D&D deterministic normalizer, registrar, Seriatim, +and typed integration tests; retain data-integrity tests and remove tests whose +only purpose was the retired generic retry/asset API. + +## Stage 25: Replace Worker Pools And Remove The Global LLM Scheduler + +**Goal:** one ready-job limit and native backend admission govern concurrent work. + +**Inspect/change:** `runner_concurrent.go`, synchronized collaborators, +CLI runtime construction, old `llm/scheduler.go`, and concurrency/cancellation tests. + +**Work:** use the existing coordinator to dispatch ready extraction and ready +merge/normalize jobs to one worker pool controlled by `concurrency.workers`. +Lane completion becomes ready only after all of that lane's extract jobs finish. +Never occupy a worker while waiting for prerequisites, or acquire another worker +for nested validation/correction. Keep sequential input/chunking/publication and +ordered steps. Remove scheduled-client/global permit plumbing and worker coupling; +share the same native engine across every role. Handle native capacity failures +through the producing operation's mode without another queue or backoff. + +**Acceptance:** one worker can complete a multi-lane pipeline without deadlock. +The limit bounds both job types together, independent of backend settings. +Completion timing cannot choose public order or the terminal scoped error. +Cancellation stops dispatch, drains started work, and releases native admission. + +**Validation:** focused runner concurrency tests with test-controlled worker and +backend limits, including overlap, serial work, ready-lane interleaving, nested +same-backend validation, capacity failure, and cancellation. Use focused race +testing when supported; it is a migration check, not a new release requirement. + +## Stage 26: Implement The Small Accepted-Artifact Filesystem Cache + +**Goal:** one best-effort cache facility replaces complex persistent records. + +**Inspect/change:** existing `internal/core/fileio` and cache permission/path tests; +add a focused accepted-artifact store under `internal/framework` or the existing +state area, with a narrow lookup/write collaborator for pipeline use. + +**Work:** read/write accepted artifact bytes at `/.json`, +using the native opaque hash unchanged. Use ordinary temporary-file/rename +publication and existing proportionate path/permission correctness helpers. +Keep roots at the CLI/state boundary. The caller owns typed codec/usability +checks. Missing entries are normal misses; unreadable/malformed/unusable entries +are misses with diagnostics. Failed writes warn without failing successful work. +Implement `auto`, `bypass`, and `refresh` I/O semantics; bypass creates no directory. +Refresh cannot remove a previous entry before a newly accepted replacement exists. +Do not add envelopes, manifests, key prefixes, expiry, migration, or collection. + +**Acceptance:** records contain only the accepted artifact/plan; no execution, +configuration, transcript copy, acceptance ledger, or historical usage is required. +Creation respects umask/setgid/default ACLs under ADR-0017, without permission +convergence or added access restrictions. Legacy stores are left untouched. + +**Validation:** focused real-temp-filesystem tests for hit/miss, unusable data, +read/write failure diagnostics, atomic replacement, modes, narrow paths, and +ordinary creation permissions. Reuse existing safety coverage where applicable. + +## Stage 27: Cache Final Typed Producer Artifacts Under The Initial Request + +**Goal:** integrate caching into the same native producer helper. + +**Inspect/change:** the helper, typed artifact codecs, operation eligibility, +resolved cache controls, and cache store from Stage 26. + +**Work:** for eligible cacheable LLM work in auto/refresh, prepare the initial +request with `PrepareExecution`; obtain `Details().MessageContentHash`, retain +the prepared details, and keep that key for the complete operation. In auto, +decode/usability-check a found artifact and discard the prepared handle on a hit. +On a miss execute that exact prepared handle using `RunPrepared`, then use the +same interpretation/assessment/coordination path. After final ordinary acceptance, +encode the final typed artifact and write it under the initial key. Normalizers +store the applied artifact, not the reconciliation proposal. Dispose of unused +handles on every exit. Bypass/ineligible work uses ordinary native execution and +performs no cache I/O. Capture must not cause another preparation. + +**Acceptance:** a hit makes no model calls, semantic validation, exchange replay, +or historical usage claims. Models/settings/session/cache hints/schema metadata +add no key dimensions; only the upstream ordered message-content value selects +reuse. Changed policy can deliberately reuse old artifacts until refresh/bypass. +Ordinary usability includes compatibility with the expected codec and current +source addressing; it does not rerun the configured domain/semantic validator chain. +Failed/rejected/incomplete/fallback and affected dependent operations neither +load nor store entries. Corrected accepted output uses the initial key. + +**Validation:** focused helper/store tests: content changes miss, settings-only +changes can hit, ordinary usability failure misses, late accepted write, no write +after terminal failure/cancellation/fallback, eligible predecessor reuse, and +native prepared details retained on a hit without an execution event. + +## Stage 28: Unify Chunk-Plan Reuse And Remove Checkpoint Execution Paths + +**Goal:** LLM chunk plans use the same accepted-result cache and no second store. + +**Inspect/change:** `runner_chunk_plan.go`, `chunk_cache.go`, `chunk_plan_store.go`, +checkpoint hooks in runner/preparation/handoff, CLI cache composition, and old +`internal/framework/{chunkplan,checkpoint}` consumers. + +**Work:** cache a generated plan only after materialization and complete chunk +assessment/continuation acceptance, under its initial request's native hash. +On a hit decode/rematerialize it against the current source with ordinary +structural/source-reference usability checks, without semantic revalidation. +An unusable cached plan is a miss, not a defective assistant response. Keep +deterministic chunkers uncached without synthetic requests. Remove source-digest +plan lookup/publication, checkpoint event/state transitions, stage/dependency +fingerprints, resume barriers, and recomputation bookkeeping from execution. +Old records are unread and unmodified, even below the selected new cache root. + +**Acceptance:** changed chunk request messages can select a different plan; +source identity alone no longer pins a partition. Default reruns reuse accepted +predecessors without `--resume`. Chunk fallback and dependent work skip reuse. +No checkpoint identity is retained solely to fill diagnostics. + +**Validation:** focused chunk/runner/cache tests with fake providers: matching +plan reuse, changed references/prompt text, settings-only reuse, invalid cached +source ranges, strict/permissive outcomes, bypass/refresh, and legacy files left +unchanged. Remove old canonical-partition/resume-policy tests as superseded. + +## Stage 29: Publish Only Domain Artifacts At Existing Paths + +**Goal:** simplify the logical output boundary without changing payload contracts. + +**Inspect/change:** `internal/modules/generic/output/json/encoder.go`, +`internal/core/artifacts`, output contracts/runner result assembly, evidence output, +and chunk-map publication. Inspect the existing lane filename mapping. + +**Work:** keep run-specific `lanes/.json`, optional `chunk-map.json`, +and optional `evidence-context.json`. Preserve domain schemas, exact lane mapping, +and union-of-cited-source-segments evidence behavior. Remove mandatory index, +manifest, rejected/warnings/diagnostics sidecar generation and reporting-only +models/metadata. Return the small application outcome and logical deliverable +files needed for publication, fallback disclosure, and optional capture. +Intrinsic domain metadata remains intrinsic, not discarded as reporting data. + +**Acceptance:** codecs and normal downstream use need no metadata bundle or +persisted receipt. Artifact count includes actual published domain deliverables, +including reused artifacts and enabled auxiliaries, not intermediates/cache/debug. +Fallback artifacts retain required downstream shape without normal-acceptance claims. + +**Validation:** focused encoder/evidence/chunk-map/codec tests verifying preserved +payloads and paths, no mandatory sidecars, optional artifact inclusion, accurate +counting, and evidence source order/window behavior. + +## Stage 30: Build The Debug Recorder And Session Lifecycle + +**Goal:** requested capture has one simple recorder and explicit failure semantics. + +**Inspect/change:** `internal/core/debugbundle`, `internal/framework/debug`, +pipeline debug collaborators, run IDs, and CLI debug/state composition. + +**Work:** replace the old summary/trace hierarchy with an application recorder +for run context, operation evidence, and one final outcome. Implement +`//` and start-time relative `latest`, without completion-time +retargeting. Resolve the selected native debug boolean and CLI/config/empty/default +directory semantics. Allocate only on the run path after config/run identity and +before engine construction/preflight. Persist incrementally and finalize once. +Allocation/link failure prevents model work; recording failure surfaces at normal +boundaries; finalization failure fails the invocation. Preserve an existing primary +failure and available data/publication paths; never retry a model for capture failure. + +**Acceptance:** a configured directory alone never enables capture; explicit empty +disables it; a nonempty CLI override can re-enable it. Disabled runs leave `latest` +untouched. Quiet mode and capture selection are independent. No automatic bundle +deletion, fallback directory, capture-depth setting, or new filesystem framework. + +**Validation:** focused temp-filesystem lifecycle tests: absent/empty/nonempty +selection, start failure, failed run retention, concurrent start/link semantics, +incremental recording failure, finalization failure, and no completion retarget. + +## Stage 31: Attach Native Execution Facts And Complete Debug Evidence + +**Goal:** use native observations/snapshots instead of parallel execution accounting. + +**Inspect/change:** helper/validator request observers, runner operation boundaries, +new recorder, old `pipeline/debug.go`, and PromptKit observation/exchange APIs. + +**Work:** create one invocation `ExecutionGroup`. Enroll the initial request once, +continuations once through `CoordinateRequest.Observer`, and nested validators +once on their own requests; do not also attach the same group to continuation +requests. Compose scope-aware diagnostic/capture observers without replacing +existing ones. Use native RunID/ParentRunID and totals, preserving unknown usage +and terminal coverage. Keep `OnPrepared` lightweight; retain its detached details +for normal boundary writes. For split prepared/cache paths use existing details. + +Record the roadmap's full available inventory: effective secret-aware config and +composition; parsed source; self-contained source/reference/normalization inputs; +prepared requests/output contracts; native events/results/raw response text; +all visible exchanges/readable messages; typed candidates, findings/guidance, +correction inputs, fallbacks, deterministic results, and cache decisions. Label +unassessed `Undelivered` responses and failures lacking prepared/result evidence. +Use convenient indented JSON/text without truncation, fabricated provider bodies, +heuristic scrubbing, or whole-run content accumulation. Exact filenames stay private. + +**Acceptance:** native totals count executions rather than producer attempts; +cache hits add none. Debug needs no second prepare or model call. Shared callbacks +are concurrency-safe; observer write failures reach the next boundary. ConfigKit +secrets/native credentials remain omitted without rewriting ordinary application +content. No duplicate usage accumulator, synthetic call-ID model, or profile ledger. + +**Validation:** focused helper/recorder tests for initial+continuation+nested calls, +result-less terminal failures, cache hit details, canceled/undelivered turns, +full untruncated selected content, native credential omissions, and concurrent +operations. Check meaningful evidence, not the complete native JSON shape. + +## Stage 32: Add Offline Preflight And Pipeline Inspection Commands + +**Goal:** execution and inspection share domain resolution and known dependencies. + +**Inspect/change:** new `internal/preflight`, CLI pipeline declarations, catalog, +profile/reference/module resolution, and existing explicit-profile preflight tests. + +**Work:** implement `pipeline list`, `show `, and +`validate `. List only enumerates configuration. Show reports resolved +steps/lanes/modules/validators/references without building an engine merely to +expand every profile. Validation reuses resolver checks and, when LLM dependencies +are selected, the same captured engine and static module declarations as runtime. +Check initial/correction/validator prompts, explicit/applicable/default profiles, +and known dependencies with pipeline/stage/lane context. Remove the old pipeline +mode of `config validate`; that standard command validates resolved configuration. + +**Acceptance:** offline preflight needs no input transcript, reference file bytes, +credential values, provider call, output/cache/debug root, or synthetic request. +Valid partial profiles pass. Dynamic request selections are checked at runtime. +Preflight does not load configuration, register modules, construct engines, or +render CLI reports; it accepts those collaborators/data from composition. + +**Validation:** focused preflight/CLI tests for known missing prompt/profile, +validator and correction dependencies, defaults with no explicit profile, valid +partial profile, no-credential/no-transcript operation, and useful scope context. + +## Stage 33: Wire Native Pipeline Run Inputs And Runtime Composition + +**Goal:** the executable runs the migrated pipeline through one native handler. + +**Inspect/change:** CLI application/pipeline commands, `run.go`, catalog, +reference selectors, `session.go`, run IDs, configuration/engine/state loaders, +and `cmd/notarius`. + +**Work:** install `pipeline run --input-file PATH` and +`commandkit.Alias("run", "pipeline run")`. Use typed supplied values for `--lane`, +`--output-dir`, `--cache`, references/unbinds, debug directory, and ckprompt +overrides. Keep repeated lanes literal (commas are not separators); validate every +singleton occurrence and retain the last successfully parsed value. Expand +reference selectors before framework resolution using the existing specificity, +required-slot, external/generated-conflict, and file-origin policies. Preserve +deterministic input-based session generation, independent of references, and +native explicit session override behavior. Apply run-wide profile/execution +overrides to producers, validators, and continuation requests. + +Reuse the one loaded selection and one engine, resolve/preflight before source +execution, then materialize references and prepare typed modules. Allocate requested +debug before engine/preflight as specified; keep all physical roots at composition. +Remove old dispatch, assembled help, parsers, comma `--only`, bespoke `--json`, +reasoning-clearing flag, resume/recompute/chunk-cache flags, and legacy aliases. + +**Acceptance:** native alias and canonical route invoke the same declarations/ +handler; `--`, operands, repeated flags, modes, and supplied empty/false/zero retain +upstream behavior. No forwarding command, recursive command invocation, fallback +to another config for explicit empty input, or request override mutation of config. + +**Validation:** focused command/reference/session tests and subprocess cases for +the two run routes, multi-reference input, literal comma lane, required input, +native overrides reaching all roles, explicit empty reasoning/debug path, and +input-hash session stability across changed reference bundles. + +## Stage 34: Finish Publication, Native Completion Reports, And Failure Outcomes + +**Goal:** subprocesses receive artifacts plus the selected small native report. + +**Inspect/change:** `run_result.go`, `run_terminal.go`, physical output placement, +CommandKit reporter/outcomes, debug finalization, and subprocess receipt tests. + +**Work:** publish domain files in the existing run-specific destination, finish +requested capture, then call `Reporter.Report` with pipeline, artifact_count, +absolute output_directory, optional debug_directory, and positive-only +fallback_count. Supply the selected human text template from that same value. +Remove separate encoders, receipt/error envelopes, automatic report persistence, +warning-count summaries, and durable execution ledgers. Send native diagnostics +to stderr and results to stdout, preserving quiet/mode/progress semantics and +fallback qualification even when warnings are hidden. + +Map native outcomes: completion `0`, operational/cancellation/publication/ +presentation failure `1`, identifiable invalid invocation/configuration `2`, native +diff differences `3`. Preserve typed underlying causes and distinguish operation +failure from report/diagnostic delivery failure. A post-publication capture/report +failure identifies available artifact/debug paths, returns nonzero, and does not +rerun or remove valid artifacts. Failed stdout may be partial; consumers check status. + +**Acceptance:** `--format json --verbose` has report-only valid JSON on stdout +for success. Positive fallback_count counts emitting operations once, not affected +artifacts, attempts, or validators. Quiet suppresses stdout but not useful failure +explanations. Native standard commands have no Notarius-wide envelope. + +**Validation:** focused subprocess tests for text/JSON/YAML success, fallback and +zero omission, verbose stderr, quiet/debug order, render/delivery failure, +publication then capture failure, useful exit categories, and no reporting sidecars. + +## Stage 35: Remove Superseded Machinery And Restore Whole-Tree Compilation + +**Goal:** delete the old architecture rather than leave dormant alternatives. + +**Inspect/change:** all production/test callers of the feature roadmap's removal +inventory; `internal/framework/llm`, `promptfs`, `readonlyfs` consumers, +checkpoint/chunkplan stores, private completion/correction/schema types, +manifest/receipt models, and old CLI tests/dependencies. + +**Work:** remove now-unused adapters, schedulers, asset/schema registries, +flattening/mount helpers, effective-YAML walking, fingerprints, manual correction +transport/capability checks, generic retries, profile/usage conversions, debug +wrappers, state ledgers, and old command surfaces. Retain useful pure helpers only +where a current consumer needs them, in the actual owning package. Do not keep an +empty `internal/framework/llm` for construction. Adapt remaining tests to native +provider boundaries and final contracts; delete tests solely asserting retired +behavior, not tests protecting unchanged domain correctness. Run tidy and remove +old-path kits/catalogs after their last imports disappear; reconcile notices. + +**Acceptance:** every package and test compiles with the declared dependencies. +Production code has one native path and no legacy compatibility scaffold. Removal +does not delete output/cache/debug directories or rewrite accepted ADR decisions. +Pure domain/codecs remain free of unnecessary library dependencies. + +**Validation:** `GOWORK=off go mod tidy`, `GOWORK=off go test ./... -run '^$'`, +`go list -m all`, and scoped code searches for the removal inventory. Classify +historical release/ADR/archive mentions rather than blindly deleting text matches. + +## Stage 36: Migrate The Maintained Minimal And Complete Examples + +**Goal:** two useful D&D configurations demonstrate the final native workflow. + +**Inspect/change:** `examples/dnd-{minimal,complete}.config.yml`, pipeline imports, +profiles/overlay files, transcript/reference fixtures, and +`internal/cli/example_contract_test.go`. + +**Work:** organize the two examples as `examples/dnd-minimal/config.yml` and +`examples/dnd-complete/config.yml`, moving their required imports/fixtures and +fixing paths. Separate directories prevent the complete example's companion +resources from changing the minimal example. Minimal uses embedded defaults and +a simple pipeline. Complete uses every configured D&D family and ordered generated +references, with optional chunk-map/evidence output; demonstrate native companion +profiles, inheritance/operator override, and a limited backend without overriding +the native queue default unnecessarily. Use version 6, qualified IDs, new worker/ +cache/mode/turn controls, and no old retry/structural-repair fields. Replace the old +development variant with a native profile/prompt customization or semantic-budget +example, not a compatibility translation. Keep all files secret-free. + +**Acceptance:** both actual config paths load, resolve, and pass offline pipeline +preflight without credential values. Imported paths are declaration-relative. +The minimal example has no accidental companion dependency. Retire superseded +copyable configurations, retaining useful shared test/reference data as needed. + +**Validation:** maintained example-loading tests plus native configuration and +pipeline validate commands for both paths. Fake-provider integration can exercise +them; live OpenRouter/local services are not default validation requirements. + +## Stage 37: Verify Assembled Recovery And Native Command Workflows + +**Goal:** prove the cross-boundary behavior most likely to regress in this cutover. + +**Inspect/change:** CLI assembled/command/cache contracts, module integration +tests, and new provider-boundary fixtures. Consolidate existing coverage rather +than creating a second comprehensive fixture framework. + +**Work:** exercise a representative ordered multi-step pipeline with typed +generated references and one native engine. Include the representative extractor, +normalizer, LLM validator, and deterministic paths. Add focused assembled cases +for downstream failure/fix/rerun reuse, correction stored under the initial key, +fallback taint across handoffs versus unaffected predecessors, chunk-plan reuse, +refresh failure preserving an old entry, native limited-backend admission and +serial workers, debug on hits/failure, and report/stream outcomes. Ensure no +unexpected external calls in help/inspection/preflight, or duplicate preparation +for debug. Cover custom operator schemas/contracts through real preparation; +where an actual request-level override is needed, use a complete native +`OutputContractInput` with `RepairAttempts` unset unless the operator supplied it. + +**Acceptance:** behavior follows the feature roadmap's full acceptance matrix, +including result-less failures, unknown usage, native status distinctions, final +turn assessment, all-validators guidance, source/session/reference preservation, +and unchanged intrinsic payloads. Avoid duplicating upstream parser/renderer/ +retry tests, pinning upstream default attempt counts, or snapshotting full help/ +native debug objects. No production model-quality claim is inferred from fakes. + +**Validation:** `GOWORK=off go test ./internal/cli ./internal/modules/integration +./internal/framework/pipeline ./internal/framework/semanticreconcile`; run focused +race checks for shared observers/coordinator when practical. Stage 46 owns the +complete final suite after documentation and remaining corrections. + +## Stage 38: Record Successor Decisions And Align Architecture Policy + +**Goal:** document deliberate boundary changes without rewriting their history. + +**Inspect/change:** `docs/policy/architecture.md`, ADRs 0005–0007, 0011, 0014, +0016, 0019, and the retained 0012/0013/0015/0017/0020 decisions. Inspect the current +next ADR number before adding sequential records. + +**Work:** write concise successor ADRs for (1) native kit composition/execution/ +admission and prompt-declared assets, (2) native continuation with application +assessment/modes, (3) unified accepted-artifact caching, and (4) artifact-first +publication/native reporting and debug ownership. Use the next available numbers, +starting at 0021 if none have since been added. Change earlier statuses/links to +identify full or partial supersession, preserving accepted decision bodies and +retained content-only asset, contextual-ID, process-warning, deployment-permission, +and ConfigKit ownership decisions. Architecture now describes native library +types as an intentional execution boundary, one worker pool/native admission, +new retry/fallback/cache semantics, optional diagnostic records, and current +output/debug responsibilities. + +**Acceptance:** no current policy still mandates the retired private LLM API, +global scheduler, latest-only replay, source-only plan partition, checkpoint +resume, manifests, or heuristic sanitization. Retained source/domain/type and +filesystem correctness boundaries remain, with no new access policy, storage +guarantee, release gate, or public debug format. Keep policy provenance headers +accurate; this is a local architecture update, not a new upstream policy adoption. + +**Validation:** review each changed invariant against the feature roadmap and +implemented behavior. Check ADR status links and accepted-body preservation; +`git diff --check`. Do not rewrite historical release notes or archived roadmaps. + +## Stage 39: Rewrite Canonical Configuration And CLI References + +**Goal:** operators can configure and invoke only the implemented native surface. + +**Inspect/change:** `docs/config.md`, `docs/cli.md`, final command declarations, +ConfigKit definition, resolver tests, and the two maintained examples. + +**Work:** document native discovery and companions under one result; version 6; +qualified resources, replacement/inheritance/null semantics and schema roots; +native profile/backend controls; worker and semantic-turn controls; strict/ +permissive precedence; and cache settings. Explain no backend concurrency default +is injected and no independent Notarius structural-repair count is pinned. +Document new pipeline commands/alias, required operands, supplied values, repeated +lanes/references, run-wide ckprompt flags, native formats/modes/progress, stdout/ +stderr, and outcome codes. Link upstream-owned standard command semantics where +appropriate, with short Notarius-specific selection examples rather than a +copied upstream manual. Link full copyable files in `examples/`. + +**Acceptance:** current references describe implemented behavior only. Retired +flags/fields occur solely in clearly identified migration material, not usable +syntax. Config fields/defaults belong in the configuration owner, invocation +syntax in the CLI owner; neither duplicates physical state or artifact schemas. + +**Validation:** compare documented commands with native help and focused parser/ +loading tests. Check relative links/anchors and example paths. Test omitted and +explicit values through declarations, not prose-based change detectors. + +## Stage 40: Update Operations And Downstream Consumer Contracts + +**Goal:** explain the intentional persistence and subprocess breaking changes. + +**Inspect/change:** `docs/operations.md`, `docs/consumers/{subprocess,dnd-pipeline}.md`, +`docs/integrations/run-result.md`, JSON/chunk-map/evidence integration contracts, +and any release-runbook command references affected by the CLI cutover. + +**Work:** describe the independent output/cache/debug roots, per-user cache default +and `/var/cache/notarius` system convention, reuse modes, best-effort I/O, unchanged +legacy data, and explicit refresh for policy/model experiments. Explain fallback +qualification and reuse exclusion, whole-source chunk fallback, debug selection/ +empty paths/start-time `latest`, retained evidence, and capture/publication failure. +Delete the superseded persisted receipt contract and replace consumer links with +the CLI report contract plus documented domain paths; do not create a new manifest +contract. Consumers check status, optionally decode the small report, then read +configured lane/auxiliary paths. Note partial stdout on delivery failure and valid +artifacts after post-publication failure. + +Add `docs/consumers/native-migration.md` covering old-to-new commands, +configuration/resources/catalog IDs, native retry intent, removed transient/repair +wrapper controls, workers/admission, cache reuse, and receipt traversal replacement. +Link it from canonical entry points. Exact old provider-attempt parity and automatic +cache conversion are not promised. Keep tagged release notes unchanged; preparing +or publishing a new release is outside this sprint. + +**Acceptance:** the complete D&D subprocess workflow uses `--input-file`, +`--output-dir`, and pipeline-level references as first-class inputs. Documentation +does not require removed sidecars, depend on debug layout as an API, claim cache +revalidation, or add new provisioning/retention restrictions. Artifact contracts +preserve their payload schemas and logical paths. + +**Validation:** exercise maintained subprocess examples with a fake provider and +compare resulting files/report. Check all references to the removed receipt, +old cache families, and legacy flags; retain necessary migration/history mentions. + +## Stage 41: Refresh Configuration, CLI, And LLM Internal Guides + +**Goal:** explain the actual native composition and execution boundary. + +**Inspect/change:** `docs/internal/{configuration,cli,llm}.md`; the final +configuration definition, composition-root loaders, command declarations, engine +construction, and native producer/validator call sites. + +**Work:** document one result handoff, source assembly, invocation reuse, standard +suite wiring, request overrides, native execution/continuation, and application +interpretation. Keep `llm.md` as an integration-focused explanation even though +its old package is removed. Link canonical operator and upstream contracts rather +than reproducing their fields, flags, or library mechanics. Remove old schemas/ +asset registration, provider-neutral adapters, scheduler, profile recording, and +inspection-conversion descriptions. + +**Acceptance:** a contributor can locate the current owner without being sent to +a deleted wrapper. Native execution types are explicit; pure domain types remain +separate. Volatile public details have one canonical owner. + +**Validation:** inspect the named packages and focused tests for every flow; +check cross-links and stale package references. + +## Stage 42: Refresh Pipeline And State Internal Guides + +**Goal:** explain acceptance, concurrency, recovery, and capture as implemented. + +**Inspect/change:** `docs/internal/{pipeline,state}.md`; typed helper/assessment, +worker coordinator, eligibility propagation, cache/store, recorder, and publication. + +**Work:** document initial/revised assessment, native semantic continuation, +strict/permissive terminal handling, supported fallback, ready-job workers, +generated-reference eligibility, accepted-artifact/plan reuse, and output/debug +ownership. Remove checkpoint fingerprints/resume, old chunk-plan canonicalization, +duplicate usage/debug wrappers, required manifest/receipt, and generic retry flows. +Link public state controls, artifact paths, and output contract definitions. + +**Acceptance:** current-flow descriptions distinguish native execution success, +domain acceptance, qualified fallback, reuse, and publication. No internal guide +promises a public debug layout or cache revalidation policy that does not exist. + +**Validation:** compare diagrams/prose with runner/cache/recorder and focused +tests; verify no stale state collaborator or operation ordering is carried forward. + +## Stage 43: Refresh Module And D&D Contributor Guides + +**Goal:** future modules can follow the native architecture without copying drift. + +**Inspect/change:** `docs/internal/{modules,dnd}.md`; production registration, +representative spell producer, registry normalizers, scene chunker/validator, +native initial/correction assets, codecs, and shared domain helpers. + +**Work:** show how a module prepares fixed inputs, builds initial/correction native +requests, interprets results, supplies fallback, and receives pipeline assessment. +Document static dependency declarations and native prompt/schema ownership instead +of manifests/registration. Explain one-time normalization handles, fresh proposal +application, domain guidance, unchanged contextual-ID/source-reference rules, and +deterministic paths without synthetic requests. Link durable schemas and public +module configuration rather than repeating them. + +**Acceptance:** the extension recipe works for the actual typed interfaces and +retains artifact-family/domain boundaries. It requires no private completion +interface, schema registry, fingerprint, or module-owned retry engine. + +**Validation:** walk the recipe against one extraction and one normalization +implementation; inspect native asset paths/IDs and retained domain regression tests. + +## Stage 44: Align The Kit Integration Contracts + +**Goal:** describe Notarius's actual use of all three native libraries. + +**Inspect/change:** `docs/integrations/{pkg-configkit,pkg-promptkit}.md`; create +`docs/integrations/pkg-commandkit.md` for the implemented CLI integration. Inspect +selected public APIs, final module versions, source/engine loaders, standard suite +wiring, request execution, and reporting. + +**Work:** update supported integration/version references and Notarius's supplied +configuration, source contexts, runtime/request values, outcomes, and ownership. +Document the single application result/focused views, native prompt/schema +selection, continuation/admission, and standard CLI composition at the consumer +boundary. Link tagged upstream guides for native mechanics and canonical local +docs for product policy. Describe no unused future API, full copied upstream +manual, compatibility adapter, or private implementation as a public protocol. + +**Acceptance:** the integration docs agree with declared dependencies and current +calls; CommandKit/ckconfig/ckprompt adoption is complete, not deferred. Notices and +integration source links resolve to the intended components/revisions. + +**Validation:** inspect imports/call sites and local/upstream link destinations; +compare claims with Stage 37's assembled boundary tests. + +## Stage 45: Refresh Orientation And Reconcile Planning Status + +**Goal:** entry points lead to current docs and preserve separate future scope. + +**Inspect/change:** `docs/internal/overview.md`, `docs/development.md`, `README.md`, +and `docs/roadmap/future.md`, plus status in this plan and the feature roadmap. + +**Work:** update package inventory/navigation for the actual composition root, +producer helper, cache, and recorder. Remove obsolete receipt/LLM-package reading +directions. Give README the shortest valid native `notarius run` quickstart and +links to the two examples, current reference docs, and migration guide. Link the +new kit integration contract where useful. Reconcile backlog items implemented +by this cutover without adding deferred features. Mark stages complete only from +actual evidence; leave final verification pending until Stage 46. Do not delete +the active plan/feature roadmap or prepare a release in this stage. + +**Acceptance:** orientation describes implemented behavior and routes to canonical +owners. Future scope stays in the backlog; historical ADR/release/archive records +are not rewritten to match new syntax. + +**Validation:** check quickstart/example paths and navigation, then search current +docs/examples for stale names while retaining meaningful migration/history mentions. + +## Stage 46: Verify The Complete Native Cutover + +**Goal:** finish only when the full end state is supported by current evidence. + +**Inspect/change:** the assembled repository, feature roadmap acceptance matrix +and removal inventory, stage evidence, maintained examples, dependencies/notices, +and documentation links. Make final focused corrections where evidence exposes +an actual gap; do not invent another feature or compatibility layer. + +**Work:** verify every acceptance row against tests or direct inspection of its +owning boundary. Close compilation blockers recorded during intermediate stages. +Confirm all production LLM roles use the one engine/native path, domain artifacts +are unchanged, new standard commands/loaders share configuration, and no completed +decision remains deferred to a later CommandKit adoption. Confirm old adapters, +retry loops, global limits, schema registration, both legacy caches, required +reporting sidecars, and copied accounting have no current consumers. Review +dependency attribution and absence of permanent replacement modules. + +Run final development checks with declared dependencies and no sibling workspaces: + +```sh +GOWORK=off go test ./... +GOWORK=off go vet ./... +GOWORK=off go build ./cmd/notarius +git diff --check +``` + +Use a temporary binary destination if an ordinary build would leave an untracked +binary. Also run maintained example/preflight and representative native help/ +version/completion/format checks not already covered by the default suite. +Inspect removal matches in current Go/config/example files, rather than requiring +historical docs to contain none. Check local links with a small read-only checker +or manual verification if the repository has no link-check facility. + +**Acceptance:** final tests/vet/build pass; retained domain regressions and all +meaningful cross-boundary cases have coverage; examples and canonical docs agree +with the executable. Native timeout/cancellation, structural-failure status, +final-turn assessment, fallback disclosure/eligibility, cache hits, worker liveness, +and post-publication errors are verified. No live-model run, production key, +release publication, permanent new gate, or comprehensive performance benchmark +is required. Report any genuinely unverified result instead of declaring success. + +## Coverage Map + +| Feature-roadmap target | Primary stages | +| --- | --- | +| U1 dependency baseline and native handoff | 1–7, 35 | +| U2 embedded/native prompt and schema ownership | 4–5, 12–24, 35 | +| U3 typed modules and shared producer assessment | 8–12, 24 | +| U4 prompt pairs, guidance, complete revalidation | 9–23, 37 | +| U5 native retries and semantic-turn allowance | 3, 9–10, 13–24 | +| U6 strict/permissive outcomes and supported fallbacks | 3, 9–24, 27–28, 34, 37 | +| U7 workers and native backend admission | 3, 17, 25, 37 | +| U8 accepted-artifact and chunk-plan cache | 3, 11, 26–28, 37 | +| U9 native execution facts, artifact delivery, capture | 11, 29–31, 34, 37 | +| U10 complete native CLI and shared offline preflight | 5–7, 32–34, 36–37 | +| Successor decisions, canonical docs, migration, backlog | 38–45 | +| Entire acceptance matrix and removal inventory | 35, 37, 46 | + +No architectural decisions remain open for this implementation. Private API names, +debug filenames, and routine serialization choices stay within the selected +boundaries; they do not require another decision cycle or stronger contracts. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 1e0ccf1f..7839d2cc 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -8,13 +8,13 @@ presentation directly. Retain the application work of producing, assessing, publishing, and reusing typed artifacts. Historical duplication of library behavior is not a requirement to preserve it. -This is the single migration plan for PromptKit, ConfigKit, CommandKit, +This is the feature roadmap for PromptKit, ConfigKit, CommandKit, accepted-artifact caching, and artifact-first output. It owns the selected target, -decision status, implementation sequence, removal inventory, acceptance -criteria, and supporting evidence. The earlier discussion documents are retired; -Git history preserves the discussion. All implementation phases are pending. -Recording these decisions has not changed application behavior. Unrelated future -work remains in the [separate backlog](future.md). +decision status, removal inventory, acceptance criteria, and supporting evidence. +The earlier discussion documents are retired; Git history preserves the discussion. +The [implementation plan](implementation.md) owns the ordered, prompt-sized stages. +The migration is pending; recording these decisions has not changed application +behavior. Unrelated future work remains in the [separate backlog](future.md). Notarius predates many modern PromptKit capabilities: the library grew out of mechanics shared by Notarius and related applications. Historical duplication @@ -721,8 +721,8 @@ Record why fallback was used and expose that qualification in the ordinary completion report, even when diagnostic verbosity hides individual warnings. Use the same application outcome for reporting and optional debug detail. Keep conversation state in process, outside ordinary artifact/cache payloads; -explicitly requested detailed debug is separate. U6 is resolved; implementation -and module-specific construction details remain in Phases 3–4. +explicitly requested detailed debug is separate. U6 is resolved; module-specific +construction remains implementation work. ### Admission And Pipeline Workers (U7 Resolved) @@ -809,7 +809,7 @@ new shared limit, but its scope now includes lane completion. Do not automatical translate a cross-backend ceiling into per-backend limits or preserve worker inheritance/caps from that field. Provide a native limited-backend example with usable queue capacity, without requiring operators to override the upstream -queue default. U7 is resolved; implementation remains in Phase 4. +queue default. U7 is resolved; implementation remains pending. ### Accepted-Artifact Cache (U8 Resolved) @@ -955,8 +955,8 @@ source partition to request-content-based plan reuse. Remove checkpoint identities, dependency fingerprints, invocation-wide barriers, recovery bookkeeping, and the separate canonical-plan lookup/publication policy. Retain domain plan interpretation and materialization without retaining a second -cache mechanism. U8 is resolved; implementation remains pending in Phases 3–5 -using U1's selected upstream hash capability. +cache mechanism. U8 is resolved; implementation remains pending using U1's +selected upstream hash capability. ### Inspection And Execution Bookkeeping @@ -1221,36 +1221,36 @@ support, applicability, and outcome distinctions. U1–U10 are resolved. There are no remaining open architectural or dependency choices in this roadmap. Observer wiring, debug filenames, routine API placement, and serialization details remain implementation work within the selected designs. -IDs retain continuity with the earlier roadmap; all implementation phases remain -pending. +IDs retain continuity with the earlier roadmap. Implementation status and ordered +stages are tracked in the [implementation plan](implementation.md). **U1 resolved:** The [reviewed dependency set](#baseline-and-dependency-target) selects PromptKit v0.24.0, ConfigKit v1.4.0, ckconfig v0.10.0, ckprompt v0.11.0, and CommandKit v0.7.0. Native `MessageContentHash` satisfies the cache requirement; typed result handoff and focused inspection enable one application-owned load. -Adopt the native resource layout, schema selections, and `OutputContractInput` -API in Phase 2 and complete cache use in Phase 5. Upstream combination checks +The native resource layout, schema selections, and `OutputContractInput` +API are part of the selected target. Upstream combination checks passed; assembled Notarius acceptance remains implementation work. **U2 resolved:** Ordinary embedded assets, prompt-declared output contracts, native schema-source selection, and removal of duplicate schema/filesystem machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved). -Implementation remains in Phase 2; U4 selects prompt roles and contracts, and U5 +Implementation remains pending; U4 selects prompt roles and contracts, and U5 selects native repair delegation and the application correction allowance. **U3 resolved:** Typed stage boundaries, composition-root engine construction, and the shared pipeline producer helper with module functions and a pipeline assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). -Implement in Phase 3 and refine signatures during Phase 4 without reopening -ownership. U4/U5 select prompt/guidance and retry behavior; U6 selects terminal +Concrete signatures remain implementation details within that ownership. +U4/U5 select prompt/guidance and retry behavior; U6 selects terminal handling, configuration precedence, and module-owned fallbacks. **U4 resolved:** Domain-specific initial/correction prompt pairs, shared response schemas, default profile behavior, validator-owned guidance, and complete applicable-chain assessment are selected in [the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved). -Implement templates and operation-specific inputs in Phases 3–4; exact prose does -not reopen ownership. U5 selects budgets; U6 selects strict/permissive handling. +Exact template prose and operation-specific inputs remain implementation work +within that ownership. U5 selects budgets; U6 selects strict/permissive handling. **U5 resolved:** `max_additional_turns` defaults to `2`, with binding over pipeline precedence and explicit zero disabling correction. Native transient and @@ -1258,7 +1258,7 @@ structural-repair defaults remain upstream-owned; bundled prompts and applicatio requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-resolved) also removes generic non-LLM retries, all producer/validator `retries` fields, and the generic outer attempt loop. Any demonstrated transient I/O recovery stays -with its specific operation. Implement in Phases 3–4; U6 selects terminal handling. +with its specific operation. U6 selects terminal handling. **U6 resolved:** [Strict/permissive handling and module-owned fallbacks](#terminal-outcomes-and-module-fallbacks-u6-resolved) replace failure-category policies. Fallbacks satisfy artifact/downstream invariants @@ -1269,7 +1269,7 @@ application default, including normalization; there are no module-declared defaults. `mode` resolves module binding, then pipeline, then that default. Chunking falls back to one whole-transcript chunk; spell extraction uses its empty artifact, and normalization uses safe final-response groups or merged input. -Implement module-specific construction details in Phases 3–4. +Module-specific construction details remain implementation work. **U7 resolved:** [One pipeline worker pool and native admission](#admission-and-pipeline-workers-u7-resolved) replace the separate worker pools and application LLM scheduler. @@ -1277,7 +1277,7 @@ replace the separate worker pools and application LLM scheduler. jobs together, independently of CPU count and backend settings. PromptKit owns backend capacity with no injected application defaults; admission failures use U6's mode without extra retries or queues. Retire `total_llm` and the old extraction -worker field with explicit migration guidance. Implement in Phase 4. +worker field with explicit migration guidance. **U8 resolved:** [The unified accepted-result cache](#accepted-artifact-cache-u8-resolved) includes LLM chunk plans and replaces both legacy caches. Root-level @@ -1286,7 +1286,7 @@ includes LLM chunk plans and replaces both legacy caches. Root-level explicit `--cache`, module-binding `cache`, pipeline `cache`, then `auto`. `auto` reads/writes, `bypass` does neither, and `refresh` writes newly accepted results without reading. Cache I/O is best effort; legacy records remain untouched -and are not imported. Implement in Phases 3–5 using U1's selected native hash API. +and are not imported. U1 supplies the selected native hash API. **U9 resolved:** [Execution bookkeeping](#inspection-and-execution-bookkeeping) uses one native execution group per invocation, with all actual initial, @@ -1297,150 +1297,21 @@ fallbacks. Preserve run-specific domain artifact paths and schemas; retire index manifest, and reporting-only sidecars and migrate consumers to documented paths. Observer wiring and [debug filenames/layout](#inventory-and-depth) are explicitly implementation choices, not unresolved architecture or stable debug contracts. -Implement in Phases 3–5. +Implementation remains pending. **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. -The former later CommandKit phase is retired. CLI compatibility is a non-goal; +CommandKit belongs to the coordinated cutover. CLI compatibility is a non-goal; existing artifact schemas remain domain contracts. Any actual artifact-format change discovered during implementation needs an explicit compatibility decision, not an assumption that native CLI adoption authorizes it. -## Implementation Sequence +## Implementation Planning -All six phases are pending. Each delivers a reviewable outcome, not a separate -release or compatibility layer. Tests and canonical documentation accompany -behavior changes; the last phase verifies integration and completeness. Record -major boundary changes in successor ADRs without rewriting accepted decisions. - -### Phase 1: Establish The Migration Baseline - -- Use U1's reviewed dependency set and release/API evidence as the implementation - baseline; the upstream cache dependency is available and resolved. -- Map affected ADRs, current callers, contracts, and removal inventory. Carry - forward Audita evidence without adopting its application wrappers. -- Carry resolved U1–U10 designs into their dependent phases without reopening - ownership. Leave observer wiring and debug filenames to implementation. - -**Completion evidence:** reviewed dependency baseline, affected callers/ADRs, -scope, and removals are mapped to implementation work. The upstream review is -complete; applying those dependencies to Notarius and verifying the migration -remain separate work. - -### Phase 2: Build Native Composition, Sources, And Preflight - -- Apply U1's dependency set and implement U2: introduce configuration version 6, - mount native embedded/companion resources beneath `promptkit`, and replace the - old schema constructor options with native prompt selections and schema roots. - Use `OutputContractInput` for necessary runtime replacements. Omit bundled - structural-repair settings and remove their application override fields. -- Build the small CommandKit application with one application-owned ConfigKit - load, composed PromptKit rules, explicit bundled underlay, and `NewFromConfig` - engine construction. Share that result with native inspection and standard - suites; add pipeline declarations/native alias and `internal/preflight`. -- Add maintained embedded-only and operator-override examples. Remove duplicated - loading and asset assembly as callers migrate. - -**Completion evidence:** runtime/inspection/preflight share the same loaded result -and intended definitions without a second PromptKit load. Source contexts, shared -message paths, prompt-declared output contracts, selected schema roots, -source-relative schemas, strict peer conflicts, and replacement work offline. Help, -version, and completion need no configuration. Preflight checks known dependencies -without credentials or synthetic requests. `config diff` handles two selections. - -### Phase 3: Prove A Representative Native Pipeline - -- Implement the resolved U3–U5 collaboration and retry policy and U6's - strict-by-default model, `mode` precedence, and representative module fallbacks. - Connect module request/interpretation functions to the pipeline assessment - callback through the shared producer helper; refine concrete signatures here. - Implement domain-specific initial/correction prompts using the same schema and - default profile, supplied overrides, and validator-owned correction guidance. - Resolve `max_additional_turns` with omission/zero distinctions and pass it to - native coordination; inherit native transport and structural-repair defaults. - Treat a native failed-validation result as a terminal failure even when its - Go error is nil, including in continuation and validator paths. -- Exercise an LLM validator on the same engine and normalization with fixed - handles and fresh plan application. Wire U9's single invocation execution group - with application scope for initial, continuation, and validator executions. -- Run through the new command surface and alias, using the small completion - report, native outcomes/streams, and useful debug capture from the outset. - Begin the resolved U8 cache integration and U9 reporting/capture implementation - without a provisional local message hash. - -**Completion evidence:** fake-provider workflows exercise accumulated history, -collection of independent validator findings without rejection short-circuiting, -full candidate revalidation through the same assessor, final-turn acceptance -without duplicate final validation, strict failure, permissive fallback, and useful -failures. Native failed-validation results never become accepted artifacts or -trigger an application extension of structural repair. Unavailable assessments -remain distinct from actionable defects. The executable publishes artifacts and -reports them through native presentation -without the private completion/correction API. - -### Phase 4: Migrate All Roles And Remove Superseded Machinery - -- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve - deterministic paths and ordered validation. Apply U4's prompt/guidance design - and U5's retry policy across roles; implement U6's module fallbacks and U7's - shared worker pool and native admission handling. -- Remove the adapter, copied completion contracts, manual replay/capability checks, - generic outer attempt loop, all producer/validator `retries` fields, global - scheduler/ceiling, and coupled worker settings. -- Replace failure-category policy fields with the selected mode configuration - and module-owned fallback behavior. -- Finish native command/inspection migration and remove custom dispatch, flags, - encoders, verbosity handling, and obsolete asset registration. Retire the - emptied `internal/framework/llm` package after moving engine assembly to the - composition root and producer collaboration to the pipeline. - -**Completion evidence:** every role uses the shared native engine and selected -admission policy. One worker limit bounds extraction and lane completion together, -including serial operation without dependency or nested-validator deadlock. -No production path depends on the private completion API, -latest-response-only correction, generic stage retries, global ceiling, or legacy -CLI infrastructure. Obsolete tests retire with the machinery; surviving domain -requirements remain. - -### Phase 5: Complete Cache, Output, And Capture Integration - -- Implement U8's unified lookup using the native prepared `MessageContentHash`, - accepted-artifact/plan storage under that initial hash, directory and mode - precedence, and best-effort cache I/O. Retire both legacy cache mechanisms. -- Complete U9's artifact-first publication/report contract. Preserve domain paths - and schemas, add `fallback_count`, and retire index/reporting sidecars. Remove - checkpoint fingerprint graphs, mandatory manifests/receipts, durable execution - ledgers, and duplicate accounting; do not retain them to populate summaries. -- Complete capture inventory, per-operation evidence, directory/empty-value - behavior, `latest`, and recording/finalization failure handling. - -**Completion evidence:** downstream failure/fix/retry reuses accepted predecessors; -message changes miss and settings-only changes can hit; corrected answers use the -initial key; hits make no model calls or fresh validation claims. Fallbacks and -affected downstream work skip cache lookup/storage; unaffected accepted work -remains reusable. Chunk plans use the same cache, rematerialize against current -source, and no longer pin one partition across changed messages. Mode precedence, -cache I/O failures, and leaving both legacy stores untouched are verified. -Artifacts need no metadata bundle and consumers use documented domain paths. -Machine output and debug capture obey the selected contracts, including omitted -zero/accurate positive fallback counts, failures after publication, and quiet-mode -capture. Observer wiring and debug organization satisfy the selected scope and -inventory without becoming separate public contracts. - -### Phase 6: Verify And Document The Coordinated Cutover - -- Complete the acceptance checks and removal inventory, including dependency - cleanup, maintained examples, and the assembled command surface. -- Finish release migration guidance, attribution, successor ADRs, and affected - canonical documentation. Update this roadmap's status and reconcile the - separate future-work backlog. -- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash - policy, or obsolete metadata requirement remains in the finished migration. - -**Completion evidence:** PromptKit, ConfigKit, and CommandKit adoption, accepted- -artifact reuse, artifact delivery, diagnostics, and maintained offline workflows -are verified together. Remaining work is explicitly separate future scope. -Release preparation/publication is governed separately by release policy. +The [implementation plan](implementation.md) owns execution order, repository +touchpoints, stage acceptance criteria, and validation commands. This roadmap +defines the complete target and acceptance criteria. Release preparation and +publication remain separate work. ## Removal Inventory @@ -1490,7 +1361,7 @@ retry tests or freeze complete help text and native debug snapshots. | Execution observations | One native invocation group enrolls actual initial/continuation/LLM validator executions once, with application scope; native totals preserve unknown-versus-zero usage without application accumulators or overlapping-group aggregation. Result-less failures remain observable; cache hits add no executions/history; native execution success remains distinct from acceptance. | | Debug controls/lifecycle | Default/CLI/config/explicit-empty paths and native mode ordering; quiet capture; allocation before engine/preflight; start-time relative `latest`, no completion retarget; disabled runs leave it untouched; capture failures preserve primary causes and publication facts. | | Debug evidence | Available input/source/native snapshots, readable text, all visible turns, assessments, normalization and deterministic outputs, cache hits, and failed-run evidence. Cache hits retain prepared details without an execution event, duplicate preparation, or model call; no invented provider transcript. ConfigKit secrets/native credentials omitted, no heuristic content scrubbing. | -| End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit phase. | +| End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit adoption. | Run focused checks during implementation and the repository-wide checks for the completed cutover under the [testing policy](../policy/testing.md): @@ -1562,7 +1433,8 @@ Record significant changes in successor ADRs, including the [canonical chunk-plan policy](../adr/0005-cache-canonical-chunk-plans-by-source.md), admission, caching, and CLI/reporting ownership. Preserve accepted decision text and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records). -This roadmap owns implementation status; Git history retains earlier discussions. +This roadmap owns the target and decision rationale; the implementation plan +owns execution stages and their status. Git history retains earlier discussions. ## Evidence And Reading