diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 7930316d..561bc76a 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -1,108 +1,193 @@ -# ConfigKit Inheritance Review Remediation +# PromptKit Configuration Migration Review Remediation -## Review Scope And Overall Result +## Scope And Overall Assessment -Reviewed on 2026-10-04, comparing implementation commits `ec99d48` through -`6450994` with the baseline `9e74df0`, the ConfigKit feature roadmap, and its -eight-stage implementation plan. +Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`, +the accepted [feature roadmap](promptkit.md), and +[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at +`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both +PromptKit `/v3` catalogs at v3.20260930.0. -The architecture matches the intended design. ConfigKit v0.5.0 owns source -composition and native sibling inheritance before decoding. Notarius retains -its typed defaults, presence-sensitive validation, declaration-relative path -normalization, detached runtime configuration, catalog-dependent resolution, -and private redacted inspection boundary. The production configuration adapter -needed no changes; the work adds the dependency update, integration tests, -documentation, and the derived development pipeline example. +The migration is substantially implemented. Native sources, combined operator +documents and imports, recursive overlays, source-relative prompt content, +captured engine lifetime, immutable checkpoint identity, and credential-free +operator inspection are in place. The provider-neutral execution boundary, +prepared execution, retries, scheduling, and durable artifact contracts remain +intact. All 16 migrated prompt bodies match the previous definitions after +removing their old embedded identity fields; prompt fragments and schemas are +unchanged. -No production regression or inadvertently strengthened requirement was found. -In particular, the implementation adds no inheritance whitelist, new schema -version, custom merging, path-containment rule, template exemption, or release -gate. The newly reserved `extends` key is the expressly adopted upstream -compatibility change. +Four bounded follow-ups remain below. No production code is changed by this +review. No additional runtime security restriction, mandatory credential check, +filesystem permission policy, compatibility shim, or release gate was found. +Eager final-definition validation and the external keyed format are intentional +upstream changes, not accidental strengthening. The previously recorded +ConfigKit remediation was completed before this work set; it is not outstanding. -Two follow-ups were identified: a path-test blind spot and an incomplete -documented command. Completion is recorded under each finding. This document -records remediation work only; it does not authorize -unrelated runtime refactoring. +## Finding 1 — Compose Registered Application Assets In One Strict Group -## Finding 1 — Make The Path Fixture Distinguish Declaring Directories +Priority: P2. Category: architecture/correctness gap. Status: open. -Priority: P2. Category: verification gap. +Evidence: -Status: addressed on 2026-10-04. The fixture now uses distinct parent-, child-, -and overlay-local paths for the source-base assertions and retains a separate -`../` reference. `GOWORK=off go test ./internal/core/config` passed. No -production resolver change was needed. +- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per + registered prompt or fallback-profile filesystem. +- `internal/framework/llm/asset_registry_sources_test.go`, + `TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that + separate registrations yield separate underlays, encoding this divergence. +- The plan's Sources And Engine Ownership section calls for one application + underlay whose inputs each retain their own filesystem context; + `docs/internal/llm.md:86–92` also describes that behavior. -Evidence: `internal/core/config/inheritance_test.go:338–436`, -`TestInheritedReferencePathsKeepDeclaringSource`. +Separate underlays have recursive overlay semantics. Conflicting application +definitions in different registrations therefore silently depend on module +registration order instead of receiving native strict-composition diagnostics. +An offline probe registered two fallback files with the same profile key and +different models. The current inspector accepted them and selected the second +model. The exact same filesystem inputs placed in one native underlay correctly +failed composition. Distinct filesystem contexts do not require separate layers. -The parent, child, and overlay files live in `root/parents`, `root/children`, -and `root/overlays`. Their resource paths all use `../fixtures/`, and -the expected paths all point into `root/fixtures`. Resolving one of these -paths from any of the three directories yields the same result. Thus, choosing -the child's directory for an inherited step/nested reference, or choosing the -parent's directory for an overlay replacement, could still pass this test. +Required changes: -`TestInheritedInspectionRetainsOriginsRedactionAndOwnership` does distinguish -the parent directory for a simple inherited `./shared.txt` reference. It does -not close the blind spot for the nested, ordered-step, direct-child replacement, -and overlay cases in the larger path fixture. +- Gather the native `FilesystemInputs` from every application prompt and + fallback-profile registration into one application underlay. Preserve each + input's existing label, filesystem, root, and disabled-import setting. Keep + schemas on their separate runtime option and omit an empty application group. +- Keep the operator and explicit local-backend contributions as later groups. + Preserve their accepted recursive overrides and source precedence. Do not + add an application-specific duplicate-key detector or flatten filesystem + contexts into an artificial shared root. +- Replace the incidental group-count test with behavioral coverage: conflicting + application contributions fail through native strict composition; distinct + declarations retain source-relative content; an operator overlay over the + resulting application base still succeeds. Reuse existing origin coverage. -Required change: +Acceptance: application conflicts are not silently resolved by registration +order; source-relative imports/content and valid later operator overlays remain +unchanged. Runtime, preflight, and reports use the corrected shared construction. -- Revise the existing path fixture to use directory-distinguishing relative - paths for these cases, such as `./shared.txt`, `./step.txt`, and a child-local - replacement. Assert inherited paths against the parent's directory, direct - replacements against the child's directory, and overlay-supplied paths - against the overlay's directory. -- Retain a separate legitimate `../` example to protect the existing host-path - behavior. Avoid making every path converge on one shared sibling directory. -- Keep this as a focused real-file configuration test. Change production - resolution only if the improved assertions expose an actual defect. +Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`. -Acceptance: choosing a parent, child, or overlay directory incorrectly produces -a different absolute path and fails the relevant assertion. Nested inheritance, -inherited ordered-step references, child replacements, and overlay replacements -still pass with the actual resolver. +## Finding 2 — Validate Explanation Syntax Before Loading Configuration -Validation: `GOWORK=off go test ./internal/core/config`. +Priority: P3. Category: CLI correctness. Status: open. -## Finding 2 — Supply Input In The Development Pipeline Command +Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads +Notarius configuration and constructs the inspection engine before validating +the pointer; the default Notarius scope parses the pointer first. -Priority: P3. Category: documentation accuracy. +Reproduction, using the native temporary build and a nonexistent config path: -Status: addressed on 2026-10-04. The example now supplies the pipeline ID, -configuration path, and transcript input path, and links to the CLI run -reference. The flags were checked against that reference, and -`GOWORK=off go test ./internal/cli` passed without a live model request. +```sh +notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2' +``` -Evidence: `docs/config.md:43–49`, in Maintained Examples. +The current PromptKit command returns exit 1 with a config-file error. The same +malformed pointer returns exit 2 with a valid config, and the Notarius scope +returns exit 2 even with the missing config. A bad PromptKit definition or +missing content file can likewise mask malformed command syntax. This differs +from the selected syntax/runtime distinction in the plan and CLI reference. -The new example says to select the development pipeline with -`notarius run dnd-session-development --config /path/to/dnd-complete.config.yml`. -The run command requires `--input`. Executing the equivalent command against -the maintained configuration returns `notarius: run requires --input` with CLI -exit status 2, before pipeline execution. +Required changes: -Required change: include a transcript input placeholder in the invocation, -for example `--input /path/to/transcript.json`, and retain the link to the CLI -reference for the full invocation contract. No CLI behavior change or new test -framework is needed. +- Validate canonical JSON Pointer syntax once, before either scope loads config + or constructs an engine. Reuse the existing parser; retain defensive parsing + inside the adapter for direct callers. +- Keep the empty root pointer, escaped literal keys, historical-only paths, + unknown-path exit 1, and writer-error behavior unchanged. +- Extend CLI coverage with a malformed pointer and unavailable configuration; + verify exit 2, no stdout, and no configuration/provider work. No broader + argument-parser refactor is needed. -Acceptance: the documented command supplies both required run inputs—the -pipeline ID and input path—and uses the implemented flag names. Verify against -the CLI reference and existing parser coverage without a live model request. +Acceptance: malformed pointers are syntax errors independent of configuration +availability for both scopes; valid selectors still load and inspect normally. -## Checks Completed During Review +Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`. + +## Finding 3 — Correct Lifecycle And Fallback Documentation + +Priority: P3. Category: documentation accuracy. Status: open. + +Evidence: + +- `docs/config.md:321–323` says prompt **and schema** definitions are validated + at engine construction. Schemas remain preparation-time resources supplied + through `WithSchemaFS`, not captured configuration definitions. +- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight + as occurring without loading a prompt. Construction now loads and validates + effective prompt definitions and captures message files/static templates; + it does not prepare a request or generate output. +- The comment on `RegisterFallbackProfileFS` in + `internal/framework/llm/asset_registry.go:61–62` still says fallback assets + are used only when the operator source lacks the ID. Under recursive overlays, + their omitted fields remain active beneath a matching operator entry. + +Required changes: + +- Distinguish constructor-time definition/message capture from preparation-time + schema loading and request-variable/artifact resolution. State accurately + that preflight needs no transcript, credential value, or provider call; do + not promise it performs no prompt-content file reads. +- Describe fallback registration as a lower-precedence application contribution + rather than an all-or-nothing lookup fallback. +- While editing this text, clarify that an operator `backends.local` declaration + can supply the backend as well as Notarius's optional `local_backend` object. + The latter is a convenience contribution, not a required second declaration. + Link canonical source rules rather than duplicating them. +- Change documentation/comments only unless verification exposes a real runtime + defect. Do not move schema loading earlier to make the stale prose true. + +Acceptance: public and internal guidance consistently explains actual lifecycle, +native layering, and operator backend configuration without inventing a stronger +preflight guarantee or an additional required config field. + +Validation: inspect current constructors/prepared execution and upstream v0.12 +contracts; review affected local links; run `git diff --check`. Reuse unchanged +behavioral checks rather than adding prose-snapshot tests. + +## Finding 4 — Preserve ADR-0010's Historical Decision Body + +Priority: P3. Category: documentation-policy compliance. Status: open. + +Evidence: the migration diff rewrites ADR-0010's Decision and Consequences +paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33` +and `:47–50`, as well as adding the appropriate status/supersession link. +`docs/policy/documentation.md` requires accepted decision content to remain +immutable; Stage 14 specified a concise status link rather than rewriting the +historical body. + +The replacement correctly acknowledges the old behavior, but the historical +record should retain its original wording. ADR-0019 already owns the new native +overlay decision and accurately limits which part of ADR-0010 it supersedes. + +Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`, +keeping its new partial-supersession status and ADR-0019 link. Leave workload +ownership and ADR-0019 unchanged. Current runtime behavior remains documented +in the canonical current references, not retrofitted into the older decision. + +Acceptance: the historical body matches the accepted original, the changed +clause's supersession remains discoverable, and no runtime behavior changes. + +Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`, +verify the ADR link, and run `git diff --check`. + +## Review Evidence And Validation - `GOWORK=off go test ./...` passed. - `GOWORK=off go vet ./...` passed. -- `GOWORK=off go build` of `./cmd/notarius` to a temporary destination passed. +- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed. - `GOWORK=off go mod tidy -diff` produced no changes. -- The selected ConfigKit module is v0.5.0; PromptKit remains v0.11.2. -- The implementation diff passed whitespace checks. -- The incomplete development invocation was reproduced without provider calls. +- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins. +- A native build outside the repository succeeded. Credential-free profile and + explanation commands and complete/development pipeline validation succeeded. +- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies + are unchanged after removing old embedded IDs/versions. Prompt fragments, + schemas, and the embedded D&D execution settings are unchanged. +- An offline Go overlay probe, stored entirely outside the repository, confirmed + the application-group conflict behavior in Finding 1. CLI invocations + reproduced Finding 2 without generation or provider calls. +- The implementation diff passed whitespace checks. No production inference + or production model-quality evaluation was performed. -These checks describe the review baseline. Subsequent remediation and its -focused validation are recorded under each finding. +Recommended remediation is limited to these findings, not another configuration +system, new security policy, or unrelated pipeline refactoring.