Record PromptKit migration review findings
This commit is contained in:
@@ -1,108 +1,193 @@
|
|||||||
# ConfigKit Inheritance Review Remediation
|
# PromptKit Configuration Migration Review Remediation
|
||||||
|
|
||||||
## Review Scope And Overall Result
|
## Scope And Overall Assessment
|
||||||
|
|
||||||
Reviewed on 2026-10-04, comparing implementation commits `ec99d48` through
|
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
|
||||||
`6450994` with the baseline `9e74df0`, the ConfigKit feature roadmap, and its
|
the accepted [feature roadmap](promptkit.md), and
|
||||||
eight-stage implementation plan.
|
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
|
||||||
|
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
|
||||||
|
PromptKit `/v3` catalogs at v3.20260930.0.
|
||||||
|
|
||||||
The architecture matches the intended design. ConfigKit v0.5.0 owns source
|
The migration is substantially implemented. Native sources, combined operator
|
||||||
composition and native sibling inheritance before decoding. Notarius retains
|
documents and imports, recursive overlays, source-relative prompt content,
|
||||||
its typed defaults, presence-sensitive validation, declaration-relative path
|
captured engine lifetime, immutable checkpoint identity, and credential-free
|
||||||
normalization, detached runtime configuration, catalog-dependent resolution,
|
operator inspection are in place. The provider-neutral execution boundary,
|
||||||
and private redacted inspection boundary. The production configuration adapter
|
prepared execution, retries, scheduling, and durable artifact contracts remain
|
||||||
needed no changes; the work adds the dependency update, integration tests,
|
intact. All 16 migrated prompt bodies match the previous definitions after
|
||||||
documentation, and the derived development pipeline example.
|
removing their old embedded identity fields; prompt fragments and schemas are
|
||||||
|
unchanged.
|
||||||
|
|
||||||
No production regression or inadvertently strengthened requirement was found.
|
Four bounded follow-ups remain below. No production code is changed by this
|
||||||
In particular, the implementation adds no inheritance whitelist, new schema
|
review. No additional runtime security restriction, mandatory credential check,
|
||||||
version, custom merging, path-containment rule, template exemption, or release
|
filesystem permission policy, compatibility shim, or release gate was found.
|
||||||
gate. The newly reserved `extends` key is the expressly adopted upstream
|
Eager final-definition validation and the external keyed format are intentional
|
||||||
compatibility change.
|
upstream changes, not accidental strengthening. The previously recorded
|
||||||
|
ConfigKit remediation was completed before this work set; it is not outstanding.
|
||||||
|
|
||||||
Two follow-ups were identified: a path-test blind spot and an incomplete
|
## Finding 1 — Compose Registered Application Assets In One Strict Group
|
||||||
documented command. Completion is recorded under each finding. This document
|
|
||||||
records remediation work only; it does not authorize
|
|
||||||
unrelated runtime refactoring.
|
|
||||||
|
|
||||||
## Finding 1 — Make The Path Fixture Distinguish Declaring Directories
|
Priority: P2. Category: architecture/correctness gap. Status: open.
|
||||||
|
|
||||||
Priority: P2. Category: verification gap.
|
Evidence:
|
||||||
|
|
||||||
Status: addressed on 2026-10-04. The fixture now uses distinct parent-, child-,
|
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
|
||||||
and overlay-local paths for the source-base assertions and retains a separate
|
registered prompt or fallback-profile filesystem.
|
||||||
`../` reference. `GOWORK=off go test ./internal/core/config` passed. No
|
- `internal/framework/llm/asset_registry_sources_test.go`,
|
||||||
production resolver change was needed.
|
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
|
||||||
|
separate registrations yield separate underlays, encoding this divergence.
|
||||||
|
- The plan's Sources And Engine Ownership section calls for one application
|
||||||
|
underlay whose inputs each retain their own filesystem context;
|
||||||
|
`docs/internal/llm.md:86–92` also describes that behavior.
|
||||||
|
|
||||||
Evidence: `internal/core/config/inheritance_test.go:338–436`,
|
Separate underlays have recursive overlay semantics. Conflicting application
|
||||||
`TestInheritedReferencePathsKeepDeclaringSource`.
|
definitions in different registrations therefore silently depend on module
|
||||||
|
registration order instead of receiving native strict-composition diagnostics.
|
||||||
|
An offline probe registered two fallback files with the same profile key and
|
||||||
|
different models. The current inspector accepted them and selected the second
|
||||||
|
model. The exact same filesystem inputs placed in one native underlay correctly
|
||||||
|
failed composition. Distinct filesystem contexts do not require separate layers.
|
||||||
|
|
||||||
The parent, child, and overlay files live in `root/parents`, `root/children`,
|
Required changes:
|
||||||
and `root/overlays`. Their resource paths all use `../fixtures/<name>`, and
|
|
||||||
the expected paths all point into `root/fixtures`. Resolving one of these
|
|
||||||
paths from any of the three directories yields the same result. Thus, choosing
|
|
||||||
the child's directory for an inherited step/nested reference, or choosing the
|
|
||||||
parent's directory for an overlay replacement, could still pass this test.
|
|
||||||
|
|
||||||
`TestInheritedInspectionRetainsOriginsRedactionAndOwnership` does distinguish
|
- Gather the native `FilesystemInputs` from every application prompt and
|
||||||
the parent directory for a simple inherited `./shared.txt` reference. It does
|
fallback-profile registration into one application underlay. Preserve each
|
||||||
not close the blind spot for the nested, ordered-step, direct-child replacement,
|
input's existing label, filesystem, root, and disabled-import setting. Keep
|
||||||
and overlay cases in the larger path fixture.
|
schemas on their separate runtime option and omit an empty application group.
|
||||||
|
- Keep the operator and explicit local-backend contributions as later groups.
|
||||||
|
Preserve their accepted recursive overrides and source precedence. Do not
|
||||||
|
add an application-specific duplicate-key detector or flatten filesystem
|
||||||
|
contexts into an artificial shared root.
|
||||||
|
- Replace the incidental group-count test with behavioral coverage: conflicting
|
||||||
|
application contributions fail through native strict composition; distinct
|
||||||
|
declarations retain source-relative content; an operator overlay over the
|
||||||
|
resulting application base still succeeds. Reuse existing origin coverage.
|
||||||
|
|
||||||
Required change:
|
Acceptance: application conflicts are not silently resolved by registration
|
||||||
|
order; source-relative imports/content and valid later operator overlays remain
|
||||||
|
unchanged. Runtime, preflight, and reports use the corrected shared construction.
|
||||||
|
|
||||||
- Revise the existing path fixture to use directory-distinguishing relative
|
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
|
||||||
paths for these cases, such as `./shared.txt`, `./step.txt`, and a child-local
|
|
||||||
replacement. Assert inherited paths against the parent's directory, direct
|
|
||||||
replacements against the child's directory, and overlay-supplied paths
|
|
||||||
against the overlay's directory.
|
|
||||||
- Retain a separate legitimate `../` example to protect the existing host-path
|
|
||||||
behavior. Avoid making every path converge on one shared sibling directory.
|
|
||||||
- Keep this as a focused real-file configuration test. Change production
|
|
||||||
resolution only if the improved assertions expose an actual defect.
|
|
||||||
|
|
||||||
Acceptance: choosing a parent, child, or overlay directory incorrectly produces
|
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
|
||||||
a different absolute path and fails the relevant assertion. Nested inheritance,
|
|
||||||
inherited ordered-step references, child replacements, and overlay replacements
|
|
||||||
still pass with the actual resolver.
|
|
||||||
|
|
||||||
Validation: `GOWORK=off go test ./internal/core/config`.
|
Priority: P3. Category: CLI correctness. Status: open.
|
||||||
|
|
||||||
## Finding 2 — Supply Input In The Development Pipeline Command
|
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
|
||||||
|
Notarius configuration and constructs the inspection engine before validating
|
||||||
|
the pointer; the default Notarius scope parses the pointer first.
|
||||||
|
|
||||||
Priority: P3. Category: documentation accuracy.
|
Reproduction, using the native temporary build and a nonexistent config path:
|
||||||
|
|
||||||
Status: addressed on 2026-10-04. The example now supplies the pipeline ID,
|
```sh
|
||||||
configuration path, and transcript input path, and links to the CLI run
|
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
|
||||||
reference. The flags were checked against that reference, and
|
```
|
||||||
`GOWORK=off go test ./internal/cli` passed without a live model request.
|
|
||||||
|
|
||||||
Evidence: `docs/config.md:43–49`, in Maintained Examples.
|
The current PromptKit command returns exit 1 with a config-file error. The same
|
||||||
|
malformed pointer returns exit 2 with a valid config, and the Notarius scope
|
||||||
|
returns exit 2 even with the missing config. A bad PromptKit definition or
|
||||||
|
missing content file can likewise mask malformed command syntax. This differs
|
||||||
|
from the selected syntax/runtime distinction in the plan and CLI reference.
|
||||||
|
|
||||||
The new example says to select the development pipeline with
|
Required changes:
|
||||||
`notarius run dnd-session-development --config /path/to/dnd-complete.config.yml`.
|
|
||||||
The run command requires `--input`. Executing the equivalent command against
|
|
||||||
the maintained configuration returns `notarius: run requires --input` with CLI
|
|
||||||
exit status 2, before pipeline execution.
|
|
||||||
|
|
||||||
Required change: include a transcript input placeholder in the invocation,
|
- Validate canonical JSON Pointer syntax once, before either scope loads config
|
||||||
for example `--input /path/to/transcript.json`, and retain the link to the CLI
|
or constructs an engine. Reuse the existing parser; retain defensive parsing
|
||||||
reference for the full invocation contract. No CLI behavior change or new test
|
inside the adapter for direct callers.
|
||||||
framework is needed.
|
- Keep the empty root pointer, escaped literal keys, historical-only paths,
|
||||||
|
unknown-path exit 1, and writer-error behavior unchanged.
|
||||||
|
- Extend CLI coverage with a malformed pointer and unavailable configuration;
|
||||||
|
verify exit 2, no stdout, and no configuration/provider work. No broader
|
||||||
|
argument-parser refactor is needed.
|
||||||
|
|
||||||
Acceptance: the documented command supplies both required run inputs—the
|
Acceptance: malformed pointers are syntax errors independent of configuration
|
||||||
pipeline ID and input path—and uses the implemented flag names. Verify against
|
availability for both scopes; valid selectors still load and inspect normally.
|
||||||
the CLI reference and existing parser coverage without a live model request.
|
|
||||||
|
|
||||||
## Checks Completed During Review
|
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
|
||||||
|
|
||||||
|
## Finding 3 — Correct Lifecycle And Fallback Documentation
|
||||||
|
|
||||||
|
Priority: P3. Category: documentation accuracy. Status: open.
|
||||||
|
|
||||||
|
Evidence:
|
||||||
|
|
||||||
|
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
|
||||||
|
at engine construction. Schemas remain preparation-time resources supplied
|
||||||
|
through `WithSchemaFS`, not captured configuration definitions.
|
||||||
|
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
|
||||||
|
as occurring without loading a prompt. Construction now loads and validates
|
||||||
|
effective prompt definitions and captures message files/static templates;
|
||||||
|
it does not prepare a request or generate output.
|
||||||
|
- The comment on `RegisterFallbackProfileFS` in
|
||||||
|
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
|
||||||
|
are used only when the operator source lacks the ID. Under recursive overlays,
|
||||||
|
their omitted fields remain active beneath a matching operator entry.
|
||||||
|
|
||||||
|
Required changes:
|
||||||
|
|
||||||
|
- Distinguish constructor-time definition/message capture from preparation-time
|
||||||
|
schema loading and request-variable/artifact resolution. State accurately
|
||||||
|
that preflight needs no transcript, credential value, or provider call; do
|
||||||
|
not promise it performs no prompt-content file reads.
|
||||||
|
- Describe fallback registration as a lower-precedence application contribution
|
||||||
|
rather than an all-or-nothing lookup fallback.
|
||||||
|
- While editing this text, clarify that an operator `backends.local` declaration
|
||||||
|
can supply the backend as well as Notarius's optional `local_backend` object.
|
||||||
|
The latter is a convenience contribution, not a required second declaration.
|
||||||
|
Link canonical source rules rather than duplicating them.
|
||||||
|
- Change documentation/comments only unless verification exposes a real runtime
|
||||||
|
defect. Do not move schema loading earlier to make the stale prose true.
|
||||||
|
|
||||||
|
Acceptance: public and internal guidance consistently explains actual lifecycle,
|
||||||
|
native layering, and operator backend configuration without inventing a stronger
|
||||||
|
preflight guarantee or an additional required config field.
|
||||||
|
|
||||||
|
Validation: inspect current constructors/prepared execution and upstream v0.12
|
||||||
|
contracts; review affected local links; run `git diff --check`. Reuse unchanged
|
||||||
|
behavioral checks rather than adding prose-snapshot tests.
|
||||||
|
|
||||||
|
## Finding 4 — Preserve ADR-0010's Historical Decision Body
|
||||||
|
|
||||||
|
Priority: P3. Category: documentation-policy compliance. Status: open.
|
||||||
|
|
||||||
|
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
|
||||||
|
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
|
||||||
|
and `:47–50`, as well as adding the appropriate status/supersession link.
|
||||||
|
`docs/policy/documentation.md` requires accepted decision content to remain
|
||||||
|
immutable; Stage 14 specified a concise status link rather than rewriting the
|
||||||
|
historical body.
|
||||||
|
|
||||||
|
The replacement correctly acknowledges the old behavior, but the historical
|
||||||
|
record should retain its original wording. ADR-0019 already owns the new native
|
||||||
|
overlay decision and accurately limits which part of ADR-0010 it supersedes.
|
||||||
|
|
||||||
|
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
|
||||||
|
keeping its new partial-supersession status and ADR-0019 link. Leave workload
|
||||||
|
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
|
||||||
|
in the canonical current references, not retrofitted into the older decision.
|
||||||
|
|
||||||
|
Acceptance: the historical body matches the accepted original, the changed
|
||||||
|
clause's supersession remains discoverable, and no runtime behavior changes.
|
||||||
|
|
||||||
|
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
|
||||||
|
verify the ADR link, and run `git diff --check`.
|
||||||
|
|
||||||
|
## Review Evidence And Validation
|
||||||
|
|
||||||
- `GOWORK=off go test ./...` passed.
|
- `GOWORK=off go test ./...` passed.
|
||||||
- `GOWORK=off go vet ./...` passed.
|
- `GOWORK=off go vet ./...` passed.
|
||||||
- `GOWORK=off go build` of `./cmd/notarius` to a temporary destination passed.
|
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
|
||||||
- `GOWORK=off go mod tidy -diff` produced no changes.
|
- `GOWORK=off go mod tidy -diff` produced no changes.
|
||||||
- The selected ConfigKit module is v0.5.0; PromptKit remains v0.11.2.
|
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
|
||||||
- The implementation diff passed whitespace checks.
|
- A native build outside the repository succeeded. Credential-free profile and
|
||||||
- The incomplete development invocation was reproduced without provider calls.
|
explanation commands and complete/development pipeline validation succeeded.
|
||||||
|
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
|
||||||
|
are unchanged after removing old embedded IDs/versions. Prompt fragments,
|
||||||
|
schemas, and the embedded D&D execution settings are unchanged.
|
||||||
|
- An offline Go overlay probe, stored entirely outside the repository, confirmed
|
||||||
|
the application-group conflict behavior in Finding 1. CLI invocations
|
||||||
|
reproduced Finding 2 without generation or provider calls.
|
||||||
|
- The implementation diff passed whitespace checks. No production inference
|
||||||
|
or production model-quality evaluation was performed.
|
||||||
|
|
||||||
These checks describe the review baseline. Subsequent remediation and its
|
Recommended remediation is limited to these findings, not another configuration
|
||||||
focused validation are recorded under each finding.
|
system, new security policy, or unrelated pipeline refactoring.
|
||||||
|
|||||||
Reference in New Issue
Block a user