2026-10-04 15:24:35 +00:00
|
|
|
|
# PromptKit Configuration Migration Review Remediation
|
|
|
|
|
|
|
|
|
|
|
|
## Scope And Overall Assessment
|
|
|
|
|
|
|
|
|
|
|
|
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
|
|
|
|
|
|
the accepted [feature roadmap](promptkit.md), and
|
|
|
|
|
|
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
|
|
|
|
|
|
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
|
|
|
|
|
|
PromptKit `/v3` catalogs at v3.20260930.0.
|
|
|
|
|
|
|
|
|
|
|
|
The migration is substantially implemented. Native sources, combined operator
|
|
|
|
|
|
documents and imports, recursive overlays, source-relative prompt content,
|
|
|
|
|
|
captured engine lifetime, immutable checkpoint identity, and credential-free
|
|
|
|
|
|
operator inspection are in place. The provider-neutral execution boundary,
|
|
|
|
|
|
prepared execution, retries, scheduling, and durable artifact contracts remain
|
|
|
|
|
|
intact. All 16 migrated prompt bodies match the previous definitions after
|
|
|
|
|
|
removing their old embedded identity fields; prompt fragments and schemas are
|
|
|
|
|
|
unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
Four bounded follow-ups remain below. No production code is changed by this
|
|
|
|
|
|
review. No additional runtime security restriction, mandatory credential check,
|
|
|
|
|
|
filesystem permission policy, compatibility shim, or release gate was found.
|
|
|
|
|
|
Eager final-definition validation and the external keyed format are intentional
|
|
|
|
|
|
upstream changes, not accidental strengthening. The previously recorded
|
|
|
|
|
|
ConfigKit remediation was completed before this work set; it is not outstanding.
|
|
|
|
|
|
|
|
|
|
|
|
## Finding 1 — Compose Registered Application Assets In One Strict Group
|
|
|
|
|
|
|
|
|
|
|
|
Priority: P2. Category: architecture/correctness gap. Status: open.
|
|
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
|
|
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
|
|
|
|
|
|
registered prompt or fallback-profile filesystem.
|
|
|
|
|
|
- `internal/framework/llm/asset_registry_sources_test.go`,
|
|
|
|
|
|
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
|
|
|
|
|
|
separate registrations yield separate underlays, encoding this divergence.
|
|
|
|
|
|
- The plan's Sources And Engine Ownership section calls for one application
|
|
|
|
|
|
underlay whose inputs each retain their own filesystem context;
|
|
|
|
|
|
`docs/internal/llm.md:86–92` also describes that behavior.
|
|
|
|
|
|
|
|
|
|
|
|
Separate underlays have recursive overlay semantics. Conflicting application
|
|
|
|
|
|
definitions in different registrations therefore silently depend on module
|
|
|
|
|
|
registration order instead of receiving native strict-composition diagnostics.
|
|
|
|
|
|
An offline probe registered two fallback files with the same profile key and
|
|
|
|
|
|
different models. The current inspector accepted them and selected the second
|
|
|
|
|
|
model. The exact same filesystem inputs placed in one native underlay correctly
|
|
|
|
|
|
failed composition. Distinct filesystem contexts do not require separate layers.
|
|
|
|
|
|
|
|
|
|
|
|
Required changes:
|
|
|
|
|
|
|
|
|
|
|
|
- Gather the native `FilesystemInputs` from every application prompt and
|
|
|
|
|
|
fallback-profile registration into one application underlay. Preserve each
|
|
|
|
|
|
input's existing label, filesystem, root, and disabled-import setting. Keep
|
|
|
|
|
|
schemas on their separate runtime option and omit an empty application group.
|
|
|
|
|
|
- Keep the operator and explicit local-backend contributions as later groups.
|
|
|
|
|
|
Preserve their accepted recursive overrides and source precedence. Do not
|
|
|
|
|
|
add an application-specific duplicate-key detector or flatten filesystem
|
|
|
|
|
|
contexts into an artificial shared root.
|
|
|
|
|
|
- Replace the incidental group-count test with behavioral coverage: conflicting
|
|
|
|
|
|
application contributions fail through native strict composition; distinct
|
|
|
|
|
|
declarations retain source-relative content; an operator overlay over the
|
|
|
|
|
|
resulting application base still succeeds. Reuse existing origin coverage.
|
|
|
|
|
|
|
|
|
|
|
|
Acceptance: application conflicts are not silently resolved by registration
|
|
|
|
|
|
order; source-relative imports/content and valid later operator overlays remain
|
|
|
|
|
|
unchanged. Runtime, preflight, and reports use the corrected shared construction.
|
|
|
|
|
|
|
|
|
|
|
|
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
|
|
|
|
|
|
|
|
|
|
|
|
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
|
|
|
|
|
|
|
|
|
|
|
|
Priority: P3. Category: CLI correctness. Status: open.
|
|
|
|
|
|
|
|
|
|
|
|
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
|
|
|
|
|
|
Notarius configuration and constructs the inspection engine before validating
|
|
|
|
|
|
the pointer; the default Notarius scope parses the pointer first.
|
|
|
|
|
|
|
|
|
|
|
|
Reproduction, using the native temporary build and a nonexistent config path:
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
The current PromptKit command returns exit 1 with a config-file error. The same
|
|
|
|
|
|
malformed pointer returns exit 2 with a valid config, and the Notarius scope
|
|
|
|
|
|
returns exit 2 even with the missing config. A bad PromptKit definition or
|
|
|
|
|
|
missing content file can likewise mask malformed command syntax. This differs
|
|
|
|
|
|
from the selected syntax/runtime distinction in the plan and CLI reference.
|
|
|
|
|
|
|
|
|
|
|
|
Required changes:
|
|
|
|
|
|
|
|
|
|
|
|
- Validate canonical JSON Pointer syntax once, before either scope loads config
|
|
|
|
|
|
or constructs an engine. Reuse the existing parser; retain defensive parsing
|
|
|
|
|
|
inside the adapter for direct callers.
|
|
|
|
|
|
- Keep the empty root pointer, escaped literal keys, historical-only paths,
|
|
|
|
|
|
unknown-path exit 1, and writer-error behavior unchanged.
|
|
|
|
|
|
- Extend CLI coverage with a malformed pointer and unavailable configuration;
|
|
|
|
|
|
verify exit 2, no stdout, and no configuration/provider work. No broader
|
|
|
|
|
|
argument-parser refactor is needed.
|
|
|
|
|
|
|
|
|
|
|
|
Acceptance: malformed pointers are syntax errors independent of configuration
|
|
|
|
|
|
availability for both scopes; valid selectors still load and inspect normally.
|
|
|
|
|
|
|
|
|
|
|
|
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
|
|
|
|
|
|
|
|
|
|
|
|
## Finding 3 — Correct Lifecycle And Fallback Documentation
|
|
|
|
|
|
|
|
|
|
|
|
Priority: P3. Category: documentation accuracy. Status: open.
|
|
|
|
|
|
|
|
|
|
|
|
Evidence:
|
|
|
|
|
|
|
|
|
|
|
|
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
|
|
|
|
|
|
at engine construction. Schemas remain preparation-time resources supplied
|
|
|
|
|
|
through `WithSchemaFS`, not captured configuration definitions.
|
|
|
|
|
|
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
|
|
|
|
|
|
as occurring without loading a prompt. Construction now loads and validates
|
|
|
|
|
|
effective prompt definitions and captures message files/static templates;
|
|
|
|
|
|
it does not prepare a request or generate output.
|
|
|
|
|
|
- The comment on `RegisterFallbackProfileFS` in
|
|
|
|
|
|
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
|
|
|
|
|
|
are used only when the operator source lacks the ID. Under recursive overlays,
|
|
|
|
|
|
their omitted fields remain active beneath a matching operator entry.
|
|
|
|
|
|
|
|
|
|
|
|
Required changes:
|
|
|
|
|
|
|
|
|
|
|
|
- Distinguish constructor-time definition/message capture from preparation-time
|
|
|
|
|
|
schema loading and request-variable/artifact resolution. State accurately
|
|
|
|
|
|
that preflight needs no transcript, credential value, or provider call; do
|
|
|
|
|
|
not promise it performs no prompt-content file reads.
|
|
|
|
|
|
- Describe fallback registration as a lower-precedence application contribution
|
|
|
|
|
|
rather than an all-or-nothing lookup fallback.
|
|
|
|
|
|
- While editing this text, clarify that an operator `backends.local` declaration
|
|
|
|
|
|
can supply the backend as well as Notarius's optional `local_backend` object.
|
|
|
|
|
|
The latter is a convenience contribution, not a required second declaration.
|
|
|
|
|
|
Link canonical source rules rather than duplicating them.
|
|
|
|
|
|
- Change documentation/comments only unless verification exposes a real runtime
|
|
|
|
|
|
defect. Do not move schema loading earlier to make the stale prose true.
|
|
|
|
|
|
|
|
|
|
|
|
Acceptance: public and internal guidance consistently explains actual lifecycle,
|
|
|
|
|
|
native layering, and operator backend configuration without inventing a stronger
|
|
|
|
|
|
preflight guarantee or an additional required config field.
|
|
|
|
|
|
|
|
|
|
|
|
Validation: inspect current constructors/prepared execution and upstream v0.12
|
|
|
|
|
|
contracts; review affected local links; run `git diff --check`. Reuse unchanged
|
|
|
|
|
|
behavioral checks rather than adding prose-snapshot tests.
|
|
|
|
|
|
|
|
|
|
|
|
## Finding 4 — Preserve ADR-0010's Historical Decision Body
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
Priority: P3. Category: documentation-policy compliance. Status: open.
|
|
|
|
|
|
|
|
|
|
|
|
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
|
|
|
|
|
|
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
|
|
|
|
|
|
and `:47–50`, as well as adding the appropriate status/supersession link.
|
|
|
|
|
|
`docs/policy/documentation.md` requires accepted decision content to remain
|
|
|
|
|
|
immutable; Stage 14 specified a concise status link rather than rewriting the
|
|
|
|
|
|
historical body.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
The replacement correctly acknowledges the old behavior, but the historical
|
|
|
|
|
|
record should retain its original wording. ADR-0019 already owns the new native
|
|
|
|
|
|
overlay decision and accurately limits which part of ADR-0010 it supersedes.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
|
|
|
|
|
|
keeping its new partial-supersession status and ADR-0019 link. Leave workload
|
|
|
|
|
|
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
|
|
|
|
|
|
in the canonical current references, not retrofitted into the older decision.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
Acceptance: the historical body matches the accepted original, the changed
|
|
|
|
|
|
clause's supersession remains discoverable, and no runtime behavior changes.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
|
|
|
|
|
|
verify the ADR link, and run `git diff --check`.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
2026-10-04 15:24:35 +00:00
|
|
|
|
## Review Evidence And Validation
|
2026-10-04 00:18:32 +00:00
|
|
|
|
|
|
|
|
|
|
- `GOWORK=off go test ./...` passed.
|
|
|
|
|
|
- `GOWORK=off go vet ./...` passed.
|
2026-10-04 15:24:35 +00:00
|
|
|
|
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
|
2026-10-04 00:18:32 +00:00
|
|
|
|
- `GOWORK=off go mod tidy -diff` produced no changes.
|
2026-10-04 15:24:35 +00:00
|
|
|
|
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
|
|
|
|
|
|
- A native build outside the repository succeeded. Credential-free profile and
|
|
|
|
|
|
explanation commands and complete/development pipeline validation succeeded.
|
|
|
|
|
|
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
|
|
|
|
|
|
are unchanged after removing old embedded IDs/versions. Prompt fragments,
|
|
|
|
|
|
schemas, and the embedded D&D execution settings are unchanged.
|
|
|
|
|
|
- An offline Go overlay probe, stored entirely outside the repository, confirmed
|
|
|
|
|
|
the application-group conflict behavior in Finding 1. CLI invocations
|
|
|
|
|
|
reproduced Finding 2 without generation or provider calls.
|
|
|
|
|
|
- The implementation diff passed whitespace checks. No production inference
|
|
|
|
|
|
or production model-quality evaluation was performed.
|
|
|
|
|
|
|
|
|
|
|
|
Recommended remediation is limited to these findings, not another configuration
|
|
|
|
|
|
system, new security policy, or unrelated pipeline refactoring.
|