194 lines
10 KiB
Markdown
194 lines
10 KiB
Markdown
# PromptKit Configuration Migration Review Remediation
|
||
|
||
## Scope And Overall Assessment
|
||
|
||
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
|
||
the accepted [feature roadmap](promptkit.md), and
|
||
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
|
||
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
|
||
PromptKit `/v3` catalogs at v3.20260930.0.
|
||
|
||
The migration is substantially implemented. Native sources, combined operator
|
||
documents and imports, recursive overlays, source-relative prompt content,
|
||
captured engine lifetime, immutable checkpoint identity, and credential-free
|
||
operator inspection are in place. The provider-neutral execution boundary,
|
||
prepared execution, retries, scheduling, and durable artifact contracts remain
|
||
intact. All 16 migrated prompt bodies match the previous definitions after
|
||
removing their old embedded identity fields; prompt fragments and schemas are
|
||
unchanged.
|
||
|
||
Four bounded follow-ups remain below. No production code is changed by this
|
||
review. No additional runtime security restriction, mandatory credential check,
|
||
filesystem permission policy, compatibility shim, or release gate was found.
|
||
Eager final-definition validation and the external keyed format are intentional
|
||
upstream changes, not accidental strengthening. The previously recorded
|
||
ConfigKit remediation was completed before this work set; it is not outstanding.
|
||
|
||
## Finding 1 — Compose Registered Application Assets In One Strict Group
|
||
|
||
Priority: P2. Category: architecture/correctness gap. Status: open.
|
||
|
||
Evidence:
|
||
|
||
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
|
||
registered prompt or fallback-profile filesystem.
|
||
- `internal/framework/llm/asset_registry_sources_test.go`,
|
||
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
|
||
separate registrations yield separate underlays, encoding this divergence.
|
||
- The plan's Sources And Engine Ownership section calls for one application
|
||
underlay whose inputs each retain their own filesystem context;
|
||
`docs/internal/llm.md:86–92` also describes that behavior.
|
||
|
||
Separate underlays have recursive overlay semantics. Conflicting application
|
||
definitions in different registrations therefore silently depend on module
|
||
registration order instead of receiving native strict-composition diagnostics.
|
||
An offline probe registered two fallback files with the same profile key and
|
||
different models. The current inspector accepted them and selected the second
|
||
model. The exact same filesystem inputs placed in one native underlay correctly
|
||
failed composition. Distinct filesystem contexts do not require separate layers.
|
||
|
||
Required changes:
|
||
|
||
- Gather the native `FilesystemInputs` from every application prompt and
|
||
fallback-profile registration into one application underlay. Preserve each
|
||
input's existing label, filesystem, root, and disabled-import setting. Keep
|
||
schemas on their separate runtime option and omit an empty application group.
|
||
- Keep the operator and explicit local-backend contributions as later groups.
|
||
Preserve their accepted recursive overrides and source precedence. Do not
|
||
add an application-specific duplicate-key detector or flatten filesystem
|
||
contexts into an artificial shared root.
|
||
- Replace the incidental group-count test with behavioral coverage: conflicting
|
||
application contributions fail through native strict composition; distinct
|
||
declarations retain source-relative content; an operator overlay over the
|
||
resulting application base still succeeds. Reuse existing origin coverage.
|
||
|
||
Acceptance: application conflicts are not silently resolved by registration
|
||
order; source-relative imports/content and valid later operator overlays remain
|
||
unchanged. Runtime, preflight, and reports use the corrected shared construction.
|
||
|
||
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
|
||
|
||
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
|
||
|
||
Priority: P3. Category: CLI correctness. Status: open.
|
||
|
||
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
|
||
Notarius configuration and constructs the inspection engine before validating
|
||
the pointer; the default Notarius scope parses the pointer first.
|
||
|
||
Reproduction, using the native temporary build and a nonexistent config path:
|
||
|
||
```sh
|
||
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
|
||
```
|
||
|
||
The current PromptKit command returns exit 1 with a config-file error. The same
|
||
malformed pointer returns exit 2 with a valid config, and the Notarius scope
|
||
returns exit 2 even with the missing config. A bad PromptKit definition or
|
||
missing content file can likewise mask malformed command syntax. This differs
|
||
from the selected syntax/runtime distinction in the plan and CLI reference.
|
||
|
||
Required changes:
|
||
|
||
- Validate canonical JSON Pointer syntax once, before either scope loads config
|
||
or constructs an engine. Reuse the existing parser; retain defensive parsing
|
||
inside the adapter for direct callers.
|
||
- Keep the empty root pointer, escaped literal keys, historical-only paths,
|
||
unknown-path exit 1, and writer-error behavior unchanged.
|
||
- Extend CLI coverage with a malformed pointer and unavailable configuration;
|
||
verify exit 2, no stdout, and no configuration/provider work. No broader
|
||
argument-parser refactor is needed.
|
||
|
||
Acceptance: malformed pointers are syntax errors independent of configuration
|
||
availability for both scopes; valid selectors still load and inspect normally.
|
||
|
||
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
|
||
|
||
## Finding 3 — Correct Lifecycle And Fallback Documentation
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: open.
|
||
|
||
Evidence:
|
||
|
||
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
|
||
at engine construction. Schemas remain preparation-time resources supplied
|
||
through `WithSchemaFS`, not captured configuration definitions.
|
||
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
|
||
as occurring without loading a prompt. Construction now loads and validates
|
||
effective prompt definitions and captures message files/static templates;
|
||
it does not prepare a request or generate output.
|
||
- The comment on `RegisterFallbackProfileFS` in
|
||
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
|
||
are used only when the operator source lacks the ID. Under recursive overlays,
|
||
their omitted fields remain active beneath a matching operator entry.
|
||
|
||
Required changes:
|
||
|
||
- Distinguish constructor-time definition/message capture from preparation-time
|
||
schema loading and request-variable/artifact resolution. State accurately
|
||
that preflight needs no transcript, credential value, or provider call; do
|
||
not promise it performs no prompt-content file reads.
|
||
- Describe fallback registration as a lower-precedence application contribution
|
||
rather than an all-or-nothing lookup fallback.
|
||
- While editing this text, clarify that an operator `backends.local` declaration
|
||
can supply the backend as well as Notarius's optional `local_backend` object.
|
||
The latter is a convenience contribution, not a required second declaration.
|
||
Link canonical source rules rather than duplicating them.
|
||
- Change documentation/comments only unless verification exposes a real runtime
|
||
defect. Do not move schema loading earlier to make the stale prose true.
|
||
|
||
Acceptance: public and internal guidance consistently explains actual lifecycle,
|
||
native layering, and operator backend configuration without inventing a stronger
|
||
preflight guarantee or an additional required config field.
|
||
|
||
Validation: inspect current constructors/prepared execution and upstream v0.12
|
||
contracts; review affected local links; run `git diff --check`. Reuse unchanged
|
||
behavioral checks rather than adding prose-snapshot tests.
|
||
|
||
## Finding 4 — Preserve ADR-0010's Historical Decision Body
|
||
|
||
Priority: P3. Category: documentation-policy compliance. Status: open.
|
||
|
||
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
|
||
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
|
||
and `:47–50`, as well as adding the appropriate status/supersession link.
|
||
`docs/policy/documentation.md` requires accepted decision content to remain
|
||
immutable; Stage 14 specified a concise status link rather than rewriting the
|
||
historical body.
|
||
|
||
The replacement correctly acknowledges the old behavior, but the historical
|
||
record should retain its original wording. ADR-0019 already owns the new native
|
||
overlay decision and accurately limits which part of ADR-0010 it supersedes.
|
||
|
||
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
|
||
keeping its new partial-supersession status and ADR-0019 link. Leave workload
|
||
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
|
||
in the canonical current references, not retrofitted into the older decision.
|
||
|
||
Acceptance: the historical body matches the accepted original, the changed
|
||
clause's supersession remains discoverable, and no runtime behavior changes.
|
||
|
||
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
|
||
verify the ADR link, and run `git diff --check`.
|
||
|
||
## Review Evidence And Validation
|
||
|
||
- `GOWORK=off go test ./...` passed.
|
||
- `GOWORK=off go vet ./...` passed.
|
||
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
|
||
- `GOWORK=off go mod tidy -diff` produced no changes.
|
||
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
|
||
- A native build outside the repository succeeded. Credential-free profile and
|
||
explanation commands and complete/development pipeline validation succeeded.
|
||
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
|
||
are unchanged after removing old embedded IDs/versions. Prompt fragments,
|
||
schemas, and the embedded D&D execution settings are unchanged.
|
||
- An offline Go overlay probe, stored entirely outside the repository, confirmed
|
||
the application-group conflict behavior in Finding 1. CLI invocations
|
||
reproduced Finding 2 without generation or provider calls.
|
||
- The implementation diff passed whitespace checks. No production inference
|
||
or production model-quality evaluation was performed.
|
||
|
||
Recommended remediation is limited to these findings, not another configuration
|
||
system, new security policy, or unrelated pipeline refactoring.
|