Files
notarius/docs/roadmap/remediation.md

194 lines
10 KiB
Markdown
Raw Normal View History

# PromptKit Configuration Migration Review Remediation
## Scope And Overall Assessment
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
the accepted [feature roadmap](promptkit.md), and
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
PromptKit `/v3` catalogs at v3.20260930.0.
The migration is substantially implemented. Native sources, combined operator
documents and imports, recursive overlays, source-relative prompt content,
captured engine lifetime, immutable checkpoint identity, and credential-free
operator inspection are in place. The provider-neutral execution boundary,
prepared execution, retries, scheduling, and durable artifact contracts remain
intact. All 16 migrated prompt bodies match the previous definitions after
removing their old embedded identity fields; prompt fragments and schemas are
unchanged.
Four bounded follow-ups remain below. No production code is changed by this
review. No additional runtime security restriction, mandatory credential check,
filesystem permission policy, compatibility shim, or release gate was found.
Eager final-definition validation and the external keyed format are intentional
upstream changes, not accidental strengthening. The previously recorded
ConfigKit remediation was completed before this work set; it is not outstanding.
## Finding 1 — Compose Registered Application Assets In One Strict Group
Priority: P2. Category: architecture/correctness gap. Status: open.
Evidence:
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
registered prompt or fallback-profile filesystem.
- `internal/framework/llm/asset_registry_sources_test.go`,
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
separate registrations yield separate underlays, encoding this divergence.
- The plan's Sources And Engine Ownership section calls for one application
underlay whose inputs each retain their own filesystem context;
`docs/internal/llm.md:86–92` also describes that behavior.
Separate underlays have recursive overlay semantics. Conflicting application
definitions in different registrations therefore silently depend on module
registration order instead of receiving native strict-composition diagnostics.
An offline probe registered two fallback files with the same profile key and
different models. The current inspector accepted them and selected the second
model. The exact same filesystem inputs placed in one native underlay correctly
failed composition. Distinct filesystem contexts do not require separate layers.
Required changes:
- Gather the native `FilesystemInputs` from every application prompt and
fallback-profile registration into one application underlay. Preserve each
input's existing label, filesystem, root, and disabled-import setting. Keep
schemas on their separate runtime option and omit an empty application group.
- Keep the operator and explicit local-backend contributions as later groups.
Preserve their accepted recursive overrides and source precedence. Do not
add an application-specific duplicate-key detector or flatten filesystem
contexts into an artificial shared root.
- Replace the incidental group-count test with behavioral coverage: conflicting
application contributions fail through native strict composition; distinct
declarations retain source-relative content; an operator overlay over the
resulting application base still succeeds. Reuse existing origin coverage.
Acceptance: application conflicts are not silently resolved by registration
order; source-relative imports/content and valid later operator overlays remain
unchanged. Runtime, preflight, and reports use the corrected shared construction.
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
Priority: P3. Category: CLI correctness. Status: open.
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
Notarius configuration and constructs the inspection engine before validating
the pointer; the default Notarius scope parses the pointer first.
Reproduction, using the native temporary build and a nonexistent config path:
```sh
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
```
The current PromptKit command returns exit 1 with a config-file error. The same
malformed pointer returns exit 2 with a valid config, and the Notarius scope
returns exit 2 even with the missing config. A bad PromptKit definition or
missing content file can likewise mask malformed command syntax. This differs
from the selected syntax/runtime distinction in the plan and CLI reference.
Required changes:
- Validate canonical JSON Pointer syntax once, before either scope loads config
or constructs an engine. Reuse the existing parser; retain defensive parsing
inside the adapter for direct callers.
- Keep the empty root pointer, escaped literal keys, historical-only paths,
unknown-path exit 1, and writer-error behavior unchanged.
- Extend CLI coverage with a malformed pointer and unavailable configuration;
verify exit 2, no stdout, and no configuration/provider work. No broader
argument-parser refactor is needed.
Acceptance: malformed pointers are syntax errors independent of configuration
availability for both scopes; valid selectors still load and inspect normally.
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
## Finding 3 — Correct Lifecycle And Fallback Documentation
Priority: P3. Category: documentation accuracy. Status: open.
Evidence:
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
at engine construction. Schemas remain preparation-time resources supplied
through `WithSchemaFS`, not captured configuration definitions.
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
as occurring without loading a prompt. Construction now loads and validates
effective prompt definitions and captures message files/static templates;
it does not prepare a request or generate output.
- The comment on `RegisterFallbackProfileFS` in
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
are used only when the operator source lacks the ID. Under recursive overlays,
their omitted fields remain active beneath a matching operator entry.
Required changes:
- Distinguish constructor-time definition/message capture from preparation-time
schema loading and request-variable/artifact resolution. State accurately
that preflight needs no transcript, credential value, or provider call; do
not promise it performs no prompt-content file reads.
- Describe fallback registration as a lower-precedence application contribution
rather than an all-or-nothing lookup fallback.
- While editing this text, clarify that an operator `backends.local` declaration
can supply the backend as well as Notarius's optional `local_backend` object.
The latter is a convenience contribution, not a required second declaration.
Link canonical source rules rather than duplicating them.
- Change documentation/comments only unless verification exposes a real runtime
defect. Do not move schema loading earlier to make the stale prose true.
Acceptance: public and internal guidance consistently explains actual lifecycle,
native layering, and operator backend configuration without inventing a stronger
preflight guarantee or an additional required config field.
Validation: inspect current constructors/prepared execution and upstream v0.12
contracts; review affected local links; run `git diff --check`. Reuse unchanged
behavioral checks rather than adding prose-snapshot tests.
## Finding 4 — Preserve ADR-0010's Historical Decision Body
Priority: P3. Category: documentation-policy compliance. Status: open.
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
and `:47–50`, as well as adding the appropriate status/supersession link.
`docs/policy/documentation.md` requires accepted decision content to remain
immutable; Stage 14 specified a concise status link rather than rewriting the
historical body.
The replacement correctly acknowledges the old behavior, but the historical
record should retain its original wording. ADR-0019 already owns the new native
overlay decision and accurately limits which part of ADR-0010 it supersedes.
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
keeping its new partial-supersession status and ADR-0019 link. Leave workload
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
in the canonical current references, not retrofitted into the older decision.
Acceptance: the historical body matches the accepted original, the changed
clause's supersession remains discoverable, and no runtime behavior changes.
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
verify the ADR link, and run `git diff --check`.
## Review Evidence And Validation
- `GOWORK=off go test ./...` passed.
- `GOWORK=off go vet ./...` passed.
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
- `GOWORK=off go mod tidy -diff` produced no changes.
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
- A native build outside the repository succeeded. Credential-free profile and
explanation commands and complete/development pipeline validation succeeded.
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
are unchanged after removing old embedded IDs/versions. Prompt fragments,
schemas, and the embedded D&D execution settings are unchanged.
- An offline Go overlay probe, stored entirely outside the repository, confirmed
the application-group conflict behavior in Finding 1. CLI invocations
reproduced Finding 2 without generation or provider calls.
- The implementation diff passed whitespace checks. No production inference
or production model-quality evaluation was performed.
Recommended remediation is limited to these findings, not another configuration
system, new security policy, or unrelated pipeline refactoring.