320 lines
18 KiB
Markdown
320 lines
18 KiB
Markdown
# Configuration Upgrade Review Remediation
|
||
|
||
The earlier PromptKit findings below retain their original scope and status.
|
||
The ConfigKit 1.2.0 review and its two new findings are recorded in the final
|
||
section; they do not reopen or attribute earlier issues to that upgrade.
|
||
|
||
## Scope And Overall Assessment
|
||
|
||
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
|
||
the accepted [feature roadmap](promptkit.md), and
|
||
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
|
||
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
|
||
PromptKit `/v3` catalogs at v3.20260930.0.
|
||
|
||
The migration is substantially implemented. Native sources, combined operator
|
||
documents and imports, recursive overlays, source-relative prompt content,
|
||
captured engine lifetime, immutable checkpoint identity, and credential-free
|
||
operator inspection are in place. The provider-neutral execution boundary,
|
||
prepared execution, retries, scheduling, and durable artifact contracts remain
|
||
intact. All 16 migrated prompt bodies match the previous definitions after
|
||
removing their old embedded identity fields; prompt fragments and schemas are
|
||
unchanged.
|
||
|
||
Four bounded follow-ups remain below. No production code is changed by this
|
||
review. No additional runtime security restriction, mandatory credential check,
|
||
filesystem permission policy, compatibility shim, or release gate was found.
|
||
Eager final-definition validation and the external keyed format are intentional
|
||
upstream changes, not accidental strengthening. The previously recorded
|
||
ConfigKit remediation was completed before this work set; it is not outstanding.
|
||
|
||
## Finding 1 — Compose Registered Application Assets In One Strict Group
|
||
|
||
Priority: P2. Category: architecture/correctness gap. Status: open.
|
||
|
||
Evidence:
|
||
|
||
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
|
||
registered prompt or fallback-profile filesystem.
|
||
- `internal/framework/llm/asset_registry_sources_test.go`,
|
||
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
|
||
separate registrations yield separate underlays, encoding this divergence.
|
||
- The plan's Sources And Engine Ownership section calls for one application
|
||
underlay whose inputs each retain their own filesystem context;
|
||
`docs/internal/llm.md:86–92` also describes that behavior.
|
||
|
||
Separate underlays have recursive overlay semantics. Conflicting application
|
||
definitions in different registrations therefore silently depend on module
|
||
registration order instead of receiving native strict-composition diagnostics.
|
||
An offline probe registered two fallback files with the same profile key and
|
||
different models. The current inspector accepted them and selected the second
|
||
model. The exact same filesystem inputs placed in one native underlay correctly
|
||
failed composition. Distinct filesystem contexts do not require separate layers.
|
||
|
||
Required changes:
|
||
|
||
- Gather the native `FilesystemInputs` from every application prompt and
|
||
fallback-profile registration into one application underlay. Preserve each
|
||
input's existing label, filesystem, root, and disabled-import setting. Keep
|
||
schemas on their separate runtime option and omit an empty application group.
|
||
- Keep the operator and explicit local-backend contributions as later groups.
|
||
Preserve their accepted recursive overrides and source precedence. Do not
|
||
add an application-specific duplicate-key detector or flatten filesystem
|
||
contexts into an artificial shared root.
|
||
- Replace the incidental group-count test with behavioral coverage: conflicting
|
||
application contributions fail through native strict composition; distinct
|
||
declarations retain source-relative content; an operator overlay over the
|
||
resulting application base still succeeds. Reuse existing origin coverage.
|
||
|
||
Acceptance: application conflicts are not silently resolved by registration
|
||
order; source-relative imports/content and valid later operator overlays remain
|
||
unchanged. Runtime, preflight, and reports use the corrected shared construction.
|
||
|
||
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
|
||
|
||
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
|
||
|
||
Priority: P3. Category: CLI correctness. Status: open.
|
||
|
||
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
|
||
Notarius configuration and constructs the inspection engine before validating
|
||
the pointer; the default Notarius scope parses the pointer first.
|
||
|
||
Reproduction, using the native temporary build and a nonexistent config path:
|
||
|
||
```sh
|
||
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
|
||
```
|
||
|
||
The current PromptKit command returns exit 1 with a config-file error. The same
|
||
malformed pointer returns exit 2 with a valid config, and the Notarius scope
|
||
returns exit 2 even with the missing config. A bad PromptKit definition or
|
||
missing content file can likewise mask malformed command syntax. This differs
|
||
from the selected syntax/runtime distinction in the plan and CLI reference.
|
||
|
||
Required changes:
|
||
|
||
- Validate canonical JSON Pointer syntax once, before either scope loads config
|
||
or constructs an engine. Reuse the existing parser; retain defensive parsing
|
||
inside the adapter for direct callers.
|
||
- Keep the empty root pointer, escaped literal keys, historical-only paths,
|
||
unknown-path exit 1, and writer-error behavior unchanged.
|
||
- Extend CLI coverage with a malformed pointer and unavailable configuration;
|
||
verify exit 2, no stdout, and no configuration/provider work. No broader
|
||
argument-parser refactor is needed.
|
||
|
||
Acceptance: malformed pointers are syntax errors independent of configuration
|
||
availability for both scopes; valid selectors still load and inspect normally.
|
||
|
||
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
|
||
|
||
## Finding 3 — Correct Lifecycle And Fallback Documentation
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: open.
|
||
|
||
Evidence:
|
||
|
||
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
|
||
at engine construction. Schemas remain preparation-time resources supplied
|
||
through `WithSchemaFS`, not captured configuration definitions.
|
||
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
|
||
as occurring without loading a prompt. Construction now loads and validates
|
||
effective prompt definitions and captures message files/static templates;
|
||
it does not prepare a request or generate output.
|
||
- The comment on `RegisterFallbackProfileFS` in
|
||
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
|
||
are used only when the operator source lacks the ID. Under recursive overlays,
|
||
their omitted fields remain active beneath a matching operator entry.
|
||
|
||
Required changes:
|
||
|
||
- Distinguish constructor-time definition/message capture from preparation-time
|
||
schema loading and request-variable/artifact resolution. State accurately
|
||
that preflight needs no transcript, credential value, or provider call; do
|
||
not promise it performs no prompt-content file reads.
|
||
- Describe fallback registration as a lower-precedence application contribution
|
||
rather than an all-or-nothing lookup fallback.
|
||
- While editing this text, clarify that an operator `backends.local` declaration
|
||
can supply the backend as well as Notarius's optional `local_backend` object.
|
||
The latter is a convenience contribution, not a required second declaration.
|
||
Link canonical source rules rather than duplicating them.
|
||
- Change documentation/comments only unless verification exposes a real runtime
|
||
defect. Do not move schema loading earlier to make the stale prose true.
|
||
|
||
Acceptance: public and internal guidance consistently explains actual lifecycle,
|
||
native layering, and operator backend configuration without inventing a stronger
|
||
preflight guarantee or an additional required config field.
|
||
|
||
Validation: inspect current constructors/prepared execution and upstream v0.12
|
||
contracts; review affected local links; run `git diff --check`. Reuse unchanged
|
||
behavioral checks rather than adding prose-snapshot tests.
|
||
|
||
## Finding 4 — Preserve ADR-0010's Historical Decision Body
|
||
|
||
Priority: P3. Category: documentation-policy compliance. Status: open.
|
||
|
||
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
|
||
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
|
||
and `:47–50`, as well as adding the appropriate status/supersession link.
|
||
`docs/policy/documentation.md` requires accepted decision content to remain
|
||
immutable; Stage 14 specified a concise status link rather than rewriting the
|
||
historical body.
|
||
|
||
The replacement correctly acknowledges the old behavior, but the historical
|
||
record should retain its original wording. ADR-0019 already owns the new native
|
||
overlay decision and accurately limits which part of ADR-0010 it supersedes.
|
||
|
||
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
|
||
keeping its new partial-supersession status and ADR-0019 link. Leave workload
|
||
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
|
||
in the canonical current references, not retrofitted into the older decision.
|
||
|
||
Acceptance: the historical body matches the accepted original, the changed
|
||
clause's supersession remains discoverable, and no runtime behavior changes.
|
||
|
||
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
|
||
verify the ADR link, and run `git diff --check`.
|
||
|
||
## Review Evidence And Validation
|
||
|
||
- `GOWORK=off go test ./...` passed.
|
||
- `GOWORK=off go vet ./...` passed.
|
||
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
|
||
- `GOWORK=off go mod tidy -diff` produced no changes.
|
||
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
|
||
- A native build outside the repository succeeded. Credential-free profile and
|
||
explanation commands and complete/development pipeline validation succeeded.
|
||
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
|
||
are unchanged after removing old embedded IDs/versions. Prompt fragments,
|
||
schemas, and the embedded D&D execution settings are unchanged.
|
||
- An offline Go overlay probe, stored entirely outside the repository, confirmed
|
||
the application-group conflict behavior in Finding 1. CLI invocations
|
||
reproduced Finding 2 without generation or provider calls.
|
||
- The implementation diff passed whitespace checks. No production inference
|
||
or production model-quality evaluation was performed.
|
||
|
||
Recommended remediation is limited to these findings, not another configuration
|
||
system, new security policy, or unrelated pipeline refactoring.
|
||
|
||
## ConfigKit 1.2.0 Review
|
||
|
||
Reviewed on 2026-11-17 against implementation revision `c9744ad`, compared with
|
||
planning revision `79088a3` and the ConfigKit feature roadmap and implementation
|
||
plan. The subsequent roadmap-only clarification in `d88a01c` does not change
|
||
the reviewed runtime. ConfigKit v1.2.0 resolves to
|
||
`c1c7e87f2cdf77a0cbf3327831ac05578e756b9e`.
|
||
|
||
The production architecture matches the target. Application configuration uses
|
||
new-path ConfigKit v1.2.0; the unchanged PromptKit v0.12.0 adapter retains
|
||
old-path ConfigKit v0.5.0. Both use YAML v4.0.0-rc.6 under Go 1.27.0. The CLI
|
||
describes one exclusive file or ordered native search source, while ConfigKit
|
||
owns acquisition, imports, collections, inheritance, decoding, diagnostics, and
|
||
inspection. Notarius supplies domain defaults, recorded normalization,
|
||
validation, detached runtime conversion, and catalog-dependent resolution.
|
||
|
||
No new production correctness defect or unwarranted restriction was found.
|
||
The changed controls, optional import defaults, post-decode CLI assignments,
|
||
step identity/upsert, search behavior, and higher Go minimum are approved scope.
|
||
The remaining gaps are two operator-documentation inaccuracies from Stage 9,
|
||
also missed by Stage 11's final documentation check. Production fixes are not
|
||
required for either finding.
|
||
|
||
### Finding CK-1 — Distinguish Mapping And Record Inheritance Syntax
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: open.
|
||
|
||
Evidence: `docs/config.md:143–149` introduces both mapping and identified-record
|
||
inheritance, then says a directive names an immediate sibling in the same
|
||
mapping. That describes mapping inheritance only. For an identified step record,
|
||
the parent belongs to the same sequence and the reference must be an identity
|
||
mapping, such as `extends: {id: parent}`. The reference never shows that form.
|
||
|
||
An offline CLI probe with a parent step providing references and a child step
|
||
providing an artifact lane confirmed that `extends: parent` fails with
|
||
`inheritance.invalid`, while changing only the directive to
|
||
`extends: {id: parent}` loads successfully and preserves the inherited reference.
|
||
`TestIdentifiedStepRecordInheritance` already exercises the correct form in
|
||
`internal/core/config/collection_composition_test.go:182–208`.
|
||
|
||
Required changes:
|
||
|
||
- Explain string-valued mapping-sibling inheritance and identity-mapping
|
||
sequence-record inheritance separately, including their distinct lookup scopes.
|
||
- Add a minimal step-record example or a direct link to the pinned upstream
|
||
syntax example. Preserve the existing rules for null cancellation, validation,
|
||
and whole-record upsert; no new application inheritance machinery is needed.
|
||
|
||
Acceptance: an operator can determine the correct syntax for both a pipeline
|
||
mapping and a step record from the canonical reference. Any added snippet loads
|
||
through the current configuration boundary. Reuse the existing behavioral test;
|
||
do not add a documentation snapshot or change runtime behavior to fit the prose.
|
||
|
||
### Finding CK-2 — Describe Dependency-Ready Step Assembly Order
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: open.
|
||
|
||
Evidence: `docs/config.md:167–170` says distinct imported step IDs combine
|
||
"in source order." ConfigKit distinguishes acquisition/import declaration order
|
||
from dependency-ready assembly order. Ready contributions are selected by their
|
||
stable source keys, so independent branches can interleave. The feature roadmap
|
||
and Stage 8 correctly rely on dependency-ready assembly rather than a Notarius
|
||
append pass.
|
||
|
||
An offline CLI probe declared imports in the order `z-first.yml`, then
|
||
`a-second.yml`, each contributing a different step and lane. Effective output
|
||
placed the step from `a-second.yml` first, then the step from `z-first.yml`.
|
||
Thus declaration order does not determine the resulting step list. This matters
|
||
because Notarius executes steps in the resolved list order and validates their
|
||
generated-reference dependencies.
|
||
|
||
Required changes:
|
||
|
||
- Replace the source-order claim with the native dependency-ready assembly rule
|
||
and link to ConfigKit v1.2.0's pinned composition contract for its full ordering
|
||
semantics. Do not promise declaration order or duplicate the upstream scheduler.
|
||
- Explain briefly why the maintained complete example's imported describe-session
|
||
step precedes the importing body's extract-events step: the imported dependency
|
||
is assembled before its importer. Keep the canonical operator rule in
|
||
`docs/config.md`; other documentation should link rather than redefine it.
|
||
|
||
Acceptance: the reference does not imply that reordering independent import
|
||
declarations necessarily reorders steps. The complete example's actual step
|
||
order and generated-reference handoffs remain unchanged. No custom sorting,
|
||
merging, or scheduler changes belong in this remediation.
|
||
|
||
### Stage-By-Stage Review Evidence
|
||
|
||
| Stage | Review result |
|
||
| --- | --- |
|
||
| 1 — Dependencies/toolchain | The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff. |
|
||
| 2 — API/handoff | Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values. |
|
||
| 3 — Normalization/origins | Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass. |
|
||
| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, record inheritance, and source-relative references. |
|
||
| 5 — Fixtures | Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes. |
|
||
| 6 — CLI discovery | The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed. |
|
||
| 7 — Discovery/failures | Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files. |
|
||
| 8 — Examples | Minimal, complete, and development pipelines validate without inference; the supporting fragment preserves module choices, step order, and handoffs. |
|
||
| 9 — Operator docs | Search, controls, path bases, CLI timing, and toolchain guidance are updated. CK-1 and CK-2 remain. |
|
||
| 10 — Internal/integration docs and notices | The mixed dependency boundary and pinned ConfigKit attribution match the selected graph. Accepted ADRs, policy, and project license/grant were not modified by the sprint. |
|
||
| 11 — Final verification | Full tests, vet, tidy, native build, dependency inspection, and provider-free validation/effective/explanation commands pass. The final documentation review missed CK-1 and CK-2. |
|
||
|
||
### ConfigKit Review Validation
|
||
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./...` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go vet ./...` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test -race ./internal/core/config ./internal/cli` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go mod tidy -diff` produced no changes.
|
||
- A native build in a temporary directory succeeded. Minimal, complete, and
|
||
development configuration validation, plus application and PromptKit effective
|
||
and explanation reports, succeeded without provider calls.
|
||
- Selected module inspection confirmed the four approved ConfigKit, PromptKit,
|
||
and YAML versions. Pinned local upstream documentation and notices were checked.
|
||
- Temporary fixtures outside the repository reproduced both documentation gaps
|
||
and verified combined identifier/reference-path normalization. No production
|
||
pipeline, credential lookup, or live model-quality evaluation was performed.
|
||
- The review document passed `git diff --check`; production code and current
|
||
operator documentation were not changed by this review.
|
||
|
||
Keep the earlier PromptKit findings separate from these two documentation fixes.
|
||
The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope.
|