Files
notarius/docs/roadmap/remediation.md

408 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Configuration Upgrade Review Remediation
The earlier PromptKit findings below retain their original scope and status.
The ConfigKit 1.2.0 review and subsequent integration audit appear below; they
do not reopen or attribute earlier issues to that upgrade.
## Scope And Overall Assessment
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
the accepted [feature roadmap](promptkit.md), and
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
PromptKit `/v3` catalogs at v3.20260930.0.
The migration is substantially implemented. Native sources, combined operator
documents and imports, recursive overlays, source-relative prompt content,
captured engine lifetime, immutable checkpoint identity, and credential-free
operator inspection are in place. The provider-neutral execution boundary,
prepared execution, retries, scheduling, and durable artifact contracts remain
intact. All 16 migrated prompt bodies match the previous definitions after
removing their old embedded identity fields; prompt fragments and schemas are
unchanged.
Four bounded follow-ups remain below. No production code is changed by this
review. No additional runtime security restriction, mandatory credential check,
filesystem permission policy, compatibility shim, or release gate was found.
Eager final-definition validation and the external keyed format are intentional
upstream changes, not accidental strengthening. The previously recorded
ConfigKit remediation was completed before this work set; it is not outstanding.
## Finding 1 — Compose Registered Application Assets In One Strict Group
Priority: P2. Category: architecture/correctness gap. Status: open.
Evidence:
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
registered prompt or fallback-profile filesystem.
- `internal/framework/llm/asset_registry_sources_test.go`,
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
separate registrations yield separate underlays, encoding this divergence.
- The plan's Sources And Engine Ownership section calls for one application
underlay whose inputs each retain their own filesystem context;
`docs/internal/llm.md:86–92` also describes that behavior.
Separate underlays have recursive overlay semantics. Conflicting application
definitions in different registrations therefore silently depend on module
registration order instead of receiving native strict-composition diagnostics.
An offline probe registered two fallback files with the same profile key and
different models. The current inspector accepted them and selected the second
model. The exact same filesystem inputs placed in one native underlay correctly
failed composition. Distinct filesystem contexts do not require separate layers.
Required changes:
- Gather the native `FilesystemInputs` from every application prompt and
fallback-profile registration into one application underlay. Preserve each
input's existing label, filesystem, root, and disabled-import setting. Keep
schemas on their separate runtime option and omit an empty application group.
- Keep the operator and explicit local-backend contributions as later groups.
Preserve their accepted recursive overrides and source precedence. Do not
add an application-specific duplicate-key detector or flatten filesystem
contexts into an artificial shared root.
- Replace the incidental group-count test with behavioral coverage: conflicting
application contributions fail through native strict composition; distinct
declarations retain source-relative content; an operator overlay over the
resulting application base still succeeds. Reuse existing origin coverage.
Acceptance: application conflicts are not silently resolved by registration
order; source-relative imports/content and valid later operator overlays remain
unchanged. Runtime, preflight, and reports use the corrected shared construction.
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
Priority: P3. Category: CLI correctness. Status: open.
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
Notarius configuration and constructs the inspection engine before validating
the pointer; the default Notarius scope parses the pointer first.
Reproduction, using the native temporary build and a nonexistent config path:
```sh
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
```
The current PromptKit command returns exit 1 with a config-file error. The same
malformed pointer returns exit 2 with a valid config, and the Notarius scope
returns exit 2 even with the missing config. A bad PromptKit definition or
missing content file can likewise mask malformed command syntax. This differs
from the selected syntax/runtime distinction in the plan and CLI reference.
Required changes:
- Validate canonical JSON Pointer syntax once, before either scope loads config
or constructs an engine. Reuse the existing parser; retain defensive parsing
inside the adapter for direct callers.
- Keep the empty root pointer, escaped literal keys, historical-only paths,
unknown-path exit 1, and writer-error behavior unchanged.
- Extend CLI coverage with a malformed pointer and unavailable configuration;
verify exit 2, no stdout, and no configuration/provider work. No broader
argument-parser refactor is needed.
Acceptance: malformed pointers are syntax errors independent of configuration
availability for both scopes; valid selectors still load and inspect normally.
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
## Finding 3 — Correct Lifecycle And Fallback Documentation
Priority: P3. Category: documentation accuracy. Status: open.
Evidence:
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
at engine construction. Schemas remain preparation-time resources supplied
through `WithSchemaFS`, not captured configuration definitions.
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
as occurring without loading a prompt. Construction now loads and validates
effective prompt definitions and captures message files/static templates;
it does not prepare a request or generate output.
- The comment on `RegisterFallbackProfileFS` in
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
are used only when the operator source lacks the ID. Under recursive overlays,
their omitted fields remain active beneath a matching operator entry.
Required changes:
- Distinguish constructor-time definition/message capture from preparation-time
schema loading and request-variable/artifact resolution. State accurately
that preflight needs no transcript, credential value, or provider call; do
not promise it performs no prompt-content file reads.
- Describe fallback registration as a lower-precedence application contribution
rather than an all-or-nothing lookup fallback.
- While editing this text, clarify that an operator `backends.local` declaration
can supply the backend as well as Notarius's optional `local_backend` object.
The latter is a convenience contribution, not a required second declaration.
Link canonical source rules rather than duplicating them.
- Change documentation/comments only unless verification exposes a real runtime
defect. Do not move schema loading earlier to make the stale prose true.
Acceptance: public and internal guidance consistently explains actual lifecycle,
native layering, and operator backend configuration without inventing a stronger
preflight guarantee or an additional required config field.
Validation: inspect current constructors/prepared execution and upstream v0.12
contracts; review affected local links; run `git diff --check`. Reuse unchanged
behavioral checks rather than adding prose-snapshot tests.
## Finding 4 — Preserve ADR-0010's Historical Decision Body
Priority: P3. Category: documentation-policy compliance. Status: open.
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
and `:47–50`, as well as adding the appropriate status/supersession link.
`docs/policy/documentation.md` requires accepted decision content to remain
immutable; Stage 14 specified a concise status link rather than rewriting the
historical body.
The replacement correctly acknowledges the old behavior, but the historical
record should retain its original wording. ADR-0019 already owns the new native
overlay decision and accurately limits which part of ADR-0010 it supersedes.
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
keeping its new partial-supersession status and ADR-0019 link. Leave workload
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
in the canonical current references, not retrofitted into the older decision.
Acceptance: the historical body matches the accepted original, the changed
clause's supersession remains discoverable, and no runtime behavior changes.
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
verify the ADR link, and run `git diff --check`.
## Review Evidence And Validation
- `GOWORK=off go test ./...` passed.
- `GOWORK=off go vet ./...` passed.
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
- `GOWORK=off go mod tidy -diff` produced no changes.
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
- A native build outside the repository succeeded. Credential-free profile and
explanation commands and complete/development pipeline validation succeeded.
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
are unchanged after removing old embedded IDs/versions. Prompt fragments,
schemas, and the embedded D&D execution settings are unchanged.
- An offline Go overlay probe, stored entirely outside the repository, confirmed
the application-group conflict behavior in Finding 1. CLI invocations
reproduced Finding 2 without generation or provider calls.
- The implementation diff passed whitespace checks. No production inference
or production model-quality evaluation was performed.
Recommended remediation is limited to these findings, not another configuration
system, new security policy, or unrelated pipeline refactoring.
## ConfigKit 1.2.0 Review
Reviewed on 2026-11-17 against implementation revision `c9744ad`, compared with
planning revision `79088a3` and the ConfigKit feature roadmap and implementation
plan. The subsequent roadmap-only clarification in `d88a01c` does not change
the reviewed runtime. ConfigKit v1.2.0 resolves to
`c1c7e87f2cdf77a0cbf3327831ac05578e756b9e`.
The production architecture matches the target. Application configuration uses
new-path ConfigKit v1.2.0; the unchanged PromptKit v0.12.0 adapter retains
old-path ConfigKit v0.5.0. Both use YAML v4.0.0-rc.6 under Go 1.27.0. The CLI
describes one exclusive file or ordered native search source, while ConfigKit
owns acquisition, imports, collections, inheritance, decoding, diagnostics, and
inspection. Notarius supplies domain defaults, recorded normalization,
validation, detached runtime conversion, and catalog-dependent resolution.
No new production correctness defect or unwarranted restriction was found.
The changed controls, optional import defaults, post-decode CLI assignments,
step identity/upsert, search behavior, and higher Go minimum are approved scope.
The remaining gaps are two operator-documentation inaccuracies from Stage 9,
also missed by Stage 11's final documentation check. Production fixes are not
required for either finding.
### Finding CK-1 — Distinguish Mapping And Record Inheritance Syntax
Priority: P3. Category: documentation accuracy. Status: open.
Evidence: `docs/config.md:143–149` introduces both mapping and identified-record
inheritance, then says a directive names an immediate sibling in the same
mapping. That describes mapping inheritance only. For an identified step record,
the parent belongs to the same sequence and the reference must be an identity
mapping, such as `extends: {id: parent}`. The reference never shows that form.
An offline CLI probe with a parent step providing references and a child step
providing an artifact lane confirmed that `extends: parent` fails with
`inheritance.invalid`, while changing only the directive to
`extends: {id: parent}` loads successfully and preserves the inherited reference.
`TestIdentifiedStepRecordInheritance` already exercises the correct form in
`internal/core/config/collection_composition_test.go:182–208`.
Required changes:
- Explain string-valued mapping-sibling inheritance and identity-mapping
sequence-record inheritance separately, including their distinct lookup scopes.
- Add a minimal step-record example or a direct link to the pinned upstream
syntax example. Preserve the existing rules for null cancellation, validation,
and whole-record upsert; no new application inheritance machinery is needed.
Acceptance: an operator can determine the correct syntax for both a pipeline
mapping and an identified record from the canonical reference. Any example
presented as a runnable pipeline must pass pipeline resolution, not just loading;
see CK-3 for step-record limitations. Reuse behavioral coverage rather than adding
a documentation snapshot or changing runtime behavior to fit the prose.
### Finding CK-2 — Describe Dependency-Ready Step Assembly Order
Priority: P3. Category: documentation accuracy. Status: open.
Evidence: `docs/config.md:167–170` says distinct imported step IDs combine
"in source order." ConfigKit distinguishes acquisition/import declaration order
from dependency-ready assembly order. Ready contributions are selected by their
stable source keys, so independent branches can interleave. The feature roadmap
and Stage 8 correctly rely on dependency-ready assembly rather than a Notarius
append pass.
An offline CLI probe declared imports in the order `z-first.yml`, then
`a-second.yml`, each contributing a different step and lane. Effective output
placed the step from `a-second.yml` first, then the step from `z-first.yml`.
Thus declaration order does not determine the resulting step list. This matters
because Notarius executes steps in the resolved list order and validates their
generated-reference dependencies.
Required changes:
- Replace the source-order claim with the native dependency-ready assembly rule
and link to ConfigKit v1.2.0's pinned composition contract for its full ordering
semantics. Do not promise declaration order or duplicate the upstream scheduler.
- Explain briefly why the maintained complete example's imported describe-session
step precedes the importing body's extract-events step: the imported dependency
is assembled before its importer. Keep the canonical operator rule in
`docs/config.md`; other documentation should link rather than redefine it.
Acceptance: the reference does not imply that reordering independent import
declarations necessarily reorders steps. The complete example's actual step
order and generated-reference handoffs remain unchanged. No custom sorting,
merging, or scheduler changes belong in this remediation.
### Stage-By-Stage Review Evidence
| Stage | Review result |
| --- | --- |
| 1 — Dependencies/toolchain | The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff. |
| 2 — API/handoff | Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values. |
| 3 — Normalization/origins | Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass. |
| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, and source-relative references. Identified-record inheritance is checked only at loading/configuration validation; CK-3 records the missing runtime-consumer check. |
| 5 — Fixtures | Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes. |
| 6 — CLI discovery | The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed. |
| 7 — Discovery/failures | Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files. |
| 8 — Examples | Minimal, complete, and development pipelines validate without inference; the supporting fragment preserves module choices, step order, and handoffs. |
| 9 — Operator docs | Search, controls, path bases, CLI timing, and toolchain guidance are updated. CK-1 and CK-2 remain. |
| 10 — Internal/integration docs and notices | The mixed dependency boundary and pinned ConfigKit attribution match the selected graph. Accepted ADRs, policy, and project license/grant were not modified by the sprint. |
| 11 — Final verification | Full tests, vet, tidy, native build, dependency inspection, and provider-free validation/effective/explanation commands pass. The final documentation review missed CK-1 and CK-2. |
### ConfigKit Review Validation
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./...` passed.
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go vet ./...` passed.
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test -race ./internal/core/config ./internal/cli` passed.
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go mod tidy -diff` produced no changes.
- A native build in a temporary directory succeeded. Minimal, complete, and
development configuration validation, plus application and PromptKit effective
and explanation reports, succeeded without provider calls.
- Selected module inspection confirmed the four approved ConfigKit, PromptKit,
and YAML versions. Pinned local upstream documentation and notices were checked.
- Temporary fixtures outside the repository reproduced both documentation gaps
and verified combined identifier/reference-path normalization. No production
pipeline, credential lookup, or live model-quality evaluation was performed.
- The review document passed `git diff --check`; production code and current
operator documentation were not changed by this review.
Keep the earlier PromptKit findings separate from these two documentation fixes.
The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope.
## ConfigKit Integration Audit
Reviewed on 2026-11-17 against `6f0a75e`, with particular attention to how the
11 implementation stages interact with the existing pipeline and PromptKit
consumers. The implementation is unchanged from `c9744ad`; the intervening
commits changed roadmaps only. CK-1 and CK-2 remain open.
The dependency/toolchain boundary, detached configuration conversion, path
normalization, CLI overrides, native discovery, selected-file reporting, and
old-format PromptKit source handoff are coherent. The maintained complete
example still reaches the existing multi-step runner and publishes its generated
registry/occurrence handoffs through an injected external client. Session,
checkpoint, warning, and output consumers continue to receive resolved domain
values, not ConfigKit source graphs. No new production correctness regression
was identified. One additional test/documentation integration gap remains.
### Finding CK-3 — Verify Identified-Record Inheritance At The Runtime Boundary
Priority: P3. Category: integration coverage and support-scope clarity. Status: open.
Affected stages: 4, 9, and 11.
`TestIdentifiedStepRecordInheritance` in
`internal/core/config/collection_composition_test.go:182–208` supplies a parent
step containing only references and a child step containing an artifact lane.
It checks `Load` and `Config.Validate`, but never calls `Config.Resolve` with a
catalog. The existing runner's resolver requires every explicit step to have at
least one artifact lane (`internal/framework/pipeline/profile.go`,
`ResolvePipeline`). The parent is an ordinary executable step, not an excluded
template. The fixture therefore cannot be used as a resolved pipeline, despite
the positive test and the reference's unqualified mention of step-record
inheritance.
Giving the parent a real artifact lane is not a simple repair to the fixture:
native inheritance recursively copies that named lane into the child, while
Notarius requires lane identifiers to be unique across steps. Adding a new
child-specific lane does not remove the inherited one. Pipeline-level inheritance
with step upsert is a different operation and remains supported and tested.
Offline reproduction through the current binary:
- A production-module equivalent of the test fixture loads for effective
inspection, but `config validate --pipeline session` fails because the parent
step must declare an artifact lane.
- A parent with `parent-spells` and a child declaring `child-spells` fails domain
validation because inherited `parent-spells` occurs in both steps.
- A native identity declaration on an extraction validator list, with
`generic/valid_json_schema` inheriting from `generic/valid_json` through
`extends: {module: generic/valid_json}`, passes production pipeline validation.
Generic identified-record inheritance is therefore usable; the gap is the
selected positive step fixture and its implied runtime applicability.
Required changes:
- Replace or reframe the positive step fixture. Cover a genuinely executable
identified-record use case in a compatible collection and verify it through
`Config.Resolve` with the appropriate catalog or the equivalent public CLI
validation path. Assert a meaningful inherited value, not only successful load.
- Make the step-specific constraints clear alongside CK-1's syntax correction.
Do not present the reference-only parent fixture as a runnable step template.
Preserve the distinction between pipeline inheritance/upsert and inheritance
between records of one final sequence.
- Preserve the existing runtime constraints. This finding does not authorize
template-only steps, implicit parent removal, lane renaming, or weaker lane-ID
uniqueness. A separate product decision would be needed for such features.
Acceptance: positive inheritance coverage reaches the actual configuration
consumer, and operator guidance distinguishes native composition from executable
pipeline validity. The complete D&D example, generated-reference validation, and
ordinary pipeline inheritance continue to behave unchanged. No production fix
is indicated by the current evidence.
### Integration Audit Validation
- Reviewed the staged changes against configuration conversion, pipeline
resolution/reference materialization, PromptKit construction/inspection,
checkpoint identity, and CLI debug/output consumers.
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline ./internal/modules/integration`
passed. This includes the maintained complete-example run, generated-reference
handoffs, command overrides, selected-path debug provenance, and profile reports.
- Temporary CLI fixtures outside the repository reproduced both step-inheritance
failures and verified the compatible validator-record alternative. An additional
author-identified validator list with whitespace-normalized module names also
passed pipeline validation.
- The full-suite, vet, race, tidy, and build results from the preceding review
remain applicable to the unchanged implementation. No live provider call or
production-quality evaluation was performed.