Files
notarius/docs/roadmap/remediation.md

18 KiB
Raw Blame History

Configuration Upgrade Review Remediation

The earlier PromptKit findings below retain their original scope and status. The ConfigKit 1.2.0 review and its two new findings are recorded in the final section; they do not reopen or attribute earlier issues to that upgrade.

Scope And Overall Assessment

Review baseline: Notarius 6700ed2, compared with planning revision f5b066b, the accepted feature roadmap, and implementation plan. PromptKit v0.12.0 is pinned at 4b8169da286917fde7875425bb32b799b12c6af3; ConfigKit remains v0.5.0, with both PromptKit /v3 catalogs at v3.20260930.0.

The migration is substantially implemented. Native sources, combined operator documents and imports, recursive overlays, source-relative prompt content, captured engine lifetime, immutable checkpoint identity, and credential-free operator inspection are in place. The provider-neutral execution boundary, prepared execution, retries, scheduling, and durable artifact contracts remain intact. All 16 migrated prompt bodies match the previous definitions after removing their old embedded identity fields; prompt fragments and schemas are unchanged.

Four bounded follow-ups remain below. No production code is changed by this review. No additional runtime security restriction, mandatory credential check, filesystem permission policy, compatibility shim, or release gate was found. Eager final-definition validation and the external keyed format are intentional upstream changes, not accidental strengthening. The previously recorded ConfigKit remediation was completed before this work set; it is not outstanding.

Finding 1 — Compose Registered Application Assets In One Strict Group

Priority: P2. Category: architecture/correctness gap. Status: open.

Evidence:

  • internal/framework/llm/asset_registry.go:106–139 creates one underlay per registered prompt or fallback-profile filesystem.
  • internal/framework/llm/asset_registry_sources_test.go, TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups, asserts that separate registrations yield separate underlays, encoding this divergence.
  • The plan's Sources And Engine Ownership section calls for one application underlay whose inputs each retain their own filesystem context; docs/internal/llm.md:86–92 also describes that behavior.

Separate underlays have recursive overlay semantics. Conflicting application definitions in different registrations therefore silently depend on module registration order instead of receiving native strict-composition diagnostics. An offline probe registered two fallback files with the same profile key and different models. The current inspector accepted them and selected the second model. The exact same filesystem inputs placed in one native underlay correctly failed composition. Distinct filesystem contexts do not require separate layers.

Required changes:

  • Gather the native FilesystemInputs from every application prompt and fallback-profile registration into one application underlay. Preserve each input's existing label, filesystem, root, and disabled-import setting. Keep schemas on their separate runtime option and omit an empty application group.
  • Keep the operator and explicit local-backend contributions as later groups. Preserve their accepted recursive overrides and source precedence. Do not add an application-specific duplicate-key detector or flatten filesystem contexts into an artificial shared root.
  • Replace the incidental group-count test with behavioral coverage: conflicting application contributions fail through native strict composition; distinct declarations retain source-relative content; an operator overlay over the resulting application base still succeeds. Reuse existing origin coverage.

Acceptance: application conflicts are not silently resolved by registration order; source-relative imports/content and valid later operator overlays remain unchanged. Runtime, preflight, and reports use the corrected shared construction.

Validation: GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/....

Finding 2 — Validate Explanation Syntax Before Loading Configuration

Priority: P3. Category: CLI correctness. Status: open.

Evidence: internal/cli/configkit_commands.go:77–106. PromptKit scope loads Notarius configuration and constructs the inspection engine before validating the pointer; the default Notarius scope parses the pointer first.

Reproduction, using the native temporary build and a nonexistent config path:

notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'

The current PromptKit command returns exit 1 with a config-file error. The same malformed pointer returns exit 2 with a valid config, and the Notarius scope returns exit 2 even with the missing config. A bad PromptKit definition or missing content file can likewise mask malformed command syntax. This differs from the selected syntax/runtime distinction in the plan and CLI reference.

Required changes:

  • Validate canonical JSON Pointer syntax once, before either scope loads config or constructs an engine. Reuse the existing parser; retain defensive parsing inside the adapter for direct callers.
  • Keep the empty root pointer, escaped literal keys, historical-only paths, unknown-path exit 1, and writer-error behavior unchanged.
  • Extend CLI coverage with a malformed pointer and unavailable configuration; verify exit 2, no stdout, and no configuration/provider work. No broader argument-parser refactor is needed.

Acceptance: malformed pointers are syntax errors independent of configuration availability for both scopes; valid selectors still load and inspect normally.

Validation: GOWORK=off go test ./internal/cli ./internal/framework/llm.

Finding 3 — Correct Lifecycle And Fallback Documentation

Priority: P3. Category: documentation accuracy. Status: open.

Evidence:

  • docs/config.md:321–323 says prompt and schema definitions are validated at engine construction. Schemas remain preparation-time resources supplied through WithSchemaFS, not captured configuration definitions.
  • docs/internal/llm.md:71–77 and docs/internal/cli.md:55–61 describe preflight as occurring without loading a prompt. Construction now loads and validates effective prompt definitions and captures message files/static templates; it does not prepare a request or generate output.
  • The comment on RegisterFallbackProfileFS in internal/framework/llm/asset_registry.go:61–62 still says fallback assets are used only when the operator source lacks the ID. Under recursive overlays, their omitted fields remain active beneath a matching operator entry.

Required changes:

  • Distinguish constructor-time definition/message capture from preparation-time schema loading and request-variable/artifact resolution. State accurately that preflight needs no transcript, credential value, or provider call; do not promise it performs no prompt-content file reads.
  • Describe fallback registration as a lower-precedence application contribution rather than an all-or-nothing lookup fallback.
  • While editing this text, clarify that an operator backends.local declaration can supply the backend as well as Notarius's optional local_backend object. The latter is a convenience contribution, not a required second declaration. Link canonical source rules rather than duplicating them.
  • Change documentation/comments only unless verification exposes a real runtime defect. Do not move schema loading earlier to make the stale prose true.

Acceptance: public and internal guidance consistently explains actual lifecycle, native layering, and operator backend configuration without inventing a stronger preflight guarantee or an additional required config field.

Validation: inspect current constructors/prepared execution and upstream v0.12 contracts; review affected local links; run git diff --check. Reuse unchanged behavioral checks rather than adding prose-snapshot tests.

Finding 4 — Preserve ADR-0010's Historical Decision Body

Priority: P3. Category: documentation-policy compliance. Status: open.

Evidence: the migration diff rewrites ADR-0010's Decision and Consequences paragraphs at docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33 and :47–50, as well as adding the appropriate status/supersession link. docs/policy/documentation.md requires accepted decision content to remain immutable; Stage 14 specified a concise status link rather than rewriting the historical body.

The replacement correctly acknowledges the old behavior, but the historical record should retain its original wording. ADR-0019 already owns the new native overlay decision and accurately limits which part of ADR-0010 it supersedes.

Required change: restore ADR-0010's Decision and Consequences body from f5b066b, keeping its new partial-supersession status and ADR-0019 link. Leave workload ownership and ADR-0019 unchanged. Current runtime behavior remains documented in the canonical current references, not retrofitted into the older decision.

Acceptance: the historical body matches the accepted original, the changed clause's supersession remains discoverable, and no runtime behavior changes.

Validation: compare the body with git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md, verify the ADR link, and run git diff --check.

Review Evidence And Validation

  • GOWORK=off go test ./... passed.
  • GOWORK=off go vet ./... passed.
  • GOWORK=off go test -race ./internal/framework/llm ./internal/cli passed.
  • GOWORK=off go mod tidy -diff produced no changes.
  • Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
  • A native build outside the repository succeeded. Credential-free profile and explanation commands and complete/development pipeline validation succeeded.
  • A YAML-aware comparison confirmed all 16 migrated prompt definition bodies are unchanged after removing old embedded IDs/versions. Prompt fragments, schemas, and the embedded D&D execution settings are unchanged.
  • An offline Go overlay probe, stored entirely outside the repository, confirmed the application-group conflict behavior in Finding 1. CLI invocations reproduced Finding 2 without generation or provider calls.
  • The implementation diff passed whitespace checks. No production inference or production model-quality evaluation was performed.

Recommended remediation is limited to these findings, not another configuration system, new security policy, or unrelated pipeline refactoring.

ConfigKit 1.2.0 Review

Reviewed on 2026-11-17 against implementation revision c9744ad, compared with planning revision 79088a3 and the ConfigKit feature roadmap and implementation plan. The subsequent roadmap-only clarification in d88a01c does not change the reviewed runtime. ConfigKit v1.2.0 resolves to c1c7e87f2cdf77a0cbf3327831ac05578e756b9e.

The production architecture matches the target. Application configuration uses new-path ConfigKit v1.2.0; the unchanged PromptKit v0.12.0 adapter retains old-path ConfigKit v0.5.0. Both use YAML v4.0.0-rc.6 under Go 1.27.0. The CLI describes one exclusive file or ordered native search source, while ConfigKit owns acquisition, imports, collections, inheritance, decoding, diagnostics, and inspection. Notarius supplies domain defaults, recorded normalization, validation, detached runtime conversion, and catalog-dependent resolution.

No new production correctness defect or unwarranted restriction was found. The changed controls, optional import defaults, post-decode CLI assignments, step identity/upsert, search behavior, and higher Go minimum are approved scope. The remaining gaps are two operator-documentation inaccuracies from Stage 9, also missed by Stage 11's final documentation check. Production fixes are not required for either finding.

Finding CK-1 — Distinguish Mapping And Record Inheritance Syntax

Priority: P3. Category: documentation accuracy. Status: open.

Evidence: docs/config.md:143–149 introduces both mapping and identified-record inheritance, then says a directive names an immediate sibling in the same mapping. That describes mapping inheritance only. For an identified step record, the parent belongs to the same sequence and the reference must be an identity mapping, such as extends: {id: parent}. The reference never shows that form.

An offline CLI probe with a parent step providing references and a child step providing an artifact lane confirmed that extends: parent fails with inheritance.invalid, while changing only the directive to extends: {id: parent} loads successfully and preserves the inherited reference. TestIdentifiedStepRecordInheritance already exercises the correct form in internal/core/config/collection_composition_test.go:182–208.

Required changes:

  • Explain string-valued mapping-sibling inheritance and identity-mapping sequence-record inheritance separately, including their distinct lookup scopes.
  • Add a minimal step-record example or a direct link to the pinned upstream syntax example. Preserve the existing rules for null cancellation, validation, and whole-record upsert; no new application inheritance machinery is needed.

Acceptance: an operator can determine the correct syntax for both a pipeline mapping and a step record from the canonical reference. Any added snippet loads through the current configuration boundary. Reuse the existing behavioral test; do not add a documentation snapshot or change runtime behavior to fit the prose.

Finding CK-2 — Describe Dependency-Ready Step Assembly Order

Priority: P3. Category: documentation accuracy. Status: open.

Evidence: docs/config.md:167–170 says distinct imported step IDs combine "in source order." ConfigKit distinguishes acquisition/import declaration order from dependency-ready assembly order. Ready contributions are selected by their stable source keys, so independent branches can interleave. The feature roadmap and Stage 8 correctly rely on dependency-ready assembly rather than a Notarius append pass.

An offline CLI probe declared imports in the order z-first.yml, then a-second.yml, each contributing a different step and lane. Effective output placed the step from a-second.yml first, then the step from z-first.yml. Thus declaration order does not determine the resulting step list. This matters because Notarius executes steps in the resolved list order and validates their generated-reference dependencies.

Required changes:

  • Replace the source-order claim with the native dependency-ready assembly rule and link to ConfigKit v1.2.0's pinned composition contract for its full ordering semantics. Do not promise declaration order or duplicate the upstream scheduler.
  • Explain briefly why the maintained complete example's imported describe-session step precedes the importing body's extract-events step: the imported dependency is assembled before its importer. Keep the canonical operator rule in docs/config.md; other documentation should link rather than redefine it.

Acceptance: the reference does not imply that reordering independent import declarations necessarily reorders steps. The complete example's actual step order and generated-reference handoffs remain unchanged. No custom sorting, merging, or scheduler changes belong in this remediation.

Stage-By-Stage Review Evidence

Stage Review result
1 — Dependencies/toolchain The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff.
2 — API/handoff Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values.
3 — Normalization/origins Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass.
4 — Collections The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, record inheritance, and source-relative references.
5 — Fixtures Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes.
6 — CLI discovery The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed.
7 — Discovery/failures Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files.
8 — Examples Minimal, complete, and development pipelines validate without inference; the supporting fragment preserves module choices, step order, and handoffs.
9 — Operator docs Search, controls, path bases, CLI timing, and toolchain guidance are updated. CK-1 and CK-2 remain.
10 — Internal/integration docs and notices The mixed dependency boundary and pinned ConfigKit attribution match the selected graph. Accepted ADRs, policy, and project license/grant were not modified by the sprint.
11 — Final verification Full tests, vet, tidy, native build, dependency inspection, and provider-free validation/effective/explanation commands pass. The final documentation review missed CK-1 and CK-2.

ConfigKit Review Validation

  • GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./... passed.
  • GOTOOLCHAIN=go1.27.0 GOWORK=off go vet ./... passed.
  • GOTOOLCHAIN=go1.27.0 GOWORK=off go test -race ./internal/core/config ./internal/cli passed.
  • GOTOOLCHAIN=go1.27.0 GOWORK=off go mod tidy -diff produced no changes.
  • A native build in a temporary directory succeeded. Minimal, complete, and development configuration validation, plus application and PromptKit effective and explanation reports, succeeded without provider calls.
  • Selected module inspection confirmed the four approved ConfigKit, PromptKit, and YAML versions. Pinned local upstream documentation and notices were checked.
  • Temporary fixtures outside the repository reproduced both documentation gaps and verified combined identifier/reference-path normalization. No production pipeline, credential lookup, or live model-quality evaluation was performed.
  • The review document passed git diff --check; production code and current operator documentation were not changed by this review.

Keep the earlier PromptKit findings separate from these two documentation fixes. The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope.