Files
notarius/docs/roadmap/configkit.md

297 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Adopt Current ConfigKit Configuration Semantics
## Purpose
Adopt `go.maximumdirect.net/configkit` v1.2.1 and its native consumer workflow.
The objective is consistent configuration semantics across ConfigKit-consuming
applications. Existing Notarius composition behavior is a migration baseline,
not a compatibility requirement to reproduce through adapters.
[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md) owns the
boundary: ConfigKit delivers the final resolved application document; Notarius
supplies application schema, defaults, normalization, validation, and subsequent
pipeline resolution. This roadmap defines the target and required work. A later
implementation plan will own sequencing. Release publication is separate work.
## Reviewed Baseline And Sources
Reviewed on 2026-11-17 against Notarius `b9899ea`:
- ConfigKit v0.5.0 under `gitea.maximumdirect.net/eric/ConfigKit`;
- PromptKit v0.12.0, which consumes that same old ConfigKit package;
- Go 1.25.5 and go-yaml v4.0.0-rc.6;
- Notarius configuration version 5, imports, overlays, sibling inheritance,
declaration-relative application paths, and effective/explanation CLI reports.
The released v1.2.1 tag in `../ConfigKit` identifies source commit
`51ef3144158db1a09fabdbd1c7ae59908c84923b`. Review references:
- [Consumer adoption guide](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.2.1/docs/consumers/README.md)
- [Configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.2.1/docs/config.md)
- [Release notes](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.2.1/docs/releases/)
All intervening release notes were read in the local checkout. The important
changes are:
| Releases | Relevance to Notarius |
| --- | --- |
| v0.6.0 | Loading/inspection optimizations; Go minimum becomes 1.26.0. No promised application-wide speedup. |
| v0.7.0–v0.7.1 | Frozen effective-configuration diffs; maintainer tooling update. |
| v0.8.0 | Input destinations, sequence-item placement, source-version checks, and identified sequences. |
| v0.9.0 | Expanded imports; imports optional by default; author-declared replace/append/upsert; identified-record inheritance; CLI assignments move after inheritance and decoding. |
| v0.10.0–v0.10.2 | Explicit configuration discovery; broader absent-main lifecycle; loading, ownership, acquisition, and history fixes. |
| v1.0.0 | New module path, source-root `config` controls, public API cleanup, source lookup, and Go 1.27.0 minimum. |
| v1.1.0 | Field-level normalization inside lists; identity-aware provenance/history across record movement. |
| v1.2.0 | Main-relative companion inputs and layered overlay context selection. |
| v1.2.1 | Stable go-yaml v4.0.0 and loader-plugin adaptation. |
### Current Integration Areas
`internal/core/config/load.go` supplies ConfigKit's definition and callbacks.
`loaded_config.go` privately retains its inspection result and publishes a
detached runtime value. `document_paths.go`, `pipeline_document.go`, and
`document_normalizer.go` implement application normalization and source tracking.
The normalizer currently records whole-list coverage even for leaf edits.
`internal/cli/configkit_load.go` and `run.go` own selection and diagnostics.
`discoverConfigPath` manually probes files before `config.Load` calls ConfigKit.
The complete example imports its pipeline through root `configkit`. Current
documentation describes unconditionally replacing whole step lists.
ConfigKit types also appear inside the PromptKit adapter's native source and
inspection APIs. That is a distinct integration boundary with the dependency
constraint below; a repository-wide import replacement would be incorrect.
## Target End State
### Native Loading And Ownership
Use the new public package and the ordinary `Load` workflow for the application
configuration. Defaults, `Resolve`, `Validate`, `VersionRule`, and sensitivity
classification remain supported application extension points. The handoff
contains fully composed, decoded, normalized, and validated application values.
Runtime conversion and catalog-dependent pipeline preparation retain their
application ownership.
ConfigKit owns import acquisition, ordering, inheritance, collection operations,
CLI assignment semantics, limits, source evidence, and frozen inspection. Pass
its results and structured failures through the established adapter boundary.
Application validation checks the resulting domain configuration; it does not
audit superseded contributions or replay structural directives.
Keep configuration `version: 5`: its typed application shape is unchanged.
Explain the deliberate ConfigKit syntax/semantic migration in operator guidance
and the eventual minor-release note. Existing files require migration rather
than a compatibility pre-parser or legacy control aliases.
### Source Controls, Imports, And Composition
Use source-root `config` controls in Notarius configuration inputs. Update real
control blocks and stored declaration pointers, including imported files,
overlays, mapping fixtures, and examples. Nested application data named `config`
retains native treatment. Former `configkit`/`_configkit` keys are ordinary data;
ordinary typed validation governs them. No custom legacy-control scanner is
needed.
This control migration applies to Notarius's application loader. PromptKit
operator files and adapter fixtures follow the version of PromptKit that reads
them; retain their old control spelling while that separate engine remains old.
Enable ConfigKit imports and accept its native path/list, directory, glob,
destination, placement, exclusion, recursion, hidden-file, symlink, and
requiredness rules. Leave `ImportDefaults` unspecified unless an application
choice is explicitly approved. In particular, do not restore former implicit
requiredness or impose a YAML-only directory filter globally. Required deployed
fragments, such as the maintained complete pipeline import, should say
`required: true` in their declaration so an incomplete installation fails there.
Optional imports remain genuinely optional.
Use the native main-only source-version default; fragments do not acquire a new
mandatory version declaration as a side effect of this upgrade.
Expose native identified sequence operations and inheritance through the source
format. Without identity, lists retain native peer-conflict/whole-replacement
behavior. Identified lists support strict peer composition and native
replace/append/upsert across tiers. Upsert replaces a complete matching record,
retaining its position; it is not a recursive merge of that record. Clearing an
append/upsert collection requires explicit replacement with an empty list.
Dependency-ready assembly order, not a Notarius sorting or append pass, controls
composition. Final Notarius checks still validate stage order, supported modules,
reference dependencies, and unique application identifiers.
Step identity registration is an explicit decision below. Other identified
collections remain available through author declarations; no broad automatic
`id` inference or universal registration is required. Demonstrate a composed
step list and a derived pipeline using native controls in maintained examples
or small canonical snippets, with behavioral coverage.
CLI overrides apply after inheritance and typed decoding, before application
resolution. Preserve explicit zero/false/empty values. Structural errors cannot
be repaired by CLI assignments; document this instead of rebuilding the former
override order. Notarius's binding-profile and validation-policy rules remain
domain semantics applied to the final configuration.
### Native Source Selection And Resources
Replace manual acquisition/probing with ConfigKit source descriptions. Explicit
`--config` and, when no CLI selection exists, `NOTARIUS_CONFIG` select exclusive
`File` sources: a missing selected file fails instead of searching elsewhere.
For default discovery use `SearchConfig` with an approved search convention;
the recommended Unix-system preset and its alternative are recorded below.
Use a required search for commands that require a configuration file, preserving
their need for a selected configuration. This does not impose requiredness on
all imports or on separate defaults-only programmatic loads.
Report the actual selected main file using retained source information; update
load/error plumbing rather than making a second discovery pass. Construction of
search descriptions remains side-effect-free, and acquisition errors follow
ConfigKit's native selected-file behavior.
Retain the deliberately selected application path policy: file-authored values
use their declaring file, CLI paths use the invocation directory, and relative
application defaults use the canonical main file. Use `EntryInfo`, source records,
and public resource contexts where appropriate. Native destinations and source
coordinates are distinct: a mounted or inherited value keeps its source base.
No new target-existence, confinement, symlink, or permission policy is added to
application resource paths. ConfigKit's own input-root rules remain upstream.
Ordinary `Load` is sufficient for current application composition. `LoadLayered`,
`MainRelative`, and overlay fallbacks are available when actual bundled/companion
inputs need them; the upgrade does not require an artificial extra input layer.
### Lean Application Resolution And Inspection
Replace `InputInfo` with `EntryInfo`, `Definition.Secret` with `Sensitive`,
generic `Record` calls with the corresponding named recording methods, and
`LayerBase`/`"base"` with `LayerMain`/`"main"` where used. Audit all affected
symbols and diagnostic strings against the v1 migration table. Use assignment-
and-record helpers when appropriate; named recording methods alone do not mutate
the application value.
Use v1.1 field-level recording for list leaf normalization. Remove the old
blanket whole-sequence coverage wrapper where its only purpose was satisfying
the old audit. Record actual structural mutations at their proper scope. Resolve
source-relative values before moving records; fixed entry positions, final
positions, historical evidence, and effective origins serve different purposes.
Keep map-key relocation bookkeeping only where actual application identifier
normalization still needs it. Do not add an independent history-replay engine.
Effective/explanation commands delegate to ConfigKit's captured inspection,
retaining redaction, presence, source history, and detached ownership. Source
lookup APIs support useful diagnostics, including empty import selections when
needed, without turning ordinary optional absence into warnings. ConfigKit's
`Diff` is available for a later operator-facing comparison command; a new CLI
diff or secrets-management workflow is not required for this adoption.
## Dependency And Toolchain Boundary
The target is ConfigKit v1.2.1, stable go-yaml v4.0.0, and a Go consumer minimum
of 1.27.0. Update the selected release-validation toolchain, setup guidance, and
actual configured build/CI environments consistently. Minimum and validation
patch remain separate choices. Include Go and configuration migration impact in
the eventual release note; patch releases cannot raise the Go minimum.
PromptKit v0.12.0 exposes **old-path ConfigKit types** through `Config.Sources`,
its source helpers, and inspection. Keep these contained in the LLM adapter
until its separate upgrade; ordinary Notarius configuration uses new-path types.
Prefer neutral scalar/domain handoffs over translating dependency-owned graphs.
There is also a concrete shared-dependency incompatibility:
- ConfigKit v1.2.1 requires stable go-yaml v4.0.0.
- Old-path ConfigKit v0.5.0 imports `go.yaml.in/yaml/v4/plugin/limit`.
- Stable go-yaml contains `plugin/loaderlimits` and no `plugin/limit`.
- Go selects one version of this shared YAML module, despite ConfigKit itself
having two distinct module paths.
Therefore keeping PromptKit v0.12.0 and its unpatched ConfigKit v0.5.0 does not
produce a buildable v1.2.1 target. Resolve the open dependency decision before
finalizing the implementation plan. Neither a local fork/replace nor silently
ending at ConfigKit v1.2.0 is the target architecture. This planning pass made
no dependency changes and ran no trial application build.
## Required Repository Work And Acceptance
- Migrate application-side dependency imports, APIs, diagnostics, source
declarations, inspection, and discovery; resolve the PromptKit/YAML boundary
through the approved dependency path. Native builds must pass at completion.
- Update real-file loading, imports, overlays, inheritance, CLI assignment,
path-origin, failure, and ownership tests under `internal/core/config` and
`internal/cli`. Add representative collection composition and step-reference
normalization cases, including imports in distinct directories and leaf
histories after list composition. Test native semantics, not the old behavior.
- Retain two top-level configuration examples. Migrate the complete example's
required import and demonstrate useful native composition without adding a
second application merger or an unnecessary example framework.
- Revise `docs/config.md`, `docs/cli.md`, `docs/internal/configuration.md`, and
affected operations/subprocess guidance for syntax, search, imports, collections,
CLI ordering, origins, and inspection. Update README/toolchain/release guidance
and affected dependency integration references. Preserve accepted ADR history;
ADR-0020 already supplies the durable ownership rationale.
- Update `docs/oss/attribution.md` from actual selected module revisions and
notices, including coexistence of old/new ConfigKit if applicable, stable YAML,
and newly selected transitive dependencies. Preserve the application grant.
Upstream ReleaseKit is maintainer tooling; adopting it as Notarius release
machinery or adding its gates is outside this feature.
- Verify focused configuration/CLI/LLM tests, full offline tests, vet, a native
build, maintained examples and report commands without provider calls, tidy
consistency, selected module identities, and affected links. No production
inference, fixed performance improvement, new release gate, or broader
filesystem/security guarantee is part of acceptance.
Completion means current native configuration semantics are usable and documented
on a coherent dependency graph, with final domain validation and no unresolved
directives at runtime. The old ConfigKit 0.5 roadmap is replaced by this feature
scope; unrelated completed plans and review findings remain separate records.
## Open Decisions
### 1. Resolve The Stable-YAML Conflict With Old PromptKit
Recommendation: seek a narrowly scoped **released upstream compatibility patch**
for the old ConfigKit module used by PromptKit, adapting it to stable go-yaml.
If upstream supports that patch, the two module identities can coexist briefly
and Notarius can complete its ConfigKit upgrade before the separate PromptKit
feature migration. This preserves the desired ordering using supported releases
rather than adding application-owned compatibility machinery. No such patch is
assumed to exist or assigned an invented release number here.
Alternative: coordinate the PromptKit adapter's module/API migration with this
cutover and select a current PromptKit release that consumes ConfigKit v1.2.1.
This is the cleaner long-term dependency graph and avoids maintaining a legacy
patch, but couples the two implementation work sets. If selected, prepare the
PromptKit roadmap before claiming a standalone ConfigKit sprint can finish.
A temporary ConfigKit v1.2.0 checkpoint can assist that coordinated migration
because it still selects YAML rc.6, but it does not complete this roadmap's
v1.2.1 target. Dependency patches/forks inside Notarius are not the recommendation.
### 2. Default Main-File Search Convention
Recommendation: use ConfigKit's `UnixSystemDirectories()` preset for default
Notarius discovery, with `SearchOptions.Required` enabled. It fits the
sysadmin-focused deployment, retains the existing installed location among its
candidates, and shares conventional names/order with other ConfigKit consumers.
Explicit CLI/environment selections remain exclusive file choices. Document the
new candidates and native first-selected-file failure behavior.
Alternative: use `SearchPaths` containing only the current installed default
`/usr/local/etc/notarius/config.yml`. This still delegates acquisition to ConfigKit
and minimizes discovery changes, but offers less convention sharing. User/home
or working-directory presets can be selected deliberately if a broader discovery
workflow is desired; they are not silently enabled by the recommended preset.
### 3. Application Identity For Pipeline Steps
Recommendation: register `id` for `/pipelines/<any-mapping-key>/steps` using
`SequenceIdentityMatching`. Step IDs already have application significance and
uniqueness checks. Native identity makes independently imported steps composable
and gives overlays/inheritance the ordinary upsert default. Authors can explicitly
select replacement or append. Document the full-record upsert rule and clearing
semantics, and exercise reference-dependent paths after composition.
Alternative: leave sequence registrations empty and let authors opt in with
source-local `config.sequences` declarations. This is also ConfigKit-native and
preserves ordinary unregistered-list replacement when no declaration is supplied,
but each composed pipeline must declare its collection identity. In either case,
avoid automatic identity inference for validator chains or other unrelated lists.