Outline native ConfigKit adoption and configuration ergonomics
This commit is contained in:
359
docs/roadmap/configkit.md
Normal file
359
docs/roadmap/configkit.md
Normal file
@@ -0,0 +1,359 @@
|
||||
# Adopt ConfigKit v1.2.1 and native configuration ergonomics
|
||||
|
||||
## Purpose
|
||||
|
||||
Adopt current ConfigKit as Narratio's configuration owner and make its native
|
||||
authoring and inspection features useful to operators. Align configuration
|
||||
semantics with other ConfigKit consumers. Existing Narratio assembly conventions
|
||||
are migration inputs rather than compatibility requirements for this work.
|
||||
|
||||
This roadmap defines the target and required changes. A subsequent implementation
|
||||
plan will define staged work after the open decisions are resolved. This pass
|
||||
adds documentation only. The earlier [v0.5.0 roadmap](config.md) and
|
||||
[implementation plan](implementation.md) describe the completed baseline.
|
||||
|
||||
The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md)
|
||||
continues to apply: ConfigKit resolves configuration before its final handoff;
|
||||
Narratio owns domain validation, application derivation, and runtime use.
|
||||
Supported callbacks and native metadata provide the integration boundary.
|
||||
|
||||
## Reviewed baseline and releases
|
||||
|
||||
Narratio currently selects `gitea.maximumdirect.net/eric/ConfigKit v0.5.0`,
|
||||
commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd`, with Go 1.26.0 as its
|
||||
consumer minimum. Release builds currently select Go 1.26.7.
|
||||
|
||||
The adoption target is `go.maximumdirect.net/configkit v1.2.1`, commit
|
||||
`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The published tag and the sibling
|
||||
checkout agree on that revision. ConfigKit requires Go 1.27.0 and uses
|
||||
`go.yaml.in/yaml/v4 v4.0.0`. Its code license remains LGPL-3.0-or-later.
|
||||
|
||||
The pinned [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
|
||||
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
|
||||
own upstream semantics. The intervening releases supply the following relevant
|
||||
changes:
|
||||
|
||||
| Release | Adoption relevance |
|
||||
| --- | --- |
|
||||
| v0.6.0 | Loading and metadata optimizations; Go minimum moves to 1.26.0. |
|
||||
| v0.7.0 | Native safe effective-configuration `Diff`. |
|
||||
| v0.7.1 | Maintainer-tool update; runtime behavior unchanged. |
|
||||
| v0.8.0 | Input/import destinations, sequence-item placement, shared source-version checks, and initial identified-sequence composition. |
|
||||
| v0.9.0 | Directory and expanded import selection; optional-by-default imports; identified-list replace/append/upsert and inheritance; CLI assignments move after decoding and inheritance. Its final collection APIs replace the initial v0.8 registrations. |
|
||||
| v0.10.0 | Native ordered configuration discovery through `SearchConfig`. |
|
||||
| v0.10.1 | Loading consolidation and import bookkeeping improvements. |
|
||||
| v0.10.2 | Correctness fixes for decoding, callback/resource lifetime, discovery, inspection budgets, and collection operations. |
|
||||
| v1.0.0 | Vanity module path, source-root `config` controls, public API cleanup, source lookup, provenance and sensitivity fixes, and Go 1.27.0 minimum. |
|
||||
| v1.1.0 | Field-level list resolution records and identity-aware inspection across list movement. |
|
||||
| v1.2.0 | Main-relative companion inputs and overlay context for layered loading; overlay filesystem fixes. |
|
||||
| v1.2.1 | Stable go-yaml v4.0.0 dependency; maintainer release-tool update. |
|
||||
|
||||
The corresponding `docs/releases/<version>.md` files at the pinned revision
|
||||
provide the migration details. Select v1.2.1 directly; intermediate releases
|
||||
provide context rather than required deployment stops.
|
||||
|
||||
## Current Narratio integration
|
||||
|
||||
The [configuration reference](../config.md) and
|
||||
[configuration internals](../internal/configuration.md) describe current behavior.
|
||||
|
||||
- Commands select a pipeline path through Narratio's search loop, bootstrap
|
||||
credentials from an explicit directory or a sibling `secrets/` directory,
|
||||
and then call ConfigKit.
|
||||
- Pipeline loading enables explicit path/glob imports and one selected overlay.
|
||||
Campaign, session, and canonical-party documents use separate loads. Session
|
||||
loading also supports command-scoped downloaded input snapshots.
|
||||
- Typed defaults, recorded normalization, domain validation, and native
|
||||
provenance are integrated. Additional path/dependency resolution and
|
||||
campaign-owned family expansion occur after the pipeline load.
|
||||
- Artifact files currently repeat enclosing `scriptorium.artifacts` mappings.
|
||||
Logical IDs come from map keys. Families expand against canonical party data.
|
||||
- Show, diff, and digest use a safe finalized-domain projection. Narratio owns
|
||||
the current mapping-diff algorithm. Native load metadata supplies source and
|
||||
artifact-catalog evidence.
|
||||
|
||||
Some public import descriptions still state restrictions from older ConfigKit
|
||||
behavior. Replace these with current native semantics during adoption rather
|
||||
than recreating them as Narratio checks.
|
||||
|
||||
## Targeted end state
|
||||
|
||||
### Native assembly and operator choice
|
||||
|
||||
ConfigKit owns source discovery, acquisition, composition, inheritance, presence,
|
||||
typed secrets, provenance, and safe native inspection. Narratio supplies its
|
||||
schema, domain defaults and validation, supported application resolution, and
|
||||
operator-selected inputs.
|
||||
|
||||
Use ordinary `Load` for an ordinary configuration. Use public native facilities
|
||||
where they serve the operator workflow. Layered loading is appropriate when
|
||||
actual lower-priority inputs are needed; typed defaults remain appropriate for
|
||||
simple defaults. Main-relative companion inputs allow such underlays to follow
|
||||
a discovered main without Narratio searching for it again.
|
||||
|
||||
Expose the library's authored `config.imports` and `config.sequences` semantics
|
||||
where applicable. Imports use ConfigKit defaults, including optional absence,
|
||||
unless the operator explicitly requests other supported behavior. An invalid
|
||||
selected source still fails. Operators mark essential imports `required: true`.
|
||||
Directory imports can select YAML with an explicit `pattern`; unrestricted
|
||||
filenames remain the library default.
|
||||
|
||||
Native destinations, nested imports, discovery filters, source-local relative
|
||||
import paths, collection operations, and inheritance work under ConfigKit's
|
||||
rules. Narratio validates the final effective domain entries. It adds no
|
||||
fragment-count rule, mandatory fragment version, import-mode allowlist, or
|
||||
parallel composition engine.
|
||||
|
||||
### Ergonomic artifact files and list customization
|
||||
|
||||
Support an artifact definition without repeated enclosing mappings through an
|
||||
explicit import destination. For example:
|
||||
|
||||
```yaml
|
||||
config:
|
||||
imports:
|
||||
- path: artifacts/session-recap.yml
|
||||
destination: [scriptorium, artifacts, session_recap]
|
||||
required: true
|
||||
```
|
||||
|
||||
The imported file contains the artifact fields such as `prompt_id`, `inputs`,
|
||||
and `output_path`. The literal destination defines the reusable key
|
||||
`session_recap`, with logical source `narratio.artifact.session_recap`.
|
||||
The filename is an organizational choice. Keyed files mounted at
|
||||
`[scriptorium, artifacts]` support declared directory or glob imports when many
|
||||
definitions should be discovered together. Apply the same pattern to families.
|
||||
|
||||
Keep keyed artifact/family mappings and campaign-owned character data. They
|
||||
already fit native inheritance and stable identity. Full-root and disjoint
|
||||
definitions remain valid native inputs; a one-artifact-per-file organization is
|
||||
guidance rather than a validity requirement.
|
||||
|
||||
Allow authored native collection declarations for existing list fields where
|
||||
their final shapes satisfy Narratio's schema. Operators can choose replacement,
|
||||
append, or upsert and identified-record inheritance. Explain that upsert
|
||||
replaces whole matching records and that an empty append/upsert contribution
|
||||
does not clear a list. Register application-owned sequence identities only
|
||||
where Narratio has a real domain identity requirement; generic library
|
||||
capabilities do not establish new uniqueness requirements for every list.
|
||||
|
||||
### Discovery, credentials, and resource paths
|
||||
|
||||
Replace Narratio-owned local candidate probing with ConfigKit search descriptions
|
||||
and `SearchConfig`. Explicit `--config` selection remains an exclusive `File`
|
||||
input. Report the source actually selected by the library. The open decisions
|
||||
below select search conventions and the independent bootstrap directory needed
|
||||
before a discovered load resolves credentials.
|
||||
|
||||
Bootstrap uses `LoadSecretsIntoEnvironment` once per invocation before typed
|
||||
credential resolution and concurrent work. Existing environment variables win,
|
||||
including explicitly empty variables. Keep explicit `--secrets-dir`, safe
|
||||
partial-report handling, and exact-name forwarding to subprocesses. The selected
|
||||
bootstrap convention must allow native discovery without a second preliminary
|
||||
configuration load or a duplicate file-selection loop.
|
||||
|
||||
Use native source/resource context for application path interpretation. The
|
||||
resource-base decision below applies to file references rather than artifact
|
||||
destinations, command names, or remote object keys. ConfigKit's consumer examples
|
||||
illustrate application policy; they do not impose it on Narratio.
|
||||
|
||||
### Final-domain inspection through native APIs
|
||||
|
||||
Keep validate, show, sources, artifact catalog, and diff useful for the operator's
|
||||
fully resolved pipeline, including campaign-driven members and publish rules.
|
||||
Use native effective rendering and `Diff` where they own the operation, with
|
||||
Narratio formatting and generated-domain context at the presentation boundary.
|
||||
|
||||
A raw pipeline `Result` is a frozen load snapshot. Mutating its `Config` later
|
||||
does not update rendering or diff. Integrate this distinction explicitly:
|
||||
record domain resolution within a supported resolver when its inputs are
|
||||
available there; where later campaign resolution remains appropriate, create
|
||||
a completed native inspection result from the finalized safe domain projection
|
||||
using the public mapping-source API. That result describes application-derived
|
||||
values; retained authoring results supply actual file provenance.
|
||||
|
||||
Use native comparison rather than maintaining a second general mapping-diff
|
||||
engine. Preserve final-domain coverage and credential identities. Narratio's
|
||||
digest and stage-owned reuse evidence remain application responsibilities.
|
||||
Source-layout changes alone do not change equivalent domain values. Source
|
||||
inspection can report composite origins, empty optional-import selections, and
|
||||
application derivation without inventing a unique defining-file contract.
|
||||
|
||||
### Scope and compatibility
|
||||
|
||||
This is a clean native cutover. Update source-root `configkit:` controls to
|
||||
`config:` and remove old-module/API references in maintained code and fixtures.
|
||||
Nested `config` data remains ordinary data under ConfigKit's source-root rules.
|
||||
The `configkit` package name and `configkit:"secret"` Go tag retain their
|
||||
upstream spelling. Mapping and identified-list `extends` follow current native
|
||||
rules.
|
||||
|
||||
This roadmap deliberately permits revised configuration semantics. Update
|
||||
tests and documentation for selected changes rather than preserving historical
|
||||
acceptance rules through compatibility loaders or keyword aliases.
|
||||
|
||||
Application identity, artifact dependency meaning, runtime output destinations,
|
||||
campaign-owned party authority, remote session retrieval, and stage reuse remain
|
||||
Narratio domain concerns. This scope does not redesign the runtime workspace,
|
||||
change subprocess adapters, adopt ReleaseKit for Narratio, or add external
|
||||
provider probes. Audita and workspace work retain their separate roadmaps.
|
||||
|
||||
## Required changes
|
||||
|
||||
### Dependency, toolchain, and notices
|
||||
|
||||
- Select the released `go.maximumdirect.net/configkit v1.2.1` module and migrate
|
||||
all imports and exchanged named types together. Remove the old requirement;
|
||||
use the released dependency rather than a filesystem replacement.
|
||||
- Raise `go.mod` to Go 1.27.0 and select an available official Go 1.27 patch for
|
||||
development/release validation and release builds. Align CI, runbook setup,
|
||||
and incorporated Go attribution with that selection.
|
||||
- Update ConfigKit source/license links to the reviewed commit and YAML v4
|
||||
attribution to the selected stable release. Check the actual resulting module
|
||||
graph. Maintainer tool dependencies are not automatically Narratio runtime
|
||||
components. Narratio's separate direct YAML use need not be rewritten merely
|
||||
because ConfigKit changed its internal parser dependency.
|
||||
|
||||
### Integration and simplification
|
||||
|
||||
- Apply the v1 API replacements actually used here, including
|
||||
`LoadSecretsIntoEnvironment`, `LayerMain`, `RecordDerivedDefault`, and
|
||||
`RecordNormalization`. Review other public names against the migration guide.
|
||||
- Use keyed expanded options and definition literals. Keep metadata projections
|
||||
compatible with current source evidence, native layer names, and safe errors.
|
||||
- Replace broad list-record workarounds with supported field-level recording
|
||||
when the application changes only fields. Record whole-list changes where
|
||||
order or membership changes warrant them.
|
||||
- Adopt native discovery, destination-based artifact authoring, authored
|
||||
collection operations, and native final-domain rendering/comparison. Remove
|
||||
replaced search, comparison, and redundant metadata machinery.
|
||||
- Follow the decisions below for resource bases and secondary-document loading.
|
||||
Share definitions and loading context between execution and inspection so an
|
||||
operator validates the configuration the pipeline will actually consume.
|
||||
- If secondary-document imports are adopted, review input materialization and
|
||||
publication as well as parsing. A copied root file can retain imports whose
|
||||
companions are absent at its new location. Materialize the effective domain
|
||||
document where a subprocess or restored session consumes it, retaining native
|
||||
authoring evidence separately. Review placeholder checks against the effective
|
||||
session input and preserve required external party/session formats.
|
||||
|
||||
### Documentation and maintained examples
|
||||
|
||||
Update the configuration reference, CLI reference, operations/bootstrap guidance,
|
||||
configuration internals, contributor prerequisites, and release procedure for
|
||||
the implemented semantics. Keep canonical facts in their respective owners.
|
||||
Update project-specific architecture wording where selected path/bootstrap
|
||||
decisions change it; preserve ADR 0003's ownership boundary.
|
||||
|
||||
Migrate the production/testing bundle to `config:`. Demonstrate destination-based
|
||||
artifact and family files, a native directory import restricted to YAML, an
|
||||
essential required import, native inheritance, and a useful selected-overlay
|
||||
list customization. Keep illustrative optional choices labeled as choices.
|
||||
Show how the declared artifact key is referenced by dependent artifacts.
|
||||
|
||||
Provide concise manual upgrade instructions for existing Narratio configurations:
|
||||
control-key/API changes, optional import defaults, collection behavior, source
|
||||
discovery, and selected path/bootstrap changes. Separately installed applications
|
||||
own their configuration versions; Narratio's migration does not automatically
|
||||
rewrite their files.
|
||||
|
||||
The eventual Narratio release requires a major version because existing valid
|
||||
configuration control blocks change. The Go minimum increase alone is permitted
|
||||
in a minor release under the adopted release policy. This roadmap does not
|
||||
authorize tagging, publication, or retrospective release-note changes.
|
||||
|
||||
### Verification
|
||||
|
||||
Use temporary inputs and controlled synthetic credentials. Exercise native
|
||||
discovery and the chosen bootstrap convention together, explicit file selection,
|
||||
selected-source errors, optional and required imports, import destinations,
|
||||
inherited artifact identities, native list customization, and final domain
|
||||
validation. Add coverage at Narratio boundaries rather than duplicating
|
||||
ConfigKit's complete structural test matrix.
|
||||
|
||||
Verify inherited presence/defaulting, party order, final campaign/family
|
||||
expansion, resource paths, source attribution, and secret-safe inspection.
|
||||
Diff/show must include derived members and publish rules; compare equal effective
|
||||
configurations across different source arrangements. Credential rotation behind
|
||||
the same reference name must remain absent from semantic diffs and digests.
|
||||
|
||||
Update tests that encode intentionally retired syntax or policies. Run focused
|
||||
configuration/command/example/documentation checks and the established full
|
||||
test, race, vet, build, and module checks on the adopted Go line. Preserve the
|
||||
current supported build targets. No new live-service or CI-completion gate is
|
||||
introduced by this migration.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
### 1. Default pipeline discovery conventions
|
||||
|
||||
**Recommendation:** use the native `WorkingDirectory`, `UserConfigDirectory`,
|
||||
and `UnixSystemDirectories` presets in that order with application name
|
||||
`narratio`. Use their documented filenames and retain `--config` for exclusive
|
||||
selection. This makes development-local configuration convenient and matches
|
||||
ConfigKit's standard naming. Existing `pipeline.yml` files remain explicitly
|
||||
selectable; migration guidance explains the new discovery names and precedence.
|
||||
|
||||
**Alternative:** use ConfigKit `SearchPaths` for the current two system
|
||||
`pipeline.yml` paths, or add those paths to a broader ordered native search.
|
||||
This reduces deployment disruption while still removing Narratio's probing
|
||||
loop, but preserves project-specific naming and gives local development less
|
||||
convenience. Adding fallback paths also creates more precedence to explain.
|
||||
|
||||
### 2. Secrets bootstrap before native discovery
|
||||
|
||||
**Recommendation:** use an invocation-owned conventional directory
|
||||
`/usr/local/etc/narratio/secrets`, with `--secrets-dir` selecting another
|
||||
directory and existing environment variables taking priority. Missing
|
||||
conventional secrets remain optional; an explicitly selected directory follows
|
||||
the existing explicit-selection behavior. This separates bootstrap from main
|
||||
file discovery and lets ConfigKit perform the first and only configuration
|
||||
selection. Service environments can provide credentials directly; development
|
||||
invocations can choose a project-local secrets directory explicitly.
|
||||
|
||||
**Alternative:** preserve implicit `secrets/` beside the selected main file.
|
||||
This is convenient for portable configuration bundles, but ConfigKit's discovery
|
||||
source selects its file during loading. Preserving pre-load bootstrap would
|
||||
require a coordinated upstream discovery/bootstrap facility or retaining
|
||||
Narratio-owned preselection. The latter reduces native consolidation; moving
|
||||
environment mutation into a hidden resolver would change the existing startup
|
||||
boundary. An upstream request is viable if sibling-directory convenience is
|
||||
more important than an immediately self-contained migration.
|
||||
|
||||
### 3. Relative resource paths in imported definitions
|
||||
|
||||
**Recommendation:** resolve relative file-resource references from their
|
||||
effective defining source's ConfigKit resource context. Inherited values follow
|
||||
their effective origin. CLI-relative paths use the invocation directory;
|
||||
application-derived paths record their dependencies. Use the main source as a
|
||||
documented fallback for origins without a distinct usable resource base.
|
||||
This makes imported artifact/integration definitions portable across bundles.
|
||||
Review actual path-bearing fields explicitly; session-relative products and
|
||||
remote keys retain their domain bases. Introduce no new target-existence or
|
||||
root-containment checks merely to adopt this policy.
|
||||
|
||||
**Alternative:** retain current per-field pipeline-root bases while using native
|
||||
metadata elsewhere. This minimizes path edits and remains a valid ConfigKit
|
||||
consumer policy, but moving reusable definitions still requires knowing the
|
||||
destination bundle's root. Absolute resource paths are another viable operator
|
||||
choice, with clear meaning but reduced portability.
|
||||
|
||||
### 4. Imports in campaign, party, and session documents
|
||||
|
||||
**Recommendation:** enable native explicit imports for locally sourced campaign,
|
||||
party, and session documents as well as pipelines. This supports split rosters
|
||||
and shared metadata with consistent library semantics. Keep campaign selection,
|
||||
party schema validation, and session domain rules application-owned. Remote
|
||||
session snapshots retain their explicit source context: remote companion-file
|
||||
retrieval needs its own supported input contract, rather than resolving relative
|
||||
imports against an incidental download directory. Pipeline overlay selection
|
||||
remains associated with the pipeline; it does not implicitly select overlays
|
||||
for every document. Adoption includes effective-document materialization where
|
||||
raw root-file copying would leave unresolved imports in downstream inputs or
|
||||
published session data.
|
||||
|
||||
**Alternative:** enable the richer facilities only for pipelines and retain
|
||||
single-source secondary documents. This bounds the migration and simplifies
|
||||
remote/local parity, but leaves an intentional project-specific difference in
|
||||
authoring ergonomics. If selected, document that scope explicitly rather than
|
||||
presenting it as a ConfigKit limitation.
|
||||
Reference in New Issue
Block a user