diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md new file mode 100644 index 0000000..475958a --- /dev/null +++ b/docs/roadmap/configkit.md @@ -0,0 +1,359 @@ +# Adopt ConfigKit v1.2.1 and native configuration ergonomics + +## Purpose + +Adopt current ConfigKit as Narratio's configuration owner and make its native +authoring and inspection features useful to operators. Align configuration +semantics with other ConfigKit consumers. Existing Narratio assembly conventions +are migration inputs rather than compatibility requirements for this work. + +This roadmap defines the target and required changes. A subsequent implementation +plan will define staged work after the open decisions are resolved. This pass +adds documentation only. The earlier [v0.5.0 roadmap](config.md) and +[implementation plan](implementation.md) describe the completed baseline. + +The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md) +continues to apply: ConfigKit resolves configuration before its final handoff; +Narratio owns domain validation, application derivation, and runtime use. +Supported callbacks and native metadata provide the integration boundary. + +## Reviewed baseline and releases + +Narratio currently selects `gitea.maximumdirect.net/eric/ConfigKit v0.5.0`, +commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd`, with Go 1.26.0 as its +consumer minimum. Release builds currently select Go 1.26.7. + +The adoption target is `go.maximumdirect.net/configkit v1.2.1`, commit +`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The published tag and the sibling +checkout agree on that revision. ConfigKit requires Go 1.27.0 and uses +`go.yaml.in/yaml/v4 v4.0.0`. Its code license remains LGPL-3.0-or-later. + +The pinned [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md) +and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md) +own upstream semantics. The intervening releases supply the following relevant +changes: + +| Release | Adoption relevance | +| --- | --- | +| v0.6.0 | Loading and metadata optimizations; Go minimum moves to 1.26.0. | +| v0.7.0 | Native safe effective-configuration `Diff`. | +| v0.7.1 | Maintainer-tool update; runtime behavior unchanged. | +| v0.8.0 | Input/import destinations, sequence-item placement, shared source-version checks, and initial identified-sequence composition. | +| v0.9.0 | Directory and expanded import selection; optional-by-default imports; identified-list replace/append/upsert and inheritance; CLI assignments move after decoding and inheritance. Its final collection APIs replace the initial v0.8 registrations. | +| v0.10.0 | Native ordered configuration discovery through `SearchConfig`. | +| v0.10.1 | Loading consolidation and import bookkeeping improvements. | +| v0.10.2 | Correctness fixes for decoding, callback/resource lifetime, discovery, inspection budgets, and collection operations. | +| v1.0.0 | Vanity module path, source-root `config` controls, public API cleanup, source lookup, provenance and sensitivity fixes, and Go 1.27.0 minimum. | +| v1.1.0 | Field-level list resolution records and identity-aware inspection across list movement. | +| v1.2.0 | Main-relative companion inputs and overlay context for layered loading; overlay filesystem fixes. | +| v1.2.1 | Stable go-yaml v4.0.0 dependency; maintainer release-tool update. | + +The corresponding `docs/releases/.md` files at the pinned revision +provide the migration details. Select v1.2.1 directly; intermediate releases +provide context rather than required deployment stops. + +## Current Narratio integration + +The [configuration reference](../config.md) and +[configuration internals](../internal/configuration.md) describe current behavior. + +- Commands select a pipeline path through Narratio's search loop, bootstrap + credentials from an explicit directory or a sibling `secrets/` directory, + and then call ConfigKit. +- Pipeline loading enables explicit path/glob imports and one selected overlay. + Campaign, session, and canonical-party documents use separate loads. Session + loading also supports command-scoped downloaded input snapshots. +- Typed defaults, recorded normalization, domain validation, and native + provenance are integrated. Additional path/dependency resolution and + campaign-owned family expansion occur after the pipeline load. +- Artifact files currently repeat enclosing `scriptorium.artifacts` mappings. + Logical IDs come from map keys. Families expand against canonical party data. +- Show, diff, and digest use a safe finalized-domain projection. Narratio owns + the current mapping-diff algorithm. Native load metadata supplies source and + artifact-catalog evidence. + +Some public import descriptions still state restrictions from older ConfigKit +behavior. Replace these with current native semantics during adoption rather +than recreating them as Narratio checks. + +## Targeted end state + +### Native assembly and operator choice + +ConfigKit owns source discovery, acquisition, composition, inheritance, presence, +typed secrets, provenance, and safe native inspection. Narratio supplies its +schema, domain defaults and validation, supported application resolution, and +operator-selected inputs. + +Use ordinary `Load` for an ordinary configuration. Use public native facilities +where they serve the operator workflow. Layered loading is appropriate when +actual lower-priority inputs are needed; typed defaults remain appropriate for +simple defaults. Main-relative companion inputs allow such underlays to follow +a discovered main without Narratio searching for it again. + +Expose the library's authored `config.imports` and `config.sequences` semantics +where applicable. Imports use ConfigKit defaults, including optional absence, +unless the operator explicitly requests other supported behavior. An invalid +selected source still fails. Operators mark essential imports `required: true`. +Directory imports can select YAML with an explicit `pattern`; unrestricted +filenames remain the library default. + +Native destinations, nested imports, discovery filters, source-local relative +import paths, collection operations, and inheritance work under ConfigKit's +rules. Narratio validates the final effective domain entries. It adds no +fragment-count rule, mandatory fragment version, import-mode allowlist, or +parallel composition engine. + +### Ergonomic artifact files and list customization + +Support an artifact definition without repeated enclosing mappings through an +explicit import destination. For example: + +```yaml +config: + imports: + - path: artifacts/session-recap.yml + destination: [scriptorium, artifacts, session_recap] + required: true +``` + +The imported file contains the artifact fields such as `prompt_id`, `inputs`, +and `output_path`. The literal destination defines the reusable key +`session_recap`, with logical source `narratio.artifact.session_recap`. +The filename is an organizational choice. Keyed files mounted at +`[scriptorium, artifacts]` support declared directory or glob imports when many +definitions should be discovered together. Apply the same pattern to families. + +Keep keyed artifact/family mappings and campaign-owned character data. They +already fit native inheritance and stable identity. Full-root and disjoint +definitions remain valid native inputs; a one-artifact-per-file organization is +guidance rather than a validity requirement. + +Allow authored native collection declarations for existing list fields where +their final shapes satisfy Narratio's schema. Operators can choose replacement, +append, or upsert and identified-record inheritance. Explain that upsert +replaces whole matching records and that an empty append/upsert contribution +does not clear a list. Register application-owned sequence identities only +where Narratio has a real domain identity requirement; generic library +capabilities do not establish new uniqueness requirements for every list. + +### Discovery, credentials, and resource paths + +Replace Narratio-owned local candidate probing with ConfigKit search descriptions +and `SearchConfig`. Explicit `--config` selection remains an exclusive `File` +input. Report the source actually selected by the library. The open decisions +below select search conventions and the independent bootstrap directory needed +before a discovered load resolves credentials. + +Bootstrap uses `LoadSecretsIntoEnvironment` once per invocation before typed +credential resolution and concurrent work. Existing environment variables win, +including explicitly empty variables. Keep explicit `--secrets-dir`, safe +partial-report handling, and exact-name forwarding to subprocesses. The selected +bootstrap convention must allow native discovery without a second preliminary +configuration load or a duplicate file-selection loop. + +Use native source/resource context for application path interpretation. The +resource-base decision below applies to file references rather than artifact +destinations, command names, or remote object keys. ConfigKit's consumer examples +illustrate application policy; they do not impose it on Narratio. + +### Final-domain inspection through native APIs + +Keep validate, show, sources, artifact catalog, and diff useful for the operator's +fully resolved pipeline, including campaign-driven members and publish rules. +Use native effective rendering and `Diff` where they own the operation, with +Narratio formatting and generated-domain context at the presentation boundary. + +A raw pipeline `Result` is a frozen load snapshot. Mutating its `Config` later +does not update rendering or diff. Integrate this distinction explicitly: +record domain resolution within a supported resolver when its inputs are +available there; where later campaign resolution remains appropriate, create +a completed native inspection result from the finalized safe domain projection +using the public mapping-source API. That result describes application-derived +values; retained authoring results supply actual file provenance. + +Use native comparison rather than maintaining a second general mapping-diff +engine. Preserve final-domain coverage and credential identities. Narratio's +digest and stage-owned reuse evidence remain application responsibilities. +Source-layout changes alone do not change equivalent domain values. Source +inspection can report composite origins, empty optional-import selections, and +application derivation without inventing a unique defining-file contract. + +### Scope and compatibility + +This is a clean native cutover. Update source-root `configkit:` controls to +`config:` and remove old-module/API references in maintained code and fixtures. +Nested `config` data remains ordinary data under ConfigKit's source-root rules. +The `configkit` package name and `configkit:"secret"` Go tag retain their +upstream spelling. Mapping and identified-list `extends` follow current native +rules. + +This roadmap deliberately permits revised configuration semantics. Update +tests and documentation for selected changes rather than preserving historical +acceptance rules through compatibility loaders or keyword aliases. + +Application identity, artifact dependency meaning, runtime output destinations, +campaign-owned party authority, remote session retrieval, and stage reuse remain +Narratio domain concerns. This scope does not redesign the runtime workspace, +change subprocess adapters, adopt ReleaseKit for Narratio, or add external +provider probes. Audita and workspace work retain their separate roadmaps. + +## Required changes + +### Dependency, toolchain, and notices + +- Select the released `go.maximumdirect.net/configkit v1.2.1` module and migrate + all imports and exchanged named types together. Remove the old requirement; + use the released dependency rather than a filesystem replacement. +- Raise `go.mod` to Go 1.27.0 and select an available official Go 1.27 patch for + development/release validation and release builds. Align CI, runbook setup, + and incorporated Go attribution with that selection. +- Update ConfigKit source/license links to the reviewed commit and YAML v4 + attribution to the selected stable release. Check the actual resulting module + graph. Maintainer tool dependencies are not automatically Narratio runtime + components. Narratio's separate direct YAML use need not be rewritten merely + because ConfigKit changed its internal parser dependency. + +### Integration and simplification + +- Apply the v1 API replacements actually used here, including + `LoadSecretsIntoEnvironment`, `LayerMain`, `RecordDerivedDefault`, and + `RecordNormalization`. Review other public names against the migration guide. +- Use keyed expanded options and definition literals. Keep metadata projections + compatible with current source evidence, native layer names, and safe errors. +- Replace broad list-record workarounds with supported field-level recording + when the application changes only fields. Record whole-list changes where + order or membership changes warrant them. +- Adopt native discovery, destination-based artifact authoring, authored + collection operations, and native final-domain rendering/comparison. Remove + replaced search, comparison, and redundant metadata machinery. +- Follow the decisions below for resource bases and secondary-document loading. + Share definitions and loading context between execution and inspection so an + operator validates the configuration the pipeline will actually consume. +- If secondary-document imports are adopted, review input materialization and + publication as well as parsing. A copied root file can retain imports whose + companions are absent at its new location. Materialize the effective domain + document where a subprocess or restored session consumes it, retaining native + authoring evidence separately. Review placeholder checks against the effective + session input and preserve required external party/session formats. + +### Documentation and maintained examples + +Update the configuration reference, CLI reference, operations/bootstrap guidance, +configuration internals, contributor prerequisites, and release procedure for +the implemented semantics. Keep canonical facts in their respective owners. +Update project-specific architecture wording where selected path/bootstrap +decisions change it; preserve ADR 0003's ownership boundary. + +Migrate the production/testing bundle to `config:`. Demonstrate destination-based +artifact and family files, a native directory import restricted to YAML, an +essential required import, native inheritance, and a useful selected-overlay +list customization. Keep illustrative optional choices labeled as choices. +Show how the declared artifact key is referenced by dependent artifacts. + +Provide concise manual upgrade instructions for existing Narratio configurations: +control-key/API changes, optional import defaults, collection behavior, source +discovery, and selected path/bootstrap changes. Separately installed applications +own their configuration versions; Narratio's migration does not automatically +rewrite their files. + +The eventual Narratio release requires a major version because existing valid +configuration control blocks change. The Go minimum increase alone is permitted +in a minor release under the adopted release policy. This roadmap does not +authorize tagging, publication, or retrospective release-note changes. + +### Verification + +Use temporary inputs and controlled synthetic credentials. Exercise native +discovery and the chosen bootstrap convention together, explicit file selection, +selected-source errors, optional and required imports, import destinations, +inherited artifact identities, native list customization, and final domain +validation. Add coverage at Narratio boundaries rather than duplicating +ConfigKit's complete structural test matrix. + +Verify inherited presence/defaulting, party order, final campaign/family +expansion, resource paths, source attribution, and secret-safe inspection. +Diff/show must include derived members and publish rules; compare equal effective +configurations across different source arrangements. Credential rotation behind +the same reference name must remain absent from semantic diffs and digests. + +Update tests that encode intentionally retired syntax or policies. Run focused +configuration/command/example/documentation checks and the established full +test, race, vet, build, and module checks on the adopted Go line. Preserve the +current supported build targets. No new live-service or CI-completion gate is +introduced by this migration. + +## Open Decisions + +### 1. Default pipeline discovery conventions + +**Recommendation:** use the native `WorkingDirectory`, `UserConfigDirectory`, +and `UnixSystemDirectories` presets in that order with application name +`narratio`. Use their documented filenames and retain `--config` for exclusive +selection. This makes development-local configuration convenient and matches +ConfigKit's standard naming. Existing `pipeline.yml` files remain explicitly +selectable; migration guidance explains the new discovery names and precedence. + +**Alternative:** use ConfigKit `SearchPaths` for the current two system +`pipeline.yml` paths, or add those paths to a broader ordered native search. +This reduces deployment disruption while still removing Narratio's probing +loop, but preserves project-specific naming and gives local development less +convenience. Adding fallback paths also creates more precedence to explain. + +### 2. Secrets bootstrap before native discovery + +**Recommendation:** use an invocation-owned conventional directory +`/usr/local/etc/narratio/secrets`, with `--secrets-dir` selecting another +directory and existing environment variables taking priority. Missing +conventional secrets remain optional; an explicitly selected directory follows +the existing explicit-selection behavior. This separates bootstrap from main +file discovery and lets ConfigKit perform the first and only configuration +selection. Service environments can provide credentials directly; development +invocations can choose a project-local secrets directory explicitly. + +**Alternative:** preserve implicit `secrets/` beside the selected main file. +This is convenient for portable configuration bundles, but ConfigKit's discovery +source selects its file during loading. Preserving pre-load bootstrap would +require a coordinated upstream discovery/bootstrap facility or retaining +Narratio-owned preselection. The latter reduces native consolidation; moving +environment mutation into a hidden resolver would change the existing startup +boundary. An upstream request is viable if sibling-directory convenience is +more important than an immediately self-contained migration. + +### 3. Relative resource paths in imported definitions + +**Recommendation:** resolve relative file-resource references from their +effective defining source's ConfigKit resource context. Inherited values follow +their effective origin. CLI-relative paths use the invocation directory; +application-derived paths record their dependencies. Use the main source as a +documented fallback for origins without a distinct usable resource base. +This makes imported artifact/integration definitions portable across bundles. +Review actual path-bearing fields explicitly; session-relative products and +remote keys retain their domain bases. Introduce no new target-existence or +root-containment checks merely to adopt this policy. + +**Alternative:** retain current per-field pipeline-root bases while using native +metadata elsewhere. This minimizes path edits and remains a valid ConfigKit +consumer policy, but moving reusable definitions still requires knowing the +destination bundle's root. Absolute resource paths are another viable operator +choice, with clear meaning but reduced portability. + +### 4. Imports in campaign, party, and session documents + +**Recommendation:** enable native explicit imports for locally sourced campaign, +party, and session documents as well as pipelines. This supports split rosters +and shared metadata with consistent library semantics. Keep campaign selection, +party schema validation, and session domain rules application-owned. Remote +session snapshots retain their explicit source context: remote companion-file +retrieval needs its own supported input contract, rather than resolving relative +imports against an incidental download directory. Pipeline overlay selection +remains associated with the pipeline; it does not implicitly select overlays +for every document. Adoption includes effective-document materialization where +raw root-file copying would leave unresolved imports in downstream inputs or +published session data. + +**Alternative:** enable the richer facilities only for pipelines and retain +single-source secondary documents. This bounds the migration and simplifies +remote/local parity, but leaves an intentional project-specific difference in +authoring ergonomics. If selected, document that scope explicitly rather than +presenting it as a ConfigKit limitation.