From 65d0e518e58b8f8930865e6da5a8236892adabf7 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 17 Nov 2026 20:41:34 +0000 Subject: [PATCH] Plan the native ConfigKit migration --- docs/roadmap/configkit.md | 10 +- docs/roadmap/implementation.md | 690 ++++++++++++++++++--------------- 2 files changed, 383 insertions(+), 317 deletions(-) diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index bdfb333..59e9093 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -7,10 +7,10 @@ authoring and inspection features useful to operators. Align configuration semantics with other ConfigKit consumers. Existing Narratio assembly conventions are migration inputs rather than compatibility requirements for this work. -This roadmap defines the target and required changes. A subsequent implementation -plan will define staged work for the agreed target. This pass adds documentation -only. The earlier [v0.5.0 roadmap](config.md) and -[implementation plan](implementation.md) describe the completed baseline. +This roadmap defines the target, user intent, and required changes. The +[implementation plan](implementation.md) defines the work needed to reach that +target. The earlier [v0.5.0 roadmap](config.md) records the completed adoption +that forms the current baseline. The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md) continues to apply: ConfigKit resolves configuration before its final handoff; @@ -200,7 +200,7 @@ does not update rendering or diff. Integrate this distinction explicitly: record domain resolution within a supported resolver when its inputs are available there; where later campaign resolution remains appropriate, create a completed native inspection result from the finalized safe domain projection -using the public mapping-source API. That result describes application-derived +using ConfigKit's public loading APIs. That result describes application-derived values; retained authoring results supply actual file provenance. Use native comparison rather than maintaining a second general mapping-diff diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index cc19ec4..f8b49d6 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,365 +1,431 @@ -# ConfigKit v0.5.0 implementation plan +# ConfigKit v1.2.1 implementation plan ## Scope and authority -Implement [the ConfigKit roadmap](config.md). It owns the target, compatibility -choices, and rationale. This plan covers Narratio's upgrade from v0.4.2 to -v0.5.0 and integration with native sibling inheritance. Audita adoption and -workspace reorganization remain subsequent, separately planned work. +Implement [the approved ConfigKit roadmap](configkit.md). It owns the target, +native-semantics intent, and four approved product decisions. This plan replaces +the completed v0.5.0 plan. Its audience is a Luna coding agent implementing one +end-to-end sprint in the numbered order below. -ConfigKit owns source assembly and native provenance. Narratio validates the -effective domain configuration and exposes the library's metadata to operators. -This revision replaces the earlier plan to repair declaration-history counting -with removal of that additional acceptance rule. It retains supported callbacks, -presence-sensitive defaults, party ordering, and provenance inspection. - -The reviewed upstream revision is -`dc09959cca4e105f49ec316c6d4ba585bd2340bd`, tagged `v0.5.0`. -Use the released module, not a local replacement. The pinned -[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/config.md) -and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/consumers/README.md) -own upstream semantics. Read their sibling-inheritance and metadata sections -when working on the relevant stages; source may also be inspected in the -selected Go module cache. The sibling checkout is optional reference material, -not a build or test dependency. +Select `go.maximumdirect.net/configkit v1.2.1`, upstream commit +`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The pinned +[consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md) +and [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md) +own upstream semantics. Read the sections identified by each stage and public +declarations in the released module as needed. The sibling checkout is optional +reference material, not a build or test prerequisite. ## Working instructions -The audience is a Luna coding agent. Complete the numbered stages in order. -Each stage is a bounded unit with its own verification and completion criteria. -Inspect current code before editing; the paths below identify owners rather -than requiring changes to every listed file. +- Start with `docs/development.md` and `docs/policy/agents.md`. Architecture, + testing, documentation, OSS, and release policies apply; reuse unchanged + policy text already read. Prefer graph tools for code discovery. +- Implement the full native configuration workflow. Discovery, destinations, + collection controls, resource bases, secondary imports, materialization, and + native inspection are part of this sprint. +- Stages are work boundaries, not deployment releases. Intermediate stages may + fail to compile or have dependent tests awaiting later work. Record that state + accurately. Avoid temporary adapters or extra abstractions merely to make + every intermediate commit green. Final validation covers the assembled system. +- Commit completed nonempty units with short plain-English messages. Add focused + behavioral tests with their owner; run meaningful available checks. The final + stage runs repository-wide validation against the finished source. +- Use ConfigKit's public loading, presence, provenance, recording, rendering, + and comparison APIs. Narratio supplies domain meaning and runtime use. + Retire obsolete source-authoring checks and tests rather than reproducing them. +- Leave import defaults unspecified so ConfigKit supplies its defaults. + Explicit false/zero/empty/null and collection semantics follow the native + contract and actual Narratio domain rules. +- Use temporary inputs, real ConfigKit, and controlled synthetic credentials. + The default tests remain offline and independent of installed subprocesses. +- Write documentation in concise, affirmative, factual language. The roadmap + owns intent; this file owns sequencing and implementation choices. -- Start with `docs/development.md`, `docs/policy/agents.md`, and `config.md` in - this directory. Follow the development policy-reading guide. Architecture, - testing, documentation, and OSS policies apply to this work; release policy - applies if discussing compatibility or eventual release preparation. -- Use Go 1.26. Prefer the repository's graph tools for code discovery and - inspect focused tests at each package boundary. -- Keep ordinary `Load`, typed defaults, explicit imports, one selected overlay, - domain validation, and current path-resolution rules. Native inheritance is - available wherever ConfigKit and Narratio's schema permit it, with no new - collection allowlist, opt-in flag, or Narratio-owned inheritance engine. -- Trust ConfigKit's structural acceptance rules. Remove Narratio's - one-base-file-per-entry check rather than making it inheritance-aware. Validate - resulting domain values; source history is diagnostic evidence, not another - acceptance gate. Parent entries that survive composition remain ordinary - entries subject to applicable domain rules. -- Use native presence and provenance APIs for defaults and operator inspection. - Keep supported callbacks and party ordering. Do not impose post-`Load` callback - relocation, relaxed empty/null semantics, or retirement of `config sources`. -- Keep invocation-owned secrets bootstrap and shared exact-name credential - forwarding unchanged. Do not introduce a second base load, file reparse, or - historical-secret lookup to reconstruct provenance or validate old input. -- Tests use temporary files, controlled synthetic secrets, and real internal - collaborators. No downstream executable, sibling checkout, cloud service, - or paid provider is needed. Protect integration behavior rather than - duplicating ConfigKit's generic algorithm tests or freezing incidental output. -- Commit each completed nonempty unit with a short plain-English message after - suitable validation. Report blockers and incomplete work accurately. If a - review-only task needs no source fix, retain useful behavioral coverage and - report that result rather than making a cosmetic rewrite. -- Update affected canonical documentation with each implemented behavior change. - In particular, Stage 2 updates the removed authoring restrictions and Stage 3 - updates changed inspection meanings. Stage 6 consolidates the policy review, - inheritance guidance, and example; it is not a reason to leave earlier - completed commits describing obsolete behavior. -- For documentation updates, state the policy affirmatively using concise, - factual language. Avoid negative parallelism. +Preserve stage ordering, artifact dependency meaning, the canonical party schema, +exact-name credential forwarding, publish commit semantics, and the existing +Linux amd64/arm64 plus best-effort macOS arm64 targets. This sprint does not +implement pending Audita/workspace features, migrate old sessions, rewrite +downstream configurations, add fragment version mandates or new path checks, +adopt ReleaseKit for Narratio, or publish a release. -Do not adopt layered loading, new source types, source-relative rebasing, -abstract artifacts, automatic YAML conversion, new execution selection rules, -new output collision policy, or changes to subprocess credentials and workspace -paths in this feature. Existing domain validation and reuse rules still apply; -do not add prohibitions, hardening, or guarantees merely because they can be -tested. This plan does not authorize a release, push, tag, ADR acceptance, or -retirement of pending roadmaps. +## Concrete integration choices -## Stage 1: Select the released dependency and update attribution [x] +- Consumer Go minimum: **1.27.0**. Development/release validation and build + toolchain: official **Go 1.27.4**, an available patch on that line. Selecting + it is not a new publication gate. +- Pipeline source when no explicit file is supplied: + `SearchConfig("narratio", Searches(WorkingDirectory(), UserConfigDirectory(), + UnixSystemDirectories()), SearchOptions{Required: true})`. + An explicit `--config` uses `File` exclusively. +- Bootstrap: optional `/usr/local/etc/narratio/secrets`, or explicit + `--secrets-dir`, through `LoadSecretsIntoEnvironment` before typed loading. + Existing environment values win. The implicit sibling directory retires. +- Local campaign/session/party loading enables imports. Only the pipeline gets + its selected overlay. Party version checking remains main-only; fragments + need no new version header. Remote session bytes are contextless snapshots. +- Existing campaign registry selection and local session search locations remain + domain conventions. Use native optional `SearchConfig` with `SearchPaths` + for local session candidates. Only actual absence permits remote fallback. +- Final-domain inspection uses a safe DTO and a defaults-backed + `Load(NoSource(), Definition.Defaults)` snapshot after domain resolution. + Credential fields contain reference names, not typed secrets that would + resolve again. Original results supply actual file provenance. Stages 8–9 + explain this boundary. -Purpose: establish the actual dependency baseline before integration work. +## Stage 1: Establish the dependency and toolchain baseline -Read `go.mod`, `go.sum`, `docs/oss/attribution.md`, the OSS policy, and the -selected release's notes, `COPYRIGHT`, and `LICENSE`. +Read `go.mod`, `go.sum`, `.woodpecker/*.yml`, release scripts/runbook, and +attribution. Read the upstream v1.0.0 migration table and v1.2.1 release note. -1. Select `gitea.maximumdirect.net/eric/ConfigKit@v0.5.0` with the normal Go - module workflow. Keep Go 1.26 and avoid unrelated dependency upgrades. -2. Update ConfigKit's attribution entry to v0.5.0 and use the full reviewed - commit for its Source, COPYRIGHT, and LICENSE links. Preserve upstream - copyright wording, SPDX expression, and applicable local license links. - Its license and YAML dependency are unchanged; no new license copy is needed. -3. Inspect module changes for unintended churn or a local `replace`. Existing - configuration loaders should compile without adopting optional APIs. +1. Select Go 1.27.0 in `go.mod` and ConfigKit v1.2.1 under its vanity path. + Migrate all production/test imports together and remove the old requirement. + Use the released module rather than a filesystem replacement. +2. Replace APIs actually used here: `LoadSecretsDir` → + `LoadSecretsIntoEnvironment`, `LayerBase` → `LayerMain`, and generic + `Resolution.Record` calls with `RecordDerivedDefault` or + `RecordNormalization`. Audit other removed public names and compared layer + or diagnostic strings. Use keyed expanded options/definition literals. +3. Rename genuine source-root `configkit:` controls to `config:` throughout + fixtures and examples. Keep the package name, `configkit:"secret"` Go tag, + and ordinary data keys distinct from source controls. +4. Align Go CI images and local/release instructions with Go 1.27.4. Preserve + platform targets and publication mechanics. Update incorporated Go notices. +5. Refresh ConfigKit and stable go-yaml v4.0.0 attribution from selected source + notices. Review actual module graph churn and tool-only dependencies. Keep + Narratio's separate YAML v3 use unless another scoped change needs it. -Verify with `go test ./internal/config ./internal/app ./internal/doccheck` -and `go mod tidy -diff`. Confirm -`go list -m gitea.maximumdirect.net/eric/ConfigKit` selects v0.5.0 without a -replacement. +Check module identity and attribution. Attempt focused config/app tests; record +failures assigned to later stages. Complete when the dependency/API baseline is +established, without preserving old behavioral expectations as compatibility. -Complete when the released dependency and notices agree and current focused -tests pass. The cross-file inheritance defect is addressed in the next stage; -this stage does not claim that new inheritance behavior is fully integrated. +## Stage 2: Integrate native pipeline discovery and independent bootstrap -## Stage 2: Remove source-authoring acceptance gates [x] +Read `internal/app/pipeline_config_path.go`, `secrets_bootstrap.go`, +`command_dependencies.go`, `loadPipelineConfigWithDependencies` and callers, +inspection loaders, and `internal/config/pipeline_configkit.go`. +Read upstream discovery and secrets-bootstrap contracts. -Purpose: accept structurally composed configuration from ConfigKit and reject -only invalid effective Narratio domain configuration. +1. Have the configuration owner accept an explicit file or the native required + discovery source above. Retain explicit-file package entry points where useful. + Commands load through one composition path instead of first probing with + `os.Stat` to obtain a default pathname. +2. Bootstrap once before typed loading or adapter creation from the independent + directory. Preserve explicit-path error behavior, safe partial reports, and + existing-empty environment precedence. +3. Obtain the actual selected root and import/overlay evidence from ConfigKit. + Carry that root into campaign selection, provenance, and command summaries. + Remove dependencies/search helpers used only for old pipeline preselection. +4. Apply this composition to run/range/alias commands, session init/plan/status/ + restore, inspection, and remote-session workflows. A selected invalid source + fails rather than falling through to another pipeline candidate. +5. For overlay diff, load the left through native selection and the right from + that selected root with its own overlay. Share invocation bootstrap; neither + side performs a preliminary base-only load. -Read `internal/config/pipeline_configkit.go`, particularly -`pipelineMetadataFromConfigKit` and `rejectDuplicateBaseDefinitions`, -`pipeline_configkit_test.go`, and `expandFamilyPublishRules` and its tests in -`internal/config/artifact_families.go` and `artifact_families_test.go`. +Test precedence and exclusive explicit selection with controlled working/user +directories and temporary native search descriptions. Test selected invalid +input, no source, bootstrap override/default behavior, and diff bootstrap once. +Keep tests isolated from the operator's system configuration and secrets. -1. Delete `rejectDuplicateBaseDefinitions` and its invocation. Do not replace - it with filtering of inheritance events, direct-declaration counting, or - another source-history validator. -2. Replace `TestPipelineConfigKitRejectsDuplicateBaseDeclarations` with behavior - coverage showing that disjoint contributions to one artifact/family compose - successfully when the resulting domain entry is valid. Check selected-overlay - replacement/removal without Narratio rejecting superseded declarations. -3. Add cross-file artifact and family inheritance regressions. Supply valid - surviving parent and child entries with distinct output destinations, and - campaign/party context for family expansion. Verify child identities and - effective fields; the inheritance regression must fail on the old check. -4. Remove the `publishDeclared` gate in `expandFamilyPublishRules`, its metadata - population, and state retained only for that gate. Validate family publication - against the effective `Publish` value, whether authored or supplied by defaults. - Preserve the missing-effective-publish check, ordinary enablement behavior, - destination validation, and conflicts between effective publish rules. - Update the test that currently fabricates an explicit-declaration flag. -5. Retain effective-domain rejection, including invalid artifacts, unresolved - dependencies, and destination collisions under current rules. Reuse existing - tests where adequate. Retain a representative structural-conflict check at - the loader boundary; ConfigKit owns the full conflict/precedence matrix. +## Stage 3: Resolve pipeline resources from native origin context -Verify with `go test ./internal/config`. Assert effective outcomes and safe -error categories, not entire diagnostic text or private metadata event order. -Do not run a separate base-only load to validate overwritten input. +Read `resolvePipelineConfig`, `resolveNotariusPaths`, +`finalizeLoadedPipeline`, `effective_digest.go`, and the upstream source-aware +path example. Resolve paths before normalization erases entry origins. -Complete when valid cross-file composition/inheritance succeeds and invalid -effective configuration still fails before execution, without authored-source -acceptance gates. Keep native metadata available for the next stage. +1. Introduce one small config-owned host-resource base helper using + `Resolution.EntryInfo`, `Resolution.Source`, and + `Resolution.SourceContext`. Use the value's effective origin rather than + its child's inheritance directive. A distinct usable source context supplies + its canonical base; missing or composite context falls back to the main source. + Explicit CLI-origin file references use the captured invocation directory. + If the main input is contextless too, use that invocation directory rather + than interpreting a diagnostic label as a filesystem base. This covers remote + byte snapshots without turning their temporary filenames into resource roots. +2. Normalize nonempty relative references with recorded helpers. Absolute values + stay absolute. Derive Notarius working directory from its already resolved + config path and record that dependency. Preserve domain validation of relevant + empty/disabled values. +3. Apply this classification: -## Stage 3: Simplify provenance inspection around native APIs [x] + | Fields | Interpretation | + | --- | --- | + | Audita/Scriptorium/Notarius `config_path`; Notarius `working_directory` | File/directory resource based on effective origin. | + | Relative `workspace.root`, `campaigns.root`, `spool.root`, `cache.root` | Host directory reference based on effective origin; absolute defaults stay unchanged. | + | Campaign template/stable-input paths; session stable-input/audio paths | Same resource policy, completed in Stage 4. | + | Artifact output paths, family output patterns, publish destinations, source IDs, prompt/profile IDs, schema identifiers, binary commands, Notarius reference bindings and chunk-cache modes | Existing domain interpretation, not file-resource rebasing. | -Purpose: preserve useful operator inspection without a parallel provenance -system or a unique-defining-file contract for composite entries. +4. Remove duplicate pipeline-root rebasing and logical-Notarius path caches once + callers use resolved domain values. Show/digest describe those domain values, + rather than reconstructing an alternative authoring-path view. +5. Keep resource access at existing consumer boundaries; base selection adds no + target-existence check, root restriction, or history acceptance gate. -Read `internal/config/artifact_catalog.go`, `artifact_families.go`, -`effective_pipeline.go`, `effective_digest.go`, and the metadata projection in -`pipeline_configkit.go`. Inspect `internal/app/config_commands_test.go`, -`config_artifacts_test.go`, and ConfigKit's public metadata contracts. +Test imported, inherited, overlay, absent-default, absolute, and ambiguous-origin +cases. Equal resolved resources have equal effective semantics across layouts; +relative text resolving to a different resource is a meaningful change. -1. Keep `config sources` and provenance-bearing `config artifacts` output. - Use public ConfigKit sources, effective origins, history, and explanations - for the information they own. Keep presence queries and recorded callbacks - where needed; retaining returned metadata is not duplicating its implementation. -2. Review existing helper/state layers for duplicated tracking. Remove redundant - cached field-ownership reconstruction where the same result is available - directly from retained metadata. Keep a small projection where needed for - Narratio's generated family/character/party context and output formatting. - Do not add a provenance service, custom ancestry graph, or second source parse. -3. Exercise Stage 2's inherited and multi-file entries through - `EffectiveArtifactCatalog` and `EffectivePipelineSources`. Report actual - contributors and child overrides. Keep historical definition evidence - distinguishable from effective field origins as exposed by ConfigKit. -4. Replace `artifactFamilySource`'s arbitrary first-contributor ownership with - native source/origin evidence. Do not infer a unique base defining file or - implement a new declaration-accounting algorithm. Composite origins may have - several sources; preserve them using existing repeated source records and - comma-separated catalog source lists. Adjust the runtime-only representation - minimally where a single-source field cannot carry that evidence. -5. Preserve logical artifact IDs, family/character identities, party provenance, - inspection command names, and read-only behavior. Retain existing headers - where their meanings remain accurate; explain that family source evidence may - be composite instead of promising a unique defining file. Do not change - native-metadata-based party ordering merely to avoid library APIs. +## Stage 4: Load campaign/session imports with correct contexts -Verify with `go test ./internal/config ./internal/app`. Cover a child family -inheriting across files, an entry composed from disjoint files, and an overlay -override. Check meaningful source records, logical identities, and absence of -secret values rather than full formatted snapshots or upstream event ordering. +Read `document_configkit.go`, `load.go`, `pathresolve.go`, +`party_resolution.go`, app session discovery/remote-session composition, and +session init. Read native import and resource-context contracts. -Complete when native provenance remains useful and truthful, redundant tracking -is removed where practical, and no acceptance rule depends on contributor count -or an inferred defining file. Application derivation context remains available. +1. Use `File` with imports enabled for local campaign/session inputs. Share + definitions with byte loading and retain successful results/source contexts. + ConfigKit owns per-source and aggregate acquisition limits. +2. Replace local session candidate probing with native optional discovery over + the existing candidate paths. Permit absent input solely for remote fallback; + distinguish absence through native source evidence. Structure callbacks to + allow that discovery-only no-source result. Acquired invalid files still + receive ordinary validation and fail without remote fallback. +3. Downloaded session bytes receive no context inferred from their temporary + filename. Retain object provenance and current explicit session/previous-ID + checks. Use native contextless-import behavior; no remote import fetcher or + custom structural rejection layer is added. +4. Resolve campaign `session_template_file`, stable-input paths, and session + overrides/audio references with Stage 3's helper. Resolve each + `audio_files` element at its entry index before ordering. Keep campaign/ + session stable-input precedence and campaign ownership of party input. + Provenance identifies the effective supplying configuration source. +5. Check unresolved template tokens in effective session string values, including + imported values. Session init still reads an authored template for intentional + substitution; runtime loading stays a configuration operation. +6. Retain detached campaign and session document values after their own loading + and resource resolution. Capture the session document before campaign-inherited + fields are copied into runtime session state. Stage 6 serializes these values. -## Stage 4: Verify inherited presence, domain resolution, and path bases [x] +Test local imports, resource bases, effective placeholder rejection, identity +mismatches, imported session-init template paths, absence versus selected-source +failure, and remote snapshots independent of temporary directories. Add no +secondary overlay flags. -Purpose: prevent Narratio's application defaults or normalization from undoing -the library's inherited explicit input. +## Stage 5: Resolve imported party definitions and canonical roster output -Read `pipelineConfigDefaults`, `resolvePipelineConfig`, and -`internal/config/load.go`, including `finalizeLoadedPipeline` and -`resolveNotariusPaths`. Inspect the current presence/defaulting tests and -the relevant domain validators. +Read `party.go`, `party_resolution.go`, `artifact_families.go`, the party +integration contract, and focused party/family tests. -1. Add representative inheritance cases for explicit false, zero, empty, - and null input versus genuinely absent input. Include a pointer boolean - and an entry-level absence-based default where Narratio consumes metadata. - For example, inherited `render_debug: false` can exercise artifact presence; - inherited empty Scriptorium timeout can exercise an existing invalid value - that must not silently become its default. -2. Preserve the existing domain outcome for each chosen field. Valid explicit - values retain their value; explicit values already invalid must still fail - ordinary validation rather than be refilled. An empty mapping merges under - ConfigKit's rules and must not be described as universal container removal. -3. Exercise overlay composition before inheritance: modify a parent's field - in the selected overlay and confirm the child's inherited result uses that - effective value. A child override still wins over its parent. -4. Verify a known root-relative field, such as enabled Notarius `config_path` - or `working_directory`, when inherited from an imported declaration. - Resolve it exactly as the existing field contract does. Do not use passive - SourceContext to rebase it to the parent's or child's source directory, or - newly normalize other path fields merely to make their behavior uniform. -5. Fix any demonstrated Narratio integration defect at its existing defaulting - or domain-resolution owner. Use ConfigKit presence and recorded operations; - retain supported defaults/resolver/validator callbacks. There is no required - move to post-`Load` functions. Do not recreate structural inheritance, inspect - old input to add acceptance rules, or relax empty/null semantics to avoid - using the library's presence API. +1. Add a file-backed party entry point with imports enabled for campaign + resolution. Keep `ParseParty(bytes)` contextless for snapshots and tests. + Reuse the definition, main-only version rule, and domain validator. +2. Preserve `narratio.party.v1`, existing character/name/class/alias validation, + and campaign ownership. Native destinations allow split roster contributions; + no fragment version mandate or abstract/template record type is added. +3. Use native metadata for roster order and evidence, preserving authored order + within a source. Extend the current ordering helper using source acquisition + order, then line/column and ID as deterministic ties across sources. Prefer a + child's own declaration/inheritance location over its parent's location. + This is a presentation convention, not a new source acceptance rule. +4. Render canonical party YAML from the effective roster: schema version, keyed + characters, singular external `alias`, classes and optional levels. Preserve + meaningful roster/list order. A small typed DTO/YAML-node writer is suitable + for this Narratio-owned external format. +5. Keep players derivation and family expansion based on that same roster. + Replace output/checksum uses of `CanonicalParty.Raw` with canonical rendering; + retain raw data only where an actual diagnostic caller needs it. -Verify with `go test ./internal/config`. Keep coverage representative; generic -missing-parent, cycle, resource-context, and expansion-limit matrices belong to -ConfigKit. Parents must remain valid Narratio entries, not special templates. +Test split/inherited rosters, aliases/classes, effective duplicate-name rejection, +order, reloadable standalone canonical output, players, and family identities. +Use native evidence rather than a second source parse to reconstruct order. -Complete when inherited explicit input and absent defaults have distinct, -correct domain outcomes and existing path bases are unchanged. +## Stage 6: Materialize effective inputs and align prepare resume checks -## Stage 5: Verify safe effective projections and inherited credentials [x] +Read `internal/stage/prepare.go`, `prepare_party.go`, `prepare_resume.go`, +prepared-input descriptors, and prepare semantic evidence/tests. -Purpose: ensure inheritance changes neither semantic comparison nor the -existing secret boundary. +1. Add configuration-owned deterministic campaign/session renderers from Stage + 4's detached document values. Serialize public schemas and resolved file + references, excluding native directives and runtime/provenance state. +2. Preserve ownership: campaign-inherited stable inputs must not become explicit + session overrides. A published session that omitted `speakers_file` must + still inherit the receiving campaign's value. Explicit session overrides + remain explicit. Include actual session identity values where application + resolution supplies them, without freezing campaign defaults into the session. +3. Prepare writes these documents and Stage 5's party bytes through existing + artifact-store helpers. Keep filenames, input kinds, provenance fields, and + write/checksum behavior. Resume uses the same renderers instead of comparing + prepared effective output with a raw imported root. +4. Ensure imported effective changes reach existing stage-owned semantic/input + evidence. Equivalent domain values remain equivalent. Full pipeline digests, + source mtimes, or every authoring file are not new blanket invalidation keys. +5. Preserve ordinary stable-file copying/audio handling. Already resolved paths + must not be rebased again against the session root. Retire raw campaign/session + file requirements used solely for copying the old authoring documents. -Read `internal/config/effective_pipeline.go`, `effective_digest.go`, -`effective_projection.go`, `scriptorium_credentials_test.go`, and the effective -projection tests in `load_validate_test.go`. For diff composition, read -`internal/app/config_diff.go`. Reuse -`TestPipelineOverlayDoesNotRequireUnusedBaseCredentials` and -`TestCredentialValuesNeverEnterEffectiveProjectionOrDigest` coverage. +Test standalone prepared documents, imported changes, equivalent layouts, and +remote-origin metadata. Canonical-byte format changes can cause a normal prepare +rerun; no session migration or new manifest schema is needed for serialization. -1. Load two configurations that produce the same effective domain values: - one uses inheritance, the other writes the expanded values explicitly. - Include ordinary artifacts and representative resolved family members. - Verify equal safe effective mappings and digests and no effective diff. - Do not compare private provenance as part of semantic equality. -2. Add a representative typed credential inheritance case using shared fields - of valid Narratio sibling mappings and controlled synthetic environment - variables. Verify resolved credentials retain their exact configured names; - serialization and inspection expose those identities without values. -3. Verify overlay removal/replacement of a credential before inheritance does - not resolve the superseded reference. Reuse the existing selected-overlay - regression rather than creating a second loader or bootstrap algorithm. -4. Check that changing only synthetic secret values leaves semantic projections - and digest unchanged. Fix only demonstrated projection defects; preserve - the existing safe typed rendering boundary and semantic reuse evidence. +## Stage 7: Close publication, restore, and remote-session round trips -Verify with `go test ./internal/config ./internal/app`. Existing shared adapter -credential-forwarding tests remain the owner of subprocess forwarding; do not -rewrite adapters or duplicate those tests simply because input can be inherited. +Read publish input selection, remote-session loading, restore contracts, and +existing in-memory storage integration tests. -Complete when resolved values, not source layout or secret bytes, determine -effective comparison, and inherited credentials obey the existing safe boundary. +1. Trace the actual bytes published as remote `session.yml`. Publish the canonical + prepared session rather than the original root or command-scoped download. + Preserve immutable object/commit-pointer ordering and lock behavior. +2. Confirm campaign/party prepared products flow through existing selections + with their identities intact and without unresolved companion imports. +3. Verify a published canonical session loads through the existing contextless + path after authoring fragments and temporary files disappear. Restore uses + recorded contents and existing safety/locking. No remote import graph or + producer-host filesystem search is introduced. +4. Verify omitted session stable fields inherit the receiving campaign's values. + Keep remote object identity evidence separate from checksums of locally + materialized canonical bytes. -## Stage 6: Align policy, document inheritance, and maintain an example [x] +Use fake/in-memory storage for prepare → publish selection → remote load and +relevant restore coverage. Fix actual producers/consumers rather than adding a +parallel publication path. Existing atomic commit tests remain authoritative. -Purpose: make the adopted behavior understandable and usable without expanding -Narratio's configuration surface. +## Stage 8: Build native final-domain inspection snapshots -Read `docs/config.md`, `docs/internal/configuration.md`, `examples/README.md`, -`docs/policy/architecture.md`, the production/testing bundle, and -`TestExamplesLoadAndValidate` in -`internal/config/load_validate_test.go`. Follow the documentation policy. +Read `effective_projection.go`, `effective_digest.go`, +`effective_pipeline.go`, pipeline finalization/family expansion, and native +completion, defaults, rendering, and typed-secret contracts. -1. Revise only the project-specific Configuration section of the architecture - policy to record the implemented boundary: ConfigKit owns loading, assembly, - inheritance, presence, secrets, and provenance; Narratio owns domain meaning - and fail-fast effective validation. Permit native metadata inspection and - supported callbacks. Align the ambiguous empty-value statement with existing - explicit-presence defaults, not a new blank/null policy. Leave general - upstream agent, testing, documentation, and OSS policy wording unchanged. -2. Add native sibling inheritance guidance to the configuration reference. - Explain exact sibling lookup, overlay-before-inheritance resolution, - explicit presence, ordinary parent validation, distinct child identity, - and native contributor inspection. Remove the one-base-file restriction and - describe acceptance of ConfigKit-composed entries. Update family-publish - guidance for effective configuration rather than explicit authoring. Link to - the pinned upstream contract for detailed syntax and structural rules. -3. Explain the reserved `extends` key throughout ConfigKit inputs, including - dynamic artifact IDs and prompt-variable mappings. Operators must rename - existing ordinary-data keys with that name; there is no opt-in or automatic - conversion. Campaign, session, and party inputs also use ConfigKit. -4. Clarify the existing statement that overlays cannot inherit: overlay files - cannot inherit or stack, but the selected overlay may supply native sibling - directives within its values. Keep imports and overlay selection unchanged. -5. Update configuration internals for inheritance before typed decoding, - removal of authoring gates, thin native contributor projection, and the safe - effective boundary. Describe implemented mechanisms, linking to the public - reference for their operator-facing meaning. Update `docs/cli.md` where source - output now reports composite evidence rather than a unique defining file. -6. In `examples/production-testing/overlays/testing.yml`, add - `extends: session_recap` to the existing `testing_notes` artifact. Keep its - disabled status and distinct output override. It inherits the imported - parent's prompt/input and the parent's testing profile after overlay - application. The parent remains an ordinary artifact. -7. Update the example guide and extend existing example validation to check - those inherited fields. Preserve the bundle's current identities and - artifact count; no new example root or template convention is necessary. +1. Keep one safe schema-specific final-domain DTO containing concrete expanded + artifacts and generated publish outputs. Omit unexpanded family declarations + and bookkeeping. Audit all credentials, including S3 fields and Audita/ + Notarius/Scriptorium lists: use exact reference names, with no resolved + `Secret` fields in the inspection DTO. +2. Build a native DTO result with `Load(NoSource(), Options{Definition: + Definition{Defaults: ...}})`, supplying a fresh safe DTO and no external + credential lookup or application callback rerun. ConfigKit copies and freezes + these already resolved values. Using defaults avoids interpreting final + `config`/`extends` data as fresh directives or converting precise numbers + through a generic mapping-source reconstruction. +3. Build/refresh the snapshot at existing domain-finalization boundaries, after + applicable path/dependency/family resolution. Keep original authoring results + separately. Synthetic default origins describe this derived snapshot, not + original file provenance. No general snapshot service is needed. +4. Route effective show and `pipeline.resolved.yml` rendering through native + `WriteEffective`. Use the same DTO's deterministic JSON for Narratio's + semantic digest. Preserve exact integers, domain null/empty values, and + credential identities. Update the digest discriminator if its format changes. +5. Remove flattening/serialization machinery used only to duplicate native + show/diff. Keep narrow path enumeration and family/party context genuinely + needed by catalog/source presentation and stage-owned reuse evidence. -Verify with `go test ./internal/config -run '^TestExamplesLoadAndValidate$'` -and `go test ./internal/doccheck`. Review examples for synthetic-only content -and validate local links. Keep unimplemented Audita/workspace behavior in its -roadmaps. Do not create a release note under an invented Narratio version; -reserved-key compatibility impact remains an explicit release-preparation item. +Test final family/publish expansion, inherited/expanded equivalence, large +integer variables, null/empty values, safe rendering, credential-reference +changes versus rotation, and no credential resolution during snapshot creation. -Complete when canonical documentation and a tested copyable example describe -the implemented adoption, without promoting new restrictions to policy. +## Stage 9: Adopt native diff and truthful operator source inspection -## Stage 7: Verify the assembled inspection workflow and close the feature [x] +Read `internal/app/config_diff.go`, config command loaders/tests, +`artifact_catalog.go`, and native Diff/source/history contracts. -Purpose: establish that real command composition consumes the integrated -configuration correctly and finish the upgrade as a reviewable baseline. +1. Fully resolve each overlay against the same selected campaign/party before + comparing Stage 8 snapshots with `configkit.Diff`. Remove + `diffEffectivePipelineValues`, its general map comparison, and consistency + checks predicated on the old flattened-leaf change model. +2. Present canonical JSON Pointer paths in native order. Derive added/removed/ + changed from snapshot `Present`, not nil or redaction text. Use compact JSON + for safe values and expose `HiddenChanged` if present. Equal values produce + `no differences`. Document pointer notation and native changed-region/list + granularity instead of emulating the old engine. +3. Source/catalog evidence comes from authoring results through public source + lookup and metadata history/origins. Add clearly labeled empty optional-import + selection evidence to `config sources`, without guessed candidates or + reasons for absence. +4. Preserve artifact/family/character IDs and generated party context. Distinguish + normalization origins from retained definition evidence. Use native `main` + where an upstream layer is reported; root/import presentation roles may remain + where they express actual application context. -Read `internal/app/config_commands_test.go`, `config_artifacts_test.go`, -and existing read-only inspection workflow coverage. Reuse the preceding -package coverage rather than repeating its entire matrix at the CLI boundary. +Test changed family members and publish rules, equivalent layouts, composite +contributors, optional empty imports, atomic lists, and pointer escaping in +allowed dynamic mapping keys. Inspection remains secret-safe and read-only, +without source reparsing or provenance-based acceptance gates. -1. Add or extend a representative command-level temporary imported bundle with - inheritance. Exercise validation, effective show, source reporting, artifact - catalog, and diff through existing command entry points. Use a campaign/party - fixture for family inspection, and paired overlays with equivalent inherited - and explicitly expanded values when testing command-level diff equality. -2. Check semantic output: resolved child values, native contributor/origin - sources, logical artifact identities, and no secret values. Keep inspection - read-only with respect to session/runtime state, as existing tests establish. - Do not invoke downstream applications or introduce new inspection modes. -3. Review the combined diff against `config.md` and this plan. Confirm that - ConfigKit v0.5.0 is selected, attribution links use the reviewed revision, - authoring-history acceptance checks were removed, native inspection and - presence semantics remain, and policy reflects that boundary. Confirm no - optional loading feature, Audita/workspace change, or compatibility shim - slipped into scope. -4. Run `go test ./...`, `go test -race ./...`, `go vet ./...`, - `go build ./...`, `go mod tidy -diff`, and `git diff --check` against the - final source. These reuse existing repository checks, not new release gates. - The full suite includes example and documentation-link validation; focused - checks need not be repeated solely for this stage if unchanged. -5. Commit the completed unit and report the dependency revision, checks, - compatibility impact, and any unfinished work. Leave the pending Audita and - workspace plans and Proposed ADRs intact. Retire completed planning files - only during separately requested cleanup/release preparation, repairing links. +## Stage 10: Exercise native authoring and maintain ergonomic examples -Complete when the feature roadmap's completion criteria are evidenced by the -released dependency, integrated behavior, maintained documentation/examples, -and passing final checks. The next implementation scope is Audita; its concrete -plan is prepared separately. Release version selection and publication are -outside this implementation pass. +Read the production/testing bundle, artifact/family tests, native destinations +and collection rules, and maintained example tests. + +1. Convert artifact/family files to destination-based definitions. IDs remain + explicit destination keys or keyed payloads; filenames do not imply IDs. +2. Demonstrate a declared directory import with `pattern: "*.yml"`, essential + `required: true` input, an optional selection, inheritance, and a useful + testing-overlay list customization through `config.sequences`. +3. Cover append/upsert/replace on an existing list using an appropriate authored + identity, such as publish output `source`. Whole-record upsert and empty-list + behavior follow ConfigKit. No application identity registration is required + merely to enable authored collection controls. +4. Demonstrate a split roster and local secondary imports using the sample + campaign, with companion resource paths following source-origin policy. + Explain that full-root and disjoint definitions remain accepted too. +5. Use ordinary `Load` for this authored-import workflow. Layered loading is + useful for actual lower-priority sources, not required for defaults alone. + If such a source is genuinely needed, use `LoadLayered` and `MainRelative` + directly with focused coverage instead of another merge/search mechanism. + +Load examples through production definitions. Check dependent logical IDs, +concrete family members, testing differences, and standalone prepared inputs. +Keep examples synthetic and credential-free. + +## Stage 11: Align canonical documentation and project-specific policy + +Read documentation/OSS/release policies, affected references, and the approved +roadmap. Finish current-behavior documentation for the assembled implementation. + +1. Update config/CLI references, operations/bootstrap guidance, troubleshooting, + configuration internals, prepare internals, and party integration docs for the + new discovery, import, resource-base, materialization, and inspection behavior. +2. Replace obsolete import restrictions with an accurate native summary and + pinned upstream links. Explain optional imports, `required`, filters, + destinations, source-local root controls, list modes, and reserved-key scope. + Keep effective domain validation Narratio-owned. +3. Align project-specific architecture configuration/startup statements and the + current raw-party-copy description in its actual owner. Preserve accepted + ADR 0003 and generic adopted policies. Routine integration details do not + require new ADRs. +4. Write manual upgrade guidance for control names, native discovery filenames/ + precedence, explicit custom filenames, secrets override, relative path changes, + required imports where desired, and list operations. Explain effective prepared + documents versus authored roots; downstream migrations remain separate. +5. Finish Go prerequisites, CI/runbook toolchain alignment, and attribution from + Stage 1. Record impact for eventual major-release preparation without inventing + a release version or changing published notes. Leave pending Audita/workspace + roadmaps and Proposed ADRs intact. + +Run doccheck and maintained example validation. Each volatile contract has one +canonical owner. Retire completed roadmap files only when separately requested. + +## Stage 12: Verify the assembled migration and close the sprint + +1. Extend a command-level temporary bundle with native discovery, explicit + secrets override, imported campaign/party/session resources, destination-based + artifacts, family expansion, and a testing overlay. Reuse preceding focused + coverage instead of repeating ConfigKit's structural matrix. +2. Exercise validate/show/sources/artifacts/diff through real composition. Check + selected root, resources, generated values, optional-import evidence, safe + credentials, and absence of runtime workspace creation by inspection. Exercise + plan or prepare with fake external seams to establish execution parity. +3. Trace remaining callers of old search, rebasing, raw-copy, and diff helpers. + Remove dead code/test seams. Review intentional test expectation changes + against the roadmap and preserve existing consequential domain/safety tests. +4. With Go 1.27.4 and dependencies prepared, run `go test ./...`, + `go test -race ./...`, `go vet ./...`, `go build ./...`, + `go mod tidy -diff`, and `git diff --check`. Use the existing release asset + builder with a temporary output directory and its existing `v0.0.0-test` label + to check the three targets; this invokes no publication or new release note. +5. Inspect `go list -m` for the exact release, absence of the old module path + and filesystem replacements, and expected notices. Review every roadmap + capability and all four approved decisions; native structural behavior must + not be narrowed by new Narratio acceptance gates. +6. Commit the finished unit and report checks, compatibility changes, revision, + and any genuine unfinished work. Leave a clean workspace. Actual release + publication and the next feature require separate authorization. + +Complete when the native workflow works end to end, prepared inputs survive +loss of authoring companions, operator inspection covers final domain values, +and final validation passes. ## Open Questions -None required for this adoption. Existing decisions select native inheritance, -native metadata inspection and callbacks, effective-domain validation instead -of authoring-history gates, current path/credential/presence semantics, and -ConfigKit before Audita before workspace work. Eventual release preparation must -account for the reserved-key compatibility impact and inspection representation -changes under the release policy; no version is selected here. +None required for this sprint. The four product choices are approved. This plan +selects concrete native snapshot, source-context, materialization, and toolchain +approaches without adding compatibility shims, structural restrictions, or +stronger runtime guarantees.