narratio/docs/roadmap/implementation.md

# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics

## Status

Implementation plan for Workstream 7, **Configuration structure and semantic
diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams 1 through 6
are complete and are prerequisites for this plan. All stages below are pending
and must be implemented in numerical order.

The purpose of this workstream is to preserve Narratio's strict, deterministic,
operator-oriented configuration behavior while making the implementation
navigable by semantic domain. It also removes incidental composition metadata
and test expectations that turn private validation choreography into a
compatibility burden. This is an internal refactor with narrowly approved
diagnostic flexibility, not a configuration-language redesign.

## Authoritative Scope And Decisions

The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md).
Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and
H42 in [`audit.md`](audit.md), C01 and C06 in
[`contract-audit.md`](contract-audit.md), and the corresponding rows in
[`audit-decisions.md`](audit-decisions.md).

The following implementation decisions are fixed:

- Keep all configuration behavior in package `internal/config`. Do not create
  subpackages, a generic configuration framework, a schema framework, a
  validator registry, or a dependency-injection layer.
- Preserve the public configuration types and entry points. `Validate`,
  `ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective
  pipeline projections, and configuration-inspection commands retain their
  current responsibilities.
- Keep `validate.go` as the short public and high-level orchestration owner.
  Organize private validation into cohesive files for core pipeline/session,
  adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path
  helpers. File names are private, but those ownership boundaries are the
  intended final structure.
- Do not replace ordinary `error` returns with a new exported validation-error
  hierarchy. No machine consumer currently requires one, and introducing one
  would add a new compatibility surface. Field paths, source identities for
  composition conflicts, wrapped causes where useful, and small semantic
  phrases provide the stable diagnostic content tests should assert.
- Continue to fail fast. The workstream does not add error aggregation and does
  not promise which of several independent invalid fields is reported first.
  Traversal of maps must nevertheless be deterministic so identical input
  produces identical behavior.
- Preserve strict single-document YAML decoding, duplicate- and unknown-field
  rejection, additive imports with no duplicate ownership, one deterministic
  profile selection, explicit-value overlay behavior, centralized defaults,
  canonical party resolution, artifact-family expansion, and secret-free
  effective digests.
- Preserve the current configuration schema. Campaign-owned
  `campaign.inputs.party_file` and canonical artifact source IDs are current
  contracts, not legacy fields. Removed player/session roster fields and old
  artifact source spellings remain unsupported; add no compatibility aliases
  or normalization branches for them.
- Treat Workstream 3's compatibility purge as authoritative. During this work,
  remove any surviving production branch that actually accepts or translates
  a retired configuration representation. A rejection supplied naturally by
  strict decoding or the canonical source validator is sufficient; do not
  maintain exhaustive legacy-form tests when representative canonical-boundary
  tests already protect the rule.
- Preserve composition source information that is consumed by conflict
  diagnostics or `config sources`. Remove only demonstrably unused node/field
  metadata. In particular, source lists and semantic field paths remain;
  redundant cached declaration-order indexes and copied line/column fields may
  be removed when no consumer reads them because slice order and the YAML
  parser already supply the required behavior.
- Preserve normalization as an all-or-nothing operation for each validator.
  Notarius reference/output maps and scalar paths are installed on the config
  only after all relevant checks succeed; a validation error must not leave a
  partially normalized value behind.
- Keep `validateAudita` intact when moving it to the adapter-validation owner.
  Its guards are cohesive; splitting them further would add names without
  reducing complexity.
- Tests must protect semantic rules and observable normalized results, not
  private helper calls, file placement, full error prose, punctuation, or one
  historical validation call order. Exact strings remain appropriate only for
  an intentionally documented CLI or machine-consumed contract.
- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9
  cleanup-stage work, or unrelated Workstream 10 cleanup.

## Instructions For Every Stage

Before each stage:

1. Read `docs/development.md`, every document under `docs/policy/`, this plan,
   and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit
   sections and decision-register rows cited above; do not restart the audits.
2. Read [`../internal/configuration.md`](../internal/configuration.md) and the
   affected portions of [`../config.md`](../config.md), then inspect the
   production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call
   tracing. Use text search for YAML fields, source IDs, error fragments,
   examples, documentation, and removal verification.
4. Confirm the worktree state and preserve unrelated changes. Do not implement
   an adjacent pending workstream merely because it shares a file.

During every stage:

- Keep the package compiling and focused tests passing. Move behavior before
  redesigning the two complex validators so reviewable commits do not combine
  unrelated mechanical and semantic changes.
- Use package-private functions with explicit inputs and return values. Avoid
  stateful validator objects, forwarding-only wrappers, generic rule tables,
  and abstractions whose only purpose is to reduce function length.
- Keep configuration mutation explicit and late. Do not mutate caller-owned
  maps while validating a temporary normalized candidate.
- Retain deterministic order wherever config maps are traversed for validation,
  normalization, canonical output, source reporting, or digest construction.
- Keep tests deterministic, offline, and independent of credentials or live
  adapters. Prefer direct package behavior and real temporary files over mocks.
- Use `gofmt` on changed Go files, run the focused checks listed for the stage,
  and run `git diff --check`. Mark a stage `Completed` only after its exit
  criteria pass.

## Stage 1 — Separate Core And Shared Validation Ownership

**Status:** Completed

### Goal

Reduce `validate.go` to configuration validation entry points and readable
high-level sequencing, while giving core pipeline/session rules and reusable
primitive checks clear package-local owners. This stage is behavior-preserving.

### Required Work

1. Keep these entry points and the visible pipeline-domain call sequence in
   `internal/config/validate.go`:
   - `Validate`;
   - `ValidatePipelineConfig`;
   - `ValidateCampaignConfig`; and
   - the short `validatePipeline` coordinator.
2. Move core configuration checks into a cohesive private file such as
   `validate_core.go`. This owner includes campaign validation, secrets,
   spool/cache, session validation, resolved-party input validation,
   session/campaign identifiers, and cross-pipeline/session rules. Keep
   `publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than
   hiding publish policy in the core owner.
3. Move general validation primitives into a small private file such as
   `validate_helpers.go`: environment-variable name checks, positive duration
   parsing, safe relative paths, and path-within-root checks. Do not turn these
   functions into a generic validation DSL.
4. Re-home the straightforward stage/adapter validators in a private adapter
   owner such as `validate_adapters.go`: notification, normalize, trim, render,
   WhisperX, Seriatim (including its output schema), and Audita. Move
   `validateAudita` intact apart from import or formatting changes.
5. Preserve current nil handling, defaults, normalization, wrapped causes,
   field identification, and high-level validation semantics. Do not use this
   mechanical move to reorder domains or rewrite diagnostics.
6. Remove imports and dead private helpers left behind in `validate.go`. Do not
   add facade functions to preserve former source-file placement; files are not
   Go API boundaries.

### Tests And Verification

- Existing core, campaign, session, duration, normalize, trim, render,
  notification, and load/validate tests should normally pass unchanged.
- Add no tests that merely assert a function moved files. Add or repair a test
  only if the move exposes an unprotected semantic rule.
- Run:

  ```sh
  go test ./internal/config
  go test ./internal/app -run 'Config|Pipeline|Session'
  git diff --check
  ```

### Exit Criteria

- `validate.go` is a concise entry/coordinator file rather than a container for
  unrelated domain implementations.
- Core, adapter, and shared helper ownership is obvious from the file layout,
  and Audita remains one cohesive validator.
- No public behavior or serialized configuration shape changed.

## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners

**Status:** Completed

### Goal

Finish the domain-level split of the former aggregate validator before changing
the internal structure of Notarius or Scriptorium validation.

### Required Work

1. Move storage, publish output, and publish lock validation into a cohesive
   private owner such as `validate_publish.go`. Move the associated helpers
   with their consumers, including `ValidatePublishLockRules`,
   `MergePublishLockRules`, configured output/source lookups, and
   `publishUploadConfiguredForS3`.
2. Preserve the exported lock-rule helper signatures because current app and
   stage code consume them. Do not redesign remote lock behavior; that belongs
   to Workstream 8.
3. Move the existing Notarius validator into `validate_notarius.go` and its
   Scriptorium counterpart, input-source validator, and exported dependency
   validator into `validate_scriptorium.go`. At this stage, move these functions
   without decomposing or semantically reordering them; Stages 3 and 4 own those
   changes.
4. Keep source-vocabulary and collision helpers with the narrowest shared
   domain owner. If both publish and Scriptorium need a small lookup derived
   from Notarius/Scriptorium configuration, use one plainly named package
   helper rather than duplicate maps or introduce a registry.
5. Ensure `validate.go` contains only entry points and high-level sequencing
   after the move. Delete the emptied blocks and unused imports.
6. Search `internal/config` production code for acceptance or translation of
   removed player/session roster fields and retired artifact source spellings.
   Delete any actual compatibility branch found. Do not mistake current
   campaign `party_file`, rejection through strict decoding, migration prose,
   or a test fixture demonstrating rejection for an active compatibility path.

### Tests And Verification

- Existing storage/publish, Notarius, Scriptorium, configuration loading, and
  application configuration tests should pass without pinning file placement.
- Run:

  ```sh
  go test ./internal/config
  go test ./internal/app ./internal/stage
  git diff --check
  ```

### Exit Criteria

- Every validation domain named by Workstream 7 has one discoverable file-level
  owner inside `internal/config`.
- There is no duplicate source/collision logic and no active retired-format
  compatibility branch in configuration production code.
- Notarius and Scriptorium semantics remain unchanged pending their dedicated
  stages.

## Stage 3 — Decompose Notarius Validation And Normalization

**Status:** Completed

### Goal

Make `validateNotarius` a short transaction that validates top-level policy,
normalizes references, normalizes output lanes, and commits the resulting
configuration only after every check succeeds.

### Required Work

1. Keep `validateNotarius` responsible for nil/enablement behavior and the
   top-level scalar contract: required binary, configuration path, pipeline ID,
   positive timeout, working directory, at least one output, and the reference
   binding limit.
2. Replace the validator's bespoke timeout parsing with the shared positive
   duration helper when it preserves the same accepted values and field
   identity. Do not embed duration policy in a Notarius-only branch.
3. Extract reference binding normalization into one private operation that:
   - iterates raw selector keys in sorted order;
   - normalizes selectors through `notariusref`;
   - rejects normalized selector collisions;
   - trims and validates each prepared-input source; and
   - returns a newly allocated normalized map without mutating the input map.
4. Extract output normalization and collision validation into a separate
   private operation that:
   - builds the reserved built-in, configured Scriptorium, and previous-session
     source set once;
   - iterates raw output keys in sorted order;
   - validates and normalizes configured keys and derived extraction source
     IDs;
   - rejects reserved-source, normalized-source, and lane-ID collisions;
   - trims lane/media/schema/module scalar fields and checks required fields;
     and
   - returns a newly allocated normalized output map.
5. Normalize binary, paths, pipeline ID, timeout, and working directory into
   local candidate values. Assign all scalar candidates, references, and
   outputs back to `cfg` only after both helper operations succeed. Preserve
   nil-versus-empty map behavior where it is observable in effective YAML or
   digests.
6. Keep all helpers Notarius-specific and private. Do not move selector policy
   out of `notariusref`, copy artifact source policy, or create a generalized
   map-normalization framework.

### Tests And Verification

- Retain lean behavior tests for disabled configuration, required scalars,
  timeout and reference-binding limits, selector normalization/collisions, supported
  prepared sources, output normalization, reserved-source collisions,
  duplicate lanes, and cross-use by Scriptorium/publish.
- Add or consolidate one test proving a failing reference or output candidate
  leaves the original Notarius config unmodified.
- Add one determinism test that constructs equivalent reference/output maps in
  different insertion orders and compares normalized success or equivalent
  diagnostic classification. Do not assert which of multiple independent
  invalid domains must win forever.
- Assert field paths and the smallest meaningful semantic fragment rather than
  complete error sentences.
- Run:

  ```sh
  go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
  go test ./internal/stage -run 'Extract|Analyze'
  git diff --check
  ```

### Exit Criteria

- `validateNotarius` visibly sequences scalar validation, reference
  normalization, output normalization/collision checks, and one final commit.
- Reference and output collision domains are separate, deterministic, and do
  not partially mutate caller-owned configuration on failure.
- No Notarius behavior moved into a generic configuration abstraction.

## Stage 4 — Decompose Deterministic Scriptorium Validation

**Status:** Completed

### Goal

Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and
variables into readable validation phases with deterministic map traversal.

### Required Work

1. Keep `validateScriptorium` as the transaction owner for nil behavior,
   required binary, optional config path, stage timeout, ordered validation
   phases, referenced-output checks, and the final dependency-graph check.
2. Produce one sorted list of configured artifact keys after validating their
   canonical key syntax. Reuse that order across phases rather than ranging
   over the artifact map independently.
3. Extract a declaration/output phase that validates enabled prompt and output
   requirements, safe output paths, artifact timeout values, and duplicate
   output ownership. Keep its collision map local to that phase.
4. Extract a dependency/input phase that:
   - validates dependency entries and self-reference using declared slice
     indexes for actionable field paths;
   - iterates input maps by sorted input name;
   - validates required source declarations through the existing canonical
     input-source owner;
   - requires configured-artifact sources to appear in `depends_on`; and
   - returns or updates only the small referenced-artifact set needed by the
     later referenced-output check.
5. Extract a variable phase that iterates variable names in sorted order,
   rejects empty names, and accepts only the currently supported string and
   boolean values.
6. Iterate the referenced-artifact set deterministically when checking required
   outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph
   validator and preserve its cycle and dependency semantics.
7. It is acceptable for phase separation to change which of several unrelated
   simultaneous mistakes is reported first. Within a given implementation,
   equivalent maps must produce the same result. Do not add compatibility code
   to reproduce the old mixed per-artifact call order.
8. Remove forwarding-only helpers and obsolete temporary sets after the split.
   Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate
   semantic owner emerges.

### Tests And Verification

- Retain semantic coverage for declaration keys, enabled requirements, output
  path safety/collisions, timeouts, dependency existence/self/cycles, input
  source classes, extraction references, required `depends_on`, referenced
  outputs, and variable types.
- Add one map-insertion-order test that feeds equivalent Scriptorium maps in
  different insertion orders and asserts equal validity or equivalent
  diagnostics. Do not assert an arbitrary globally preferred independent-error
  order.
- Consolidate table cases that repeat the same source-vocabulary rule through
  different retired spellings when a representative canonical rejection test
  already protects unsupported-source behavior.
- Run:

  ```sh
  go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
  go test ./internal/stage -run 'Analyze'
  go test ./internal/app -run 'Artifact|Config|Plan'
  git diff --check
  ```

### Exit Criteria

- Scriptorium validation has explicit declaration/output,
  dependency/input, variable, and graph phases.
- All map traversal that can affect validation or normalization is
  deterministic.
- Tests protect fields and semantic rules without freezing the former mixed
  validator choreography.

## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata

**Status:** Pending

### Goal

Separate composition-envelope shape decoding, cross-field semantics, and
root-only metadata removal while retaining the source-aware configuration model
that imports, profiles, diagnostics, and inspection commands require.

### Required Work

1. Refactor `splitPipelineCompositionEnvelope` in
   `internal/config/pipeline_composition.go` into a short coordinator over
   three explicit private phases:
   - locate and decode the root `composition` mapping and its `imports`,
     `default_profile`, and `profiles` shapes;
   - validate cross-field semantics, including default-profile membership; and
   - clone/remove the root-only envelope to produce the ordinary pipeline body.
2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization.
   Keep unknown-field rejection, non-empty profile declarations, imports and
   profiles in declaration order, explicit-profile selection behavior, and
   source-qualified shape/conflict diagnostics.
3. Do not mutate the parsed root document during validation. Return a distinct
   body document only after the envelope is valid; the original document must
   remain reusable by callers and tests.
4. Remove `compositionNode` and `compositionField` metadata that has no runtime
   reader after the refactor. Specifically verify and remove redundant cached
   `order`, `line`, and `column` fields and their clone/renumber assignments if
   they remain write-only. Slice order remains the declaration-order
   representation; YAML parser nodes remain the source of parse-time line and
   column diagnostics.
5. Retain each node's semantic path and source list because merge conflict
   reporting, ownership projection, effective digests, artifact-family source
   projection, and `config sources` consume them. Do not reduce actionable
   conflict diagnostics or delete provenance that has a real consumer.
6. Keep composition product-specific. Do not replace the tree with generic map
   decoding or adopt a third-party composition library.

### Tests And Verification

- Test malformed envelope shapes, unknown fields, invalid profile declarations,
  missing default membership, successful envelope removal, and absence of
  caller-document mutation through the composition boundary.
- Preserve tests for explicit empty/false/zero/list/map values, additive
  conflicts naming all claiming sources, overlay semantics, canonical YAML,
  effective digest stability, profile selection, and `config sources`
  ownership. These are semantic contracts, not incidental provenance.
- Remove assertions of deleted private cached metadata. Continue asserting
  declaration behavior through ordered outputs rather than private index
  fields.
- Run:

  ```sh
  go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
  go test ./internal/app -run 'Config'
  git diff --check
  ```

### Exit Criteria

- Envelope decoding, cross-field validation, and body production have separate
  focused owners under one coordinator.
- Only provenance and ordering data with a demonstrated consumer remains.
- Imports, profiles, overlays, source reporting, and effective digests retain
  deterministic behavior.

## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7

**Status:** Pending

### Goal

Review the completed configuration refactor as one system, remove obsolete test
friction and documentation language, and close Workstream 7 with repository-wide
evidence.

### Required Work

1. Review every configuration test changed by Stages 1–5. Retain coverage of
   schema strictness, normalization, semantic invariants, meaningful failure
   categories, deterministic outcomes, composition source conflicts, effective
   projections/digests, and representative assembled loads.
2. Rewrite assertions that pin complete diagnostic prose, punctuation,
   formatting, private helper order, or an arbitrary independent-error winner.
   Prefer a relevant field path plus one small semantic fragment. Use
   `errors.Is`/`errors.As` only where an existing wrapped or typed cause is a
   real contract; do not introduce types solely to change test style.
3. Delete redundant legacy-form enumerations whose only purpose is to preserve
   knowledge of retired fields or spellings already rejected by the canonical
   strict/source boundary. Retain representative tests proving unknown fields
   and unsupported sources fail, and retain migration guidance tests only when
   that guidance is an intentional user-facing contract.
4. Consolidate oversized table fixtures only where multiple cases protect the
   same rule. Do not delete distinct risks merely to reduce test LOC, and do not
   rewrite healthy semantic tests just because they use a small stable error
   fragment.
5. Update [`../internal/configuration.md`](../internal/configuration.md) to
   describe the implemented validation-domain ownership, deterministic phase
   behavior, composition-envelope phases, and diagnostic guarantees without
   listing private helper choreography. Update [`../config.md`](../config.md)
   only if its user-visible current-behavior wording is inaccurate; no schema
   change is expected.
6. Search current production, tests, examples, and current-behavior docs for
   active acceptance of the retired configuration forms named by C01. Remove
   actual compatibility code and stale current-behavior claims, while leaving
   immutable audit/history documents unchanged.
7. Run the full repository validation from `docs/development.md`:

   ```sh
   go test ./...
   go test -race ./...
   go vet ./...
   go build ./...
   go test ./internal/doccheck
   go test ./internal/config -run '^TestExamplesLoadAndValidate$'
   ```

8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app`
   to stress map-order independence and assembled configuration behavior.
9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark
   only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`.
   Workstream 8 must remain the next pending workstream.

### Exit Criteria

- Configuration remains strict, deterministic, source-aware where useful, and
  unchanged in public schema and normalized semantics.
- Validation code is organized by semantic owner; Notarius, Scriptorium, and
  composition coordinators expose clear phases without generic frameworks.
- No active retired configuration compatibility path, write-only composition
  metadata, or test dependency on incidental diagnostic choreography remains.
- Current documentation describes the implemented boundaries, all validation
  commands pass, Workstream 7 is complete, and Workstream 8 remains pending.

## Open Questions

None. The audits, completed compatibility purge, current configuration
contracts, and fixed decisions above resolve the product and implementation
choices needed for a `gpt-5.6-terra` coding agent to execute this plan.