narratio/docs/roadmap/remote.md

# Roadmap: Campaign State, Remote Sessions, and Archive Locks

## Purpose

This roadmap tracks implementation work for three related feature areas:

1. `campaign.yml` configuration for stable campaign-level inputs.
2. Remote `session.yml` loading from the existing S3 object-store backend.
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.

Treat future sections of this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each item is implemented and canonical docs are updated.

## Current Code Facts

The current codebase already settles several design choices:

- CLI commands use short noun flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--force`, and `--artifacts`.
- `run-stage` uses flags before the positional stage name, for example:

      narratio run-stage --session ./session.yml prepare

- `session.yml` is represented by `config.SessionConfig` and currently owns `session_id`, `previous_session_id`, `campaign`, `date`, `title`, and `inputs`.
- Strict YAML decoding is already implemented with `yaml.Decoder.KnownFields(true)`.
- Local session discovery is already ordered as `./session.yml`, `/usr/local/etc/narratio/session.yml`, then `/etc/narratio/session.yml`.
- The canonical S3 session prefix is already:

      {root_prefix}/campaigns/{campaign}/sessions/{session_id}/

- Archive promotion is already source-based through `archive.promote_artifacts[].source`, with destination derivation and validation in `internal/config`.
- Storage adapters receive bucket-relative keys and do not infer campaign, session, run, or root-prefix semantics.

## Guardrails

Keep Narratio explicit and stage-driven. Do not introduce a generic workflow engine, broad config language, or stage behavior that reaches through adapter boundaries.

Implementation must preserve these constraints:

- Keep storage details behind `internal/adapters/storage`.
- Compute session, campaign, archive, and remote config keys in app/artifact/path helpers, not inside storage implementations.
- Use centralized path helpers in `internal/artifacts` or the established local path model.
- Preserve manifest-driven resume and stage status semantics.
- Keep strict YAML decoding for `pipeline.yml`, `campaign.yml`, and `session.yml`.
- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
- Update canonical user-facing docs only after behavior is implemented.

## Phase 1: Add `campaign.yml` (implemented)

Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.

### CLI and Discovery

Add `--campaign <path>` to `run`, `plan`, `resume`, `run-stage`, and `restore`.

Examples:

      narratio run --campaign ./campaign.yml --session ./session.yml
      narratio plan --campaign ./campaign.yml --session ./session.yml
      narratio resume --campaign ./campaign.yml --session ./session.yml
      narratio run-stage --campaign ./campaign.yml --session ./session.yml prepare
      narratio restore --campaign ./campaign.yml --session ./session.yml

Campaign config discovery order:

1. explicit `--campaign <path>`;
2. `./campaign.yml`;
3. `/usr/local/etc/narratio/campaign.yml`;
4. `/etc/narratio/campaign.yml`.

Implement this in the same style as `resolvePipelineConfigPath` and `resolveSessionConfigPath`. Add default path constants and a search-path variable in `internal/config/defaults.go`.

### Config Shape

Initial `campaign.yml` fields:

      campaign: icewind-dale
      inputs:
        speakers_file: ./speakers.yml
        autocorrect_file: ./autocorrect.yml
        glossary_file: ./glossary.yml

Do not add speculative campaign artifact defaults, prompt defaults, or title conventions in the first implementation.

### Merge Behavior

Add `CampaignConfig` and keep the final stage-facing config explicit.

Required behavior:

- `pipeline.yml` remains host/runtime configuration.
- `campaign.yml` supplies campaign identity and stable input file defaults.
- `session.yml` remains the source for `session_id`, `previous_session_id`, `date`, `title`, and audio input.
- Campaign-level `speakers_file`, `autocorrect_file`, and `glossary_file` fill missing session-level stable input fields.
- Session-level stable input fields override campaign-level stable input fields.
- If both `campaign.yml` and `session.yml` specify `campaign`, the values must match.
- The resolved session must satisfy the existing session validation rules before stages run.
- Unknown fields in `campaign.yml` fail strict decode.

Path resolution must preserve source-file locality:

- campaign-provided stable input paths resolve relative to `campaign.yml`;
- session-provided stable input overrides resolve relative to `session.yml`;
- absolute paths keep existing behavior.

Track enough provenance in the resolved config or prepare inputs so `prepare` can copy the correct source files without guessing which file supplied each path.

### Prepare Behavior

Update `prepare` to materialize the resolved campaign/session inputs into canonical session input paths:

      inputs/campaign.yml
      inputs/session.yml
      inputs/pipeline.resolved.yml
      inputs/speakers.yml
      inputs/autocorrect.yml
      inputs/glossary.yml

Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.

## Phase 2: Load Remote `session.yml` (implemented)

Support running with no local session file when a remote session file exists under the canonical session prefix.

### Preconditions

Build this phase after `campaign.yml`, because campaign identity is required to compute the remote session key. Do not infer campaign identity from object-store listing.

### Loading Precedence

Session loading order:

1. If `--session <path>` is supplied, load that local file.
2. If `--session` is omitted, use existing local discovery: `./session.yml`, `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`.
3. If no local session file is found, `--session-id` is present, storage is configured, and campaign identity is resolved, load remote `session.yml`.
4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable.

Do not make remote loading mask local discovery. Existing local discovery remains the local fallback before remote is attempted. Once remote loading is attempted, a missing remote object, storage init error, or malformed remote YAML fails clearly because no local session was available.

### Remote Key Layout

Use the existing canonical S3 layout:

      session prefix: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
      session file:   {session_prefix}/session.yml
      audio prefix:   {session_prefix}/{session.inputs.audio_s3.prefix}

Add a centralized helper near `internal/artifacts/s3_keys.go`:

      S3SessionConfigKey(sessionPrefix string) string

The helper should return `{session_prefix}/session.yml` using the same key normalization style as `S3CurrentManifestKey`, `S3CurrentRunPointerKey`, and `S3PromotedArtifactKey`.

### Decode, Template, and Provenance

Remote `session.yml` uses the same template variables and mismatch checks as local sessions:

- `{{session_id}}`
- `{{ session_id }}`
- `{{previous_session_id}}`
- `{{ previous_session_id }}`

Decode remote session YAML with strict known-field validation. Reuse the current session template/render/decode path by adding a byte/string-based loader rather than duplicating YAML decode logic.

When `prepare` materializes a remote session into `inputs/session.yml`, record that it came from S3. Preserve useful non-secret provenance when available:

- bucket;
- key;
- ETag;
- size;
- local checksum;
- downloaded temp/materialized path.

## Phase 3: Add Logical Archive Locks (implemented)

Narratio supports source-based archive locks under `pipeline.archive.locks`. The promotion system is source-based; destination-based locks are not supported.

### Config Shape

Lock entries:

      archive:
        locks:
          - source: narratio.transcript.polished
            reason: Human-reviewed transcript; do not overwrite automatically.
          - source: narratio.artifact.session_recap
            reason: Final recap was manually edited.

Validation rules:

- `source` is required.
- `source` must be a built-in source ID or configured `narratio.artifact.<key>` accepted by the same source validation used for `promote_artifacts`.
- `reason` is optional and non-secret.
- duplicate lock sources fail validation.
- lock entries do not support `dest` in the first implementation; unknown fields already fail strict decode.

### Archive Behavior

Archive continues uploading complete run-local outputs under `runs/{run_id}/`.

Promotion behavior:

1. Resolve promotion source and destination using existing source-based promotion logic.
2. If the promotion source is unlocked, upload the top-level promoted object normally.
3. If the promotion source is locked, skip only the top-level promotion overwrite.
4. Treat locked required promotions as intentional successful skips by default.
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last.

Ordinary `--force` does not override locks. A lock-break override remains out of scope.

### Metadata

Archive records locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.

Include:

- source ID;
- destination relative path and remote key;
- reason;
- local resolved path;
- resolved provenance;
- whether the original promotion rule was required.

Keep existing metadata such as `promoted_paths`, `skipped_optional_promotions`, `current_manifest_key`, `current_run_id_key`, and `current_pointer_written`.

## Phase 4: Future Operator Helpers

These commands are future work only. Do not implement them with the first campaign, remote-session, or lock changes.

Potential helper shapes:

      narratio session validate --session-id 2026-06-07
      narratio session init --session-id 2026-06-07 --title "The Black Cabin"
      narratio status --session-id 2026-06-07
      narratio locks --session-id 2026-06-07
      narratio lock narratio.artifact.session_recap --session-id 2026-06-07
      narratio unlock narratio.artifact.session_recap --session-id 2026-06-07

Potential behavior:

- validate remote session config;
- check audio object availability;
- inspect committed remote current state;
- list promoted transcripts/artifacts;
- list archive lock status;
- initialize a remote session skeleton;
- publish or sync campaign assets.

## Implementation Sequence

Use small, reviewable commits.

1. Campaign config types and discovery:
   add `CampaignConfig`, strict loading, defaults/search paths, `--campaign` flags, and config/app tests.
2. Campaign/session merge:
   implement resolved stable input merge, path provenance, validation, and prepare materialization.
3. Campaign docs after implementation:
   update canonical docs and examples only for implemented behavior.
4. Remote session key and loader:
   add `S3SessionConfigKey`, byte/string session loading, remote download through `ObjectStore`, and app-level precedence tests.
5. Remote session prepare provenance:
   materialize downloaded session config and record S3 provenance.
6. Remote session docs after implementation:
   update canonical docs and examples only after behavior exists.
7. Archive lock config:
   add lock config structs, strict decode coverage, source validation, and duplicate detection.
8. Archive lock enforcement:
   skip locked top-level promotions, preserve run-local uploads, record lock metadata, and protect commit ordering.
9. Final sweep:
   run focused tests, then `go test ./...`; verify planned behavior remains only in roadmap docs until implemented.

## Test Plan

Add focused coverage in these packages:

- `internal/config`: campaign load, strict decode, discovery constants, merge validation, campaign/session mismatch, lock validation, duplicate lock rejection.
- `internal/app`: `--campaign` parsing on `run`, `plan`, `resume`, `run-stage`, and `restore`; campaign discovery; explicit `--session` precedence; local discovery before remote; remote session fallback when local discovery misses.
- `internal/artifacts`: `S3SessionConfigKey`; canonical session prefix compatibility; source ID validation for lock sources.
- `internal/stage/prepare`: campaign/session stable input materialization; campaign-relative and session-relative path resolution; remote session provenance in `manifest.Inputs`.
- `internal/stage/archive`: locked required promotion succeeds as skipped; unlocked promotion uploads; run-local outputs upload when top-level promotion is locked; `--force` does not break locks; `current/run_id.txt` remains the last upload.
- `internal/adapters/storage`: fake object key normalization and remote session download expectations.

Run at least:

      go test ./internal/config -v
      go test ./internal/app -run TestExecute -v
      go test ./internal/artifacts -v
      go test ./internal/stage -run 'Prepare|Archive' -v
      go test ./internal/adapters/storage -v
      go test ./...

Use fake storage for remote-session and archive-lock behavior. Ordinary tests must not require live S3.

## Documentation Updates After Implementation

After each phase is implemented, update only docs for behavior that exists.

Likely files:

- `docs/config.md`
- `docs/cli.md`
- `docs/operations.md`
- `docs/internal/stage-prepare.md`
- `docs/internal/stage-archive.md`
- `docs/internal/storage.md`
- `docs/internal/artifacts.md`
- relevant examples under `examples/`

Do not document remote campaign loading, helper commands, or lock override flags as current behavior until implemented.

## Remaining Open Decisions

The codebase resolves the campaign flag name, campaign discovery order, remote session layout, local-vs-remote session precedence, source-based archive lock model, and locked required promotion policy.

Remaining decisions:

1. Whether `campaign.yml` should eventually be loadable from S3. Do not implement remote campaign loading in the first phase.
2. Whether a future explicit lock override command or flag is needed. Do not make ordinary `--force` break locks.
3. Whether future helper commands should be top-level commands or subcommands. Keep them out of the first implementation.