narratio/docs/operations.md

# Operations

This guide describes the implemented operator lifecycle for Narratio.

For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).

## Normal workflow (S3-first path)

1. Create or upload `session.yml`, or pass a local `session.yml` explicitly.
2. Upload session `.flac` files to object storage under the configured session audio prefix.
3. Run Narratio:

```bash
narratio run --session-id 2026-04-04
```

4. Read success output:
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
- use `manifest=<path>` with `status` for inspection.

Notes:
- default config/campaign/session discovery checks system config locations unless `--config`, `--campaign`, and `--session` are passed.
- when local `session.yml` discovery misses, `--session-id` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.

Initialize a remote session skeleton:

```bash
narratio session init --config /etc/narratio/pipeline.yml --campaign /etc/narratio/campaign.yml --session-id 2026-04-04 --remote
```

Remote init writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. It fails if the object already exists unless `--force` is passed.

Validate before running:

```bash
narratio session validate --session-id 2026-04-04
```

## Restore workflow

Use restore when local durable session state is missing or stale and archive current state is authoritative.

Dry-run (no local writes):

```bash
narratio restore --session-id 2026-04-04 --dry-run
```

Execution:

```bash
narratio restore --session-id 2026-04-04
```

Post-restore analyze rerun pattern:

```bash
narratio analyze --session-id 2026-04-04
```

Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session.

Restore default scope:
- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would.
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)

Reset local state before restore testing:

```bash
narratio clean --session-id 2026-04-04 --dry-run
narratio clean --session-id 2026-04-04
narratio restore --session-id 2026-04-04 --include-audio
```

`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.

## Local filesystem layout and state artifacts

Session root:
- `{workspace.root}/work/{campaign}/{session_id}/`

Primary state:
- `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a modifying command is active.
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.

Canonical session directories:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `previous/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`

Run-local stage directories:
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.

Behavior:
- directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.

## Analyze artifact execution lifecycle

Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.

Execution model:
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
- artifact-to-artifact dependencies are declared via `depends_on`.
- selected artifacts run in deterministic dependency order.
- after each successful artifact run, output is promoted to configured canonical `output_path`.

Configured artifact source reuse:
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.

`--artifacts` behavior:
- accepted on `run`, `resume`, `run-stage analyze`, and `analyze`.
- filters analyze execution only.
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.

Canonical previous-session input behavior:
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist.
- if analyze fails due to missing canonical previous cache, rerun:
  - `narratio run-stage --session-id <id> --force prepare`
  - or `narratio restore --session-id <id>` when remote archive current state is authoritative.

## Remote archive layout and publish contract

When archive is enabled and run upload is enabled, archive publishes under:

- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- run prefix: `{session_prefix}/runs/{run_id}/`

Archive uploads:
- run record files from run root (excluding `audio/`).
- promoted files from explicit `archive.promote_artifacts` rules.
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.

Publish order:
1. upload `current/manifest.json`
2. upload `current/run_id.txt` last

`current/run_id.txt` is the remote commit marker.

Archive promotion is explicit and source-based:
- Narratio does not auto-promote all generated analyze artifacts.
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.

Lock helper behavior:
- `narratio locks --session-id <id>` lists effective static and remote locks.
- `narratio locks add --session-id <id> --reason <text> <source>` writes a remote lock.
- `narratio locks add --session-id <id> --force --reason <text> <source>` updates an existing remote lock reason.
- `narratio locks remove --session-id <id> <source>` removes only a remote lock.
- `locks remove` cannot remove static pipeline locks.
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.

## Resume, retry, restore, and safe rerun behavior

Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.

Resume:
- `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order.

Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.

Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- ordinary `--force` does not override archive locks.

Safe rerun pattern:
1. rerun the changed stage with `--force`.
2. run `resume` to rebuild downstream stages.

## Cleanup behavior

Automatic post-archive cleanup is considered only when archive stage executed and succeeded.

Automatic cleanup toggles:
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.

Manual cleanup:
- `narratio clean --session-id <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
- `--dry-run` prints targets without deleting.
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.

The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.

Cleanup eligibility gates:
- archive enabled
- archive run upload enabled
- run record upload completed
- current pointer write completed (`current/run_id.txt` written)

No cleanup for failed/incomplete/unarchived/archive-skipped runs.

## Failure and recovery playbooks

After run failure, Narratio keeps:
- session manifest
- run manifest
- run-local artifacts/logs/config/reports

Failed or incomplete runs remain local-only.

After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.

Recommended recovery:

1. inspect state:

```bash
narratio status --session-id 2026-04-04
```

This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks.

2. for one manifest file, run:

```bash
narratio status --manifest <manifest-path>
```

3. for restore-specific checks, run:

```bash
narratio restore --session-id 2026-04-04 --dry-run
```

4. fix root cause (config/input/credentials/storage/service availability).
5. continue with `resume`, or targeted `run-stage --force` followed by `resume`.

## Restore report

Non-dry-run restore writes a durable report at:
- `reports/restore-latest.json`

Report content includes:
- identity (`campaign`, `session_id`, `run_id`)
- mode flags (`dry_run`, `force`, `include_audio`)
- plan counts and execution counts
- per-action status

Dry-run does not write restore report files.

## Operational caveats

- `status` with no config/session flags still requires explicit `--manifest`.
- `status --session-id <id>` uses normal config/session loading, including remote session fallback.
- `status --session-id <id>` includes the same promoted remote output availability view as `artifacts list --remote` when storage is configured.
- local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`.
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
- restore requires configured remote object storage and committed remote current state.