narratio/docs/roadmap/previous.md

# Roadmap: Previous-Session Artifacts

## Status

Completed.

This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.

## Summary

Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session.

The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just `session_recap`.

The canonical source syntax should be:

```yaml
source: narratio.previous_session.artifact.<artifact_name>
```

For example:

```yaml
source: narratio.previous_session.artifact.session_recap
```

Previous-session artifacts are materialized during the `prepare` stage into a current-session top-level `previous/` directory. Downstream stages consume only the local `previous/` copies. The S3 backend is authoritative for previous-session state.

## Design Decisions

### 1. Add `previous_session_id` to `session.yml`

Add an optional top-level session key:

```yaml
session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign
```

Rules:

- `previous_session_id` is optional.
- If present, it identifies the previous session within the same campaign.
- It must not equal `session_id`.
- It should use the same validation rules as `session_id`.
- It may be supplied through session templating.
- Add a CLI/template value such as `--previous-session-id <id>` if required by the existing session templating implementation.
- If a template placeholder for `previous_session_id` is present and no value is supplied, loading should fail with a clear unresolved-template error.

### 2. Use canonical previous-session artifact source IDs

Support this source pattern in Scriptorium artifact input definitions:

```text
narratio.previous_session.artifact.<artifact_name>
```

Examples:

```yaml
inputs:
  previous_recap:
    source: narratio.previous_session.artifact.session_recap
    required: false

  previous_quest_log:
    source: narratio.previous_session.artifact.quest_log
    required: true
```

Rules:

- `<artifact_name>` must be a valid configured artifact key.
- Use the same artifact key validation rules as current-session runtime artifacts.
- Do not special-case `session_recap`.
- Do not limit implementation to a fixed list of previous artifacts.

### 3. Add a top-level `previous/` workspace directory

Extend the session workspace layout with:

```text
previous/
  manifest.json
  artifacts/
    <artifact outputs copied from the previous session>
```

The `previous/` directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace.

Conceptually:

```text
work/<campaign>/<current_session_id>/
  previous/
    manifest.json
    artifacts/
      session_recap.md
      quest_log.json
```

The current session should not read directly from the previous session's local workspace during ordinary operation.

### 4. S3 is authoritative for previous-session state

For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend.

Do not compare local and remote copies.

Do not prefer local previous-session workspace state.

Do not implement a `--local` override in this roadmap. That can be considered later.

The simplified stage behavior is:

1. If the local manifest indicates `prepare` already succeeded and `--force` is not supplied, the runner skips `prepare`. No previous-session download occurs.
2. If `prepare` has not succeeded, `prepare` runs and downloads referenced previous-session artifacts from S3.
3. If `prepare` previously succeeded but `--force` is supplied, `prepare` runs again and overwrites local `previous/` state from S3.

### 5. Materialize only referenced previous-session artifacts

During `prepare`, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches:

```text
narratio.previous_session.artifact.<artifact_name>
```

Only those referenced previous-session artifacts need to be downloaded.

Do not blindly download every previous-session artifact.

If no previous-session artifact sources are referenced, `prepare` should not require `previous_session_id` and should not touch `previous/`.

### 6. Preserve stage boundaries

`prepare` owns previous-session artifact materialization because these files are inputs to later stages.

`analyze` should not talk to S3.

The Scriptorium adapter should not know about previous sessions.

The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys.

### 7. Archive and restore `previous/`

After implementation:

- `archive` should include `previous/` as durable current-session prepared input state.
- `restore` should restore `previous/` along with the rest of the durable session state it already restores.
- `previous/` should not be treated as current-session generated artifacts.
- `previous/` entries should be recorded as inputs/provenance, not as outputs produced by the current session.

## Target User Workflow

A typical session config:

```yaml
session_id: "{{ session_id }}"
previous_session_id: "{{ previous_session_id }}"
campaign: sample-campaign

inputs:
  audio_dir: ./audio
  speakers_file: ./examples/speakers.yml
  autocorrect_file: ./examples/autocorrect.yml
  glossary_file: ./examples/glossary.yml
```

A typical artifact config:

```yaml
scriptorium:
  artifacts:
    session_recap:
      enabled: true
      prompt_id: dnd_session.session_recap
      output_path: artifacts/session_recap.md
      inputs:
        transcript:
          source: narratio.transcript.trimmed
          required: true
        previous_recap:
          source: narratio.previous_session.artifact.session_recap
          required: false
```

Typical command:

```bash
narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04
```

Expected behavior:

1. `prepare` sees a referenced previous-session artifact: `session_recap`.
2. `prepare` downloads the previous session's `narratio.artifact.session_recap` from S3.
3. `prepare` writes it under the current session workspace, for example `previous/artifacts/session_recap.md`.
4. `prepare` records provenance in the current session manifest.
5. `analyze` resolves `narratio.previous_session.artifact.session_recap` from the local `previous/` directory.
6. Scriptorium receives the previous recap as a normal input file.

## Implementation Plan

### Phase 1: Session config and templating

Update session configuration structs to include:

```yaml
previous_session_id: ""
```

Implementation steps:

1. Add `PreviousSessionID` or equivalent to the session config type.
2. Add validation:
   - optional;
   - same format constraints as `session_id`;
   - must not equal `session_id`.
3. Extend session templating support to include:
   - `{{previous_session_id}}`
   - `{{ previous_session_id }}`
4. Add a CLI flag if required by current templating flow:
   - `--previous-session-id <id>`
5. Ensure unresolved `previous_session_id` placeholders fail clearly.
6. Update config tests for:
   - no previous session;
   - valid previous session;
   - previous session equal to current session;
   - unresolved placeholder;
   - CLI/template rendering.

Do not add future workflow flags in this phase.

### Phase 2: Workspace path helpers

Add centralized path helpers for current-session previous-state paths.

Suggested helpers:

```text
SessionPreviousDir()
SessionPreviousManifestPath()
SessionPreviousArtifactsDir()
SessionPreviousArtifactPath(name or relative output path)
```

The exact names should match existing path-helper style.

Rules:

- Do not construct `previous/` paths through scattered string concatenation.
- Keep paths session-relative where possible.
- Ensure workspace layout creation includes `previous/` only when appropriate, or creates it idempotently with the rest of the layout if simpler.

Tests:

- path helper tests;
- workspace layout tests;
- ensure cleanup logic does not accidentally delete configured roots;
- ensure `previous/` is treated as session-durable state, not run-local state.

### Phase 3: Previous-session source parsing

Add parsing/recognition for:

```text
narratio.previous_session.artifact.<artifact_name>
```

Implementation steps:

1. Add constants/helpers in the artifact/source parsing layer.
2. Validate artifact names using the same rules as current runtime artifact keys.
3. Add helpers such as:
   - `IsPreviousSessionArtifactSource(source string) bool`
   - `PreviousSessionArtifactName(source string) (string, bool)`
4. Ensure config validation accepts this source pattern.
5. Ensure invalid sources fail clearly.

Tests:

- valid previous-session artifact source;
- invalid/missing artifact name;
- invalid artifact key characters;
- ordinary current-session sources still validate;
- unknown sources still fail.

### Phase 4: Scan configured artifacts for previous-session inputs

Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names.

Rules:

- Scan enabled artifacts according to current artifact-enable semantics.
- Include all inputs whose source matches `narratio.previous_session.artifact.<name>`.
- Deduplicate artifact names.
- Sort results deterministically.
- Preserve required/optional information per reference.
- If the same previous artifact is referenced both required and optional, treat it as required.

Suggested output model:

```go
type PreviousArtifactRequirement struct {
    Name     string
    Required bool
    Sources  []string // optional diagnostics
}
```

Tests:

- no artifacts;
- no previous inputs;
- one optional previous input;
- one required previous input;
- duplicate references;
- required plus optional reference to the same artifact;
- deterministic ordering.

### Phase 5: Resolve previous-session archive keys

Implement a narrow service/helper used by `prepare` to resolve previous-session artifact files from S3.

Responsibilities:

1. Locate the previous session's current remote state.
2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs.
3. Resolve `narratio.artifact.<name>` inside the previous session's artifact catalog/manifest.
4. Download the resolved artifact into the current session's `previous/` directory.
5. Download/store the previous session manifest as `previous/manifest.json`.
6. Return provenance records for manifest input recording.

Important archive invariant:

- Remote current state must be based on the committed archive marker.
- Do not treat incomplete archive uploads as current state.
- Use the existing archive/current remote layout and commit-marker rules.
- `current/run_id.txt` is the final remote commit marker and should be respected when locating current remote state.

Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter.

Required vs optional behavior:

- Required previous artifact missing from S3/current manifest: fail `prepare`.
- Optional previous artifact missing from S3/current manifest: continue without materializing that input.
- Previous session missing entirely:
  - fail if any referenced previous artifact is required;
  - continue if all referenced previous artifacts are optional.
- If previous_session_id is unset:
  - fail if any referenced previous artifact is required;
  - continue and omit all previous-session inputs if all are optional.

Validation behavior:

- Downloaded artifacts should pass the same validation rules as current-session artifacts where practical.
- Generic Scriptorium artifacts should at least be non-empty.
- Invalid required artifact: fail.
- Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence.

Tests:

- downloads previous manifest;
- downloads required previous artifact;
- skips missing optional previous artifact;
- fails missing required previous artifact;
- fails required previous artifact when previous_session_id is unset;
- optional previous artifact with no previous_session_id does not fail;
- respects current remote commit marker;
- does not use local previous-session workspace state;
- uses storage adapter with explicit keys.

### Phase 6: Integrate with `prepare`

Extend the `prepare` stage:

1. Run existing input materialization as before.
2. Detect previous-session artifact requirements.
3. If requirements exist, hydrate `previous/` from S3 according to the rules above.
4. Record hydrated previous artifacts in `manifest.Inputs`.
5. Preserve existing prepare outputs and provenance behavior.

Overwrite behavior:

- If `prepare` runs, it owns `previous/`.
- Before hydrating, clear the managed `previous/` directory, or clear the managed previous artifact paths.
- Prefer clearing the whole `previous/` directory if no other feature writes there.
- Under `--force`, this naturally overwrites local `previous/` state.
- Do not compare local and remote copies.

Skip behavior:

- Do not add stage-local skip logic.
- Runner-level skip remains authoritative.
- If the manifest says `prepare` succeeded and `--force` is not supplied, `prepare` does not run and no S3 downloads occur.
- If users add a new previous-session input after `prepare` already succeeded, they must rerun prepare with `--force`.

Error message requirement:

If an analyze-stage input cannot be resolved because `previous/` is missing or stale, the error should tell the operator to run:

```bash
narratio run-stage --force prepare
```

or the appropriate existing CLI command shape.

Tests:

- ordinary prepare without previous_session_id remains unchanged;
- prepare with optional previous artifact and no previous_session_id succeeds;
- prepare with required previous artifact and no previous_session_id fails;
- prepare with required previous artifact downloads to `previous/`;
- force prepare overwrites `previous/`;
- prepare records manifest inputs for previous artifacts;
- prepare skip behavior remains controlled by runner tests;
- no regression in S3 audio prepare behavior.

### Phase 7: Artifact resolver support

Update artifact resolution so Scriptorium inputs can resolve:

```text
narratio.previous_session.artifact.<artifact_name>
```

from the current session's `previous/` directory.

Rules:

- The resolver should not call S3.
- The resolver should not read the previous session's local workspace.
- The resolver should map the previous-session source ID to the local prepared copy under `previous/`.
- Resolution should use manifest input records when available.
- Fallback to the local `previous/` path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred.
- Missing required input should fail with a clear prepare-oriented message.
- Optional missing input should be omitted.

Suggested provenance:

```text
previous_session.manifest.outputs
previous_session.archive.current
current_session.previous_cache
```

Use names that fit the existing manifest/resolver vocabulary.

Tests:

- resolves prepared previous artifact through manifest input record;
- resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy;
- missing optional previous artifact is omitted;
- missing required previous artifact fails clearly;
- current-session `narratio.artifact.<name>` behavior is unchanged.

### Phase 8: Analyze-stage integration

The analyze stage should require little or no special previous-session logic if the resolver is designed correctly.

Confirm:

- Scriptorium input resolution accepts previous-session source IDs.
- The Scriptorium adapter receives a normal local input path.
- Render-debug and run modes behave the same as for ordinary inputs.
- Stage metadata includes useful input provenance if current structures support it.

Tests:

- configured artifact receives previous recap input;
- optional previous recap omitted when not prepared;
- required previous recap fails when not prepared;
- render-debug path works with previous-session inputs;
- no S3 calls occur from analyze.

### Phase 9: Archive `previous/`

Update archive behavior so the current session's durable `previous/` directory is uploaded/preserved.

Rules:

- `previous/` is current-session input/provenance state.
- It is not a generated current-session artifact.
- Archive it with the current session's durable state, alongside other durable session files according to the current archive layout.
- Preserve existing archive commit ordering.
- Do not make `previous/` upload the final commit marker.
- Do not treat missing `previous/` as an error when no previous-session inputs were prepared.

Tests:

- archive includes `previous/manifest.json` and prepared previous artifacts when present;
- archive omits or tolerates absent `previous/` when unused;
- archive commit ordering remains valid;
- cleanup behavior does not delete durable `previous/` before archive commit.

### Phase 10: Restore `previous/`

Update `narratio restore` so it restores the current session's archived `previous/` directory.

Rules:

- Restore `previous/` as durable current-session state.
- Do not infer or restore the previous session workspace merely because `previous_session_id` exists.
- Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived `previous/` cache.
- Respect existing restore flags and overwrite behavior.

Tests:

- restore downloads `previous/` when present;
- restore succeeds when `previous/` is absent;
- restore with force overwrites local `previous/` according to existing restore semantics;
- restored current session can run analyze using `previous/` without needing previous session archive access.

### Phase 11: Documentation updates after implementation

After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap.

Likely files:

- `docs/config.md`
- `docs/cli.md`
- `docs/operations.md`
- `docs/internal/stage-prepare.md`
- `docs/internal/artifacts.md`
- `docs/internal/storage.md` only if storage contracts change
- `docs/internal/workspace.md`
- relevant maintained examples under `examples/`

Docs should explain:

- `session.previous_session_id`;
- `--previous-session-id` if added;
- `narratio.previous_session.artifact.<artifact_name>`;
- `previous/` workspace directory;
- S3-authoritative previous-session behavior;
- required vs optional previous-session artifact handling;
- when to run `narratio run-stage --force prepare`.

Do not document future `--local` support as current behavior.

## Future Work

These items are explicitly out of scope for the initial implementation.

### `--local` previous-session source

A future flag may allow `prepare` to copy previous-session artifacts from a local previous session workspace instead of S3.

Possible future command shape:

```bash
narratio run-stage --force prepare --local
```

or a more specific flag such as:

```bash
narratio run-stage --force prepare --previous-source local
```

Do not implement this now.

### `narratio run --restore`

A future flag may run `narratio restore` before starting the regular pipeline:

```bash
narratio run --restore --session-id 2026-04-11
```

Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation.

### Multi-previous-session support

A future design may support more than one previous/reference session.

Do not implement this now.

### Previous-session artifact version pinning

A future design may pin previous-session artifact inputs to a specific previous run ID or checksum.

Do not implement this now.

## Test Plan Summary

Run at least:

```bash
go test ./internal/config -v
go test ./internal/artifacts -v
go test ./internal/stage -run Prepare -v
go test ./internal/stage -run Analyze -v
go test ./internal/app -run TestExecute -v
go test ./...
```

Add focused tests for:

- session config and templating;
- previous-session source parsing;
- previous artifact requirement scanning;
- S3-backed previous artifact download;
- prepare skip/force semantics;
- manifest input provenance;
- resolver behavior;
- analyze integration;
- archive/restore `previous/` persistence;
- examples load/validate.

## Acceptance Criteria

The feature is complete when:

1. `session.yml` supports optional `previous_session_id`.
2. Session templating supports `previous_session_id`.
3. Scriptorium artifact inputs accept `narratio.previous_session.artifact.<artifact_name>`.
4. `prepare` scans enabled Scriptorium artifacts for previous-session artifact inputs.
5. `prepare` downloads referenced previous-session artifacts from S3 into `previous/`.
6. Required/optional behavior is correct.
7. `prepare` uses stage-level skip/force behavior and does not compare local and remote copies.
8. `analyze` resolves previous-session artifacts only from local prepared `previous/` state.
9. `archive` persists `previous/`.
10. `restore` restores `previous/`.
11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior.
12. Current-behavior docs and maintained examples are updated after implementation.
13. No storage adapter implementation infers campaign/session/root-prefix semantics.
14. No previous-session S3 logic is added to `analyze` or the Scriptorium adapter.