narratio/docs/roadmap/notarius-extract-stage.md

# Notarius Extraction Stage

## Status

Complete.

The completed implementation sequence is retained in
[implementation.md](implementation.md) as decision and delivery history.
Current behavior is documented in the canonical configuration, CLI,
operations, integration, and internal references linked from this roadmap.

## Purpose

Add a first-class Narratio `extract` stage that runs Notarius against the
session's final trimmed transcript and makes validated structured artifacts
available to later analysis and publish work.

Notarius remains responsible for its D&D extraction pipeline, prompts,
references, LLM profiles, retries, validation, normalization, and published
schemas. Narratio owns invocation, required-output policy, safe bundle
ingestion, artifact identity, manifest state, resume, and downstream
availability.

The integration must preserve Narratio's explicit stage model. It must not
turn Narratio into a generic workflow engine or reproduce Notarius's
configuration language.

## User Outcome

An operator can enable one Notarius pipeline for a Narratio campaign. During a
normal run, Narratio will:

1. produce the final trimmed Seriatim JSON transcript;
2. invoke Notarius once with that transcript;
3. discover the exact run bundle through Notarius's machine-readable receipt;
4. validate every output contract Narratio is configured to require;
5. promote the complete validated bundle into immutable session storage;
6. record exact artifact paths, checksums, contracts, and external provenance;
7. make configured lanes available as `narratio.extraction.<key>` sources; and
8. allow individual Scriptorium artifacts and publish rules to select those
   sources explicitly.

The maintained complete D&D example will require all ten lanes published by
Notarius's `dnd-session` pipeline. Ordinary deployments may configure a
narrower required set.

## Chosen Architecture

### First-Class Stage

Extraction is an independently observable and resumable pipeline stage. It is
not part of `analyze`, a Scriptorium artifact producer, or an implicit external
preprocessing requirement.

This boundary is required because Notarius work is expensive, produces
multiple durable outputs, has its own compatibility and diagnostic contracts,
and may be consumed by both `analyze` and `publish`.

### Canonical Order

The canonical stage order becomes:

```text
prepare -> transcribe -> merge -> polish -> normalize -> trim
        -> extract -> render -> analyze -> publish -> notify
```

`extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It
does not consume rendered Markdown.

Narratio currently invalidates succeeded stages by canonical downstream order.
Placing `extract` before `render` means forcing extraction may rerun the less
expensive deterministic render stage, while forcing render does not rerun the
more expensive Notarius pipeline. This is preferable to placing extraction
after render and does not require dependency-aware scheduling or a DAG.

Adding the stage must update every canonical-stage inventory, full-plan and
single-stage selection, prerequisite validation, downstream invalidation,
resume behavior, run manifests, CLI validation and help, and focused tests.

### Stage Boundary

The stage owns Narratio policy:

- resolve the final trimmed transcript through the manifest-aware artifact
  resolver;
- allocate run-local receipt, log, and output-root paths;
- build a transport-neutral Notarius request from resolved configuration;
- call the configured Notarius adapter once;
- enforce Narratio's configured required-output policy;
- validate selected payloads as regular, non-empty, syntactically valid JSON;
- promote the validated bundle to immutable session artifact storage;
- return explicit source IDs, checksums, contracts, and external provenance;
  and
- report an explicit skipped outcome when Notarius is disabled.

The stage must not construct subprocess arguments, guess Notarius filenames,
parse interactive output, decode D&D payload structures, or reproduce
Notarius pipeline configuration.

### Adapter Boundary

Add `internal/adapters/notarius` with a narrow runner interface, production
subprocess implementation, and small fake.

The request contains only:

- resolved executable path or name;
- absolute Notarius configuration path;
- pipeline ID;
- absolute transcript path;
- absolute output root;
- working directory;
- timeout; and
- stdout receipt and stderr log destinations.

The adapter owns:

- exact `notarius run ... --json` argument construction;
- stdout and stderr separation;
- context cancellation and timeout through Narratio's shared subprocess
  boundary;
- environment inheritance;
- exit-status handling;
- bounded receipt loading after exit status zero;
- tolerant decoding of supported `notarius.run-result.v1` documents;
- validation of required receipt fields;
- confinement of the receipt's absolute `output_directory` beneath the
  absolute output root Narratio supplied for this invocation;
- confinement of `index_file` beneath the receipt's absolute
  `output_directory`;
- tolerant decoding of the supported `index.json` contract;
- confinement of every index descriptor path beneath the bundle root; and
- bounded tolerant decoding of `rejected.json` and `warnings.json` into
  transport-neutral summaries without reading lane payload bodies; summaries
  persisted by Narratio contain structured scope, lane, and reason fields but
  not unbounded free-form external messages; and
- returning transport-neutral receipt, descriptor, diagnostic, and bundle
  information.

Only exit status zero permits receipt decoding. Unsupported schema versions,
malformed documents, missing required fields, absolute logical paths, path
escapes, symlinks at consumed paths, and incompatible structural metadata are
integration failures.

The production runner will not execute `notarius config validate` before every
session. Notarius run-time validation remains authoritative, while operators
may use the separate validation command as deployment preflight. A second
automatic subprocess can be added later only if operational evidence warrants
it.

The adapter does not write Narratio manifests, decide required lanes, choose
analysis inputs, or interpret D&D payloads.

## External Contract Baseline

The integration consumes the contracts documented by Notarius in
`../notarius/docs/consumers/dnd-pipeline.md` and its linked canonical
integration documents.

The initial compatibility baseline is:

- `notarius run <pipeline-id> --config ... --input ... --output-dir ... --json`;
- successful receipt schema `notarius.run-result.v1`;
- an absolute receipt `output_directory`;
- logical `index_file` discovery beneath that directory;
- the production JSON `index.json` descriptor model; and
- the exact media type and schema identity configured for each required lane.

Compatibility is decided from these published contracts, not by parsing
`notarius --version`. Unknown fields in supported receipt and index versions
are tolerated. Unsupported versions or incompatible required descriptors fail
before any artifact becomes current in Narratio.

## Configuration Contract

Add a strict optional `pipeline.notarius` section:

```yaml
notarius:
  enabled: true
  binary: notarius
  config_path: /absolute/path/to/notarius.yml
  pipeline_id: dnd-session
  timeout: 3h
  working_directory: /absolute/path/to/deployment
  outputs:
    npc_registry:
      lane_id: npc-registry
      media_type: application/json
      schema_id: <published-schema-id>
      schema_version: <published-schema-version>
```

Fields:

- `enabled` is an explicit opt-in and defaults to false;
- `binary` defaults to `notarius`;
- `config_path` is required when enabled;
- `pipeline_id` is required when enabled;
- `timeout` defaults to `3h` and must be positive;
- `working_directory` is optional and defaults to the directory containing
  `config_path`; and
- `outputs` maps stable Narratio extraction keys to required Notarius lane
  contracts.

All configured paths become absolute during configuration resolution. Notarius
reference paths continue to follow Notarius's configuration-relative rules,
while PromptKit profile paths remain relative to the chosen process working
directory where Notarius permits that behavior.

Each output entry contains:

- exact `lane_id`;
- exact `media_type`;
- exact `schema_id`;
- exact `schema_version`; and
- optional `module_key`.

The map key produces `narratio.extraction.<key>`. Keys use Narratio's existing
path-safe configured-artifact key grammar. Keys, normalized source IDs, and
lane IDs must be non-empty and unique. Extraction source IDs must not collide
with built-ins or configured Scriptorium sources.

Every configured output is required for extraction-stage success. Operators
who need only a subset configure only that subset. Narratio does not add a
second `required_lanes` list or a runtime lane-selection flag.

Narratio does not configure Notarius lane topology, references, prompts, LLM
profiles, model settings, concurrency, retry behavior, or session IDs. One
stage execution runs the configured Notarius pipeline as a unit. Narratio does
not pass Notarius's `--session-id` override.

## Stage Outcomes And Resume

### Explicit Self-Skip

Extend the stage-result contract with an explicit disposition whose zero value
remains successful for backward compatibility. A stage may return:

- succeeded; or
- skipped with a stable reason.

When Notarius is absent or disabled, `extract` returns skipped with reason
`notarius_disabled`, no outputs, and bounded metadata. The runner records a
skipped session-stage and run-stage outcome rather than a successful empty
stage. Skipped stages are reconsidered on later invocations, so subsequently
enabling Notarius causes extraction to run without requiring force.

A genuine self-skip clears any older outputs for that stage before persisting
the new skipped state. Downstream consumers cannot resolve artifacts retained
from an earlier extraction after the stage is disabled.

### Resume Validation

Add a small optional resume-validation interface implemented by `extract`.
Before skipping an already-succeeded extract stage, the runner asks it whether
the recorded result remains resumable.

The resume validator confirms:

- Notarius is still enabled;
- recorded configuration identity matches the current config path, pipeline
  ID, and output-contract map;
- the immutable bundle and canonical index still exist;
- every configured source is present in the succeeded extract record;
- lane files remain confined regular files;
- stored checksums still match; and
- stored descriptor contracts and external provenance remain compatible.

An ordinary contract mismatch returns a non-resumable decision with a bounded
reason. The runner marks the stage stale and executes it. An environmental
error that prevents making a safe decision returns an error and stops the run.

Narratio does not recursively interpret Notarius configuration, PromptKit
profiles, or reference files. Changes to those external inputs are therefore
not automatically detectable. Operator documentation must require
`--force extract` after changing them.

The configuration fingerprint covers the resolved binary, config path,
pipeline ID, timeout, working directory, and the deterministically sorted
output-contract map. It identifies Narratio's invocation contract, not the
transitive content of files owned by Notarius.

### Force And Invalidation

- Forcing `trim` or an earlier stage stales succeeded `extract` and all later
  stages.
- Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and
  `notify` under the existing canonical-order rule.
- Forcing `render` does not stale `extract` because extraction precedes it.
- A failed, skipped, stale, or interrupted extract stage never supplies current
  extraction sources.

## Bundle Storage And Commit

### Run-Local Execution

Notarius runs against a run-local output root beneath the Narratio run's
`extract` directory. Receipt bytes and stderr remain run-local diagnostics.
Failed and malformed bundles remain outside durable artifact storage for
inspection and never become current merely because files exist.

### Immutable Durable Bundles

After complete validation, promote the exact Notarius bundle into:

```text
artifacts/notarius/<narratio-run-id>/
```

The destination is unique and must not already exist. Promotion uses a sibling
temporary directory on the same filesystem, recursively copies only regular
files and directories, rejects symlinks and special files, preserves relative
layout, and renames the completed temporary tree into place.

The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`,
`warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as
`chunk-map.json` and `evidence-context.json`. Unknown regular files may be
preserved because the complete bundle is provenance, but no unknown file is
registered as a stable Narratio source.

There is no mutable filesystem `current` directory or symlink. The atomically
saved Narratio session manifest selects the current successful bundle. Older
successful bundles remain immutable until explicit cleanup policy removes
them.

The raw Notarius receipt is retained without rewriting its original
`output_directory`. Narratio's manifest records promoted durable paths.

## Artifact And Manifest Model

### Explicit Source Identity

Extend `artifacts.Ref` with an optional explicit `SourceID`. The application
runner prefers that value and retains existing stage-specific inference only
as a backward-compatible fallback. `extract` must not require another
stage-name special case in output mapping.

Add neutral optional artifact metadata models under `internal/artifactmodel`:

- contract metadata: media type, schema ID, schema version, and optional
  module key; and
- external provenance: system, external run ID, pipeline ID, and external
  artifact ID.

Both `artifacts.Ref` and `manifest.ArtifactRecord` carry these nested models.
Existing manifests remain readable because the fields are optional and use
`omitempty` encoding.

For a Notarius lane, external provenance uses:

- system `notarius`;
- receipt run ID;
- receipt pipeline ID; and
- lane ID as the external artifact ID.

### Registered Sources

For each configured output, `extract` records one artifact with:

- source ID `narratio.extraction.<key>`;
- durable lane path discovered through the promoted canonical index;
- producer stage and Narratio run ID;
- SHA-256 checksum;
- configured and observed contract metadata; and
- Notarius external provenance.

The canonical promoted `index.json` is also a stage output with kind
`notarius_index`, but it is not a selectable extraction source. Extract-stage
metadata records the durable bundle root, receipt path, Notarius validation
status, counts, warnings/rejections paths and summaries, the producing Narratio
run ID, and the normalized configured-contract fingerprint used by resume
validation.

The producing Narratio run ID belongs to the successful extract result. It is
not compared with the session manifest's top-level `run_id`, which advances on
later invocations even when extraction is validly resumed. Resume and catalog
checks instead require the extract outputs, immutable bundle path, and stored
extract-stage producer identity to agree with one another.

## Artifact Catalog And Resolution

Add extraction as a first-class artifact-policy and runtime-catalog family:

- source kind `extraction`;
- canonical prefix `narratio.extraction.`;
- registration from `pipeline.notarius.outputs`; and
- manifest-backed availability from the current successful `extract` record.

Extraction availability is never inferred by scanning
`artifacts/notarius/`. A source is available only when:

- it is declared in current configuration;
- the session manifest records `extract` as succeeded and not stale;
- the exact matching source output is present;
- its durable path is confined and valid;
- its checksum matches; and
- its recorded contract and external provenance are compatible.

The catalog should expose one shared registration and hydration path used by
both `analyze` and `publish`. Avoid parallel extract-specific resolution logic
inside each stage.

## Analyze Integration

An enabled Scriptorium artifact may select an extraction source through the
existing input contract:

```yaml
inputs:
  npcs:
    source: narratio.extraction.npc_registry
    required: true
```

Required missing extraction inputs fail with guidance to enable/configure or
rerun `extract`. Optional missing inputs follow the existing Scriptorium input
contract.

Narratio never injects every extraction output into every analysis. Each
Scriptorium artifact chooses the smallest useful set. This limits context
size, cost, and the risk of treating derived claims as transcript authority.
Analysis prompts should continue to treat the transcript as authoritative and
Notarius artifacts as structured, cited, derived evidence.

The existing `--artifacts` selection remains scoped to Scriptorium artifacts.
It does not select Notarius lanes or partially run the Notarius pipeline.

## Publish Integration

Publish source validation and resolution accept configured
`narratio.extraction.<key>` sources through the shared runtime artifact
catalog. Operators may publish individual structured lanes without manually
copying files.

The Notarius bundle is not automatically published wholesale. Bundle files or
lanes are published only through explicit configured publish rules. Existing
publish locking, destination safety, commit ordering, and required/unselected
artifact behavior remain unchanged.

## Failure And Diagnostic Semantics

The stage fails before invoking Notarius when enabled configuration or the
final trimmed transcript is invalid.

The stage fails after invocation for:

- cancellation or timeout;
- nonzero process exit;
- malformed, oversized, or unsupported receipt data;
- malformed or unsupported index data;
- unsafe receipt, descriptor, or filesystem paths;
- symlinks or special files in the promoted bundle;
- mismatched receipt pipeline identity;
- missing configured lanes;
- duplicate lane descriptors;
- rejected required lanes;
- incompatible media type, schema identity/version, or module key;
- empty or syntactically invalid required JSON payloads;
- checksum or promotion failure; or
- manifest persistence failure.

Process success alone does not establish consumer success. Notarius may exit
zero while omitting or rejecting a lane, and Narratio's configured required
set remains authoritative.

Failure retains bounded receipt and stderr diagnostics plus the run-local
bundle where available. Durable artifact storage and extraction source records
are updated only after complete validation and promotion. A promoted unique
bundle whose later manifest save fails is unreferenced and may be reclaimed by
explicit cleanup; it is never inferred as current.

## Security And Privacy

Transcripts, Notarius lanes, evidence context, manifests, receipts, warnings,
rejections, debug data, and logs are private campaign material.

- Secrets do not appear in command arguments, generated configuration,
  artifact metadata, logs, examples, or documentation.
- Credentials continue to enter through Notarius and PromptKit's documented
  environment or secret mechanisms.
- Receipt and index paths are untrusted external input until confined.
- Recursive promotion never follows symlinks or copies special files.
- Diagnostics remain bounded and do not echo payload bodies.
- Automatic cleanup follows Narratio's existing post-publish gates and path
  safety rules; it does not silently remove immutable extraction bundles
  outside an explicit covered policy.

## Maintained Complete D&D Example

Add a maintained example that enables Notarius's complete `dnd-session`
pipeline and maps these required lanes:

| Extraction key | Notarius lane ID |
| --- | --- |
| `item_registry` | `item-registry` |
| `npc_registry` | `npc-registry` |
| `location_registry` | `location-registry` |
| `scene_descriptions` | `scene-descriptions` |
| `item_occurrences` | `item-occurrences` |
| `spells` | `spells` |
| `combat_turns` | `combat-turns` |
| `npc_occurrences` | `npc-occurrences` |
| `location_occurrences` | `location-occurrences` |
| `enemy_events` | `enemy-events` |

The example obtains exact media types and schema identities from Notarius's
published contracts at implementation time. It demonstrates at least one
Scriptorium artifact consuming a small, purpose-specific subset of extraction
sources. It uses only placeholders and repository-relative example paths,
contains no credentials, and passes maintained example validation.

## Documentation Deliverables

When implementation lands, update current-behavior documentation in the same
change:

- add `docs/integrations/notarius.md` for the consumed CLI, receipt, index, and
  compatibility contract, linking to Notarius's canonical documentation;
- add `docs/internal/stage-extract.md` for stage flow, collaborators, state,
  failures, resume validation, and focused tests;
- update `docs/internal/overview.md`, `docs/internal/adapters.md`,
  `docs/internal/artifacts.md`, `docs/internal/manifest.md`, and
  `docs/internal/workspace.md` within their canonical scopes;
- update `docs/policy/architecture.md` for the Notarius boundary and explicit
  skipped/resume-validation contracts;
- update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
  `docs/troubleshooting.md`, `README.md`, and maintained examples only within
  their canonical scopes; and
- update `docs/development.md` only if its contributor routing changes.

Outside this roadmap, do not describe the feature as implemented until its
code and documentation are complete.

## Testing Expectations

Tests should protect contracts and meaningful risks rather than private helper
structure. The implementation plan assigns detailed ownership, with coverage
for:

- strict configuration, defaults, normalization, cross-source validation, and
  maintained examples;
- stage result dispositions and clearing of self-skipped outputs;
- backward-compatible artifact metadata serialization;
- exact adapter arguments, streams, cancellation, timeout, exit behavior,
  receipt/index compatibility, and every path-confinement boundary;
- recursive promotion safety, atomic visibility, cleanup on failure, symlink
  rejection, and immutable destination behavior;
- required-lane policy, descriptor compatibility, JSON syntax, checksums, and
  provenance;
- canonical order, single-stage selection, force, staleness, resume
  validation, and enable-after-skip behavior;
- manifest-backed extraction catalog resolution for required, optional,
  missing, stale, skipped, incompatible, and tampered artifacts;
- analyze and publish integration without automatic lane injection; and
- representative assembled execution with a fake Notarius runner and no live
  LLM, credentials, or external subprocess in the ordinary test suite.

Repository-wide tests, vet, build, and maintained-example validation are
required after focused tests pass.

## Acceptance Criteria

- `extract` is a first-class stage between `trim` and `render` everywhere
  Narratio models stage order and lifecycle.
- Narratio invokes Notarius only through a narrow tested adapter.
- Extraction consumes the manifest-resolved final trimmed Seriatim JSON.
- Disabled extraction is recorded as skipped and runs normally if later
  enabled.
- Successful resume requires valid manifest-recorded immutable outputs rather
  than filesystem presence alone.
- The complete bundle is promoted to a unique immutable directory without
  following symlinks or exposing a partial destination.
- Every configured lane is discovered by lane ID, contract-checked, checksummed,
  and recorded with explicit source identity and external provenance.
- Analysis and publish resolve extraction sources only from a current
  successful extract manifest record.
- Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered
  output never becomes a current input.
- The complete D&D example maps all ten current lanes and demonstrates curated
  analysis inputs.
- Tests remain deterministic, offline, and independent of real Notarius,
  PromptKit, LLM providers, and credentials.
- Current-behavior documentation is updated only as implementation becomes
  complete.

## Non-Goals

- Reimplementing Notarius extraction, configuration, schemas, prompts,
  references, retries, profiles, validation, or lane orchestration.
- Supporting arbitrary extractor programs or multiple Notarius pipelines in
  one Narratio run.
- Turning canonical stage execution into a DAG or generic workflow engine.
- Folding extraction into `analyze` or Scriptorium.
- Automatically injecting all structured outputs into every prompt.
- Selecting Notarius lanes through Narratio's `--artifacts` flag.
- Decoding D&D payload bodies in the generic adapter or stage.
- Automatically detecting changes throughout Notarius's referenced config,
  PromptKit profile, and campaign-reference closure.
- Supporting previous-session extraction sources in the initial feature.
- Automatically publishing the complete Notarius bundle.
- Requiring live Notarius, PromptKit, an LLM provider, or external services in
  the ordinary test suite.