narratio/docs/roadmap/notarius-extract-stage.md

# Notarius Extraction Stage

## Status

Proposed.

## Purpose

Add a first-class Narratio `extract` stage that runs Notarius against the
session's final trimmed transcript, validates and collects the resulting
structured D&D artifacts, and registers those artifacts for later use by the
`analyze` and `publish` stages.

This feature should integrate Notarius through Narratio's existing stage,
adapter, manifest, workspace, and artifact-catalog boundaries. It must not turn
Narratio into a generic workflow engine or a second configuration language for
Notarius pipelines.

## User Outcome

An operator can enable one configured Notarius pipeline for a Narratio
campaign. During a normal run, Narratio will:

1. finish producing the session transcript tiers;
2. invoke Notarius once with the final trimmed Seriatim JSON transcript;
3. collect and validate the configured structured artifact lanes;
4. record their exact files and provenance in the Narratio manifest; and
5. make those artifacts selectable as inputs to Scriptorium artifacts in the
   later `analyze` stage.

The maintained D&D example should demonstrate all ten lanes emitted by
Notarius's complete `dnd-session` pipeline.

## Target Stage Architecture

### Canonical Order

The canonical stage order becomes:

```text
prepare -> transcribe -> merge -> polish -> normalize -> trim -> render
        -> extract -> analyze -> publish -> notify
```

`extract` is deliberately after all transcript-producing stages and before
analysis. Its source document is the manifest-resolved
`narratio.transcript.final_trimmed` artifact, normally
`transcripts/final.trimmed.json`. It does not consume rendered Markdown.

Adding the stage must update full-plan construction, explicit stage selection,
downstream invalidation, prerequisite checks, resume behavior, run manifests,
CLI stage validation and help, and every canonical-stage inventory. Forcing an
upstream transcript stage must stale a previously successful `extract` stage
and its downstream stages. Forcing `extract` must stale `analyze`, `publish`,
and `notify` according to existing rules.

### Stage Boundary

The stage owns Narratio policy and state transitions:

- resolve the final trimmed transcript through the runtime artifact catalog;
- build a Narratio-level Notarius request from validated configuration and
  run-local paths;
- call a narrow Notarius adapter;
- apply the configured required-output policy;
- materialize the validated bundle into its canonical session location;
- return manifest-ready artifact references and bounded metadata; and
- fail without marking the stage successful when any required contract or
  materialization step fails.

The stage must not construct subprocess arguments, infer Notarius output
filenames, parse provider logs, or decode individual D&D payload bodies.

### Adapter Boundary

Add a dedicated Notarius adapter package with a small interface, production
subprocess implementation, and test fake. Its request should contain only the
resolved Notarius binary, configuration path, pipeline ID, transcript path,
output root, working directory, timeout, and process-log destinations needed
for one run.

The adapter owns:

- optional `notarius config validate` preflight for the configured pipeline;
- exact `notarius run ... --json` argument construction;
- stdout and stderr separation;
- context cancellation and timeout propagation through Narratio's shared
  subprocess boundary;
- exit-status handling;
- decoding the `notarius.run-result.v1` success receipt;
- receipt and index path-confinement checks;
- decoding `index.json` and resolving descriptor paths safely beneath the
  reported output directory; and
- returning a transport-neutral result containing the bundle location,
  receipt summary, lane descriptors, pipeline-wide descriptors, warnings and
  rejection locations, and diagnostic log paths.

Only exit status zero permits receipt decoding. Receipt, index, or descriptor
paths that are absolute where a logical relative path is required, or that
escape their owning root, are integration failures. Unknown fields in a
supported receipt or index schema should be tolerated. Unsupported schema
versions and incompatible descriptor metadata should fail clearly.

The adapter must not write Narratio manifests, choose required lanes, decide
analysis inputs, or contain D&D domain logic.

## Configuration Contract

Add a strict optional `pipeline.notarius` configuration section. Omission or
`enabled: false` keeps the current workflow usable and causes `extract` to
self-skip without outputs.

The section should provide:

- `enabled`: explicit opt-in;
- `binary`: Notarius executable, defaulting to `notarius`;
- `config_path`: required when enabled;
- `pipeline_id`: required when enabled;
- `timeout`: a positive stage timeout with a documented default;
- `working_directory`: optional explicit subprocess working directory,
  defaulting to the directory containing `config_path`; and
- an `outputs` map defining the Notarius lane artifacts Narratio promises to
  collect.

Each output-map key is a stable Narratio extraction key. Each value must define:

- the exact Notarius `lane_id`;
- the expected `media_type`;
- the expected `schema_id`;
- the expected `schema_version`; and
- optionally an expected `module_key` when the operator needs to constrain the
  producing module as part of compatibility.

Narratio derives the downstream source ID
`narratio.extraction.<output-key>` from the map key. Keys and lane IDs must be
non-empty, unique after normalization, path-safe under the existing artifact
policy, and collision-free with built-in and configured artifact identities.
Every configured output is required: a successful Notarius process that omits
one, rejects it, or reports incompatible descriptor metadata fails the
`extract` stage.

This explicit map keeps Narratio's consumer contract stable when a Notarius
lane ID or schema changes and avoids hard-coding the current D&D family into a
generic adapter. It also replaces a separate `required_lanes` list, which would
duplicate configuration.

Narratio should not reproduce Notarius lane selection, references, LLM
profiles, model settings, retries, concurrency, or prompt configuration. Those
remain in the referenced Notarius configuration. Narratio should not expose a
runtime lane-selection flag for `extract`; one stage invocation runs the
configured Notarius pipeline as a unit.

All configured paths should become absolute during Narratio configuration
resolution. The deterministic default working directory allows a Notarius
profile path relative to that directory, but operator documentation should
still recommend absolute deployment paths where practical. Notarius reference
paths continue to follow Notarius's own configuration-relative rules.

## Output And Artifact Model

### Canonical Bundle

Run Notarius against a run-local output root. After all configured descriptors
are validated, materialize the contents of the exact run-specific Notarius
bundle into a fixed canonical session directory:

```text
artifacts/notarius/
```

Preserve its relative layout, including `index.json`, `manifest.json`,
`rejected.json`, `warnings.json`, `lanes/`, and any indexed `chunk-map.json` or
`evidence-context.json`. Materialize the complete directory as one narrow,
transactional replacement so a failed or interrupted rerun cannot mix files
from different Notarius runs.

The raw subprocess receipt and stderr log belong in the run-local `extract`
report and log directories. The raw receipt identifies the original run-local
Notarius bundle and must not be rewritten to pretend that the canonical copy
was its original `output_directory`. Narratio's manifest is the durable ledger
for the canonical materialized paths.

### Registered Artifact Sources

For each configured output, locate the lane through the canonical copy of
`index.json` and record a manifest artifact with:

- source ID `narratio.extraction.<output-key>`;
- canonical lane-file path discovered from the index;
- producer stage and Narratio run ID;
- checksum;
- Notarius lane ID; and
- descriptor media type, schema identity/version, and module key when present.

If the current manifest model cannot carry descriptor compatibility metadata,
extend its artifact metadata in a backward-tolerant way rather than encoding
that information in filenames or source IDs.

Also record the canonical Notarius index as a stage output or stage metadata so
operators can discover the complete bundle, including non-lane artifacts. The
configured lane sources are the stable interface for analysis; the index and
bundle remain the provenance and inspection interface.

## Analysis And Publish Integration

Extend the runtime artifact catalog and configured Scriptorium input validation
so an enabled analysis artifact can declare, for example:

```yaml
inputs:
  npc_registry:
    source: narratio.extraction.npc_registry
```

Resolution must remain manifest-first and verify that the recorded artifact
was produced by a successful current `extract` stage. A required extraction
source that is unavailable must fail analysis with guidance to configure or
rerun `extract`; an optional source may be omitted according to the existing
Scriptorium input contract.

Publish source resolution should accept configured
`narratio.extraction.<output-key>` sources through the same artifact catalog so
operators may publish selected structured artifacts without manually copying
paths. The existing `--artifacts` flag remains scoped to Scriptorium artifact
selection and must not partially execute the Notarius pipeline.

No current-session analysis artifact should consume an incidental file from a
failed, stale, skipped, or superseded extraction run.

## Failure, Skip, Resume, And Diagnostics

- Missing or invalid enabled Notarius configuration fails configuration
  validation before stage execution where statically discoverable.
- A disabled or absent Notarius configuration makes `extract` skip with clear
  stage metadata and no new outputs.
- A missing or invalid final trimmed transcript fails `extract` before starting
  Notarius.
- Preflight failure, nonzero Notarius exit, cancellation, timeout, malformed or
  unsupported receipt/index data, unsafe paths, incompatible descriptors,
  rejected required outputs, or missing configured lanes fails the entire
  stage.
- Process success does not override Narratio's required-output policy.
- A failed run retains bounded run-local receipt bytes, stderr, and the
  unpublished Notarius bundle for diagnosis, subject to Narratio's existing
  sensitive-data and cleanup policies.
- The canonical bundle and manifest artifacts are updated only after complete
  validation and materialization.
- Resume skips a succeeded, non-stale `extract` stage only when its
  manifest-recorded canonical index and configured lane outputs still validate.
- Force and staleness behavior follows the ordinary stage contract; it must not
  depend on merely finding `artifacts/notarius/` on disk.

Transcripts, Notarius outputs, evidence context, manifests, receipts, and logs
are private campaign material. Subprocess arguments and manifest metadata must
not contain secrets. Credentials remain in the environment or in mechanisms
owned by Notarius and PromptKit.

## Maintained D&D Example

Add or update a Narratio example that enables Notarius's complete
`dnd-session` pipeline and maps these ten required lanes to stable extraction
keys:

| Output key | Notarius lane ID |
| --- | --- |
| `item_registry` | `item-registry` |
| `npc_registry` | `npc-registry` |
| `location_registry` | `location-registry` |
| `scene_descriptions` | `scene-descriptions` |
| `item_occurrences` | `item-occurrences` |
| `spells` | `spells` |
| `combat_turns` | `combat-turns` |
| `npc_occurrences` | `npc-occurrences` |
| `location_occurrences` | `location-occurrences` |
| `enemy_events` | `enemy-events` |

The example must include each lane's current media type and schema identity
from Notarius's published contracts. It should also demonstrate at least one
Scriptorium analysis artifact consuming one or more
`narratio.extraction.*` sources. The example must use placeholders and relative
paths suitable for the example tree, contain no credentials, and pass the
repository's configuration validation tests.

## Compatibility Policy

The initial integration baseline is the public subprocess contract available
in Notarius v0.3.0:

- successful JSON receipt schema `notarius.run-result.v1`;
- production JSON bundle discovery through `index.json`; and
- the schema IDs and versions explicitly configured for required lanes.

Runtime compatibility should be decided from those published contracts, not
from textual parsing of `notarius --version`. New optional receipt or index
fields must not break Narratio. An unsupported receipt version or lane schema
must fail before the artifact is registered for analysis.

## Documentation Deliverables When Implemented

Update current-behavior documentation in the same change that implements the
feature:

- add `docs/integrations/notarius.md` for the external CLI, receipt, bundle,
  and adapter contract, linking to Notarius's canonical documentation;
- add `docs/internal/stage-extract.md` for stage inputs, outputs, collaborators,
  state transitions, failures, and focused tests;
- update `docs/internal/adapters.md`, `docs/internal/artifacts.md`,
  `docs/internal/manifest.md`, and the internal stage inventory;
- update `docs/architecture.md` to list Notarius among isolated external
  systems and preserve the adapter/stage boundary;
- update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
  `docs/troubleshooting.md`, `README.md`, and maintained examples only to the
  extent their canonical scopes require; and
- add the missing `docs/development.md` required by the repository's
  documentation policy, or track that repository-wide documentation gap in a
  separate roadmap if it is intentionally outside the implementation scope.

Outside this roadmap, do not describe `extract`, Notarius configuration, or
`narratio.extraction.*` sources as implemented until the code exists.

## Testing And Validation Expectations

Implementation should provide focused tests for:

- strict configuration decoding, defaults, required fields, path resolution,
  output-map validation, normalized-key collisions, and example loading;
- exact stage order, selection, downstream staleness, resume, force, and
  prerequisite behavior;
- adapter command construction, deterministic working directory, environment
  inheritance, stdout/stderr separation, cancellation, timeout, and nonzero
  exits;
- supported and unsupported receipt versions, unknown optional fields,
  malformed receipts, index decoding, and path escapes at every boundary;
- descriptor lookup by lane ID rather than filename, expected metadata checks,
  missing/rejected configured lanes, and tolerated unconfigured lanes;
- run-local execution, transactional canonical-bundle replacement, checksums,
  failed-run preservation, and manifest recording;
- artifact-catalog resolution from `narratio.extraction.*` into analysis and
  publish, including required, optional, missing, stale, and skipped cases; and
- end-to-end stage execution with a fake Notarius adapter, without live LLM or
  external subprocess requirements in the ordinary test suite.

Run the repository-wide Go tests, vet, build, and maintained example validation
after focused tests pass.

## Acceptance Criteria

- `extract` is a first-class transactional stage between `render` and
  `analyze` everywhere Narratio models stage order or state.
- Narratio invokes Notarius only through a narrow, tested adapter.
- The stage consumes the manifest-resolved final trimmed Seriatim transcript.
- The Notarius configuration remains owned by Notarius; Narratio configures
  only invocation and its downstream consumer contract.
- Every configured output is discovered through the receipt and `index.json`,
  contract-checked, materialized transactionally, and recorded with a stable
  `narratio.extraction.*` source ID.
- The complete D&D example maps all ten current lanes and passes strict config
  validation.
- Analysis can consume extraction sources through the existing artifact input
  model, and publish can select them through the artifact catalog.
- Failed, partial, rejected, unsafe, stale, or incompatible output never becomes
  a current analysis input.
- Resume and force behavior remains manifest-driven.
- Documentation accurately describes the implemented stage, adapter,
  configuration, operations, and compatibility boundary without duplicating
  Notarius's canonical schemas.

## Non-Goals

- Reimplementing Notarius extraction, prompts, schemas, references, retries,
  profiles, or lane orchestration in Narratio.
- Allowing one Narratio run to invoke arbitrary extractor programs or multiple
  Notarius pipelines.
- Making `extract` a configurable DAG or folding it into the Scriptorium
  `analyze` stage.
- Partially selecting Notarius lanes through Narratio's `--artifacts` flag.
- Decoding D&D payload bodies in the generic Notarius adapter.
- Supporting previous-session extraction artifacts in the initial feature.
- Requiring live Notarius, PromptKit, an LLM provider, or external services in
  the ordinary unit test suite.