narratio/docs/internal/stage-extract.md

# Internal: Extract Stage

## Responsibility

`extract` runs after `render` and before `analyze`. It converts the canonical
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
An omitted or disabled Notarius section makes the stage explicitly self-skip
with reason `notarius_disabled`, no outputs, and no Notarius runner.

The external protocol is documented in the
[Notarius integration contract](../integrations/notarius.md). Configuration
fields belong in [Configuration](../config.md), and physical paths and force
procedures belong in [Operations](../operations.md).

## Lifecycle

`internal/stage/extract.go`:

1. resolves the final trimmed transcript from the shared artifact catalog;
2. resolves every configured prepared reference through the shared
   manifest-authoritative identity resolver before creating run-local output;
3. streams each verified reference into an invocation-local snapshot and
   rejects any source change observed while copying;
4. fingerprints the Notarius invocation contract, including sorted reference
   identities;
5. creates a run-local staging directory and invokes the injected
   `notarius.Runner`;
6. revalidates the reference snapshots, then validates the v2 successful
   receipt, confined index, management documents, configured required lane
   descriptors, validation summaries, and regular payload files;
7. atomically promotes the complete bundle to its immutable durable location;
8. records one non-selectable `notarius_index` output and one selectable
   `notarius_lane` output per configured lane; and
9. registers each lane as `narratio.extraction.<output_key>` for downstream
   Scriptorium and publish resolution.

Lane records retain checksum, contract, producer run ID, and Notarius system,
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
root, receipt, diagnostic paths, rejection/warning summaries, producing
Narratio run ID, the resolved trimmed-input identity, and invocation
fingerprint. The input identity binds the exact transcript bytes, canonical
source ID, producer stage/output/run identity, and resolution provenance.
Reference metadata contains only selector, source ID, canonical session-relative
path, checksum, and size; adapter requests receive selector and absolute
invocation-local snapshot path, never payload contents. Snapshot bytes must
match the prepared identity both before and after Notarius runs, so a concurrent
prepared-file replacement cannot make recorded provenance describe different
bytes from those supplied to Notarius.
Validation completes before
promotion, so a rejected result cannot expose a partial durable bundle.

Any executed extraction outcome that replaces a different effective outcome
marks succeeded analysis and delivery dependents stale. Render is an independent
sibling and remains current. Repeating the same disabled self-skip with no
outputs is stable and does not repeatedly invalidate dependent stages.

## Resume Validation

`internal/stage/extract_resume.go` permits a skip only when the existing stage
record succeeded and still matches the current invocation fingerprint. The
fingerprint covers the resolved executable and config paths, pipeline ID,
timeout, working directory, sorted configured output contracts, the current
direct trimmed-transcript identity, and sorted prepared-reference identities.
The same reference helper and transcript identity are resolved again for
artifact evidence, so changing the current transcript bytes or producer
identity makes the prior extraction obsolete.

A valid prepared-reference change makes extraction non-resumable. Missing,
unsafe, or checksum-inconsistent prepared evidence is a hard validation error
with prepare-force guidance because an immediate extract rerun cannot succeed.

The validator then checks the producing run identity, canonical immutable
bundle root, path confinement and absence of symlink components, receipt
identity, exactly one canonical index, the exact configured source set,
contracts and provenance, regular-file status, and stored checksums. Missing or
obsolete results are non-resumable and run again; unsafe filesystem conditions
return an error rather than silently accepting or replacing data.

The fingerprint cannot observe files imported by Notarius configuration,
profile contents, prompt/module definitions, or other transitive inputs.
Operators must force extraction after changing any such input.

## Failure Behavior

Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
index compatibility, inconsistent warning or diagnostic envelopes,
required-lane rejection or incomplete validation, payload inspection,
checksum, or promotion errors fail the stage through ordinary manifest
transition handling.
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
not given selectable extraction sources unless the complete configured result
has passed validation and promotion.

When a replacement attempt begins, the current session-stage record no longer
advertises payload from the previous success. A failed replacement therefore
has no current outputs, logs, generated configuration references, or metadata,
while the earlier invocation manifest and immutable promoted bundle remain
available for audit and recovery.

## Implementation And Focused Tests

- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
  `internal/stage/extract_resume.go`,
  `internal/stage/extract_test.go`
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
  `internal/adapters/notarius/subprocess_test.go`
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
  `internal/artifacts/extraction_catalog_test.go`
- Composition and downstream behavior: `internal/app/runner_test.go`,
  `internal/stage/analyze_test.go`, `internal/stage/publish_test.go`