narratio/docs/internal/stage-extract.md

# Internal: Extract Stage

## Responsibility

`extract` runs after `render` and before `analyze`. It converts the canonical
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
An omitted or disabled Notarius section makes the stage explicitly self-skip
with reason `notarius_disabled`, no outputs, and no Notarius runner.

The external protocol is documented in the
[Notarius integration contract](../integrations/notarius.md). Configuration
fields belong in [Configuration](../config.md), and physical paths and force
procedures belong in [Operations](../operations.md).

## Lifecycle

`internal/stage/extract.go`:

1. resolves the final trimmed transcript from the shared artifact catalog;
2. resolves every configured prepared reference through the shared
   manifest-authoritative identity resolver before creating run-local output;
3. streams each verified reference into an invocation-local snapshot and
   rejects any source change observed while copying;
4. fingerprints the byte- and provenance-bearing Notarius invocation evidence,
   including sorted reference identities;
5. creates a run-local staging directory and invokes the injected
   `notarius.Runner`;
6. revalidates the reference snapshots, then validates the v2 successful
   receipt, confined index, management documents, configured required lane
   descriptors, validation summaries, and regular payload files;
7. atomically promotes the complete bundle to its immutable durable location;
8. records one non-selectable `notarius_index` output and one selectable
   `notarius_lane` output per configured lane; and
9. registers each lane as `narratio.extraction.<output_key>` for downstream
   Scriptorium and publish resolution.

Lane records retain checksum, contract, producer run ID, and Notarius system,
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
root, receipt, diagnostic paths, rejection/warning summaries, producing
Narratio run ID, the resolved trimmed-input identity, and invocation
fingerprint. The input identity binds the exact transcript bytes, canonical
source ID, producer stage/output/run identity, and resolution provenance.
Reference metadata contains only selector, source ID, canonical session-relative
path, checksum, and size; adapter requests receive selector and absolute
invocation-local snapshot path, never payload contents. Snapshot bytes must
match the prepared identity both before and after Notarius runs, so a concurrent
prepared-file replacement cannot make recorded provenance describe different
bytes from those supplied to Notarius.
Validation completes before the single durable rename, so a rejected result
cannot expose a partial durable bundle.

Any executed extraction outcome that replaces a different effective outcome
marks succeeded analysis and delivery dependents stale. Render is an independent
sibling and remains current. Repeating the same disabled self-skip with no
outputs is stable and does not repeatedly invalidate dependent stages.

## Resume Validation

Before the focused validator runs, the application compares extract's versioned
semantic fingerprint. It covers enablement, Notarius pipeline identity, sorted
reference selector/source mappings, sorted declared output contracts, and each
canonical `narratio.extraction.<key>` output identity. It excludes executable,
timeout, working directory, config path, and private Notarius config contents.

`internal/stage/extract_resume.go` then permits a skip only when the existing
stage record still matches the current byte- and provenance-bearing invocation
evidence. That evidence covers the current direct trimmed-transcript identity,
sorted prepared-reference identities, pipeline identity, and configured output
contracts. The same reference helper and transcript identity are resolved again
for artifact evidence, so changing current transcript bytes, reference bytes,
or producer identity makes the prior extraction obsolete. Operational runner
settings do not invalidate otherwise current durable evidence.

A valid prepared-reference change makes extraction non-resumable. Missing,
unsafe, or checksum-inconsistent prepared evidence is a hard validation error
with prepare-force guidance because an immediate extract rerun cannot succeed.

The validator then checks the producing run identity, canonical immutable
bundle root, path confinement and absence of symlink components, receipt
identity, exactly one canonical index, the exact configured source set,
contracts and provenance, regular-file status, and stored checksums. Missing or
obsolete results are non-resumable and run again; unsafe filesystem conditions
return an error rather than silently accepting or replacing data.

Neither contract can observe files imported by Notarius configuration, profile
contents, prompt/module definitions, or other transitive inputs. Operators must
force extraction after changing any such private input behind a stable
identifier.

## Failure Behavior

Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
index compatibility, inconsistent warning or diagnostic envelopes,
required-lane rejection or incomplete validation, payload inspection,
checksum, or installation errors fail the stage through ordinary manifest
transition handling.
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
not given selectable extraction sources unless the complete configured result
has passed validation and installation.

When a replacement attempt begins, the current session-stage record no longer
advertises payload from the previous success. A failed replacement therefore
has no current outputs, logs, generated configuration references, or metadata,
while the earlier immutable promoted bundle remains available for audit and
recovery.

## Implementation And Focused Tests

- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
  `internal/stage/extract_resume.go`,
  `internal/stage/extract_test.go`,
  `internal/stage/semantic_contracts_delivery.go`
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
  `internal/adapters/notarius/subprocess_test.go`
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
  `internal/artifacts/extraction_catalog_test.go`
- Composition and downstream behavior: `internal/app/runner_test.go`,
  `internal/stage/analyze_test.go`, `internal/stage/publish_test.go`