narratio/docs/internal/stage-extract.md

# Internal: Extract Stage

## Responsibility

`extract` runs after `render` and before `analyze`. It converts the canonical
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
An omitted or disabled Notarius section makes the stage explicitly self-skip
with reason `notarius_disabled`, no outputs, and no Notarius runner.

The external protocol is documented in the
[Notarius integration contract](../integrations/notarius.md). Configuration
fields belong in [Configuration](../config.md), and physical paths and force
procedures belong in [Operations](../operations.md).

## Lifecycle

`internal/stage/extract.go`:

1. resolves the final trimmed transcript from the shared artifact catalog;
2. resolves every configured prepared reference through the shared
   manifest-authoritative identity resolver before creating run-local output;
3. resolves each reference to a stable content-addressed session snapshot,
   reusing verified bytes or reconstructing a missing/corrupt regular entry,
   and rejects any source change observed while copying;
4. fingerprints the byte- and provenance-bearing Notarius invocation evidence,
   including sorted reference identities and the explicit downstream
   configuration-overlay selection;
5. creates a run-local staging directory and invokes the injected
   `notarius.Runner`, carrying the optional already-resolved credential
   reference without exposing its value, the optional exact configuration
   overlay, plus effective resume and chunk-cache controls after force
   precedence;
6. revalidates the reference snapshots, then validates the v2 successful
   receipt, confined index, management documents, configured required lane
   descriptors, validation summaries, and regular payload files;
7. atomically promotes the complete bundle to its immutable durable location;
8. records one non-selectable `notarius_index` output and one selectable
   `notarius_lane` output per configured lane; and
9. registers each lane as `narratio.extraction.<output_key>` for downstream
   Scriptorium and publish resolution.

Lane records retain checksum, contract, producer run ID, and Notarius system,
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
root, receipt, diagnostic paths, rejection/warning summaries, producing
Narratio run ID, the resolved trimmed-input identity, and invocation
fingerprint. The input identity binds the exact transcript bytes, canonical
source ID, producer stage/output/run identity, and resolution provenance.
Reference metadata contains only selector, source ID, canonical session-relative
path, checksum, and size; adapter requests receive selector and absolute stable
snapshot path, never payload contents. Snapshot bytes must
match the prepared identity both before and after Notarius runs, so a concurrent
prepared-file replacement cannot make recorded provenance describe different
bytes from those supplied to Notarius.
Validation completes before the single durable rename, so a rejected result
cannot expose a partial durable bundle.

Snapshot destinations are owned by `internal/artifacts` under
`cache/notarius/references/<sha256>/<prepared-basename>`. They contain no run ID
or timestamp and can be reused across invocations. Identical bytes and basename
share a verified file; changed bytes select a different digest directory.
Unsafe path components and non-regular conflicts fail without recursive
deletion. Snapshot paths are delivery details and never become manifest output
evidence or prerequisites for reuse of an already complete extract bundle.

Any executed extraction outcome that replaces a different effective outcome
marks succeeded analysis and delivery dependents stale. Render is an independent
sibling and remains current. Repeating the same disabled self-skip with no
outputs is stable and does not repeatedly invalidate dependent stages.

## Resume Validation

Before the focused validator runs, the application compares extract's versioned
semantic fingerprint. It covers enablement, Notarius pipeline identity, sorted
reference selector/source mappings, sorted declared output contracts, and each
canonical `narratio.extraction.<key>` output identity. It also covers the
explicit Notarius configuration-overlay selection; omitted, empty, and exact
`default` selectors share the base-selection identity. It excludes executable,
timeout, working directory, config path, and private Notarius config contents.
Credential presence, reference name, and resolved value are likewise
operational-only and excluded. Ordinary command loading still resolves a
selected reference before resume evaluation, but a credential-only change does
not rerun a valid completed extraction or stale its dependents.

`internal/stage/extract_resume.go` then permits a skip only when the existing
stage record still matches the current byte- and provenance-bearing invocation
evidence. That evidence covers the current direct trimmed-transcript identity,
sorted prepared-reference identities, pipeline identity, and configured output
contracts, plus the same downstream overlay-selection identity. The same
reference helper and transcript identity are resolved again for artifact
evidence, so changing current transcript bytes, reference bytes, producer
identity, or a named downstream selection makes the prior extraction obsolete.
Operational runner settings do not invalidate otherwise current durable
evidence. This includes `resume` and `chunk_cache`; changing them alone cannot
make a complete extraction non-resumable. If extraction executes, ordinary
runs forward the configured values, while force disables resume and selects
`refresh`.

A valid prepared-reference change makes extraction non-resumable. Missing,
unsafe, or checksum-inconsistent prepared evidence is a hard validation error
with prepare-force guidance because an immediate extract rerun cannot succeed.

The validator then checks the producing run identity, canonical immutable
bundle root, path confinement and absence of symlink components, receipt
identity, exactly one canonical index, the exact configured source set,
contracts and provenance, regular-file status, and stored checksums. Missing or
obsolete results are non-resumable and run again; unsafe filesystem conditions
return an error rather than silently accepting or replacing data.

This sensitivity is limited to the explicit downstream selector. Narratio's
own overlay label and whole effective-configuration digest are not blanket
extraction invalidators. Neither contract can observe files imported by
Notarius configuration, profile contents, prompt/module definitions, or other
transitive inputs. Operators must force extraction after changing any such
private input behind a stable identifier.

## Failure Behavior

Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
index compatibility, inconsistent warning or diagnostic envelopes,
required-lane rejection or incomplete validation, payload inspection,
checksum, or installation errors fail the stage through ordinary manifest
transition handling.
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
not given selectable extraction sources unless the complete configured result
has passed validation and installation.

When a replacement attempt begins, the current session-stage record no longer
advertises payload from the previous success. A failed replacement therefore
has no current outputs, logs, generated configuration references, or metadata,
while the earlier immutable promoted bundle remains available for audit and
recovery.

## Implementation And Focused Tests

- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
  `internal/stage/extract_resume.go`,
  `internal/stage/extract_test.go`,
  `internal/stage/semantic_contracts_delivery.go`
- Stable snapshot materialization and verification:
  `internal/stage/extract_references.go`
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
  `internal/adapters/notarius/subprocess_test.go`
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
  `internal/artifacts/extraction_catalog_test.go`
- Composition and downstream behavior: `internal/app/runner_test.go`,
  `internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
- Opt-in Notarius v0.8.1 production-boundary coverage:
  `internal/stage/notarius_cache_integration_test.go`