narratio/docs/internal/stage-analyze.md

# Stage: analyze

## Purpose

Reconcile configured Scriptorium artifacts, execute only required work in
dependency order, and safely materialize validated outputs.

## Inputs

- ordinary configured artifacts from `pipeline.scriptorium.artifacts`; canonical
  party artifact families have already expanded into this map during
  configuration resolution, including corresponding member dependencies and
  rewritten member-artifact input sources. Each artifact's dependency list is
  already the normalized union of explicit ordering edges and configured-artifact
  input edges.
- optional selected artifact keys supplied through the stage environment
- built-in, configured, extraction, and previous-session source references in
  artifact inputs

Supported source families:
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
  `narratio.input.glossary`, `narratio.input.spell_catalog`
- configured artifacts: `narratio.artifact.<key>`
- extraction lanes: `narratio.extraction.<key>`
- previous-session cache: `narratio.previous_session.artifact.<key>`

## Outputs

- one current per-artifact manifest record per validated materialized output
- stage metadata describing selected/generated/reused artifacts

## Key Behavior

- when `pipeline.scriptorium` is absent or no configured artifact is
  executable, completes successfully and records explanatory metadata. Before
  returning, it reconciles and checkpoints the complete session collection:
  records removed from configuration are dropped, while valid records for
  configured but unselected artifacts remain authoritative. This is not an
  explicit self-skip: the session manifest records success, satisfies publish's
  prerequisite, and an ordinary later run reuses the result while the effective
  set remains empty. Enabling or selecting an artifact later makes missing
  versioned evidence non-resumable and schedules it without requiring force.
- builds a runtime artifact catalog containing built-ins, configured artifacts,
  and configured extraction lanes. Extraction availability is hydrated only
  from compatible successful extraction evidence.
- uses enabled configured artifacts by default. An explicit `--artifacts`
  selection is a one-invocation override that makes exactly the named
  configured artifacts explicit targets even when disabled. The work planner
  adds required configured prerequisites, reuses current ones, and schedules
  stale, missing, or otherwise non-current prerequisites before dependents.
- makes a non-executable configured artifact reusable only when its current
  manifest record and durable output pass the configured-artifact evidence
  contract; an incidental or stale canonical file is unavailable.
- validates selected artifact dependency order (cycle-safe topo ordering).
- consumes only the concrete dependency graph finalized by configuration;
  planning, prerequisite closure, execution, reuse, invalidation, and semantic
  evidence do not infer a second graph from inputs.
- resolves required/optional inputs per artifact source definition into an
  ordered semantic identity. Each identity records the configured input name,
  canonical source ID, required policy, explicit presence, source contract,
  checksum, size, and a source-based logical identity. Workspace paths and
  producer run IDs are excluded.
- orders input identities by configured input name independently of Go map
  iteration. Runtime adapter paths remain a separate execution-only map.
- omits an unavailable optional input from the adapter request while retaining
  explicit absence in its semantic identity; an unavailable required input
  fails.
- resolves prepared stable input sources through the shared manifest-authoritative
  identity resolver; it does not accept incidental files or fall back to
  campaign/session source paths.
- reuses checksums and sizes from validated prepared, extraction, and current
  configured-artifact evidence. Other resolved inputs are hashed as confined
  regular files with streaming reads and the central resolved-artifact size
  limit.
- owns a versioned SHA-256 fingerprint contract with one fixed-field canonical
  JSON payload and no map serialization. Configured artifacts are fingerprinted
  in deterministic dependency order.
- fingerprints the normalized artifact key, prompt and profile identifiers,
  effective render-debug behavior, session-relative output identity, sorted
  dependency keys, ordered input declarations and semantic identities,
  validated current dependency-output identities, and sorted effective
  Scriptorium variables (including Narratio's sticky session variable).
- provides read-only reconciliation that classifies each configured record as
  current, stale, missing, failed, or otherwise non-resumable, and
  separately identifies manifest records removed from current configuration.
  A record is current only when its fingerprint version and value match and its
  configured output still passes manifest-authoritative evidence validation.
- owns a read-only typed work planner. Its explicit targets are enabled
  artifacts by default or the exact normalized `--artifacts` selection when
  supplied. It closes targets over configured prerequisites, orders the closure
  topologically, reuses current members, and schedules every non-current member
  before its dependents.
- force applies only to explicit targets. A current prerequisite is reused
  unless it is itself an explicit forced target; disabled prerequisites may be
  rebuilt when required, while unrelated disabled artifacts are excluded.
- the work plan carries explicit targets, prerequisite-only work, deterministic
  execution and reuse lists, invalidated and removed records, and a cloned
  reconciled starting collection. Valid unrelated configured records survive
  reconciliation, removed records are omitted, and incidental files never
  become current without manifest-backed regeneration.
- implements aggregate resume validation by running the same read-only catalog,
  fingerprint reconciliation, and work planner used by execution. A succeeded
  aggregate record is reusable exactly when the selected closure schedules no
  artifact work and no removed configuration records require a checkpoint;
  stale unrelated records do not block a partial selection.
- exposes the typed artifact decision to `session plan`. Planning applies it to
  a cloned manifest after modeling earlier selected stage transitions, so
  aggregate run/skip and artifact execute/reuse decisions match the ordinary
  runner without creating durable state or invoking Scriptorium.
- executes only the work plan's scheduled entries. Manifest-validated current
  prerequisites remain available through the runtime catalog without invoking
  Scriptorium; newly produced prerequisites enter that catalog with the same
  contract, checksum, and size identity used for persisted current evidence.
- before the first adapter call, checkpoints the reconciled starting collection
  when it removed configuration records or cleared obsolete authority. After
  each successful artifact, checkpoints the complete collection before making
  that artifact available to the next scheduled dependent. Analyze supplies
  its typed update to the shared
  [checkpoint persistence boundary](manifest.md#checkpoint-persistence-boundary)
  and refreshes its working and committed collections only after the save
  succeeds.
- keeps adapter output in the invocation's run-local analyze directory until
  it is a safe, non-empty, bounded regular file with a calculated checksum and
  complete output contract. Canonical replacement uses the shared atomic file
  operation boundary and verifies that the installed checksum matches the
  validated run-local bytes.
- records each successful artifact's freshly computed fingerprint, canonical
  relative output path, contract, checksum, size, producer run ID, bounded
  Scriptorium provenance, logs, and generated configuration references in the
  analyze-owned manifest collection.
- preserves valid unrelated current records during partial execution. If a
  rebuilt output's bytes and contract are unchanged, unselected dependents may
  remain current. If that semantic identity changes, unselected transitive
  dependents become stale without being executed; dependents included in the
  invocation are evaluated in dependency order instead.
- returns aggregate outputs from every current session record in normalized-key
  order after the complete reconciled collection has been checkpointed;
  unrelated stale records do not make an otherwise successful partial
  invocation fail.
- resolves previous-session sources from local `previous/` cache only.
- runs optional render-debug, then artifact execution.
- validates non-empty output files and materializes canonical outputs.

## Failure Semantics

- required missing configured/previous-session inputs fail.
- missing required prepared stable input source includes prepare rerun guidance.
- missing required previous-session source includes prepare rerun guidance.
- missing required `narratio.transcript.final_markdown` or
  `narratio.transcript.final_trimmed_markdown` inputs includes render rerun
  guidance.
- dependency cycles or unavailable required dependencies fail.
- adapter validation failures fail stage.
- a scheduled artifact failure installs a retry collection on the live manifest
  with the active artifact marked `failed`, a bounded error, and no output
  authority. Current transitive dependents become stale without execution.
- earlier artifacts remain current only after their run-local output passed
  validation, canonical materialization, and a successful complete-collection
  manifest checkpoint; unattempted later artifacts retain their reconciled
  session state.
- unrelated current records survive a partial failure. Old canonical bytes for
  the failed artifact and newly materialized bytes whose checkpoint fails are
  incidental, not current evidence.
- a checkpoint failure stops the queue immediately. The in-memory retry
  collection starts from the last successful checkpoint, marks the active
  artifact failed without output authority, and stales its current transitive
  dependents.
- the runner's ordinary error path marks aggregate analyze failed, clears its
  aggregate outputs, invalidates publish, notify, and cleanup through the
  application dependency relation, and persists the live retry collection.
  Terminal-save errors are joined with the original artifact or checkpoint
  failure.

## Invariants

- `analyze` performs no remote storage calls for previous-session source resolution.
- input-identity resolution is read-only: it does not invoke adapters,
  materialize outputs, update status, or create run-local files.
- fingerprints exclude timeouts, retries, timestamps, producer and Narratio run
  IDs, executable and config paths, workspace roots, diagnostic locations, and
  executable or private transitive configuration contents. A change that is
  visible only inside Scriptorium—such as a file privately loaded by its config
  path—requires an explicit forced regeneration.
- output provenance and metadata are deterministic per execution.
- a canonical file without current per-artifact manifest evidence is never
  promoted to current state.
- an ordinary retry after interruption or partial failure reuses every valid
  earlier checkpoint and schedules only failed, stale, missing, or otherwise
  non-current work in dependency order.

## Related Contracts And Tests

- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
  fields and source-selection rules, including
  [artifact families](../config.md#scriptorium-artifact-families).
- [CLI](../cli.md) owns user-visible artifact selection.
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
- Implementation and tests: `internal/stage/analyze.go`,
  `internal/stage/analyze_artifact_execution.go`,
  `internal/stage/analyze_execution_state.go`,
  `internal/stage/analyze_input_identity.go`,
  `internal/stage/analyze_fingerprint.go`,
  `internal/stage/analyze_reconciliation.go`, `internal/stage/analyze_plan.go`,
  `internal/stage/analyze_resume.go`, `internal/stage/analyze_test.go`,
  `internal/stage/analyze_checkpoint_test.go`,
  `internal/stage/analyze_input_identity_test.go`,
  `internal/stage/analyze_fingerprint_test.go`,
  `internal/stage/analyze_reconciliation_test.go`,
  `internal/stage/analyze_plan_test.go`,
  `internal/stage/analyze_incremental_execution_test.go`,
  `internal/stage/analyze_failure_test.go`,
  `internal/stage/analyze_resume_test.go`, and
  `internal/app/analyze_checkpoint_recovery_test.go`.