Files
notarius/docs/internal/pipeline.md

205 lines
12 KiB
Markdown
Raw Normal View History

# Pipeline Internals
This guide explains resolution, typed execution, candidate assessment, and
ordering in `internal/framework/pipeline`.
[Configuration](../config.md#pipelines) owns bindings and producer controls;
[Operations](../operations.md) owns state/recovery workflows;
[integration contracts](../integrations/json-output.md) own deliverable shapes.
[Module Internals](modules.md) owns extension recipes.
## Boundary And Fixed Shape
The framework receives prepared composition, source bytes, a supplied session,
native execution options, and explicit cache/capture collaborators. It returns
logical deliverable files, process diagnostics, and fallback count. Normalized
artifacts and reuse eligibility remain private run state for handoffs/evidence;
there is no reporting ledger in the output boundary.
```text
input -> chunk -> ordered steps [extract -> merge -> normalize] -> output
```
Input and chunking are pipeline-wide. Each ordered step contains typed artifact
lanes; the next step waits for the current step and its generated handoffs.
This fixed pipeline is not a general workflow graph. The CLI owns arguments,
configuration loading, session derivation, physical roots, publication, and
native reporting; see [CLI Internals](cli.md).
## Resolve, Materialize, Prepare
[ResolvePipeline](../../internal/framework/pipeline/profile.go) selects modules,
capabilities, exact artifact kinds, validators, references, options, and concrete
producer controls. It copies static native prompt dependencies from selected
specifications. Effective profiles apply only to model-backed roles; an empty
profile selection leaves native prompt defaults applicable.
[Control resolution](../../internal/framework/pipeline/validation_policy.go)
settles presence-aware controls before execution; modules do not repeat inheritance.
The composition digest describes resolved composition, not cache identity.
Resolution receives exact reference bindings/unbinds, not CLI selector grammar.
It preserves required slots, external/generated conflicts, prior-step
dependencies, and artifact compatibility. [Reference materialization](../../internal/framework/pipeline/references.go)
reads selected external bytes with declaration/invocation origins, media/UTF-8
and declared size checks. Generated selections remain declarations until their
producer step completes. Public selection semantics belong in
[References And Ordered Handoffs](../config.md#references-and-ordered-handoffs).
[Prepare](../../internal/framework/pipeline/prepare.go) clones resolved data and
constructs input, chunker, validators, typed lanes, and output through supplied
registries and the shared native engine. Builders receive independently owned
build requests. Exact Go type, codec, evidence-projector, registration, and
constructor checks happen before source parsing. There is no correction transport,
asset registry, or persistent execution identity to collect.
Offline known-resource [preflight](../../internal/preflight/preflight.go) is
a separate composition check, not request execution or module construction.
## Typed Operations And Assessment
[Typed contracts](../../internal/framework/contracts/typed_pipeline.go) carry
a `ProducerOperation[T]` with resolved controls, one assessor, native execution
options, incoming reuse eligibility, and an optional typed cache binding.
[Runner operation completion](../../internal/framework/pipeline/runner_operation.go)
assesses deterministic production directly. Model modules use
[RunProducer](../../internal/framework/pipeline/producer.go); the runner consumes
its acceptance/fallback outcome without assessing it again.
[Candidate assessment](../../internal/framework/pipeline/candidate_assessment.go)
invokes each applicable validator once in configured order. It continues after
rejections and unavailable/failed assessments; caller cancellation and framework/
capture failures interrupt it. Typed candidates are encoded once for assessment,
then decoded separately for each typed validator. Serialized validators receive
owned bytes/schema metadata; chunk validators receive detached materialized
chunks. Source, references, annotations, and metadata views are isolated too.
[Assessment values](../../internal/framework/contracts/assessment.go) distinguish
approval, semantic rejection, failure, and unavailable judgment. An empty configured
chain adds no ordinary acceptance criteria. Intrinsic codecs, source parsing,
materialization, and necessary module/fallback/cache checks remain; ordinary
artifact citation checks depend on the [configured chain](../config.md#module-bindings-and-validators).
Rejection guidance is module-authored, ordered, and
deduplicated separately from reason codes/operator messages. Invalid guidance is
a contract failure, not invented correction text. LLM validators execute directly
on the shared engine and do not own another semantic loop.
## Native Continuation And Terminal Handling
For executed model work, `RunProducer` checks native errors and structural
validation status before interpretation, including a failed result with no Go
error. Module interpretation produces a usable typed candidate and any actionable
domain guidance. The assessor runs the complete chain on every delivered usable
candidate, including unchanged revisions and the final allowed response.
When correction is possible, the helper starts native `Coordinate` from the
captured exchange. Its first callback reuses already-collected initial findings.
Native coordination owns turn iteration and growing history; the module builds
only new correction material from fixed operation inputs. Successful native
execution, stopped/exhausted coordination, and domain acceptance are distinct.
Native repair/transient retries are not extended by outer application retries.
[LLM Integration](llm.md) owns the native request/capture integration.
[Terminal disposition](../../internal/framework/pipeline/producer_outcome.go)
either fails or invokes one supported module fallback. Fallback construction
preserves intrinsic/source/downstream invariants without ordinary reassessment
or semantic-approval claims. It may fail normally. A failed continuation cannot
recover an earlier candidate; caller cancellation and undelivered responses
produce no acceptance/fallback. Native operational timeouts retain typed causes
rather than becoming caller cancellation solely by error spelling.
Deterministic work produces once, then uses direct assessment and any supported
module fallback. An operation/parse/materialization/encoding error remains an
error, not a generic rerun or permissive recovery mechanism.
## One Ready-Job Worker Pool
[The lane coordinator](../../internal/framework/pipeline/runner_concurrent.go)
dispatches extraction jobs and ready lane continuations to the same bounded
worker pool. A continuation becomes ready after all extracts for its lane finish;
one worker runs its merge then normalization. Ready continuations can interleave
with unrelated extraction, and prerequisite waits occupy no worker.
Input parsing, chunking, step barriers, and final output remain sequential.
Results are restored to chunk/lane order before merging. Stable operation/lane/
chunk positions select the scoped terminal error among reported failures rather
than response timing. Parent cancellation takes precedence. On failure/cancellation
the coordinator stops dispatch, drains started results, closes jobs, and joins
workers before returning. Output encoding does not run after terminal failure.
Nested validation and correction use the operation's worker, not another worker
permit. Native backend admission is independent and released before candidate
assessment, allowing a nested validator on the same limited backend. Worker
configuration does not create another model scheduler, queue, or retry policy.
## Handoffs And Reuse Eligibility
[Generated handoffs](../../internal/framework/pipeline/handoff.go) locate one
earlier normalized producer, decode/re-encode through its registered codec, check
full artifact/schema/media compatibility, and clone bytes into the consumer's
declared reference set. Valid qualified fallback is usable current-run context;
it does not become ordinary acceptance through a handoff.
Eligibility travels separately from artifact bytes:
- A chunk fallback disables reuse for all derived lane work.
- An extract fallback disables reuse for its dependent merge/normalization.
- A fallback-derived generated reference disables reuse from the consuming
operation onward; unrelated siblings and earlier accepted work remain eligible.
An emitting fallback increments the run count once. Merely consuming its data or
inheriting ineligibility adds no fallback. These are transient dependency facts,
not persisted fingerprints or a second recovery protocol.
[Typed cache bindings](../../internal/framework/pipeline/runner_cache.go) use the
registered durable codec and available exact-type evidence projection for
current-source usability. [Chunk planning](../../internal/framework/pipeline/runner_chunk_plan.go)
uses the same producer cache with plan decoding/rematerialization. Hits do not
run semantic assessment. Deterministic work has no synthetic request/cache key.
[Run State Internals](state.md) owns prepared-handle/store mechanics and recovery.
## Output And Capture Boundaries
[Runner](../../internal/framework/pipeline/runner.go) parses/validates source,
materializes the final plan, runs ordered steps, aggregates diagnostics, and
builds optional chunk-map/evidence values. Normalized values are codec-checked
before handoff/output. Evidence reconstruction also works from reused artifacts;
it projects selected citations in source order, not a second typed output channel.
[Evidence internals](../../internal/framework/pipeline/evidence_output.go) and
the [evidence contract](../integrations/evidence-context.md) define those owners.
The encoder receives owned normalized/auxiliary values and returns logical files.
It does not choose a directory, publish a report, or serialize execution ledgers.
[JSON output](../integrations/json-output.md) owns logical paths and payloads;
[State Internals](state.md) owns physical publication and failure retention.
Optional scoped evidence records inputs, candidates, findings, outcomes, and
cache decisions. Native observers enroll actual executions once in one invocation
group, preserving unknown usage and terminal coverage. Capture failures are
ordinary boundary failures, never semantic feedback or reasons to rerun a model.
Debug layout is private; it is not a recovery source or consumer protocol.
## Focused Verification
- [Resolution](../../internal/framework/pipeline/typed_resolution_test.go),
[preparation](../../internal/framework/pipeline/preparation_test.go), and
[references](../../internal/framework/pipeline/references_test.go) cover static
compatibility and construction/materialization ownership.
- [Assessment](../../internal/framework/pipeline/candidate_assessment_test.go)
and [producer tests](../../internal/framework/pipeline/producer_test.go) cover
isolation, whole-chain/final-turn assessment, guidance, native status, and cancellation.
- [Runner operations](../../internal/framework/pipeline/runner_operation_test.go)
cover no outer reassessment, deterministic errors, fallback counts/eligibility,
and actual generated consumers.
- [Worker tests](../../internal/framework/pipeline/runner_concurrency_test.go)
and [ready-job tests](../../internal/framework/pipeline/ready_jobs_test.go)
cover mixed jobs, stable order, serial liveness, nested admission, and draining.
- [Producer reuse](../../internal/framework/pipeline/producer_cache_test.go),
[runner reuse](../../internal/framework/pipeline/runner_cache_test.go), and
[plan reuse](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
initial keys, applied artifacts, no reassessment on hits, and current-source checks.
- [Capture](../../internal/framework/pipeline/native_capture_test.go),
[evidence](../../internal/framework/pipeline/evidence_output_test.go), and
[assembled recovery](../../internal/cli/recovery_contract_test.go) cover
native facts, output reconstruction, and cross-boundary reuse/failure.
Use focused pipeline and CLI suites for boundary changes.
[Development](../development.md#validation) owns repository-wide checks.