205 lines
12 KiB
Markdown
205 lines
12 KiB
Markdown
# Pipeline Internals
|
|
|
|
This guide explains resolution, typed execution, candidate assessment, and
|
|
ordering in `internal/framework/pipeline`.
|
|
[Configuration](../config.md#pipelines) owns bindings and producer controls;
|
|
[Operations](../operations.md) owns state/recovery workflows;
|
|
[integration contracts](../integrations/json-output.md) own deliverable shapes.
|
|
[Module Internals](modules.md) owns extension recipes.
|
|
|
|
## Boundary And Fixed Shape
|
|
|
|
The framework receives prepared composition, source bytes, a supplied session,
|
|
native execution options, and explicit cache/capture collaborators. It returns
|
|
logical deliverable files, process diagnostics, and fallback count. Normalized
|
|
artifacts and reuse eligibility remain private run state for handoffs/evidence;
|
|
there is no reporting ledger in the output boundary.
|
|
|
|
```text
|
|
input -> chunk -> ordered steps [extract -> merge -> normalize] -> output
|
|
```
|
|
|
|
Input and chunking are pipeline-wide. Each ordered step contains typed artifact
|
|
lanes; the next step waits for the current step and its generated handoffs.
|
|
This fixed pipeline is not a general workflow graph. The CLI owns arguments,
|
|
configuration loading, session derivation, physical roots, publication, and
|
|
native reporting; see [CLI Internals](cli.md).
|
|
|
|
## Resolve, Materialize, Prepare
|
|
|
|
[ResolvePipeline](../../internal/framework/pipeline/profile.go) selects modules,
|
|
capabilities, exact artifact kinds, validators, references, options, and concrete
|
|
producer controls. It copies static native prompt dependencies from selected
|
|
specifications. Effective profiles apply only to model-backed roles; an empty
|
|
profile selection leaves native prompt defaults applicable.
|
|
[Control resolution](../../internal/framework/pipeline/validation_policy.go)
|
|
settles presence-aware controls before execution; modules do not repeat inheritance.
|
|
The composition digest describes resolved composition, not cache identity.
|
|
|
|
Resolution receives exact reference bindings/unbinds, not CLI selector grammar.
|
|
It preserves required slots, external/generated conflicts, prior-step
|
|
dependencies, and artifact compatibility. [Reference materialization](../../internal/framework/pipeline/references.go)
|
|
reads selected external bytes with declaration/invocation origins, media/UTF-8
|
|
and declared size checks. Generated selections remain declarations until their
|
|
producer step completes. Public selection semantics belong in
|
|
[References And Ordered Handoffs](../config.md#references-and-ordered-handoffs).
|
|
|
|
[Prepare](../../internal/framework/pipeline/prepare.go) clones resolved data and
|
|
constructs input, chunker, validators, typed lanes, and output through supplied
|
|
registries and the shared native engine. Builders receive independently owned
|
|
build requests. Exact Go type, codec, evidence-projector, registration, and
|
|
constructor checks happen before source parsing. There is no correction transport,
|
|
asset registry, or persistent execution identity to collect.
|
|
Offline known-resource [preflight](../../internal/preflight/preflight.go) is
|
|
a separate composition check, not request execution or module construction.
|
|
|
|
## Typed Operations And Assessment
|
|
|
|
[Typed contracts](../../internal/framework/contracts/typed_pipeline.go) carry
|
|
a `ProducerOperation[T]` with resolved controls, one assessor, native execution
|
|
options, incoming reuse eligibility, and an optional typed cache binding.
|
|
[Runner operation completion](../../internal/framework/pipeline/runner_operation.go)
|
|
assesses deterministic production directly. Model modules use
|
|
[RunProducer](../../internal/framework/pipeline/producer.go); the runner consumes
|
|
its acceptance/fallback outcome without assessing it again.
|
|
|
|
[Candidate assessment](../../internal/framework/pipeline/candidate_assessment.go)
|
|
invokes each applicable validator once in configured order. It continues after
|
|
rejections and unavailable/failed assessments; caller cancellation and framework/
|
|
capture failures interrupt it. Typed candidates are encoded once for assessment,
|
|
then decoded separately for each typed validator. Serialized validators receive
|
|
owned bytes/schema metadata; chunk validators receive detached materialized
|
|
chunks. Source, references, annotations, and metadata views are isolated too.
|
|
|
|
[Assessment values](../../internal/framework/contracts/assessment.go) distinguish
|
|
approval, semantic rejection, failure, and unavailable judgment. An empty configured
|
|
chain adds no ordinary acceptance criteria. Intrinsic codecs, source parsing,
|
|
materialization, and necessary module/fallback/cache checks remain; ordinary
|
|
artifact citation checks depend on the [configured chain](../config.md#module-bindings-and-validators).
|
|
Rejection guidance is module-authored, ordered, and
|
|
deduplicated separately from reason codes/operator messages. Invalid guidance is
|
|
a contract failure, not invented correction text. LLM validators execute directly
|
|
on the shared engine and do not own another semantic loop.
|
|
|
|
## Native Continuation And Terminal Handling
|
|
|
|
For executed model work, `RunProducer` checks native errors and structural
|
|
validation status before interpretation, including a failed result with no Go
|
|
error. Module interpretation produces a usable typed candidate and any actionable
|
|
domain guidance. The assessor runs the complete chain on every delivered usable
|
|
candidate, including unchanged revisions and the final allowed response.
|
|
|
|
When correction is possible, the helper starts native `Coordinate` from the
|
|
captured exchange. Its first callback reuses already-collected initial findings.
|
|
Native coordination owns turn iteration and growing history; the module builds
|
|
only new correction material from fixed operation inputs. Successful native
|
|
execution, stopped/exhausted coordination, and domain acceptance are distinct.
|
|
Native repair/transient retries are not extended by outer application retries.
|
|
[LLM Integration](llm.md) owns the native request/capture integration.
|
|
|
|
[Terminal disposition](../../internal/framework/pipeline/producer_outcome.go)
|
|
either fails or invokes one supported module fallback. Fallback construction
|
|
preserves intrinsic/source/downstream invariants without ordinary reassessment
|
|
or semantic-approval claims. It may fail normally. A failed continuation cannot
|
|
recover an earlier candidate; caller cancellation and undelivered responses
|
|
produce no acceptance/fallback. Native operational timeouts retain typed causes
|
|
rather than becoming caller cancellation solely by error spelling.
|
|
|
|
Deterministic work produces once, then uses direct assessment and any supported
|
|
module fallback. An operation/parse/materialization/encoding error remains an
|
|
error, not a generic rerun or permissive recovery mechanism.
|
|
|
|
## One Ready-Job Worker Pool
|
|
|
|
[The lane coordinator](../../internal/framework/pipeline/runner_concurrent.go)
|
|
dispatches extraction jobs and ready lane continuations to the same bounded
|
|
worker pool. A continuation becomes ready after all extracts for its lane finish;
|
|
one worker runs its merge then normalization. Ready continuations can interleave
|
|
with unrelated extraction, and prerequisite waits occupy no worker.
|
|
Input parsing, chunking, step barriers, and final output remain sequential.
|
|
|
|
Results are restored to chunk/lane order before merging. Stable operation/lane/
|
|
chunk positions select the scoped terminal error among reported failures rather
|
|
than response timing. Parent cancellation takes precedence. On failure/cancellation
|
|
the coordinator stops dispatch, drains started results, closes jobs, and joins
|
|
workers before returning. Output encoding does not run after terminal failure.
|
|
|
|
Nested validation and correction use the operation's worker, not another worker
|
|
permit. Native backend admission is independent and released before candidate
|
|
assessment, allowing a nested validator on the same limited backend. Worker
|
|
configuration does not create another model scheduler, queue, or retry policy.
|
|
|
|
## Handoffs And Reuse Eligibility
|
|
|
|
[Generated handoffs](../../internal/framework/pipeline/handoff.go) locate one
|
|
earlier normalized producer, decode/re-encode through its registered codec, check
|
|
full artifact/schema/media compatibility, and clone bytes into the consumer's
|
|
declared reference set. Valid qualified fallback is usable current-run context;
|
|
it does not become ordinary acceptance through a handoff.
|
|
|
|
Eligibility travels separately from artifact bytes:
|
|
|
|
- A chunk fallback disables reuse for all derived lane work.
|
|
- An extract fallback disables reuse for its dependent merge/normalization.
|
|
- A fallback-derived generated reference disables reuse from the consuming
|
|
operation onward; unrelated siblings and earlier accepted work remain eligible.
|
|
|
|
An emitting fallback increments the run count once. Merely consuming its data or
|
|
inheriting ineligibility adds no fallback. These are transient dependency facts,
|
|
not persisted fingerprints or a second recovery protocol.
|
|
|
|
[Typed cache bindings](../../internal/framework/pipeline/runner_cache.go) use the
|
|
registered durable codec and available exact-type evidence projection for
|
|
current-source usability. [Chunk planning](../../internal/framework/pipeline/runner_chunk_plan.go)
|
|
uses the same producer cache with plan decoding/rematerialization. Hits do not
|
|
run semantic assessment. Deterministic work has no synthetic request/cache key.
|
|
[Run State Internals](state.md) owns prepared-handle/store mechanics and recovery.
|
|
|
|
## Output And Capture Boundaries
|
|
|
|
[Runner](../../internal/framework/pipeline/runner.go) parses/validates source,
|
|
materializes the final plan, runs ordered steps, aggregates diagnostics, and
|
|
builds optional chunk-map/evidence values. Normalized values are codec-checked
|
|
before handoff/output. Evidence reconstruction also works from reused artifacts;
|
|
it projects selected citations in source order, not a second typed output channel.
|
|
[Evidence internals](../../internal/framework/pipeline/evidence_output.go) and
|
|
the [evidence contract](../integrations/evidence-context.md) define those owners.
|
|
|
|
The encoder receives owned normalized/auxiliary values and returns logical files.
|
|
It does not choose a directory, publish a report, or serialize execution ledgers.
|
|
[JSON output](../integrations/json-output.md) owns logical paths and payloads;
|
|
[State Internals](state.md) owns physical publication and failure retention.
|
|
|
|
Optional scoped evidence records inputs, candidates, findings, outcomes, and
|
|
cache decisions. Native observers enroll actual executions once in one invocation
|
|
group, preserving unknown usage and terminal coverage. Capture failures are
|
|
ordinary boundary failures, never semantic feedback or reasons to rerun a model.
|
|
Debug layout is private; it is not a recovery source or consumer protocol.
|
|
|
|
## Focused Verification
|
|
|
|
- [Resolution](../../internal/framework/pipeline/typed_resolution_test.go),
|
|
[preparation](../../internal/framework/pipeline/preparation_test.go), and
|
|
[references](../../internal/framework/pipeline/references_test.go) cover static
|
|
compatibility and construction/materialization ownership.
|
|
- [Assessment](../../internal/framework/pipeline/candidate_assessment_test.go)
|
|
and [producer tests](../../internal/framework/pipeline/producer_test.go) cover
|
|
isolation, whole-chain/final-turn assessment, guidance, native status, and cancellation.
|
|
- [Runner operations](../../internal/framework/pipeline/runner_operation_test.go)
|
|
cover no outer reassessment, deterministic errors, fallback counts/eligibility,
|
|
and actual generated consumers.
|
|
- [Worker tests](../../internal/framework/pipeline/runner_concurrency_test.go)
|
|
and [ready-job tests](../../internal/framework/pipeline/ready_jobs_test.go)
|
|
cover mixed jobs, stable order, serial liveness, nested admission, and draining.
|
|
- [Producer reuse](../../internal/framework/pipeline/producer_cache_test.go),
|
|
[runner reuse](../../internal/framework/pipeline/runner_cache_test.go), and
|
|
[plan reuse](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
|
|
initial keys, applied artifacts, no reassessment on hits, and current-source checks.
|
|
- [Capture](../../internal/framework/pipeline/native_capture_test.go),
|
|
[evidence](../../internal/framework/pipeline/evidence_output_test.go), and
|
|
[assembled recovery](../../internal/cli/recovery_contract_test.go) cover
|
|
native facts, output reconstruction, and cross-boundary reuse/failure.
|
|
|
|
Use focused pipeline and CLI suites for boundary changes.
|
|
[Development](../development.md#validation) owns repository-wide checks.
|