Refresh pipeline execution and run state internal guides
This commit is contained in:
@@ -1,279 +1,202 @@
|
||||
# Pipeline Internals
|
||||
|
||||
This document describes the framework-owned pipeline mechanics in
|
||||
**internal/framework/pipeline**. [Configuration](../config.md) owns selectable
|
||||
profiles, bindings, and retry settings; [Operations](../operations.md) owns
|
||||
state lifecycle and recovery; and the [integration contracts](../integrations/)
|
||||
own durable output shapes. Concrete production extensions are covered by
|
||||
[Module Internals](modules.md).
|
||||
This guide explains resolution, typed execution, candidate assessment, and
|
||||
ordering in `internal/framework/pipeline`.
|
||||
[Configuration](../config.md#pipelines) owns bindings and producer controls;
|
||||
[Operations](../operations.md) owns state/recovery workflows;
|
||||
[integration contracts](../integrations/json-output.md) own deliverable shapes.
|
||||
[Module Internals](modules.md) owns extension recipes.
|
||||
|
||||
## Boundary
|
||||
## Boundary And Fixed Shape
|
||||
|
||||
The pipeline framework accepts a resolved composition, registries, shared
|
||||
dependencies, input bytes, a supplied prompt session, and state/debug
|
||||
collaborators. It returns logical output files, normalized artifacts, recorded
|
||||
rejections, grouped diagnostics, manifest provenance, and checkpoint decisions. The
|
||||
CLI owns process arguments, configuration discovery, session resolution,
|
||||
physical roots, and placement of returned output files.
|
||||
The framework receives prepared composition, source bytes, a supplied session,
|
||||
native execution options, and explicit cache/capture collaborators. It returns
|
||||
logical deliverable files, process diagnostics, and fallback count. Normalized
|
||||
artifacts and reuse eligibility remain private run state for handoffs/evidence;
|
||||
there is no reporting ledger in the output boundary.
|
||||
|
||||
The framework has one fixed shape:
|
||||
```text
|
||||
input -> chunk -> ordered steps [extract -> merge -> normalize] -> output
|
||||
```
|
||||
|
||||
~~~
|
||||
input -> chunk -> extract -> merge -> normalize -> output
|
||||
~~~
|
||||
|
||||
Input and chunking are pipeline-wide. A selected artifact lane owns extract,
|
||||
merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
|
||||
is an ordered list of steps, not an arbitrary workflow graph.
|
||||
Input and chunking are pipeline-wide. Each ordered step contains typed artifact
|
||||
lanes; the next step waits for the current step and its generated handoffs.
|
||||
This fixed pipeline is not a general workflow graph. The CLI owns arguments,
|
||||
configuration loading, session derivation, physical roots, publication, and
|
||||
native reporting; see [CLI Internals](cli.md).
|
||||
|
||||
## Resolve, Materialize, Prepare
|
||||
|
||||
Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
|
||||
It normalizes the pipeline and lane identities, applies stage defaults, selects
|
||||
requested lanes where that is supported, resolves validator chains, checks
|
||||
module capabilities and typed artifact compatibility, validates options, and
|
||||
assigns a deterministic resolved-composition digest. A correction protocol is
|
||||
selected from each eligible LLM-backed producer specification and becomes part
|
||||
of that resolved identity; only `single_response_v1` is currently supported.
|
||||
Preparation rejects an LLM-backed producer that combines a non-empty validator
|
||||
chain with positive producer retries unless it declares that protocol. Producers
|
||||
without validators or without retries remain valid without correction support.
|
||||
The resolved pipeline contains bindings and declared reference targets, not
|
||||
external reference bytes.
|
||||
After selection, the resolver applies command, binding, and pipeline profile
|
||||
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
||||
an empty resolved binding profile. It resolves structural output repair
|
||||
separately: a binding's `structured_output_repair_attempts` value wins, then a
|
||||
pipeline value applies to LLM-backed bindings and validators, and omission
|
||||
leaves the prompt-owned policy intact. An explicit repair value on a
|
||||
deterministic binding is rejected. Resolved bindings own copied repair values,
|
||||
and these effective values are part of the digest, so execution and checkpoint
|
||||
consumers do not repeat profile inheritance or configuration resolution.
|
||||
Each LLM request receives its own copy of that resolved value. PromptKit spends
|
||||
it only for structural correction inside one completion; the runner's binding
|
||||
retry policy remains the separate outer budget for complete stage attempts.
|
||||
Configuration resolution supplies the selected overlay and catalog; see
|
||||
[Configuration Internals](configuration.md).
|
||||
[ResolvePipeline](../../internal/framework/pipeline/profile.go) selects modules,
|
||||
capabilities, exact artifact kinds, validators, references, options, and concrete
|
||||
producer controls. It copies static native prompt dependencies from selected
|
||||
specifications. Effective profiles apply only to model-backed roles; an empty
|
||||
profile selection leaves native prompt defaults applicable.
|
||||
[Control resolution](../../internal/framework/pipeline/validation_policy.go)
|
||||
settles presence-aware controls before execution; modules do not repeat inheritance.
|
||||
The composition digest describes resolved composition, not cache identity.
|
||||
|
||||
External reference materialization happens before preparation. The materializer
|
||||
checks that each slot is declared by the selected module, resolves a file path
|
||||
relative to the correct configuration or working-directory origin, reads
|
||||
UTF-8 text, verifies media type and size limits, and retains bounded
|
||||
provenance. For a positive slot limit, it reads at most the limit plus one byte
|
||||
and rejects overflow before retaining content. A generated-artifact selector
|
||||
remains declared but has no bytes until its producing step completes.
|
||||
Resolution receives exact reference bindings/unbinds, not CLI selector grammar.
|
||||
It preserves required slots, external/generated conflicts, prior-step
|
||||
dependencies, and artifact compatibility. [Reference materialization](../../internal/framework/pipeline/references.go)
|
||||
reads selected external bytes with declaration/invocation origins, media/UTF-8
|
||||
and declared size checks. Generated selections remain declarations until their
|
||||
producer step completes. Public selection semantics belong in
|
||||
[References And Ordered Handoffs](../config.md#references-and-ordered-handoffs).
|
||||
|
||||
Preparation is the construction boundary. It validates the resolved shape and
|
||||
registry set, clones the resolved data, then constructs the input adapter,
|
||||
chunker, stage-local validators, every typed lane, and output encoder. The
|
||||
prepared producer metadata preserves each selected correction protocol, and
|
||||
the resolved digest carrying that metadata participates in checkpoint identity.
|
||||
Each registered builder receives its own cloned build request immediately
|
||||
before its module-owned code runs. Preparation also collects stable checkpoint
|
||||
fingerprints. Missing registrations, incompatible typed entries, nil
|
||||
implementations, and constructor failures are reported before source parsing
|
||||
or any stage operation begins.
|
||||
[Prepare](../../internal/framework/pipeline/prepare.go) clones resolved data and
|
||||
constructs input, chunker, validators, typed lanes, and output through supplied
|
||||
registries and the shared native engine. Builders receive independently owned
|
||||
build requests. Exact Go type, codec, evidence-projector, registration, and
|
||||
constructor checks happen before source parsing. There is no correction transport,
|
||||
asset registry, or persistent execution identity to collect.
|
||||
Offline known-resource [preflight](../../internal/preflight/preflight.go) is
|
||||
a separate composition check, not request execution or module construction.
|
||||
|
||||
An output encoder can opt into source-evidence publication through its output
|
||||
policy. Preparation keeps the configured lane allowlist and active lanes
|
||||
separate, then verifies an exact typed evidence projector and registered codec
|
||||
for each active lane. The resulting private plan is immutable; lanes excluded
|
||||
by invocation filtering remain configured but do not acquire a projector for
|
||||
that run.
|
||||
## Typed Operations And Assessment
|
||||
|
||||
## Typed Lanes And References
|
||||
[Typed contracts](../../internal/framework/contracts/typed_pipeline.go) carry
|
||||
a `ProducerOperation[T]` with resolved controls, one assessor, native execution
|
||||
options, incoming reuse eligibility, and an optional typed cache binding.
|
||||
[Runner operation completion](../../internal/framework/pipeline/runner_operation.go)
|
||||
assesses deterministic production directly. Model modules use
|
||||
[RunProducer](../../internal/framework/pipeline/producer.go); the runner consumes
|
||||
its acceptance/fallback outcome without assessing it again.
|
||||
|
||||
Each resolved lane has one artifact kind, codec, and exact Go type. The
|
||||
framework uses private type erasure only around those typed operations; every
|
||||
handoff checks exact type and codec identity and reports incompatibility as an
|
||||
error rather than panicking. Encoding through the registered codec is the
|
||||
boundary for output, checkpoints, debug records, and generated references.
|
||||
[Candidate assessment](../../internal/framework/pipeline/candidate_assessment.go)
|
||||
invokes each applicable validator once in configured order. It continues after
|
||||
rejections and unavailable/failed assessments; caller cancellation and framework/
|
||||
capture failures interrupt it. Typed candidates are encoded once for assessment,
|
||||
then decoded separately for each typed validator. Serialized validators receive
|
||||
owned bytes/schema metadata; chunk validators receive detached materialized
|
||||
chunks. Source, references, annotations, and metadata views are isolated too.
|
||||
|
||||
Reference targets are stage- and lane-specific. External reference bytes are
|
||||
cloned into the operation request. Generated references are built at the next
|
||||
step boundary from exactly one accepted normalized producer output. The
|
||||
framework decodes and re-encodes that output with the registered producer
|
||||
codec, checks its complete schema and media identity, and records a content
|
||||
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
|
||||
incompatible producer prevents the consumer step from starting.
|
||||
[Assessment values](../../internal/framework/contracts/assessment.go) distinguish
|
||||
approval, semantic rejection, failure, and unavailable judgment. An empty configured
|
||||
chain adds no ordinary acceptance criteria, but cannot remove intrinsic codec or
|
||||
source-addressing checks. Rejection guidance is module-authored, ordered, and
|
||||
deduplicated separately from reason codes/operator messages. Invalid guidance is
|
||||
a contract failure, not invented correction text. LLM validators execute directly
|
||||
on the shared engine and do not own another semantic loop.
|
||||
|
||||
Resolution receives only exact stage-and-lane operational reference overrides.
|
||||
The CLI may offer broader pipeline- or lane-scoped selectors, but expands and
|
||||
arbitrates those before entering the framework. External overrides are applied
|
||||
after configured external defaults and local bindings. They cannot coexist
|
||||
with a generated binding for the same target and slot, and external unbinds do
|
||||
not remove generated handoffs.
|
||||
## Native Continuation And Terminal Handling
|
||||
|
||||
## Execution And Ordering
|
||||
For executed model work, `RunProducer` checks native errors and structural
|
||||
validation status before interpretation, including a failed result with no Go
|
||||
error. Module interpretation produces a usable typed candidate and any actionable
|
||||
domain guidance. The assessor runs the complete chain on every delivered usable
|
||||
candidate, including unchanged revisions and the final allowed response.
|
||||
|
||||
The runner validates its input, installs no-op state collaborators when none
|
||||
were supplied, and serially performs source parsing and chunk-plan selection.
|
||||
It transports the supplied session unchanged to prompt-facing operations and
|
||||
run-manifest metadata; it neither derives a session nor substitutes a parsed
|
||||
source document identifier. The public session contract is owned by the
|
||||
[CLI reference](../cli.md#run).
|
||||
An accepted plan is materialized into source-addressed chunks and passes the
|
||||
configured chunk validators before any lane runs. A chunk rejection is a
|
||||
recorded pipeline outcome: lanes do not start, but the output stage can encode
|
||||
the terminal result.
|
||||
When correction is possible, the helper starts native `Coordinate` from the
|
||||
captured exchange. Its first callback reuses already-collected initial findings.
|
||||
Native coordination owns turn iteration and growing history; the module builds
|
||||
only new correction material from fixed operation inputs. Successful native
|
||||
execution, stopped/exhausted coordination, and domain acceptance are distinct.
|
||||
Native repair/transient retries are not extended by outer application retries.
|
||||
[LLM Integration](llm.md) owns the native request/capture integration.
|
||||
|
||||
For each ordered step, the runner first builds generated reference sets from
|
||||
the accepted normalized outputs of earlier steps. It then executes the step's
|
||||
lanes. Later steps do not begin until the current step is terminal and its
|
||||
generated handoffs have succeeded.
|
||||
[Terminal disposition](../../internal/framework/pipeline/producer_outcome.go)
|
||||
either fails or invokes one supported module fallback. Fallback construction
|
||||
preserves intrinsic/source/downstream invariants without ordinary reassessment
|
||||
or semantic-approval claims. It may fail normally. A failed continuation cannot
|
||||
recover an earlier candidate; caller cancellation and undelivered responses
|
||||
produce no acceptance/fallback. Native operational timeouts retain typed causes
|
||||
rather than becoming caller cancellation solely by error spelling.
|
||||
|
||||
Within a step, the lane engine dispatches extraction jobs in deterministic
|
||||
chunk-first, lane-second order to a bounded worker group. When all extraction
|
||||
jobs for one lane are terminal, a bounded continuation group can run that
|
||||
lane's merge and normalize work while extraction for other lanes continues.
|
||||
The framework does not create an unbounded goroutine per chunk or lane.
|
||||
Deterministic work produces once, then uses direct assessment and any supported
|
||||
module fallback. An operation/parse/materialization/encoding error remains an
|
||||
error, not a generic rerun or permissive recovery mechanism.
|
||||
|
||||
Completion timing does not determine public results. The coordinator restores
|
||||
lane and chunk order before merging results, and selects a framework error by
|
||||
stable stage, lane, and chunk position. A validator rejection records a lane
|
||||
outcome without cancelling unrelated work. A framework error or parent
|
||||
cancellation cancels derived work, prevents queued work from starting, waits
|
||||
for started workers, and prevents output encoding.
|
||||
## One Ready-Job Worker Pool
|
||||
|
||||
## Validation, Retries, And Output
|
||||
[The lane coordinator](../../internal/framework/pipeline/runner_concurrent.go)
|
||||
dispatches extraction jobs and ready lane continuations to the same bounded
|
||||
worker pool. A continuation becomes ready after all extracts for its lane finish;
|
||||
one worker runs its merge then normalization. Ready continuations can interleave
|
||||
with unrelated extraction, and prerequisite waits occupy no worker.
|
||||
Input parsing, chunking, step barriers, and final output remain sequential.
|
||||
|
||||
Every chunk, extract, merge, and normalize candidate passes its resolved
|
||||
validator chain. Validators receive immutable canonical input appropriate to
|
||||
their target: chunks, codec-decoded typed candidates, or serialized codec
|
||||
bytes. Each typed validator receives a newly decoded value from the one
|
||||
candidate serialization for that attempt, while serialized validators receive
|
||||
separately owned representation bytes and schema metadata. They may approve,
|
||||
approve with warnings, reject, fail, or be skipped when a runtime prerequisite
|
||||
is unavailable. The shared executor settles every configured validator in
|
||||
order. A failed LLM-backed validator retries only itself against the same
|
||||
immutable candidate; it does not regenerate the producer or alter the
|
||||
validator request. Rejections stop that validator, while other configured
|
||||
validators still run. The executor retains ordered results, bounded
|
||||
deduplicated correction guidance from rejections, and only the final exhausted
|
||||
failure outcome for each validator. The correction builder keeps first
|
||||
occurrence order, omits internal reason codes, validator names, and operator
|
||||
messages, and requests one complete replacement. Missing guidance or an
|
||||
oversized aggregate is a framework contract error; guidance is never inferred
|
||||
or truncated.
|
||||
Results are restored to chunk/lane order before merging. Stable operation/lane/
|
||||
chunk positions select the scoped terminal error among reported failures rather
|
||||
than response timing. Parent cancellation takes precedence. On failure/cancellation
|
||||
the coordinator stops dispatch, drains started results, closes jobs, and joins
|
||||
workers before returning. Output encoding does not run after terminal failure.
|
||||
|
||||
The runner applies the binding's retry policy around a stage operation and its
|
||||
complete validation chain. It preserves terminal diagnostics only from the final accepted
|
||||
or rejected attempt, plus one fixed validation-incomplete warning per validator whose execution
|
||||
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
||||
Normalizer-specific retry directives consume this same budget and validate any
|
||||
final safe fallback through the normalizer chain. A directive may carry bounded
|
||||
correction guidance only when it also exposes the exact latest
|
||||
`single_response_v1` candidate. The state machine then uses the same replacement
|
||||
request shape as validator correction. A directive without guidance clears any
|
||||
prior correction and starts a fresh attempt, which preserves structural retry
|
||||
behavior when no valid response exists.
|
||||
Nested validation and correction use the operation's worker, not another worker
|
||||
permit. Native backend admission is independent and released before candidate
|
||||
assessment, allowing a nested validator on the same limited backend. Worker
|
||||
configuration does not create another model scheduler, queue, or retry policy.
|
||||
|
||||
The artifact-neutral producer-attempt state machine owns that shared budget,
|
||||
attempt provenance, semantic-correction material, and terminal-policy
|
||||
selection. It accepts producer and complete-validation closures, so artifact
|
||||
materialization, cache handling, checkpoints, and debug output stay at the
|
||||
operation boundary. It distinguishes operational, structural, module-requested,
|
||||
and validator-semantic retries. Model feedback from either semantic source is
|
||||
available only for a valid latest `single_response_v1` candidate. A
|
||||
deterministic or no-model validator rejection instead settles the semantic
|
||||
policy immediately, while a feedback-free module directive remains an ordinary
|
||||
fresh retry. Structural-output errors alone use the structural policy, and
|
||||
validation failure without rejection settles the validator-failure policy
|
||||
without regenerating the producer.
|
||||
## Handoffs And Reuse Eligibility
|
||||
|
||||
Chunk planning uses this state machine for generated plans. A rejected or
|
||||
validation-incomplete automatic cache hit is not model material and therefore
|
||||
falls through to a fresh initial generation at producer attempt one; it neither
|
||||
receives a correction, consumes retry budget, promotes cached-candidate
|
||||
diagnostics, nor overwrites the stored record. An incomplete cache validation
|
||||
under `fail_run` terminates instead. Only a newly generated, completely
|
||||
validated plan is published to the chunk-plan store. Rejected plans never
|
||||
advance, and validation-incomplete plans remain unpublishable.
|
||||
[Generated handoffs](../../internal/framework/pipeline/handoff.go) locate one
|
||||
earlier normalized producer, decode/re-encode through its registered codec, check
|
||||
full artifact/schema/media compatibility, and clone bytes into the consumer's
|
||||
declared reference set. Valid qualified fallback is usable current-run context;
|
||||
it does not become ordinary acceptance through a handoff.
|
||||
|
||||
After terminal lane work, the runner assembles manifest provenance, normalized
|
||||
artifacts, rejections, final grouped diagnostics, and an optional accepted chunk map. When an
|
||||
output policy selected evidence lanes, it decodes accepted serialized normalize
|
||||
outputs through their registered codecs and invokes the prepared typed
|
||||
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
|
||||
also used after normalized-checkpoint reuse, so no second typed output channel
|
||||
is retained. The runner passes the resulting owned artifact to the output
|
||||
encoder, which returns logical files and does not choose a physical directory.
|
||||
The CLI publishes those files only after the runner returns without a framework
|
||||
error. Logical file names and schemas are defined by the [output integration
|
||||
contracts](../integrations/).
|
||||
Eligibility travels separately from artifact bytes:
|
||||
|
||||
For every completed producer disposition, the runner projects one bounded
|
||||
validation summary to the manifest, the affected rejection when present, and
|
||||
the CLI result receipt. The summary records status, configured-order rejecting
|
||||
validators and reason codes, incomplete validators, producer-attempt count,
|
||||
and terminal action. It contains no operator message, correction guidance, or
|
||||
model response. `complete`, `rejected`, and `incomplete` describe the final
|
||||
candidate disposition; a run-level `incomplete` status indicates at least one
|
||||
current-run output advanced under `warn_continue`.
|
||||
- A chunk fallback disables reuse for all derived lane work.
|
||||
- An extract fallback disables reuse for its dependent merge/normalization.
|
||||
- A fallback-derived generated reference disables reuse from the consuming
|
||||
operation onward; unrelated siblings and earlier accepted work remain eligible.
|
||||
|
||||
## Checkpoint And Debug Hooks
|
||||
An emitting fallback increments the run count once. Merely consuming its data or
|
||||
inheriting ineligibility adds no fallback. These are transient dependency facts,
|
||||
not persisted fingerprints or a second recovery protocol.
|
||||
|
||||
The runner receives checkpoint and debug interfaces rather than roots. It
|
||||
records workflow transitions and reuse decisions through the supplied
|
||||
collaborators, and clones reusable artifacts before they re-enter normal typed
|
||||
handoff. Generated-reference dependencies participate in checkpoint decisions.
|
||||
Selective recomputation can require a canonical accepted normalized predecessor
|
||||
before a dependent lane starts.
|
||||
[Typed cache bindings](../../internal/framework/pipeline/runner_cache.go) use the
|
||||
registered durable codec and available exact-type evidence projection for
|
||||
current-source usability. [Chunk planning](../../internal/framework/pipeline/runner_chunk_plan.go)
|
||||
uses the same producer cache with plan decoding/rematerialization. Hits do not
|
||||
run semantic assessment. Deterministic work has no synthetic request/cache key.
|
||||
[Run State Internals](state.md) owns prepared-handle/store mechanics and recovery.
|
||||
|
||||
The runner writes successful checkpoint artifacts only after complete accepted
|
||||
validation. Chunk plans follow the same rule for publication. A rejection,
|
||||
invalid structured response, or incomplete validation is never reusable state;
|
||||
the current run may still hand off an otherwise valid `warn_continue` result
|
||||
according to its terminal policy. The runner carries private reuse eligibility
|
||||
through extract, merge, normalize, and generated-reference handoff. Any stage
|
||||
derived from incomplete validation skips both checkpoint lookup and all
|
||||
checkpoint publication even when that stage's own validation completes.
|
||||
External references and fully validated generated references remain eligible.
|
||||
Attempt debug records retain safe kind,
|
||||
validator, repair-usage, policy, and terminal-decision provenance. Full
|
||||
assistant and correction content remains confined to the requested detailed
|
||||
LLM trace.
|
||||
## Output And Capture Boundaries
|
||||
|
||||
Debug recording is attempt-scoped and application-owned. A failure to persist
|
||||
required debug data is a framework error. State roots, persistence, reason-code
|
||||
meanings, resume, and cleanup are intentionally owned by
|
||||
[Run State Internals](state.md) and [Operations](../operations.md).
|
||||
Extract-validator trace scopes include the current chunk ordinal so concurrent
|
||||
chunks cannot overwrite one another's validator attempts or LLM artifacts.
|
||||
[Runner](../../internal/framework/pipeline/runner.go) parses/validates source,
|
||||
materializes the final plan, runs ordered steps, aggregates diagnostics, and
|
||||
builds optional chunk-map/evidence values. Normalized values are codec-checked
|
||||
before handoff/output. Evidence reconstruction also works from reused artifacts;
|
||||
it projects selected citations in source order, not a second typed output channel.
|
||||
[Evidence internals](../../internal/framework/pipeline/evidence_output.go) and
|
||||
the [evidence contract](../integrations/evidence-context.md) define those owners.
|
||||
|
||||
## Invariants To Preserve
|
||||
The encoder receives owned normalized/auxiliary values and returns logical files.
|
||||
It does not choose a directory, publish a report, or serialize execution ledgers.
|
||||
[JSON output](../integrations/json-output.md) owns logical paths and payloads;
|
||||
[State Internals](state.md) owns physical publication and failure retention.
|
||||
|
||||
- The six fixed stages remain explicit; a pipeline is not a general DAG.
|
||||
- Resolution and preparation reject statically discoverable incompatibility
|
||||
before parsing or execution.
|
||||
- Every typed lane uses one compatible artifact kind, codec, and exact Go type.
|
||||
- Generated references come only from one earlier accepted normalized producer
|
||||
and carry canonical identity rather than an unverified value.
|
||||
- Rejections are recorded outcomes; framework errors cancel derived work and
|
||||
prevent output encoding.
|
||||
- Public ordering and selected errors are independent of goroutine completion
|
||||
order.
|
||||
- Pipeline modules receive collaborators and data, never CLI streams or
|
||||
physical output, cache, or debug roots.
|
||||
Optional scoped evidence records inputs, candidates, findings, outcomes, and
|
||||
cache decisions. Native observers enroll actual executions once in one invocation
|
||||
group, preserving unknown usage and terminal coverage. Capture failures are
|
||||
ordinary boundary failures, never semantic feedback or reasons to rerun a model.
|
||||
Debug layout is private; it is not a recovery source or consumer protocol.
|
||||
|
||||
## Focused Tests
|
||||
## Focused Verification
|
||||
|
||||
- **internal/framework/pipeline/profile_test.go** and
|
||||
**typed_resolution_test.go** cover resolution, defaults, ordered steps,
|
||||
compatibility, validators, references, and resolved identity.
|
||||
- **internal/framework/pipeline/preparation_test.go** covers complete
|
||||
construction before execution and contextual construction failures.
|
||||
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
|
||||
cover external materialization, generated references, provenance, and typed
|
||||
producer checks.
|
||||
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
|
||||
execution, ordered steps, stable error selection, rejections, and
|
||||
cancellation.
|
||||
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
|
||||
**runner_typed_checkpoint_test.go**, and
|
||||
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
|
||||
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
|
||||
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
|
||||
- [Resolution](../../internal/framework/pipeline/typed_resolution_test.go),
|
||||
[preparation](../../internal/framework/pipeline/preparation_test.go), and
|
||||
[references](../../internal/framework/pipeline/references_test.go) cover static
|
||||
compatibility and construction/materialization ownership.
|
||||
- [Assessment](../../internal/framework/pipeline/candidate_assessment_test.go)
|
||||
and [producer tests](../../internal/framework/pipeline/producer_test.go) cover
|
||||
isolation, whole-chain/final-turn assessment, guidance, native status, and cancellation.
|
||||
- [Runner operations](../../internal/framework/pipeline/runner_operation_test.go)
|
||||
cover no outer reassessment, deterministic errors, fallback counts/eligibility,
|
||||
and actual generated consumers.
|
||||
- [Worker tests](../../internal/framework/pipeline/runner_concurrency_test.go)
|
||||
and [ready-job tests](../../internal/framework/pipeline/ready_jobs_test.go)
|
||||
cover mixed jobs, stable order, serial liveness, nested admission, and draining.
|
||||
- [Producer reuse](../../internal/framework/pipeline/producer_cache_test.go),
|
||||
[runner reuse](../../internal/framework/pipeline/runner_cache_test.go), and
|
||||
[plan reuse](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
|
||||
initial keys, applied artifacts, no reassessment on hits, and current-source checks.
|
||||
- [Capture](../../internal/framework/pipeline/native_capture_test.go),
|
||||
[evidence](../../internal/framework/pipeline/evidence_output_test.go), and
|
||||
[assembled recovery](../../internal/cli/recovery_contract_test.go) cover
|
||||
native facts, output reconstruction, and cross-boundary reuse/failure.
|
||||
|
||||
Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
|
||||
pipeline boundary. Use the more focused tests above while iterating.
|
||||
Use focused pipeline and CLI suites for boundary changes.
|
||||
[Development](../development.md#validation) owns repository-wide checks.
|
||||
|
||||
@@ -1,165 +1,193 @@
|
||||
# Run State Internals
|
||||
|
||||
LLM debug response artifacts and their safe per-call references retain an
|
||||
optional terminal execution summary when one is available. This metadata is
|
||||
diagnostic only and does not change receipts, checkpoints, or aggregate usage.
|
||||
This guide explains physical cache, capture, and publication collaborators.
|
||||
[Operations](../operations.md) owns layout, permissions, retention, and recovery
|
||||
procedures; [Configuration](../config.md#concurrency-output-cache-and-debug) owns
|
||||
settings/defaults; [CLI Reference](../cli.md#completion-report) owns reporting.
|
||||
[Pipeline Internals](pipeline.md) owns assessment, workers, and dependency eligibility.
|
||||
|
||||
This document describes the implementation collaborators behind output, cache,
|
||||
and debug state. User-visible fields belong in [Configuration](../config.md),
|
||||
and physical layout, retention, recovery, reason codes, and cleanup belong in
|
||||
[Operations](../operations.md).
|
||||
## Composition And Independent State
|
||||
|
||||
## Composition
|
||||
[The run handler](../../internal/cli/pipeline_run.go) is the physical-path
|
||||
composition root. It obtains loaded configuration and a safe run identity,
|
||||
allocates requested capture before engine/preflight, resolves/materializes/
|
||||
prepares, and then constructs the cache/recorder collaborators for the runner.
|
||||
Shared inspection loaders create no run storage.
|
||||
|
||||
`internal/cli` is the only physical-path composition root. It resolves the
|
||||
effective configuration, selects exact roots, allocates requested debug bundles,
|
||||
constructs cache collaborators, writes logical output files, and reports paths.
|
||||
Pipeline modules receive interfaces and request data, never output, cache, or
|
||||
debug roots.
|
||||
Modules receive explicit operation/cache/evidence interfaces and data, not roots
|
||||
or CommandKit reporters. Output, accepted-result cache, and optional capture
|
||||
remain independent. None is a required metadata bundle for another.
|
||||
In-memory normalized artifacts support handoffs/evidence; they are not a
|
||||
persisted resume ledger.
|
||||
|
||||
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
|
||||
recorder only when recording is enabled and a checkpoint loader only for a
|
||||
resume invocation. It allocates debug state only after a safe run identity has
|
||||
been generated and only when debug capture was requested. These choices keep
|
||||
the three state families independently composable.
|
||||
## Prepared Result Reuse
|
||||
|
||||
## Output And Cache
|
||||
[RunProducer](../../internal/framework/pipeline/producer.go) owns native request
|
||||
selection and cache interaction for an eligible model operation:
|
||||
|
||||
The pipeline runner returns logical output files. After validating every
|
||||
logical name, the CLI exclusively creates the run directory beneath the
|
||||
selected output root and performs confined, atomic file writes within it.
|
||||
The runner supplies an accepted chunk map as an optional, defensively owned
|
||||
output-request artifact. The JSON encoder alone decides whether its explicit
|
||||
option writes the map and optional index descriptor; neither the map payload
|
||||
nor its annotations are copied into the run manifest. The durable fields are
|
||||
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
1. Prepare the initial request once with `PrepareExecution`, retain its detached
|
||||
details and opaque `MessageContentHash`.
|
||||
2. For a read-enabled selection, decode a found entry through the expected
|
||||
durable codec and current-source usability checks. On a usable hit, discard
|
||||
the prepared handle and return the artifact without model work.
|
||||
3. On a miss, execute that exact handle with `RunPrepared`, then use ordinary
|
||||
interpretation, complete assessment, and native continuation.
|
||||
4. Store only final ordinary acceptance, encoded as the applied typed artifact
|
||||
or checked logical plan, under the initial key. Dispose of unused
|
||||
prepared handles on every exit.
|
||||
|
||||
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
|
||||
and atomic publication. Its store is constructed only when the selected mode is
|
||||
not `bypass`. It preserves rooted entry checks while creating new cache roots
|
||||
and source-digest directories with requested mode `0777`, then creates an
|
||||
atomic temporary file through rooted `O_CREATE|O_EXCL` with requested mode
|
||||
`0666` before sync and rename.
|
||||
[Typed bindings](../../internal/framework/pipeline/runner_cache.go) attach codec
|
||||
and source-evidence checks; [chunk-plan bindings](../../internal/framework/pipeline/runner_chunk_plan.go)
|
||||
decode and rematerialize against the current document. A hit has no ordinary
|
||||
semantic revalidation, native execution event, restored exchange, or historical
|
||||
usage. Normalizers store applied domain results, not reconciliation proposals.
|
||||
Deterministic operations stay uncached without manufactured requests.
|
||||
|
||||
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
|
||||
codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
|
||||
recording is enabled and constructs a loader only for a `--resume` invocation.
|
||||
The recorder writes manifest and payload JSON through
|
||||
`fileio.WriteJSONWithProcessPermissions`, whose confined atomic writer creates
|
||||
new directories with requested mode `0777` and temporary files with requested
|
||||
mode `0666` before publication.
|
||||
Identity incorporates explicit stable semantic fingerprints collected from
|
||||
prepared modules and validators in addition to configuration, input,
|
||||
references, runtime overrides, observed LLM profiles, and the LLM runtime's
|
||||
non-secret effective profile-source identity. A profile source change therefore
|
||||
causes a cold miss even when the configured profile ID remains unchanged.
|
||||
The serialized
|
||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||
they do not describe a current public state surface.
|
||||
The key is the native initial message-content value, not a configuration,
|
||||
policy, model/session, implementation, or source-only fingerprint. Rendered
|
||||
message changes can change selection; settings-only changes need not.
|
||||
Correction retains the initial key. Public reuse choices and their consequences
|
||||
belong in [Operations](../operations.md#accepted-result-cache-and-recovery).
|
||||
|
||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||
Accepted step and lane identities are encoded injectively before becoming
|
||||
filesystem path components, while ordinary safe identifiers retain their
|
||||
readable paths.
|
||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||
include the producer's artifact kind, complete schema identity, media type,
|
||||
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||
when progressively loading consumer stage checkpoints, so changed producer
|
||||
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||
Selective recomputation instead requires each unselected producer's accepted
|
||||
normalized artifact; invalid accepted state records its specific bounded reason
|
||||
and stops before the dependent. The selected step and its transitive dependents
|
||||
record `forced_recompute`.
|
||||
Reuse eligibility is separate from the selected cache mode. Fallback and actual
|
||||
dependent work perform neither lookup nor storage, even if they later produce
|
||||
otherwise accepted output. Bypass/ineligible work uses ordinary native execution
|
||||
without cache-specific split preparation. Capture reuses available prepared
|
||||
details; it never adds preparation to explain a hit.
|
||||
|
||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||
and may execute later lane stages after an earlier cache miss. Selective
|
||||
recomputation instead asks the loader for the required producer's accepted
|
||||
normalize artifact. That lookup reuses the existing normalize files, requires
|
||||
workspace schema v4 plus an exact non-empty invocation identity, and deliberately
|
||||
does not require extract or merge checkpoint files or dependency fingerprints.
|
||||
The runner performs canonical codec and producer-provenance validation before
|
||||
cloning the artifact into normal step output. Success restores only stored
|
||||
normalize diagnostics and emits one normalize decision; failure retains the
|
||||
files, records the decision, and stops without executing the producer or
|
||||
consumer.
|
||||
## Filesystem Store And Failure Handling
|
||||
|
||||
The loader assigns a typed category and reason code at each validation site;
|
||||
diagnostic prose is not classified after the fact. The runner then applies
|
||||
forced-execution policy, validates reusable artifact bytes through the prepared
|
||||
codec once, returns the canonical hydrated value to the stage, and records the
|
||||
final decision before enforcing a required-predecessor failure. That failure
|
||||
names only the step, lane, and stable reason code. Decision detail is selected
|
||||
from code-owned descriptions by reason code and then UTF-8 normalized and
|
||||
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
|
||||
codes remain intact through pipeline events and become strings only in manifest
|
||||
and debug-summary JSON.
|
||||
[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
|
||||
canonical operator-facing reason-code reference.
|
||||
[artifactcache.FilesystemStore](../../internal/framework/artifactcache/filesystem.go)
|
||||
implements the narrow `contracts.ArtifactCache` interface. Construction retains
|
||||
the exact selected root without creating it. Lookup checks one confined regular
|
||||
JSON entry; missing data is an ordinary miss. Stored bytes are the unchanged
|
||||
accepted artifact, without provenance/execution envelopes or alternate key
|
||||
dimensions. Physical placement belongs in
|
||||
[Operations](../operations.md#state-surfaces).
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. Its exact-mode writer remains available to output and debug
|
||||
callers; its process-permission writer is used by checkpoints. The chunk-plan
|
||||
store retains its stronger rooted entry validation. [Operations](../operations.md#cache-provisioning)
|
||||
owns cache mode and provisioning guidance.
|
||||
The store honors the resolved read/write mode. Refresh does not read or remove
|
||||
previous acceptance in anticipation of execution. Only a newly supplied accepted
|
||||
value can replace it. Failed refresh execution or a failed cache write preserves
|
||||
the old entry.
|
||||
Legacy data is not scanned, converted, rewritten, or deleted.
|
||||
|
||||
The CLI constructs selective-recomputation policy from resolved generated
|
||||
artifact dependencies. It forces the selected step and transitive consumers,
|
||||
while marking unforced producers as required reusable inputs. The runner owns
|
||||
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
|
||||
owns the operator workflow and stable reason-code meanings.
|
||||
Publication uses [fileio](../../internal/core/fileio/fileio.go)'s confined atomic
|
||||
process-permission writer. New cache objects use process/deployment creation
|
||||
permissions; there is no permission convergence on an existing root.
|
||||
[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md) owns that decision.
|
||||
|
||||
## Debug Bundles
|
||||
[Cache integration](../../internal/framework/pipeline/producer_cache.go) treats
|
||||
unreadable/malformed/codec-unusable entries as diagnostic misses. Store failures
|
||||
become scoped process warnings without failing successful production or calling
|
||||
the provider again. Artifact encoding, caller cancellation, requested evidence,
|
||||
and observer failures retain their own error boundaries; they are not swallowed
|
||||
as best-effort storage failures.
|
||||
|
||||
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
||||
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
|
||||
resolution, run, final grouped diagnostic, and failure artifacts. `internal/framework/debug`
|
||||
implements the pipeline-facing trace recorder under the trace root.
|
||||
Accepted predecessors can survive downstream failure and be reused by an ordinary
|
||||
rerun. There is no checkpoint loader, resume barrier, forced-recompute policy, or
|
||||
canonical partition selected solely by source digest. New message selections
|
||||
populate independently; an accepted plan is rematerialized, not restored with a
|
||||
historic execution claim.
|
||||
|
||||
The CLI allocates a bundle before pipeline resolution and treats requested
|
||||
summary or trace persistence failures as command failures. The pipeline's debug
|
||||
boundaries redact sensitive metadata and credential-shaped bytes while allowing
|
||||
application-owned trace material. Debug data is never a checkpoint source or
|
||||
cache input.
|
||||
## Artifact Publication And Native Reporting
|
||||
|
||||
Generated reference bytes exist only in cloned operation requests and are not
|
||||
written as paths into checkpoints, manifests, or debug summaries. Those state
|
||||
surfaces retain canonical identities and bounded producer provenance so that a
|
||||
resume decision can be explained without copying generated campaign content.
|
||||
The runner returns logical deliverable files only after successful execution/
|
||||
encoding. The [JSON encoder](../../internal/modules/generic/output/json/encoder.go)
|
||||
owns domain-file selection and optional auxiliary inclusion.
|
||||
[JSON output](../integrations/json-output.md),
|
||||
[chunk maps](../integrations/chunk-map.md), and
|
||||
[evidence context](../integrations/evidence-context.md) own logical paths/schemas,
|
||||
not the store or recorder.
|
||||
|
||||
After allocation, one CLI-owned state value accumulates the known report paths,
|
||||
pipeline outcome counts, and validation status. A single guarded terminalization
|
||||
operation writes the success report, or makes one attempt each to write the
|
||||
failure report and error log. Terminal persistence failures are reported
|
||||
separately and never replace the command's primary error.
|
||||
[writeOutputFiles](../../internal/cli/run_output.go) validates every logical
|
||||
path before allocating the run directory, refuses an existing destination, and
|
||||
writes files with confined atomic helpers. This is per-file publication, not
|
||||
an all-files transaction. A later write can leave a partial new directory;
|
||||
existing siblings are preserved. No rollback deletes available artifacts.
|
||||
|
||||
## Invariants To Preserve
|
||||
After complete publication, the run handler finalizes requested capture before
|
||||
[reportRunResult](../../internal/cli/run_result.go) submits the small native
|
||||
completion value. The report is not automatically persisted or required to decode
|
||||
domain files. No index, manifest, receipt, or process-diagnostic sidecar is emitted.
|
||||
Artifact count counts returned deliverables, including selected available auxiliaries,
|
||||
not cache/debug/intermediate files.
|
||||
|
||||
- Modules receive state collaborators and request data, never physical roots.
|
||||
- Output logical paths are validated before a run directory is allocated, and
|
||||
files are atomically written within that directory.
|
||||
- Chunk-plan publication occurs only for accepted plans; bypass does not
|
||||
construct or touch a plan store.
|
||||
- Checkpoint recording and checkpoint loading remain separate collaborators.
|
||||
- Debug state is opt-in, is not cache input, and terminal reporting does not
|
||||
obscure the command's primary failure.
|
||||
Post-publication capture/render/delivery failure remains nonzero with available
|
||||
locations. Valid artifacts remain; model work and capture terminalization are not
|
||||
repeated. Native delivery evidence is separate from the operation cause, and
|
||||
failed stdout may be partial. Public outcomes and caller responsibilities belong
|
||||
in [CLI Reference](../cli.md#output-streams-and-exit-statuses) and the
|
||||
[subprocess guide](../consumers/subprocess.md).
|
||||
|
||||
## Tests To Inspect
|
||||
## Capture Lifecycle
|
||||
|
||||
- `internal/cli/run_contract_test.go`: command-owned state allocation,
|
||||
terminalization, and output/report boundaries.
|
||||
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
|
||||
and resume collaborator construction.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/cli/recompute_policy_test.go`: forced dependents and required
|
||||
reusable predecessors for selective recomputation.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration validation at the CLI boundary.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
|
||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
|
||||
reuse.
|
||||
- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
|
||||
publication, and permissions.
|
||||
[Debug selection](../../internal/cli/run_debug.go) preserves native boolean and
|
||||
supplied directory semantics independently of verbosity. The run path alone
|
||||
calls [debug.Start](../../internal/framework/debug/recorder.go).
|
||||
[Bundle allocation](../../internal/core/debugbundle/bundle.go) validates identity,
|
||||
creates a new absolute per-run directory, and atomically updates a relative
|
||||
latest link at start. Failure can return an available directory for explanation/
|
||||
retention. Disabled capture creates nothing; completion never retargets the link.
|
||||
|
||||
One synchronized recorder writes evidence incrementally, retaining lifecycle
|
||||
state and the first recording error rather than accumulating whole-run content.
|
||||
Callback failures are exposed at the next boundary that can return an error.
|
||||
`Finalize` attempts terminal evidence once, joins primary and capture causes,
|
||||
and retains its result. Repeated calls do not rewrite evidence or retarget latest.
|
||||
A later presentation failure does not change capture into another terminal run.
|
||||
Allocation/link/record/finalization failures never trigger model retries or a
|
||||
fallback directory. Available files are retained.
|
||||
|
||||
## Native And Application Evidence
|
||||
|
||||
[Run capture](../../internal/cli/run_capture.go) snapshots the original secret-aware
|
||||
application/PromptKit configuration and source evidence. Scoped pipeline capture
|
||||
records source/reference/incoming artifacts, normalization originals/mappings/
|
||||
proposals, candidates, complete findings/guidance, fallback/deterministic outcomes,
|
||||
and cache selection/current artifact bytes.
|
||||
|
||||
[NativeCapture](../../internal/framework/pipeline/native_capture.go) composes
|
||||
existing observers and preparation callbacks. Preparation callbacks retain
|
||||
detached details cheaply; ordinary boundaries write prepared contracts, native
|
||||
results, raw response text, and full readable exchanges. Events are written
|
||||
incrementally. Scope includes operation/step/lane/chunk/validator identity to
|
||||
avoid concurrent evidence collisions.
|
||||
|
||||
One invocation `ExecutionGroup` enrolls actual initial, continuation, and nested
|
||||
validator executions once. Native identity, terminal coverage, and unknown usage
|
||||
remain native facts, not a copied profile/attempt accounting model.
|
||||
Hits can retain prepared evidence without execution or usage. Result-less failures
|
||||
retain availability/cause evidence; undelivered responses are explicitly unassessed.
|
||||
|
||||
Native credential omissions and ConfigKit marked-secret paths are preserved.
|
||||
Arbitrary source/prompt/response text is not heuristically rewritten or made safe
|
||||
for general logging. Exact evidence names/nesting/serialized forms are private,
|
||||
not a public protocol or reconstruction requirement. Debug is neither cache
|
||||
input nor an artifact-discovery interface; [Operations](../operations.md#debug-bundles)
|
||||
owns handling.
|
||||
|
||||
## Focused Verification
|
||||
|
||||
- [Filesystem cache](../../internal/framework/artifactcache/filesystem_test.go)
|
||||
and [permissions](../../internal/framework/artifactcache/filesystem_permissions_unix_test.go)
|
||||
cover raw bytes, modes, atomic replacement, diagnostics, confinement, deployment
|
||||
permissions, and unchanged legacy files.
|
||||
- [Producer cache](../../internal/framework/pipeline/producer_cache_test.go),
|
||||
[typed runner cache](../../internal/framework/pipeline/runner_cache_test.go), and
|
||||
[plan cache](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
|
||||
initial handles/keys, final acceptance, current-source checks, and failed refresh.
|
||||
- [Recorder](../../internal/framework/debug/recorder_test.go) and
|
||||
[allocation](../../internal/core/debugbundle/bundle_test.go) cover incremental
|
||||
failure retention, concurrent starts, and one terminalization.
|
||||
- [Native evidence](../../internal/framework/pipeline/native_capture_test.go) and
|
||||
[configuration capture](../../internal/cli/run_capture_test.go) cover available
|
||||
content, exact sensitivity, observer composition, and no duplicate preparation.
|
||||
- [Output writes](../../internal/cli/run_id_test.go) and
|
||||
[native result boundaries](../../internal/cli/run_result_command_test.go) cover
|
||||
collisions, partial directories, capture/presentation failure, and retained causes.
|
||||
- [Assembled recovery](../../internal/cli/recovery_contract_test.go) covers
|
||||
downstream fix/rerun, initial-key correction reuse, qualified fallback, actual
|
||||
generated consumers, and debug on hits/failure.
|
||||
|
||||
Use focused pipeline/cache/recorder/CLI checks for state-boundary changes.
|
||||
[Development](../development.md#validation) owns wider validation.
|
||||
|
||||
Reference in New Issue
Block a user