Refresh pipeline execution and run state internal guides
This commit is contained in:
@@ -1,165 +1,193 @@
|
||||
# Run State Internals
|
||||
|
||||
LLM debug response artifacts and their safe per-call references retain an
|
||||
optional terminal execution summary when one is available. This metadata is
|
||||
diagnostic only and does not change receipts, checkpoints, or aggregate usage.
|
||||
This guide explains physical cache, capture, and publication collaborators.
|
||||
[Operations](../operations.md) owns layout, permissions, retention, and recovery
|
||||
procedures; [Configuration](../config.md#concurrency-output-cache-and-debug) owns
|
||||
settings/defaults; [CLI Reference](../cli.md#completion-report) owns reporting.
|
||||
[Pipeline Internals](pipeline.md) owns assessment, workers, and dependency eligibility.
|
||||
|
||||
This document describes the implementation collaborators behind output, cache,
|
||||
and debug state. User-visible fields belong in [Configuration](../config.md),
|
||||
and physical layout, retention, recovery, reason codes, and cleanup belong in
|
||||
[Operations](../operations.md).
|
||||
## Composition And Independent State
|
||||
|
||||
## Composition
|
||||
[The run handler](../../internal/cli/pipeline_run.go) is the physical-path
|
||||
composition root. It obtains loaded configuration and a safe run identity,
|
||||
allocates requested capture before engine/preflight, resolves/materializes/
|
||||
prepares, and then constructs the cache/recorder collaborators for the runner.
|
||||
Shared inspection loaders create no run storage.
|
||||
|
||||
`internal/cli` is the only physical-path composition root. It resolves the
|
||||
effective configuration, selects exact roots, allocates requested debug bundles,
|
||||
constructs cache collaborators, writes logical output files, and reports paths.
|
||||
Pipeline modules receive interfaces and request data, never output, cache, or
|
||||
debug roots.
|
||||
Modules receive explicit operation/cache/evidence interfaces and data, not roots
|
||||
or CommandKit reporters. Output, accepted-result cache, and optional capture
|
||||
remain independent. None is a required metadata bundle for another.
|
||||
In-memory normalized artifacts support handoffs/evidence; they are not a
|
||||
persisted resume ledger.
|
||||
|
||||
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
|
||||
recorder only when recording is enabled and a checkpoint loader only for a
|
||||
resume invocation. It allocates debug state only after a safe run identity has
|
||||
been generated and only when debug capture was requested. These choices keep
|
||||
the three state families independently composable.
|
||||
## Prepared Result Reuse
|
||||
|
||||
## Output And Cache
|
||||
[RunProducer](../../internal/framework/pipeline/producer.go) owns native request
|
||||
selection and cache interaction for an eligible model operation:
|
||||
|
||||
The pipeline runner returns logical output files. After validating every
|
||||
logical name, the CLI exclusively creates the run directory beneath the
|
||||
selected output root and performs confined, atomic file writes within it.
|
||||
The runner supplies an accepted chunk map as an optional, defensively owned
|
||||
output-request artifact. The JSON encoder alone decides whether its explicit
|
||||
option writes the map and optional index descriptor; neither the map payload
|
||||
nor its annotations are copied into the run manifest. The durable fields are
|
||||
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
|
||||
1. Prepare the initial request once with `PrepareExecution`, retain its detached
|
||||
details and opaque `MessageContentHash`.
|
||||
2. For a read-enabled selection, decode a found entry through the expected
|
||||
durable codec and current-source usability checks. On a usable hit, discard
|
||||
the prepared handle and return the artifact without model work.
|
||||
3. On a miss, execute that exact handle with `RunPrepared`, then use ordinary
|
||||
interpretation, complete assessment, and native continuation.
|
||||
4. Store only final ordinary acceptance, encoded as the applied typed artifact
|
||||
or checked logical plan, under the initial key. Dispose of unused
|
||||
prepared handles on every exit.
|
||||
|
||||
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
|
||||
and atomic publication. Its store is constructed only when the selected mode is
|
||||
not `bypass`. It preserves rooted entry checks while creating new cache roots
|
||||
and source-digest directories with requested mode `0777`, then creates an
|
||||
atomic temporary file through rooted `O_CREATE|O_EXCL` with requested mode
|
||||
`0666` before sync and rename.
|
||||
[Typed bindings](../../internal/framework/pipeline/runner_cache.go) attach codec
|
||||
and source-evidence checks; [chunk-plan bindings](../../internal/framework/pipeline/runner_chunk_plan.go)
|
||||
decode and rematerialize against the current document. A hit has no ordinary
|
||||
semantic revalidation, native execution event, restored exchange, or historical
|
||||
usage. Normalizers store applied domain results, not reconciliation proposals.
|
||||
Deterministic operations stay uncached without manufactured requests.
|
||||
|
||||
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
|
||||
codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
|
||||
recording is enabled and constructs a loader only for a `--resume` invocation.
|
||||
The recorder writes manifest and payload JSON through
|
||||
`fileio.WriteJSONWithProcessPermissions`, whose confined atomic writer creates
|
||||
new directories with requested mode `0777` and temporary files with requested
|
||||
mode `0666` before publication.
|
||||
Identity incorporates explicit stable semantic fingerprints collected from
|
||||
prepared modules and validators in addition to configuration, input,
|
||||
references, runtime overrides, observed LLM profiles, and the LLM runtime's
|
||||
non-secret effective profile-source identity. A profile source change therefore
|
||||
causes a cold miss even when the configured profile ID remains unchanged.
|
||||
The serialized
|
||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||
they do not describe a current public state surface.
|
||||
The key is the native initial message-content value, not a configuration,
|
||||
policy, model/session, implementation, or source-only fingerprint. Rendered
|
||||
message changes can change selection; settings-only changes need not.
|
||||
Correction retains the initial key. Public reuse choices and their consequences
|
||||
belong in [Operations](../operations.md#accepted-result-cache-and-recovery).
|
||||
|
||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||
Accepted step and lane identities are encoded injectively before becoming
|
||||
filesystem path components, while ordinary safe identifiers retain their
|
||||
readable paths.
|
||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||
include the producer's artifact kind, complete schema identity, media type,
|
||||
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||
when progressively loading consumer stage checkpoints, so changed producer
|
||||
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||
Selective recomputation instead requires each unselected producer's accepted
|
||||
normalized artifact; invalid accepted state records its specific bounded reason
|
||||
and stops before the dependent. The selected step and its transitive dependents
|
||||
record `forced_recompute`.
|
||||
Reuse eligibility is separate from the selected cache mode. Fallback and actual
|
||||
dependent work perform neither lookup nor storage, even if they later produce
|
||||
otherwise accepted output. Bypass/ineligible work uses ordinary native execution
|
||||
without cache-specific split preparation. Capture reuses available prepared
|
||||
details; it never adds preparation to explain a hit.
|
||||
|
||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||
and may execute later lane stages after an earlier cache miss. Selective
|
||||
recomputation instead asks the loader for the required producer's accepted
|
||||
normalize artifact. That lookup reuses the existing normalize files, requires
|
||||
workspace schema v4 plus an exact non-empty invocation identity, and deliberately
|
||||
does not require extract or merge checkpoint files or dependency fingerprints.
|
||||
The runner performs canonical codec and producer-provenance validation before
|
||||
cloning the artifact into normal step output. Success restores only stored
|
||||
normalize diagnostics and emits one normalize decision; failure retains the
|
||||
files, records the decision, and stops without executing the producer or
|
||||
consumer.
|
||||
## Filesystem Store And Failure Handling
|
||||
|
||||
The loader assigns a typed category and reason code at each validation site;
|
||||
diagnostic prose is not classified after the fact. The runner then applies
|
||||
forced-execution policy, validates reusable artifact bytes through the prepared
|
||||
codec once, returns the canonical hydrated value to the stage, and records the
|
||||
final decision before enforcing a required-predecessor failure. That failure
|
||||
names only the step, lane, and stable reason code. Decision detail is selected
|
||||
from code-owned descriptions by reason code and then UTF-8 normalized and
|
||||
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
|
||||
codes remain intact through pipeline events and become strings only in manifest
|
||||
and debug-summary JSON.
|
||||
[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
|
||||
canonical operator-facing reason-code reference.
|
||||
[artifactcache.FilesystemStore](../../internal/framework/artifactcache/filesystem.go)
|
||||
implements the narrow `contracts.ArtifactCache` interface. Construction retains
|
||||
the exact selected root without creating it. Lookup checks one confined regular
|
||||
JSON entry; missing data is an ordinary miss. Stored bytes are the unchanged
|
||||
accepted artifact, without provenance/execution envelopes or alternate key
|
||||
dimensions. Physical placement belongs in
|
||||
[Operations](../operations.md#state-surfaces).
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. Its exact-mode writer remains available to output and debug
|
||||
callers; its process-permission writer is used by checkpoints. The chunk-plan
|
||||
store retains its stronger rooted entry validation. [Operations](../operations.md#cache-provisioning)
|
||||
owns cache mode and provisioning guidance.
|
||||
The store honors the resolved read/write mode. Refresh does not read or remove
|
||||
previous acceptance in anticipation of execution. Only a newly supplied accepted
|
||||
value can replace it. Failed refresh execution or a failed cache write preserves
|
||||
the old entry.
|
||||
Legacy data is not scanned, converted, rewritten, or deleted.
|
||||
|
||||
The CLI constructs selective-recomputation policy from resolved generated
|
||||
artifact dependencies. It forces the selected step and transitive consumers,
|
||||
while marking unforced producers as required reusable inputs. The runner owns
|
||||
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
|
||||
owns the operator workflow and stable reason-code meanings.
|
||||
Publication uses [fileio](../../internal/core/fileio/fileio.go)'s confined atomic
|
||||
process-permission writer. New cache objects use process/deployment creation
|
||||
permissions; there is no permission convergence on an existing root.
|
||||
[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md) owns that decision.
|
||||
|
||||
## Debug Bundles
|
||||
[Cache integration](../../internal/framework/pipeline/producer_cache.go) treats
|
||||
unreadable/malformed/codec-unusable entries as diagnostic misses. Store failures
|
||||
become scoped process warnings without failing successful production or calling
|
||||
the provider again. Artifact encoding, caller cancellation, requested evidence,
|
||||
and observer failures retain their own error boundaries; they are not swallowed
|
||||
as best-effort storage failures.
|
||||
|
||||
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
||||
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
|
||||
resolution, run, final grouped diagnostic, and failure artifacts. `internal/framework/debug`
|
||||
implements the pipeline-facing trace recorder under the trace root.
|
||||
Accepted predecessors can survive downstream failure and be reused by an ordinary
|
||||
rerun. There is no checkpoint loader, resume barrier, forced-recompute policy, or
|
||||
canonical partition selected solely by source digest. New message selections
|
||||
populate independently; an accepted plan is rematerialized, not restored with a
|
||||
historic execution claim.
|
||||
|
||||
The CLI allocates a bundle before pipeline resolution and treats requested
|
||||
summary or trace persistence failures as command failures. The pipeline's debug
|
||||
boundaries redact sensitive metadata and credential-shaped bytes while allowing
|
||||
application-owned trace material. Debug data is never a checkpoint source or
|
||||
cache input.
|
||||
## Artifact Publication And Native Reporting
|
||||
|
||||
Generated reference bytes exist only in cloned operation requests and are not
|
||||
written as paths into checkpoints, manifests, or debug summaries. Those state
|
||||
surfaces retain canonical identities and bounded producer provenance so that a
|
||||
resume decision can be explained without copying generated campaign content.
|
||||
The runner returns logical deliverable files only after successful execution/
|
||||
encoding. The [JSON encoder](../../internal/modules/generic/output/json/encoder.go)
|
||||
owns domain-file selection and optional auxiliary inclusion.
|
||||
[JSON output](../integrations/json-output.md),
|
||||
[chunk maps](../integrations/chunk-map.md), and
|
||||
[evidence context](../integrations/evidence-context.md) own logical paths/schemas,
|
||||
not the store or recorder.
|
||||
|
||||
After allocation, one CLI-owned state value accumulates the known report paths,
|
||||
pipeline outcome counts, and validation status. A single guarded terminalization
|
||||
operation writes the success report, or makes one attempt each to write the
|
||||
failure report and error log. Terminal persistence failures are reported
|
||||
separately and never replace the command's primary error.
|
||||
[writeOutputFiles](../../internal/cli/run_output.go) validates every logical
|
||||
path before allocating the run directory, refuses an existing destination, and
|
||||
writes files with confined atomic helpers. This is per-file publication, not
|
||||
an all-files transaction. A later write can leave a partial new directory;
|
||||
existing siblings are preserved. No rollback deletes available artifacts.
|
||||
|
||||
## Invariants To Preserve
|
||||
After complete publication, the run handler finalizes requested capture before
|
||||
[reportRunResult](../../internal/cli/run_result.go) submits the small native
|
||||
completion value. The report is not automatically persisted or required to decode
|
||||
domain files. No index, manifest, receipt, or process-diagnostic sidecar is emitted.
|
||||
Artifact count counts returned deliverables, including selected available auxiliaries,
|
||||
not cache/debug/intermediate files.
|
||||
|
||||
- Modules receive state collaborators and request data, never physical roots.
|
||||
- Output logical paths are validated before a run directory is allocated, and
|
||||
files are atomically written within that directory.
|
||||
- Chunk-plan publication occurs only for accepted plans; bypass does not
|
||||
construct or touch a plan store.
|
||||
- Checkpoint recording and checkpoint loading remain separate collaborators.
|
||||
- Debug state is opt-in, is not cache input, and terminal reporting does not
|
||||
obscure the command's primary failure.
|
||||
Post-publication capture/render/delivery failure remains nonzero with available
|
||||
locations. Valid artifacts remain; model work and capture terminalization are not
|
||||
repeated. Native delivery evidence is separate from the operation cause, and
|
||||
failed stdout may be partial. Public outcomes and caller responsibilities belong
|
||||
in [CLI Reference](../cli.md#output-streams-and-exit-statuses) and the
|
||||
[subprocess guide](../consumers/subprocess.md).
|
||||
|
||||
## Tests To Inspect
|
||||
## Capture Lifecycle
|
||||
|
||||
- `internal/cli/run_contract_test.go`: command-owned state allocation,
|
||||
terminalization, and output/report boundaries.
|
||||
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
|
||||
and resume collaborator construction.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/cli/recompute_policy_test.go`: forced dependents and required
|
||||
reusable predecessors for selective recomputation.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration validation at the CLI boundary.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
|
||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
|
||||
reuse.
|
||||
- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
|
||||
publication, and permissions.
|
||||
[Debug selection](../../internal/cli/run_debug.go) preserves native boolean and
|
||||
supplied directory semantics independently of verbosity. The run path alone
|
||||
calls [debug.Start](../../internal/framework/debug/recorder.go).
|
||||
[Bundle allocation](../../internal/core/debugbundle/bundle.go) validates identity,
|
||||
creates a new absolute per-run directory, and atomically updates a relative
|
||||
latest link at start. Failure can return an available directory for explanation/
|
||||
retention. Disabled capture creates nothing; completion never retargets the link.
|
||||
|
||||
One synchronized recorder writes evidence incrementally, retaining lifecycle
|
||||
state and the first recording error rather than accumulating whole-run content.
|
||||
Callback failures are exposed at the next boundary that can return an error.
|
||||
`Finalize` attempts terminal evidence once, joins primary and capture causes,
|
||||
and retains its result. Repeated calls do not rewrite evidence or retarget latest.
|
||||
A later presentation failure does not change capture into another terminal run.
|
||||
Allocation/link/record/finalization failures never trigger model retries or a
|
||||
fallback directory. Available files are retained.
|
||||
|
||||
## Native And Application Evidence
|
||||
|
||||
[Run capture](../../internal/cli/run_capture.go) snapshots the original secret-aware
|
||||
application/PromptKit configuration and source evidence. Scoped pipeline capture
|
||||
records source/reference/incoming artifacts, normalization originals/mappings/
|
||||
proposals, candidates, complete findings/guidance, fallback/deterministic outcomes,
|
||||
and cache selection/current artifact bytes.
|
||||
|
||||
[NativeCapture](../../internal/framework/pipeline/native_capture.go) composes
|
||||
existing observers and preparation callbacks. Preparation callbacks retain
|
||||
detached details cheaply; ordinary boundaries write prepared contracts, native
|
||||
results, raw response text, and full readable exchanges. Events are written
|
||||
incrementally. Scope includes operation/step/lane/chunk/validator identity to
|
||||
avoid concurrent evidence collisions.
|
||||
|
||||
One invocation `ExecutionGroup` enrolls actual initial, continuation, and nested
|
||||
validator executions once. Native identity, terminal coverage, and unknown usage
|
||||
remain native facts, not a copied profile/attempt accounting model.
|
||||
Hits can retain prepared evidence without execution or usage. Result-less failures
|
||||
retain availability/cause evidence; undelivered responses are explicitly unassessed.
|
||||
|
||||
Native credential omissions and ConfigKit marked-secret paths are preserved.
|
||||
Arbitrary source/prompt/response text is not heuristically rewritten or made safe
|
||||
for general logging. Exact evidence names/nesting/serialized forms are private,
|
||||
not a public protocol or reconstruction requirement. Debug is neither cache
|
||||
input nor an artifact-discovery interface; [Operations](../operations.md#debug-bundles)
|
||||
owns handling.
|
||||
|
||||
## Focused Verification
|
||||
|
||||
- [Filesystem cache](../../internal/framework/artifactcache/filesystem_test.go)
|
||||
and [permissions](../../internal/framework/artifactcache/filesystem_permissions_unix_test.go)
|
||||
cover raw bytes, modes, atomic replacement, diagnostics, confinement, deployment
|
||||
permissions, and unchanged legacy files.
|
||||
- [Producer cache](../../internal/framework/pipeline/producer_cache_test.go),
|
||||
[typed runner cache](../../internal/framework/pipeline/runner_cache_test.go), and
|
||||
[plan cache](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
|
||||
initial handles/keys, final acceptance, current-source checks, and failed refresh.
|
||||
- [Recorder](../../internal/framework/debug/recorder_test.go) and
|
||||
[allocation](../../internal/core/debugbundle/bundle_test.go) cover incremental
|
||||
failure retention, concurrent starts, and one terminalization.
|
||||
- [Native evidence](../../internal/framework/pipeline/native_capture_test.go) and
|
||||
[configuration capture](../../internal/cli/run_capture_test.go) cover available
|
||||
content, exact sensitivity, observer composition, and no duplicate preparation.
|
||||
- [Output writes](../../internal/cli/run_id_test.go) and
|
||||
[native result boundaries](../../internal/cli/run_result_command_test.go) cover
|
||||
collisions, partial directories, capture/presentation failure, and retained causes.
|
||||
- [Assembled recovery](../../internal/cli/recovery_contract_test.go) covers
|
||||
downstream fix/rerun, initial-key correction reuse, qualified fallback, actual
|
||||
generated consumers, and debug on hits/failure.
|
||||
|
||||
Use focused pipeline/cache/recorder/CLI checks for state-boundary changes.
|
||||
[Development](../development.md#validation) owns wider validation.
|
||||
|
||||
Reference in New Issue
Block a user