Refresh pipeline execution and run state internal guides

This commit is contained in:
2026-11-22 06:45:28 +00:00
parent 5db09d0203
commit 41f6079040
3 changed files with 355 additions and 383 deletions

View File

@@ -1,165 +1,193 @@
# Run State Internals
LLM debug response artifacts and their safe per-call references retain an
optional terminal execution summary when one is available. This metadata is
diagnostic only and does not change receipts, checkpoints, or aggregate usage.
This guide explains physical cache, capture, and publication collaborators.
[Operations](../operations.md) owns layout, permissions, retention, and recovery
procedures; [Configuration](../config.md#concurrency-output-cache-and-debug) owns
settings/defaults; [CLI Reference](../cli.md#completion-report) owns reporting.
[Pipeline Internals](pipeline.md) owns assessment, workers, and dependency eligibility.
This document describes the implementation collaborators behind output, cache,
and debug state. User-visible fields belong in [Configuration](../config.md),
and physical layout, retention, recovery, reason codes, and cleanup belong in
[Operations](../operations.md).
## Composition And Independent State
## Composition
[The run handler](../../internal/cli/pipeline_run.go) is the physical-path
composition root. It obtains loaded configuration and a safe run identity,
allocates requested capture before engine/preflight, resolves/materializes/
prepares, and then constructs the cache/recorder collaborators for the runner.
Shared inspection loaders create no run storage.
`internal/cli` is the only physical-path composition root. It resolves the
effective configuration, selects exact roots, allocates requested debug bundles,
constructs cache collaborators, writes logical output files, and reports paths.
Pipeline modules receive interfaces and request data, never output, cache, or
debug roots.
Modules receive explicit operation/cache/evidence interfaces and data, not roots
or CommandKit reporters. Output, accepted-result cache, and optional capture
remain independent. None is a required metadata bundle for another.
In-memory normalized artifacts support handoffs/evidence; they are not a
persisted resume ledger.
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
recorder only when recording is enabled and a checkpoint loader only for a
resume invocation. It allocates debug state only after a safe run identity has
been generated and only when debug capture was requested. These choices keep
the three state families independently composable.
## Prepared Result Reuse
## Output And Cache
[RunProducer](../../internal/framework/pipeline/producer.go) owns native request
selection and cache interaction for an eligible model operation:
The pipeline runner returns logical output files. After validating every
logical name, the CLI exclusively creates the run directory beneath the
selected output root and performs confined, atomic file writes within it.
The runner supplies an accepted chunk map as an optional, defensively owned
output-request artifact. The JSON encoder alone decides whether its explicit
option writes the map and optional index descriptor; neither the map payload
nor its annotations are copied into the run manifest. The durable fields are
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
1. Prepare the initial request once with `PrepareExecution`, retain its detached
details and opaque `MessageContentHash`.
2. For a read-enabled selection, decode a found entry through the expected
durable codec and current-source usability checks. On a usable hit, discard
the prepared handle and return the artifact without model work.
3. On a miss, execute that exact handle with `RunPrepared`, then use ordinary
interpretation, complete assessment, and native continuation.
4. Store only final ordinary acceptance, encoded as the applied typed artifact
or checked logical plan, under the initial key. Dispose of unused
prepared handles on every exit.
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
and atomic publication. Its store is constructed only when the selected mode is
not `bypass`. It preserves rooted entry checks while creating new cache roots
and source-digest directories with requested mode `0777`, then creates an
atomic temporary file through rooted `O_CREATE|O_EXCL` with requested mode
`0666` before sync and rename.
[Typed bindings](../../internal/framework/pipeline/runner_cache.go) attach codec
and source-evidence checks; [chunk-plan bindings](../../internal/framework/pipeline/runner_chunk_plan.go)
decode and rematerialize against the current document. A hit has no ordinary
semantic revalidation, native execution event, restored exchange, or historical
usage. Normalizers store applied domain results, not reconciliation proposals.
Deterministic operations stay uncached without manufactured requests.
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
recording is enabled and constructs a loader only for a `--resume` invocation.
The recorder writes manifest and payload JSON through
`fileio.WriteJSONWithProcessPermissions`, whose confined atomic writer creates
new directories with requested mode `0777` and temporary files with requested
mode `0666` before publication.
Identity incorporates explicit stable semantic fingerprints collected from
prepared modules and validators in addition to configuration, input,
references, runtime overrides, observed LLM profiles, and the LLM runtime's
non-secret effective profile-source identity. A profile source change therefore
causes a cold miss even when the configured profile ID remains unchanged.
The serialized
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
they do not describe a current public state surface.
The key is the native initial message-content value, not a configuration,
policy, model/session, implementation, or source-only fingerprint. Rendered
message changes can change selection; settings-only changes need not.
Correction retains the initial key. Public reuse choices and their consequences
belong in [Operations](../operations.md#accepted-result-cache-and-recovery).
Ordered-step lane checkpoints include the step identity in their storage scope.
Accepted step and lane identities are encoded injectively before becoming
filesystem path components, while ordinary safe identifiers retain their
readable paths.
When a later lane consumes a generated artifact, its dependency fingerprints
include the producer's artifact kind, complete schema identity, media type,
canonical content digest, and size. Ordinary resume compares those fingerprints
when progressively loading consumer stage checkpoints, so changed producer
content produces `dependency_invalidated` rather than stale downstream reuse.
Selective recomputation instead requires each unselected producer's accepted
normalized artifact; invalid accepted state records its specific bounded reason
and stops before the dependent. The selected step and its transitive dependents
record `forced_recompute`.
Reuse eligibility is separate from the selected cache mode. Fallback and actual
dependent work perform neither lookup nor storage, even if they later produce
otherwise accepted output. Bypass/ineligible work uses ordinary native execution
without cache-specific split preparation. Capture reuses available prepared
details; it never adds preparation to explain a hit.
Ordinary resume loads extract, merge, and normalize checkpoints progressively
and may execute later lane stages after an earlier cache miss. Selective
recomputation instead asks the loader for the required producer's accepted
normalize artifact. That lookup reuses the existing normalize files, requires
workspace schema v4 plus an exact non-empty invocation identity, and deliberately
does not require extract or merge checkpoint files or dependency fingerprints.
The runner performs canonical codec and producer-provenance validation before
cloning the artifact into normal step output. Success restores only stored
normalize diagnostics and emits one normalize decision; failure retains the
files, records the decision, and stops without executing the producer or
consumer.
## Filesystem Store And Failure Handling
The loader assigns a typed category and reason code at each validation site;
diagnostic prose is not classified after the fact. The runner then applies
forced-execution policy, validates reusable artifact bytes through the prepared
codec once, returns the canonical hydrated value to the stage, and records the
final decision before enforcing a required-predecessor failure. That failure
names only the step, lane, and stable reason code. Decision detail is selected
from code-owned descriptions by reason code and then UTF-8 normalized and
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
codes remain intact through pipeline events and become strings only in manifest
and debug-summary JSON.
[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
canonical operator-facing reason-code reference.
[artifactcache.FilesystemStore](../../internal/framework/artifactcache/filesystem.go)
implements the narrow `contracts.ArtifactCache` interface. Construction retains
the exact selected root without creating it. Lookup checks one confined regular
JSON entry; missing data is an ordinary miss. Stored bytes are the unchanged
accepted artifact, without provenance/execution envelopes or alternate key
dimensions. Physical placement belongs in
[Operations](../operations.md#state-surfaces).
`internal/core/fileio` provides confined atomic file writes used by state
collaborators. Its exact-mode writer remains available to output and debug
callers; its process-permission writer is used by checkpoints. The chunk-plan
store retains its stronger rooted entry validation. [Operations](../operations.md#cache-provisioning)
owns cache mode and provisioning guidance.
The store honors the resolved read/write mode. Refresh does not read or remove
previous acceptance in anticipation of execution. Only a newly supplied accepted
value can replace it. Failed refresh execution or a failed cache write preserves
the old entry.
Legacy data is not scanned, converted, rewritten, or deleted.
The CLI constructs selective-recomputation policy from resolved generated
artifact dependencies. It forces the selected step and transitive consumers,
while marking unforced producers as required reusable inputs. The runner owns
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
owns the operator workflow and stable reason-code meanings.
Publication uses [fileio](../../internal/core/fileio/fileio.go)'s confined atomic
process-permission writer. New cache objects use process/deployment creation
permissions; there is no permission convergence on an existing root.
[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md) owns that decision.
## Debug Bundles
[Cache integration](../../internal/framework/pipeline/producer_cache.go) treats
unreadable/malformed/codec-unusable entries as diagnostic misses. Store failures
become scoped process warnings without failing successful production or calling
the provider again. Artifact encoding, caller cancellation, requested evidence,
and observer failures retain their own error boundaries; they are not swallowed
as best-effort storage failures.
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
resolution, run, final grouped diagnostic, and failure artifacts. `internal/framework/debug`
implements the pipeline-facing trace recorder under the trace root.
Accepted predecessors can survive downstream failure and be reused by an ordinary
rerun. There is no checkpoint loader, resume barrier, forced-recompute policy, or
canonical partition selected solely by source digest. New message selections
populate independently; an accepted plan is rematerialized, not restored with a
historic execution claim.
The CLI allocates a bundle before pipeline resolution and treats requested
summary or trace persistence failures as command failures. The pipeline's debug
boundaries redact sensitive metadata and credential-shaped bytes while allowing
application-owned trace material. Debug data is never a checkpoint source or
cache input.
## Artifact Publication And Native Reporting
Generated reference bytes exist only in cloned operation requests and are not
written as paths into checkpoints, manifests, or debug summaries. Those state
surfaces retain canonical identities and bounded producer provenance so that a
resume decision can be explained without copying generated campaign content.
The runner returns logical deliverable files only after successful execution/
encoding. The [JSON encoder](../../internal/modules/generic/output/json/encoder.go)
owns domain-file selection and optional auxiliary inclusion.
[JSON output](../integrations/json-output.md),
[chunk maps](../integrations/chunk-map.md), and
[evidence context](../integrations/evidence-context.md) own logical paths/schemas,
not the store or recorder.
After allocation, one CLI-owned state value accumulates the known report paths,
pipeline outcome counts, and validation status. A single guarded terminalization
operation writes the success report, or makes one attempt each to write the
failure report and error log. Terminal persistence failures are reported
separately and never replace the command's primary error.
[writeOutputFiles](../../internal/cli/run_output.go) validates every logical
path before allocating the run directory, refuses an existing destination, and
writes files with confined atomic helpers. This is per-file publication, not
an all-files transaction. A later write can leave a partial new directory;
existing siblings are preserved. No rollback deletes available artifacts.
## Invariants To Preserve
After complete publication, the run handler finalizes requested capture before
[reportRunResult](../../internal/cli/run_result.go) submits the small native
completion value. The report is not automatically persisted or required to decode
domain files. No index, manifest, receipt, or process-diagnostic sidecar is emitted.
Artifact count counts returned deliverables, including selected available auxiliaries,
not cache/debug/intermediate files.
- Modules receive state collaborators and request data, never physical roots.
- Output logical paths are validated before a run directory is allocated, and
files are atomically written within that directory.
- Chunk-plan publication occurs only for accepted plans; bypass does not
construct or touch a plan store.
- Checkpoint recording and checkpoint loading remain separate collaborators.
- Debug state is opt-in, is not cache input, and terminal reporting does not
obscure the command's primary failure.
Post-publication capture/render/delivery failure remains nonzero with available
locations. Valid artifacts remain; model work and capture terminalization are not
repeated. Native delivery evidence is separate from the operation cause, and
failed stdout may be partial. Public outcomes and caller responsibilities belong
in [CLI Reference](../cli.md#output-streams-and-exit-statuses) and the
[subprocess guide](../consumers/subprocess.md).
## Tests To Inspect
## Capture Lifecycle
- `internal/cli/run_contract_test.go`: command-owned state allocation,
terminalization, and output/report boundaries.
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
and resume collaborator construction.
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
permissions, cleanup, and redaction.
- `internal/cli/recompute_policy_test.go`: forced dependents and required
reusable predecessors for selective recomputation.
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
filesystem recovery, deterministic decisions, and failed predecessor state.
- `internal/cli/production_contract_test.go`: production composition and
configuration validation at the CLI boundary.
- `internal/cli/example_contract_test.go`: maintained example ownership.
- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
reuse.
- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
publication, and permissions.
[Debug selection](../../internal/cli/run_debug.go) preserves native boolean and
supplied directory semantics independently of verbosity. The run path alone
calls [debug.Start](../../internal/framework/debug/recorder.go).
[Bundle allocation](../../internal/core/debugbundle/bundle.go) validates identity,
creates a new absolute per-run directory, and atomically updates a relative
latest link at start. Failure can return an available directory for explanation/
retention. Disabled capture creates nothing; completion never retargets the link.
One synchronized recorder writes evidence incrementally, retaining lifecycle
state and the first recording error rather than accumulating whole-run content.
Callback failures are exposed at the next boundary that can return an error.
`Finalize` attempts terminal evidence once, joins primary and capture causes,
and retains its result. Repeated calls do not rewrite evidence or retarget latest.
A later presentation failure does not change capture into another terminal run.
Allocation/link/record/finalization failures never trigger model retries or a
fallback directory. Available files are retained.
## Native And Application Evidence
[Run capture](../../internal/cli/run_capture.go) snapshots the original secret-aware
application/PromptKit configuration and source evidence. Scoped pipeline capture
records source/reference/incoming artifacts, normalization originals/mappings/
proposals, candidates, complete findings/guidance, fallback/deterministic outcomes,
and cache selection/current artifact bytes.
[NativeCapture](../../internal/framework/pipeline/native_capture.go) composes
existing observers and preparation callbacks. Preparation callbacks retain
detached details cheaply; ordinary boundaries write prepared contracts, native
results, raw response text, and full readable exchanges. Events are written
incrementally. Scope includes operation/step/lane/chunk/validator identity to
avoid concurrent evidence collisions.
One invocation `ExecutionGroup` enrolls actual initial, continuation, and nested
validator executions once. Native identity, terminal coverage, and unknown usage
remain native facts, not a copied profile/attempt accounting model.
Hits can retain prepared evidence without execution or usage. Result-less failures
retain availability/cause evidence; undelivered responses are explicitly unassessed.
Native credential omissions and ConfigKit marked-secret paths are preserved.
Arbitrary source/prompt/response text is not heuristically rewritten or made safe
for general logging. Exact evidence names/nesting/serialized forms are private,
not a public protocol or reconstruction requirement. Debug is neither cache
input nor an artifact-discovery interface; [Operations](../operations.md#debug-bundles)
owns handling.
## Focused Verification
- [Filesystem cache](../../internal/framework/artifactcache/filesystem_test.go)
and [permissions](../../internal/framework/artifactcache/filesystem_permissions_unix_test.go)
cover raw bytes, modes, atomic replacement, diagnostics, confinement, deployment
permissions, and unchanged legacy files.
- [Producer cache](../../internal/framework/pipeline/producer_cache_test.go),
[typed runner cache](../../internal/framework/pipeline/runner_cache_test.go), and
[plan cache](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover
initial handles/keys, final acceptance, current-source checks, and failed refresh.
- [Recorder](../../internal/framework/debug/recorder_test.go) and
[allocation](../../internal/core/debugbundle/bundle_test.go) cover incremental
failure retention, concurrent starts, and one terminalization.
- [Native evidence](../../internal/framework/pipeline/native_capture_test.go) and
[configuration capture](../../internal/cli/run_capture_test.go) cover available
content, exact sensitivity, observer composition, and no duplicate preparation.
- [Output writes](../../internal/cli/run_id_test.go) and
[native result boundaries](../../internal/cli/run_result_command_test.go) cover
collisions, partial directories, capture/presentation failure, and retained causes.
- [Assembled recovery](../../internal/cli/recovery_contract_test.go) covers
downstream fix/rerun, initial-key correction reuse, qualified fallback, actual
generated consumers, and debug on hits/failure.
Use focused pipeline/cache/recorder/CLI checks for state-boundary changes.
[Development](../development.md#validation) owns wider validation.