diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index b6c6dc17..33b130f9 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -1,279 +1,202 @@ # Pipeline Internals -This document describes the framework-owned pipeline mechanics in -**internal/framework/pipeline**. [Configuration](../config.md) owns selectable -profiles, bindings, and retry settings; [Operations](../operations.md) owns -state lifecycle and recovery; and the [integration contracts](../integrations/) -own durable output shapes. Concrete production extensions are covered by -[Module Internals](modules.md). +This guide explains resolution, typed execution, candidate assessment, and +ordering in `internal/framework/pipeline`. +[Configuration](../config.md#pipelines) owns bindings and producer controls; +[Operations](../operations.md) owns state/recovery workflows; +[integration contracts](../integrations/json-output.md) own deliverable shapes. +[Module Internals](modules.md) owns extension recipes. -## Boundary +## Boundary And Fixed Shape -The pipeline framework accepts a resolved composition, registries, shared -dependencies, input bytes, a supplied prompt session, and state/debug -collaborators. It returns logical output files, normalized artifacts, recorded -rejections, grouped diagnostics, manifest provenance, and checkpoint decisions. The -CLI owns process arguments, configuration discovery, session resolution, -physical roots, and placement of returned output files. +The framework receives prepared composition, source bytes, a supplied session, +native execution options, and explicit cache/capture collaborators. It returns +logical deliverable files, process diagnostics, and fallback count. Normalized +artifacts and reuse eligibility remain private run state for handoffs/evidence; +there is no reporting ledger in the output boundary. -The framework has one fixed shape: +```text +input -> chunk -> ordered steps [extract -> merge -> normalize] -> output +``` -~~~ -input -> chunk -> extract -> merge -> normalize -> output -~~~ - -Input and chunking are pipeline-wide. A selected artifact lane owns extract, -merge, and normalize; output aggregates the terminal lane outcomes. A pipeline -is an ordered list of steps, not an arbitrary workflow graph. +Input and chunking are pipeline-wide. Each ordered step contains typed artifact +lanes; the next step waits for the current step and its generated handoffs. +This fixed pipeline is not a general workflow graph. The CLI owns arguments, +configuration loading, session derivation, physical roots, publication, and +native reporting; see [CLI Internals](cli.md). ## Resolve, Materialize, Prepare -Resolution turns a configured pipeline profile into a **ResolvedPipeline**. -It normalizes the pipeline and lane identities, applies stage defaults, selects -requested lanes where that is supported, resolves validator chains, checks -module capabilities and typed artifact compatibility, validates options, and -assigns a deterministic resolved-composition digest. A correction protocol is -selected from each eligible LLM-backed producer specification and becomes part -of that resolved identity; only `single_response_v1` is currently supported. -Preparation rejects an LLM-backed producer that combines a non-empty validator -chain with positive producer retries unless it declares that protocol. Producers -without validators or without retries remain valid without correction support. -The resolved pipeline contains bindings and declared reference targets, not -external reference bytes. -After selection, the resolver applies command, binding, and pipeline profile -precedence to LLM-backed bindings and validators only; prompt defaults remain -an empty resolved binding profile. It resolves structural output repair -separately: a binding's `structured_output_repair_attempts` value wins, then a -pipeline value applies to LLM-backed bindings and validators, and omission -leaves the prompt-owned policy intact. An explicit repair value on a -deterministic binding is rejected. Resolved bindings own copied repair values, -and these effective values are part of the digest, so execution and checkpoint -consumers do not repeat profile inheritance or configuration resolution. -Each LLM request receives its own copy of that resolved value. PromptKit spends -it only for structural correction inside one completion; the runner's binding -retry policy remains the separate outer budget for complete stage attempts. -Configuration resolution supplies the selected overlay and catalog; see -[Configuration Internals](configuration.md). +[ResolvePipeline](../../internal/framework/pipeline/profile.go) selects modules, +capabilities, exact artifact kinds, validators, references, options, and concrete +producer controls. It copies static native prompt dependencies from selected +specifications. Effective profiles apply only to model-backed roles; an empty +profile selection leaves native prompt defaults applicable. +[Control resolution](../../internal/framework/pipeline/validation_policy.go) +settles presence-aware controls before execution; modules do not repeat inheritance. +The composition digest describes resolved composition, not cache identity. -External reference materialization happens before preparation. The materializer -checks that each slot is declared by the selected module, resolves a file path -relative to the correct configuration or working-directory origin, reads -UTF-8 text, verifies media type and size limits, and retains bounded -provenance. For a positive slot limit, it reads at most the limit plus one byte -and rejects overflow before retaining content. A generated-artifact selector -remains declared but has no bytes until its producing step completes. +Resolution receives exact reference bindings/unbinds, not CLI selector grammar. +It preserves required slots, external/generated conflicts, prior-step +dependencies, and artifact compatibility. [Reference materialization](../../internal/framework/pipeline/references.go) +reads selected external bytes with declaration/invocation origins, media/UTF-8 +and declared size checks. Generated selections remain declarations until their +producer step completes. Public selection semantics belong in +[References And Ordered Handoffs](../config.md#references-and-ordered-handoffs). -Preparation is the construction boundary. It validates the resolved shape and -registry set, clones the resolved data, then constructs the input adapter, -chunker, stage-local validators, every typed lane, and output encoder. The -prepared producer metadata preserves each selected correction protocol, and -the resolved digest carrying that metadata participates in checkpoint identity. -Each registered builder receives its own cloned build request immediately -before its module-owned code runs. Preparation also collects stable checkpoint -fingerprints. Missing registrations, incompatible typed entries, nil -implementations, and constructor failures are reported before source parsing -or any stage operation begins. +[Prepare](../../internal/framework/pipeline/prepare.go) clones resolved data and +constructs input, chunker, validators, typed lanes, and output through supplied +registries and the shared native engine. Builders receive independently owned +build requests. Exact Go type, codec, evidence-projector, registration, and +constructor checks happen before source parsing. There is no correction transport, +asset registry, or persistent execution identity to collect. +Offline known-resource [preflight](../../internal/preflight/preflight.go) is +a separate composition check, not request execution or module construction. -An output encoder can opt into source-evidence publication through its output -policy. Preparation keeps the configured lane allowlist and active lanes -separate, then verifies an exact typed evidence projector and registered codec -for each active lane. The resulting private plan is immutable; lanes excluded -by invocation filtering remain configured but do not acquire a projector for -that run. +## Typed Operations And Assessment -## Typed Lanes And References +[Typed contracts](../../internal/framework/contracts/typed_pipeline.go) carry +a `ProducerOperation[T]` with resolved controls, one assessor, native execution +options, incoming reuse eligibility, and an optional typed cache binding. +[Runner operation completion](../../internal/framework/pipeline/runner_operation.go) +assesses deterministic production directly. Model modules use +[RunProducer](../../internal/framework/pipeline/producer.go); the runner consumes +its acceptance/fallback outcome without assessing it again. -Each resolved lane has one artifact kind, codec, and exact Go type. The -framework uses private type erasure only around those typed operations; every -handoff checks exact type and codec identity and reports incompatibility as an -error rather than panicking. Encoding through the registered codec is the -boundary for output, checkpoints, debug records, and generated references. +[Candidate assessment](../../internal/framework/pipeline/candidate_assessment.go) +invokes each applicable validator once in configured order. It continues after +rejections and unavailable/failed assessments; caller cancellation and framework/ +capture failures interrupt it. Typed candidates are encoded once for assessment, +then decoded separately for each typed validator. Serialized validators receive +owned bytes/schema metadata; chunk validators receive detached materialized +chunks. Source, references, annotations, and metadata views are isolated too. -Reference targets are stage- and lane-specific. External reference bytes are -cloned into the operation request. Generated references are built at the next -step boundary from exactly one accepted normalized producer output. The -framework decodes and re-encodes that output with the registered producer -codec, checks its complete schema and media identity, and records a content -digest plus bounded producer provenance. A missing, ambiguous, invalid, or -incompatible producer prevents the consumer step from starting. +[Assessment values](../../internal/framework/contracts/assessment.go) distinguish +approval, semantic rejection, failure, and unavailable judgment. An empty configured +chain adds no ordinary acceptance criteria, but cannot remove intrinsic codec or +source-addressing checks. Rejection guidance is module-authored, ordered, and +deduplicated separately from reason codes/operator messages. Invalid guidance is +a contract failure, not invented correction text. LLM validators execute directly +on the shared engine and do not own another semantic loop. -Resolution receives only exact stage-and-lane operational reference overrides. -The CLI may offer broader pipeline- or lane-scoped selectors, but expands and -arbitrates those before entering the framework. External overrides are applied -after configured external defaults and local bindings. They cannot coexist -with a generated binding for the same target and slot, and external unbinds do -not remove generated handoffs. +## Native Continuation And Terminal Handling -## Execution And Ordering +For executed model work, `RunProducer` checks native errors and structural +validation status before interpretation, including a failed result with no Go +error. Module interpretation produces a usable typed candidate and any actionable +domain guidance. The assessor runs the complete chain on every delivered usable +candidate, including unchanged revisions and the final allowed response. -The runner validates its input, installs no-op state collaborators when none -were supplied, and serially performs source parsing and chunk-plan selection. -It transports the supplied session unchanged to prompt-facing operations and -run-manifest metadata; it neither derives a session nor substitutes a parsed -source document identifier. The public session contract is owned by the -[CLI reference](../cli.md#run). -An accepted plan is materialized into source-addressed chunks and passes the -configured chunk validators before any lane runs. A chunk rejection is a -recorded pipeline outcome: lanes do not start, but the output stage can encode -the terminal result. +When correction is possible, the helper starts native `Coordinate` from the +captured exchange. Its first callback reuses already-collected initial findings. +Native coordination owns turn iteration and growing history; the module builds +only new correction material from fixed operation inputs. Successful native +execution, stopped/exhausted coordination, and domain acceptance are distinct. +Native repair/transient retries are not extended by outer application retries. +[LLM Integration](llm.md) owns the native request/capture integration. -For each ordered step, the runner first builds generated reference sets from -the accepted normalized outputs of earlier steps. It then executes the step's -lanes. Later steps do not begin until the current step is terminal and its -generated handoffs have succeeded. +[Terminal disposition](../../internal/framework/pipeline/producer_outcome.go) +either fails or invokes one supported module fallback. Fallback construction +preserves intrinsic/source/downstream invariants without ordinary reassessment +or semantic-approval claims. It may fail normally. A failed continuation cannot +recover an earlier candidate; caller cancellation and undelivered responses +produce no acceptance/fallback. Native operational timeouts retain typed causes +rather than becoming caller cancellation solely by error spelling. -Within a step, the lane engine dispatches extraction jobs in deterministic -chunk-first, lane-second order to a bounded worker group. When all extraction -jobs for one lane are terminal, a bounded continuation group can run that -lane's merge and normalize work while extraction for other lanes continues. -The framework does not create an unbounded goroutine per chunk or lane. +Deterministic work produces once, then uses direct assessment and any supported +module fallback. An operation/parse/materialization/encoding error remains an +error, not a generic rerun or permissive recovery mechanism. -Completion timing does not determine public results. The coordinator restores -lane and chunk order before merging results, and selects a framework error by -stable stage, lane, and chunk position. A validator rejection records a lane -outcome without cancelling unrelated work. A framework error or parent -cancellation cancels derived work, prevents queued work from starting, waits -for started workers, and prevents output encoding. +## One Ready-Job Worker Pool -## Validation, Retries, And Output +[The lane coordinator](../../internal/framework/pipeline/runner_concurrent.go) +dispatches extraction jobs and ready lane continuations to the same bounded +worker pool. A continuation becomes ready after all extracts for its lane finish; +one worker runs its merge then normalization. Ready continuations can interleave +with unrelated extraction, and prerequisite waits occupy no worker. +Input parsing, chunking, step barriers, and final output remain sequential. -Every chunk, extract, merge, and normalize candidate passes its resolved -validator chain. Validators receive immutable canonical input appropriate to -their target: chunks, codec-decoded typed candidates, or serialized codec -bytes. Each typed validator receives a newly decoded value from the one -candidate serialization for that attempt, while serialized validators receive -separately owned representation bytes and schema metadata. They may approve, -approve with warnings, reject, fail, or be skipped when a runtime prerequisite -is unavailable. The shared executor settles every configured validator in -order. A failed LLM-backed validator retries only itself against the same -immutable candidate; it does not regenerate the producer or alter the -validator request. Rejections stop that validator, while other configured -validators still run. The executor retains ordered results, bounded -deduplicated correction guidance from rejections, and only the final exhausted -failure outcome for each validator. The correction builder keeps first -occurrence order, omits internal reason codes, validator names, and operator -messages, and requests one complete replacement. Missing guidance or an -oversized aggregate is a framework contract error; guidance is never inferred -or truncated. +Results are restored to chunk/lane order before merging. Stable operation/lane/ +chunk positions select the scoped terminal error among reported failures rather +than response timing. Parent cancellation takes precedence. On failure/cancellation +the coordinator stops dispatch, drains started results, closes jobs, and joins +workers before returning. Output encoding does not run after terminal failure. -The runner applies the binding's retry policy around a stage operation and its -complete validation chain. It preserves terminal diagnostics only from the final accepted -or rejected attempt, plus one fixed validation-incomplete warning per validator whose execution -budget was exhausted under `warn_continue`. Cancellation stops retries. -Normalizer-specific retry directives consume this same budget and validate any -final safe fallback through the normalizer chain. A directive may carry bounded -correction guidance only when it also exposes the exact latest -`single_response_v1` candidate. The state machine then uses the same replacement -request shape as validator correction. A directive without guidance clears any -prior correction and starts a fresh attempt, which preserves structural retry -behavior when no valid response exists. +Nested validation and correction use the operation's worker, not another worker +permit. Native backend admission is independent and released before candidate +assessment, allowing a nested validator on the same limited backend. Worker +configuration does not create another model scheduler, queue, or retry policy. -The artifact-neutral producer-attempt state machine owns that shared budget, -attempt provenance, semantic-correction material, and terminal-policy -selection. It accepts producer and complete-validation closures, so artifact -materialization, cache handling, checkpoints, and debug output stay at the -operation boundary. It distinguishes operational, structural, module-requested, -and validator-semantic retries. Model feedback from either semantic source is -available only for a valid latest `single_response_v1` candidate. A -deterministic or no-model validator rejection instead settles the semantic -policy immediately, while a feedback-free module directive remains an ordinary -fresh retry. Structural-output errors alone use the structural policy, and -validation failure without rejection settles the validator-failure policy -without regenerating the producer. +## Handoffs And Reuse Eligibility -Chunk planning uses this state machine for generated plans. A rejected or -validation-incomplete automatic cache hit is not model material and therefore -falls through to a fresh initial generation at producer attempt one; it neither -receives a correction, consumes retry budget, promotes cached-candidate -diagnostics, nor overwrites the stored record. An incomplete cache validation -under `fail_run` terminates instead. Only a newly generated, completely -validated plan is published to the chunk-plan store. Rejected plans never -advance, and validation-incomplete plans remain unpublishable. +[Generated handoffs](../../internal/framework/pipeline/handoff.go) locate one +earlier normalized producer, decode/re-encode through its registered codec, check +full artifact/schema/media compatibility, and clone bytes into the consumer's +declared reference set. Valid qualified fallback is usable current-run context; +it does not become ordinary acceptance through a handoff. -After terminal lane work, the runner assembles manifest provenance, normalized -artifacts, rejections, final grouped diagnostics, and an optional accepted chunk map. When an -output policy selected evidence lanes, it decodes accepted serialized normalize -outputs through their registered codecs and invokes the prepared typed -projectors. Rejected or absent lanes contribute nothing. This reconstruction is -also used after normalized-checkpoint reuse, so no second typed output channel -is retained. The runner passes the resulting owned artifact to the output -encoder, which returns logical files and does not choose a physical directory. -The CLI publishes those files only after the runner returns without a framework -error. Logical file names and schemas are defined by the [output integration -contracts](../integrations/). +Eligibility travels separately from artifact bytes: -For every completed producer disposition, the runner projects one bounded -validation summary to the manifest, the affected rejection when present, and -the CLI result receipt. The summary records status, configured-order rejecting -validators and reason codes, incomplete validators, producer-attempt count, -and terminal action. It contains no operator message, correction guidance, or -model response. `complete`, `rejected`, and `incomplete` describe the final -candidate disposition; a run-level `incomplete` status indicates at least one -current-run output advanced under `warn_continue`. +- A chunk fallback disables reuse for all derived lane work. +- An extract fallback disables reuse for its dependent merge/normalization. +- A fallback-derived generated reference disables reuse from the consuming + operation onward; unrelated siblings and earlier accepted work remain eligible. -## Checkpoint And Debug Hooks +An emitting fallback increments the run count once. Merely consuming its data or +inheriting ineligibility adds no fallback. These are transient dependency facts, +not persisted fingerprints or a second recovery protocol. -The runner receives checkpoint and debug interfaces rather than roots. It -records workflow transitions and reuse decisions through the supplied -collaborators, and clones reusable artifacts before they re-enter normal typed -handoff. Generated-reference dependencies participate in checkpoint decisions. -Selective recomputation can require a canonical accepted normalized predecessor -before a dependent lane starts. +[Typed cache bindings](../../internal/framework/pipeline/runner_cache.go) use the +registered durable codec and available exact-type evidence projection for +current-source usability. [Chunk planning](../../internal/framework/pipeline/runner_chunk_plan.go) +uses the same producer cache with plan decoding/rematerialization. Hits do not +run semantic assessment. Deterministic work has no synthetic request/cache key. +[Run State Internals](state.md) owns prepared-handle/store mechanics and recovery. -The runner writes successful checkpoint artifacts only after complete accepted -validation. Chunk plans follow the same rule for publication. A rejection, -invalid structured response, or incomplete validation is never reusable state; -the current run may still hand off an otherwise valid `warn_continue` result -according to its terminal policy. The runner carries private reuse eligibility -through extract, merge, normalize, and generated-reference handoff. Any stage -derived from incomplete validation skips both checkpoint lookup and all -checkpoint publication even when that stage's own validation completes. -External references and fully validated generated references remain eligible. -Attempt debug records retain safe kind, -validator, repair-usage, policy, and terminal-decision provenance. Full -assistant and correction content remains confined to the requested detailed -LLM trace. +## Output And Capture Boundaries -Debug recording is attempt-scoped and application-owned. A failure to persist -required debug data is a framework error. State roots, persistence, reason-code -meanings, resume, and cleanup are intentionally owned by -[Run State Internals](state.md) and [Operations](../operations.md). -Extract-validator trace scopes include the current chunk ordinal so concurrent -chunks cannot overwrite one another's validator attempts or LLM artifacts. +[Runner](../../internal/framework/pipeline/runner.go) parses/validates source, +materializes the final plan, runs ordered steps, aggregates diagnostics, and +builds optional chunk-map/evidence values. Normalized values are codec-checked +before handoff/output. Evidence reconstruction also works from reused artifacts; +it projects selected citations in source order, not a second typed output channel. +[Evidence internals](../../internal/framework/pipeline/evidence_output.go) and +the [evidence contract](../integrations/evidence-context.md) define those owners. -## Invariants To Preserve +The encoder receives owned normalized/auxiliary values and returns logical files. +It does not choose a directory, publish a report, or serialize execution ledgers. +[JSON output](../integrations/json-output.md) owns logical paths and payloads; +[State Internals](state.md) owns physical publication and failure retention. -- The six fixed stages remain explicit; a pipeline is not a general DAG. -- Resolution and preparation reject statically discoverable incompatibility - before parsing or execution. -- Every typed lane uses one compatible artifact kind, codec, and exact Go type. -- Generated references come only from one earlier accepted normalized producer - and carry canonical identity rather than an unverified value. -- Rejections are recorded outcomes; framework errors cancel derived work and - prevent output encoding. -- Public ordering and selected errors are independent of goroutine completion - order. -- Pipeline modules receive collaborators and data, never CLI streams or - physical output, cache, or debug roots. +Optional scoped evidence records inputs, candidates, findings, outcomes, and +cache decisions. Native observers enroll actual executions once in one invocation +group, preserving unknown usage and terminal coverage. Capture failures are +ordinary boundary failures, never semantic feedback or reasons to rerun a model. +Debug layout is private; it is not a recovery source or consumer protocol. -## Focused Tests +## Focused Verification -- **internal/framework/pipeline/profile_test.go** and - **typed_resolution_test.go** cover resolution, defaults, ordered steps, - compatibility, validators, references, and resolved identity. -- **internal/framework/pipeline/preparation_test.go** covers complete - construction before execution and contextual construction failures. -- **internal/framework/pipeline/references_test.go** and **handoff_test.go** - cover external materialization, generated references, provenance, and typed - producer checks. -- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded - execution, ordered steps, stable error selection, rejections, and - cancellation. -- **internal/framework/pipeline/runner_chunk_plan_test.go**, - **runner_typed_checkpoint_test.go**, and - **runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior. -- **internal/framework/pipeline/runner_attempt_debug_test.go** and - **runner_terminal_debug_test.go** cover attempt and terminal debug behavior. +- [Resolution](../../internal/framework/pipeline/typed_resolution_test.go), + [preparation](../../internal/framework/pipeline/preparation_test.go), and + [references](../../internal/framework/pipeline/references_test.go) cover static + compatibility and construction/materialization ownership. +- [Assessment](../../internal/framework/pipeline/candidate_assessment_test.go) + and [producer tests](../../internal/framework/pipeline/producer_test.go) cover + isolation, whole-chain/final-turn assessment, guidance, native status, and cancellation. +- [Runner operations](../../internal/framework/pipeline/runner_operation_test.go) + cover no outer reassessment, deterministic errors, fallback counts/eligibility, + and actual generated consumers. +- [Worker tests](../../internal/framework/pipeline/runner_concurrency_test.go) + and [ready-job tests](../../internal/framework/pipeline/ready_jobs_test.go) + cover mixed jobs, stable order, serial liveness, nested admission, and draining. +- [Producer reuse](../../internal/framework/pipeline/producer_cache_test.go), + [runner reuse](../../internal/framework/pipeline/runner_cache_test.go), and + [plan reuse](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover + initial keys, applied artifacts, no reassessment on hits, and current-source checks. +- [Capture](../../internal/framework/pipeline/native_capture_test.go), + [evidence](../../internal/framework/pipeline/evidence_output_test.go), and + [assembled recovery](../../internal/cli/recovery_contract_test.go) cover + native facts, output reconstruction, and cross-boundary reuse/failure. -Run **go test ./internal/framework/pipeline ./internal/cli** after changing a -pipeline boundary. Use the more focused tests above while iterating. +Use focused pipeline and CLI suites for boundary changes. +[Development](../development.md#validation) owns repository-wide checks. diff --git a/docs/internal/state.md b/docs/internal/state.md index e7d02886..1b56853f 100644 --- a/docs/internal/state.md +++ b/docs/internal/state.md @@ -1,165 +1,193 @@ # Run State Internals -LLM debug response artifacts and their safe per-call references retain an -optional terminal execution summary when one is available. This metadata is -diagnostic only and does not change receipts, checkpoints, or aggregate usage. +This guide explains physical cache, capture, and publication collaborators. +[Operations](../operations.md) owns layout, permissions, retention, and recovery +procedures; [Configuration](../config.md#concurrency-output-cache-and-debug) owns +settings/defaults; [CLI Reference](../cli.md#completion-report) owns reporting. +[Pipeline Internals](pipeline.md) owns assessment, workers, and dependency eligibility. -This document describes the implementation collaborators behind output, cache, -and debug state. User-visible fields belong in [Configuration](../config.md), -and physical layout, retention, recovery, reason codes, and cleanup belong in -[Operations](../operations.md). +## Composition And Independent State -## Composition +[The run handler](../../internal/cli/pipeline_run.go) is the physical-path +composition root. It obtains loaded configuration and a safe run identity, +allocates requested capture before engine/preflight, resolves/materializes/ +prepares, and then constructs the cache/recorder collaborators for the runner. +Shared inspection loaders create no run storage. -`internal/cli` is the only physical-path composition root. It resolves the -effective configuration, selects exact roots, allocates requested debug bundles, -constructs cache collaborators, writes logical output files, and reports paths. -Pipeline modules receive interfaces and request data, never output, cache, or -debug roots. +Modules receive explicit operation/cache/evidence interfaces and data, not roots +or CommandKit reporters. Output, accepted-result cache, and optional capture +remain independent. None is a required metadata bundle for another. +In-memory normalized artifacts support handoffs/evidence; they are not a +persisted resume ledger. -The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint -recorder only when recording is enabled and a checkpoint loader only for a -resume invocation. It allocates debug state only after a safe run identity has -been generated and only when debug capture was requested. These choices keep -the three state families independently composable. +## Prepared Result Reuse -## Output And Cache +[RunProducer](../../internal/framework/pipeline/producer.go) owns native request +selection and cache interaction for an eligible model operation: -The pipeline runner returns logical output files. After validating every -logical name, the CLI exclusively creates the run directory beneath the -selected output root and performs confined, atomic file writes within it. -The runner supplies an accepted chunk map as an optional, defensively owned -output-request artifact. The JSON encoder alone decides whether its explicit -option writes the map and optional index descriptor; neither the map payload -nor its annotations are copied into the run manifest. The durable fields are -owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md). +1. Prepare the initial request once with `PrepareExecution`, retain its detached + details and opaque `MessageContentHash`. +2. For a read-enabled selection, decode a found entry through the expected + durable codec and current-source usability checks. On a usable hit, discard + the prepared handle and return the artifact without model work. +3. On a miss, execute that exact handle with `RunPrepared`, then use ordinary + interpretation, complete assessment, and native continuation. +4. Store only final ordinary acceptance, encoded as the applied typed artifact + or checked logical plan, under the initial key. Dispose of unused + prepared handles on every exit. -`internal/framework/chunkplan` owns source-addressed plan storage, validation, -and atomic publication. Its store is constructed only when the selected mode is -not `bypass`. It preserves rooted entry checks while creating new cache roots -and source-digest directories with requested mode `0777`, then creates an -atomic temporary file through rooted `O_CREATE|O_EXCL` with requested mode -`0666` before sync and rename. +[Typed bindings](../../internal/framework/pipeline/runner_cache.go) attach codec +and source-evidence checks; [chunk-plan bindings](../../internal/framework/pipeline/runner_chunk_plan.go) +decode and rematerialize against the current document. A hit has no ordinary +semantic revalidation, native execution event, restored exchange, or historical +usage. Normalizers store applied domain results, not reconciliation proposals. +Deterministic operations stay uncached without manufactured requests. -`internal/framework/checkpoint` owns checkpoint identity, manifests, payload -codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint -recording is enabled and constructs a loader only for a `--resume` invocation. -The recorder writes manifest and payload JSON through -`fileio.WriteJSONWithProcessPermissions`, whose confined atomic writer creates -new directories with requested mode `0777` and temporary files with requested -mode `0666` before publication. -Identity incorporates explicit stable semantic fingerprints collected from -prepared modules and validators in addition to configuration, input, -references, runtime overrides, observed LLM profiles, and the LLM runtime's -non-secret effective profile-source identity. A profile source change therefore -causes a cold miss even when the configured profile ID remains unchanged. -The serialized -`workspace_schema_version` identifiers are frozen wire-compatibility fields; -they do not describe a current public state surface. +The key is the native initial message-content value, not a configuration, +policy, model/session, implementation, or source-only fingerprint. Rendered +message changes can change selection; settings-only changes need not. +Correction retains the initial key. Public reuse choices and their consequences +belong in [Operations](../operations.md#accepted-result-cache-and-recovery). -Ordered-step lane checkpoints include the step identity in their storage scope. -Accepted step and lane identities are encoded injectively before becoming -filesystem path components, while ordinary safe identifiers retain their -readable paths. -When a later lane consumes a generated artifact, its dependency fingerprints -include the producer's artifact kind, complete schema identity, media type, -canonical content digest, and size. Ordinary resume compares those fingerprints -when progressively loading consumer stage checkpoints, so changed producer -content produces `dependency_invalidated` rather than stale downstream reuse. -Selective recomputation instead requires each unselected producer's accepted -normalized artifact; invalid accepted state records its specific bounded reason -and stops before the dependent. The selected step and its transitive dependents -record `forced_recompute`. +Reuse eligibility is separate from the selected cache mode. Fallback and actual +dependent work perform neither lookup nor storage, even if they later produce +otherwise accepted output. Bypass/ineligible work uses ordinary native execution +without cache-specific split preparation. Capture reuses available prepared +details; it never adds preparation to explain a hit. -Ordinary resume loads extract, merge, and normalize checkpoints progressively -and may execute later lane stages after an earlier cache miss. Selective -recomputation instead asks the loader for the required producer's accepted -normalize artifact. That lookup reuses the existing normalize files, requires -workspace schema v4 plus an exact non-empty invocation identity, and deliberately -does not require extract or merge checkpoint files or dependency fingerprints. -The runner performs canonical codec and producer-provenance validation before -cloning the artifact into normal step output. Success restores only stored -normalize diagnostics and emits one normalize decision; failure retains the -files, records the decision, and stops without executing the producer or -consumer. +## Filesystem Store And Failure Handling -The loader assigns a typed category and reason code at each validation site; -diagnostic prose is not classified after the fact. The runner then applies -forced-execution policy, validates reusable artifact bytes through the prepared -codec once, returns the canonical hydrated value to the stage, and records the -final decision before enforcing a required-predecessor failure. That failure -names only the step, lane, and stable reason code. Decision detail is selected -from code-owned descriptions by reason code and then UTF-8 normalized and -bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and -codes remain intact through pipeline events and become strings only in manifest -and debug-summary JSON. -[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the -canonical operator-facing reason-code reference. +[artifactcache.FilesystemStore](../../internal/framework/artifactcache/filesystem.go) +implements the narrow `contracts.ArtifactCache` interface. Construction retains +the exact selected root without creating it. Lookup checks one confined regular +JSON entry; missing data is an ordinary miss. Stored bytes are the unchanged +accepted artifact, without provenance/execution envelopes or alternate key +dimensions. Physical placement belongs in +[Operations](../operations.md#state-surfaces). -`internal/core/fileio` provides confined atomic file writes used by state -collaborators. Its exact-mode writer remains available to output and debug -callers; its process-permission writer is used by checkpoints. The chunk-plan -store retains its stronger rooted entry validation. [Operations](../operations.md#cache-provisioning) -owns cache mode and provisioning guidance. +The store honors the resolved read/write mode. Refresh does not read or remove +previous acceptance in anticipation of execution. Only a newly supplied accepted +value can replace it. Failed refresh execution or a failed cache write preserves +the old entry. +Legacy data is not scanned, converted, rewritten, or deleted. -The CLI constructs selective-recomputation policy from resolved generated -artifact dependencies. It forces the selected step and transitive consumers, -while marking unforced producers as required reusable inputs. The runner owns -the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute) -owns the operator workflow and stable reason-code meanings. +Publication uses [fileio](../../internal/core/fileio/fileio.go)'s confined atomic +process-permission writer. New cache objects use process/deployment creation +permissions; there is no permission convergence on an existing root. +[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md) owns that decision. -## Debug Bundles +[Cache integration](../../internal/framework/pipeline/producer_cache.go) treats +unreadable/malformed/codec-unusable entries as diagnostic misses. Store failures +become scoped process warnings without failing successful production or calling +the provider again. Artifact encoding, caller cancellation, requested evidence, +and observer failures retain their own error boundaries; they are not swallowed +as best-effort storage failures. -`internal/core/debugbundle` allocates an explicitly requested per-run bundle -with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command, -resolution, run, final grouped diagnostic, and failure artifacts. `internal/framework/debug` -implements the pipeline-facing trace recorder under the trace root. +Accepted predecessors can survive downstream failure and be reused by an ordinary +rerun. There is no checkpoint loader, resume barrier, forced-recompute policy, or +canonical partition selected solely by source digest. New message selections +populate independently; an accepted plan is rematerialized, not restored with a +historic execution claim. -The CLI allocates a bundle before pipeline resolution and treats requested -summary or trace persistence failures as command failures. The pipeline's debug -boundaries redact sensitive metadata and credential-shaped bytes while allowing -application-owned trace material. Debug data is never a checkpoint source or -cache input. +## Artifact Publication And Native Reporting -Generated reference bytes exist only in cloned operation requests and are not -written as paths into checkpoints, manifests, or debug summaries. Those state -surfaces retain canonical identities and bounded producer provenance so that a -resume decision can be explained without copying generated campaign content. +The runner returns logical deliverable files only after successful execution/ +encoding. The [JSON encoder](../../internal/modules/generic/output/json/encoder.go) +owns domain-file selection and optional auxiliary inclusion. +[JSON output](../integrations/json-output.md), +[chunk maps](../integrations/chunk-map.md), and +[evidence context](../integrations/evidence-context.md) own logical paths/schemas, +not the store or recorder. -After allocation, one CLI-owned state value accumulates the known report paths, -pipeline outcome counts, and validation status. A single guarded terminalization -operation writes the success report, or makes one attempt each to write the -failure report and error log. Terminal persistence failures are reported -separately and never replace the command's primary error. +[writeOutputFiles](../../internal/cli/run_output.go) validates every logical +path before allocating the run directory, refuses an existing destination, and +writes files with confined atomic helpers. This is per-file publication, not +an all-files transaction. A later write can leave a partial new directory; +existing siblings are preserved. No rollback deletes available artifacts. -## Invariants To Preserve +After complete publication, the run handler finalizes requested capture before +[reportRunResult](../../internal/cli/run_result.go) submits the small native +completion value. The report is not automatically persisted or required to decode +domain files. No index, manifest, receipt, or process-diagnostic sidecar is emitted. +Artifact count counts returned deliverables, including selected available auxiliaries, +not cache/debug/intermediate files. -- Modules receive state collaborators and request data, never physical roots. -- Output logical paths are validated before a run directory is allocated, and - files are atomically written within that directory. -- Chunk-plan publication occurs only for accepted plans; bypass does not - construct or touch a plan store. -- Checkpoint recording and checkpoint loading remain separate collaborators. -- Debug state is opt-in, is not cache input, and terminal reporting does not - obscure the command's primary failure. +Post-publication capture/render/delivery failure remains nonzero with available +locations. Valid artifacts remain; model work and capture terminalization are not +repeated. Native delivery evidence is separate from the operation cause, and +failed stdout may be partial. Public outcomes and caller responsibilities belong +in [CLI Reference](../cli.md#output-streams-and-exit-statuses) and the +[subprocess guide](../consumers/subprocess.md). -## Tests To Inspect +## Capture Lifecycle -- `internal/cli/run_contract_test.go`: command-owned state allocation, - terminalization, and output/report boundaries. -- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection, - and resume collaborator construction. -- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, - permissions, cleanup, and redaction. -- `internal/cli/recompute_policy_test.go`: forced dependents and required - reusable predecessors for selective recomputation. -- `internal/cli/recompute_execution_contract_test.go`: selective recomputation, - filesystem recovery, deterministic decisions, and failed predecessor state. -- `internal/cli/production_contract_test.go`: production composition and - configuration validation at the CLI boundary. -- `internal/cli/example_contract_test.go`: maintained example ownership. -- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes. -- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and - reuse. -- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement, - publication, and permissions. +[Debug selection](../../internal/cli/run_debug.go) preserves native boolean and +supplied directory semantics independently of verbosity. The run path alone +calls [debug.Start](../../internal/framework/debug/recorder.go). +[Bundle allocation](../../internal/core/debugbundle/bundle.go) validates identity, +creates a new absolute per-run directory, and atomically updates a relative +latest link at start. Failure can return an available directory for explanation/ +retention. Disabled capture creates nothing; completion never retargets the link. + +One synchronized recorder writes evidence incrementally, retaining lifecycle +state and the first recording error rather than accumulating whole-run content. +Callback failures are exposed at the next boundary that can return an error. +`Finalize` attempts terminal evidence once, joins primary and capture causes, +and retains its result. Repeated calls do not rewrite evidence or retarget latest. +A later presentation failure does not change capture into another terminal run. +Allocation/link/record/finalization failures never trigger model retries or a +fallback directory. Available files are retained. + +## Native And Application Evidence + +[Run capture](../../internal/cli/run_capture.go) snapshots the original secret-aware +application/PromptKit configuration and source evidence. Scoped pipeline capture +records source/reference/incoming artifacts, normalization originals/mappings/ +proposals, candidates, complete findings/guidance, fallback/deterministic outcomes, +and cache selection/current artifact bytes. + +[NativeCapture](../../internal/framework/pipeline/native_capture.go) composes +existing observers and preparation callbacks. Preparation callbacks retain +detached details cheaply; ordinary boundaries write prepared contracts, native +results, raw response text, and full readable exchanges. Events are written +incrementally. Scope includes operation/step/lane/chunk/validator identity to +avoid concurrent evidence collisions. + +One invocation `ExecutionGroup` enrolls actual initial, continuation, and nested +validator executions once. Native identity, terminal coverage, and unknown usage +remain native facts, not a copied profile/attempt accounting model. +Hits can retain prepared evidence without execution or usage. Result-less failures +retain availability/cause evidence; undelivered responses are explicitly unassessed. + +Native credential omissions and ConfigKit marked-secret paths are preserved. +Arbitrary source/prompt/response text is not heuristically rewritten or made safe +for general logging. Exact evidence names/nesting/serialized forms are private, +not a public protocol or reconstruction requirement. Debug is neither cache +input nor an artifact-discovery interface; [Operations](../operations.md#debug-bundles) +owns handling. + +## Focused Verification + +- [Filesystem cache](../../internal/framework/artifactcache/filesystem_test.go) + and [permissions](../../internal/framework/artifactcache/filesystem_permissions_unix_test.go) + cover raw bytes, modes, atomic replacement, diagnostics, confinement, deployment + permissions, and unchanged legacy files. +- [Producer cache](../../internal/framework/pipeline/producer_cache_test.go), + [typed runner cache](../../internal/framework/pipeline/runner_cache_test.go), and + [plan cache](../../internal/framework/pipeline/chunk_plan_cache_test.go) cover + initial handles/keys, final acceptance, current-source checks, and failed refresh. +- [Recorder](../../internal/framework/debug/recorder_test.go) and + [allocation](../../internal/core/debugbundle/bundle_test.go) cover incremental + failure retention, concurrent starts, and one terminalization. +- [Native evidence](../../internal/framework/pipeline/native_capture_test.go) and + [configuration capture](../../internal/cli/run_capture_test.go) cover available + content, exact sensitivity, observer composition, and no duplicate preparation. +- [Output writes](../../internal/cli/run_id_test.go) and + [native result boundaries](../../internal/cli/run_result_command_test.go) cover + collisions, partial directories, capture/presentation failure, and retained causes. +- [Assembled recovery](../../internal/cli/recovery_contract_test.go) covers + downstream fix/rerun, initial-key correction reuse, qualified fallback, actual + generated consumers, and debug on hits/failure. + +Use focused pipeline/cache/recorder/CLI checks for state-boundary changes. +[Development](../development.md#validation) owns wider validation. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 4c451aa5..c67fd138 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2054,6 +2054,27 @@ check cross-links and stale package references. ## Stage 42: Refresh Pipeline And State Internal Guides +**Status:** Complete. Rewrote both guides around current typed resolution, +materialization/preparation, operation-scoped complete assessment, native +continuation, strict/permissive disposition, and supported fallback. Documented +one ready-job pool, ordered barriers/results, cancellation draining, and separate +native backend admission. Handoffs distinguish usable qualified artifacts from +ordinary acceptance and propagate reuse exclusion only to actual dependents. + +State guidance now traces the retained initial prepared handle/hash, durable +typed/plan usability on hits without semantic reassessment, final acceptance +writes, best-effort cache I/O, independent publication, and one incremental +capture lifecycle with native observations. Removed retired checkpoint/resume, +separate pool, generic retry, reporting-sidecar, copied accounting, and heuristic +redaction flows. Public controls, paths, payloads, and handling link to canonical +owners; debug representation remains private. Inspected current runner/helper, +assessment, coordinator, handoff, cache, allocation/recorder, evidence, and CLI +publication code and focused regressions. All 84 local links/anchors resolve; +no current incoming anchors require repair. Pipeline, cache, recorder, bundle, +file I/O, and CLI package tests and vet pass; removal searches and +`git diff --check` pass. No runtime code/data or operator contract changed. +Module and D&D contributor recipes remain assigned to the next stage. + **Goal:** explain acceptance, concurrency, recovery, and capture as implemented. **Inspect/change:** `docs/internal/{pipeline,state}.md`; typed helper/assessment,