Replace checkpoint proposals with accepted-artifact caching
This commit is contained in:
@@ -30,15 +30,19 @@ explicit-empty directory behavior are resolved below. Item 6 selects verificatio
|
||||
migration guidance, and incorporation into the master roadmap; that incorporation
|
||||
is still pending.
|
||||
|
||||
The major architectural decisions are settled. Remaining choices are run-report
|
||||
fields/versioning and durable record formats (item 4/U9). Function signatures,
|
||||
bundle filenames other than `latest`, and similar implementation details do not
|
||||
The cache follow-up in item 4 replaces the earlier checkpoint design with an
|
||||
accepted-artifact cache keyed by initial message content. The output/report
|
||||
contract has been reopened around consumer needs; it is not merely waiting for
|
||||
field names and versions. Function signatures, bundle filenames other than
|
||||
`latest`, and similar implementation details do not
|
||||
reopen the selected architecture. Broader PromptKit choices remain with the
|
||||
master roadmap.
|
||||
|
||||
The PromptKit architecture and checkpoint decisions remain in force. This
|
||||
discussion develops the CommandKit portion of that migration and its interactions
|
||||
with U1 (dependencies), U9 (records/reporting), and U10 (CommandKit adoption).
|
||||
The PromptKit ownership direction remains in force. The revised
|
||||
[cache decision](promptkit-architecture-discussion.md#5-accepted-artifact-cache)
|
||||
supersedes earlier checkpoint matching and provenance requirements. Master-roadmap
|
||||
incorporation is pending, including U1 (the upstream content-hash capability),
|
||||
U8 (cache migration), U9 (records/reporting), and U10 (CommandKit adoption).
|
||||
|
||||
## Agreed Proposal
|
||||
|
||||
@@ -60,10 +64,10 @@ Documenting the transition replaces a requirement to keep the old CLI working.
|
||||
Update canonical CLI and consumer documentation when the behavior lands.
|
||||
|
||||
Application responsibilities such as producing artifacts, reporting publication
|
||||
outcomes, and managing checkpoint/debug storage remain meaningful. Their existing
|
||||
outcomes, and managing cache/debug storage remain meaningful. Their existing
|
||||
CLI expression is not protected by an implicit compatibility exception. This
|
||||
guiding rule does not itself choose stored artifact or checkpoint formats; those
|
||||
remain separate application decisions in the master roadmap.
|
||||
guiding rule does not itself choose stored artifact formats. The revised cache
|
||||
decision below selects simple artifact storage rather than checkpoint envelopes.
|
||||
|
||||
### Why Adopt Together
|
||||
|
||||
@@ -88,7 +92,7 @@ spending migration effort recreating existing CLI conventions.
|
||||
| Command hosting, standard input/help mechanics, common report formatting and visibility | CommandKit. |
|
||||
| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. |
|
||||
| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. |
|
||||
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, checkpoints, and domain outcomes | Notarius application handlers and framework. |
|
||||
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, accepted-artifact caching, and domain outcomes | Notarius application handlers and framework. |
|
||||
| Receipt content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
|
||||
|
||||
Adopt the standard capability suite selected in item 1, including upstream
|
||||
@@ -104,9 +108,9 @@ meaning and selects report content. CommandKit supplies presentation mechanics.
|
||||
Two application needs already identified for deliberate treatment are:
|
||||
|
||||
- The current `run --json` publishes a receipt after output publication. Item 4
|
||||
replaces its presentation with one compact application report rendered natively
|
||||
for humans and machines. Exact fields/versioning remain with U9; reports must
|
||||
describe the actual publication and domain outcome.
|
||||
records native reporting mechanics, but the output/report contract has been
|
||||
reopened around artifact delivery and optional metadata. Its earlier mandatory
|
||||
run-report proposal is subject to that discussion.
|
||||
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when an
|
||||
effective debug directory is available, including an application default.
|
||||
Capture selection remains independent of the final reporting mode.
|
||||
@@ -191,8 +195,11 @@ are selected native features; this does not reintroduce legacy Notarius aliases.
|
||||
- Retain application handlers for pipeline execution and discovery, with native
|
||||
commands, inputs, and result presentation as selected in items 2–5.
|
||||
|
||||
No upstream feature request has been identified as a blocker for this scope.
|
||||
Evaluate any later demonstrated gap on its actual application need. On roadmap
|
||||
The standard capability suite needs no identified upstream addition. The later
|
||||
cache decision does require a native message-content hash, which is not the
|
||||
existing v0.23.3 rendered-prompt hash. Confirm the supplying API/release before
|
||||
implementing cache lookup; the version table remains the reviewed baseline, not
|
||||
a claim that the new hash is already available. On roadmap
|
||||
reconciliation, incorporate these versions into U1, replace U10's open standard
|
||||
capability selection with this decision, and incorporate the application wiring
|
||||
and presentation decisions from items 2–5. Preserve the remaining record choices
|
||||
@@ -268,7 +275,7 @@ omitted; application validation still decides whether a supplied value is valid.
|
||||
| Lane selection | Repeatable `--lane ID`; omission selects the normal pipeline lane set. Replace comma-separated `--only`. |
|
||||
| Output placement | Application-owned `--output-dir PATH`. |
|
||||
| Chunk-plan caching | `--chunk-cache auto\|bypass\|refresh`. |
|
||||
| Checkpoint reuse | Keep `--resume` and `--recompute-step ID` with the agreed recovery policy. |
|
||||
| Accepted-artifact cache | Replace checkpoint-centric controls with cache reuse/bypass/refresh behavior; concrete CLI spelling remains to be selected. `--resume` and `--recompute-step` are migration context, not compatibility requirements. |
|
||||
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
|
||||
| Configuration-valued tuning | Supported application `--set POINTER=JSON` overrides, including worker and semantic-budget settings once their configuration design is settled. |
|
||||
|
||||
@@ -500,8 +507,8 @@ preserving supplied operator values. Pipeline modules construct their own domain
|
||||
requests; share concrete request-building functions when useful without
|
||||
simulating pipeline execution to populate an inspection command.
|
||||
|
||||
Output publication, checkpoints, debug capture, and execution records belong to
|
||||
the run path. Loaders do not initialize those resources; item 5 selects debug
|
||||
Output publication, artifact caching, debug capture, and execution records belong
|
||||
to the run path. Loaders do not initialize those resources; item 5 selects debug
|
||||
allocation timing. Remove command-specific loading paths, copied configuration
|
||||
and resource inspection, eager runtime setup, and duplicated preflight
|
||||
orchestration. No broader application/package reorganization is required by this
|
||||
@@ -514,10 +521,56 @@ Evidence: [native suite and loader ownership](../../../commandkit/docs/consumers
|
||||
|
||||
## 4. Reports, Machine Output, And Outcomes
|
||||
|
||||
**Discussion / decision:** Agreed. Use native CommandKit reports for command
|
||||
results, native outcomes for command completion, and a compact application-owned
|
||||
run report for artifact discovery and domain status. Distinguish pipeline results,
|
||||
command completion, and presentation delivery. Implementation is pending.
|
||||
**Discussion / decision:** Native reporting and outcome mechanics remain selected.
|
||||
The output/report contract is reopened: consumers primarily need resulting
|
||||
artifacts, metadata may be optional, failures need useful nonzero outcomes, and
|
||||
operators/developers use debug capture for detailed investigation. The earlier
|
||||
run-report and durable-metadata proposals below remain discussion material for
|
||||
that review, not requirements to preserve or expand. No new report/manifest
|
||||
schemas have been selected. Finalize the cache decision first; return to the
|
||||
consumer output/report contract afterward. Implementation is pending.
|
||||
|
||||
### Accepted-Artifact Cache And PromptKit Ownership
|
||||
|
||||
**Agreed follow-up:** Rename and replace the checkpoint subsystem with the
|
||||
[accepted-artifact cache](promptkit-architecture-discussion.md#5-accepted-artifact-cache).
|
||||
That decision owns the matching and storage policy; this section records its
|
||||
CommandKit and execution-boundary implications.
|
||||
|
||||
The cached value is an accepted answer to the original task. Compute the key
|
||||
from the initial prepared request using a proposed upstream `MessageContentHash`
|
||||
over ordered message roles and exact text, excluding settings, session/execution
|
||||
IDs, timestamps, and provider cache hints. Do not hash the entire prepared
|
||||
request or introduce another Notarius-owned fingerprint policy.
|
||||
|
||||
Look up `cache/<hash>.json` before model execution. A hit loads the final accepted
|
||||
Notarius artifact directly, subject only to ordinary decoding/usability checks.
|
||||
It does not execute a model request, reconstruct a result/exchange, replay
|
||||
corrections, or repeat semantic validation merely because the artifact was cached.
|
||||
|
||||
On a miss, execute the complete Notarius operation: native PromptKit calls,
|
||||
domain interpretation and validation, and any application-directed correction
|
||||
passes. Only after final acceptance save the artifact under the initial hash,
|
||||
even if correction required several turns. For normalization, cache the accepted
|
||||
normalized artifact after applying the plan, not the model's proposed plan.
|
||||
Rejected, failed, and validation-incomplete work is not published to the cache.
|
||||
|
||||
PromptKit owns structural response validation, internal retries, repairs, and
|
||||
other mechanics inside each request execution. Notarius consumes a usable
|
||||
response or terminal failure at the public boundary and owns domain acceptance.
|
||||
Internal PromptKit attempts do not determine cache eligibility. Native debug
|
||||
evidence about those attempts does not transfer their ownership to Notarius.
|
||||
|
||||
The upstream requirement is message-content hashing. It does not require native
|
||||
cached-response injection, replay, exchange restoration, or write-through
|
||||
response caching. A small Notarius filesystem lookup/write facility stores the
|
||||
accepted artifact without checkpoint manifests, dependency fingerprints,
|
||||
acceptance ledgers, or required execution provenance. A cache hit is not a new
|
||||
model execution or a claim of validation under a changed policy.
|
||||
|
||||
Concrete cache flags and legacy-checkpoint handling remain implementation-planning
|
||||
choices. The output/report discussion must not reintroduce cache metadata solely
|
||||
to populate a report. The master roadmap still needs this replacement policy.
|
||||
|
||||
### One Report Value For Human And Machine Output
|
||||
|
||||
@@ -633,10 +686,10 @@ status reflects presentation failure while the published artifacts remain a
|
||||
fact. Do not rewrite pipeline history as failed execution or automatically rerun
|
||||
model work because report delivery failed.
|
||||
|
||||
Notarius owns stage/lane/chunk progress, domain acceptance, and checkpoint reuse.
|
||||
Notarius owns stage/lane/chunk progress, domain acceptance, and cache reuse.
|
||||
PromptKit supplies native execution observations and `ExecutionGroup` totals;
|
||||
CommandKit presents the selected facts. Explicitly enroll the relevant initial,
|
||||
continuation, and validator executions without double counting. Checkpoint reuse
|
||||
continuation, and validator executions without double counting. Cache reuse
|
||||
does not add historical token usage to current execution totals, and unavailable
|
||||
usage remains distinguishable from zero.
|
||||
|
||||
@@ -742,7 +795,7 @@ context where useful:
|
||||
| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
|
||||
| Returned `RunResult` | Available model output and native validation result. |
|
||||
| `Exchange.Details()` | Explicit inspection of captured conversation history. |
|
||||
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and checkpoint reuse. |
|
||||
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and cache reuse. |
|
||||
|
||||
Compose capture and accounting observers. Remove the debug completion wrapper,
|
||||
copied execution summaries, synthetic call identities, and duplicate terminal
|
||||
@@ -757,12 +810,14 @@ Capture only available evidence. Preparation/admission failures may have no
|
||||
`OnPrepared` notification; observer events have metadata, not response bodies;
|
||||
operational failures may return no `RunResult`. Do not present the bundle as a
|
||||
complete provider transcript. Use ordinary `Run` with native hooks, without
|
||||
forcing split preparation/execution solely for recording.
|
||||
forcing split preparation/execution solely for recording. The cache's lookup
|
||||
before execution is a separate reason to use native preparation; `OnPrepared`
|
||||
notifications alone do not provide cache short-circuiting.
|
||||
|
||||
Conversation capture required for continuation remains in-process execution
|
||||
state. Persisting its history requires enabled debug capture. Normal manifests,
|
||||
checkpoints, and run reports retain their minimal projections independently of
|
||||
debug selection.
|
||||
state. Persisting its history requires enabled debug capture. Cached values are
|
||||
accepted domain artifacts, not conversation state or provenance records. The
|
||||
separate output/report contract remains under discussion.
|
||||
|
||||
### Capture Lifecycle And Failure Policy
|
||||
|
||||
@@ -783,7 +838,7 @@ Treat link-update failures under the requested-capture failure policy below.
|
||||
One application-owned recorder records initial context, accepts native execution
|
||||
evidence and application decisions, and finalizes once with the known pipeline,
|
||||
publication, and capture outcomes. Debug remains separate from output/cache,
|
||||
never supplies checkpoint reuse, and is not automatically deleted.
|
||||
never supplies cache reuse, and is not automatically deleted.
|
||||
|
||||
Requested capture is required work:
|
||||
|
||||
@@ -891,8 +946,9 @@ application outcome used for the CLI report. Reference detailed records where
|
||||
useful; include convenient summary fields without maintaining a second execution
|
||||
ledger. Configuration and resolved composition normally need one snapshot each.
|
||||
Use native execution IDs with application scope to connect records.
|
||||
Reused work has a restored-payload snapshot and reuse decision linked to original
|
||||
provenance, not a fabricated model execution.
|
||||
Reused work has a loaded-artifact snapshot and current cache-hit decision, not
|
||||
a fabricated model execution. Historical provenance is not required in cache
|
||||
files or reconstructed solely for debug capture.
|
||||
|
||||
Retain enough available context to investigate a run that fails before publishing
|
||||
output. Do not depend entirely on a successful output manifest or automatically
|
||||
@@ -905,7 +961,7 @@ competing transcript model or deduplication system to eliminate repeated history
|
||||
**Resolved:** Use one default capture depth preserving every application-visible
|
||||
turn, its useful inputs and outputs, and the decisions made about it. An operator
|
||||
should be able to inspect a failed operation without reconstructing deltas,
|
||||
finding a checkpoint, or extracting model text from escaped JSON strings.
|
||||
finding a cache entry, or extracting model text from escaped JSON strings.
|
||||
Convenience governs both capture and inspection; direct native serialization
|
||||
and a simple readable rendering of the same values can both be useful.
|
||||
|
||||
@@ -988,9 +1044,9 @@ correction inputs even if the next execution fails during preparation; successfu
|
||||
preparation captures the actual rendered correction in its native messages.
|
||||
|
||||
Retain deterministic stage outputs too: chunk plans and materialized chunks,
|
||||
merged artifacts, and normalized artifacts. For checkpoint reuse, retain the
|
||||
restored payload and reuse decision linked to original provenance. These records
|
||||
do not fabricate a current execution or new validation of reused work.
|
||||
merged artifacts, and normalized artifacts. For cache reuse, retain the loaded
|
||||
artifact and current hit decision without requiring historical provenance. These
|
||||
records do not fabricate a current execution or new validation of reused work.
|
||||
|
||||
#### Failed Runs And Native Availability
|
||||
|
||||
@@ -1046,9 +1102,11 @@ Select a small set of workflows that collectively establish:
|
||||
flags, supplied-value distinctions, and overrides mapped correctly.
|
||||
- Reports, diagnostics, and exit outcomes follow item 4, distinguishing domain
|
||||
rejection, command failure, and presentation failure after publication.
|
||||
- A downstream failure followed by resume reuses successful predecessors under
|
||||
the simpler checkpoint policy, with explicit recomputation and consumed-input
|
||||
changes handled as selected in the master roadmap.
|
||||
- A downstream failure followed by another invocation reuses accepted artifacts
|
||||
under the revised cache policy. Final corrected artifacts are stored under the
|
||||
initial message hash; changed messages miss, settings-only changes can hit,
|
||||
and hits neither make model calls nor repeat domain validation. Rejected or
|
||||
validation-incomplete work is never stored as an accepted cache value.
|
||||
- Debug capture follows item 5's directory selection, reporting-mode independence,
|
||||
`latest` update timing, useful evidence, and allocation/write/finalization
|
||||
failure behavior.
|
||||
@@ -1090,17 +1148,17 @@ old-to-new migration map covering:
|
||||
- Application and PromptKit configuration changes, source discovery, profiles,
|
||||
and companion files.
|
||||
- Native backend admission, independent worker settings, and revised retry controls.
|
||||
- Checkpoint reuse, explicit recomputation, and the selected treatment of existing
|
||||
checkpoint files.
|
||||
- Accepted-artifact cache reuse, explicit refresh/bypass, and the selected
|
||||
treatment of old checkpoint files.
|
||||
- Machine reports, exit outcomes, and the distinction between successful command
|
||||
execution and accepted domain results.
|
||||
- Debug capture under `/tmp/notarius-debug`, `latest`, and the interaction between
|
||||
capture and reporting modes.
|
||||
|
||||
CLI backward compatibility remains a non-goal; migration guidance does not imply
|
||||
compatibility shims or preservation of old command behavior. Existing checkpoint
|
||||
and published-artifact compatibility remain explicit application decisions about
|
||||
stored work and consumer data, separate from CLI syntax.
|
||||
compatibility shims or preservation of old command behavior. Legacy checkpoint
|
||||
handling and published-artifact compatibility remain explicit application
|
||||
decisions, without assuming that old checkpoint envelopes must survive.
|
||||
|
||||
Maintain runnable examples for embedded defaults and operator overrides. Update
|
||||
canonical CLI, configuration, operations, integration/consumer, and internal
|
||||
@@ -1121,7 +1179,7 @@ master roadmap. Use this phase sequence:
|
||||
including reporting and useful debug capture.
|
||||
4. Migrate remaining roles and remove the adapter, scheduler, retry machinery,
|
||||
and custom CLI infrastructure.
|
||||
5. Complete checkpoint migration, durable records, and capture behavior.
|
||||
5. Complete cache migration, the selected output/report contract, and capture behavior.
|
||||
6. Verify the assembled application and finish migration documentation.
|
||||
|
||||
Each phase delivers a reviewable result with appropriate verification. Tests and
|
||||
@@ -1130,8 +1188,10 @@ completeness. The phases do not require separate releases or an intermediate
|
||||
compatibility framework.
|
||||
|
||||
Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow
|
||||
U9 to remaining durable-field and accounting choices, and debug implementation
|
||||
U9 to the reopened consumer output/report contract, and debug implementation
|
||||
details within item 5's resolved inventory, depth, and directory controls.
|
||||
Update U1 for the upstream message-content hash and replace U8's checkpoint
|
||||
identity/provenance plan with the accepted-artifact cache decision.
|
||||
Reconcile removal inventory and acceptance criteria, including old
|
||||
inspection-command and machine-output compatibility assumptions. Preserve
|
||||
unresolved schema handling, role-specific retry/fallback policy, worker defaults,
|
||||
|
||||
@@ -2,11 +2,18 @@
|
||||
|
||||
## Purpose And Status
|
||||
|
||||
The decisions in items 1–6 have been incorporated into the
|
||||
The original decisions in items 1–6 were incorporated into the
|
||||
[PromptKit-native architecture migration roadmap](promptkit.md). That roadmap
|
||||
is the implementation plan and owns current unresolved choices, sequencing,
|
||||
acceptance criteria, and implementation status.
|
||||
|
||||
The subsequent CommandKit discussion replaces item 5's checkpoint proposal with
|
||||
the accepted-artifact cache below and clarifies PromptKit's execution ownership.
|
||||
Those updates have not yet been incorporated into the master roadmap. They
|
||||
supersede its earlier checkpoint matching and provenance requirements. The
|
||||
[CommandKit notepad](commandkit-migration-discussion.md) records the related CLI,
|
||||
debug, and output/report discussion; the output/report contract is being revisited.
|
||||
|
||||
This document retains the discussion, rationale, and evidence, including item 7's
|
||||
completed Audita comparison. Item-level questions and follow-ups below record
|
||||
the discussion as it developed; they are not a second implementation plan.
|
||||
@@ -57,7 +64,7 @@ resulting change to the adoption target remains to be selected.
|
||||
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
|
||||
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
|
||||
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
|
||||
| Application records and CLI | Store artifacts and checkpoints, attach pipeline meaning to execution facts, and present results through CommandKit. |
|
||||
| Application records and CLI | Store artifacts and accepted-artifact cache entries, attach pipeline meaning to execution facts, and present results through CommandKit. |
|
||||
|
||||
Engine construction becomes ordinary application wiring using item 2's embedded
|
||||
defaults and operator sources. Profile selection is expressed through native
|
||||
@@ -69,8 +76,8 @@ Use native requests, results, exchanges, errors, and observers directly in
|
||||
model-executing application code. Pure domain models, codecs, source-reference
|
||||
resolution, and deterministic transformations need no PromptKit dependency.
|
||||
Retain application interfaces for extracting artifacts, validating typed
|
||||
candidates, encoding output, and loading checkpoints. Deterministic stages
|
||||
continue to operate without conversation machinery.
|
||||
candidates, encoding output, and loading accepted cached artifacts. Deterministic
|
||||
stages continue to operate without conversation machinery.
|
||||
|
||||
### Connect Modules To Pipeline Assessment
|
||||
|
||||
@@ -116,8 +123,9 @@ that wrapper stack around native calls.
|
||||
|
||||
Attach pipeline scope through native callbacks and observers. Use native
|
||||
`ExecutionGroup` totals and small application projections for manifests and
|
||||
optional debug artifacts, following item 6. Checkpoint matching and durable
|
||||
acceptance provenance remain the application responsibilities selected in item 5.
|
||||
optional debug artifacts, following item 6 and the reopened output/report
|
||||
discussion. Item 5 now selects a simple accepted-artifact cache, without required
|
||||
durable acceptance or execution-provenance records.
|
||||
|
||||
Audita demonstrates shared engine assembly, native continuation, and CommandKit
|
||||
integration. Its `RoleClient`, structured-output interface, and opaque exchange
|
||||
@@ -225,20 +233,23 @@ exchange wrapper are not part of the selected direction.
|
||||
The flow and removal/adaptation direction below are agreed. Begin with one
|
||||
conversation per producer operation and complete replacement model responses.
|
||||
Retain application assessment, safe-fallback decisions, artifact validation,
|
||||
and checkpoint eligibility. Detailed response contracts, budget migration, and
|
||||
and cache eligibility. Detailed response contracts, budget migration, and
|
||||
the open policy questions below remain to be resolved. No application changes
|
||||
have been implemented.
|
||||
|
||||
### Agreed Flow
|
||||
|
||||
For cacheable operations, item 5's initial preparation and lookup precede this
|
||||
flow. The steps below describe execution on a miss.
|
||||
|
||||
1. Prepare the source context, references, and any request-local candidate-handle
|
||||
mapping once for the producer operation. Run its initial native request with
|
||||
exchange capture when continuation is enabled. Capture is independent of
|
||||
detailed debug recording.
|
||||
2. Inspect native structural validation, decode the response, and perform the
|
||||
module's deterministic interpretation and safety checks. Run the configured
|
||||
validation chain on a materialized candidate. A module can identify actionable
|
||||
defects before a candidate is ready for that chain.
|
||||
2. Handle the native call's terminal success or failure, decode a usable response,
|
||||
and perform the module's deterministic interpretation and safety checks. Run
|
||||
the configured validation chain on a materialized candidate. A module can
|
||||
identify actionable defects before a candidate is ready for that chain.
|
||||
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
|
||||
and a configured `AdditionalTurns` budget. The callback selects a native
|
||||
repair prompt and supplies actionable findings as variables. PromptKit retains
|
||||
@@ -295,18 +306,27 @@ plan checks, typed application rules, and candidate isolation. In particular,
|
||||
normalization must still reject unknown or overlapping candidate handles and
|
||||
apply only safe groups. A final fallback can retain safe merges while leaving
|
||||
unsafe groups separate; it must go through configured artifact validation and
|
||||
terminal policy before it can advance. After native structural-repair exhaustion,
|
||||
a supported deterministic fallback can be assessed directly instead of first
|
||||
terminal policy before it can advance. After a native terminal failure covered
|
||||
by the selected fallback policy, a supported deterministic fallback can be assessed
|
||||
directly instead of first
|
||||
restarting the same model operation through a feedback-free module retry. Which
|
||||
earlier safe candidate, if any, may survive a later failed turn remains an explicit
|
||||
policy question; do not introduce implicit best-so-far acceptance.
|
||||
|
||||
Retain the distinction between a rejection and unavailable validation. Under the
|
||||
current `warn_continue` policy, incomplete validation can advance a structurally
|
||||
valid candidate but prevents checkpoint reuse/publication, including for derived
|
||||
results. Continuation adoption does not itself change that policy, safe-fallback
|
||||
eligibility, or durable provenance. Keep raw conversation state in process and
|
||||
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
|
||||
valid candidate but prevents accepted-artifact cache reuse/publication, including
|
||||
for derived results. Continuation adoption does not itself change that policy
|
||||
or safe-fallback eligibility. Keep raw conversation state in process and
|
||||
out of cached artifacts; explicit detailed debug remains separate. Item 5
|
||||
supersedes checkpoint-specific publication and provenance machinery.
|
||||
|
||||
PromptKit owns everything within an individual request execution, including
|
||||
structural validation, internal retries, and repairs. Notarius consumes the
|
||||
public result or failure and owns domain interpretation, validation, and any
|
||||
application-directed correction passes. Do not inspect or reproduce internal
|
||||
PromptKit attempts to decide cache eligibility; cache only the final accepted
|
||||
Notarius artifact after its complete application process succeeds.
|
||||
|
||||
PromptKit should own validation and repair of the private model response.
|
||||
Notarius's serialized schema validator currently receives the codec-produced
|
||||
@@ -334,7 +354,7 @@ and the native continuation evidence in item 7.
|
||||
- Failure handling for non-LLM operations and any justified recovery beyond
|
||||
native execution budgets.
|
||||
- Item 2's deferred filesystem and schema assembly choices, item 5's remaining
|
||||
checkpoint implementation details, and item 6's durable bookkeeping design.
|
||||
cache implementation details, and the reopened output/report contract.
|
||||
|
||||
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
|
||||
roadmap. Record the changed architecture in a successor to
|
||||
@@ -342,7 +362,7 @@ roadmap. Record the changed architecture in a successor to
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
Migration checks should cover accumulated history, fixed handle mappings,
|
||||
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
|
||||
validator failure, cancellation, and checkpoint eligibility. Replace tests of
|
||||
validator failure, cancellation, and cache eligibility. Replace tests of
|
||||
the retired message-reconstruction protocol rather than preserving its mechanics.
|
||||
|
||||
## 4. Global Scheduling And Admission
|
||||
@@ -415,103 +435,118 @@ Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
|
||||
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
|
||||
and record the changed ownership in an ADR. Update architecture, configuration,
|
||||
operations, and implementation documentation when the migration lands. Continue
|
||||
with item 5's execution identity and checkpoint reuse discussion.
|
||||
with item 5's accepted-artifact cache discussion.
|
||||
|
||||
## 5. Execution Identity And Checkpoint Reuse
|
||||
## 5. Accepted-Artifact Cache
|
||||
|
||||
**Discussion / decision:** Direction agreed: resuming preserves successful stage
|
||||
results for the same logical work and data; explicit recomputation applies
|
||||
changed execution behavior. Use a simple Notarius-owned matching policy instead
|
||||
of comprehensive execution fingerprints. No application changes have been
|
||||
implemented.
|
||||
**Discussion / decision:** Agreed. Rename and replace the checkpoint subsystem
|
||||
with a simple cache surrounding the complete Notarius producer operation.
|
||||
Compute its key early from the initial prepared request's message contents;
|
||||
store its value late, after Notarius has accepted the final artifact. This
|
||||
supersedes both comprehensive execution fingerprints and the earlier
|
||||
stage/input/prompt-ID matching proposal. Implementation is pending.
|
||||
|
||||
### Purpose And Matching Rule
|
||||
### Purpose And Key
|
||||
|
||||
Checkpoints preserve a successful producer's result when a dependent stage fails.
|
||||
After fixing the failed stage, an operator should be able to retry it without
|
||||
paying to rerun the successful producer. Matching every configuration parameter
|
||||
or implementation revision would undermine that recovery purpose.
|
||||
The cached value is an accepted answer to the original task. After a downstream
|
||||
failure, rerunning the pipeline can reuse that artifact without repeating the
|
||||
expensive model work that produced it. Cheap deterministic work elsewhere in
|
||||
the pipeline does not need a general stage-checkpoint framework.
|
||||
|
||||
The agreed conceptual key is:
|
||||
Use a PromptKit-provided `MessageContentHash` (proposed name) over the initial
|
||||
request's ordered messages, including each message's role and exact text. Exclude
|
||||
model settings, session IDs, execution IDs, timestamps, and provider cache hints.
|
||||
Do not add whole-configuration, stage, prompt-ID, library-version, or validation-
|
||||
policy fingerprints to this matching rule.
|
||||
|
||||
Changing the actual initial messages changes the key. Changing a prompt ID,
|
||||
model, or setting without changing those messages does not. Fixing deterministic
|
||||
interpretation or validation may therefore require explicit refresh/bypass to
|
||||
replace a previously accepted artifact; a hit does not claim fresh validation.
|
||||
|
||||
This is a concrete upstream hash requirement. The reviewed PromptKit v0.23.3
|
||||
`RenderedPromptHash` includes session identity and provider cache hints, while
|
||||
hashing the entire `PreparedRun` would include volatile execution metadata.
|
||||
Neither is the selected key. Expose the content hash through native preparation
|
||||
rather than implementing another message-hashing policy in Notarius. Confirm
|
||||
the supplying upstream API/release during roadmap incorporation; do not claim
|
||||
the proposed field already exists or silently substitute a broader fingerprint.
|
||||
|
||||
### Compute Early, Store After Acceptance
|
||||
|
||||
```text
|
||||
logical stage identity + input-data identity + prompt ID, where applicable
|
||||
Prepare initial request -> obtain message-content hash -> look up cache/<hash>.json
|
||||
Hit: decode the accepted Notarius artifact and continue the pipeline
|
||||
Miss: execute the complete Notarius operation
|
||||
-> native PromptKit calls as needed
|
||||
-> domain interpretation, validation, and correction passes
|
||||
-> on final acceptance, save the artifact under the initial hash
|
||||
```
|
||||
|
||||
Scope the checkpoint to the relevant step/lane/stage and artifact kind so that
|
||||
shared prompt IDs do not mix unrelated results. These are logical identifiers,
|
||||
not hashes of implementations or complete configuration. Deterministic stages
|
||||
need no synthetic prompt identity.
|
||||
Lookup occurs before model execution. Use native preparation to inspect the
|
||||
initial request; preserve that prepared request for execution on a miss where
|
||||
the native API supports it. This is a reason to prepare before deciding whether
|
||||
to execute, distinct from preparing solely to obtain logging details.
|
||||
|
||||
Input data includes the source or chunk being processed, reference material,
|
||||
and upstream artifacts actually consumed. Matching only the original source
|
||||
would miss changes to generated dependencies. Content digests can compactly
|
||||
identify these inputs and check stored payload integrity without requiring an
|
||||
exact fingerprint of the execution environment.
|
||||
Retain the initial hash throughout application-directed correction. Save the
|
||||
final accepted artifact under that hash, not under a later correction request's
|
||||
hash. A subsequent run can then reuse the accepted answer without replaying the
|
||||
failed turns that led to it.
|
||||
|
||||
| Change | Reuse behavior |
|
||||
| --- | --- |
|
||||
| Fix the implementation of a later failed stage | Preserve earlier successful stages with matching work and inputs. |
|
||||
| Edit prompt text while retaining its ID | Reuse; explicit recomputation applies the edit. |
|
||||
| Change model, generation settings, or implementation/validation policy | Reuse otherwise compatible completed work; explicit recomputation applies the change. |
|
||||
| Change the prompt ID or data consumed by the stage | No matching checkpoint under this rule. |
|
||||
| Explicitly request recomputation | Recompute the selected work and its dependents. |
|
||||
The value is the final Notarius artifact, serialized through its existing codec.
|
||||
For normalization, it is the accepted normalized artifact after plan application,
|
||||
not the model's proposed reconciliation plan. Do not store an intermediate raw
|
||||
response, `RunResult`, exchange, or prepared-execution handle as the cache value.
|
||||
|
||||
The existing controls are `--resume` and `--recompute-step`, documented in the
|
||||
[CLI reference](../cli.md#run) and
|
||||
[selective recompute workflow](../operations.md#selective-recompute). Their
|
||||
current strict compatibility checks still need migration to this agreed policy.
|
||||
The accepted tradeoff is that a checkpoint may preserve a successful result
|
||||
produced by code, prompts, or settings that have since improved.
|
||||
Only publish after the whole Notarius process succeeds, including configured
|
||||
domain validation and any application-directed retries/correction passes.
|
||||
Rejected, failed, and validation-incomplete results are not cacheable. Retain
|
||||
the existing exclusion of derivatives of validation-incomplete work without
|
||||
restoring checkpoint manifests or fingerprint graphs to implement it.
|
||||
|
||||
### Matching, Loading, And Provenance
|
||||
A hit loads the accepted artifact directly. Ordinary decoding and usability
|
||||
checks apply; absent or unusable entries are misses. Do not execute a model
|
||||
request, reconstruct native results/exchanges, replay corrections, or repeat
|
||||
semantic validation merely because the value came from cache. Do not count
|
||||
historical model work as current execution.
|
||||
|
||||
A matching key selects a checkpoint; it does not make an unreadable or
|
||||
incompatible artifact usable. Retain completion/acceptance status, integrity
|
||||
checks, and the ability to decode and consume the stored artifact. Exact schema
|
||||
hash equality is not required when the artifact remains compatible.
|
||||
### PromptKit Ownership Boundary
|
||||
|
||||
Only accepted, completely validated results are reusable. Rejected, structurally
|
||||
invalid, or validation-incomplete results and their ineligible derivatives
|
||||
remain outside reusable state. Acceptance on resume refers to the earlier
|
||||
execution: do not automatically rerun semantic validators merely because their
|
||||
configuration changed, or claim the artifact passed the new validation policy.
|
||||
Preserve original provenance; explicit recomputation applies changed behavior.
|
||||
PromptKit owns the mechanics within each request execution, including structural
|
||||
response validation, internal retries, and repairs. Notarius handles a usable
|
||||
response or terminal failure at the public call boundary, then applies its own
|
||||
domain interpretation, validation, correction, and acceptance process.
|
||||
|
||||
Useful execution metadata can remain available for explanation without becoming
|
||||
a matching requirement. Neither complete conversation history nor native
|
||||
prepared-execution handles are needed to locate and reuse a completed artifact.
|
||||
The separate source-addressed canonical chunk-plan policy remains distinct.
|
||||
Internal PromptKit attempts are not inputs to the cache design. Native debug
|
||||
observations may expose evidence about them, but Notarius does not inspect or
|
||||
reproduce those mechanics to determine cache eligibility. The upstream addition
|
||||
needed here is the message-content hash; no cached-response injection, replay,
|
||||
exchange restoration, or PromptKit-owned write-through cache is required.
|
||||
|
||||
### Simplification And Ownership
|
||||
### Storage And Removal
|
||||
|
||||
- Remove the broad PromptKit configuration fingerprint from checkpoint matching,
|
||||
including effective-configuration parsing and whole-catalog enumeration.
|
||||
- Remove library/catalog versions, model settings, prompt-content hashes, and
|
||||
component-policy fingerprints as automatic invalidators under this policy.
|
||||
- Keep straightforward input/dependency identity, successful-stage records,
|
||||
artifact loading checks, storage, and explicit recomputation in Notarius.
|
||||
- Keep PromptKit hashes available for native inspection and diagnostics without
|
||||
making them checkpoint prerequisites.
|
||||
Use a small filesystem lookup/write facility with `cache/<hash>.json` containing
|
||||
the accepted artifact. Ordinary temporary-file/rename publication suffices to
|
||||
avoid exposing partial entries. Cache lookup must not require original output
|
||||
or debug directories, acceptance ledgers, execution provenance, or a separate
|
||||
versioned checkpoint envelope. Debug capture can record the current hit and
|
||||
loaded artifact without requiring historical metadata in the cache file.
|
||||
|
||||
Comprehensive upstream execution identity is no longer required for this
|
||||
design. It does not require a later PromptKit release or a temporary conservative
|
||||
configuration fingerprint. Item 5 supersedes the earlier proposal to make an
|
||||
upstream semantic-equality feature part of the migration's checkpoint design.
|
||||
Retire checkpoint-specific identities, invocation-wide lookup barriers,
|
||||
stage/dependency fingerprints, manifests, and recovery bookkeeping. Do not
|
||||
preserve that machinery solely to cache cheap deterministic stages. The existing
|
||||
standalone canonical chunk-plan cache remains a separate concern.
|
||||
|
||||
### Remaining Implementation Work
|
||||
The old `--resume`/`--recompute-step` conventions are migration context, not a
|
||||
compatibility requirement. Select straightforward cache reuse/bypass/refresh
|
||||
controls and legacy-file handling during implementation planning; do not infer
|
||||
new flag spellings or a migration framework from this decision.
|
||||
|
||||
Define the concrete logical stage/prompt identifiers and consumed-input
|
||||
representation. Revise checkpoint lookup and storage scoping so that an
|
||||
invocation-wide pipeline/configuration digest cannot still prevent reuse of
|
||||
successful predecessors. Decide how existing checkpoint records transition to
|
||||
the simpler format and how incompatible payloads are reported.
|
||||
|
||||
**Follow-up:** Reconcile the adoption roadmap and record the changed reuse
|
||||
contract in an ADR. Update current-behavior documentation when implemented.
|
||||
Verify recovery after a downstream fix, reuse across prompt/settings changes,
|
||||
misses after consumed-data changes, explicit recomputation, and rejection of
|
||||
unusable or unaccepted stored artifacts. Continue with item 6.
|
||||
**Follow-up:** Incorporate this design and the upstream hash requirement into the
|
||||
master roadmap. Verify that a downstream failure leaves earlier accepted
|
||||
artifacts reusable, corrected answers are stored under the initial hash, changed
|
||||
messages miss, settings-only changes can hit, and a hit makes no model calls or
|
||||
new semantic-validation claims. Keep output/report contract decisions separate.
|
||||
|
||||
## 6. Inspection And Execution Bookkeeping
|
||||
|
||||
@@ -529,7 +564,7 @@ No application changes have been implemented.
|
||||
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
|
||||
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
|
||||
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
|
||||
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
|
||||
| Acceptance, rejection, fallback, accepted-artifact reuse, and pipeline outcome | Notarius. |
|
||||
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
|
||||
|
||||
### Native Inspection And Operator Presentation
|
||||
@@ -592,15 +627,21 @@ artifact, which validators assessed it, whether continuation or a safe fallback
|
||||
was used, and whether the artifact was generated now or reused. PromptKit cannot
|
||||
infer that meaning from provider execution alone.
|
||||
|
||||
Under item 5, a reused artifact retains its original execution and acceptance
|
||||
provenance. The current invocation records the reuse decision without counting
|
||||
the earlier execution's token usage as current work or claiming new validation.
|
||||
Under the revised item 5, cache files contain accepted artifacts without required
|
||||
historical execution or acceptance records. A current debug record can identify
|
||||
reuse and show the loaded artifact without counting earlier token usage as
|
||||
current work or claiming new validation.
|
||||
|
||||
Retain small, deliberate application projections for the
|
||||
The output/report contract is reopened in the CommandKit discussion; the earlier
|
||||
manifest and receipt proposals below are not requirements to retain metadata
|
||||
without a consumer need. They do not impose any additional cache records.
|
||||
|
||||
Previously proposed small application projections for the
|
||||
[published manifest](../integrations/json-output.md) and
|
||||
[run-result receipt](../integrations/run-result.md), with explicit compatibility
|
||||
decisions for changed fields. Durable contracts should not become raw upstream
|
||||
type dumps or require parallel runtime representations of every native result.
|
||||
decisions for changed fields, remain subject to that review. Durable contracts
|
||||
should not become raw upstream type dumps or require parallel runtime
|
||||
representations of every native result.
|
||||
|
||||
Keep application debug artifact placement, opt-in capture, retention, and
|
||||
disclosure handling. Native callbacks supply execution data; they do not own
|
||||
@@ -627,7 +668,7 @@ remaining upstream request should identify a concrete unmet requirement.
|
||||
significant boundary changes in successor ADRs. Update current-behavior
|
||||
documentation when implemented. Verify failure reporting without a result,
|
||||
continuation/validator accounting, unknown usage, actual-target provenance,
|
||||
checkpoint reuse accounting, and optional debug capture during implementation.
|
||||
cache reuse accounting, and optional debug capture during implementation.
|
||||
|
||||
## 7. Compare Audita's PromptKit Workflow
|
||||
|
||||
@@ -766,7 +807,7 @@ and [prompt guidance](../../../audita/docs/internal/prompts.md).
|
||||
individual correction within a proposal response. Notarius must deliberately
|
||||
choose its correction unit rather than assume per-entity branching.
|
||||
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
|
||||
reuse identity or persisted validation-provenance requirements.
|
||||
cache design; item 5 records that application decision independently.
|
||||
|
||||
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
|
||||
[observations](../../../audita/internal/framework/llm/observations.go),
|
||||
@@ -786,10 +827,10 @@ flow and the remaining repair-prompt, response-contract, and policy details;
|
||||
the comparison alone did not settle those details.
|
||||
|
||||
Item 4 now records agreement to use native admission with bounded application
|
||||
workers. Item 5 selects simple application-owned recovery matching and removes
|
||||
the upstream execution-fingerprint prerequisite. Item 6 adopts native execution
|
||||
observations and CommandKit presentation, retaining application attribution,
|
||||
domain outcomes, and durable provenance.
|
||||
workers. The revised item 5 selects an accepted-artifact cache and a focused
|
||||
upstream message-content hash instead of comprehensive execution fingerprints.
|
||||
Item 6 adopts native execution observations and CommandKit presentation;
|
||||
consumer output/report requirements are being revisited separately.
|
||||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||||
feature is needed merely to provide replaceable defaults or domain-directed
|
||||
continuation. Any remaining upstream request should identify the specific unmet
|
||||
|
||||
Reference in New Issue
Block a user