815 lines
49 KiB
Markdown
815 lines
49 KiB
Markdown
# PromptKit Architecture Discussion Record
|
||
|
||
## Purpose And Status
|
||
|
||
The decisions in items 1–6 have been incorporated into the
|
||
[PromptKit-native architecture migration roadmap](promptkit.md). That roadmap
|
||
is the implementation plan and owns current unresolved choices, sequencing,
|
||
acceptance criteria, and implementation status.
|
||
|
||
This document retains the discussion, rationale, and evidence, including item 7's
|
||
completed Audita comparison. Item-level questions and follow-ups below record
|
||
the discussion as it developed; they are not a second implementation plan.
|
||
|
||
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
|
||
capabilities as fully as practical. Application ownership of a policy does not
|
||
necessarily require application ownership of its implementation. Evaluate
|
||
remaining machinery against this goal, including opportunities for upstream
|
||
features where PromptKit does not yet meet an application need.
|
||
|
||
No application changes have been implemented through this discussion or the
|
||
roadmap rewrite. Significant changes to accepted architecture need successor ADRs
|
||
under the [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||
Current boundaries are described in [Architecture](../policy/architecture.md)
|
||
and [LLM Runtime Internals](../internal/llm.md).
|
||
|
||
## 1. LLM API And Diagnostic Boundary
|
||
|
||
**Discussion / decision:** Direction agreed: remove the Notarius PromptKit
|
||
adapter and compose the application directly from native PromptKit execution,
|
||
typed artifact operations, and pipeline orchestration. A small application
|
||
function connects generation to configured validation and acceptance. The design
|
||
below incorporates items 2–6 and the Audita review; exact signatures and migration
|
||
details remain open. No application changes have been implemented.
|
||
|
||
### Background And Intent
|
||
|
||
Notarius and related applications, including Audita, originally developed their
|
||
own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by
|
||
extracting their duplicated mechanics into a shared library. Notarius adopted
|
||
an early version and predates many of the library's modern capabilities.
|
||
|
||
The maintainer clarified that duplication of PromptKit capabilities, including
|
||
slight behavioral differences, does not reflect an intentional requirement for
|
||
an independent Notarius implementation. Existing wrappers, tests, and historical
|
||
decisions are migration evidence, not sufficient reasons to preserve that
|
||
duplication. Identify actual application needs before choosing what to retain.
|
||
|
||
Missing reusable capabilities may justify upstream feature requests. The
|
||
discussion is not constrained to what v0.23.2 can replace immediately; any
|
||
resulting change to the adoption target remains to be selected.
|
||
|
||
### Application Structure
|
||
|
||
| Part | Responsibility |
|
||
| --- | --- |
|
||
| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. |
|
||
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
|
||
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
|
||
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
|
||
| Application records and CLI | Store artifacts and checkpoints, attach pipeline meaning to execution facts, and present results through CommandKit. |
|
||
|
||
Engine construction becomes ordinary application wiring using item 2's embedded
|
||
defaults and operator sources. Profile selection is expressed through native
|
||
requests. The current `internal/framework/llm` package can shrink to engine
|
||
assembly and focused helpers; whether those warrant a separate package remains
|
||
an organizational choice, not a reason to retain an adapter.
|
||
|
||
Use native requests, results, exchanges, errors, and observers directly in
|
||
model-executing application code. Pure domain models, codecs, source-reference
|
||
resolution, and deterministic transformations need no PromptKit dependency.
|
||
Retain application interfaces for extracting artifacts, validating typed
|
||
candidates, encoding output, and loading checkpoints. Deterministic stages
|
||
continue to operate without conversation machinery.
|
||
|
||
### Connect Modules To Pipeline Assessment
|
||
|
||
The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
|
||
prepares inputs, invokes the private completion interface, interprets a response,
|
||
and returns a copied raw model candidate to the
|
||
[producer retry loop](../../internal/framework/pipeline/producer_attempts.go).
|
||
Replace that arrangement with direct collaboration between module interpretation,
|
||
pipeline assessment, and native execution:
|
||
|
||
1. The module prepares source material, references, and any candidate-handle
|
||
mappings once, and constructs the native initial request.
|
||
2. Application execution calls `Engine.Run`, capturing an exchange when
|
||
continuation is enabled.
|
||
3. The module interprets the structurally valid response into a typed candidate
|
||
or actionable domain findings.
|
||
4. The pipeline runs configured candidate validation and decides whether to
|
||
accept, request correction, or apply terminal policy.
|
||
5. If correction is needed, native `Coordinate` invokes the same interpretation
|
||
and assessment for revised responses. The module supplies the repair request;
|
||
PromptKit maintains history and enforces the turn budget.
|
||
|
||
A small shared application function can connect these responsibilities: the
|
||
module supplies native request construction and response interpretation, while
|
||
the pipeline supplies configured validation and terminal policy. It uses native
|
||
PromptKit types and returns a typed application outcome. It does not introduce
|
||
another completion API, conversation protocol, or retry engine.
|
||
|
||
Item 3 owns the detailed continuation and assessment rules, including final-turn
|
||
assessment, failure handling, complete replacement responses, and safe fallback.
|
||
For normalization, a revised reconciliation plan is applied to the original
|
||
inputs to produce a fresh candidate. LLM-backed validators can call the same
|
||
native engine and return application validation findings; they do not need the
|
||
producer's conversation machinery.
|
||
|
||
### Removal, Records, And Testing
|
||
|
||
Retire `StructuredLLMClient`, the private structured-completion request/response
|
||
family, adapter conversions, and opaque correction transport. Items 3, 4, and 6
|
||
specify removal of the outer LLM retry loop, scheduler, debug completion wrapper,
|
||
profile-recording plumbing, and duplicate execution accounting. Do not recreate
|
||
that wrapper stack around native calls.
|
||
|
||
Attach pipeline scope through native callbacks and observers. Use native
|
||
`ExecutionGroup` totals and small application projections for manifests and
|
||
optional debug artifacts, following item 6. Checkpoint matching and durable
|
||
acceptance provenance remain the application responsibilities selected in item 5.
|
||
|
||
Audita demonstrates shared engine assembly, native continuation, and CommandKit
|
||
integration. Its `RoleClient`, structured-output interface, and opaque exchange
|
||
wrapper are not part of this design; item 7 records the comparison's evidence
|
||
and limits.
|
||
|
||
Test domain interpretation and assessment directly. Exercise model workflows
|
||
with a real PromptKit engine and its injectable client or a fake HTTP provider.
|
||
Pipeline orchestration can still use fake stage implementations. Testing does
|
||
not require preserving a private completion interface.
|
||
|
||
### Remaining Implementation Work
|
||
|
||
- Select concrete module/pipeline function signatures that connect request
|
||
construction, interpretation, configured validation, and terminal disposition.
|
||
- Choose the location of engine assembly and focused application helpers without
|
||
introducing a replacement LLM service layer.
|
||
- Resolve item 2's deferred filesystem and schema choices and the remaining
|
||
migration details recorded under items 3–6, including durable compatibility.
|
||
- Evaluate any remaining upstream gap against an actual application requirement,
|
||
rather than preserving historical machinery by default.
|
||
|
||
**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6.
|
||
Record the significant boundary change in successor ADRs. Update architecture
|
||
and implementation documentation when the change lands, explicitly replacing
|
||
the current transport-neutral completion boundary with PromptKit as the execution
|
||
dependency of application workflows. Keep domain operations and application
|
||
outcomes as the meaningful Notarius-owned boundaries.
|
||
|
||
## 2. Prompt Assets And Schema Assembly
|
||
|
||
**Discussion / decision:** Direction agreed: use native PromptKit declarations
|
||
as the sole prompt assembly mechanism and remove the duplicate application
|
||
assembly plumbing. Resolution of the custom read-only filesystem and schemas
|
||
is explicitly deferred; the next topic may clarify those choices.
|
||
|
||
**Selected direction:** Keep the embedded content tree and load its prompt and
|
||
profile definitions as native PromptKit configuration in the application
|
||
collection. Use native filesystem contexts and imports to reference shared
|
||
content directly. Application composition assembles the embedded and operator
|
||
sources once; modules select prompt IDs, supply domain inputs, and interpret
|
||
typed results.
|
||
|
||
The spell extractor illustrates the current duplication: its prompt YAML
|
||
declares ordered messages and a schema, while a separate Go asset manifest
|
||
lists files, mounts shared fragments, registers filesystems, and hashes them.
|
||
The selected direction makes native declarations the sole prompt assembly source.
|
||
|
||
Candidates for removal:
|
||
|
||
- Prompt and fallback-profile registration plumbing in `AssetRegistry` and
|
||
individual modules; schema responsibilities remain under review.
|
||
- `ModulePromptFS`, `PromptAssetManifest`, shared-name lookup tables, synthetic
|
||
`sharedassets` mounts.
|
||
- Manual prompt-file enumeration for checkpoint fingerprints, retired under
|
||
item 5's simpler reuse policy. Native prompt hashes remain available for
|
||
inspection and diagnostics.
|
||
|
||
Keep prompt prose, ordered messages, cache controls, workload profiles, private
|
||
response schema documents, and domain interpretation. Durable artifact schemas
|
||
remain separate. Content ownership does not require runtime asset registration.
|
||
|
||
Native schema paths resolve against the configured schema root, separately from
|
||
message-file source contexts. PromptKit v0.23.2 loads schemas during preparation
|
||
and requires self-contained schema documents. A captured prompt hash alone does
|
||
not establish schema equality. Existing schema metadata in manifests and
|
||
checkpoint inputs must be deliberately replaced or retired; its presence does
|
||
not require preserving the complete wrapper system.
|
||
|
||
Deferred decisions:
|
||
|
||
- Whether the custom read-only filesystem can be removed after reviewing its
|
||
remaining responsibilities.
|
||
- Schema assembly, root/path selection, wrapper removal, and schema metadata.
|
||
Supplying the embedded filesystem as a native schema root remains a candidate,
|
||
not a selected design.
|
||
- Schema metadata for manifests and artifact compatibility. Item 5 settles that
|
||
checkpoint matching does not require exact schema identity.
|
||
|
||
Implementation questions:
|
||
|
||
- Can native sources represent the existing shared fragments and module assets?
|
||
- Which registry, remounting, and hashing helpers could disappear?
|
||
- What schema metadata remains useful for inspection and artifact loading under
|
||
item 5's simpler reuse policy?
|
||
- How do we preserve message ordering and cache controls while simplifying
|
||
assembly?
|
||
|
||
**Follow-up:** Continue with item 3, then revisit the deferred filesystem and
|
||
schema decisions with any new clarity. Confirm native source contexts for shared
|
||
fragments and operator replacements. Apply item 5's reuse policy and resolve
|
||
remaining metadata with item 6. Verify the resulting definitions through native
|
||
preparation and fake-provider execution; filesystem remounting tests can retire
|
||
with the removed implementation. The agreed direction has not yet been implemented.
|
||
|
||
## 3. Retry And Semantic Correction Ownership
|
||
|
||
**Discussion / decision:** Direction agreed: adopt Audita's native continuation
|
||
approach. PromptKit owns captured exchanges, accumulated conversation history,
|
||
continuation execution, and turn-budget enforcement. Notarius supplies semantic
|
||
assessment, corrective findings, and acceptance or fallback decisions. Use native
|
||
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
|
||
exchange wrapper are not part of the selected direction.
|
||
|
||
The flow and removal/adaptation direction below are agreed. Begin with one
|
||
conversation per producer operation and complete replacement model responses.
|
||
Retain application assessment, safe-fallback decisions, artifact validation,
|
||
and checkpoint eligibility. Detailed response contracts, budget migration, and
|
||
the open policy questions below remain to be resolved. No application changes
|
||
have been implemented.
|
||
|
||
### Agreed Flow
|
||
|
||
1. Prepare the source context, references, and any request-local candidate-handle
|
||
mapping once for the producer operation. Run its initial native request with
|
||
exchange capture when continuation is enabled. Capture is independent of
|
||
detailed debug recording.
|
||
2. Inspect native structural validation, decode the response, and perform the
|
||
module's deterministic interpretation and safety checks. Run the configured
|
||
validation chain on a materialized candidate. A module can identify actionable
|
||
defects before a candidate is ready for that chain.
|
||
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
|
||
and a configured `AdditionalTurns` budget. The callback selects a native
|
||
repair prompt and supplies actionable findings as variables. PromptKit retains
|
||
the original exchange and each subsequent repair turn.
|
||
4. Assess every delivered response, including the last permitted turn. Rebuild
|
||
the typed candidate from the fixed operation inputs and the revised response;
|
||
do not repeatedly apply revisions to an already modified artifact. A changed
|
||
candidate receives the complete configured validation chain again, including
|
||
validators that approved an earlier version.
|
||
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
|
||
whether to reject, fail, or evaluate an explicitly supported safe fallback.
|
||
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
|
||
Cancellation never turns an undelivered response into an accepted candidate.
|
||
|
||
Start with one conversation per producer operation: a chunk-plan generation,
|
||
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
|
||
with Notarius's whole-output validation boundary. Audita's per-proposal branches
|
||
are useful evidence, but per-entity repair would require its own combination and
|
||
validation design. It is not necessary to adopt native continuation.
|
||
|
||
Repair prompts should supply a new user task, with enduring principles in the
|
||
original system instructions. Avoid replaying the initial task or embedding
|
||
initial-response-only constraints in system messages. Begin with complete
|
||
replacement model responses; for normalization, the response can remain a
|
||
reconciliation plan that Notarius applies to construct the durable artifact.
|
||
A different repair schema or profile is supported but not yet selected. Keep
|
||
source context and candidate-handle meanings stable throughout the conversation.
|
||
Accumulated history increases context and token use across turns; the semantic
|
||
turn budget should reflect that tradeoff.
|
||
|
||
### Machinery To Remove Or Adapt
|
||
|
||
| Current machinery | Agreed direction |
|
||
| --- | --- |
|
||
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
|
||
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
|
||
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
|
||
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
|
||
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
|
||
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
|
||
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
|
||
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
|
||
|
||
Native transient retries, structural repair, and semantic additional turns remain
|
||
distinct controls, but Notarius need not implement all three loops. Existing
|
||
producer and validator `retries` settings cannot be mechanically renamed: their
|
||
current budgets cover different behavior. Non-LLM failure handling should be
|
||
reviewed separately rather than routed through conversation coordination.
|
||
|
||
### Validation And Safe Fallback Still Have Application Meaning
|
||
|
||
Keep source grounding, reference resolution, domain invariants, reconciliation
|
||
plan checks, typed application rules, and candidate isolation. In particular,
|
||
normalization must still reject unknown or overlapping candidate handles and
|
||
apply only safe groups. A final fallback can retain safe merges while leaving
|
||
unsafe groups separate; it must go through configured artifact validation and
|
||
terminal policy before it can advance. After native structural-repair exhaustion,
|
||
a supported deterministic fallback can be assessed directly instead of first
|
||
restarting the same model operation through a feedback-free module retry. Which
|
||
earlier safe candidate, if any, may survive a later failed turn remains an explicit
|
||
policy question; do not introduce implicit best-so-far acceptance.
|
||
|
||
Retain the distinction between a rejection and unavailable validation. Under the
|
||
current `warn_continue` policy, incomplete validation can advance a structurally
|
||
valid candidate but prevents checkpoint reuse/publication, including for derived
|
||
results. Continuation adoption does not itself change that policy, safe-fallback
|
||
eligibility, or durable provenance. Keep raw conversation state in process and
|
||
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
|
||
|
||
PromptKit should own validation and repair of the private model response.
|
||
Notarius's serialized schema validator currently receives the codec-produced
|
||
artifact and its artifact schema, which can differ from the model response.
|
||
Do not remove that boundary merely because both use JSON Schema. Audit actual
|
||
duplicate response checks individually; artifact validation and Go/domain
|
||
interpretation are not automatically redundant. Item 2's filesystem and schema
|
||
assembly decisions remain open.
|
||
|
||
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
|
||
[validation executor](../../internal/framework/pipeline/validation_executor.go),
|
||
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
|
||
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
|
||
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
|
||
and the native continuation evidence in item 7.
|
||
|
||
### Open Implementation And Policy Questions
|
||
|
||
- Exact repair prompt definitions, response schemas, and profile selection within
|
||
the agreed complete-replacement approach.
|
||
- Migration of producer and validator `retries` settings to native controls,
|
||
including configuration names, defaults, and compatibility.
|
||
- Detailed terminal/fallback handling, including whether an earlier safe
|
||
candidate may survive a later failed turn.
|
||
- Failure handling for non-LLM operations and any justified recovery beyond
|
||
native execution budgets.
|
||
- Item 2's deferred filesystem and schema assembly choices, item 5's remaining
|
||
checkpoint implementation details, and item 6's durable bookkeeping design.
|
||
|
||
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
|
||
roadmap. Record the changed architecture in a successor to
|
||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||
Migration checks should cover accumulated history, fixed handle mappings,
|
||
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
|
||
validator failure, cancellation, and checkpoint eligibility. Replace tests of
|
||
the retired message-reconstruction protocol rather than preserving its mechanics.
|
||
|
||
## 4. Global Scheduling And Admission
|
||
|
||
**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and
|
||
retire the separate cross-backend ceiling. Rely on PromptKit's native backend
|
||
admission and capacity limits while keeping bounded Notarius pipeline workers.
|
||
This deliberately changes the existing global-limit contract; reproducing that
|
||
ceiling in PromptKit is not a migration prerequisite. No application changes
|
||
have been implemented.
|
||
|
||
### Agreed Ownership
|
||
|
||
| Concern | Owner |
|
||
| --- | --- |
|
||
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
|
||
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
|
||
| Application disposition after admission or execution failure | Notarius application policy |
|
||
|
||
For limited backends, PromptKit v0.23.2 bounds active model attempts by
|
||
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||
An operation holds admission
|
||
across preparation, retries, and structural repair, while active permits cover
|
||
individual model attempts and are released during retry waits. A completed
|
||
continuation turn releases admission before application assessment, so its
|
||
callback can invoke an LLM validator using the same backend.
|
||
|
||
The current Notarius wrapper holds a global permit around the complete
|
||
structured call, including backend waiting and repairs. Removing it eliminates
|
||
that additional queue and avoids making independent backends share those
|
||
permits. Independent native backend limits may allow aggregate concurrency above
|
||
the former global ceiling; that is an accepted behavior change. Pipeline worker
|
||
bounds remain application work controls, not a replacement global LLM ceiling.
|
||
|
||
### Removal And Configuration Consequences
|
||
|
||
- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom
|
||
queue/cancellation/permit bookkeeping.
|
||
- Retire `concurrency.total_llm`. Give pipeline worker settings an independent
|
||
default and remove the extraction-worker maximum derived from `total_llm`.
|
||
- Use one shared execution engine for producers, validators, and repair turns.
|
||
Native capacity belongs to an engine and selected backend ID; separate engine
|
||
instances have independent pools.
|
||
- Configure native limits at the intended backend capacity boundary. Profiles
|
||
selecting the same backend ID share its pool; separate IDs pointing to the
|
||
same endpoint do not automatically share capacity. Endpoint-only profiles and
|
||
backends without a positive limit remain unrestricted by backend admission.
|
||
- Handle native `ErrCapacityExceeded` as an admission failure through application
|
||
policy, consistently with item 3's removal of blanket execution retries.
|
||
- Retire scheduler implementation tests with the removed code. Retain focused
|
||
integration checks for shared native admission across roles, independent
|
||
backends, bounded pipeline work, queue exhaustion, and cancellation.
|
||
|
||
### Remaining Details
|
||
|
||
Select independent worker defaults, native backend/queue defaults where the
|
||
application supplies them, and the configuration migration treatment of the
|
||
retired field. Specify how admission failures are reported and handled for each
|
||
application role. These details do not reopen the selected ownership boundary.
|
||
|
||
A future demonstrated need for capacity shared across several backend IDs could
|
||
justify an upstream PromptKit feature request. No such feature is required for
|
||
this agreed design, and it does not justify retaining the Notarius scheduler.
|
||
|
||
Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
|
||
[scheduler](../../internal/framework/llm/scheduler.go),
|
||
[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and
|
||
[native capacity behavior](../../../promptkit/docs/internal/capacity.md).
|
||
|
||
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
|
||
and record the changed ownership in an ADR. Update architecture, configuration,
|
||
operations, and implementation documentation when the migration lands. Continue
|
||
with item 5's execution identity and checkpoint reuse discussion.
|
||
|
||
## 5. Execution Identity And Checkpoint Reuse
|
||
|
||
**Discussion / decision:** Direction agreed: resuming preserves successful stage
|
||
results for the same logical work and data; explicit recomputation applies
|
||
changed execution behavior. Use a simple Notarius-owned matching policy instead
|
||
of comprehensive execution fingerprints. No application changes have been
|
||
implemented.
|
||
|
||
### Purpose And Matching Rule
|
||
|
||
Checkpoints preserve a successful producer's result when a dependent stage fails.
|
||
After fixing the failed stage, an operator should be able to retry it without
|
||
paying to rerun the successful producer. Matching every configuration parameter
|
||
or implementation revision would undermine that recovery purpose.
|
||
|
||
The agreed conceptual key is:
|
||
|
||
```text
|
||
logical stage identity + input-data identity + prompt ID, where applicable
|
||
```
|
||
|
||
Scope the checkpoint to the relevant step/lane/stage and artifact kind so that
|
||
shared prompt IDs do not mix unrelated results. These are logical identifiers,
|
||
not hashes of implementations or complete configuration. Deterministic stages
|
||
need no synthetic prompt identity.
|
||
|
||
Input data includes the source or chunk being processed, reference material,
|
||
and upstream artifacts actually consumed. Matching only the original source
|
||
would miss changes to generated dependencies. Content digests can compactly
|
||
identify these inputs and check stored payload integrity without requiring an
|
||
exact fingerprint of the execution environment.
|
||
|
||
| Change | Reuse behavior |
|
||
| --- | --- |
|
||
| Fix the implementation of a later failed stage | Preserve earlier successful stages with matching work and inputs. |
|
||
| Edit prompt text while retaining its ID | Reuse; explicit recomputation applies the edit. |
|
||
| Change model, generation settings, or implementation/validation policy | Reuse otherwise compatible completed work; explicit recomputation applies the change. |
|
||
| Change the prompt ID or data consumed by the stage | No matching checkpoint under this rule. |
|
||
| Explicitly request recomputation | Recompute the selected work and its dependents. |
|
||
|
||
The existing controls are `--resume` and `--recompute-step`, documented in the
|
||
[CLI reference](../cli.md#run) and
|
||
[selective recompute workflow](../operations.md#selective-recompute). Their
|
||
current strict compatibility checks still need migration to this agreed policy.
|
||
The accepted tradeoff is that a checkpoint may preserve a successful result
|
||
produced by code, prompts, or settings that have since improved.
|
||
|
||
### Matching, Loading, And Provenance
|
||
|
||
A matching key selects a checkpoint; it does not make an unreadable or
|
||
incompatible artifact usable. Retain completion/acceptance status, integrity
|
||
checks, and the ability to decode and consume the stored artifact. Exact schema
|
||
hash equality is not required when the artifact remains compatible.
|
||
|
||
Only accepted, completely validated results are reusable. Rejected, structurally
|
||
invalid, or validation-incomplete results and their ineligible derivatives
|
||
remain outside reusable state. Acceptance on resume refers to the earlier
|
||
execution: do not automatically rerun semantic validators merely because their
|
||
configuration changed, or claim the artifact passed the new validation policy.
|
||
Preserve original provenance; explicit recomputation applies changed behavior.
|
||
|
||
Useful execution metadata can remain available for explanation without becoming
|
||
a matching requirement. Neither complete conversation history nor native
|
||
prepared-execution handles are needed to locate and reuse a completed artifact.
|
||
The separate source-addressed canonical chunk-plan policy remains distinct.
|
||
|
||
### Simplification And Ownership
|
||
|
||
- Remove the broad PromptKit configuration fingerprint from checkpoint matching,
|
||
including effective-configuration parsing and whole-catalog enumeration.
|
||
- Remove library/catalog versions, model settings, prompt-content hashes, and
|
||
component-policy fingerprints as automatic invalidators under this policy.
|
||
- Keep straightforward input/dependency identity, successful-stage records,
|
||
artifact loading checks, storage, and explicit recomputation in Notarius.
|
||
- Keep PromptKit hashes available for native inspection and diagnostics without
|
||
making them checkpoint prerequisites.
|
||
|
||
Comprehensive upstream execution identity is no longer required for this
|
||
design. It does not require a later PromptKit release or a temporary conservative
|
||
configuration fingerprint. Item 5 supersedes the earlier proposal to make an
|
||
upstream semantic-equality feature part of the migration's checkpoint design.
|
||
|
||
### Remaining Implementation Work
|
||
|
||
Define the concrete logical stage/prompt identifiers and consumed-input
|
||
representation. Revise checkpoint lookup and storage scoping so that an
|
||
invocation-wide pipeline/configuration digest cannot still prevent reuse of
|
||
successful predecessors. Decide how existing checkpoint records transition to
|
||
the simpler format and how incompatible payloads are reported.
|
||
|
||
**Follow-up:** Reconcile the adoption roadmap and record the changed reuse
|
||
contract in an ADR. Update current-behavior documentation when implemented.
|
||
Verify recovery after a downstream fix, reuse across prompt/settings changes,
|
||
misses after consumed-data changes, explicit recomputation, and rejection of
|
||
unusable or unaccepted stored artifacts. Continue with item 6.
|
||
|
||
## 6. Inspection And Execution Bookkeeping
|
||
|
||
**Discussion / decision:** Direction agreed: use native PromptKit inspection
|
||
and execution facts, with CommandKit owning standardized operator presentation.
|
||
Remove duplicate LLM reporting and accounting machinery. Notarius retains
|
||
application context, domain decisions, durable artifacts, and their provenance.
|
||
No application changes have been implemented.
|
||
|
||
### Ownership
|
||
|
||
| Responsibility | Owner |
|
||
| --- | --- |
|
||
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
|
||
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
|
||
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
|
||
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
|
||
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
|
||
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
|
||
|
||
### Native Inspection And Operator Presentation
|
||
|
||
Remove the profile-inspector wrapper, copied effective-model-parameter types,
|
||
and local reconstruction of upstream configuration. Use native inspection for
|
||
configured resources and native prepared/result data for the target actually
|
||
executed, including request overrides. Profile inspection can describe a partial
|
||
configured profile; it is not evidence of the completed execution target.
|
||
|
||
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
|
||
preview, and opt-in smoke commands. Notarius supplies configuration/engine
|
||
loaders and application request builders. Avoid building temporary custom
|
||
commands or report machinery that the planned CommandKit adoption would replace.
|
||
|
||
Retain application pipeline preflight: resolve modules, artifact compatibility,
|
||
references, and known prompt/profile selections. Use native definition checks
|
||
without simulating stage execution or requiring provider credentials. Concrete
|
||
request preparation remains the place to check the completed target, inputs,
|
||
schema, and credential requirements.
|
||
|
||
CommandKit supplies formatting, visibility, progress presentation, and delivery
|
||
error handling. Notarius supplies report content, domain outcomes, and exit-status
|
||
mapping. Preserve deliberate machine-output contracts rather than treating every
|
||
receipt as a generic formatted report.
|
||
|
||
### Native Execution Facts
|
||
|
||
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
|
||
for provenance or optional debug capture. Remove unconditional
|
||
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
|
||
- Use native observations for terminal execution facts, including failures that
|
||
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
|
||
preparation or admission failure; it cannot serve as a completion record.
|
||
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
|
||
Remove the private debug completion wrapper and duplicate synthetic call-ID
|
||
machinery where native identity and application attribution suffice.
|
||
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
|
||
cloning, and independent attempt/usage accumulators. Use native types internally
|
||
and project selected fields only at an application reporting or durable boundary.
|
||
- Replace the adapter's `LLMProfileRecorder` and provider-forwarding plumbing with
|
||
the application projection needed for actual-run provenance. Do not substitute
|
||
a catalog inventory for the profiles and targets actually used.
|
||
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
|
||
initial run, continuation runs, and nested validator runs as applicable.
|
||
Preserve terminal-coverage and token-usage completeness; unknown usage is not
|
||
zero. Do not count both attempt observations and terminal aggregates, or add
|
||
overlapping groups as though they were disjoint work.
|
||
|
||
Application execution code should attach scope and observers, run and assess the
|
||
native result, and record its disposition and execution references. This does
|
||
not require a replacement general-purpose LLM interface. A successful native
|
||
execution can still produce a domain-rejected candidate; provider success and
|
||
application acceptance remain separate facts.
|
||
|
||
### Application Records And Durable Boundaries
|
||
|
||
Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an
|
||
artifact, which validators assessed it, whether continuation or a safe fallback
|
||
was used, and whether the artifact was generated now or reused. PromptKit cannot
|
||
infer that meaning from provider execution alone.
|
||
|
||
Under item 5, a reused artifact retains its original execution and acceptance
|
||
provenance. The current invocation records the reuse decision without counting
|
||
the earlier execution's token usage as current work or claiming new validation.
|
||
|
||
Retain small, deliberate application projections for the
|
||
[published manifest](../integrations/json-output.md) and
|
||
[run-result receipt](../integrations/run-result.md), with explicit compatibility
|
||
decisions for changed fields. Durable contracts should not become raw upstream
|
||
type dumps or require parallel runtime representations of every native result.
|
||
|
||
Keep application debug artifact placement, opt-in capture, retention, and
|
||
disclosure handling. Native callbacks supply execution data; they do not own
|
||
Notarius's storage policy. Detailed prepared messages belong in the selected
|
||
debug capture path, while ordinary provenance can retain only the needed fields.
|
||
|
||
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
|
||
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
|
||
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
|
||
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
|
||
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
|
||
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
|
||
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
|
||
|
||
### Remaining Implementation Work
|
||
|
||
Select the minimal application scope and durable fields, observer/group
|
||
enrollment, and compatibility treatment for existing manifests, receipts, and
|
||
debug artifacts. Coordinate command migration with the roadmap's CommandKit
|
||
follow-up. The core ownership change needs no new PromptKit capability; any
|
||
remaining upstream request should identify a concrete unmet requirement.
|
||
|
||
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
|
||
significant boundary changes in successor ADRs. Update current-behavior
|
||
documentation when implemented. Verify failure reporting without a result,
|
||
continuation/validator accounting, unknown usage, actual-target provenance,
|
||
checkpoint reuse accounting, and optional debug capture during implementation.
|
||
|
||
## 7. Compare Audita's PromptKit Workflow
|
||
|
||
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
|
||
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
|
||
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
|
||
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
|
||
not new Notarius contracts. The source links below assume the sibling checkout.
|
||
|
||
### Embedded Defaults And Operator Replacement
|
||
|
||
- One captured engine is assembled from PromptKit's default catalog, the
|
||
embedded Audita underlay, and a higher-priority operator underlay.
|
||
`FilesystemInputs` supplies eleven native prompt definitions in
|
||
`collections.audita`: four proposal prompts, four repair prompts, and three
|
||
validator prompts.
|
||
- Operator sources are optional `prompts.yml` and recursively selected YAML
|
||
under `prompts/`, anchored to the application main source's recorded context.
|
||
Both locations feed one operator underlay and are peers; conflicting peer
|
||
definitions fail rather than receiving filename-based precedence.
|
||
- Replacement is by resource identity, not filename. For example, a complete
|
||
prompt record with local ID `modules.grammar.repair` under
|
||
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
|
||
other embedded definitions remain available. Native same-ID replacement is
|
||
whole-record replacement, not recursive merging of individual prompt fields.
|
||
- Native imports and message-file context are retained. Tests show that an
|
||
existing engine retains captured message content after a file edit and a new
|
||
engine sees the edit. They also verify inherited message-file resolution,
|
||
main-relative discovery independent of the working directory, and peer
|
||
conflict rejection.
|
||
- A diagnostic embedded-only construction can classify a loading error; it
|
||
does not silently fall back to defaults after an operator-source failure.
|
||
|
||
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
|
||
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
|
||
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
|
||
|
||
### Prompt Content And Schema Ownership Are Separate
|
||
|
||
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
|
||
small prompt registry records IDs and embedded paths; it does not reconstruct
|
||
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
|
||
supplied as template variables. Processing supplies the application-owned values
|
||
for those shared names, but an operator replacement decides whether its template
|
||
references them at all. They are not forcibly appended to replacement messages.
|
||
|
||
The processing request independently supplies a complete native output contract:
|
||
JSON Schema validation, the schema path selected for that operation, and its
|
||
structural-repair budget. The three schemas describe a correction set, a repair
|
||
decision, and validator decisions. PromptKit receives their canonical embedded
|
||
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
|
||
change the processing code's expected response shape. A test replaces a prompt
|
||
with a text/no-validation declaration and still exercises the processing
|
||
request's JSON-schema repair contract.
|
||
|
||
Audita still keeps a response-schema key/metadata/hash registry and application
|
||
decoding checks. Its use of native loading does not prove that every schema
|
||
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
|
||
a concrete option: module-owned response requirements expressed directly through
|
||
native per-request contracts. Standalone native preview/preparation can use the
|
||
prompt declaration's contract; processing supplies its own override.
|
||
|
||
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
|
||
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
|
||
`prepareCall` and `OutputContract` in the
|
||
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
|
||
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
|
||
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
|
||
|
||
### Native Continuation With Application Assessment
|
||
|
||
The implemented flow is:
|
||
|
||
1. Generate a section's proposal set, capturing a native immutable exchange when
|
||
correction repair is enabled. This retains the actual proposal messages and
|
||
response; it does not require debug recording.
|
||
2. Apply application validation. An actionable rejected proposal starts its own
|
||
branch from the original proposal exchange. Unavailable-only findings and
|
||
no-effect corrections do not start repair. Existing approvals remain separate.
|
||
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
|
||
Each follow-up selects a distinct repair prompt, repair profile, and repair
|
||
response schema. The request carries the original proposal reference, current
|
||
correction, ordered findings, and available before/after text previews.
|
||
4. PromptKit assembles history and progresses the conversation. A second turn
|
||
retains the first repair instruction and response. A sibling branch starts
|
||
from the original exchange and does not inherit another branch's repairs.
|
||
Audita does not rerender the original proposal or manually replay messages.
|
||
5. Audita decodes each delivered decision. Withdrawal ends the branch without
|
||
applying that correction. A revision undergoes the full application validation
|
||
chain again, including validators that approved its earlier version. Approval
|
||
stops the branch; actionable rejection can request another turn.
|
||
6. The final permitted response is assessed before exhaustion is determined.
|
||
Requesting another turn at that boundary does not allocate another logical
|
||
call. Recoverable provider/invalid-output failures finish the branch as
|
||
unavailable; fatal errors or parent cancellation interrupt processing.
|
||
Canceled, undelivered responses are diagnostic evidence only, never approvals.
|
||
|
||
Transport retries and structural repair operate inside each PromptKit execution;
|
||
they are separate from the semantic branch's additional-turn budget. Tests show
|
||
one semantic repair turn containing multiple provider attempts for structural
|
||
repair. The application retains acceptance, withdrawal, application ordering,
|
||
and reporting; PromptKit retains conversation and execution mechanics.
|
||
|
||
Audita's prompt roles support this model deliberately: system instructions carry
|
||
durable principles, the initial user message carries the initial module task,
|
||
and the repair prompt supplies a new user assignment. Adopting continuation in
|
||
Notarius would require reviewing the complete conversation and response-format
|
||
transitions, rather than mechanically appending to prompts written for fresh
|
||
replacement requests.
|
||
|
||
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
|
||
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
|
||
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
|
||
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
|
||
and [prompt guidance](../../../audita/docs/internal/prompts.md).
|
||
|
||
### Other Boundaries And Limits Of The Comparison
|
||
|
||
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
|
||
wrapper, and application diagnostic types. Adopt the demonstrated native
|
||
mechanisms without assuming Notarius should copy those wrappers; item 1's
|
||
adapter-removal direction remains in force.
|
||
- Proposal, validator, and repair roles share one engine and its per-backend
|
||
admission. The runner separately bounds section work. This demonstrates useful
|
||
separation of application work scheduling and provider capacity. Item 4 now
|
||
selects that ownership and retires Notarius's independent cross-backend ceiling.
|
||
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
|
||
feed a native `ExecutionGroup`; application records add logical-call and
|
||
stage/section/proposal attribution. Nested validator executions enroll
|
||
explicitly. Counts preserve terminal coverage and token-usage completeness.
|
||
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
|
||
commands using Audita's engine construction, including request preparation,
|
||
validation, and smoke diagnostics.
|
||
- Audita edits independently assessable transcript corrections; Notarius accepts
|
||
typed stage artifacts and has cross-stage dependencies. Audita branches at an
|
||
individual correction within a proposal response. Notarius must deliberately
|
||
choose its correction unit rather than assume per-entity branching.
|
||
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
|
||
reuse identity or persisted validation-provenance requirements.
|
||
|
||
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
|
||
[observations](../../../audita/internal/framework/llm/observations.go),
|
||
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
|
||
[component scope](../../../audita/docs/internal/overview.md).
|
||
|
||
### Implications For The Remaining Discussion
|
||
|
||
Item 2's native underlay direction has a working application precedent. Audita
|
||
also offers evidence for ordinary embedded filesystems and per-request schema
|
||
selection, while our filesystem and schema decisions remain open.
|
||
|
||
The subsequent item 3 discussion selects native continuation and accumulated
|
||
conversation history. Keeping domain validation local does not require keeping
|
||
conversation assembly or the turn loop local. Item 3 owns the agreed Notarius
|
||
flow and the remaining repair-prompt, response-contract, and policy details;
|
||
the comparison alone did not settle those details.
|
||
|
||
Item 4 now records agreement to use native admission with bounded application
|
||
workers. Item 5 selects simple application-owned recovery matching and removes
|
||
the upstream execution-fingerprint prerequisite. Item 6 adopts native execution
|
||
observations and CommandKit presentation, retaining application attribution,
|
||
domain outcomes, and durable provenance.
|
||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||
feature is needed merely to provide replaceable defaults or domain-directed
|
||
continuation. Any remaining upstream request should identify the specific unmet
|
||
requirement after reconsidering Notarius's historical behavior.
|
||
|
||
### Verification
|
||
|
||
Read the production composition, adapter, coordinator, domain repair callbacks,
|
||
prompt/schema definitions, and focused behavior tests. The following checks
|
||
passed from the Audita checkout:
|
||
|
||
```sh
|
||
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
|
||
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
|
||
```
|
||
|
||
The subprocess checks exercise assembled configuration, real native continuation
|
||
messages, operator replacement, final-turn assessment, sibling isolation,
|
||
repair-disabled behavior, and cancellation using a local fake provider. These
|
||
checks establish implementation behavior, not live-model quality. No live model
|
||
calls or application source changes were made. Broader migration decisions and
|
||
changes to the adoption roadmap remain for discussion.
|