Files
notarius/docs/roadmap/promptkit-architecture-discussion.md

51 KiB
Raw Blame History

PromptKit Architecture Discussion Record

Purpose And Status

The decisions in items 1–6, the subsequent CommandKit discussion, and the simplified cache/output designs are incorporated into the master native migration roadmap. It owns the selected target, unresolved choices, sequencing, acceptance criteria, and implementation status.

The CommandKit discussion record retains the related CLI, preflight, output, debug, and migration rationale. The master roadmap replaces earlier checkpoint matching/provenance proposals with the accepted-artifact cache and retires mandatory manifests, extensive receipts, and durable execution ledgers. The upstream message-content hash remains pending; incorporation does not claim it is already available or implemented.

This document retains the discussion, rationale, and evidence, including item 7's completed Audita comparison. Item-level questions and follow-ups below record the discussion as it developed; they are not a second implementation plan.

The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not necessarily require application ownership of its implementation. Evaluate remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need.

No application changes have been implemented through this discussion or the roadmap rewrite. Significant changes to accepted architecture need successor ADRs under the documentation policy. Current boundaries are described in Architecture and LLM Runtime Internals.

1. LLM API And Diagnostic Boundary

Discussion / decision: Direction agreed: remove the Notarius PromptKit adapter and compose the application directly from native PromptKit execution, typed artifact operations, and pipeline orchestration. A small application function connects generation to configured validation and acceptance. The design below incorporates items 2–6 and the Audita review; exact signatures and migration details remain open. No application changes have been implemented.

Background And Intent

Notarius and related applications, including Audita, originally developed their own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by extracting their duplicated mechanics into a shared library. Notarius adopted an early version and predates many of the library's modern capabilities.

The maintainer clarified that duplication of PromptKit capabilities, including slight behavioral differences, does not reflect an intentional requirement for an independent Notarius implementation. Existing wrappers, tests, and historical decisions are migration evidence, not sufficient reasons to preserve that duplication. Identify actual application needs before choosing what to retain.

Missing reusable capabilities may justify upstream feature requests. The discussion is not constrained to what v0.23.2 can replace immediately; any resulting change to the adoption target remains to be selected.

Application Structure

Part Responsibility
Application composition Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs.
Artifact-family modules Supply prompt inputs and response requirements, interpret responses, and apply domain rules.
Pipeline execution Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers.
PromptKit Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts.
Application records and CLI Store artifacts and accepted-artifact cache entries, attach pipeline meaning to execution facts, and present results through CommandKit.

Engine construction becomes ordinary application wiring using item 2's embedded defaults and operator sources. Profile selection is expressed through native requests. U3 now places shared loaders and engine construction in internal/cli and the producer helper beside pipeline assessment. Retire the emptied internal/framework/llm package rather than retaining a constructor-only package. The master roadmap owns the resolved design; implementation is pending.

Use native requests, results, exchanges, errors, and observers directly in model-executing application code. Pure domain models, codecs, source-reference resolution, and deterministic transformations need no PromptKit dependency. Retain application interfaces for extracting artifacts, validating typed candidates, encoding output, and loading accepted cached artifacts. Deterministic stages continue to operate without conversation machinery.

Connect Modules To Pipeline Assessment

The current spell extractor prepares inputs, invokes the private completion interface, interprets a response, and returns a copied raw model candidate to the producer retry loop. Replace that arrangement with direct collaboration between module interpretation, pipeline assessment, and native execution:

  1. The module prepares source material, references, and any candidate-handle mappings once, and constructs the native initial request.
  2. Application execution calls Engine.Run, capturing an exchange when continuation is enabled.
  3. The module interprets the structurally valid response into a typed candidate or actionable domain findings.
  4. The pipeline runs configured candidate validation and decides whether to accept, request correction, or apply terminal policy.
  5. If correction is needed, native Coordinate invokes the same interpretation and assessment for revised responses. The module supplies the repair request; PromptKit maintains history and enforces the turn budget.

The resolved U3 design uses a small shared helper in internal/framework/pipeline. Modules supply native initial/correction requests and response interpretation; the pipeline supplies an operation-scoped assessment callback and relevant cache/debug collaborators. Local variables or closures hold operation state. Use the same assessor for initial and revised candidates without an additional final pass. Native preparation supports cache lookup before execution; hits load the accepted artifact directly. The helper returns a typed application outcome and saves only final accepted artifacts under the initial hash.

Keep deterministic stages on their ordinary typed production/validation path. LLM validators call the shared engine directly and return findings without an automatic producer correction loop. No replacement completion API, execution-plan language, or copied conversation model is selected.

Item 3 owns the detailed continuation and assessment rules, including final-turn assessment, failure handling, complete replacement responses, and safe fallback. For normalization, a revised reconciliation plan is applied to the original inputs to produce a fresh candidate. LLM-backed validators can call the same native engine and return application validation findings; they do not need the producer's conversation machinery.

Removal, Records, And Testing

Retire StructuredLLMClient, the private structured-completion request/response family, adapter conversions, and opaque correction transport. Items 3, 4, and 6 specify removal of the outer LLM retry loop, scheduler, debug completion wrapper, profile-recording plumbing, and duplicate execution accounting. Do not recreate that wrapper stack around native calls.

Attach pipeline scope through native callbacks and observers. Use native ExecutionGroup totals for operational diagnostics and optional debug artifacts, following item 6. The small completion report follows the CommandKit output contract. Item 5 now selects a simple accepted-artifact cache, without required durable acceptance or execution-provenance records.

Audita demonstrates shared engine assembly, native continuation, and CommandKit integration. Its RoleClient, structured-output interface, and opaque exchange wrapper are not part of this design; item 7 records the comparison's evidence and limits.

Test domain interpretation and assessment directly. Exercise model workflows with a real PromptKit engine and its injectable client or a fake HTTP provider. Pipeline orchestration can still use fake stage implementations. Testing does not require preserving a private completion interface.

Remaining Implementation Work

  • Implement the resolved U3 ownership and placement; refine function signatures and argument grouping against representative extraction and normalization. These implementation details do not reopen the selected boundaries.
  • Implement item 2's resolved asset/schema design and settle the remaining migration details recorded under items 3–6, including durable compatibility.
  • Evaluate any remaining upstream gap against an actual application requirement, rather than preserving historical machinery by default.

Follow-up: Reconcile the adoption roadmap with this design and items 2–6. Record the significant boundary change in successor ADRs. Update architecture and implementation documentation when the change lands, explicitly replacing the current transport-neutral completion boundary with PromptKit as the execution dependency of application workflows. Keep domain operations and application outcomes as the meaningful Notarius-owned boundaries.

2. Prompt Assets And Schema Assembly

Discussion / decision: Agreed and incorporated as resolved U2 in the master roadmap. The filesystem and schema questions initially deferred here are now settled. Implementation is pending; the master roadmap owns the detailed design and checks.

Use ordinary embedded assets, native prompt declarations, and native schema loading. Keep the content tree and module ownership of prompt meaning and domain interpretation. Shared files use real native filesystem paths. Remove duplicate asset registration, per-module manifests, synthetic mounts, flattened schema namespaces, and the custom read-only filesystem that served that assembly.

Prompt definitions select output contracts by default. Initial and correction prompts can share schema files. Use native per-request replacement only for a demonstrated runtime-dependent contract, without an application wrapper or schema registry. Audita demonstrates ordinary embedded schema loading and per-call contracts; its per-call selection is not required for every Notarius operation.

The default schema source is the embedded asset tree. Operator-selected external schema directories use native whole-root replacement, not an application overlay. Prompt underlays do not implicitly overlay schema files. PromptKit owns private response-schema loading, structural checks, and repair; Notarius keeps typed interpretation, domain validation, and distinct durable artifact schemas/codecs.

Retire response-schema hashes and copied metadata used only by the adapter, checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic snapshots. Neither asset nor schema metadata adds another cache fingerprint.

The spell extractor illustrated the duplication: prompt YAML declared ordered messages and a schema while a Go manifest separately listed, mounted, registered, and hashed assets. Native declarations now supply the selected assembly model.

Follow-up: Implement the master roadmap's Phase 2 and retire tests of removed mounting/registry machinery. Verify shared paths, operator replacements, native schema sources, and any necessary complete request-contract replacement.

3. Retry And Semantic Correction Ownership

Discussion / decision: Direction agreed: adopt Audita's native continuation approach. PromptKit owns captured exchanges, accumulated conversation history, continuation execution, and turn-budget enforcement. Notarius supplies semantic assessment, corrective findings, and acceptance or fallback decisions. Use native PromptKit types directly, consistent with item 1; Audita's adapter and opaque exchange wrapper are not part of the selected direction.

The flow and removal/adaptation direction below are agreed. Begin with one conversation per producer operation and complete replacement model responses. Retain application assessment, safe-fallback decisions, artifact validation, and cache eligibility. Detailed response contracts, budget migration, and the open policy questions below remain to be resolved. No application changes have been implemented.

Agreed Flow

For cacheable operations, item 5's initial preparation and lookup precede this flow. The steps below describe execution on a miss.

  1. Prepare the source context, references, and any request-local candidate-handle mapping once for the producer operation. Run its initial native request with exchange capture when continuation is enabled. Capture is independent of detailed debug recording.
  2. Handle the native call's terminal success or failure, decode a usable response, and perform the module's deterministic interpretation and safety checks. Run the configured validation chain on a materialized candidate. A module can identify actionable defects before a candidate is ready for that chain.
  3. If correction is needed, call Engine.Coordinate with the initial exchange and a configured AdditionalTurns budget. The callback selects a native repair prompt and supplies actionable findings as variables. PromptKit retains the original exchange and each subsequent repair turn.
  4. Assess every delivered response, including the last permitted turn. Rebuild the typed candidate from the fixed operation inputs and the revised response; do not repeatedly apply revisions to an already modified artifact. A changed candidate receives the complete configured validation chain again, including validators that approved an earlier version.
  5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides whether to reject, fail, or evaluate an explicitly supported safe fallback. Native stopped and exhausted outcomes describe coordination, not acceptance. Cancellation never turns an undelivered response into an accepted candidate.

Start with one conversation per producer operation: a chunk-plan generation, one lane's extraction of one chunk, or one lane's normalization pass. This aligns with Notarius's whole-output validation boundary. Audita's per-proposal branches are useful evidence, but per-entity repair would require its own combination and validation design. It is not necessary to adopt native continuation.

Repair prompts should supply a new user task, with enduring principles in the original system instructions. Avoid replaying the initial task or embedding initial-response-only constraints in system messages. Begin with complete replacement model responses; for normalization, the response can remain a reconciliation plan that Notarius applies to construct the durable artifact. A different repair schema or profile is supported but not yet selected. Keep source context and candidate-handle meanings stable throughout the conversation. Accumulated history increases context and token use across turns; the semantic turn budget should reflect that tradeoff.

Machinery To Remove Or Adapt

Current machinery Agreed direction
CorrectionProtocolSingleResponseV1, raw-response ModelCandidate, and SemanticCorrection Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native RunResult / Exchange for continuation state.
Adapter code that rebuilds the original request and appends an assistant/user pair Remove with the adapter. Native continuation assembles history; native repair definitions render findings.
runProducerAttempts combining operational, structural, module-requested, and validator-semantic retries Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine.
Outer LLM retries after transport or structural repair exhaustion Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose.
executeValidationChain's repeated LLM validator invocations Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate.
NormalizeRetry and producerRetryDirective Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result.
CorrectionRequest's hard-coded retry-message construction Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions.
Correction capability declarations and preparation checks Retire checks tied to single_response_v1; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange.

Native transient retries, structural repair, and semantic additional turns remain distinct controls, but Notarius need not implement all three loops. Existing producer and validator retries settings cannot be mechanically renamed: their current budgets cover different behavior. Non-LLM failure handling should be reviewed separately rather than routed through conversation coordination.

Validation And Safe Fallback Still Have Application Meaning

Keep source grounding, reference resolution, domain invariants, reconciliation plan checks, typed application rules, and candidate isolation. In particular, normalization must still reject unknown or overlapping candidate handles and apply only safe groups. A final fallback can retain safe merges while leaving unsafe groups separate; it must go through configured artifact validation and terminal policy before it can advance. After a native terminal failure covered by the selected fallback policy, a supported deterministic fallback can be assessed directly instead of first restarting the same model operation through a feedback-free module retry. Which earlier safe candidate, if any, may survive a later failed turn remains an explicit policy question; do not introduce implicit best-so-far acceptance.

Retain the distinction between a rejection and unavailable validation. Under the current warn_continue policy, incomplete validation can advance a structurally valid candidate but prevents accepted-artifact cache reuse/publication, including for derived results. Continuation adoption does not itself change that policy or safe-fallback eligibility. Keep raw conversation state in process and out of cached artifacts; explicit detailed debug remains separate. Item 5 supersedes checkpoint-specific publication and provenance machinery.

PromptKit owns everything within an individual request execution, including structural validation, internal retries, and repairs. Notarius consumes the public result or failure and owns domain interpretation, validation, and any application-directed correction passes. Do not inspect or reproduce internal PromptKit attempts to decide cache eligibility; cache only the final accepted Notarius artifact after its complete application process succeeds.

PromptKit should own validation and repair of the private model response. Notarius's serialized schema validator currently receives the codec-produced artifact and its artifact schema, which can differ from the model response. Do not remove that boundary merely because both use JSON Schema. Audit actual duplicate response checks individually; artifact validation and Go/domain interpretation are not automatically redundant. Item 2's resolved native asset and schema design preserves this distinction.

Evidence: producer attempt loop, validation executor, typed artifact validation, NPC normalizer, semantic reconciliation, and the native continuation evidence in item 7.

Open Implementation And Policy Questions

  • Exact repair prompt definitions, response schemas, and profile selection within the agreed complete-replacement approach.
  • Migration of producer and validator retries settings to native controls, including configuration names, defaults, and compatibility.
  • Detailed terminal/fallback handling, including whether an earlier safe candidate may survive a later failed turn.
  • Failure handling for non-LLM operations and any justified recovery beyond native execution budgets.
  • Item 5's remaining cache implementation details and migration to the selected CommandKit output contract.

Follow-up: Resolve the remaining details above, then reconcile the adoption roadmap. Record the changed architecture in a successor to ADR-0014 and ADR-0016. Migration checks should cover accumulated history, fixed handle mappings, revalidation, final-turn acceptance, native budget exhaustion, safe fallback, validator failure, cancellation, and cache eligibility. Replace tests of the retired message-reconstruction protocol rather than preserving its mechanics.

4. Global Scheduling And Admission

Discussion / decision: Direction agreed: remove Notarius's LLM scheduler and retire the separate cross-backend ceiling. Rely on PromptKit's native backend admission and capacity limits while keeping bounded Notarius pipeline workers. This deliberately changes the existing global-limit contract; reproducing that ceiling in PromptKit is not a migration prerequisite. No application changes have been implemented.

Agreed Ownership

Concern Owner
Bounded extraction jobs and lane continuations Notarius pipeline workers
Backend admission, active model attempts, queues, cancellation, and permit release PromptKit
Application disposition after admission or execution failure Notarius application policy

For limited backends, PromptKit v0.23.2 bounds active model attempts by concurrency_limit and admitted operations by that limit plus queue_capacity. An operation holds admission across preparation, retries, and structural repair, while active permits cover individual model attempts and are released during retry waits. A completed continuation turn releases admission before application assessment, so its callback can invoke an LLM validator using the same backend.

The current Notarius wrapper holds a global permit around the complete structured call, including backend waiting and repairs. Removing it eliminates that additional queue and avoids making independent backends share those permits. Independent native backend limits may allow aggregate concurrency above the former global ceiling; that is an accepted behavior change. Pipeline worker bounds remain application work controls, not a replacement global LLM ceiling.

Removal And Configuration Consequences

  • Remove Scheduler, scheduledClient, their production wiring, and custom queue/cancellation/permit bookkeeping.
  • Retire concurrency.total_llm. Give pipeline worker settings an independent default and remove the extraction-worker maximum derived from total_llm.
  • Use one shared execution engine for producers, validators, and repair turns. Native capacity belongs to an engine and selected backend ID; separate engine instances have independent pools.
  • Configure native limits at the intended backend capacity boundary. Profiles selecting the same backend ID share its pool; separate IDs pointing to the same endpoint do not automatically share capacity. Endpoint-only profiles and backends without a positive limit remain unrestricted by backend admission.
  • Handle native ErrCapacityExceeded as an admission failure through application policy, consistently with item 3's removal of blanket execution retries.
  • Retire scheduler implementation tests with the removed code. Retain focused integration checks for shared native admission across roles, independent backends, bounded pipeline work, queue exhaustion, and cancellation.

Remaining Details

Select independent worker defaults, native backend/queue defaults where the application supplies them, and the configuration migration treatment of the retired field. Specify how admission failures are reported and handled for each application role. These details do not reopen the selected ownership boundary.

A future demonstrated need for capacity shared across several backend IDs could justify an upstream PromptKit feature request. No such feature is required for this agreed design, and it does not justify retaining the Notarius scheduler.

Evidence: scheduled client, scheduler, pipeline workers, and native capacity behavior.

Follow-up: Reconcile the adoption roadmap's scheduler-preservation language and record the changed ownership in an ADR. Update architecture, configuration, operations, and implementation documentation when the migration lands. Continue with item 5's accepted-artifact cache discussion.

5. Accepted-Artifact Cache

Discussion / decision: Agreed. Rename and replace the checkpoint subsystem with a simple cache surrounding the complete Notarius producer operation. Compute its key early from the initial prepared request's message contents; store its value late, after Notarius has accepted the final artifact. This supersedes both comprehensive execution fingerprints and the earlier stage/input/prompt-ID matching proposal. Implementation is pending.

Purpose And Key

The cached value is an accepted answer to the original task. After a downstream failure, rerunning the pipeline can reuse that artifact without repeating the expensive model work that produced it. Cheap deterministic work elsewhere in the pipeline does not need a general stage-checkpoint framework.

Use a PromptKit-provided MessageContentHash (proposed name) over the initial request's ordered messages, including each message's role and exact text. Exclude model settings, session IDs, execution IDs, timestamps, and provider cache hints. Do not add whole-configuration, stage, prompt-ID, library-version, or validation- policy fingerprints to this matching rule.

Changing the actual initial messages changes the key. Changing a prompt ID, model, or setting without changing those messages does not. Fixing deterministic interpretation or validation may therefore require explicit refresh/bypass to replace a previously accepted artifact; a hit does not claim fresh validation.

This is a concrete upstream hash requirement. The reviewed PromptKit v0.23.3 RenderedPromptHash includes session identity and provider cache hints, while hashing the entire PreparedRun would include volatile execution metadata. Neither is the selected key. Expose the content hash through native preparation rather than implementing another message-hashing policy in Notarius. Confirm the supplying upstream API/release before cache implementation; do not claim the proposed field already exists or silently substitute a broader fingerprint.

Compute Early, Store After Acceptance

Prepare initial request -> obtain message-content hash -> look up cache/<hash>.json
    Hit: decode the accepted Notarius artifact and continue the pipeline
    Miss: execute the complete Notarius operation
          -> native PromptKit calls as needed
          -> domain interpretation, validation, and correction passes
          -> on final acceptance, save the artifact under the initial hash

Lookup occurs before model execution. Use native preparation to inspect the initial request; preserve that prepared request for execution on a miss where the native API supports it. This is a reason to prepare before deciding whether to execute, distinct from preparing solely to obtain logging details.

Retain the initial hash throughout application-directed correction. Save the final accepted artifact under that hash, not under a later correction request's hash. A subsequent run can then reuse the accepted answer without replaying the failed turns that led to it.

The value is the final Notarius artifact, serialized through its existing codec. For normalization, it is the accepted normalized artifact after plan application, not the model's proposed reconciliation plan. Do not store an intermediate raw response, RunResult, exchange, or prepared-execution handle as the cache value.

Only publish after the whole Notarius process succeeds, including configured domain validation and any application-directed retries/correction passes. Rejected, failed, and validation-incomplete results are not cacheable. Retain the existing exclusion of derivatives of validation-incomplete work without restoring checkpoint manifests or fingerprint graphs to implement it.

A hit loads the accepted artifact directly. Ordinary decoding and usability checks apply; absent or unusable entries are misses. Do not execute a model request, reconstruct native results/exchanges, replay corrections, or repeat semantic validation merely because the value came from cache. Do not count historical model work as current execution.

PromptKit Ownership Boundary

PromptKit owns the mechanics within each request execution, including structural response validation, internal retries, and repairs. Notarius handles a usable response or terminal failure at the public call boundary, then applies its own domain interpretation, validation, correction, and acceptance process.

Internal PromptKit attempts are not inputs to the cache design. Native debug observations may expose evidence about them, but Notarius does not inspect or reproduce those mechanics to determine cache eligibility. The upstream addition needed here is the message-content hash; no cached-response injection, replay, exchange restoration, or PromptKit-owned write-through cache is required.

Storage And Removal

Use a small filesystem lookup/write facility with cache/<hash>.json containing the accepted artifact. Ordinary temporary-file/rename publication suffices to avoid exposing partial entries. Cache lookup must not require original output or debug directories, acceptance ledgers, execution provenance, or a separate versioned checkpoint envelope. Debug capture can record the current hit and loaded artifact without requiring historical metadata in the cache file.

Retire checkpoint-specific identities, invocation-wide lookup barriers, stage/dependency fingerprints, manifests, and recovery bookkeeping. Do not preserve that machinery solely to cache cheap deterministic stages. The existing standalone canonical chunk-plan cache remains a separate concern.

The old --resume/--recompute-step conventions are migration context, not a compatibility requirement. Select straightforward cache reuse/bypass/refresh controls and legacy-file handling during implementation planning; do not infer new flag spellings or a migration framework from this decision.

Follow-up: The master roadmap now includes this design and the upstream hash requirement. During implementation, verify that a downstream failure leaves earlier accepted artifacts reusable, corrected answers are stored under the initial hash, changed messages miss, settings-only changes can hit, and a hit makes no model calls or new semantic-validation claims. Keep output/report contract decisions separate.

6. Inspection And Execution Bookkeeping

Discussion / decision: Direction agreed: use native PromptKit inspection and execution facts, with CommandKit owning standardized operator presentation. Remove duplicate LLM reporting and accounting machinery. Notarius retains application context, domain decisions, artifact delivery, completion-summary content, and optional debug capture. The CommandKit output decision below removes mandatory durable execution provenance. No application changes have been implemented.

Ownership

Responsibility Owner
Configuration provenance, resource inspection, and request preparation ConfigKit and PromptKit through their native APIs.
Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility CommandKit and its ckconfig/ckprompt integrations.
Provider attempts, transport retries, structural repairs, execution failures, and token observations PromptKit observations and results; ExecutionGroup for selected execution totals.
Step/lane/chunk/validator attribution and pipeline progress meaning Notarius.
Acceptance, rejection, fallback, accepted-artifact reuse, and pipeline outcome Notarius.
Artifact storage, completion-summary content, and optional debug storage Notarius.

Native Inspection And Operator Presentation

Remove the profile-inspector wrapper, copied effective-model-parameter types, and local reconstruction of upstream configuration. Use native inspection for configured resources and native prepared/result data for the target actually executed, including request overrides. Profile inspection can describe a partial configured profile; it is not evidence of the completed execution target.

Adopt ckconfig and ckprompt for standard inspection, requirements, validation, preview, and opt-in smoke commands. Notarius supplies configuration/engine loaders and application request builders. Avoid building temporary custom commands or report machinery that the planned CommandKit adoption would replace.

Retain application pipeline preflight: resolve modules, artifact compatibility, references, and known prompt/profile selections. Use native definition checks without simulating stage execution or requiring provider credentials. Concrete request preparation remains the place to check the completed target, inputs, schema, and credential requirements.

CommandKit supplies formatting, visibility, progress presentation, and delivery error handling. Notarius supplies report content, domain outcomes, and exit-status mapping. Follow the small structured completion report and native text template selected in CommandKit item 4; do not preserve the old receipt shape or independent encoders.

Native Execution Facts

  • Use ordinary Run, with OnPrepared when actual prepared details are needed for provenance or optional debug capture. Remove unconditional PrepareExecution/RunPrepared use solely to obtain those details.
  • Use native observations for terminal execution facts, including failures that return no RunResult. OnPrepared is conditional and is not delivered after preparation or admission failure; it cannot serve as a completion record.
  • Use native RunID/ParentRunID with application scope to correlate executions. Remove the private debug completion wrapper and duplicate synthetic call-ID machinery where native identity and application attribution suffice.
  • Remove parallel LLMExecutionSummary, token-usage types, enum conversions, cloning, and independent attempt/usage accumulators. Use native types internally and project selected fields only at an application reporting or durable boundary.
  • Remove the adapter's LLMProfileRecorder and provider-forwarding plumbing. Native snapshots provide the profiles and targets actually used for optional diagnostics; do not reconstruct a durable profile inventory.
  • Use ExecutionGroup for totals over selected executions. Explicitly enroll the initial run, continuation runs, and nested validator runs as applicable. Preserve terminal-coverage and token-usage completeness; unknown usage is not zero. Do not count both attempt observations and terminal aggregates, or add overlapping groups as though they were disjoint work.

Application execution code should attach scope and observers, run and assess the native result, and record its disposition and execution references. This does not require a replacement general-purpose LLM interface. A successful native execution can still produce a domain-rejected candidate; provider success and application acceptance remain separate facts.

Application Records And Durable Boundaries

Attach the application context needed to explain a pipeline in optional debug capture: which stage/lane/chunk produced an artifact, which validators assessed it, whether continuation or a safe fallback was used, and whether the artifact was generated now or reused. PromptKit cannot infer that meaning from provider execution alone. These diagnostic facts do not require consumer-facing histories.

Under the revised item 5, cache files contain accepted artifacts without required historical execution or acceptance records. A current debug record can identify reuse and show the loaded artifact without counting earlier token usage as current work or claiming new validation.

The selected CommandKit output contract delivers artifacts as files and a small completion report through native rendering. It does not require a run manifest, metadata bundle, separately persisted receipt, or durable execution ledger. Existing manifest and receipt contracts document migration impact; their field inventories are not requirements to recreate. Preserve intrinsic domain artifact content and reuse existing outcome facts for the completion summary and debug overview.

Keep application debug artifact placement, opt-in capture, and lifecycle under CommandKit item 5. Native callbacks supply execution data; they do not own Notarius's storage policy. Detailed prepared messages and execution history belong in that optional capture, not a mandatory consumer metadata bundle.

Evidence: current inspection wrapper, adapter bookkeeping, debug completion wrapper, native PromptKit APIs, ckconfig integration, ckprompt integration, and CommandKit reporting.

Remaining Implementation Work

Select application scope and observer/group enrollment for the agreed diagnostics. Implement the selected completion report and retire superseded manifest, receipt, and execution-ledger machinery. Coordinate command migration and debug capture with the incorporated design in the master roadmap. The core bookkeeping ownership change needs no new PromptKit capability; any remaining upstream request should identify a concrete unmet requirement.

Follow-up: Reconcile the adoption roadmap with items 1–6 and record the significant boundary changes in successor ADRs. Update current-behavior documentation when implemented. Verify failure reporting without a result, continuation/validator accounting, unknown usage, actual-target provenance, cache reuse accounting, and optional debug capture during implementation.

7. Compare Audita's PromptKit Workflow

Review status: Completed on 2026-11-18 against the clean sibling checkout ../audita, revision da5b8913d1206a8e8f0a784b2322760385548a60. Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision, not new Notarius contracts. The source links below assume the sibling checkout.

Embedded Defaults And Operator Replacement

  • One captured engine is assembled from PromptKit's default catalog, the embedded Audita underlay, and a higher-priority operator underlay. FilesystemInputs supplies eleven native prompt definitions in collections.audita: four proposal prompts, four repair prompts, and three validator prompts.
  • Operator sources are optional prompts.yml and recursively selected YAML under prompts/, anchored to the application main source's recorded context. Both locations feed one operator underlay and are peers; conflicting peer definitions fail rather than receiving filename-based precedence.
  • Replacement is by resource identity, not filename. For example, a complete prompt record with local ID modules.grammar.repair under collections.audita.prompts replaces audita/modules.grammar.repair while other embedded definitions remain available. Native same-ID replacement is whole-record replacement, not recursive merging of individual prompt fields.
  • Native imports and message-file context are retained. Tests show that an existing engine retains captured message content after a file edit and a new engine sees the edit. They also verify inherited message-file resolution, main-relative discovery independent of the working directory, and peer conflict rejection.
  • A diagnostic embedded-only construction can classify a loading error; it does not silently fall back to defaults after an operator-source failure.

Evidence: runtime assembly, source tests, and assembly failure tests.

Prompt Content And Schema Ownership Are Separate

Audita uses ordinary embedded filesystems for prompt and schema assets. Its small prompt registry records IDs and embedded paths; it does not reconstruct Notarius-style per-module virtual prompt trees. Shared instruction fragments are supplied as template variables. Processing supplies the application-owned values for those shared names, but an operator replacement decides whether its template references them at all. They are not forcibly appended to replacement messages.

The processing request independently supplies a complete native output contract: JSON Schema validation, the schema path selected for that operation, and its structural-repair budget. The three schemas describe a correction set, a repair decision, and validator decisions. PromptKit receives their canonical embedded filesystem via WithSchemaFS. Thus operator control of prompt wording does not change the processing code's expected response shape. A test replaces a prompt with a text/no-validation declaration and still exercises the processing request's JSON-schema repair contract.

Audita still keeps a response-schema key/metadata/hash registry and application decoding checks. Its use of native loading does not prove that every schema wrapper is necessary, nor decide Notarius's deferred schema design. It supplies a concrete option: module-owned response requirements expressed directly through native per-request contracts. Standalone native preview/preparation can use the prompt declaration's contract; processing supplies its own override.

Evidence: prompt registry, schema registry, prepareCall and OutputContract in the processing adapter, and TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText in adapter tests.

Native Continuation With Application Assessment

The implemented flow is:

  1. Generate a section's proposal set, capturing a native immutable exchange when correction repair is enabled. This retains the actual proposal messages and response; it does not require debug recording.
  2. Apply application validation. An actionable rejected proposal starts its own branch from the original proposal exchange. Unavailable-only findings and no-effect corrections do not start repair. Existing approvals remain separate.
  3. Call Engine.Coordinate with that exchange and an additional-turn budget. Each follow-up selects a distinct repair prompt, repair profile, and repair response schema. The request carries the original proposal reference, current correction, ordered findings, and available before/after text previews.
  4. PromptKit assembles history and progresses the conversation. A second turn retains the first repair instruction and response. A sibling branch starts from the original exchange and does not inherit another branch's repairs. Audita does not rerender the original proposal or manually replay messages.
  5. Audita decodes each delivered decision. Withdrawal ends the branch without applying that correction. A revision undergoes the full application validation chain again, including validators that approved its earlier version. Approval stops the branch; actionable rejection can request another turn.
  6. The final permitted response is assessed before exhaustion is determined. Requesting another turn at that boundary does not allocate another logical call. Recoverable provider/invalid-output failures finish the branch as unavailable; fatal errors or parent cancellation interrupt processing. Canceled, undelivered responses are diagnostic evidence only, never approvals.

Transport retries and structural repair operate inside each PromptKit execution; they are separate from the semantic branch's additional-turn budget. Tests show one semantic repair turn containing multiple provider attempts for structural repair. The application retains acceptance, withdrawal, application ordering, and reporting; PromptKit retains conversation and execution mechanics.

Audita's prompt roles support this model deliberately: system instructions carry durable principles, the initial user message carries the initial module task, and the repair prompt supplies a new user assignment. Adopting continuation in Notarius would require reviewing the complete conversation and response-format transitions, rather than mechanically appending to prompts written for fresh replacement requests.

Evidence: native coordination bridge, domain repair flow, wire-level conversation tests, runner repair tests, and prompt guidance.

Other Boundaries And Limits Of The Comparison

  • Audita still has RoleClient, a structured-call interface, an opaque exchange wrapper, and application diagnostic types. Adopt the demonstrated native mechanisms without assuming Notarius should copy those wrappers; item 1's adapter-removal direction remains in force.
  • Proposal, validator, and repair roles share one engine and its per-backend admission. The runner separately bounds section work. This demonstrates useful separation of application work scheduling and provider capacity. Item 4 now selects that ownership and retires Notarius's independent cross-backend ceiling.
  • Ordinary calls use Run, OperationTimeout, and OnPrepared. Observations feed a native ExecutionGroup; application records add logical-call and stage/section/proposal attribution. Nested validator executions enroll explicitly. Counts preserve terminal coverage and token-usage completeness.
  • CommandKit's ckconfig and ckprompt integrations supply configuration and prompt commands using Audita's engine construction, including request preparation, validation, and smoke diagnostics.
  • Audita edits independently assessable transcript corrections; Notarius accepts typed stage artifacts and has cross-stage dependencies. Audita branches at an individual correction within a proposal response. Notarius must deliberately choose its correction unit rather than assume per-entity branching.
  • Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's cache design; item 5 records that application decision independently.

Evidence: LLM runtime, observations, CLI integrations, and component scope.

Implications For The Remaining Discussion

Item 2's native underlay direction has a working application precedent. Audita also offers evidence for ordinary embedded filesystems and per-request schema selection. U2 now selects ordinary embedded assets and prompt-declared contracts by default, reserving per-request replacement for demonstrated runtime needs.

The subsequent item 3 discussion selects native continuation and accumulated conversation history. Keeping domain validation local does not require keeping conversation assembly or the turn loop local. Item 3 owns the agreed Notarius flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details.

Item 4 now records agreement to use native admission with bounded application workers. The revised item 5 selects an accepted-artifact cache and a focused upstream message-content hash instead of comprehensive execution fingerprints. Item 6 adopts native execution observations and CommandKit presentation; the CommandKit output decision selects artifact files and a small native completion report. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet requirement after reconsidering Notarius's historical behavior.

Verification

Read the production composition, adapter, coordinator, domain repair callbacks, prompt/schema definitions, and focused behavior tests. The following checks passed from the Audita checkout:

go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1

The subprocess checks exercise assembled configuration, real native continuation messages, operator replacement, final-turn assessment, sibling isolation, repair-disabled behavior, and cancellation using a local fake provider. These checks establish implementation behavior, not live-model quality. No live model calls or application source changes were made. Broader migration decisions and changes to the adoption roadmap remain for discussion.