49 KiB
PromptKit Architecture Discussion Record
Purpose And Status
The decisions in items 1–6 have been incorporated into the PromptKit-native architecture migration roadmap. That roadmap is the implementation plan and owns current unresolved choices, sequencing, acceptance criteria, and implementation status.
This document retains the discussion, rationale, and evidence, including item 7's completed Audita comparison. Item-level questions and follow-ups below record the discussion as it developed; they are not a second implementation plan.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not necessarily require application ownership of its implementation. Evaluate remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need.
No application changes have been implemented through this discussion or the roadmap rewrite. Significant changes to accepted architecture need successor ADRs under the documentation policy. Current boundaries are described in Architecture and LLM Runtime Internals.
1. LLM API And Diagnostic Boundary
Discussion / decision: Direction agreed: remove the Notarius PromptKit adapter and compose the application directly from native PromptKit execution, typed artifact operations, and pipeline orchestration. A small application function connects generation to configured validation and acceptance. The design below incorporates items 2–6 and the Audita review; exact signatures and migration details remain open. No application changes have been implemented.
Background And Intent
Notarius and related applications, including Audita, originally developed their own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by extracting their duplicated mechanics into a shared library. Notarius adopted an early version and predates many of the library's modern capabilities.
The maintainer clarified that duplication of PromptKit capabilities, including slight behavioral differences, does not reflect an intentional requirement for an independent Notarius implementation. Existing wrappers, tests, and historical decisions are migration evidence, not sufficient reasons to preserve that duplication. Identify actual application needs before choosing what to retain.
Missing reusable capabilities may justify upstream feature requests. The discussion is not constrained to what v0.23.2 can replace immediately; any resulting change to the adoption target remains to be selected.
Application Structure
| Part | Responsibility |
|---|---|
| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. |
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
| Application records and CLI | Store artifacts and checkpoints, attach pipeline meaning to execution facts, and present results through CommandKit. |
Engine construction becomes ordinary application wiring using item 2's embedded
defaults and operator sources. Profile selection is expressed through native
requests. The current internal/framework/llm package can shrink to engine
assembly and focused helpers; whether those warrant a separate package remains
an organizational choice, not a reason to retain an adapter.
Use native requests, results, exchanges, errors, and observers directly in model-executing application code. Pure domain models, codecs, source-reference resolution, and deterministic transformations need no PromptKit dependency. Retain application interfaces for extracting artifacts, validating typed candidates, encoding output, and loading checkpoints. Deterministic stages continue to operate without conversation machinery.
Connect Modules To Pipeline Assessment
The current spell extractor prepares inputs, invokes the private completion interface, interprets a response, and returns a copied raw model candidate to the producer retry loop. Replace that arrangement with direct collaboration between module interpretation, pipeline assessment, and native execution:
- The module prepares source material, references, and any candidate-handle mappings once, and constructs the native initial request.
- Application execution calls
Engine.Run, capturing an exchange when continuation is enabled. - The module interprets the structurally valid response into a typed candidate or actionable domain findings.
- The pipeline runs configured candidate validation and decides whether to accept, request correction, or apply terminal policy.
- If correction is needed, native
Coordinateinvokes the same interpretation and assessment for revised responses. The module supplies the repair request; PromptKit maintains history and enforces the turn budget.
A small shared application function can connect these responsibilities: the module supplies native request construction and response interpretation, while the pipeline supplies configured validation and terminal policy. It uses native PromptKit types and returns a typed application outcome. It does not introduce another completion API, conversation protocol, or retry engine.
Item 3 owns the detailed continuation and assessment rules, including final-turn assessment, failure handling, complete replacement responses, and safe fallback. For normalization, a revised reconciliation plan is applied to the original inputs to produce a fresh candidate. LLM-backed validators can call the same native engine and return application validation findings; they do not need the producer's conversation machinery.
Removal, Records, And Testing
Retire StructuredLLMClient, the private structured-completion request/response
family, adapter conversions, and opaque correction transport. Items 3, 4, and 6
specify removal of the outer LLM retry loop, scheduler, debug completion wrapper,
profile-recording plumbing, and duplicate execution accounting. Do not recreate
that wrapper stack around native calls.
Attach pipeline scope through native callbacks and observers. Use native
ExecutionGroup totals and small application projections for manifests and
optional debug artifacts, following item 6. Checkpoint matching and durable
acceptance provenance remain the application responsibilities selected in item 5.
Audita demonstrates shared engine assembly, native continuation, and CommandKit
integration. Its RoleClient, structured-output interface, and opaque exchange
wrapper are not part of this design; item 7 records the comparison's evidence
and limits.
Test domain interpretation and assessment directly. Exercise model workflows with a real PromptKit engine and its injectable client or a fake HTTP provider. Pipeline orchestration can still use fake stage implementations. Testing does not require preserving a private completion interface.
Remaining Implementation Work
- Select concrete module/pipeline function signatures that connect request construction, interpretation, configured validation, and terminal disposition.
- Choose the location of engine assembly and focused application helpers without introducing a replacement LLM service layer.
- Resolve item 2's deferred filesystem and schema choices and the remaining migration details recorded under items 3–6, including durable compatibility.
- Evaluate any remaining upstream gap against an actual application requirement, rather than preserving historical machinery by default.
Follow-up: Reconcile the adoption roadmap with this design and items 2–6. Record the significant boundary change in successor ADRs. Update architecture and implementation documentation when the change lands, explicitly replacing the current transport-neutral completion boundary with PromptKit as the execution dependency of application workflows. Keep domain operations and application outcomes as the meaningful Notarius-owned boundaries.
2. Prompt Assets And Schema Assembly
Discussion / decision: Direction agreed: use native PromptKit declarations as the sole prompt assembly mechanism and remove the duplicate application assembly plumbing. Resolution of the custom read-only filesystem and schemas is explicitly deferred; the next topic may clarify those choices.
Selected direction: Keep the embedded content tree and load its prompt and profile definitions as native PromptKit configuration in the application collection. Use native filesystem contexts and imports to reference shared content directly. Application composition assembles the embedded and operator sources once; modules select prompt IDs, supply domain inputs, and interpret typed results.
The spell extractor illustrates the current duplication: its prompt YAML declares ordered messages and a schema, while a separate Go asset manifest lists files, mounts shared fragments, registers filesystems, and hashes them. The selected direction makes native declarations the sole prompt assembly source.
Candidates for removal:
- Prompt and fallback-profile registration plumbing in
AssetRegistryand individual modules; schema responsibilities remain under review. ModulePromptFS,PromptAssetManifest, shared-name lookup tables, syntheticsharedassetsmounts.- Manual prompt-file enumeration for checkpoint fingerprints, retired under item 5's simpler reuse policy. Native prompt hashes remain available for inspection and diagnostics.
Keep prompt prose, ordered messages, cache controls, workload profiles, private response schema documents, and domain interpretation. Durable artifact schemas remain separate. Content ownership does not require runtime asset registration.
Native schema paths resolve against the configured schema root, separately from message-file source contexts. PromptKit v0.23.2 loads schemas during preparation and requires self-contained schema documents. A captured prompt hash alone does not establish schema equality. Existing schema metadata in manifests and checkpoint inputs must be deliberately replaced or retired; its presence does not require preserving the complete wrapper system.
Deferred decisions:
- Whether the custom read-only filesystem can be removed after reviewing its remaining responsibilities.
- Schema assembly, root/path selection, wrapper removal, and schema metadata. Supplying the embedded filesystem as a native schema root remains a candidate, not a selected design.
- Schema metadata for manifests and artifact compatibility. Item 5 settles that checkpoint matching does not require exact schema identity.
Implementation questions:
- Can native sources represent the existing shared fragments and module assets?
- Which registry, remounting, and hashing helpers could disappear?
- What schema metadata remains useful for inspection and artifact loading under item 5's simpler reuse policy?
- How do we preserve message ordering and cache controls while simplifying assembly?
Follow-up: Continue with item 3, then revisit the deferred filesystem and schema decisions with any new clarity. Confirm native source contexts for shared fragments and operator replacements. Apply item 5's reuse policy and resolve remaining metadata with item 6. Verify the resulting definitions through native preparation and fake-provider execution; filesystem remounting tests can retire with the removed implementation. The agreed direction has not yet been implemented.
3. Retry And Semantic Correction Ownership
Discussion / decision: Direction agreed: adopt Audita's native continuation approach. PromptKit owns captured exchanges, accumulated conversation history, continuation execution, and turn-budget enforcement. Notarius supplies semantic assessment, corrective findings, and acceptance or fallback decisions. Use native PromptKit types directly, consistent with item 1; Audita's adapter and opaque exchange wrapper are not part of the selected direction.
The flow and removal/adaptation direction below are agreed. Begin with one conversation per producer operation and complete replacement model responses. Retain application assessment, safe-fallback decisions, artifact validation, and checkpoint eligibility. Detailed response contracts, budget migration, and the open policy questions below remain to be resolved. No application changes have been implemented.
Agreed Flow
- Prepare the source context, references, and any request-local candidate-handle mapping once for the producer operation. Run its initial native request with exchange capture when continuation is enabled. Capture is independent of detailed debug recording.
- Inspect native structural validation, decode the response, and perform the module's deterministic interpretation and safety checks. Run the configured validation chain on a materialized candidate. A module can identify actionable defects before a candidate is ready for that chain.
- If correction is needed, call
Engine.Coordinatewith the initial exchange and a configuredAdditionalTurnsbudget. The callback selects a native repair prompt and supplies actionable findings as variables. PromptKit retains the original exchange and each subsequent repair turn. - Assess every delivered response, including the last permitted turn. Rebuild the typed candidate from the fixed operation inputs and the revised response; do not repeatedly apply revisions to an already modified artifact. A changed candidate receives the complete configured validation chain again, including validators that approved an earlier version.
- Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
whether to reject, fail, or evaluate an explicitly supported safe fallback.
Native
stoppedandexhaustedoutcomes describe coordination, not acceptance. Cancellation never turns an undelivered response into an accepted candidate.
Start with one conversation per producer operation: a chunk-plan generation, one lane's extraction of one chunk, or one lane's normalization pass. This aligns with Notarius's whole-output validation boundary. Audita's per-proposal branches are useful evidence, but per-entity repair would require its own combination and validation design. It is not necessary to adopt native continuation.
Repair prompts should supply a new user task, with enduring principles in the original system instructions. Avoid replaying the initial task or embedding initial-response-only constraints in system messages. Begin with complete replacement model responses; for normalization, the response can remain a reconciliation plan that Notarius applies to construct the durable artifact. A different repair schema or profile is supported but not yet selected. Keep source context and candidate-handle meanings stable throughout the conversation. Accumulated history increases context and token use across turns; the semantic turn budget should reflect that tradeoff.
Machinery To Remove Or Adapt
| Current machinery | Agreed direction |
|---|---|
CorrectionProtocolSingleResponseV1, raw-response ModelCandidate, and SemanticCorrection |
Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native RunResult / Exchange for continuation state. |
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
runProducerAttempts combining operational, structural, module-requested, and validator-semantic retries |
Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
executeValidationChain's repeated LLM validator invocations |
Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
NormalizeRetry and producerRetryDirective |
Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
CorrectionRequest's hard-coded retry-message construction |
Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
| Correction capability declarations and preparation checks | Retire checks tied to single_response_v1; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
Native transient retries, structural repair, and semantic additional turns remain
distinct controls, but Notarius need not implement all three loops. Existing
producer and validator retries settings cannot be mechanically renamed: their
current budgets cover different behavior. Non-LLM failure handling should be
reviewed separately rather than routed through conversation coordination.
Validation And Safe Fallback Still Have Application Meaning
Keep source grounding, reference resolution, domain invariants, reconciliation plan checks, typed application rules, and candidate isolation. In particular, normalization must still reject unknown or overlapping candidate handles and apply only safe groups. A final fallback can retain safe merges while leaving unsafe groups separate; it must go through configured artifact validation and terminal policy before it can advance. After native structural-repair exhaustion, a supported deterministic fallback can be assessed directly instead of first restarting the same model operation through a feedback-free module retry. Which earlier safe candidate, if any, may survive a later failed turn remains an explicit policy question; do not introduce implicit best-so-far acceptance.
Retain the distinction between a rejection and unavailable validation. Under the
current warn_continue policy, incomplete validation can advance a structurally
valid candidate but prevents checkpoint reuse/publication, including for derived
results. Continuation adoption does not itself change that policy, safe-fallback
eligibility, or durable provenance. Keep raw conversation state in process and
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
PromptKit should own validation and repair of the private model response. Notarius's serialized schema validator currently receives the codec-produced artifact and its artifact schema, which can differ from the model response. Do not remove that boundary merely because both use JSON Schema. Audit actual duplicate response checks individually; artifact validation and Go/domain interpretation are not automatically redundant. Item 2's filesystem and schema assembly decisions remain open.
Evidence: producer attempt loop, validation executor, typed artifact validation, NPC normalizer, semantic reconciliation, and the native continuation evidence in item 7.
Open Implementation And Policy Questions
- Exact repair prompt definitions, response schemas, and profile selection within the agreed complete-replacement approach.
- Migration of producer and validator
retriessettings to native controls, including configuration names, defaults, and compatibility. - Detailed terminal/fallback handling, including whether an earlier safe candidate may survive a later failed turn.
- Failure handling for non-LLM operations and any justified recovery beyond native execution budgets.
- Item 2's deferred filesystem and schema assembly choices, item 5's remaining checkpoint implementation details, and item 6's durable bookkeeping design.
Follow-up: Resolve the remaining details above, then reconcile the adoption roadmap. Record the changed architecture in a successor to ADR-0014 and ADR-0016. Migration checks should cover accumulated history, fixed handle mappings, revalidation, final-turn acceptance, native budget exhaustion, safe fallback, validator failure, cancellation, and checkpoint eligibility. Replace tests of the retired message-reconstruction protocol rather than preserving its mechanics.
4. Global Scheduling And Admission
Discussion / decision: Direction agreed: remove Notarius's LLM scheduler and retire the separate cross-backend ceiling. Rely on PromptKit's native backend admission and capacity limits while keeping bounded Notarius pipeline workers. This deliberately changes the existing global-limit contract; reproducing that ceiling in PromptKit is not a migration prerequisite. No application changes have been implemented.
Agreed Ownership
| Concern | Owner |
|---|---|
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
| Application disposition after admission or execution failure | Notarius application policy |
For limited backends, PromptKit v0.23.2 bounds active model attempts by
concurrency_limit and admitted operations by that limit plus queue_capacity.
An operation holds admission
across preparation, retries, and structural repair, while active permits cover
individual model attempts and are released during retry waits. A completed
continuation turn releases admission before application assessment, so its
callback can invoke an LLM validator using the same backend.
The current Notarius wrapper holds a global permit around the complete structured call, including backend waiting and repairs. Removing it eliminates that additional queue and avoids making independent backends share those permits. Independent native backend limits may allow aggregate concurrency above the former global ceiling; that is an accepted behavior change. Pipeline worker bounds remain application work controls, not a replacement global LLM ceiling.
Removal And Configuration Consequences
- Remove
Scheduler,scheduledClient, their production wiring, and custom queue/cancellation/permit bookkeeping. - Retire
concurrency.total_llm. Give pipeline worker settings an independent default and remove the extraction-worker maximum derived fromtotal_llm. - Use one shared execution engine for producers, validators, and repair turns. Native capacity belongs to an engine and selected backend ID; separate engine instances have independent pools.
- Configure native limits at the intended backend capacity boundary. Profiles selecting the same backend ID share its pool; separate IDs pointing to the same endpoint do not automatically share capacity. Endpoint-only profiles and backends without a positive limit remain unrestricted by backend admission.
- Handle native
ErrCapacityExceededas an admission failure through application policy, consistently with item 3's removal of blanket execution retries. - Retire scheduler implementation tests with the removed code. Retain focused integration checks for shared native admission across roles, independent backends, bounded pipeline work, queue exhaustion, and cancellation.
Remaining Details
Select independent worker defaults, native backend/queue defaults where the application supplies them, and the configuration migration treatment of the retired field. Specify how admission failures are reported and handled for each application role. These details do not reopen the selected ownership boundary.
A future demonstrated need for capacity shared across several backend IDs could justify an upstream PromptKit feature request. No such feature is required for this agreed design, and it does not justify retaining the Notarius scheduler.
Evidence: scheduled client, scheduler, pipeline workers, and native capacity behavior.
Follow-up: Reconcile the adoption roadmap's scheduler-preservation language and record the changed ownership in an ADR. Update architecture, configuration, operations, and implementation documentation when the migration lands. Continue with item 5's execution identity and checkpoint reuse discussion.
5. Execution Identity And Checkpoint Reuse
Discussion / decision: Direction agreed: resuming preserves successful stage results for the same logical work and data; explicit recomputation applies changed execution behavior. Use a simple Notarius-owned matching policy instead of comprehensive execution fingerprints. No application changes have been implemented.
Purpose And Matching Rule
Checkpoints preserve a successful producer's result when a dependent stage fails. After fixing the failed stage, an operator should be able to retry it without paying to rerun the successful producer. Matching every configuration parameter or implementation revision would undermine that recovery purpose.
The agreed conceptual key is:
logical stage identity + input-data identity + prompt ID, where applicable
Scope the checkpoint to the relevant step/lane/stage and artifact kind so that shared prompt IDs do not mix unrelated results. These are logical identifiers, not hashes of implementations or complete configuration. Deterministic stages need no synthetic prompt identity.
Input data includes the source or chunk being processed, reference material, and upstream artifacts actually consumed. Matching only the original source would miss changes to generated dependencies. Content digests can compactly identify these inputs and check stored payload integrity without requiring an exact fingerprint of the execution environment.
| Change | Reuse behavior |
|---|---|
| Fix the implementation of a later failed stage | Preserve earlier successful stages with matching work and inputs. |
| Edit prompt text while retaining its ID | Reuse; explicit recomputation applies the edit. |
| Change model, generation settings, or implementation/validation policy | Reuse otherwise compatible completed work; explicit recomputation applies the change. |
| Change the prompt ID or data consumed by the stage | No matching checkpoint under this rule. |
| Explicitly request recomputation | Recompute the selected work and its dependents. |
The existing controls are --resume and --recompute-step, documented in the
CLI reference and
selective recompute workflow. Their
current strict compatibility checks still need migration to this agreed policy.
The accepted tradeoff is that a checkpoint may preserve a successful result
produced by code, prompts, or settings that have since improved.
Matching, Loading, And Provenance
A matching key selects a checkpoint; it does not make an unreadable or incompatible artifact usable. Retain completion/acceptance status, integrity checks, and the ability to decode and consume the stored artifact. Exact schema hash equality is not required when the artifact remains compatible.
Only accepted, completely validated results are reusable. Rejected, structurally invalid, or validation-incomplete results and their ineligible derivatives remain outside reusable state. Acceptance on resume refers to the earlier execution: do not automatically rerun semantic validators merely because their configuration changed, or claim the artifact passed the new validation policy. Preserve original provenance; explicit recomputation applies changed behavior.
Useful execution metadata can remain available for explanation without becoming a matching requirement. Neither complete conversation history nor native prepared-execution handles are needed to locate and reuse a completed artifact. The separate source-addressed canonical chunk-plan policy remains distinct.
Simplification And Ownership
- Remove the broad PromptKit configuration fingerprint from checkpoint matching, including effective-configuration parsing and whole-catalog enumeration.
- Remove library/catalog versions, model settings, prompt-content hashes, and component-policy fingerprints as automatic invalidators under this policy.
- Keep straightforward input/dependency identity, successful-stage records, artifact loading checks, storage, and explicit recomputation in Notarius.
- Keep PromptKit hashes available for native inspection and diagnostics without making them checkpoint prerequisites.
Comprehensive upstream execution identity is no longer required for this design. It does not require a later PromptKit release or a temporary conservative configuration fingerprint. Item 5 supersedes the earlier proposal to make an upstream semantic-equality feature part of the migration's checkpoint design.
Remaining Implementation Work
Define the concrete logical stage/prompt identifiers and consumed-input representation. Revise checkpoint lookup and storage scoping so that an invocation-wide pipeline/configuration digest cannot still prevent reuse of successful predecessors. Decide how existing checkpoint records transition to the simpler format and how incompatible payloads are reported.
Follow-up: Reconcile the adoption roadmap and record the changed reuse contract in an ADR. Update current-behavior documentation when implemented. Verify recovery after a downstream fix, reuse across prompt/settings changes, misses after consumed-data changes, explicit recomputation, and rejection of unusable or unaccepted stored artifacts. Continue with item 6.
6. Inspection And Execution Bookkeeping
Discussion / decision: Direction agreed: use native PromptKit inspection and execution facts, with CommandKit owning standardized operator presentation. Remove duplicate LLM reporting and accounting machinery. Notarius retains application context, domain decisions, durable artifacts, and their provenance. No application changes have been implemented.
Ownership
| Responsibility | Owner |
|---|---|
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; ExecutionGroup for selected execution totals. |
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
Native Inspection And Operator Presentation
Remove the profile-inspector wrapper, copied effective-model-parameter types, and local reconstruction of upstream configuration. Use native inspection for configured resources and native prepared/result data for the target actually executed, including request overrides. Profile inspection can describe a partial configured profile; it is not evidence of the completed execution target.
Adopt ckconfig and ckprompt for standard inspection, requirements, validation, preview, and opt-in smoke commands. Notarius supplies configuration/engine loaders and application request builders. Avoid building temporary custom commands or report machinery that the planned CommandKit adoption would replace.
Retain application pipeline preflight: resolve modules, artifact compatibility, references, and known prompt/profile selections. Use native definition checks without simulating stage execution or requiring provider credentials. Concrete request preparation remains the place to check the completed target, inputs, schema, and credential requirements.
CommandKit supplies formatting, visibility, progress presentation, and delivery error handling. Notarius supplies report content, domain outcomes, and exit-status mapping. Preserve deliberate machine-output contracts rather than treating every receipt as a generic formatted report.
Native Execution Facts
- Use ordinary
Run, withOnPreparedwhen actual prepared details are needed for provenance or optional debug capture. Remove unconditionalPrepareExecution/RunPrepareduse solely to obtain those details. - Use native observations for terminal execution facts, including failures that
return no
RunResult.OnPreparedis conditional and is not delivered after preparation or admission failure; it cannot serve as a completion record. - Use native
RunID/ParentRunIDwith application scope to correlate executions. Remove the private debug completion wrapper and duplicate synthetic call-ID machinery where native identity and application attribution suffice. - Remove parallel
LLMExecutionSummary, token-usage types, enum conversions, cloning, and independent attempt/usage accumulators. Use native types internally and project selected fields only at an application reporting or durable boundary. - Replace the adapter's
LLMProfileRecorderand provider-forwarding plumbing with the application projection needed for actual-run provenance. Do not substitute a catalog inventory for the profiles and targets actually used. - Use
ExecutionGroupfor totals over selected executions. Explicitly enroll the initial run, continuation runs, and nested validator runs as applicable. Preserve terminal-coverage and token-usage completeness; unknown usage is not zero. Do not count both attempt observations and terminal aggregates, or add overlapping groups as though they were disjoint work.
Application execution code should attach scope and observers, run and assess the native result, and record its disposition and execution references. This does not require a replacement general-purpose LLM interface. A successful native execution can still produce a domain-rejected candidate; provider success and application acceptance remain separate facts.
Application Records And Durable Boundaries
Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an artifact, which validators assessed it, whether continuation or a safe fallback was used, and whether the artifact was generated now or reused. PromptKit cannot infer that meaning from provider execution alone.
Under item 5, a reused artifact retains its original execution and acceptance provenance. The current invocation records the reuse decision without counting the earlier execution's token usage as current work or claiming new validation.
Retain small, deliberate application projections for the published manifest and run-result receipt, with explicit compatibility decisions for changed fields. Durable contracts should not become raw upstream type dumps or require parallel runtime representations of every native result.
Keep application debug artifact placement, opt-in capture, retention, and disclosure handling. Native callbacks supply execution data; they do not own Notarius's storage policy. Detailed prepared messages belong in the selected debug capture path, while ordinary provenance can retain only the needed fields.
Evidence: current inspection wrapper, adapter bookkeeping, debug completion wrapper, native PromptKit APIs, ckconfig integration, ckprompt integration, and CommandKit reporting.
Remaining Implementation Work
Select the minimal application scope and durable fields, observer/group enrollment, and compatibility treatment for existing manifests, receipts, and debug artifacts. Coordinate command migration with the roadmap's CommandKit follow-up. The core ownership change needs no new PromptKit capability; any remaining upstream request should identify a concrete unmet requirement.
Follow-up: Reconcile the adoption roadmap with items 1–6 and record the significant boundary changes in successor ADRs. Update current-behavior documentation when implemented. Verify failure reporting without a result, continuation/validator accounting, unknown usage, actual-target provenance, checkpoint reuse accounting, and optional debug capture during implementation.
7. Compare Audita's PromptKit Workflow
Review status: Completed on 2026-11-18 against the clean sibling checkout
../audita, revision da5b8913d1206a8e8f0a784b2322760385548a60.
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
not new Notarius contracts. The source links below assume the sibling checkout.
Embedded Defaults And Operator Replacement
- One captured engine is assembled from PromptKit's default catalog, the
embedded Audita underlay, and a higher-priority operator underlay.
FilesystemInputssupplies eleven native prompt definitions incollections.audita: four proposal prompts, four repair prompts, and three validator prompts. - Operator sources are optional
prompts.ymland recursively selected YAML underprompts/, anchored to the application main source's recorded context. Both locations feed one operator underlay and are peers; conflicting peer definitions fail rather than receiving filename-based precedence. - Replacement is by resource identity, not filename. For example, a complete
prompt record with local ID
modules.grammar.repairundercollections.audita.promptsreplacesaudita/modules.grammar.repairwhile other embedded definitions remain available. Native same-ID replacement is whole-record replacement, not recursive merging of individual prompt fields. - Native imports and message-file context are retained. Tests show that an existing engine retains captured message content after a file edit and a new engine sees the edit. They also verify inherited message-file resolution, main-relative discovery independent of the working directory, and peer conflict rejection.
- A diagnostic embedded-only construction can classify a loading error; it does not silently fall back to defaults after an operator-source failure.
Evidence: runtime assembly, source tests, and assembly failure tests.
Prompt Content And Schema Ownership Are Separate
Audita uses ordinary embedded filesystems for prompt and schema assets. Its small prompt registry records IDs and embedded paths; it does not reconstruct Notarius-style per-module virtual prompt trees. Shared instruction fragments are supplied as template variables. Processing supplies the application-owned values for those shared names, but an operator replacement decides whether its template references them at all. They are not forcibly appended to replacement messages.
The processing request independently supplies a complete native output contract:
JSON Schema validation, the schema path selected for that operation, and its
structural-repair budget. The three schemas describe a correction set, a repair
decision, and validator decisions. PromptKit receives their canonical embedded
filesystem via WithSchemaFS. Thus operator control of prompt wording does not
change the processing code's expected response shape. A test replaces a prompt
with a text/no-validation declaration and still exercises the processing
request's JSON-schema repair contract.
Audita still keeps a response-schema key/metadata/hash registry and application decoding checks. Its use of native loading does not prove that every schema wrapper is necessary, nor decide Notarius's deferred schema design. It supplies a concrete option: module-owned response requirements expressed directly through native per-request contracts. Standalone native preview/preparation can use the prompt declaration's contract; processing supplies its own override.
Evidence: prompt registry,
schema registry,
prepareCall and OutputContract in the
processing adapter, and
TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText in
adapter tests.
Native Continuation With Application Assessment
The implemented flow is:
- Generate a section's proposal set, capturing a native immutable exchange when correction repair is enabled. This retains the actual proposal messages and response; it does not require debug recording.
- Apply application validation. An actionable rejected proposal starts its own branch from the original proposal exchange. Unavailable-only findings and no-effect corrections do not start repair. Existing approvals remain separate.
- Call
Engine.Coordinatewith that exchange and an additional-turn budget. Each follow-up selects a distinct repair prompt, repair profile, and repair response schema. The request carries the original proposal reference, current correction, ordered findings, and available before/after text previews. - PromptKit assembles history and progresses the conversation. A second turn retains the first repair instruction and response. A sibling branch starts from the original exchange and does not inherit another branch's repairs. Audita does not rerender the original proposal or manually replay messages.
- Audita decodes each delivered decision. Withdrawal ends the branch without applying that correction. A revision undergoes the full application validation chain again, including validators that approved its earlier version. Approval stops the branch; actionable rejection can request another turn.
- The final permitted response is assessed before exhaustion is determined. Requesting another turn at that boundary does not allocate another logical call. Recoverable provider/invalid-output failures finish the branch as unavailable; fatal errors or parent cancellation interrupt processing. Canceled, undelivered responses are diagnostic evidence only, never approvals.
Transport retries and structural repair operate inside each PromptKit execution; they are separate from the semantic branch's additional-turn budget. Tests show one semantic repair turn containing multiple provider attempts for structural repair. The application retains acceptance, withdrawal, application ordering, and reporting; PromptKit retains conversation and execution mechanics.
Audita's prompt roles support this model deliberately: system instructions carry durable principles, the initial user message carries the initial module task, and the repair prompt supplies a new user assignment. Adopting continuation in Notarius would require reviewing the complete conversation and response-format transitions, rather than mechanically appending to prompts written for fresh replacement requests.
Evidence: native coordination bridge, domain repair flow, wire-level conversation tests, runner repair tests, and prompt guidance.
Other Boundaries And Limits Of The Comparison
- Audita still has
RoleClient, a structured-call interface, an opaque exchange wrapper, and application diagnostic types. Adopt the demonstrated native mechanisms without assuming Notarius should copy those wrappers; item 1's adapter-removal direction remains in force. - Proposal, validator, and repair roles share one engine and its per-backend admission. The runner separately bounds section work. This demonstrates useful separation of application work scheduling and provider capacity. Item 4 now selects that ownership and retires Notarius's independent cross-backend ceiling.
- Ordinary calls use
Run,OperationTimeout, andOnPrepared. Observations feed a nativeExecutionGroup; application records add logical-call and stage/section/proposal attribution. Nested validator executions enroll explicitly. Counts preserve terminal coverage and token-usage completeness. - CommandKit's ckconfig and ckprompt integrations supply configuration and prompt commands using Audita's engine construction, including request preparation, validation, and smoke diagnostics.
- Audita edits independently assessable transcript corrections; Notarius accepts typed stage artifacts and has cross-stage dependencies. Audita branches at an individual correction within a proposal response. Notarius must deliberately choose its correction unit rather than assume per-entity branching.
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's reuse identity or persisted validation-provenance requirements.
Evidence: LLM runtime, observations, CLI integrations, and component scope.
Implications For The Remaining Discussion
Item 2's native underlay direction has a working application precedent. Audita also offers evidence for ordinary embedded filesystems and per-request schema selection, while our filesystem and schema decisions remain open.
The subsequent item 3 discussion selects native continuation and accumulated conversation history. Keeping domain validation local does not require keeping conversation assembly or the turn loop local. Item 3 owns the agreed Notarius flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details.
Item 4 now records agreement to use native admission with bounded application workers. Item 5 selects simple application-owned recovery matching and removes the upstream execution-fingerprint prerequisite. Item 6 adopts native execution observations and CommandKit presentation, retaining application attribution, domain outcomes, and durable provenance. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet requirement after reconsidering Notarius's historical behavior.
Verification
Read the production composition, adapter, coordinator, domain repair callbacks, prompt/schema definitions, and focused behavior tests. The following checks passed from the Audita checkout:
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
The subprocess checks exercise assembled configuration, real native continuation messages, operator replacement, final-turn assessment, sibling isolation, repair-disabled behavior, and cancellation using a local fake provider. These checks establish implementation behavior, not live-model quality. No live model calls or application source changes were made. Broader migration decisions and changes to the adoption roadmap remain for discussion.