30 KiB
PromptKit Architecture Discussion
Purpose And Status
Working notepad for reviewing the ownership boundaries in the PromptKit v0.23.2 adoption roadmap. Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. Items 1–3 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. Item 7 records the completed Audita comparison and its implications for the remaining decisions.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not necessarily require application ownership of its implementation. Evaluate remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need.
No application changes have been implemented through this notepad. The adoption roadmap still needs reconciliation with the selected direction, including its adapter-preservation and latest-response-only language. Significant changes to accepted architecture need successor ADRs under the documentation policy. Current boundaries are described in Architecture and LLM Runtime Internals.
1. LLM API And Diagnostic Boundary
Discussion / decision: Direction selected: remove the Notarius PromptKit adapter and make the application fully PromptKit-native. Detailed design and migration consequences remain to be worked through.
Background And Intent
Notarius and related applications, including Audita, originally developed their own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by extracting their duplicated mechanics into a shared library. Notarius adopted an early version and predates many of the library's modern capabilities.
The maintainer clarified that duplication of PromptKit capabilities, including slight behavioral differences, does not reflect an intentional requirement for an independent Notarius implementation. Existing wrappers, tests, and historical decisions are migration evidence, not sufficient reasons to preserve that duplication. Identify actual application needs before choosing what to retain.
Missing reusable capabilities may justify upstream feature requests. The discussion is not constrained to what v0.23.2 can replace immediately; any resulting change to the adoption target remains to be selected.
Design Consequences To Work Through
- Use native PromptKit APIs and contracts for LLM execution instead of preserving a parallel provider-neutral completion API or rebuilding the adapter under another name.
- Review copied request, result, error, inspection, observation, and debug types for removal. Keep application-owned representations only where they serve a concrete application need, with compatibility decisions made explicitly.
- Let PromptKit provide reusable execution mechanics. Determine how Notarius supplies domain inputs, interprets typed outputs, and expresses its acceptance decisions through the resulting design.
- Review application composition, test seams, diagnostic storage, and disclosure handling without assuming they require an LLM abstraction layer.
- Evaluate each upstream gap as a requirement to understand and potentially address in PromptKit, rather than automatically preserving local machinery.
Open questions:
- What application collaborators and helpers remain after the adapter is removed?
- Which durable output or diagnostic contracts need deliberate migration?
- Which existing behavioral differences should disappear through native adoption, and which reflect an application need that should be supplied to PromptKit?
Follow-up: Apply this direction to items 2–6, compare Audita under item 7, and reconcile the adoption roadmap. Record the significant boundary change in a successor ADR and update architecture and implementation documentation when the change lands. Do not treat the remaining topics as decided by this direction.
2. Prompt Assets And Schema Assembly
Discussion / decision: Direction agreed: use native PromptKit declarations as the sole prompt assembly mechanism and remove the duplicate application assembly plumbing. Resolution of the custom read-only filesystem and schemas is explicitly deferred; the next topic may clarify those choices.
Selected direction: Keep the embedded content tree and load its prompt and profile definitions as native PromptKit configuration in the application collection. Use native filesystem contexts and imports to reference shared content directly. Application composition assembles the embedded and operator sources once; modules select prompt IDs, supply domain inputs, and interpret typed results.
The spell extractor illustrates the current duplication: its prompt YAML declares ordered messages and a schema, while a separate Go asset manifest lists files, mounts shared fragments, registers filesystems, and hashes them. The selected direction makes native declarations the sole prompt assembly source.
Candidates for removal:
- Prompt and fallback-profile registration plumbing in
AssetRegistryand individual modules; schema responsibilities remain under review. ModulePromptFS,PromptAssetManifest, shared-name lookup tables, syntheticsharedassetsmounts.- Manual prompt-file enumeration for fingerprints, replaced by captured native prompt identities subject to the broader identity design in item 5.
Keep prompt prose, ordered messages, cache controls, workload profiles, private response schema documents, and domain interpretation. Durable artifact schemas remain separate. Content ownership does not require runtime asset registration.
Native schema paths resolve against the configured schema root, separately from message-file source contexts. PromptKit v0.23.2 loads schemas during preparation and requires self-contained schema documents. A captured prompt hash alone does not establish schema equality. Existing schema metadata in manifests and checkpoint inputs must be deliberately replaced or retired; its presence does not require preserving the complete wrapper system.
Deferred decisions:
- Whether the custom read-only filesystem can be removed after reviewing its remaining responsibilities.
- Schema assembly, root/path selection, wrapper removal, and schema metadata. Supplying the embedded filesystem as a native schema root remains a candidate, not a selected design.
- How schema identity contributes to manifests and checkpoint reuse.
Implementation questions:
- Can native sources represent the existing shared fragments and module assets?
- Which registry, remounting, and hashing helpers could disappear?
- What private schema identity is still needed for checkpoint reuse?
- How do we preserve message ordering and cache controls while simplifying assembly?
Follow-up: Continue with item 3, then revisit the deferred filesystem and schema decisions with any new clarity. Confirm native source contexts for shared fragments and operator replacements. Resolve metadata and reuse identity with items 5–6. Verify the resulting definitions through native preparation and fake-provider execution; filesystem remounting tests can retire with the removed implementation. The agreed direction has not yet been implemented.
3. Retry And Semantic Correction Ownership
Discussion / decision: Direction agreed: adopt Audita's native continuation approach. PromptKit owns captured exchanges, accumulated conversation history, continuation execution, and turn-budget enforcement. Notarius supplies semantic assessment, corrective findings, and acceptance or fallback decisions. Use native PromptKit types directly, consistent with item 1; Audita's adapter and opaque exchange wrapper are not part of the selected direction.
The flow and removal/adaptation direction below are agreed. Begin with one conversation per producer operation and complete replacement model responses. Retain application assessment, safe-fallback decisions, artifact validation, and checkpoint eligibility. Detailed response contracts, budget migration, and the open policy questions below remain to be resolved. No application changes have been implemented.
Agreed Flow
- Prepare the source context, references, and any request-local candidate-handle mapping once for the producer operation. Run its initial native request with exchange capture when continuation is enabled. Capture is independent of detailed debug recording.
- Inspect native structural validation, decode the response, and perform the module's deterministic interpretation and safety checks. Run the configured validation chain on a materialized candidate. A module can identify actionable defects before a candidate is ready for that chain.
- If correction is needed, call
Engine.Coordinatewith the initial exchange and a configuredAdditionalTurnsbudget. The callback selects a native repair prompt and supplies actionable findings as variables. PromptKit retains the original exchange and each subsequent repair turn. - Assess every delivered response, including the last permitted turn. Rebuild the typed candidate from the fixed operation inputs and the revised response; do not repeatedly apply revisions to an already modified artifact. A changed candidate receives the complete configured validation chain again, including validators that approved an earlier version.
- Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
whether to reject, fail, or evaluate an explicitly supported safe fallback.
Native
stoppedandexhaustedoutcomes describe coordination, not acceptance. Cancellation never turns an undelivered response into an accepted candidate.
Start with one conversation per producer operation: a chunk-plan generation, one lane's extraction of one chunk, or one lane's normalization pass. This aligns with Notarius's whole-output validation boundary. Audita's per-proposal branches are useful evidence, but per-entity repair would require its own combination and validation design. It is not necessary to adopt native continuation.
Repair prompts should supply a new user task, with enduring principles in the original system instructions. Avoid replaying the initial task or embedding initial-response-only constraints in system messages. Begin with complete replacement model responses; for normalization, the response can remain a reconciliation plan that Notarius applies to construct the durable artifact. A different repair schema or profile is supported but not yet selected. Keep source context and candidate-handle meanings stable throughout the conversation. Accumulated history increases context and token use across turns; the semantic turn budget should reflect that tradeoff.
Machinery To Remove Or Adapt
| Current machinery | Agreed direction |
|---|---|
CorrectionProtocolSingleResponseV1, raw-response ModelCandidate, and SemanticCorrection |
Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native RunResult / Exchange for continuation state. |
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
runProducerAttempts combining operational, structural, module-requested, and validator-semantic retries |
Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
executeValidationChain's repeated LLM validator invocations |
Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
NormalizeRetry and producerRetryDirective |
Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
CorrectionRequest's hard-coded retry-message construction |
Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
| Correction capability declarations and preparation checks | Retire checks tied to single_response_v1; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
Native transient retries, structural repair, and semantic additional turns remain
distinct controls, but Notarius need not implement all three loops. Existing
producer and validator retries settings cannot be mechanically renamed: their
current budgets cover different behavior. Non-LLM failure handling should be
reviewed separately rather than routed through conversation coordination.
Validation And Safe Fallback Still Have Application Meaning
Keep source grounding, reference resolution, domain invariants, reconciliation plan checks, typed application rules, and candidate isolation. In particular, normalization must still reject unknown or overlapping candidate handles and apply only safe groups. A final fallback can retain safe merges while leaving unsafe groups separate; it must go through configured artifact validation and terminal policy before it can advance. After native structural-repair exhaustion, a supported deterministic fallback can be assessed directly instead of first restarting the same model operation through a feedback-free module retry. Which earlier safe candidate, if any, may survive a later failed turn remains an explicit policy question; do not introduce implicit best-so-far acceptance.
Retain the distinction between a rejection and unavailable validation. Under the
current warn_continue policy, incomplete validation can advance a structurally
valid candidate but prevents checkpoint reuse/publication, including for derived
results. Continuation adoption does not itself change that policy, safe-fallback
eligibility, or durable provenance. Keep raw conversation state in process and
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
PromptKit should own validation and repair of the private model response. Notarius's serialized schema validator currently receives the codec-produced artifact and its artifact schema, which can differ from the model response. Do not remove that boundary merely because both use JSON Schema. Audit actual duplicate response checks individually; artifact validation and Go/domain interpretation are not automatically redundant. Item 2's filesystem and schema assembly decisions remain open.
Evidence: producer attempt loop, validation executor, typed artifact validation, NPC normalizer, semantic reconciliation, and the native continuation evidence in item 7.
Open Implementation And Policy Questions
- Exact repair prompt definitions, response schemas, and profile selection within the agreed complete-replacement approach.
- Migration of producer and validator
retriessettings to native controls, including configuration names, defaults, and compatibility. - Detailed terminal/fallback handling, including whether an earlier safe candidate may survive a later failed turn.
- Failure handling for non-LLM operations and any justified recovery beyond native execution budgets.
- Item 2's deferred filesystem and schema assembly choices, and items 5–6's execution identity and durable bookkeeping design.
Follow-up: Resolve the remaining details above, then reconcile the adoption roadmap. Record the changed architecture in a successor to ADR-0014 and ADR-0016. Migration checks should cover accumulated history, fixed handle mappings, revalidation, final-turn acceptance, native budget exhaustion, safe fallback, validator failure, cancellation, and checkpoint eligibility. Replace tests of the retired message-reconstruction protocol rather than preserving its mechanics.
4. Global Scheduling And Admission
Consider whether backend capacity limits and pipeline worker bounds meet the operational need, or whether a separate cross-backend ceiling remains useful. If it remains useful, consider implementing that reusable mechanism upstream.
Questions:
- What resource or operational budget does each limit protect?
- Should a global permit remain occupied during backend queueing and repair?
- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide admission support?
Discussion / decision: Pending.
Follow-up: Account for the existing global-limit contract. PromptKit v0.23.2's per-backend limits do not directly replace it.
5. Execution Identity And Checkpoint Reuse
Consider moving identity for PromptKit-owned execution semantics upstream. Notarius would combine opaque upstream identities with its source, module, reference, schema, and policy identities and retain checkpoint eligibility and storage decisions.
Questions:
- Which identity can be computed before a stage runs, and which needs a concrete prepared request?
- Which changes should invalidate reuse, and at what granularity?
- Should a conservative application fingerprint be an interim implementation rather than the architectural destination?
- Does this justify upstream work and a later adoption target?
Discussion / decision: Pending.
Follow-up: Review PromptKit's deferred semantic execution-target fingerprint proposal. This capability is not supplied by the v0.23.2 adoption target.
6. Inspection And Execution Bookkeeping
Extend the roadmap's use of native inspection, Run, OnPrepared, observations,
and ExecutionGroup to reconsider surrounding wrappers and copied report types.
Notarius retains stage correlation, artifact provenance, storage, and disclosure
policy; CommandKit is the planned owner of standardized operator presentation.
Questions:
- Which reports and accounting mechanisms can use upstream results directly?
- Which application projections remain necessary for durable contracts?
- Can the PromptKit migration avoid building temporary machinery that the CommandKit adoption would immediately replace?
Discussion / decision: Pending.
Follow-up: Coordinate this topic with item 1 and the roadmap's CommandKit follow-up so report and adapter decisions are made once.
7. Compare Audita's PromptKit Workflow
Review status: Completed on 2026-11-18 against the clean sibling checkout
../audita, revision da5b8913d1206a8e8f0a784b2322760385548a60.
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
not new Notarius contracts. The source links below assume the sibling checkout.
Embedded Defaults And Operator Replacement
- One captured engine is assembled from PromptKit's default catalog, the
embedded Audita underlay, and a higher-priority operator underlay.
FilesystemInputssupplies eleven native prompt definitions incollections.audita: four proposal prompts, four repair prompts, and three validator prompts. - Operator sources are optional
prompts.ymland recursively selected YAML underprompts/, anchored to the application main source's recorded context. Both locations feed one operator underlay and are peers; conflicting peer definitions fail rather than receiving filename-based precedence. - Replacement is by resource identity, not filename. For example, a complete
prompt record with local ID
modules.grammar.repairundercollections.audita.promptsreplacesaudita/modules.grammar.repairwhile other embedded definitions remain available. Native same-ID replacement is whole-record replacement, not recursive merging of individual prompt fields. - Native imports and message-file context are retained. Tests show that an existing engine retains captured message content after a file edit and a new engine sees the edit. They also verify inherited message-file resolution, main-relative discovery independent of the working directory, and peer conflict rejection.
- A diagnostic embedded-only construction can classify a loading error; it does not silently fall back to defaults after an operator-source failure.
Evidence: runtime assembly, source tests, and assembly failure tests.
Prompt Content And Schema Ownership Are Separate
Audita uses ordinary embedded filesystems for prompt and schema assets. Its small prompt registry records IDs and embedded paths; it does not reconstruct Notarius-style per-module virtual prompt trees. Shared instruction fragments are supplied as template variables. Processing supplies the application-owned values for those shared names, but an operator replacement decides whether its template references them at all. They are not forcibly appended to replacement messages.
The processing request independently supplies a complete native output contract:
JSON Schema validation, the schema path selected for that operation, and its
structural-repair budget. The three schemas describe a correction set, a repair
decision, and validator decisions. PromptKit receives their canonical embedded
filesystem via WithSchemaFS. Thus operator control of prompt wording does not
change the processing code's expected response shape. A test replaces a prompt
with a text/no-validation declaration and still exercises the processing
request's JSON-schema repair contract.
Audita still keeps a response-schema key/metadata/hash registry and application decoding checks. Its use of native loading does not prove that every schema wrapper is necessary, nor decide Notarius's deferred schema design. It supplies a concrete option: module-owned response requirements expressed directly through native per-request contracts. Standalone native preview/preparation can use the prompt declaration's contract; processing supplies its own override.
Evidence: prompt registry,
schema registry,
prepareCall and OutputContract in the
processing adapter, and
TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText in
adapter tests.
Native Continuation With Application Assessment
The implemented flow is:
- Generate a section's proposal set, capturing a native immutable exchange when correction repair is enabled. This retains the actual proposal messages and response; it does not require debug recording.
- Apply application validation. An actionable rejected proposal starts its own branch from the original proposal exchange. Unavailable-only findings and no-effect corrections do not start repair. Existing approvals remain separate.
- Call
Engine.Coordinatewith that exchange and an additional-turn budget. Each follow-up selects a distinct repair prompt, repair profile, and repair response schema. The request carries the original proposal reference, current correction, ordered findings, and available before/after text previews. - PromptKit assembles history and progresses the conversation. A second turn retains the first repair instruction and response. A sibling branch starts from the original exchange and does not inherit another branch's repairs. Audita does not rerender the original proposal or manually replay messages.
- Audita decodes each delivered decision. Withdrawal ends the branch without applying that correction. A revision undergoes the full application validation chain again, including validators that approved its earlier version. Approval stops the branch; actionable rejection can request another turn.
- The final permitted response is assessed before exhaustion is determined. Requesting another turn at that boundary does not allocate another logical call. Recoverable provider/invalid-output failures finish the branch as unavailable; fatal errors or parent cancellation interrupt processing. Canceled, undelivered responses are diagnostic evidence only, never approvals.
Transport retries and structural repair operate inside each PromptKit execution; they are separate from the semantic branch's additional-turn budget. Tests show one semantic repair turn containing multiple provider attempts for structural repair. The application retains acceptance, withdrawal, application ordering, and reporting; PromptKit retains conversation and execution mechanics.
Audita's prompt roles support this model deliberately: system instructions carry durable principles, the initial user message carries the initial module task, and the repair prompt supplies a new user assignment. Adopting continuation in Notarius would require reviewing the complete conversation and response-format transitions, rather than mechanically appending to prompts written for fresh replacement requests.
Evidence: native coordination bridge, domain repair flow, wire-level conversation tests, runner repair tests, and prompt guidance.
Other Boundaries And Limits Of The Comparison
- Audita still has
RoleClient, a structured-call interface, an opaque exchange wrapper, and application diagnostic types. Adopt the demonstrated native mechanisms without assuming Notarius should copy those wrappers; item 1's adapter-removal direction remains in force. - Proposal, validator, and repair roles share one engine and its per-backend admission. The runner separately bounds section work. This demonstrates useful separation of application work scheduling and provider capacity, but does not supply Notarius's independent cross-backend ceiling.
- Ordinary calls use
Run,OperationTimeout, andOnPrepared. Observations feed a nativeExecutionGroup; application records add logical-call and stage/section/proposal attribution. Nested validator executions enroll explicitly. Counts preserve terminal coverage and token-usage completeness. - CommandKit's ckconfig and ckprompt integrations supply configuration and prompt commands using Audita's engine construction, including request preparation, validation, and smoke diagnostics.
- Audita edits independently assessable transcript corrections; Notarius accepts typed stage artifacts and has cross-stage dependencies. Audita branches at an individual correction within a proposal response. Notarius must deliberately choose its correction unit rather than assume per-entity branching.
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's reuse identity or persisted validation-provenance requirements.
Evidence: LLM runtime, observations, CLI integrations, and component scope.
Implications For The Remaining Discussion
Item 2's native underlay direction has a working application precedent. Audita also offers evidence for ordinary embedded filesystems and per-request schema selection, while our filesystem and schema decisions remain open.
The subsequent item 3 discussion selects native continuation and accumulated conversation history. Keeping domain validation local does not require keeping conversation assembly or the turn loop local. Item 3 owns the agreed Notarius flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details.
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit integration. Item 5 remains independent upstream/application identity work. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet requirement after reconsidering Notarius's historical behavior.
Verification
Read the production composition, adapter, coordinator, domain repair callbacks, prompt/schema definitions, and focused behavior tests. The following checks passed from the Audita checkout:
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
The subprocess checks exercise assembled configuration, real native continuation messages, operator replacement, final-turn assessment, sibling isolation, repair-disabled behavior, and cancellation using a local fake provider. These checks establish implementation behavior, not live-model quality. No live model calls or application source changes were made. Broader migration decisions and changes to the adoption roadmap remain for discussion.