Files
notarius/docs/roadmap/promptkit-architecture-discussion.md

815 lines
49 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PromptKit Architecture Discussion Record
## Purpose And Status
The decisions in items 1–6 have been incorporated into the
[PromptKit-native architecture migration roadmap](promptkit.md). That roadmap
is the implementation plan and owns current unresolved choices, sequencing,
acceptance criteria, and implementation status.
This document retains the discussion, rationale, and evidence, including item 7's
completed Audita comparison. Item-level questions and follow-ups below record
the discussion as it developed; they are not a second implementation plan.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
capabilities as fully as practical. Application ownership of a policy does not
necessarily require application ownership of its implementation. Evaluate
remaining machinery against this goal, including opportunities for upstream
features where PromptKit does not yet meet an application need.
No application changes have been implemented through this discussion or the
roadmap rewrite. Significant changes to accepted architecture need successor ADRs
under the [documentation policy](../policy/documentation.md#architecture-decision-records).
Current boundaries are described in [Architecture](../policy/architecture.md)
and [LLM Runtime Internals](../internal/llm.md).
## 1. LLM API And Diagnostic Boundary
**Discussion / decision:** Direction agreed: remove the Notarius PromptKit
adapter and compose the application directly from native PromptKit execution,
typed artifact operations, and pipeline orchestration. A small application
function connects generation to configured validation and acceptance. The design
below incorporates items 2–6 and the Audita review; exact signatures and migration
details remain open. No application changes have been implemented.
### Background And Intent
Notarius and related applications, including Audita, originally developed their
own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by
extracting their duplicated mechanics into a shared library. Notarius adopted
an early version and predates many of the library's modern capabilities.
The maintainer clarified that duplication of PromptKit capabilities, including
slight behavioral differences, does not reflect an intentional requirement for
an independent Notarius implementation. Existing wrappers, tests, and historical
decisions are migration evidence, not sufficient reasons to preserve that
duplication. Identify actual application needs before choosing what to retain.
Missing reusable capabilities may justify upstream feature requests. The
discussion is not constrained to what v0.23.2 can replace immediately; any
resulting change to the adoption target remains to be selected.
### Application Structure
| Part | Responsibility |
| --- | --- |
| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. |
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
| Application records and CLI | Store artifacts and checkpoints, attach pipeline meaning to execution facts, and present results through CommandKit. |
Engine construction becomes ordinary application wiring using item 2's embedded
defaults and operator sources. Profile selection is expressed through native
requests. The current `internal/framework/llm` package can shrink to engine
assembly and focused helpers; whether those warrant a separate package remains
an organizational choice, not a reason to retain an adapter.
Use native requests, results, exchanges, errors, and observers directly in
model-executing application code. Pure domain models, codecs, source-reference
resolution, and deterministic transformations need no PromptKit dependency.
Retain application interfaces for extracting artifacts, validating typed
candidates, encoding output, and loading checkpoints. Deterministic stages
continue to operate without conversation machinery.
### Connect Modules To Pipeline Assessment
The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
prepares inputs, invokes the private completion interface, interprets a response,
and returns a copied raw model candidate to the
[producer retry loop](../../internal/framework/pipeline/producer_attempts.go).
Replace that arrangement with direct collaboration between module interpretation,
pipeline assessment, and native execution:
1. The module prepares source material, references, and any candidate-handle
mappings once, and constructs the native initial request.
2. Application execution calls `Engine.Run`, capturing an exchange when
continuation is enabled.
3. The module interprets the structurally valid response into a typed candidate
or actionable domain findings.
4. The pipeline runs configured candidate validation and decides whether to
accept, request correction, or apply terminal policy.
5. If correction is needed, native `Coordinate` invokes the same interpretation
and assessment for revised responses. The module supplies the repair request;
PromptKit maintains history and enforces the turn budget.
A small shared application function can connect these responsibilities: the
module supplies native request construction and response interpretation, while
the pipeline supplies configured validation and terminal policy. It uses native
PromptKit types and returns a typed application outcome. It does not introduce
another completion API, conversation protocol, or retry engine.
Item 3 owns the detailed continuation and assessment rules, including final-turn
assessment, failure handling, complete replacement responses, and safe fallback.
For normalization, a revised reconciliation plan is applied to the original
inputs to produce a fresh candidate. LLM-backed validators can call the same
native engine and return application validation findings; they do not need the
producer's conversation machinery.
### Removal, Records, And Testing
Retire `StructuredLLMClient`, the private structured-completion request/response
family, adapter conversions, and opaque correction transport. Items 3, 4, and 6
specify removal of the outer LLM retry loop, scheduler, debug completion wrapper,
profile-recording plumbing, and duplicate execution accounting. Do not recreate
that wrapper stack around native calls.
Attach pipeline scope through native callbacks and observers. Use native
`ExecutionGroup` totals and small application projections for manifests and
optional debug artifacts, following item 6. Checkpoint matching and durable
acceptance provenance remain the application responsibilities selected in item 5.
Audita demonstrates shared engine assembly, native continuation, and CommandKit
integration. Its `RoleClient`, structured-output interface, and opaque exchange
wrapper are not part of this design; item 7 records the comparison's evidence
and limits.
Test domain interpretation and assessment directly. Exercise model workflows
with a real PromptKit engine and its injectable client or a fake HTTP provider.
Pipeline orchestration can still use fake stage implementations. Testing does
not require preserving a private completion interface.
### Remaining Implementation Work
- Select concrete module/pipeline function signatures that connect request
construction, interpretation, configured validation, and terminal disposition.
- Choose the location of engine assembly and focused application helpers without
introducing a replacement LLM service layer.
- Resolve item 2's deferred filesystem and schema choices and the remaining
migration details recorded under items 3–6, including durable compatibility.
- Evaluate any remaining upstream gap against an actual application requirement,
rather than preserving historical machinery by default.
**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6.
Record the significant boundary change in successor ADRs. Update architecture
and implementation documentation when the change lands, explicitly replacing
the current transport-neutral completion boundary with PromptKit as the execution
dependency of application workflows. Keep domain operations and application
outcomes as the meaningful Notarius-owned boundaries.
## 2. Prompt Assets And Schema Assembly
**Discussion / decision:** Direction agreed: use native PromptKit declarations
as the sole prompt assembly mechanism and remove the duplicate application
assembly plumbing. Resolution of the custom read-only filesystem and schemas
is explicitly deferred; the next topic may clarify those choices.
**Selected direction:** Keep the embedded content tree and load its prompt and
profile definitions as native PromptKit configuration in the application
collection. Use native filesystem contexts and imports to reference shared
content directly. Application composition assembles the embedded and operator
sources once; modules select prompt IDs, supply domain inputs, and interpret
typed results.
The spell extractor illustrates the current duplication: its prompt YAML
declares ordered messages and a schema, while a separate Go asset manifest
lists files, mounts shared fragments, registers filesystems, and hashes them.
The selected direction makes native declarations the sole prompt assembly source.
Candidates for removal:
- Prompt and fallback-profile registration plumbing in `AssetRegistry` and
individual modules; schema responsibilities remain under review.
- `ModulePromptFS`, `PromptAssetManifest`, shared-name lookup tables, synthetic
`sharedassets` mounts.
- Manual prompt-file enumeration for checkpoint fingerprints, retired under
item 5's simpler reuse policy. Native prompt hashes remain available for
inspection and diagnostics.
Keep prompt prose, ordered messages, cache controls, workload profiles, private
response schema documents, and domain interpretation. Durable artifact schemas
remain separate. Content ownership does not require runtime asset registration.
Native schema paths resolve against the configured schema root, separately from
message-file source contexts. PromptKit v0.23.2 loads schemas during preparation
and requires self-contained schema documents. A captured prompt hash alone does
not establish schema equality. Existing schema metadata in manifests and
checkpoint inputs must be deliberately replaced or retired; its presence does
not require preserving the complete wrapper system.
Deferred decisions:
- Whether the custom read-only filesystem can be removed after reviewing its
remaining responsibilities.
- Schema assembly, root/path selection, wrapper removal, and schema metadata.
Supplying the embedded filesystem as a native schema root remains a candidate,
not a selected design.
- Schema metadata for manifests and artifact compatibility. Item 5 settles that
checkpoint matching does not require exact schema identity.
Implementation questions:
- Can native sources represent the existing shared fragments and module assets?
- Which registry, remounting, and hashing helpers could disappear?
- What schema metadata remains useful for inspection and artifact loading under
item 5's simpler reuse policy?
- How do we preserve message ordering and cache controls while simplifying
assembly?
**Follow-up:** Continue with item 3, then revisit the deferred filesystem and
schema decisions with any new clarity. Confirm native source contexts for shared
fragments and operator replacements. Apply item 5's reuse policy and resolve
remaining metadata with item 6. Verify the resulting definitions through native
preparation and fake-provider execution; filesystem remounting tests can retire
with the removed implementation. The agreed direction has not yet been implemented.
## 3. Retry And Semantic Correction Ownership
**Discussion / decision:** Direction agreed: adopt Audita's native continuation
approach. PromptKit owns captured exchanges, accumulated conversation history,
continuation execution, and turn-budget enforcement. Notarius supplies semantic
assessment, corrective findings, and acceptance or fallback decisions. Use native
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
exchange wrapper are not part of the selected direction.
The flow and removal/adaptation direction below are agreed. Begin with one
conversation per producer operation and complete replacement model responses.
Retain application assessment, safe-fallback decisions, artifact validation,
and checkpoint eligibility. Detailed response contracts, budget migration, and
the open policy questions below remain to be resolved. No application changes
have been implemented.
### Agreed Flow
1. Prepare the source context, references, and any request-local candidate-handle
mapping once for the producer operation. Run its initial native request with
exchange capture when continuation is enabled. Capture is independent of
detailed debug recording.
2. Inspect native structural validation, decode the response, and perform the
module's deterministic interpretation and safety checks. Run the configured
validation chain on a materialized candidate. A module can identify actionable
defects before a candidate is ready for that chain.
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
and a configured `AdditionalTurns` budget. The callback selects a native
repair prompt and supplies actionable findings as variables. PromptKit retains
the original exchange and each subsequent repair turn.
4. Assess every delivered response, including the last permitted turn. Rebuild
the typed candidate from the fixed operation inputs and the revised response;
do not repeatedly apply revisions to an already modified artifact. A changed
candidate receives the complete configured validation chain again, including
validators that approved an earlier version.
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
whether to reject, fail, or evaluate an explicitly supported safe fallback.
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
Cancellation never turns an undelivered response into an accepted candidate.
Start with one conversation per producer operation: a chunk-plan generation,
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
with Notarius's whole-output validation boundary. Audita's per-proposal branches
are useful evidence, but per-entity repair would require its own combination and
validation design. It is not necessary to adopt native continuation.
Repair prompts should supply a new user task, with enduring principles in the
original system instructions. Avoid replaying the initial task or embedding
initial-response-only constraints in system messages. Begin with complete
replacement model responses; for normalization, the response can remain a
reconciliation plan that Notarius applies to construct the durable artifact.
A different repair schema or profile is supported but not yet selected. Keep
source context and candidate-handle meanings stable throughout the conversation.
Accumulated history increases context and token use across turns; the semantic
turn budget should reflect that tradeoff.
### Machinery To Remove Or Adapt
| Current machinery | Agreed direction |
| --- | --- |
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
Native transient retries, structural repair, and semantic additional turns remain
distinct controls, but Notarius need not implement all three loops. Existing
producer and validator `retries` settings cannot be mechanically renamed: their
current budgets cover different behavior. Non-LLM failure handling should be
reviewed separately rather than routed through conversation coordination.
### Validation And Safe Fallback Still Have Application Meaning
Keep source grounding, reference resolution, domain invariants, reconciliation
plan checks, typed application rules, and candidate isolation. In particular,
normalization must still reject unknown or overlapping candidate handles and
apply only safe groups. A final fallback can retain safe merges while leaving
unsafe groups separate; it must go through configured artifact validation and
terminal policy before it can advance. After native structural-repair exhaustion,
a supported deterministic fallback can be assessed directly instead of first
restarting the same model operation through a feedback-free module retry. Which
earlier safe candidate, if any, may survive a later failed turn remains an explicit
policy question; do not introduce implicit best-so-far acceptance.
Retain the distinction between a rejection and unavailable validation. Under the
current `warn_continue` policy, incomplete validation can advance a structurally
valid candidate but prevents checkpoint reuse/publication, including for derived
results. Continuation adoption does not itself change that policy, safe-fallback
eligibility, or durable provenance. Keep raw conversation state in process and
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
PromptKit should own validation and repair of the private model response.
Notarius's serialized schema validator currently receives the codec-produced
artifact and its artifact schema, which can differ from the model response.
Do not remove that boundary merely because both use JSON Schema. Audit actual
duplicate response checks individually; artifact validation and Go/domain
interpretation are not automatically redundant. Item 2's filesystem and schema
assembly decisions remain open.
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
[validation executor](../../internal/framework/pipeline/validation_executor.go),
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
and the native continuation evidence in item 7.
### Open Implementation And Policy Questions
- Exact repair prompt definitions, response schemas, and profile selection within
the agreed complete-replacement approach.
- Migration of producer and validator `retries` settings to native controls,
including configuration names, defaults, and compatibility.
- Detailed terminal/fallback handling, including whether an earlier safe
candidate may survive a later failed turn.
- Failure handling for non-LLM operations and any justified recovery beyond
native execution budgets.
- Item 2's deferred filesystem and schema assembly choices, item 5's remaining
checkpoint implementation details, and item 6's durable bookkeeping design.
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
roadmap. Record the changed architecture in a successor to
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
Migration checks should cover accumulated history, fixed handle mappings,
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
validator failure, cancellation, and checkpoint eligibility. Replace tests of
the retired message-reconstruction protocol rather than preserving its mechanics.
## 4. Global Scheduling And Admission
**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and
retire the separate cross-backend ceiling. Rely on PromptKit's native backend
admission and capacity limits while keeping bounded Notarius pipeline workers.
This deliberately changes the existing global-limit contract; reproducing that
ceiling in PromptKit is not a migration prerequisite. No application changes
have been implemented.
### Agreed Ownership
| Concern | Owner |
| --- | --- |
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
| Application disposition after admission or execution failure | Notarius application policy |
For limited backends, PromptKit v0.23.2 bounds active model attempts by
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
An operation holds admission
across preparation, retries, and structural repair, while active permits cover
individual model attempts and are released during retry waits. A completed
continuation turn releases admission before application assessment, so its
callback can invoke an LLM validator using the same backend.
The current Notarius wrapper holds a global permit around the complete
structured call, including backend waiting and repairs. Removing it eliminates
that additional queue and avoids making independent backends share those
permits. Independent native backend limits may allow aggregate concurrency above
the former global ceiling; that is an accepted behavior change. Pipeline worker
bounds remain application work controls, not a replacement global LLM ceiling.
### Removal And Configuration Consequences
- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom
queue/cancellation/permit bookkeeping.
- Retire `concurrency.total_llm`. Give pipeline worker settings an independent
default and remove the extraction-worker maximum derived from `total_llm`.
- Use one shared execution engine for producers, validators, and repair turns.
Native capacity belongs to an engine and selected backend ID; separate engine
instances have independent pools.
- Configure native limits at the intended backend capacity boundary. Profiles
selecting the same backend ID share its pool; separate IDs pointing to the
same endpoint do not automatically share capacity. Endpoint-only profiles and
backends without a positive limit remain unrestricted by backend admission.
- Handle native `ErrCapacityExceeded` as an admission failure through application
policy, consistently with item 3's removal of blanket execution retries.
- Retire scheduler implementation tests with the removed code. Retain focused
integration checks for shared native admission across roles, independent
backends, bounded pipeline work, queue exhaustion, and cancellation.
### Remaining Details
Select independent worker defaults, native backend/queue defaults where the
application supplies them, and the configuration migration treatment of the
retired field. Specify how admission failures are reported and handled for each
application role. These details do not reopen the selected ownership boundary.
A future demonstrated need for capacity shared across several backend IDs could
justify an upstream PromptKit feature request. No such feature is required for
this agreed design, and it does not justify retaining the Notarius scheduler.
Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
[scheduler](../../internal/framework/llm/scheduler.go),
[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and
[native capacity behavior](../../../promptkit/docs/internal/capacity.md).
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
and record the changed ownership in an ADR. Update architecture, configuration,
operations, and implementation documentation when the migration lands. Continue
with item 5's execution identity and checkpoint reuse discussion.
## 5. Execution Identity And Checkpoint Reuse
**Discussion / decision:** Direction agreed: resuming preserves successful stage
results for the same logical work and data; explicit recomputation applies
changed execution behavior. Use a simple Notarius-owned matching policy instead
of comprehensive execution fingerprints. No application changes have been
implemented.
### Purpose And Matching Rule
Checkpoints preserve a successful producer's result when a dependent stage fails.
After fixing the failed stage, an operator should be able to retry it without
paying to rerun the successful producer. Matching every configuration parameter
or implementation revision would undermine that recovery purpose.
The agreed conceptual key is:
```text
logical stage identity + input-data identity + prompt ID, where applicable
```
Scope the checkpoint to the relevant step/lane/stage and artifact kind so that
shared prompt IDs do not mix unrelated results. These are logical identifiers,
not hashes of implementations or complete configuration. Deterministic stages
need no synthetic prompt identity.
Input data includes the source or chunk being processed, reference material,
and upstream artifacts actually consumed. Matching only the original source
would miss changes to generated dependencies. Content digests can compactly
identify these inputs and check stored payload integrity without requiring an
exact fingerprint of the execution environment.
| Change | Reuse behavior |
| --- | --- |
| Fix the implementation of a later failed stage | Preserve earlier successful stages with matching work and inputs. |
| Edit prompt text while retaining its ID | Reuse; explicit recomputation applies the edit. |
| Change model, generation settings, or implementation/validation policy | Reuse otherwise compatible completed work; explicit recomputation applies the change. |
| Change the prompt ID or data consumed by the stage | No matching checkpoint under this rule. |
| Explicitly request recomputation | Recompute the selected work and its dependents. |
The existing controls are `--resume` and `--recompute-step`, documented in the
[CLI reference](../cli.md#run) and
[selective recompute workflow](../operations.md#selective-recompute). Their
current strict compatibility checks still need migration to this agreed policy.
The accepted tradeoff is that a checkpoint may preserve a successful result
produced by code, prompts, or settings that have since improved.
### Matching, Loading, And Provenance
A matching key selects a checkpoint; it does not make an unreadable or
incompatible artifact usable. Retain completion/acceptance status, integrity
checks, and the ability to decode and consume the stored artifact. Exact schema
hash equality is not required when the artifact remains compatible.
Only accepted, completely validated results are reusable. Rejected, structurally
invalid, or validation-incomplete results and their ineligible derivatives
remain outside reusable state. Acceptance on resume refers to the earlier
execution: do not automatically rerun semantic validators merely because their
configuration changed, or claim the artifact passed the new validation policy.
Preserve original provenance; explicit recomputation applies changed behavior.
Useful execution metadata can remain available for explanation without becoming
a matching requirement. Neither complete conversation history nor native
prepared-execution handles are needed to locate and reuse a completed artifact.
The separate source-addressed canonical chunk-plan policy remains distinct.
### Simplification And Ownership
- Remove the broad PromptKit configuration fingerprint from checkpoint matching,
including effective-configuration parsing and whole-catalog enumeration.
- Remove library/catalog versions, model settings, prompt-content hashes, and
component-policy fingerprints as automatic invalidators under this policy.
- Keep straightforward input/dependency identity, successful-stage records,
artifact loading checks, storage, and explicit recomputation in Notarius.
- Keep PromptKit hashes available for native inspection and diagnostics without
making them checkpoint prerequisites.
Comprehensive upstream execution identity is no longer required for this
design. It does not require a later PromptKit release or a temporary conservative
configuration fingerprint. Item 5 supersedes the earlier proposal to make an
upstream semantic-equality feature part of the migration's checkpoint design.
### Remaining Implementation Work
Define the concrete logical stage/prompt identifiers and consumed-input
representation. Revise checkpoint lookup and storage scoping so that an
invocation-wide pipeline/configuration digest cannot still prevent reuse of
successful predecessors. Decide how existing checkpoint records transition to
the simpler format and how incompatible payloads are reported.
**Follow-up:** Reconcile the adoption roadmap and record the changed reuse
contract in an ADR. Update current-behavior documentation when implemented.
Verify recovery after a downstream fix, reuse across prompt/settings changes,
misses after consumed-data changes, explicit recomputation, and rejection of
unusable or unaccepted stored artifacts. Continue with item 6.
## 6. Inspection And Execution Bookkeeping
**Discussion / decision:** Direction agreed: use native PromptKit inspection
and execution facts, with CommandKit owning standardized operator presentation.
Remove duplicate LLM reporting and accounting machinery. Notarius retains
application context, domain decisions, durable artifacts, and their provenance.
No application changes have been implemented.
### Ownership
| Responsibility | Owner |
| --- | --- |
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
### Native Inspection And Operator Presentation
Remove the profile-inspector wrapper, copied effective-model-parameter types,
and local reconstruction of upstream configuration. Use native inspection for
configured resources and native prepared/result data for the target actually
executed, including request overrides. Profile inspection can describe a partial
configured profile; it is not evidence of the completed execution target.
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
preview, and opt-in smoke commands. Notarius supplies configuration/engine
loaders and application request builders. Avoid building temporary custom
commands or report machinery that the planned CommandKit adoption would replace.
Retain application pipeline preflight: resolve modules, artifact compatibility,
references, and known prompt/profile selections. Use native definition checks
without simulating stage execution or requiring provider credentials. Concrete
request preparation remains the place to check the completed target, inputs,
schema, and credential requirements.
CommandKit supplies formatting, visibility, progress presentation, and delivery
error handling. Notarius supplies report content, domain outcomes, and exit-status
mapping. Preserve deliberate machine-output contracts rather than treating every
receipt as a generic formatted report.
### Native Execution Facts
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
for provenance or optional debug capture. Remove unconditional
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
- Use native observations for terminal execution facts, including failures that
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
preparation or admission failure; it cannot serve as a completion record.
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
Remove the private debug completion wrapper and duplicate synthetic call-ID
machinery where native identity and application attribution suffice.
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
cloning, and independent attempt/usage accumulators. Use native types internally
and project selected fields only at an application reporting or durable boundary.
- Replace the adapter's `LLMProfileRecorder` and provider-forwarding plumbing with
the application projection needed for actual-run provenance. Do not substitute
a catalog inventory for the profiles and targets actually used.
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
initial run, continuation runs, and nested validator runs as applicable.
Preserve terminal-coverage and token-usage completeness; unknown usage is not
zero. Do not count both attempt observations and terminal aggregates, or add
overlapping groups as though they were disjoint work.
Application execution code should attach scope and observers, run and assess the
native result, and record its disposition and execution references. This does
not require a replacement general-purpose LLM interface. A successful native
execution can still produce a domain-rejected candidate; provider success and
application acceptance remain separate facts.
### Application Records And Durable Boundaries
Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an
artifact, which validators assessed it, whether continuation or a safe fallback
was used, and whether the artifact was generated now or reused. PromptKit cannot
infer that meaning from provider execution alone.
Under item 5, a reused artifact retains its original execution and acceptance
provenance. The current invocation records the reuse decision without counting
the earlier execution's token usage as current work or claiming new validation.
Retain small, deliberate application projections for the
[published manifest](../integrations/json-output.md) and
[run-result receipt](../integrations/run-result.md), with explicit compatibility
decisions for changed fields. Durable contracts should not become raw upstream
type dumps or require parallel runtime representations of every native result.
Keep application debug artifact placement, opt-in capture, retention, and
disclosure handling. Native callbacks supply execution data; they do not own
Notarius's storage policy. Detailed prepared messages belong in the selected
debug capture path, while ordinary provenance can retain only the needed fields.
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
### Remaining Implementation Work
Select the minimal application scope and durable fields, observer/group
enrollment, and compatibility treatment for existing manifests, receipts, and
debug artifacts. Coordinate command migration with the roadmap's CommandKit
follow-up. The core ownership change needs no new PromptKit capability; any
remaining upstream request should identify a concrete unmet requirement.
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
significant boundary changes in successor ADRs. Update current-behavior
documentation when implemented. Verify failure reporting without a result,
continuation/validator accounting, unknown usage, actual-target provenance,
checkpoint reuse accounting, and optional debug capture during implementation.
## 7. Compare Audita's PromptKit Workflow
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
not new Notarius contracts. The source links below assume the sibling checkout.
### Embedded Defaults And Operator Replacement
- One captured engine is assembled from PromptKit's default catalog, the
embedded Audita underlay, and a higher-priority operator underlay.
`FilesystemInputs` supplies eleven native prompt definitions in
`collections.audita`: four proposal prompts, four repair prompts, and three
validator prompts.
- Operator sources are optional `prompts.yml` and recursively selected YAML
under `prompts/`, anchored to the application main source's recorded context.
Both locations feed one operator underlay and are peers; conflicting peer
definitions fail rather than receiving filename-based precedence.
- Replacement is by resource identity, not filename. For example, a complete
prompt record with local ID `modules.grammar.repair` under
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
other embedded definitions remain available. Native same-ID replacement is
whole-record replacement, not recursive merging of individual prompt fields.
- Native imports and message-file context are retained. Tests show that an
existing engine retains captured message content after a file edit and a new
engine sees the edit. They also verify inherited message-file resolution,
main-relative discovery independent of the working directory, and peer
conflict rejection.
- A diagnostic embedded-only construction can classify a loading error; it
does not silently fall back to defaults after an operator-source failure.
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
### Prompt Content And Schema Ownership Are Separate
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
small prompt registry records IDs and embedded paths; it does not reconstruct
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
supplied as template variables. Processing supplies the application-owned values
for those shared names, but an operator replacement decides whether its template
references them at all. They are not forcibly appended to replacement messages.
The processing request independently supplies a complete native output contract:
JSON Schema validation, the schema path selected for that operation, and its
structural-repair budget. The three schemas describe a correction set, a repair
decision, and validator decisions. PromptKit receives their canonical embedded
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
change the processing code's expected response shape. A test replaces a prompt
with a text/no-validation declaration and still exercises the processing
request's JSON-schema repair contract.
Audita still keeps a response-schema key/metadata/hash registry and application
decoding checks. Its use of native loading does not prove that every schema
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
a concrete option: module-owned response requirements expressed directly through
native per-request contracts. Standalone native preview/preparation can use the
prompt declaration's contract; processing supplies its own override.
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
`prepareCall` and `OutputContract` in the
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
### Native Continuation With Application Assessment
The implemented flow is:
1. Generate a section's proposal set, capturing a native immutable exchange when
correction repair is enabled. This retains the actual proposal messages and
response; it does not require debug recording.
2. Apply application validation. An actionable rejected proposal starts its own
branch from the original proposal exchange. Unavailable-only findings and
no-effect corrections do not start repair. Existing approvals remain separate.
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
Each follow-up selects a distinct repair prompt, repair profile, and repair
response schema. The request carries the original proposal reference, current
correction, ordered findings, and available before/after text previews.
4. PromptKit assembles history and progresses the conversation. A second turn
retains the first repair instruction and response. A sibling branch starts
from the original exchange and does not inherit another branch's repairs.
Audita does not rerender the original proposal or manually replay messages.
5. Audita decodes each delivered decision. Withdrawal ends the branch without
applying that correction. A revision undergoes the full application validation
chain again, including validators that approved its earlier version. Approval
stops the branch; actionable rejection can request another turn.
6. The final permitted response is assessed before exhaustion is determined.
Requesting another turn at that boundary does not allocate another logical
call. Recoverable provider/invalid-output failures finish the branch as
unavailable; fatal errors or parent cancellation interrupt processing.
Canceled, undelivered responses are diagnostic evidence only, never approvals.
Transport retries and structural repair operate inside each PromptKit execution;
they are separate from the semantic branch's additional-turn budget. Tests show
one semantic repair turn containing multiple provider attempts for structural
repair. The application retains acceptance, withdrawal, application ordering,
and reporting; PromptKit retains conversation and execution mechanics.
Audita's prompt roles support this model deliberately: system instructions carry
durable principles, the initial user message carries the initial module task,
and the repair prompt supplies a new user assignment. Adopting continuation in
Notarius would require reviewing the complete conversation and response-format
transitions, rather than mechanically appending to prompts written for fresh
replacement requests.
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
and [prompt guidance](../../../audita/docs/internal/prompts.md).
### Other Boundaries And Limits Of The Comparison
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
wrapper, and application diagnostic types. Adopt the demonstrated native
mechanisms without assuming Notarius should copy those wrappers; item 1's
adapter-removal direction remains in force.
- Proposal, validator, and repair roles share one engine and its per-backend
admission. The runner separately bounds section work. This demonstrates useful
separation of application work scheduling and provider capacity. Item 4 now
selects that ownership and retires Notarius's independent cross-backend ceiling.
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
feed a native `ExecutionGroup`; application records add logical-call and
stage/section/proposal attribution. Nested validator executions enroll
explicitly. Counts preserve terminal coverage and token-usage completeness.
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
commands using Audita's engine construction, including request preparation,
validation, and smoke diagnostics.
- Audita edits independently assessable transcript corrections; Notarius accepts
typed stage artifacts and has cross-stage dependencies. Audita branches at an
individual correction within a proposal response. Notarius must deliberately
choose its correction unit rather than assume per-entity branching.
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
reuse identity or persisted validation-provenance requirements.
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
[observations](../../../audita/internal/framework/llm/observations.go),
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
[component scope](../../../audita/docs/internal/overview.md).
### Implications For The Remaining Discussion
Item 2's native underlay direction has a working application precedent. Audita
also offers evidence for ordinary embedded filesystems and per-request schema
selection, while our filesystem and schema decisions remain open.
The subsequent item 3 discussion selects native continuation and accumulated
conversation history. Keeping domain validation local does not require keeping
conversation assembly or the turn loop local. Item 3 owns the agreed Notarius
flow and the remaining repair-prompt, response-contract, and policy details;
the comparison alone did not settle those details.
Item 4 now records agreement to use native admission with bounded application
workers. Item 5 selects simple application-owned recovery matching and removes
the upstream execution-fingerprint prerequisite. Item 6 adopts native execution
observations and CommandKit presentation, retaining application attribution,
domain outcomes, and durable provenance.
The two requested capabilities already work with v0.23.2; no new PromptKit
feature is needed merely to provide replaceable defaults or domain-directed
continuation. Any remaining upstream request should identify the specific unmet
requirement after reconsidering Notarius's historical behavior.
### Verification
Read the production composition, adapter, coordinator, domain repair callbacks,
prompt/schema definitions, and focused behavior tests. The following checks
passed from the Audita checkout:
```sh
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
```
The subprocess checks exercise assembled configuration, real native continuation
messages, operator replacement, final-turn assessment, sibling isolation,
repair-disabled behavior, and cancellation using a local fake provider. These
checks establish implementation behavior, not live-model quality. No live model
calls or application source changes were made. Broader migration decisions and
changes to the adoption roadmap remain for discussion.