Consolidate migration planning into the master roadmap
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -1,30 +1,18 @@
|
||||
# Future Work
|
||||
|
||||
Current Notarius behavior is documented in the canonical README, CLI,
|
||||
configuration, operations, internal, and integration docs. This roadmap records
|
||||
future work only. Items are ordered roughly by current value and specificity,
|
||||
not as committed release dates.
|
||||
|
||||
## Near-Term Validation And LLM Reliability
|
||||
|
||||
PromptKit now owns structural output repair within one completion. Notarius
|
||||
owns stage candidates, validator chains, semantic rejection policy, bounded
|
||||
feedback-aware stage retries, validation provenance, and reusable-state
|
||||
eligibility, and the separation of actionable process warnings from quality
|
||||
diagnostics. The remaining near-term work applies those completed foundations
|
||||
to domain review and empirical evaluation.
|
||||
|
||||
Near-term reliability work should now be selected from the concrete evaluation
|
||||
and extension opportunities below. The retry, validation, and subprocess
|
||||
foundations described above are implemented current behavior.
|
||||
configuration, operations, internal, and integration docs. This backlog records
|
||||
future work outside the [native migration roadmap](promptkit.md), which owns
|
||||
PromptKit/ConfigKit/CommandKit adoption, continuation, admission, cache, output,
|
||||
and debug decisions. Items below are ordered roughly by value and specificity,
|
||||
not as committed release dates or prerequisites for that migration.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
- Evaluate ordinary extraction retries and the completed normalization path
|
||||
against a human-reviewed transcript set before and after adopting the shared
|
||||
PromptKit repair and Notarius validation-retry policies above.
|
||||
- Evaluate extraction, normalization, and correction quality against a
|
||||
human-reviewed transcript set before and after the native migration.
|
||||
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
|
||||
validator, and normalizer development. Treat model-quality review as an
|
||||
iterative human evaluation aid, not a deterministic correctness gate.
|
||||
@@ -46,8 +34,8 @@ The sections below keep broader extensions deferred.
|
||||
### Large-Collection Semantic Reconciliation
|
||||
|
||||
- Evaluate deterministic candidate blocking only after representative registry
|
||||
inputs exceed the active roadmap's bounded single-request limits. Blocking
|
||||
should use cheap, explainable signals to form plausible comparison sets while
|
||||
inputs exceed practical single-request capacity. Blocking should use cheap,
|
||||
explainable signals to form plausible comparison sets while
|
||||
preserving the possibility that a duplicate appears outside a lexical name
|
||||
match.
|
||||
- Define correctness for candidates that appear in more than one block,
|
||||
@@ -61,16 +49,15 @@ The sections below keep broader extensions deferred.
|
||||
|
||||
### Operator-Selected Semantic Policies
|
||||
|
||||
- Consider allowing an operator to select an approved semantic-policy prompt
|
||||
for a typed reconciliation module without replacing the shared protocol,
|
||||
response schema, or deterministic safety rules.
|
||||
- Define the trusted asset source, configuration syntax, compatibility checks,
|
||||
startup validation, provenance, prompt fingerprinting, checkpoint effects,
|
||||
and support boundary before exposing the option.
|
||||
- Prefer selection among registered, typed-policy-compatible prompt assets over
|
||||
arbitrary filesystem prompt paths. Do not add this flexibility until an
|
||||
operator workflow requires it; artifact-family-owned policy remains simpler
|
||||
and safer for the initial implementation.
|
||||
- Consider a dedicated semantic-policy selection option for a typed
|
||||
reconciliation module when a concrete operator workflow needs more than the
|
||||
migration's [native prompt replacement](promptkit.md#configuration-sources-and-prompt-assets).
|
||||
- Use native prompt IDs and sources while retaining typed interpretation and
|
||||
deterministic safety rules. Define any additional configuration and domain
|
||||
compatibility checks from that workflow; do not recreate asset registration,
|
||||
prompt fingerprints, or checkpoint dependency tracking.
|
||||
- Reuse the selected [accepted-artifact cache policy](promptkit.md#accepted-artifact-cache)
|
||||
when evaluating policy changes and the need for explicit refresh.
|
||||
|
||||
### Broader Reconciliation Inputs And Module Selection
|
||||
|
||||
@@ -89,9 +76,8 @@ The sections below keep broader extensions deferred.
|
||||
artifact.
|
||||
- Add production LLM-backed validators only when a concrete review policy
|
||||
benefits from model judgment and deterministic checks are insufficient.
|
||||
- Add validator diagnostics and timing summaries if operators need more detail
|
||||
than the current [durable output bundle](../integrations/json-output.md)
|
||||
provides.
|
||||
- Consider additional validator diagnostic views only if a concrete need remains
|
||||
after the selected [native observations and debug capture](promptkit.md#inspection-and-execution-bookkeeping).
|
||||
- Add validator compatibility metadata if deployments need config-time proof
|
||||
that a validator is suitable for a particular stage, module, or artifact
|
||||
kind.
|
||||
@@ -133,10 +119,11 @@ occurred.
|
||||
|
||||
Define compatibility and migration policy before generated-reference chains
|
||||
must span multiple schema versions or long-lived historical artifacts. The
|
||||
policy should address stable identifier semantics, which schema changes permit
|
||||
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
|
||||
producer or all dependents must be recomputed. Do not add a general migration
|
||||
framework until an actual contract change requires one.
|
||||
policy should address stable identifier semantics, when an older artifact may
|
||||
be decoded or adapted, and when consumers need regenerated artifacts. For cached
|
||||
artifacts, use the [selected reuse policy](promptkit.md#accepted-artifact-cache)
|
||||
and explicit refresh where needed. Do not add a general migration framework
|
||||
until an actual contract change requires one.
|
||||
|
||||
### Artifact-family-oriented physical packaging
|
||||
|
||||
@@ -170,7 +157,6 @@ section only after a concrete workflow, contract, and priority emerge.
|
||||
|
||||
### Workspace And Storage
|
||||
|
||||
- Default-idempotent run behavior with an explicit force override.
|
||||
- Remote workspace storage.
|
||||
- Workspace garbage collection and archival policies.
|
||||
- Cross-machine checkpoint reuse.
|
||||
- Cross-machine sharing of accepted-artifact cache entries.
|
||||
|
||||
@@ -1,842 +0,0 @@
|
||||
# PromptKit Architecture Discussion Record
|
||||
|
||||
## Purpose And Status
|
||||
|
||||
The decisions in items 1–6, the subsequent CommandKit discussion, and the
|
||||
simplified cache/output designs are incorporated into the
|
||||
[master native migration roadmap](promptkit.md). It owns the selected target,
|
||||
unresolved choices, sequencing, acceptance criteria, and implementation status.
|
||||
|
||||
The [CommandKit discussion record](commandkit-migration-discussion.md) retains
|
||||
the related CLI, preflight, output, debug, and migration rationale. The master
|
||||
roadmap replaces earlier checkpoint matching/provenance proposals with the
|
||||
accepted-artifact cache and retires mandatory manifests, extensive receipts,
|
||||
and durable execution ledgers. The upstream message-content hash remains pending;
|
||||
incorporation does not claim it is already available or implemented.
|
||||
|
||||
This document retains the discussion, rationale, and evidence, including item 7's
|
||||
completed Audita comparison. Item-level questions and follow-ups below record
|
||||
the discussion as it developed; they are not a second implementation plan.
|
||||
|
||||
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
|
||||
capabilities as fully as practical. Application ownership of a policy does not
|
||||
necessarily require application ownership of its implementation. Evaluate
|
||||
remaining machinery against this goal, including opportunities for upstream
|
||||
features where PromptKit does not yet meet an application need.
|
||||
|
||||
No application changes have been implemented through this discussion or the
|
||||
roadmap rewrite. Significant changes to accepted architecture need successor ADRs
|
||||
under the [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||||
Current boundaries are described in [Architecture](../policy/architecture.md)
|
||||
and [LLM Runtime Internals](../internal/llm.md).
|
||||
|
||||
## 1. LLM API And Diagnostic Boundary
|
||||
|
||||
**Discussion / decision:** Direction agreed: remove the Notarius PromptKit
|
||||
adapter and compose the application directly from native PromptKit execution,
|
||||
typed artifact operations, and pipeline orchestration. A small application
|
||||
function connects generation to configured validation and acceptance. The design
|
||||
below incorporates items 2–6 and the Audita review; exact signatures and migration
|
||||
details remain open. No application changes have been implemented.
|
||||
|
||||
### Background And Intent
|
||||
|
||||
Notarius and related applications, including Audita, originally developed their
|
||||
own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by
|
||||
extracting their duplicated mechanics into a shared library. Notarius adopted
|
||||
an early version and predates many of the library's modern capabilities.
|
||||
|
||||
The maintainer clarified that duplication of PromptKit capabilities, including
|
||||
slight behavioral differences, does not reflect an intentional requirement for
|
||||
an independent Notarius implementation. Existing wrappers, tests, and historical
|
||||
decisions are migration evidence, not sufficient reasons to preserve that
|
||||
duplication. Identify actual application needs before choosing what to retain.
|
||||
|
||||
Missing reusable capabilities may justify upstream feature requests. The
|
||||
discussion is not constrained to what v0.23.2 can replace immediately; any
|
||||
resulting change to the adoption target remains to be selected.
|
||||
|
||||
### Application Structure
|
||||
|
||||
| Part | Responsibility |
|
||||
| --- | --- |
|
||||
| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. |
|
||||
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
|
||||
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
|
||||
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
|
||||
| Application records and CLI | Store artifacts and accepted-artifact cache entries, attach pipeline meaning to execution facts, and present results through CommandKit. |
|
||||
|
||||
Engine construction becomes ordinary application wiring using item 2's embedded
|
||||
defaults and operator sources. Profile selection is expressed through native
|
||||
requests. U3 now places shared loaders and engine construction in `internal/cli`
|
||||
and the producer helper beside pipeline assessment. Retire the emptied
|
||||
`internal/framework/llm` package rather than retaining a constructor-only package.
|
||||
The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns
|
||||
the resolved design; implementation is pending.
|
||||
|
||||
Use native requests, results, exchanges, errors, and observers directly in
|
||||
model-executing application code. Pure domain models, codecs, source-reference
|
||||
resolution, and deterministic transformations need no PromptKit dependency.
|
||||
Retain application interfaces for extracting artifacts, validating typed
|
||||
candidates, encoding output, and loading accepted cached artifacts. Deterministic
|
||||
stages continue to operate without conversation machinery.
|
||||
|
||||
### Connect Modules To Pipeline Assessment
|
||||
|
||||
The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
|
||||
prepares inputs, invokes the private completion interface, interprets a response,
|
||||
and returns a copied raw model candidate to the
|
||||
[producer retry loop](../../internal/framework/pipeline/producer_attempts.go).
|
||||
Replace that arrangement with direct collaboration between module interpretation,
|
||||
pipeline assessment, and native execution:
|
||||
|
||||
1. The module prepares source material, references, and any candidate-handle
|
||||
mappings once, and constructs the native initial request.
|
||||
2. Application execution calls `Engine.Run`, capturing an exchange when
|
||||
continuation is enabled.
|
||||
3. The module interprets the structurally valid response into a typed candidate
|
||||
or actionable domain findings.
|
||||
4. The pipeline runs configured candidate validation and decides whether to
|
||||
accept, request correction, or apply terminal policy.
|
||||
5. If correction is needed, native `Coordinate` invokes the same interpretation
|
||||
and assessment for revised responses. The module supplies the repair request;
|
||||
PromptKit maintains history and enforces the turn budget.
|
||||
|
||||
The resolved U3 design uses a small shared helper in `internal/framework/pipeline`.
|
||||
Modules supply native initial/correction requests and response interpretation;
|
||||
the pipeline supplies an operation-scoped assessment callback and relevant
|
||||
cache/debug collaborators. Local variables or closures hold operation state.
|
||||
Use the same assessor for initial and revised candidates without an additional
|
||||
final pass. Native preparation supports cache lookup before execution; hits load
|
||||
the accepted artifact directly. The helper returns a typed application outcome
|
||||
and saves only final accepted artifacts under the initial hash.
|
||||
|
||||
Keep deterministic stages on their ordinary typed production/validation path.
|
||||
LLM validators call the shared engine directly and return findings without an
|
||||
automatic producer correction loop. No replacement completion API, execution-plan
|
||||
language, or copied conversation model is selected.
|
||||
|
||||
Item 3 owns the detailed continuation and assessment rules, including final-turn
|
||||
assessment, failure handling, complete replacement responses, and safe fallback.
|
||||
For normalization, a revised reconciliation plan is applied to the original
|
||||
inputs to produce a fresh candidate. LLM-backed validators can call the same
|
||||
native engine and return application validation findings; they do not need the
|
||||
producer's conversation machinery.
|
||||
|
||||
### Removal, Records, And Testing
|
||||
|
||||
Retire `StructuredLLMClient`, the private structured-completion request/response
|
||||
family, adapter conversions, and opaque correction transport. Items 3, 4, and 6
|
||||
specify removal of the outer LLM retry loop, scheduler, debug completion wrapper,
|
||||
profile-recording plumbing, and duplicate execution accounting. Do not recreate
|
||||
that wrapper stack around native calls.
|
||||
|
||||
Attach pipeline scope through native callbacks and observers. Use native
|
||||
`ExecutionGroup` totals for operational diagnostics and optional debug artifacts,
|
||||
following item 6. The small completion report follows the
|
||||
[CommandKit output contract](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes).
|
||||
Item 5 now selects a simple accepted-artifact cache, without required durable
|
||||
acceptance or execution-provenance records.
|
||||
|
||||
Audita demonstrates shared engine assembly, native continuation, and CommandKit
|
||||
integration. Its `RoleClient`, structured-output interface, and opaque exchange
|
||||
wrapper are not part of this design; item 7 records the comparison's evidence
|
||||
and limits.
|
||||
|
||||
Test domain interpretation and assessment directly. Exercise model workflows
|
||||
with a real PromptKit engine and its injectable client or a fake HTTP provider.
|
||||
Pipeline orchestration can still use fake stage implementations. Testing does
|
||||
not require preserving a private completion interface.
|
||||
|
||||
### Remaining Implementation Work
|
||||
|
||||
- Implement the resolved U3 ownership and placement; refine function signatures
|
||||
and argument grouping against representative extraction and normalization.
|
||||
These implementation details do not reopen the selected boundaries.
|
||||
- Implement item 2's resolved asset/schema design and settle the remaining
|
||||
migration details recorded under items 3–6, including durable compatibility.
|
||||
- Evaluate any remaining upstream gap against an actual application requirement,
|
||||
rather than preserving historical machinery by default.
|
||||
|
||||
**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6.
|
||||
Record the significant boundary change in successor ADRs. Update architecture
|
||||
and implementation documentation when the change lands, explicitly replacing
|
||||
the current transport-neutral completion boundary with PromptKit as the execution
|
||||
dependency of application workflows. Keep domain operations and application
|
||||
outcomes as the meaningful Notarius-owned boundaries.
|
||||
|
||||
## 2. Prompt Assets And Schema Assembly
|
||||
|
||||
**Discussion / decision:** Agreed and incorporated as resolved U2 in the
|
||||
[master roadmap](promptkit.md#prompt-assets-and-response-schemas-u2-resolved).
|
||||
The filesystem and schema questions initially deferred here are now settled.
|
||||
Implementation is pending; the master roadmap owns the detailed design and checks.
|
||||
|
||||
Use ordinary embedded assets, native prompt declarations, and native schema
|
||||
loading. Keep the content tree and module ownership of prompt meaning and domain
|
||||
interpretation. Shared files use real native filesystem paths. Remove duplicate
|
||||
asset registration, per-module manifests, synthetic mounts, flattened schema
|
||||
namespaces, and the custom read-only filesystem that served that assembly.
|
||||
|
||||
Prompt definitions select output contracts by default. Initial and correction
|
||||
prompts can share schema files. Use native per-request replacement only for a
|
||||
demonstrated runtime-dependent contract, without an application wrapper or schema
|
||||
registry. Audita demonstrates ordinary embedded schema loading and per-call
|
||||
contracts; its per-call selection is not required for every Notarius operation.
|
||||
|
||||
The default schema source is the embedded asset tree. Operator-selected external
|
||||
schema directories use native whole-root replacement, not an application overlay.
|
||||
Prompt underlays do not implicitly overlay schema files. PromptKit owns private
|
||||
response-schema loading, structural checks, and repair; Notarius keeps typed
|
||||
interpretation, domain validation, and distinct durable artifact schemas/codecs.
|
||||
|
||||
Retire response-schema hashes and copied metadata used only by the adapter,
|
||||
checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic
|
||||
snapshots. Neither asset nor schema metadata adds another cache fingerprint.
|
||||
|
||||
The spell extractor illustrated the duplication: prompt YAML declared ordered
|
||||
messages and a schema while a Go manifest separately listed, mounted, registered,
|
||||
and hashed assets. Native declarations now supply the selected assembly model.
|
||||
|
||||
**Follow-up:** Implement the master roadmap's Phase 2 and retire tests of removed
|
||||
mounting/registry machinery. Verify shared paths, operator replacements, native
|
||||
schema sources, and any necessary complete request-contract replacement.
|
||||
|
||||
## 3. Retry And Semantic Correction Ownership
|
||||
|
||||
**Discussion / decision:** Direction agreed: adopt Audita's native continuation
|
||||
approach. PromptKit owns captured exchanges, accumulated conversation history,
|
||||
continuation execution, and turn-budget enforcement. Notarius supplies semantic
|
||||
assessment, corrective findings, and acceptance or fallback decisions. Use native
|
||||
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
|
||||
exchange wrapper are not part of the selected direction.
|
||||
|
||||
The flow and removal/adaptation direction below are agreed. Begin with one
|
||||
conversation per producer operation and complete replacement model responses.
|
||||
Retain application assessment, safe-fallback decisions, artifact validation,
|
||||
and cache eligibility. Detailed response contracts, budget migration, and
|
||||
the open policy questions below remain to be resolved. No application changes
|
||||
have been implemented.
|
||||
|
||||
### Agreed Flow
|
||||
|
||||
For cacheable operations, item 5's initial preparation and lookup precede this
|
||||
flow. The steps below describe execution on a miss.
|
||||
|
||||
1. Prepare the source context, references, and any request-local candidate-handle
|
||||
mapping once for the producer operation. Run its initial native request with
|
||||
exchange capture when continuation is enabled. Capture is independent of
|
||||
detailed debug recording.
|
||||
2. Handle the native call's terminal success or failure, decode a usable response,
|
||||
and perform the module's deterministic interpretation and safety checks. Run
|
||||
the configured validation chain on a materialized candidate. A module can
|
||||
identify actionable defects before a candidate is ready for that chain.
|
||||
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
|
||||
and a configured `AdditionalTurns` budget. The callback selects a native
|
||||
repair prompt and supplies actionable findings as variables. PromptKit retains
|
||||
the original exchange and each subsequent repair turn.
|
||||
4. Assess every delivered response, including the last permitted turn. Rebuild
|
||||
the typed candidate from the fixed operation inputs and the revised response;
|
||||
do not repeatedly apply revisions to an already modified artifact. A changed
|
||||
candidate receives the complete configured validation chain again, including
|
||||
validators that approved an earlier version.
|
||||
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
|
||||
whether to reject, fail, or evaluate an explicitly supported safe fallback.
|
||||
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
|
||||
Cancellation never turns an undelivered response into an accepted candidate.
|
||||
|
||||
Start with one conversation per producer operation: a chunk-plan generation,
|
||||
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
|
||||
with Notarius's whole-output validation boundary. Audita's per-proposal branches
|
||||
are useful evidence, but per-entity repair would require its own combination and
|
||||
validation design. It is not necessary to adopt native continuation.
|
||||
|
||||
Repair prompts should supply a new user task, with enduring principles in the
|
||||
original system instructions. Avoid replaying the initial task or embedding
|
||||
initial-response-only constraints in system messages. Begin with complete
|
||||
replacement model responses; for normalization, the response can remain a
|
||||
reconciliation plan that Notarius applies to construct the durable artifact.
|
||||
A different repair schema or profile is supported but not yet selected. Keep
|
||||
source context and candidate-handle meanings stable throughout the conversation.
|
||||
Accumulated history increases context and token use across turns; the semantic
|
||||
turn budget should reflect that tradeoff.
|
||||
|
||||
### Machinery To Remove Or Adapt
|
||||
|
||||
| Current machinery | Agreed direction |
|
||||
| --- | --- |
|
||||
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
|
||||
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
|
||||
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
|
||||
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
|
||||
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
|
||||
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
|
||||
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
|
||||
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
|
||||
|
||||
Native transient retries, structural repair, and semantic additional turns remain
|
||||
distinct controls, but Notarius need not implement all three loops. Existing
|
||||
producer and validator `retries` settings cannot be mechanically renamed: their
|
||||
current budgets cover different behavior. Non-LLM failure handling should be
|
||||
reviewed separately rather than routed through conversation coordination.
|
||||
|
||||
### Validation And Safe Fallback Still Have Application Meaning
|
||||
|
||||
Keep source grounding, reference resolution, domain invariants, reconciliation
|
||||
plan checks, typed application rules, and candidate isolation. In particular,
|
||||
normalization must still reject unknown or overlapping candidate handles and
|
||||
apply only safe groups. A final fallback can retain safe merges while leaving
|
||||
unsafe groups separate; it must go through configured artifact validation and
|
||||
terminal policy before it can advance. After a native terminal failure covered
|
||||
by the selected fallback policy, a supported deterministic fallback can be assessed
|
||||
directly instead of first
|
||||
restarting the same model operation through a feedback-free module retry. Which
|
||||
earlier safe candidate, if any, may survive a later failed turn remains an explicit
|
||||
policy question; do not introduce implicit best-so-far acceptance.
|
||||
|
||||
Retain the distinction between a rejection and unavailable validation. Under the
|
||||
current `warn_continue` policy, incomplete validation can advance a structurally
|
||||
valid candidate but prevents accepted-artifact cache reuse/publication, including
|
||||
for derived results. Continuation adoption does not itself change that policy
|
||||
or safe-fallback eligibility. Keep raw conversation state in process and
|
||||
out of cached artifacts; explicit detailed debug remains separate. Item 5
|
||||
supersedes checkpoint-specific publication and provenance machinery.
|
||||
|
||||
PromptKit owns everything within an individual request execution, including
|
||||
structural validation, internal retries, and repairs. Notarius consumes the
|
||||
public result or failure and owns domain interpretation, validation, and any
|
||||
application-directed correction passes. Do not inspect or reproduce internal
|
||||
PromptKit attempts to decide cache eligibility; cache only the final accepted
|
||||
Notarius artifact after its complete application process succeeds.
|
||||
|
||||
PromptKit should own validation and repair of the private model response.
|
||||
Notarius's serialized schema validator currently receives the codec-produced
|
||||
artifact and its artifact schema, which can differ from the model response.
|
||||
Do not remove that boundary merely because both use JSON Schema. Audit actual
|
||||
duplicate response checks individually; artifact validation and Go/domain
|
||||
interpretation are not automatically redundant. Item 2's resolved native asset
|
||||
and schema design preserves this distinction.
|
||||
|
||||
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
|
||||
[validation executor](../../internal/framework/pipeline/validation_executor.go),
|
||||
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
|
||||
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
|
||||
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
|
||||
and the native continuation evidence in item 7.
|
||||
|
||||
### Open Implementation And Policy Questions
|
||||
|
||||
- Exact repair prompt definitions, response schemas, and profile selection within
|
||||
the agreed complete-replacement approach.
|
||||
- Migration of producer and validator `retries` settings to native controls,
|
||||
including configuration names, defaults, and compatibility.
|
||||
- Detailed terminal/fallback handling, including whether an earlier safe
|
||||
candidate may survive a later failed turn.
|
||||
- Failure handling for non-LLM operations and any justified recovery beyond
|
||||
native execution budgets.
|
||||
- Item 5's remaining cache implementation details and migration to the selected
|
||||
CommandKit output contract.
|
||||
|
||||
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
|
||||
roadmap. Record the changed architecture in a successor to
|
||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
Migration checks should cover accumulated history, fixed handle mappings,
|
||||
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
|
||||
validator failure, cancellation, and cache eligibility. Replace tests of
|
||||
the retired message-reconstruction protocol rather than preserving its mechanics.
|
||||
|
||||
## 4. Global Scheduling And Admission
|
||||
|
||||
**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and
|
||||
retire the separate cross-backend ceiling. Rely on PromptKit's native backend
|
||||
admission and capacity limits while keeping bounded Notarius pipeline workers.
|
||||
This deliberately changes the existing global-limit contract; reproducing that
|
||||
ceiling in PromptKit is not a migration prerequisite. No application changes
|
||||
have been implemented.
|
||||
|
||||
### Agreed Ownership
|
||||
|
||||
| Concern | Owner |
|
||||
| --- | --- |
|
||||
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
|
||||
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
|
||||
| Application disposition after admission or execution failure | Notarius application policy |
|
||||
|
||||
For limited backends, PromptKit v0.23.2 bounds active model attempts by
|
||||
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||||
An operation holds admission
|
||||
across preparation, retries, and structural repair, while active permits cover
|
||||
individual model attempts and are released during retry waits. A completed
|
||||
continuation turn releases admission before application assessment, so its
|
||||
callback can invoke an LLM validator using the same backend.
|
||||
|
||||
The current Notarius wrapper holds a global permit around the complete
|
||||
structured call, including backend waiting and repairs. Removing it eliminates
|
||||
that additional queue and avoids making independent backends share those
|
||||
permits. Independent native backend limits may allow aggregate concurrency above
|
||||
the former global ceiling; that is an accepted behavior change. Pipeline worker
|
||||
bounds remain application work controls, not a replacement global LLM ceiling.
|
||||
|
||||
### Removal And Configuration Consequences
|
||||
|
||||
- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom
|
||||
queue/cancellation/permit bookkeeping.
|
||||
- Retire `concurrency.total_llm`. Give pipeline worker settings an independent
|
||||
default and remove the extraction-worker maximum derived from `total_llm`.
|
||||
- Use one shared execution engine for producers, validators, and repair turns.
|
||||
Native capacity belongs to an engine and selected backend ID; separate engine
|
||||
instances have independent pools.
|
||||
- Configure native limits at the intended backend capacity boundary. Profiles
|
||||
selecting the same backend ID share its pool; separate IDs pointing to the
|
||||
same endpoint do not automatically share capacity. Endpoint-only profiles and
|
||||
backends without a positive limit remain unrestricted by backend admission.
|
||||
- Handle native `ErrCapacityExceeded` as an admission failure through application
|
||||
policy, consistently with item 3's removal of blanket execution retries.
|
||||
- Retire scheduler implementation tests with the removed code. Retain focused
|
||||
integration checks for shared native admission across roles, independent
|
||||
backends, bounded pipeline work, queue exhaustion, and cancellation.
|
||||
|
||||
### Remaining Details
|
||||
|
||||
Select independent worker defaults, native backend/queue defaults where the
|
||||
application supplies them, and the configuration migration treatment of the
|
||||
retired field. Specify how admission failures are reported and handled for each
|
||||
application role. These details do not reopen the selected ownership boundary.
|
||||
|
||||
A future demonstrated need for capacity shared across several backend IDs could
|
||||
justify an upstream PromptKit feature request. No such feature is required for
|
||||
this agreed design, and it does not justify retaining the Notarius scheduler.
|
||||
|
||||
Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
|
||||
[scheduler](../../internal/framework/llm/scheduler.go),
|
||||
[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and
|
||||
[native capacity behavior](../../../promptkit/docs/internal/capacity.md).
|
||||
|
||||
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
|
||||
and record the changed ownership in an ADR. Update architecture, configuration,
|
||||
operations, and implementation documentation when the migration lands. Continue
|
||||
with item 5's accepted-artifact cache discussion.
|
||||
|
||||
## 5. Accepted-Artifact Cache
|
||||
|
||||
**Discussion / decision:** Agreed. Rename and replace the checkpoint subsystem
|
||||
with a simple cache surrounding the complete Notarius producer operation.
|
||||
Compute its key early from the initial prepared request's message contents;
|
||||
store its value late, after Notarius has accepted the final artifact. This
|
||||
supersedes both comprehensive execution fingerprints and the earlier
|
||||
stage/input/prompt-ID matching proposal. Implementation is pending.
|
||||
|
||||
### Purpose And Key
|
||||
|
||||
The cached value is an accepted answer to the original task. After a downstream
|
||||
failure, rerunning the pipeline can reuse that artifact without repeating the
|
||||
expensive model work that produced it. Cheap deterministic work elsewhere in
|
||||
the pipeline does not need a general stage-checkpoint framework.
|
||||
|
||||
Use a PromptKit-provided `MessageContentHash` (proposed name) over the initial
|
||||
request's ordered messages, including each message's role and exact text. Exclude
|
||||
model settings, session IDs, execution IDs, timestamps, and provider cache hints.
|
||||
Do not add whole-configuration, stage, prompt-ID, library-version, or validation-
|
||||
policy fingerprints to this matching rule.
|
||||
|
||||
Changing the actual initial messages changes the key. Changing a prompt ID,
|
||||
model, or setting without changing those messages does not. Fixing deterministic
|
||||
interpretation or validation may therefore require explicit refresh/bypass to
|
||||
replace a previously accepted artifact; a hit does not claim fresh validation.
|
||||
|
||||
This is a concrete upstream hash requirement. The reviewed PromptKit v0.23.3
|
||||
`RenderedPromptHash` includes session identity and provider cache hints, while
|
||||
hashing the entire `PreparedRun` would include volatile execution metadata.
|
||||
Neither is the selected key. Expose the content hash through native preparation
|
||||
rather than implementing another message-hashing policy in Notarius. Confirm
|
||||
the supplying upstream API/release before cache implementation; do not claim
|
||||
the proposed field already exists or silently substitute a broader fingerprint.
|
||||
|
||||
### Compute Early, Store After Acceptance
|
||||
|
||||
```text
|
||||
Prepare initial request -> obtain message-content hash -> look up cache/<hash>.json
|
||||
Hit: decode the accepted Notarius artifact and continue the pipeline
|
||||
Miss: execute the complete Notarius operation
|
||||
-> native PromptKit calls as needed
|
||||
-> domain interpretation, validation, and correction passes
|
||||
-> on final acceptance, save the artifact under the initial hash
|
||||
```
|
||||
|
||||
Lookup occurs before model execution. Use native preparation to inspect the
|
||||
initial request; preserve that prepared request for execution on a miss where
|
||||
the native API supports it. This is a reason to prepare before deciding whether
|
||||
to execute, distinct from preparing solely to obtain logging details.
|
||||
|
||||
Retain the initial hash throughout application-directed correction. Save the
|
||||
final accepted artifact under that hash, not under a later correction request's
|
||||
hash. A subsequent run can then reuse the accepted answer without replaying the
|
||||
failed turns that led to it.
|
||||
|
||||
The value is the final Notarius artifact, serialized through its existing codec.
|
||||
For normalization, it is the accepted normalized artifact after plan application,
|
||||
not the model's proposed reconciliation plan. Do not store an intermediate raw
|
||||
response, `RunResult`, exchange, or prepared-execution handle as the cache value.
|
||||
|
||||
Only publish after the whole Notarius process succeeds, including configured
|
||||
domain validation and any application-directed retries/correction passes.
|
||||
Rejected, failed, and validation-incomplete results are not cacheable. Retain
|
||||
the existing exclusion of derivatives of validation-incomplete work without
|
||||
restoring checkpoint manifests or fingerprint graphs to implement it.
|
||||
|
||||
A hit loads the accepted artifact directly. Ordinary decoding and usability
|
||||
checks apply; absent or unusable entries are misses. Do not execute a model
|
||||
request, reconstruct native results/exchanges, replay corrections, or repeat
|
||||
semantic validation merely because the value came from cache. Do not count
|
||||
historical model work as current execution.
|
||||
|
||||
### PromptKit Ownership Boundary
|
||||
|
||||
PromptKit owns the mechanics within each request execution, including structural
|
||||
response validation, internal retries, and repairs. Notarius handles a usable
|
||||
response or terminal failure at the public call boundary, then applies its own
|
||||
domain interpretation, validation, correction, and acceptance process.
|
||||
|
||||
Internal PromptKit attempts are not inputs to the cache design. Native debug
|
||||
observations may expose evidence about them, but Notarius does not inspect or
|
||||
reproduce those mechanics to determine cache eligibility. The upstream addition
|
||||
needed here is the message-content hash; no cached-response injection, replay,
|
||||
exchange restoration, or PromptKit-owned write-through cache is required.
|
||||
|
||||
### Storage And Removal
|
||||
|
||||
Use a small filesystem lookup/write facility with `cache/<hash>.json` containing
|
||||
the accepted artifact. Ordinary temporary-file/rename publication suffices to
|
||||
avoid exposing partial entries. Cache lookup must not require original output
|
||||
or debug directories, acceptance ledgers, execution provenance, or a separate
|
||||
versioned checkpoint envelope. Debug capture can record the current hit and
|
||||
loaded artifact without requiring historical metadata in the cache file.
|
||||
|
||||
Retire checkpoint-specific identities, invocation-wide lookup barriers,
|
||||
stage/dependency fingerprints, manifests, and recovery bookkeeping. Do not
|
||||
preserve that machinery solely to cache cheap deterministic stages. The existing
|
||||
standalone canonical chunk-plan cache remains a separate concern.
|
||||
|
||||
The old `--resume`/`--recompute-step` conventions are migration context, not a
|
||||
compatibility requirement. Select straightforward cache reuse/bypass/refresh
|
||||
controls and legacy-file handling during implementation planning; do not infer
|
||||
new flag spellings or a migration framework from this decision.
|
||||
|
||||
**Follow-up:** The master roadmap now includes this design and the upstream hash
|
||||
requirement. During implementation, verify that a downstream failure leaves
|
||||
earlier accepted artifacts reusable, corrected answers are stored under the
|
||||
initial hash, changed messages miss, settings-only changes can hit, and a hit makes no model calls or
|
||||
new semantic-validation claims. Keep output/report contract decisions separate.
|
||||
|
||||
## 6. Inspection And Execution Bookkeeping
|
||||
|
||||
**Discussion / decision:** Direction agreed: use native PromptKit inspection
|
||||
and execution facts, with CommandKit owning standardized operator presentation.
|
||||
Remove duplicate LLM reporting and accounting machinery. Notarius retains
|
||||
application context, domain decisions, artifact delivery, completion-summary
|
||||
content, and optional debug capture. The CommandKit output decision below removes
|
||||
mandatory durable execution provenance. No application changes have been
|
||||
implemented.
|
||||
|
||||
### Ownership
|
||||
|
||||
| Responsibility | Owner |
|
||||
| --- | --- |
|
||||
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
|
||||
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
|
||||
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
|
||||
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
|
||||
| Acceptance, rejection, fallback, accepted-artifact reuse, and pipeline outcome | Notarius. |
|
||||
| Artifact storage, completion-summary content, and optional debug storage | Notarius. |
|
||||
|
||||
### Native Inspection And Operator Presentation
|
||||
|
||||
Remove the profile-inspector wrapper, copied effective-model-parameter types,
|
||||
and local reconstruction of upstream configuration. Use native inspection for
|
||||
configured resources and native prepared/result data for the target actually
|
||||
executed, including request overrides. Profile inspection can describe a partial
|
||||
configured profile; it is not evidence of the completed execution target.
|
||||
|
||||
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
|
||||
preview, and opt-in smoke commands. Notarius supplies configuration/engine
|
||||
loaders and application request builders. Avoid building temporary custom
|
||||
commands or report machinery that the planned CommandKit adoption would replace.
|
||||
|
||||
Retain application pipeline preflight: resolve modules, artifact compatibility,
|
||||
references, and known prompt/profile selections. Use native definition checks
|
||||
without simulating stage execution or requiring provider credentials. Concrete
|
||||
request preparation remains the place to check the completed target, inputs,
|
||||
schema, and credential requirements.
|
||||
|
||||
CommandKit supplies formatting, visibility, progress presentation, and delivery
|
||||
error handling. Notarius supplies report content, domain outcomes, and exit-status
|
||||
mapping. Follow the small structured completion report and native text template
|
||||
selected in [CommandKit item 4](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes);
|
||||
do not preserve the old receipt shape or independent encoders.
|
||||
|
||||
### Native Execution Facts
|
||||
|
||||
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
|
||||
for provenance or optional debug capture. Remove unconditional
|
||||
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
|
||||
- Use native observations for terminal execution facts, including failures that
|
||||
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
|
||||
preparation or admission failure; it cannot serve as a completion record.
|
||||
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
|
||||
Remove the private debug completion wrapper and duplicate synthetic call-ID
|
||||
machinery where native identity and application attribution suffice.
|
||||
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
|
||||
cloning, and independent attempt/usage accumulators. Use native types internally
|
||||
and project selected fields only at an application reporting or durable boundary.
|
||||
- Remove the adapter's `LLMProfileRecorder` and provider-forwarding plumbing.
|
||||
Native snapshots provide the profiles and targets actually used for optional
|
||||
diagnostics; do not reconstruct a durable profile inventory.
|
||||
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
|
||||
initial run, continuation runs, and nested validator runs as applicable.
|
||||
Preserve terminal-coverage and token-usage completeness; unknown usage is not
|
||||
zero. Do not count both attempt observations and terminal aggregates, or add
|
||||
overlapping groups as though they were disjoint work.
|
||||
|
||||
Application execution code should attach scope and observers, run and assess the
|
||||
native result, and record its disposition and execution references. This does
|
||||
not require a replacement general-purpose LLM interface. A successful native
|
||||
execution can still produce a domain-rejected candidate; provider success and
|
||||
application acceptance remain separate facts.
|
||||
|
||||
### Application Records And Durable Boundaries
|
||||
|
||||
Attach the application context needed to explain a pipeline in optional debug
|
||||
capture: which stage/lane/chunk produced an artifact, which validators assessed
|
||||
it, whether continuation or a safe fallback was used, and whether the artifact
|
||||
was generated now or reused. PromptKit cannot infer that meaning from provider
|
||||
execution alone. These diagnostic facts do not require consumer-facing histories.
|
||||
|
||||
Under the revised item 5, cache files contain accepted artifacts without required
|
||||
historical execution or acceptance records. A current debug record can identify
|
||||
reuse and show the loaded artifact without counting earlier token usage as
|
||||
current work or claiming new validation.
|
||||
|
||||
The selected CommandKit output contract delivers artifacts as files and a small
|
||||
completion report through native rendering. It does not require a run manifest,
|
||||
metadata bundle, separately persisted receipt, or durable execution ledger.
|
||||
Existing [manifest](../integrations/json-output.md) and
|
||||
[receipt](../integrations/run-result.md) contracts document migration impact;
|
||||
their field inventories are not requirements to recreate. Preserve intrinsic
|
||||
domain artifact content and reuse existing outcome facts for the completion
|
||||
summary and debug overview.
|
||||
|
||||
Keep application debug artifact placement, opt-in capture, and lifecycle under
|
||||
[CommandKit item 5](commandkit-migration-discussion.md#5-debug-capture-and-optional-diagnostics).
|
||||
Native callbacks supply execution data; they do not own Notarius's storage policy.
|
||||
Detailed prepared messages and execution history belong in that optional capture,
|
||||
not a mandatory consumer metadata bundle.
|
||||
|
||||
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
|
||||
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
|
||||
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
|
||||
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
|
||||
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
|
||||
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
|
||||
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
|
||||
|
||||
### Remaining Implementation Work
|
||||
|
||||
Select application scope and observer/group enrollment for the agreed diagnostics.
|
||||
Implement the selected completion report and retire superseded manifest, receipt,
|
||||
and execution-ledger machinery. Coordinate command migration and debug capture
|
||||
with the incorporated design in the master roadmap. The core
|
||||
bookkeeping ownership change needs no new PromptKit capability; any remaining
|
||||
upstream request should identify a concrete unmet requirement.
|
||||
|
||||
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
|
||||
significant boundary changes in successor ADRs. Update current-behavior
|
||||
documentation when implemented. Verify failure reporting without a result,
|
||||
continuation/validator accounting, unknown usage, actual-target provenance,
|
||||
cache reuse accounting, and optional debug capture during implementation.
|
||||
|
||||
## 7. Compare Audita's PromptKit Workflow
|
||||
|
||||
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
|
||||
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
|
||||
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
|
||||
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
|
||||
not new Notarius contracts. The source links below assume the sibling checkout.
|
||||
|
||||
### Embedded Defaults And Operator Replacement
|
||||
|
||||
- One captured engine is assembled from PromptKit's default catalog, the
|
||||
embedded Audita underlay, and a higher-priority operator underlay.
|
||||
`FilesystemInputs` supplies eleven native prompt definitions in
|
||||
`collections.audita`: four proposal prompts, four repair prompts, and three
|
||||
validator prompts.
|
||||
- Operator sources are optional `prompts.yml` and recursively selected YAML
|
||||
under `prompts/`, anchored to the application main source's recorded context.
|
||||
Both locations feed one operator underlay and are peers; conflicting peer
|
||||
definitions fail rather than receiving filename-based precedence.
|
||||
- Replacement is by resource identity, not filename. For example, a complete
|
||||
prompt record with local ID `modules.grammar.repair` under
|
||||
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
|
||||
other embedded definitions remain available. Native same-ID replacement is
|
||||
whole-record replacement, not recursive merging of individual prompt fields.
|
||||
- Native imports and message-file context are retained. Tests show that an
|
||||
existing engine retains captured message content after a file edit and a new
|
||||
engine sees the edit. They also verify inherited message-file resolution,
|
||||
main-relative discovery independent of the working directory, and peer
|
||||
conflict rejection.
|
||||
- A diagnostic embedded-only construction can classify a loading error; it
|
||||
does not silently fall back to defaults after an operator-source failure.
|
||||
|
||||
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
|
||||
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
|
||||
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
|
||||
|
||||
### Prompt Content And Schema Ownership Are Separate
|
||||
|
||||
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
|
||||
small prompt registry records IDs and embedded paths; it does not reconstruct
|
||||
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
|
||||
supplied as template variables. Processing supplies the application-owned values
|
||||
for those shared names, but an operator replacement decides whether its template
|
||||
references them at all. They are not forcibly appended to replacement messages.
|
||||
|
||||
The processing request independently supplies a complete native output contract:
|
||||
JSON Schema validation, the schema path selected for that operation, and its
|
||||
structural-repair budget. The three schemas describe a correction set, a repair
|
||||
decision, and validator decisions. PromptKit receives their canonical embedded
|
||||
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
|
||||
change the processing code's expected response shape. A test replaces a prompt
|
||||
with a text/no-validation declaration and still exercises the processing
|
||||
request's JSON-schema repair contract.
|
||||
|
||||
Audita still keeps a response-schema key/metadata/hash registry and application
|
||||
decoding checks. Its use of native loading does not prove that every schema
|
||||
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
|
||||
a concrete option: module-owned response requirements expressed directly through
|
||||
native per-request contracts. Standalone native preview/preparation can use the
|
||||
prompt declaration's contract; processing supplies its own override.
|
||||
|
||||
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
|
||||
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
|
||||
`prepareCall` and `OutputContract` in the
|
||||
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
|
||||
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
|
||||
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
|
||||
|
||||
### Native Continuation With Application Assessment
|
||||
|
||||
The implemented flow is:
|
||||
|
||||
1. Generate a section's proposal set, capturing a native immutable exchange when
|
||||
correction repair is enabled. This retains the actual proposal messages and
|
||||
response; it does not require debug recording.
|
||||
2. Apply application validation. An actionable rejected proposal starts its own
|
||||
branch from the original proposal exchange. Unavailable-only findings and
|
||||
no-effect corrections do not start repair. Existing approvals remain separate.
|
||||
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
|
||||
Each follow-up selects a distinct repair prompt, repair profile, and repair
|
||||
response schema. The request carries the original proposal reference, current
|
||||
correction, ordered findings, and available before/after text previews.
|
||||
4. PromptKit assembles history and progresses the conversation. A second turn
|
||||
retains the first repair instruction and response. A sibling branch starts
|
||||
from the original exchange and does not inherit another branch's repairs.
|
||||
Audita does not rerender the original proposal or manually replay messages.
|
||||
5. Audita decodes each delivered decision. Withdrawal ends the branch without
|
||||
applying that correction. A revision undergoes the full application validation
|
||||
chain again, including validators that approved its earlier version. Approval
|
||||
stops the branch; actionable rejection can request another turn.
|
||||
6. The final permitted response is assessed before exhaustion is determined.
|
||||
Requesting another turn at that boundary does not allocate another logical
|
||||
call. Recoverable provider/invalid-output failures finish the branch as
|
||||
unavailable; fatal errors or parent cancellation interrupt processing.
|
||||
Canceled, undelivered responses are diagnostic evidence only, never approvals.
|
||||
|
||||
Transport retries and structural repair operate inside each PromptKit execution;
|
||||
they are separate from the semantic branch's additional-turn budget. Tests show
|
||||
one semantic repair turn containing multiple provider attempts for structural
|
||||
repair. The application retains acceptance, withdrawal, application ordering,
|
||||
and reporting; PromptKit retains conversation and execution mechanics.
|
||||
|
||||
Audita's prompt roles support this model deliberately: system instructions carry
|
||||
durable principles, the initial user message carries the initial module task,
|
||||
and the repair prompt supplies a new user assignment. Adopting continuation in
|
||||
Notarius would require reviewing the complete conversation and response-format
|
||||
transitions, rather than mechanically appending to prompts written for fresh
|
||||
replacement requests.
|
||||
|
||||
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
|
||||
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
|
||||
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
|
||||
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
|
||||
and [prompt guidance](../../../audita/docs/internal/prompts.md).
|
||||
|
||||
### Other Boundaries And Limits Of The Comparison
|
||||
|
||||
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
|
||||
wrapper, and application diagnostic types. Adopt the demonstrated native
|
||||
mechanisms without assuming Notarius should copy those wrappers; item 1's
|
||||
adapter-removal direction remains in force.
|
||||
- Proposal, validator, and repair roles share one engine and its per-backend
|
||||
admission. The runner separately bounds section work. This demonstrates useful
|
||||
separation of application work scheduling and provider capacity. Item 4 now
|
||||
selects that ownership and retires Notarius's independent cross-backend ceiling.
|
||||
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
|
||||
feed a native `ExecutionGroup`; application records add logical-call and
|
||||
stage/section/proposal attribution. Nested validator executions enroll
|
||||
explicitly. Counts preserve terminal coverage and token-usage completeness.
|
||||
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
|
||||
commands using Audita's engine construction, including request preparation,
|
||||
validation, and smoke diagnostics.
|
||||
- Audita edits independently assessable transcript corrections; Notarius accepts
|
||||
typed stage artifacts and has cross-stage dependencies. Audita branches at an
|
||||
individual correction within a proposal response. Notarius must deliberately
|
||||
choose its correction unit rather than assume per-entity branching.
|
||||
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
|
||||
cache design; item 5 records that application decision independently.
|
||||
|
||||
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
|
||||
[observations](../../../audita/internal/framework/llm/observations.go),
|
||||
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
|
||||
[component scope](../../../audita/docs/internal/overview.md).
|
||||
|
||||
### Implications For The Remaining Discussion
|
||||
|
||||
Item 2's native underlay direction has a working application precedent. Audita
|
||||
also offers evidence for ordinary embedded filesystems and per-request schema
|
||||
selection. U2 now selects ordinary embedded assets and prompt-declared contracts
|
||||
by default, reserving per-request replacement for demonstrated runtime needs.
|
||||
|
||||
The subsequent item 3 discussion selects native continuation and accumulated
|
||||
conversation history. Keeping domain validation local does not require keeping
|
||||
conversation assembly or the turn loop local. Item 3 owns the agreed Notarius
|
||||
flow and the remaining repair-prompt, response-contract, and policy details;
|
||||
the comparison alone did not settle those details.
|
||||
|
||||
Item 4 now records agreement to use native admission with bounded application
|
||||
workers. The revised item 5 selects an accepted-artifact cache and a focused
|
||||
upstream message-content hash instead of comprehensive execution fingerprints.
|
||||
Item 6 adopts native execution observations and CommandKit presentation;
|
||||
the [CommandKit output decision](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes)
|
||||
selects artifact files and a small native completion report.
|
||||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||||
feature is needed merely to provide replaceable defaults or domain-directed
|
||||
continuation. Any remaining upstream request should identify the specific unmet
|
||||
requirement after reconsidering Notarius's historical behavior.
|
||||
|
||||
### Verification
|
||||
|
||||
Read the production composition, adapter, coordinator, domain repair callbacks,
|
||||
prompt/schema definitions, and focused behavior tests. The following checks
|
||||
passed from the Audita checkout:
|
||||
|
||||
```sh
|
||||
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
|
||||
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
|
||||
```
|
||||
|
||||
The subprocess checks exercise assembled configuration, real native continuation
|
||||
messages, operator replacement, final-turn assessment, sibling isolation,
|
||||
repair-disabled behavior, and cancellation using a local fake provider. These
|
||||
checks establish implementation behavior, not live-model quality. No live model
|
||||
calls or application source changes were made. Broader migration decisions and
|
||||
changes to the adoption roadmap remain for discussion.
|
||||
@@ -8,13 +8,19 @@ presentation directly. Retain the application work of producing, assessing,
|
||||
publishing, and reusing typed artifacts. Historical duplication of library
|
||||
behavior is not a requirement to preserve it.
|
||||
|
||||
This roadmap incorporates the [PromptKit architecture discussion](promptkit-architecture-discussion.md)
|
||||
and all six items of the [CommandKit discussion](commandkit-migration-discussion.md),
|
||||
including the subsequent accepted-artifact cache and artifact-first output
|
||||
decisions. The notepads retain rationale and Audita evidence; this document owns
|
||||
the selected target, unresolved choices, implementation sequence, removal
|
||||
inventory, and acceptance criteria. All implementation phases are pending.
|
||||
Recording these decisions has not changed application behavior.
|
||||
This is the single migration plan for PromptKit, ConfigKit, CommandKit,
|
||||
accepted-artifact caching, and artifact-first output. It owns the selected target,
|
||||
remaining choices, implementation sequence, removal inventory, acceptance
|
||||
criteria, and supporting evidence. The earlier discussion documents are retired;
|
||||
Git history preserves the discussion. All implementation phases are pending.
|
||||
Recording these decisions has not changed application behavior. Unrelated future
|
||||
work remains in the [separate backlog](future.md).
|
||||
|
||||
Notarius predates many modern PromptKit capabilities: the library grew out of
|
||||
mechanics shared by Notarius and related applications. Historical duplication
|
||||
and slight behavioral differences do not express an intentional application
|
||||
requirement. Evaluate remaining machinery against actual domain needs, and
|
||||
consider upstream features when a reusable capability is missing.
|
||||
|
||||
Complete one coordinated cutover for configuration, PromptKit resources, and the
|
||||
CLI. Adopt native public conventions fully; preserving old CLI behavior is a
|
||||
@@ -23,6 +29,10 @@ parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is
|
||||
part of the main migration, not a later follow-up. Release preparation and
|
||||
publication remain separate work.
|
||||
|
||||
Adopting the kits together lets standard inspection and execution share the same
|
||||
loaders and native facts. It avoids building temporary command and reporting
|
||||
adapters while those boundaries are already being replaced.
|
||||
|
||||
## Baseline And Dependency Target
|
||||
|
||||
At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application
|
||||
@@ -38,22 +48,27 @@ The selected reviewed baseline is:
|
||||
| `go.maximumdirect.net/ckconfig` | `v0.9.0` |
|
||||
| `go.maximumdirect.net/ckprompt` | `v0.10.0` |
|
||||
| `go.maximumdirect.net/configkit` | `v1.2.1` |
|
||||
| `go.maximumdirect.net/promptkit` | `v0.23.3`, pending the message-content hash addition below |
|
||||
| `go.maximumdirect.net/promptkit` | `v0.23.3`, subject to U1's hash-capable release review |
|
||||
|
||||
Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible
|
||||
validation dependencies. PromptKit v0.23.3 supersedes the initial v0.23.2 target
|
||||
for its timeout/cancellation fixes. These are selected versions, not a claim
|
||||
that they are the latest releases. The local tag review and integration guidance
|
||||
are recorded in [CommandKit item 1](commandkit-migration-discussion.md#1-adoption-scope-and-dependency-baseline).
|
||||
The tagged [release notes][release], [migration guide][migration],
|
||||
[format reference][formats], and [Go consumer guide][consumer] supply the baseline.
|
||||
validation dependencies. The selected CommandKit integration releases document
|
||||
compatibility with PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout
|
||||
attribution and cancellation/error identity fixes; that patch requires no
|
||||
configuration or request API migration from v0.23.2 and retains its dependencies.
|
||||
The original review used local release tags and tagged dependency manifests;
|
||||
remote release lookup was unavailable, so these are selected versions, not a
|
||||
claim that they are the latest releases. See the
|
||||
[integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
|
||||
and tagged [release notes][release], [migration guide][migration],
|
||||
[format reference][formats], and [Go consumer guide][consumer].
|
||||
|
||||
The cache requires the proposed native `MessageContentHash`. The
|
||||
[upstream request](../../../promptkit/docs/roadmap/notarius-request.md) is pending;
|
||||
v0.23.3's `RenderedPromptHash` is not a substitute. Confirm the supplying API and
|
||||
tagged release, then verify the assembled dependency set. Independent composition
|
||||
and pipeline work can proceed while that addition is pending; the completed
|
||||
cache must use the upstream capability. Do not add a local hash-policy fork.
|
||||
The cache requires native `MessageContentHash`; v0.23.3's `RenderedPromptHash`
|
||||
is not a substitute. The sibling PromptKit repository has retired the original
|
||||
feature request, and its [v0.24.0 release notes](../../../promptkit/docs/releases/v0.24.0.md)
|
||||
document the addition and a ConfigKit v1.3.0 dependency. U1 remains open to review
|
||||
the supplying API/release and verify a compatible assembled dependency set before
|
||||
changing the selected baseline. Independent work can proceed; completed cache
|
||||
integration must use the upstream capability. Do not add a local hash-policy fork.
|
||||
Neither a cross-backend global limit nor a comprehensive execution fingerprint
|
||||
is required. Remove obsolete imports and catalog dependencies when unused;
|
||||
avoid permanent `replace` directives and update [attribution](../oss/attribution.md)
|
||||
@@ -480,8 +495,8 @@ Prepare initial request -> obtain MessageContentHash -> look up cache/<hash>.jso
|
||||
-> on final acceptance, store the artifact under the INITIAL hash
|
||||
```
|
||||
|
||||
Use the proposed upstream hash of ordered message roles and exact text. Include
|
||||
message boundaries and order, excluding model/settings, session/execution IDs,
|
||||
Use the upstream message-content hash of ordered message roles and exact text.
|
||||
Include message boundaries and order, excluding model/settings, session/execution IDs,
|
||||
timing, provider cache hints, and other non-content metadata. Add no stage,
|
||||
prompt-ID, library-version, whole-configuration, schema-policy, or implementation
|
||||
fingerprint. Actual input or schema text rendered into messages participates as
|
||||
@@ -499,7 +514,7 @@ prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared`
|
||||
where supported, and discard the handle on a hit. `OnPrepared` execution hooks
|
||||
alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes
|
||||
session identity and cache hints; it is not the requested hash. The upstream
|
||||
request in U1 must land before completing this cache integration.
|
||||
API/release review in U1 must finish before completing this cache integration.
|
||||
|
||||
Keep the initial key throughout corrections. Cache the final Notarius artifact
|
||||
through its existing codec, not an intermediate response, `RunResult`, exchange,
|
||||
@@ -548,7 +563,9 @@ diagnostics; split preparation has a separate purpose for cache lookup. Native
|
||||
observations provide terminal facts even without a returned `RunResult`.
|
||||
`OnPrepared` is absent on preparation/admission failures and is not a completion
|
||||
record. Keep synchronous callbacks lightweight, compose capture/accounting
|
||||
observers, and synchronize shared state where operations overlap.
|
||||
observers, and synchronize shared state where operations overlap. `OnPrepared`
|
||||
runs while admission is held; persist retained evidence at normal operation
|
||||
boundaries.
|
||||
|
||||
Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope.
|
||||
Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning,
|
||||
@@ -751,7 +768,7 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
|
||||
| ID | Remaining choice or dependency | Resolve for |
|
||||
| --- | --- | --- |
|
||||
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
|
||||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||||
@@ -787,7 +804,7 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
|
||||
### Phase 1: Establish The Migration Baseline
|
||||
|
||||
- Record the selected dependency set and track U1 against the upstream request.
|
||||
- Record the selected dependency set and review U1 against the upstream release.
|
||||
Keep the cache dependency explicit without blocking independent work.
|
||||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||||
forward Audita evidence without adopting its application wrappers.
|
||||
@@ -795,8 +812,9 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
execution, assets/schemas, CommandKit, debug, cache, or output ownership.
|
||||
|
||||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||||
and outstanding choices are explicit. This phase does not claim that the hash
|
||||
already exists or that the migration is implemented.
|
||||
and outstanding choices are explicit. Upstream availability alone does not
|
||||
establish compatibility of the assembled Notarius dependencies or completion
|
||||
of the migration.
|
||||
|
||||
### Phase 2: Build Native Composition, Sources, And Preflight
|
||||
|
||||
@@ -869,7 +887,8 @@ including failures after publication and quiet-mode capture.
|
||||
- Complete the acceptance checks and removal inventory, including dependency
|
||||
cleanup, maintained examples, and the assembled command surface.
|
||||
- Finish release migration guidance, attribution, successor ADRs, and affected
|
||||
canonical documentation. Reconcile future-work notes and notepad status.
|
||||
canonical documentation. Update this roadmap's status and reconcile the
|
||||
separate future-work backlog.
|
||||
- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash
|
||||
policy, or obsolete metadata requirement remains in the finished migration.
|
||||
|
||||
@@ -968,7 +987,52 @@ Record significant changes in successor ADRs, including
|
||||
[PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md),
|
||||
admission, caching, and CLI/reporting ownership. Preserve accepted decision text
|
||||
and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||||
Notepads retain discussion history; this roadmap owns implementation status.
|
||||
This roadmap owns implementation status; Git history retains earlier discussions.
|
||||
|
||||
## Evidence And Reading
|
||||
|
||||
These sources support the selected design, not a claim of completed Notarius
|
||||
implementation. Sibling links assume the local repositories are available; the
|
||||
tagged PromptKit links below preserve the reviewed baseline.
|
||||
|
||||
### Native Library Guidance
|
||||
|
||||
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md) and
|
||||
[CLI conventions](../../../commandkit/docs/cli.md): module selection, native
|
||||
aliases/inputs, shared loaders, reporting modes, outcomes, and delivery failures.
|
||||
- [ckconfig](../../../commandkit/integrations/ckconfig/README.md) and
|
||||
[ckprompt](../../../commandkit/integrations/ckprompt/README.md): standard suites
|
||||
supplied with application loaders and request builders.
|
||||
- [ConfigKit](../../../ConfigKit/README.md): native loading and inspection.
|
||||
- [PromptKit consumer guide][consumer] and
|
||||
[capacity behavior](../../../promptkit/docs/internal/capacity.md): native
|
||||
preparation/execution, continuation, observations, and admission.
|
||||
|
||||
### Audita Comparison
|
||||
|
||||
The comparison was completed on 2026-11-18 against Audita revision
|
||||
`da5b8913d1206a8e8f0a784b2322760385548a60`, using PromptKit v0.23.2,
|
||||
ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
|
||||
The observations below concern that revision; linked sibling files may evolve.
|
||||
|
||||
| Evidence | Lesson carried into this roadmap |
|
||||
| --- | --- |
|
||||
| [Engine assembly](../../../audita/internal/framework/llm/runtime.go), [source tests](../../../audita/internal/framework/llm/engine_sources_test.go), and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go) | One captured engine, native underlays and main-source-relative companions, whole-record replacement, preserved file contexts, strict peer conflicts, and no silent fallback on operator-source errors. |
|
||||
| [Adapter tests](../../../audita/internal/framework/llm/adapter_test.go) | Ordinary embedded schemas and complete per-request output contracts work with operator prompt replacement. U2 chooses prompt-declared defaults for Notarius; Audita's schema registry and per-call overrides are not requirements to copy. |
|
||||
| [Native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), [repair flow](../../../audita/internal/framework/runner/repair.go), [conversation tests](../../../audita/internal/framework/llm/conversation_test.go), and [repair tests](../../../audita/internal/framework/runner/repair_test.go) | Captured exchanges, accumulated history, complete candidate revalidation, final-turn assessment, and canceled/undelivered response handling. |
|
||||
| [Prompt guidance](../../../audita/docs/internal/prompts.md) | Durable system instructions, an initial user task, and subsequent repair assignments suit native continuation. |
|
||||
| [Observations](../../../audita/internal/framework/llm/observations.go) and [CLI composition](../../../audita/internal/cli/app_wiring.go) | Native execution totals with application scope and explicit nested-validator enrollment; standard CommandKit suites using shared loaders. |
|
||||
|
||||
Audita repairs individual transcript corrections in independent branches;
|
||||
Notarius's selected unit is a whole producer operation and typed artifact.
|
||||
Audita's wrappers, response-schema registry, per-proposal policies, and absence
|
||||
of a cache do not define Notarius requirements.
|
||||
|
||||
At the reviewed revision, focused LLM, prompt/schema, runner, and proposal-generation
|
||||
tests passed, together with subprocess checks for correction conversations,
|
||||
cancellation during repair, and repair-disabled/output selections using a local
|
||||
fake provider. These historical checks do not establish live-model quality or
|
||||
completion of Notarius's migration acceptance checks.
|
||||
|
||||
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md
|
||||
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md
|
||||
|
||||
Reference in New Issue
Block a user