From 8fb21b7daeffd68e099ee1c165debad41c27a684 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 18 Nov 2026 19:48:00 +0000 Subject: [PATCH] Record native continuation direction and retry simplifications --- .../promptkit-architecture-discussion.md | 137 +++++++++++++++--- 1 file changed, 114 insertions(+), 23 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 6f6ed296..35496c9c 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -5,7 +5,7 @@ Working notepad for reviewing the ownership boundaries in the [PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. -Items 1 and 2 record selected architectural directions. Detailed design, +Items 1–3 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. Item 7 records the completed Audita comparison and its implications for the @@ -141,29 +141,120 @@ implementation. The agreed direction has not yet been implemented. ## 3. Retry And Semantic Correction Ownership -Consider narrowing outer stage retries once PromptKit has exhausted its own -transport and structural-repair budgets. Notarius would retain application -assessment, semantic correction, fallback, and acceptance decisions. +**Discussion / decision:** Direction selected: adopt Audita's native continuation +approach. PromptKit owns captured exchanges, accumulated conversation history, +continuation execution, and turn-budget enforcement. Notarius supplies semantic +assessment, corrective findings, and acceptance or fallback decisions. Use native +PromptKit types directly, consistent with item 1; Audita's adapter and opaque +exchange wrapper are not part of the selected direction. -Questions: +The following is a proposed application of that direction. Correction scope, +response contracts, budget migration, and terminal policy details still need +agreement. -- Which operational failures justify repeating an entire stage? -- Should semantic correction preserve only the latest response or accumulate - conversation history? -- How should retries for non-LLM operations differ from LLM execution retries? -- Would a native orchestration API reduce machinery without obscuring typed - validation and terminal disposition? +### Proposed Flow -**Discussion / decision:** Pending. +1. Prepare the source context, references, and any request-local candidate-handle + mapping once for the producer operation. Run its initial native request with + exchange capture when continuation is enabled. Capture is independent of + detailed debug recording. +2. Inspect native structural validation, decode the response, and perform the + module's deterministic interpretation and safety checks. Run the configured + validation chain on a materialized candidate. A module can identify actionable + defects before a candidate is ready for that chain. +3. If correction is needed, call `Engine.Coordinate` with the initial exchange + and a configured `AdditionalTurns` budget. The callback selects a native + repair prompt and supplies actionable findings as variables. PromptKit retains + the original exchange and each subsequent repair turn. +4. Assess every delivered response, including the last permitted turn. Rebuild + the typed candidate from the fixed operation inputs and the revised response; + do not repeatedly apply revisions to an already modified artifact. A changed + candidate receives the complete configured validation chain again, including + validators that approved an earlier version. +5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides + whether to reject, fail, or evaluate an explicitly supported safe fallback. + Native `stopped` and `exhausted` outcomes describe coordination, not acceptance. + Cancellation never turns an undelivered response into an accepted candidate. -**Follow-up:** Evaluate behavior changes against +Start with one conversation per producer operation: a chunk-plan generation, +one lane's extraction of one chunk, or one lane's normalization pass. This aligns +with Notarius's whole-output validation boundary. Audita's per-proposal branches +are useful evidence, but per-entity repair would require its own combination and +validation design. It is not necessary to adopt native continuation. + +Repair prompts should supply a new user task, with enduring principles in the +original system instructions. Avoid replaying the initial task or embedding +initial-response-only constraints in system messages. A complete replacement +model response is a simple starting point; for normalization that can remain a +replacement reconciliation plan, rather than a model-generated durable artifact. +A different repair schema or profile is supported but not yet selected. Keep +source context and candidate-handle meanings stable throughout the conversation. +Accumulated history increases context and token use across turns; the semantic +turn budget should reflect that tradeoff. + +### Machinery To Remove Or Adapt + +| Current machinery | Proposed treatment | +| --- | --- | +| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. | +| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. | +| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. | +| Outer LLM retries after transport or structural repair exhaustion | Prefer settling the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs a concrete application reason. | +| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. | +| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. | +| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. | +| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. | + +Native transient retries, structural repair, and semantic additional turns remain +distinct controls, but Notarius need not implement all three loops. Existing +producer and validator `retries` settings cannot be mechanically renamed: their +current budgets cover different behavior. Non-LLM failure handling should be +reviewed separately rather than routed through conversation coordination. + +### Validation And Safe Fallback Still Have Application Meaning + +Keep source grounding, reference resolution, domain invariants, reconciliation +plan checks, typed application rules, and candidate isolation. In particular, +normalization must still reject unknown or overlapping candidate handles and +apply only safe groups. A final fallback can retain safe merges while leaving +unsafe groups separate; it must go through configured artifact validation and +terminal policy before it can advance. After native structural-repair exhaustion, +a supported deterministic fallback can be assessed directly instead of first +restarting the same model operation through a feedback-free module retry. Which earlier safe +candidate, if any, may survive a later failed turn remains an explicit policy +question; do not introduce implicit best-so-far acceptance. + +Retain the distinction between a rejection and unavailable validation. Under the +current `warn_continue` policy, incomplete validation can advance a structurally +valid candidate but prevents checkpoint reuse/publication, including for derived +results. Continuation adoption does not itself change that policy, safe-fallback +eligibility, or durable provenance. Keep raw conversation state in process and +out of ordinary manifests/checkpoints; explicit detailed debug remains separate. + +PromptKit should own validation and repair of the private model response. +Notarius's serialized schema validator currently receives the codec-produced +artifact and its artifact schema, which can differ from the model response. +Do not remove that boundary merely because both use JSON Schema. Audit actual +duplicate response checks individually; artifact validation and Go/domain +interpretation are not automatically redundant. Item 2's filesystem and schema +assembly decisions remain open. + +Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go), +[validation executor](../../internal/framework/pipeline/validation_executor.go), +[typed artifact validation](../../internal/framework/pipeline/runner_typed.go), +[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go), +[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go), +and the native continuation evidence in item 7. + +**Follow-up:** Agree correction scope and repair response contracts; settle budget +migration and terminal/fallback behavior; then reconcile the adoption roadmap. +Record the changed architecture in a successor to [ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and [ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). -PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct -replacement for the existing latest-response-only policy or full stage loop. -The Audita review in item 7 demonstrates application-owned validation inside -native continuation coordination. Evaluate adopting that model on its merits; -the existing Notarius history policy is subject to reconsideration under item 1. +Migration checks should cover accumulated history, fixed handle mappings, +revalidation, final-turn acceptance, native budget exhaustion, safe fallback, +validator failure, cancellation, and checkpoint eligibility. Replace tests of +the retired message-reconstruction protocol rather than preserving its mechanics. ## 4. Global Scheduling And Admission @@ -373,11 +464,11 @@ Item 2's native underlay direction has a working application precedent. Audita also offers evidence for ordinary embedded filesystems and per-request schema selection, while our filesystem and schema decisions remain open. -For item 3, native continuation is a practical alternative to the current -latest-response-only protocol. Keeping domain validation local does not require -keeping conversation assembly or the turn loop local. Decide whether Notarius -should adopt accumulated branch history, separate repair prompts, and revised -response contracts; none has been selected merely by this comparison. +The subsequent item 3 discussion selects native continuation and accumulated +conversation history. Keeping domain validation local does not require keeping +conversation assembly or the turn loop local. Item 3 owns the proposed Notarius +flow and the remaining repair-prompt, response-contract, and policy decisions; +the comparison alone did not settle those details. Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit integration. Item 5 remains independent upstream/application identity work.