From ae8e3ec6981ec93298392183adb43d4e2f0b35d0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 18 Nov 2026 19:54:55 +0000 Subject: [PATCH] Record agreement on native continuation and retry ownership --- .../promptkit-architecture-discussion.md | 57 ++++++++++++------- 1 file changed, 36 insertions(+), 21 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 35496c9c..3d7d9bfd 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -18,9 +18,9 @@ remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need. No application changes have been implemented through this notepad. The adoption -roadmap still needs reconciliation with the selected direction; its adapter -preservation language is subject to this review. Significant changes to accepted -architecture need successor ADRs under the +roadmap still needs reconciliation with the selected direction, including its +adapter-preservation and latest-response-only language. Significant changes to +accepted architecture need successor ADRs under the [documentation policy](../policy/documentation.md#architecture-decision-records). Current boundaries are described in [Architecture](../policy/architecture.md) and [LLM Runtime Internals](../internal/llm.md). @@ -141,18 +141,21 @@ implementation. The agreed direction has not yet been implemented. ## 3. Retry And Semantic Correction Ownership -**Discussion / decision:** Direction selected: adopt Audita's native continuation +**Discussion / decision:** Direction agreed: adopt Audita's native continuation approach. PromptKit owns captured exchanges, accumulated conversation history, continuation execution, and turn-budget enforcement. Notarius supplies semantic assessment, corrective findings, and acceptance or fallback decisions. Use native PromptKit types directly, consistent with item 1; Audita's adapter and opaque exchange wrapper are not part of the selected direction. -The following is a proposed application of that direction. Correction scope, -response contracts, budget migration, and terminal policy details still need -agreement. +The flow and removal/adaptation direction below are agreed. Begin with one +conversation per producer operation and complete replacement model responses. +Retain application assessment, safe-fallback decisions, artifact validation, +and checkpoint eligibility. Detailed response contracts, budget migration, and +the open policy questions below remain to be resolved. No application changes +have been implemented. -### Proposed Flow +### Agreed Flow 1. Prepare the source context, references, and any request-local candidate-handle mapping once for the producer operation. Run its initial native request with @@ -184,9 +187,9 @@ validation design. It is not necessary to adopt native continuation. Repair prompts should supply a new user task, with enduring principles in the original system instructions. Avoid replaying the initial task or embedding -initial-response-only constraints in system messages. A complete replacement -model response is a simple starting point; for normalization that can remain a -replacement reconciliation plan, rather than a model-generated durable artifact. +initial-response-only constraints in system messages. Begin with complete +replacement model responses; for normalization, the response can remain a +reconciliation plan that Notarius applies to construct the durable artifact. A different repair schema or profile is supported but not yet selected. Keep source context and candidate-handle meanings stable throughout the conversation. Accumulated history increases context and token use across turns; the semantic @@ -194,12 +197,12 @@ turn budget should reflect that tradeoff. ### Machinery To Remove Or Adapt -| Current machinery | Proposed treatment | +| Current machinery | Agreed direction | | --- | --- | | `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. | | Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. | | `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. | -| Outer LLM retries after transport or structural repair exhaustion | Prefer settling the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs a concrete application reason. | +| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. | | `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. | | `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. | | `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. | @@ -220,9 +223,9 @@ apply only safe groups. A final fallback can retain safe merges while leaving unsafe groups separate; it must go through configured artifact validation and terminal policy before it can advance. After native structural-repair exhaustion, a supported deterministic fallback can be assessed directly instead of first -restarting the same model operation through a feedback-free module retry. Which earlier safe -candidate, if any, may survive a later failed turn remains an explicit policy -question; do not introduce implicit best-so-far acceptance. +restarting the same model operation through a feedback-free module retry. Which +earlier safe candidate, if any, may survive a later failed turn remains an explicit +policy question; do not introduce implicit best-so-far acceptance. Retain the distinction between a rejection and unavailable validation. Under the current `warn_continue` policy, incomplete validation can advance a structurally @@ -246,9 +249,21 @@ Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_att [semantic reconciliation](../../internal/framework/semanticreconcile/engine.go), and the native continuation evidence in item 7. -**Follow-up:** Agree correction scope and repair response contracts; settle budget -migration and terminal/fallback behavior; then reconcile the adoption roadmap. -Record the changed architecture in a successor to +### Open Implementation And Policy Questions + +- Exact repair prompt definitions, response schemas, and profile selection within + the agreed complete-replacement approach. +- Migration of producer and validator `retries` settings to native controls, + including configuration names, defaults, and compatibility. +- Detailed terminal/fallback handling, including whether an earlier safe + candidate may survive a later failed turn. +- Failure handling for non-LLM operations and any justified recovery beyond + native execution budgets. +- Item 2's deferred filesystem and schema assembly choices, and items 5–6's + execution identity and durable bookkeeping design. + +**Follow-up:** Resolve the remaining details above, then reconcile the adoption +roadmap. Record the changed architecture in a successor to [ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and [ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). Migration checks should cover accumulated history, fixed handle mappings, @@ -466,8 +481,8 @@ selection, while our filesystem and schema decisions remain open. The subsequent item 3 discussion selects native continuation and accumulated conversation history. Keeping domain validation local does not require keeping -conversation assembly or the turn loop local. Item 3 owns the proposed Notarius -flow and the remaining repair-prompt, response-contract, and policy decisions; +conversation assembly or the turn loop local. Item 3 owns the agreed Notarius +flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details. Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit