Record native continuation direction and retry simplifications

This commit is contained in:
2026-11-18 19:48:00 +00:00
parent 5a527c789d
commit 8fb21b7dae

View File

@@ -5,7 +5,7 @@
Working notepad for reviewing the ownership boundaries in the
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
below in order, recording questions, evidence, decisions, and follow-up work.
Items 1 and 2 record selected architectural directions. Detailed design,
Items 1–3 record selected architectural directions. Detailed design,
read-only filesystem and schema decisions, and the remaining topics are still
under discussion.
Item 7 records the completed Audita comparison and its implications for the
@@ -141,29 +141,120 @@ implementation. The agreed direction has not yet been implemented.
## 3. Retry And Semantic Correction Ownership
Consider narrowing outer stage retries once PromptKit has exhausted its own
transport and structural-repair budgets. Notarius would retain application
assessment, semantic correction, fallback, and acceptance decisions.
**Discussion / decision:** Direction selected: adopt Audita's native continuation
approach. PromptKit owns captured exchanges, accumulated conversation history,
continuation execution, and turn-budget enforcement. Notarius supplies semantic
assessment, corrective findings, and acceptance or fallback decisions. Use native
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
exchange wrapper are not part of the selected direction.
Questions:
The following is a proposed application of that direction. Correction scope,
response contracts, budget migration, and terminal policy details still need
agreement.
- Which operational failures justify repeating an entire stage?
- Should semantic correction preserve only the latest response or accumulate
conversation history?
- How should retries for non-LLM operations differ from LLM execution retries?
- Would a native orchestration API reduce machinery without obscuring typed
validation and terminal disposition?
### Proposed Flow
**Discussion / decision:** Pending.
1. Prepare the source context, references, and any request-local candidate-handle
mapping once for the producer operation. Run its initial native request with
exchange capture when continuation is enabled. Capture is independent of
detailed debug recording.
2. Inspect native structural validation, decode the response, and perform the
module's deterministic interpretation and safety checks. Run the configured
validation chain on a materialized candidate. A module can identify actionable
defects before a candidate is ready for that chain.
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
and a configured `AdditionalTurns` budget. The callback selects a native
repair prompt and supplies actionable findings as variables. PromptKit retains
the original exchange and each subsequent repair turn.
4. Assess every delivered response, including the last permitted turn. Rebuild
the typed candidate from the fixed operation inputs and the revised response;
do not repeatedly apply revisions to an already modified artifact. A changed
candidate receives the complete configured validation chain again, including
validators that approved an earlier version.
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
whether to reject, fail, or evaluate an explicitly supported safe fallback.
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
Cancellation never turns an undelivered response into an accepted candidate.
**Follow-up:** Evaluate behavior changes against
Start with one conversation per producer operation: a chunk-plan generation,
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
with Notarius's whole-output validation boundary. Audita's per-proposal branches
are useful evidence, but per-entity repair would require its own combination and
validation design. It is not necessary to adopt native continuation.
Repair prompts should supply a new user task, with enduring principles in the
original system instructions. Avoid replaying the initial task or embedding
initial-response-only constraints in system messages. A complete replacement
model response is a simple starting point; for normalization that can remain a
replacement reconciliation plan, rather than a model-generated durable artifact.
A different repair schema or profile is supported but not yet selected. Keep
source context and candidate-handle meanings stable throughout the conversation.
Accumulated history increases context and token use across turns; the semantic
turn budget should reflect that tradeoff.
### Machinery To Remove Or Adapt
| Current machinery | Proposed treatment |
| --- | --- |
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
| Outer LLM retries after transport or structural repair exhaustion | Prefer settling the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs a concrete application reason. |
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
Native transient retries, structural repair, and semantic additional turns remain
distinct controls, but Notarius need not implement all three loops. Existing
producer and validator `retries` settings cannot be mechanically renamed: their
current budgets cover different behavior. Non-LLM failure handling should be
reviewed separately rather than routed through conversation coordination.
### Validation And Safe Fallback Still Have Application Meaning
Keep source grounding, reference resolution, domain invariants, reconciliation
plan checks, typed application rules, and candidate isolation. In particular,
normalization must still reject unknown or overlapping candidate handles and
apply only safe groups. A final fallback can retain safe merges while leaving
unsafe groups separate; it must go through configured artifact validation and
terminal policy before it can advance. After native structural-repair exhaustion,
a supported deterministic fallback can be assessed directly instead of first
restarting the same model operation through a feedback-free module retry. Which earlier safe
candidate, if any, may survive a later failed turn remains an explicit policy
question; do not introduce implicit best-so-far acceptance.
Retain the distinction between a rejection and unavailable validation. Under the
current `warn_continue` policy, incomplete validation can advance a structurally
valid candidate but prevents checkpoint reuse/publication, including for derived
results. Continuation adoption does not itself change that policy, safe-fallback
eligibility, or durable provenance. Keep raw conversation state in process and
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
PromptKit should own validation and repair of the private model response.
Notarius's serialized schema validator currently receives the codec-produced
artifact and its artifact schema, which can differ from the model response.
Do not remove that boundary merely because both use JSON Schema. Audit actual
duplicate response checks individually; artifact validation and Go/domain
interpretation are not automatically redundant. Item 2's filesystem and schema
assembly decisions remain open.
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
[validation executor](../../internal/framework/pipeline/validation_executor.go),
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
and the native continuation evidence in item 7.
**Follow-up:** Agree correction scope and repair response contracts; settle budget
migration and terminal/fallback behavior; then reconcile the adoption roadmap.
Record the changed architecture in a successor to
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
replacement for the existing latest-response-only policy or full stage loop.
The Audita review in item 7 demonstrates application-owned validation inside
native continuation coordination. Evaluate adopting that model on its merits;
the existing Notarius history policy is subject to reconsideration under item 1.
Migration checks should cover accumulated history, fixed handle mappings,
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
validator failure, cancellation, and checkpoint eligibility. Replace tests of
the retired message-reconstruction protocol rather than preserving its mechanics.
## 4. Global Scheduling And Admission
@@ -373,11 +464,11 @@ Item 2's native underlay direction has a working application precedent. Audita
also offers evidence for ordinary embedded filesystems and per-request schema
selection, while our filesystem and schema decisions remain open.
For item 3, native continuation is a practical alternative to the current
latest-response-only protocol. Keeping domain validation local does not require
keeping conversation assembly or the turn loop local. Decide whether Notarius
should adopt accumulated branch history, separate repair prompts, and revised
response contracts; none has been selected merely by this comparison.
The subsequent item 3 discussion selects native continuation and accumulated
conversation history. Keeping domain validation local does not require keeping
conversation assembly or the turn loop local. Item 3 owns the proposed Notarius
flow and the remaining repair-prompt, response-contract, and policy decisions;
the comparison alone did not settle those details.
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit
integration. Item 5 remains independent upstream/application identity work.