From d515a7e49b4b8cb242378e7a133b0f14c3a2e73c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 18:53:07 +0000 Subject: [PATCH] Record conversation continuation design decisions --- docs/roadmap/conversation-continuation.md | 220 ++++++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 docs/roadmap/conversation-continuation.md diff --git a/docs/roadmap/conversation-continuation.md b/docs/roadmap/conversation-continuation.md new file mode 100644 index 0000000..41c0efd --- /dev/null +++ b/docs/roadmap/conversation-continuation.md @@ -0,0 +1,220 @@ +# Conversation Continuation And Application-Directed Repair + +Status: proposed feature roadmap; open decisions await discussion. No +implementation is authorized by this document. Recommendations below are not +accepted contracts. + +## Purpose And Scope + +This roadmap records PromptKit's design questions arising from +[Audita's feature request](audita-request.md). That request owns the consumer +workflow and requested acceptance scenarios; this document owns the proposed +PromptKit direction, discussion order, and eventual implementation sequencing. + +The proposed capabilities are: + +- Capture a rendered request and its final assistant content, then continue it + using ordinary configured prompts without reconstructing the original request. +- Allow independent continuations from one captured exchange. +- Coordinate bounded revisions for one branch using application-supplied + assessment and follow-up request construction. + +Continuation has direct value by removing consumer capture, reconstruction, +prefix comparison, and history assembly. The revision coordinator could also +remove repeated lifecycle work, provided its public contract remains small. +Audita would continue owning candidate selection, domain validation, feedback +meaning, branch scheduling, accepted sibling proposals, and transcript changes. +Its domain-specific repair code would not move wholesale into PromptKit. + +The request needs in-process use. Persistent sessions, restartable workflows, +conversation storage, and batch-item reconciliation are outside its scope. +The existing [prepared execution handle](../../prepared_execution.go) remains a +single-use execution capability; continuation needs its own semantics. + +## Open Decisions In Discussion Order + +### 1. What Does A Captured Exchange Contain? + +**Status: open.** Decide whether capture represents a logical conversation or +the complete sequence of provider requests, including structural repair. + +**Recommendation:** capture the complete rendered request, preserving message +content, roles, and cache metadata, followed by the final assistant content +returned by the run. An initial run that needed structural repair would retain +its final content without adding the internal corrective instructions to the +logical conversation. This matches Audita's requested history and keeps +transport attempts and structural repair separate from application turns. + +Make capture opt-in. A continuation would append its rendered follow-up messages +and final response to produce a new exchange, leaving the source exchange and +other branches unchanged. Previously rendered history would not require the +original input files or templates to be rendered again. + +**Alternative:** retain the actual corrective messages and responses that led +to the final output. This provides more execution detail, but increases retained +content and carries internal repair instructions into later model context. +Attempt diagnostics can serve that need separately if a consumer requires it. + +**Details to settle:** capture availability after structural validation +exhaustion or an operational failure; how callers request capture for ordinary +and prepared runs; and the ownership and engine-binding of captured exchanges. +The request's “full final assistant response” should refer to the content +supported by PromptKit, without silently adding a raw provider-envelope API. + +### 2. How Are Follow-Up Execution Settings Selected? + +**Status: open.** Decide which settings come from the follow-up request and +which, if any, carry forward from the originating exchange. + +**Recommendation:** preserve conversation content while resolving the new +profile and output contract through the follow-up prompt and ordinary request +overrides. The follow-up can therefore use a different model and response +schema without modifying the original exchange. Reuse existing selection and +precedence rules rather than adding another settings-resolution system. + +In particular, prefer the follow-up prompt's default profile when no explicit +profile is supplied, with the ordinary missing-profile behavior if neither is +available. Do not retain the original direct credential merely to retain +conversation history; resolve credential supply for the new execution. + +**Alternative:** inherit the originating execution's profile and settings unless +overridden. This is convenient for repeated turns against the same model, but +adds another precedence source and can make a follow-up prompt's defaults +surprising. Inheriting an old output contract is especially unsuitable when +proposal and revision responses have different shapes. + +**Details to settle:** session-ID inheritance versus follow-up template +resolution, explicit overrides, credential lifetime, and the ordering of +follow-up definition messages and caller-appended messages. Preserve authored +roles rather than silently rewriting follow-up system or developer messages. + +### 3. Where Does Application Assessment Run In The Capacity Lifecycle? + +**Status: open.** Select a lifecycle that permits application validators to +call the same engine and backend, including with capacity limited to one. + +The current [run flow](../internal/runner.md#run-flow) retains logical admission +through structural validation and repair. Releasing an active generation permit +alone does not release that admission lease. + +**Recommendation:** coordinate completed executions. Finish a revision run and +release its admission lease before invoking application assessment. Each later +revision receives ordinary admission. The coordinator itself holds no backend +capacity while awaiting application work. This reuses the existing execution +machinery and supports validators that make their own model calls. + +**Alternative:** extend the existing structural-repair lifecycle with application +callbacks and explicit lease release/reacquisition. This could share more loop +code, but would complicate admission ownership and combine operations with +different schemas and validation responsibilities. It would still have to meet +the same nested-call requirement. + +**Details to settle:** callback cancellation boundaries and the sequence of +request building, execution, response interpretation, and assessment. The +coordinator should remain scoped to one branch; applications retain decisions +about scheduling branches and stopping other work after a failure. + +### 4. How Are Assessment Outcomes And Failures Represented? + +**Status: open.** Define how acceptance, another requested revision, withdrawal, +unavailable assessment, exhaustion, and interruption are distinguished. + +**Recommendation:** use explicit application assessment outcomes separately +from operational errors. Applications interpret responses and feedback; +PromptKit records the selected disposition and coordinates subsequent work. +Accept supplied initial rejection findings without repeating the original +assessment. A revised candidate becomes accepted only after application +assessment accepts it. + +Return sequence diagnostics that preserve completed attempts when work stops, +while clearly distinguishing the latest candidate from an accepted candidate. +Cancellation and operational errors should retain useful Go error identities. +Applications decide whether a stopped branch affects independent work. + +**Alternative:** use an error-only assessment callback, treating nil as +acceptance and typed errors as rejection, withdrawal, or unavailability. This +looks smaller, but overloads errors with ordinary domain decisions and makes +continuation policy harder to understand. + +**Details to settle:** the small outcome vocabulary, who supplies each outcome, +callback/result types, result-plus-error behavior, zero-budget behavior, and +what consumes an additional-revision attempt when preparation or generation +fails. Keep this budget distinct from structural repair and transport retry. + +### 5. How Are Related Executions And Usage Accounted For? + +**Status: open.** Define accounting for the initial exchange, each revision, +its structural repairs and transport retries, and model calls made by +application validators. + +The current [observation boundary](../internal/observation.md) keeps nested +invocations independent. A callback's arbitrary model calls cannot simply be +assumed to belong to the enclosing revision's totals. + +**Recommendation:** preserve independent run accounting and add explicit +relationships between a sequence, its revision attempts, and associated runs. +Choose a small association mechanism, such as a scoped observation facility or +explicit execution references, so consumers need not rebuild lifecycle tracking. +Shared initial work should be referenced by branches and counted once when +reporting the combined operation. Report available usage with its existing +presence/completeness meaning; do not promise complete cost accounting. + +**Alternative:** have PromptKit automatically capture every nested run during an +assessment callback. This minimizes explicit association but can include +unrelated work and changes the existing invocation-isolation boundary. +Another simpler option is to report revision usage only and let applications +attach validator accounting; that leaves more work in Audita and needs checking +against its stated reporting requirement. + +**Details to settle:** association of concurrent validator calls, observer +composition, failure-path diagnostics, and which totals include which child +executions. A revision budget bounds revision attempts, not all provider calls +performed by structural repair, transport retries, or application validators. + +### 6. Where Should Continuation Inspection Live? + +**Status: open.** Clarify the request's references to inspection without changing +the established resource-inspection boundary. + +**Recommendation:** expose originating exchange, follow-up prompt, and execution +relationships in continuation details, prepared execution metadata, sequence +reports, and observations as appropriate. Keep `InspectPrompt`, `InspectProfile`, +and `InspectBackend` focused on their configured resources. Reuse the existing +approach to generically renderable inspection data where new details are exposed. + +**Alternative:** introduce a dedicated conversation inspection API. This may +be useful if a retained exchange needs a richer view, but should be justified +against a simple details method. Extending resource inspection with execution +history would give those APIs a different responsibility. + +**Details to settle:** the minimum inspectable metadata, how callers explicitly +access captured message content, and separation of content-bearing details from +metadata-only observations. Public type names and serialization details belong +in implementation planning after this boundary is settled. + +## Proposed Delivery And Validation + +Resolve the decisions above before selecting exact public signatures or writing +an implementation plan. The recommended sequence is: + +1. Deliver capture and independent continuation through the public facade, + allowing Audita to remove its reconstruction and prefix-verification work. +2. Build the single-branch revision coordinator on that capability, reusing + ordinary execution, validation, capacity, and observation machinery. + +Use [Audita's acceptance scenarios](audita-request.md#suggested-delivery-and-acceptance-scenarios) +as the consumer checks. Additional proposed coverage should establish: + +- The selected conversation history when the initial run needed structural + repair, including use of its full final assistant content. +- Continuation after the original input files change or disappear, without + altering the captured history. +- Independent branches retaining their own history and outcomes. +- Assessment calls against the same backend at capacity one, without capacity + retained by the coordinator blocking them. +- Associated validator usage and shared initial usage reported without + double-counting, including incomplete usage and operational failures. + +These are planning targets, not new implemented guarantees. Keep additional +mechanisms and constraints tied to the selected decisions and demonstrated +consumer needs.