736 lines
41 KiB
Markdown
736 lines
41 KiB
Markdown
# Conversation Continuation And Application-Directed Interaction
|
|
|
|
Status: continuation and coordinator behavioral decisions accepted; optional
|
|
execution groups accepted as a separable addition with open decisions below.
|
|
No implementation is authorized by this document. Exact public types and
|
|
signatures remain for implementation planning.
|
|
|
|
## North Star: Content-Neutral, Generally Useful Capabilities
|
|
|
|
**Accepted direction:** evaluate each capability for its usefulness across
|
|
different downstream applications. PromptKit must be able to implement it and
|
|
explain its value without assuming or inferring the meaning of the content
|
|
passed through prompts and responses.
|
|
|
|
PromptKit can own conversation and execution mechanics: captured messages,
|
|
continuations, independent branches, execution relationships, budgets,
|
|
cancellation, observations, and usage. Applications own the interpretation of
|
|
responses and their reasons for choosing the next action. Repair, analysis,
|
|
clarification, and branching exploration are examples of consumer workflows
|
|
that could use the same mechanisms.
|
|
|
|
Decision 3 selects an optional sequential coordinator built on independently
|
|
usable continuation primitives. Its value is generally useful execution and
|
|
conversation management, rather than implementation of Audita's repair loop.
|
|
It executes application-directed interactions without needing to understand why
|
|
the application requests another turn or stops. Applications retain orchestration
|
|
across branches; that initial scope does not prohibit future capabilities.
|
|
|
|
Apply two questions to proposed public concepts and implementation machinery:
|
|
|
|
1. Can their value be explained using conversation and execution concepts alone?
|
|
2. Can different applications use them without translating their domain into
|
|
repair-specific concepts?
|
|
|
|
Audita supplies concrete evidence of need and acceptance examples; its workflow
|
|
does not define PromptKit's abstraction. Terms such as rejected candidate,
|
|
accepted revision, and withdrawal belong to the consuming application. This
|
|
direction concerns the new capabilities and does not rename or expand
|
|
PromptKit's existing structural-output repair functionality.
|
|
|
|
## Purpose And Scope
|
|
|
|
This roadmap records PromptKit's design questions arising from
|
|
[Audita's feature request](audita-request.md). That request owns the consumer
|
|
workflow and requested acceptance scenarios; this document owns the proposed
|
|
PromptKit direction, discussion order, and eventual implementation sequencing.
|
|
|
|
The proposed capabilities are:
|
|
|
|
- Capture a rendered request and its final assistant content, then continue it
|
|
using ordinary configured prompts without reconstructing the original request.
|
|
- Allow independent continuations from one captured exchange.
|
|
- Provide an optional sequential coordinator for bounded looping using
|
|
application-supplied next actions and follow-up requests.
|
|
- Provide optional execution groups for explicitly selected runs and summaries
|
|
of their available execution facts, independently of conversation continuation.
|
|
|
|
Continuation has direct value by removing consumer capture, reconstruction,
|
|
prefix comparison, and history assembly. The coordinator centralizes turn
|
|
progression, cancellation, and budgets while reusing ordinary execution
|
|
observations. Captured exchanges supply the minimal parent relationship selected
|
|
in Decision 5. Its value comes from removing repeated lifecycle work, beyond
|
|
merely replacing a consumer's loop counter.
|
|
Audita would continue owning candidate selection, domain validation, feedback
|
|
meaning, branch scheduling, accepted sibling proposals, and transcript changes.
|
|
Its domain-specific repair code would not move wholesale into PromptKit.
|
|
|
|
The request needs in-process use. Persistent sessions, restartable workflows,
|
|
conversation storage, and batch-item reconciliation are outside its scope.
|
|
The existing [prepared execution handle](../../prepared_execution.go) remains a
|
|
single-use execution capability; continuation needs its own semantics.
|
|
|
|
## Decisions In Discussion Order
|
|
|
|
### 1. What Does A Captured Exchange Contain?
|
|
|
|
**Status: accepted; exact capture API remains for implementation planning.** Capture
|
|
represents the logical conversation. Structural-output repair attempts and
|
|
corrective instructions are internal execution details, not conversation turns.
|
|
|
|
**Accepted approach:** capture the complete rendered request, preserving message
|
|
content, roles, and cache metadata, followed by the final assistant content
|
|
returned by the run. When structural repair succeeds, retain only the final
|
|
corrected assistant content, excluding intermediate candidates and corrective
|
|
instructions from the logical conversation. This supplies reusable context for
|
|
subsequent turns while keeping transport attempts and structural repair separate
|
|
from application turns. It also matches Audita's requested history.
|
|
|
|
**Accepted capture availability:** when capture is requested, follow the ordinary
|
|
distinction between a completed result and an operational error:
|
|
|
|
| Execution outcome | Capture behavior |
|
|
| --- | --- |
|
|
| Completed response that satisfies its output contract | Capture the logical exchange. |
|
|
| Completed response with failed structural validation, including repair exhaustion | Capture the final returned response; preserve the failed validation status in the ordinary result. |
|
|
| Operational error with no completed result | Produce no new exchange; the preceding exchange remains usable. |
|
|
| Completed response followed by cancellation before coordinator callback delivery | Preserve the undelivered response under Decision 4's accepted optional-result behavior. |
|
|
|
|
Capture does not imply validation success or application acceptance. The
|
|
coordinator delivers completed results with their ordinary validation status;
|
|
the application decides whether to stop or continue, potentially using another
|
|
output contract. Failed structural validation is not reclassified as an
|
|
operational error. Only the final returned content enters the logical history,
|
|
never intermediate structural-repair responses or corrective instructions.
|
|
|
|
**Accepted opt-in behavior:** support capture per execution through both ordinary
|
|
and prepared execution. Ordinary execution returns the captured exchange
|
|
alongside its usual result. For prepared execution, retain the capture choice
|
|
with the prepared handle and create the exchange when that exact execution
|
|
completes. Preparation alone produces no completed exchange because it has no
|
|
assistant response. A continuation operation implies capture so it can produce
|
|
the next reusable exchange.
|
|
|
|
Construct capture from the request actually executed. Consumers must not need
|
|
to reconstruct history or pair independently prepared data with a response.
|
|
An ordinary request option is the preferred API direction; separate capture
|
|
methods are also viable. Select the precise representation during API planning.
|
|
|
|
**Accepted ownership and lifetime:** a captured exchange is an immutable,
|
|
reusable snapshot held by the caller. Continuing it appends the rendered
|
|
follow-up messages and final response to produce a new exchange, leaving the
|
|
source exchange and other branches unchanged. Independent branches may share
|
|
their starting exchange. Inspection values cannot mutate retained history.
|
|
Previously rendered history does not require the original input files or
|
|
templates to be rendered again.
|
|
|
|
A completed exchange retains no backend admission lease, direct credential,
|
|
or execution callback. Ordinary Go memory management handles its lifetime when
|
|
references are released; it adds no `Close` or `Discard` lifecycle. The existing
|
|
single-use prepared execution lifecycle remains separate.
|
|
|
|
**Accepted engine independence:** another engine instance in the same process
|
|
may continue a captured exchange. The receiving engine resolves the new prompt,
|
|
profile, output contract, and credentials under Decision 2's rules. It does not
|
|
need the originating engine to rerender prior messages. This permits retaining
|
|
conversations while an application rebuilds its engine with new configuration.
|
|
It does not imply persistence, restartable sessions, serialization for later
|
|
resumption, or cross-process portability.
|
|
|
|
**Alternative considered, not selected:** retain the corrective messages and
|
|
responses that led to the final output. This provides more execution detail,
|
|
but increases retained content and carries internal repair instructions into
|
|
later model context.
|
|
Fine-grained execution tracing is outside this feature's purpose. Existing
|
|
attempt and usage observations remain separate from logical conversation
|
|
content; this decision adds no requirement to capture internal repair text.
|
|
|
|
**Additional alternatives considered, not selected:**
|
|
|
|
- Capture only responses that satisfy their output contract. This would offer a
|
|
validated-output condition but unnecessarily restrict continuing from otherwise
|
|
available conversation content. Applications retain that choice.
|
|
- Bind exchanges to their originating engine. Prepared executions need ownership
|
|
of execution-specific state, but captured conversation data has no equivalent
|
|
requirement under the selected fresh-execution settings model.
|
|
- Use a mutable conversation object. This would make appending convenient but
|
|
require explicit copying or forking to preserve independent branches. Immutable
|
|
snapshots fit the selected branch behavior directly.
|
|
|
|
**Implementation planning:** select the opt-in API and exchange representation
|
|
within these behaviors. The request's “full final assistant response” refers to
|
|
content supported by PromptKit, without adding a raw provider-envelope API.
|
|
|
|
### 2. How Are Follow-Up Execution Settings Selected?
|
|
|
|
**Status: accepted.** A continuation supplies existing conversational context
|
|
to a normally configured new execution. Conversation history does not introduce
|
|
another source of execution-setting inheritance.
|
|
|
|
**Accepted approach:** preserve conversation content while resolving the new
|
|
profile and output contract through the follow-up prompt and ordinary request
|
|
overrides. The follow-up can therefore use a different model and response
|
|
schema without modifying the original exchange. Explicit request overrides use
|
|
ordinary precedence, including complete replacement of the output contract
|
|
when supplied. Reuse existing selection and precedence rules.
|
|
|
|
In particular, prefer the follow-up prompt's default profile when no explicit
|
|
profile is supplied, with the ordinary missing-profile behavior if neither is
|
|
available. Do not retain the original direct credential merely to retain
|
|
conversation history; resolve credential supply for the new execution using
|
|
the ordinary credential rules.
|
|
|
|
Select the session ID using the new request's explicit value or the follow-up
|
|
prompt's session template under the existing rules. Do not implicitly inherit
|
|
the preceding execution's session ID. An application can explicitly supply the
|
|
same session ID across turns. Conversation relationships are recorded separately
|
|
and do not depend on a shared session ID.
|
|
|
|
**Alternative considered, not selected:** inherit the originating execution's
|
|
profile and settings unless overridden. This is convenient for repeated turns
|
|
against the same model, but adds another precedence source and can make a
|
|
follow-up prompt's defaults surprising. Inheriting an old output contract is
|
|
especially unsuitable when successive turns expect different response shapes.
|
|
An explicit inheritance option is not selected for this feature; reconsider it
|
|
only if ordinary selection leaves a demonstrated usability gap.
|
|
|
|
**Implementation planning detail:** specify message assembly order for captured
|
|
history, follow-up definition messages, and caller-appended messages, preserving
|
|
authored roles. This does not reopen the accepted execution-setting precedence.
|
|
|
|
### 3. Should PromptKit Coordinate Turns, And Where Does Application Work Run?
|
|
|
|
**Status: accepted; exact public types remain for implementation planning.** Provide
|
|
independently usable continuation primitives and an optional sequential
|
|
coordinator for one branch. The application selects each next request or decides
|
|
to stop. PromptKit manages turn progression and execution bookkeeping under the
|
|
caller's chosen additional-turn budget and context.
|
|
|
|
Applications can create several independent continuations or coordinators from
|
|
the same exchange. Scheduling branches, joining results, and deciding whether a
|
|
branch failure should stop other work remain application responsibilities in
|
|
this initial scope. Consumers can also use continuation primitives in their own
|
|
loops without adopting the coordinator.
|
|
|
|
The current [run flow](../internal/runner.md#run-flow) retains logical admission
|
|
through structural validation and repair. Releasing an active generation permit
|
|
alone does not release that admission lease.
|
|
|
|
**Accepted execution lifecycle:** compose completed executions. Finish a turn
|
|
and release its logical admission lease before invoking application work. Each
|
|
later turn receives ordinary admission. The coordinator holds no backend
|
|
capacity while awaiting application work. Callbacks can inspect responses,
|
|
consult application data, wait for application input, or make model calls against
|
|
the same engine and backend, including at capacity one. PromptKit need not
|
|
interpret the purpose of that work.
|
|
|
|
**Alternatives considered, not selected:**
|
|
|
|
- Provide only continuation and observation primitives. This would keep the
|
|
public API smaller, but leave repeated lifecycle and accounting coordination
|
|
downstream. Independent primitive use remains supported alongside the selected
|
|
coordinator.
|
|
- Provide a coordinator for an entire branching tree. This could manage more
|
|
work, but introduces scheduling, concurrency, joining, and failure-policy
|
|
questions without demonstrated shared answers across consumers. Applications
|
|
can build those workflows from the selected branch-level capabilities.
|
|
|
|
**Implementation planning:** select exact request/stop and outcome types under
|
|
Decision 4's accepted response delivery, turn-budget consumption, zero-turn
|
|
behavior, cancellation, and minimal error/result semantics.
|
|
Decision 5 records the selected execution association and accounting approach.
|
|
The sketches below illustrate the selected responsibility split; they do not
|
|
settle the remaining public contracts.
|
|
|
|
#### Illustrative Consumer Sketches
|
|
|
|
These are pseudocode, not implemented Go APIs or selected exported names.
|
|
`coordinate`, `Continue`, `Stop`, and callback state fields are illustrative.
|
|
`request` represents an ordinary PromptKit request, with the selection rules
|
|
accepted in Decision 2. Each sketch starts from an already captured exchange.
|
|
The callback can keep application state in a closure; PromptKit need not define
|
|
a universal domain-state or feedback schema.
|
|
|
|
**Analysis loop:** the application examines the latest response and selects a
|
|
question or decides that its investigation is complete.
|
|
|
|
```text
|
|
analysis = application.start_analysis(initial_exchange)
|
|
|
|
choose_next(ctx, turn):
|
|
if turn.new_result exists:
|
|
analysis.record(turn.new_result)
|
|
|
|
question = application.choose_next_question(analysis)
|
|
if question is absent:
|
|
return Stop()
|
|
|
|
return Continue(request(
|
|
prompt_id = "research/investigate",
|
|
vars = {"question": question}
|
|
))
|
|
|
|
termination, error = coordinate(
|
|
ctx, start = initial_exchange,
|
|
additional_turns = 3, next = choose_next
|
|
)
|
|
application.present_analysis(analysis, termination, error)
|
|
```
|
|
|
|
PromptKit preserves the history and performs each selected execution. It does
|
|
not decide whether the response answered the question, whether the evidence is
|
|
sufficient, or what to investigate next. The application callback may consult
|
|
other sources or call the engine before returning its next action.
|
|
|
|
**Audita branch:** the application starts with one already-assessed correction
|
|
and its rejection findings. It interprets and assesses each new response; the
|
|
coordinator uses the same generic controls as the analysis loop.
|
|
|
|
```text
|
|
branch = audita.branch(candidate, existing_findings)
|
|
|
|
choose_next(ctx, turn):
|
|
if turn.new_result exists:
|
|
response = audita.decode_response(turn.new_result)
|
|
if audita.is_withdrawal(response):
|
|
branch.mark_withdrawn()
|
|
return Stop()
|
|
|
|
branch.update_candidate(response)
|
|
assessment = audita.assess(ctx, engine, branch.candidate)
|
|
branch.record(assessment)
|
|
if audita.should_stop(assessment):
|
|
return Stop()
|
|
|
|
return Continue(request(
|
|
prompt_id = branch.configured_follow_up_prompt,
|
|
vars = audita.follow_up_variables(branch)
|
|
))
|
|
|
|
termination, error = coordinate(
|
|
ctx, start = initial_exchange,
|
|
additional_turns = 3, next = choose_next
|
|
)
|
|
audita.record_branch_outcome(branch, termination, error)
|
|
```
|
|
|
|
The first callback uses existing findings, so initial batch validation is not
|
|
repeated. `audita.assess` may make model calls after the turn's admission lease
|
|
has been released. Only Audita decides that a correction is accepted, withdrawn,
|
|
or unavailable, and retains those meanings in its own state. Further corrections
|
|
can start separate coordinators from the same initial exchange; accepted sibling
|
|
corrections and other branches remain with Audita.
|
|
|
|
Both sketches omit application-specific error handling, including checks for
|
|
failed structural validation before schema-specific decoding, and depict normal
|
|
turn progression. The final permitted response follows the same callback path as
|
|
earlier responses, as accepted in Decision 4. That decision also defines zero-turn
|
|
budgets, cancellation around callback delivery, and a minimal termination result.
|
|
The sketches do not imply an archive of responses in that result: applications
|
|
already receive each completed response through the callback. Completed work
|
|
must not be mistaken for application acceptance of its content.
|
|
The concrete interface should let both examples use the same coordination
|
|
mechanism without manual message-history assembly or per-turn lifecycle
|
|
bookkeeping. Calls made inside callbacks retain independent observations;
|
|
applications explicitly group them as selected in Decision 5.
|
|
|
|
### 4. How Are Application Next Actions And Execution Outcomes Represented?
|
|
|
|
**Status: accepted; exact public types remain for implementation planning.**
|
|
The coordinator uses content-neutral next actions and distinguishes
|
|
application-directed completion from exhaustion.
|
|
|
|
**Accepted normal flow:** deliver each completed response once through the
|
|
ordinary application callback, including the last permitted response. If the
|
|
callback returns `Stop`, report application-directed completion. If it requests
|
|
another turn and none remain, report exhaustion without executing that request.
|
|
Using the entire budget does not itself imply exhaustion; exhaustion means the
|
|
application requested additional work that the budget would not permit.
|
|
|
|
This gives consumers one response-processing path, including any application
|
|
validation, without a separate path for the final response. Internal transport
|
|
retries and structural repairs do not produce additional conversation-response
|
|
callbacks. Cancellation and operational errors can interrupt normal delivery as
|
|
specified below. `Stop` and other API spellings remain illustrative until public
|
|
types are selected.
|
|
|
|
**Alternative considered, not selected:** stop immediately when the turn budget
|
|
is used and expose the final response only in the report. This avoids another
|
|
callback but requires separate consumer processing of the final response.
|
|
|
|
**Accepted zero-turn behavior:** with an active context, invoke the initial
|
|
callback once even when the additional-turn budget is zero. The callback receives
|
|
the starting exchange and knows that no additional turns remain; this is not
|
|
delivery of a newly generated response. It can use existing application state
|
|
without repeating earlier assessment.
|
|
|
|
- `Stop` produces application-directed completion.
|
|
- Requesting another turn produces exhaustion without executing the request.
|
|
- Returning an error produces an operational failure, subject to the
|
|
cancellation checks below.
|
|
|
|
Zero bounds coordinator-driven continuation executions, not work inside the
|
|
application callback. The callback can still consult data or make model calls.
|
|
Applications wanting to disable all such work can skip invoking the coordinator.
|
|
The alternative of treating zero as an immediate no-op with no callback is not
|
|
selected: it would introduce special behavior and would not distinguish an
|
|
already-complete interaction from one requesting more work.
|
|
|
|
**Accepted turn-budget consumption:** at most N additional continuation
|
|
executions may be started. The first operational error ends coordination.
|
|
Starting a continuation consumes one turn even if preparation or generation
|
|
subsequently fails. Callback invocations do not consume turns, and internal
|
|
transport retries and structural repairs remain within the same turn. The
|
|
starting exchange is outside the additional-turn budget. Requesting a turn
|
|
when none remain produces exhaustion without starting an execution.
|
|
|
|
For example, with a three-turn budget, a preparation failure in the second
|
|
continuation returns that error and ends coordination. It does not start a
|
|
third continuation or report exhaustion. The budget does not measure provider
|
|
calls, work performed inside application callbacks, or cost.
|
|
|
|
Counting only completed continuations was considered and not selected. Because
|
|
operational errors already stop coordination, that alternative would primarily
|
|
change bookkeeping. Use the straightforward executions-started interpretation
|
|
without introducing public counters, budget refunds, or recovery machinery for
|
|
this purpose.
|
|
|
|
**Accepted cancellation behavior:** check the caller's context immediately
|
|
before callback invocation, including the initial callback. If cancellation is
|
|
already observed, do not invoke the callback; return the context error with its
|
|
identity preserved. If the coordinator already obtained a completed response
|
|
that has not been delivered to the callback, preserve that single response in
|
|
the optional result described below. This does not imply application processing
|
|
or acceptance, or require recovering a response that a failed underlying
|
|
execution did not return.
|
|
|
|
Cancellation can race with callback invocation. The callback receives the same
|
|
context and is responsible for cooperating with cancellation during its work.
|
|
PromptKit checks the context again after the callback returns and before starting
|
|
another execution. It does not forcibly terminate callback code. Thus ordinary
|
|
response delivery is once per completed turn during normal progression, while
|
|
cancellation can interrupt that progression. Preserving available completed work
|
|
on interruption does not create a separate final-response processing path during
|
|
normal budget exhaustion.
|
|
|
|
The alternative of invoking the callback after cancellation with a canceled
|
|
context is not selected. It could help callbacks that only record results, but
|
|
callbacks may perform substantial application work or model calls; PromptKit
|
|
cannot assume they are merely bookkeeping.
|
|
|
|
**Accepted operational-failure behavior:** each continuation is an ordinary
|
|
execution supplied with captured conversation history. Applications receive
|
|
completed responses through the callback before another continuation begins.
|
|
An execution or callback error stops coordination and propagates to the caller,
|
|
preserving useful Go error identities. PromptKit does not automatically replay
|
|
the failed continuation or callback. Existing transport retries and structural
|
|
repair remain within ordinary execution. Earlier callback deliveries and
|
|
application work remain intact; applications decide whether the failure affects
|
|
other branches or the overall workflow.
|
|
|
|
**Accepted minimal return contract:** provide only the following information:
|
|
|
|
- An ordinary Go error for execution failures, callback failures, and
|
|
cancellation, preserving `errors.Is`/`errors.As` behavior.
|
|
- A small outcome value distinguishing application-directed stopping from
|
|
budget exhaustion on normal termination. These outcomes are not errors.
|
|
- An optional completed response when cancellation prevented its delivery to
|
|
the callback. Reuse existing response types where practical.
|
|
|
|
Delivery means the callback was invoked with the response; it does not imply
|
|
the application finished processing or accepted it. If that callback returns
|
|
an error, the response is not classified as undelivered or automatically
|
|
delivered again. The coordinator returns no archive of previously delivered
|
|
responses. A small optional result accommodates the undelivered-response case;
|
|
exact Go declarations remain for implementation planning.
|
|
|
|
Failure-stage fields, additional run-ID references, a separate failure
|
|
classification hierarchy, and aggregate reporting are not part of this minimal
|
|
termination contract. They require a demonstrated consumer need before addition.
|
|
This does not remove fields from existing response values or observation APIs.
|
|
Decision 5 reuses ordinary run accounting without expanding this result.
|
|
Decision 6 exposes captured-exchange details without expanding the coordinator's
|
|
termination result.
|
|
|
|
Audita retains its own meanings for acceptance, rejection, withdrawal, and
|
|
assessment unavailability. A normal stop does not assert any of those outcomes.
|
|
|
|
**Alternative considered, not selected for normal termination:** use callback
|
|
errors to signal ordinary termination as well as operational failure. This can
|
|
reduce the number of result types, but makes normal control flow harder to
|
|
distinguish from failure. Normal stopping uses an explicit action, while failures
|
|
use ordinary Go errors. A broader structured termination report was also
|
|
considered and not selected: its additional metadata is unnecessary for the
|
|
demonstrated minimal contract.
|
|
|
|
**Implementation planning:** select exact types for next actions, outcomes, and
|
|
the optional result under these accepted behaviors. As selected in Decision 1,
|
|
a completed run with failed structural validation produces a reusable exchange
|
|
when capture is requested and retains its ordinary validation result. That
|
|
result follows the normal response callback path, subject to cancellation;
|
|
the operational-error behavior does not reclassify it.
|
|
|
|
### 5. How Are Related Executions And Usage Accounted For?
|
|
|
|
**Status: accepted; exact placement of relationship metadata remains for
|
|
implementation planning.** Reuse ordinary per-run accounting, expose a minimal
|
|
parent relationship for continuations, and let applications select accounting
|
|
membership explicitly. The separable execution-group addition below supplies
|
|
optional summaries without expanding the coordinator. This applies to both
|
|
standalone continuations and coordinated turns.
|
|
|
|
The current [observation boundary](../internal/observation.md) keeps nested
|
|
invocations independent. A callback's arbitrary model calls cannot simply be
|
|
assumed to belong to the enclosing interaction's totals.
|
|
|
|
**Accepted per-run accounting:** each continuation reports its own usage and
|
|
ordinary execution observations, including the existing distinctions between
|
|
transport attempts, structural repairs, and available failure-path diagnostics.
|
|
Keep existing usage presence/completeness semantics. The coordinator adds no
|
|
second accounting system or cumulative totals to its termination result.
|
|
|
|
An initial exchange A that produces branches B and C contributes its usage only
|
|
to A. B and C do not inherit that usage. Applications reporting the combined work
|
|
can count A, B, and C once each. Usage reported for sending captured history again
|
|
in B or C belongs to those new executions and still counts. Reusing content does
|
|
not eliminate newly reported usage. Existing observation totals must not be
|
|
added to the attempt events they already summarize.
|
|
|
|
**Accepted relationship metadata:** retain the originating run identity with
|
|
each captured exchange and expose the immediate parent relationship in
|
|
continuation metadata. Use existing run identifiers without introducing mandatory
|
|
conversation, branch, or sequence identifiers. This supplies information that
|
|
PromptKit knows directly without requiring consumers to reconstruct which
|
|
exchange each continuation extends. Its exact public placement remains for
|
|
implementation planning and need not enlarge the coordinator's termination
|
|
result.
|
|
|
|
**Accepted application boundary:** model calls made inside a callback remain
|
|
independently observed and accounted for. PromptKit does not infer that they
|
|
belong to a continuation or aggregate merely because they execute inside its
|
|
callback. Applications explicitly select which runs belong together, including
|
|
concurrent calls, and may use the optional execution groups below to collect and
|
|
summarize those facts. This revises the earlier choice to leave all aggregation
|
|
to applications; membership and domain interpretation remain application-owned.
|
|
Audita retains assembly of its overall usage report and mappings to its domain
|
|
identifiers. PromptKit supplies execution facts without owning an
|
|
application-wide reporting model or promising complete cost accounting.
|
|
|
|
**Alternatives considered, not selected:**
|
|
|
|
- Automatically group nested calls. This could simplify obtaining an overall
|
|
total, but requires rules about which calls belong together and how observation
|
|
context propagates, and could include unrelated work.
|
|
- Add no relationship metadata. This would keep the API smaller, but observation
|
|
consumers would have to reconstruct continuation relationships themselves.
|
|
|
|
**Implementation planning:** place the minimal parent reference and preserve
|
|
ordinary per-request observation behavior for every continuation. Execution
|
|
groups address the separately demonstrated need for explicit accounting across
|
|
related calls; broader reporting remains outside this scope. The additional-turn
|
|
budget continues to bound continuation executions, not all provider calls
|
|
performed inside those executions or application callbacks.
|
|
|
|
### 6. Where Should Continuation Inspection Live?
|
|
|
|
**Status: accepted; exact method and type shape remain for implementation
|
|
planning.** Provide explicit access to the captured exchange's existing content
|
|
and minimal identity metadata, following the general pattern of
|
|
`PreparedExecution.Details()`.
|
|
|
|
**Accepted inspection scope:** expose the logical message history, including
|
|
the final assistant response, the exchange's originating run identity, and the
|
|
immediate parent relationship selected in Decision 5. Existing execution results
|
|
and preparation details continue supplying their ordinary information. Reuse
|
|
existing types where useful rather than creating a second comprehensive
|
|
execution report.
|
|
|
|
Details should be generically renderable through ordinary encoders so consumers
|
|
can display them without field-by-field adapters. Reading or modifying a
|
|
returned details value must not alter retained history or another branch's
|
|
execution. Access reads captured values without rerendering templates, reloading
|
|
inputs, or making provider calls. Captured message content remains separate
|
|
from ordinary metadata-only execution observations.
|
|
|
|
The exchange contains logical conversation history because continuation needs
|
|
it. This does not create a separate archive of execution results or expand the
|
|
coordinator's minimal termination result. `InspectPrompt`, `InspectProfile`, and
|
|
`InspectBackend` retain their configured-resource responsibilities.
|
|
|
|
**Representation alternatives for implementation planning:** a small details
|
|
accessor on a retained exchange is the preferred pattern. A public data value
|
|
may also satisfy the accepted behavior with fewer API elements, provided its
|
|
ownership rules prevent caller mutations from affecting retained branch state.
|
|
Choose the exact representation and serialization details during API planning.
|
|
|
|
**Alternative considered, not selected:** an engine-level conversation lookup
|
|
and inspection API. Such an API could serve a searchable conversation registry,
|
|
but this feature does not select that registry. Explicit access through an
|
|
already-held exchange satisfies the present need. Richer reporting or registry
|
|
capabilities require separate consumer justification.
|
|
|
|
## Separable Addition: Optional Execution Groups
|
|
|
|
**Status: accepted direction; the four decisions below remain open.** Audita's
|
|
follow-up request identifies a distinct need: a validator call can belong to the
|
|
same logical operation as a continuation without extending its conversation.
|
|
Conversation ancestry describes which exchange supplies history; accounting
|
|
membership describes which executions the application chooses to report together.
|
|
Neither relationship implies the other.
|
|
|
|
An application creates a group for a logical operation and explicitly associates
|
|
selected ordinary runs or coordinated continuations with it. Calls made by a
|
|
validator or other callback join only when the application chooses. Groups use
|
|
existing run identities and observation/usage semantics to summarize available
|
|
execution facts. They are usable without capture or a coordinator, and continuation
|
|
and coordination remain usable without them. Selecting a group does not change
|
|
execution scheduling, capacity, cancellation, retry, or validation behavior.
|
|
|
|
For example, an application may group a continuation and a separate validator
|
|
run while keeping their conversations independent. It may also include the
|
|
initial run shared by several branches. Replaying that run's captured history
|
|
does not enroll it or count its usage again; newly reported usage for sending
|
|
history in later executions still belongs to those later runs.
|
|
|
|
The group does not infer content meaning, automatically capture arbitrary nested
|
|
calls, schedule work, determine application outcomes, or introduce a reporting
|
|
framework. Applications retain domain identifiers, presentation, and any broader
|
|
report assembly. Group summaries are obtained separately from the coordinator's
|
|
minimal termination result. They preserve unavailable versus explicit zero usage
|
|
and do not add terminal totals to the attempts they already summarize.
|
|
|
|
### Group Decision 1: How Does An Application Select Membership?
|
|
|
|
**Recommendation:** first evaluate a group collector compatible with the existing
|
|
observer API. Attaching it explicitly to a request would enroll the observed run;
|
|
a coordinator could receive an explicit selection for its own continuation runs.
|
|
Callback calls would still require their own selection. This reuses the existing
|
|
source of execution facts and avoids a second accounting path.
|
|
|
|
The design must allow the application to keep its existing observer alongside
|
|
the group. Specify ordinary and prepared execution attachment and coordinator
|
|
propagation without inheriting membership into arbitrary nested calls. Preparation
|
|
alone is not an executed member; discarded handles and failed claims do not
|
|
currently emit execution observations.
|
|
|
|
**Alternative:** explicit group options on requests and the coordinator, backed
|
|
by ordinary observation internally. This makes membership clearer and separates
|
|
it from application event delivery, but adds public API plumbing. Choose after
|
|
sketching the smallest workable interfaces for ordinary, prepared, coordinated,
|
|
and callback-initiated calls. Do not select a context-based implicit scope.
|
|
|
|
### Group Decision 2: What Is The Group's Lifetime?
|
|
|
|
**Recommendation:** expose summary snapshots from a caller-held group, with no
|
|
mandatory close or wait operation. Applications already own the work and decide
|
|
when to read or discard the group. Support concurrent selected executions and
|
|
summary reads without imposing serial scheduling.
|
|
|
|
A snapshot must distinguish observed terminal runs from members whose terminal
|
|
facts are not yet available. A missing terminal event can reflect ongoing work
|
|
or interruption such as a panic; it is not evidence of zero usage or success.
|
|
Do not promise terminal observations beyond the existing observation contract.
|
|
|
|
**Alternative:** an explicit seal/close or wait lifecycle could provide a fixed
|
|
membership boundary or completion wait. It adds lifecycle rules and overlaps
|
|
with application orchestration; defer unless a consumer demonstrates that need.
|
|
|
|
### Group Decision 3: May Groups Overlap?
|
|
|
|
**Recommendation:** allow a run to belong to more than one explicitly selected
|
|
group, counting each run identity once within each group. This accommodates both
|
|
an operation summary and a broader application summary without requiring a
|
|
hierarchical group model. Overlapping group totals are not necessarily additive.
|
|
|
|
**Alternative:** require one group per run. This simplifies membership plumbing,
|
|
but restricts useful accounting views and makes applications rebuild the broader
|
|
aggregation themselves. There is no demonstrated need for that restriction.
|
|
Do not add group nesting, automatic parent membership, or group scheduling merely
|
|
to support overlap.
|
|
|
|
### Group Decision 4: Which Facts Does A Summary Expose And Aggregate?
|
|
|
|
**Recommendation:** start with member run identities, terminal-observation status,
|
|
and a small selection of existing execution counters and presence-aware usage
|
|
totals. Select the exact fields during API design; avoid a comprehensive new
|
|
report schema or new outcome hierarchy. Summaries describe execution facts, not
|
|
application acceptance or monetary cost. Summed run durations, if included,
|
|
must not be presented as elapsed group time when runs overlap.
|
|
|
|
Prefer aggregating one terminal event per run. Terminal observations already
|
|
summarize retries, structural repairs, reported usage, and failure-path facts,
|
|
including executions that return no ordinary result. This is simpler than
|
|
reconstructing those totals from attempt events. Preserve each usage field's
|
|
known/completeness distinctions and expose missing terminal coverage separately,
|
|
so a total over finished members cannot imply complete accounting for all members.
|
|
|
|
**Alternative:** accumulate attempt events for live partial totals, reconciling
|
|
terminal events without counting the same work twice. This exposes more facts
|
|
before completion or after a missing terminal event, but needs additional
|
|
reconciliation rules. Prefer terminal-based totals unless live attempt accounting
|
|
has demonstrated value. Whichever representation is selected, absent evidence
|
|
must not become a known zero, and terminal summaries must never be added to their
|
|
already-counted attempts.
|
|
|
|
## Proposed Delivery And Validation
|
|
|
|
The six continuation/coordinator behavioral decisions are accepted. Their
|
|
remaining work is API and staged implementation planning: exact types and
|
|
signatures, capture opt-in syntax, message assembly details, placement of
|
|
relationship metadata, and the concrete inspection representation. Execution
|
|
groups have four open decisions above and a separate delivery track; resolving
|
|
or implementing them is not a prerequisite for continuation. The recommended
|
|
continuation delivery sequence is:
|
|
|
|
1. Deliver capture and independent continuation through the public facade,
|
|
allowing Audita to remove its reconstruction and prefix-verification work.
|
|
2. Build the optional sequential coordinator on that capability, reusing
|
|
ordinary execution, validation, capacity, and observation machinery. Refine
|
|
both consumer sketches against the selected public contracts to verify that
|
|
the same mechanism removes useful repeated work in each application.
|
|
|
|
Plan execution groups independently after resolving their open decisions,
|
|
reusing ordinary observations and preserving existing application observers.
|
|
They can ship separately without changing the coordinator termination contract.
|
|
|
|
Use [Audita's acceptance scenarios](audita-request.md#suggested-delivery-and-acceptance-scenarios)
|
|
as consumer checks, alongside analysis and branching-conversation examples that
|
|
exercise the same APIs without repair-specific concepts. Additional proposed
|
|
coverage should establish:
|
|
|
|
- The selected conversation history when the initial run needed structural
|
|
repair, including use of its full final assistant content.
|
|
- Capture of the final returned response after structural validation fails,
|
|
preserving its ordinary validation status through the coordinator callback.
|
|
Operational failures with no completed result create no new exchange.
|
|
- Capture opt-in through ordinary and prepared execution, using the executed
|
|
snapshot and producing no completed exchange during preparation alone.
|
|
- Continuation through another engine instance in the same process, retaining
|
|
prior content while applying the receiving engine's follow-up configuration.
|
|
- Continuation after the original input files change or disappear, without
|
|
altering the captured history.
|
|
- Independent branches retaining their own history and outcomes.
|
|
- Assessment calls against the same backend at capacity one, without capacity
|
|
retained by the coordinator blocking them.
|
|
- Per-run usage and failure diagnostics available through ordinary observations,
|
|
including incomplete usage. Consumer examples explicitly select accounting
|
|
membership and count shared initial work once, without coordinator-provided
|
|
totals; optional groups may supply those summaries when available.
|
|
- Immediate parent relationships distinguish successive turns and independent
|
|
branches using existing run identities.
|
|
- Explicit exchange details expose captured logical history and minimal identity
|
|
metadata through ordinary encoding. Mutating returned details does not affect
|
|
retained branch state, and inspection requires no rendering or external reads.
|
|
|
|
Execution-group validation should separately cover explicit ordinary, prepared,
|
|
coordinator, and callback membership; unrelated nested calls remaining outside
|
|
the group; independent ancestry and membership; concurrent collection alongside
|
|
an existing application observer; failure-path terminal facts; unavailable and
|
|
explicit-zero usage; missing terminal observations; and no attempt/terminal
|
|
double counting. If overlapping groups are selected, verify once-per-run counting
|
|
within each group without implying that overlapping totals can be added.
|
|
|
|
These are planning targets, not new implemented guarantees. Keep additional
|
|
mechanisms and constraints tied to the selected decisions and demonstrated
|
|
consumer needs.
|