Define the execution controls implementation plan
This commit is contained in:
139
docs/roadmap/audita-request.md
Normal file
139
docs/roadmap/audita-request.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Audita request: coordinator execution controls
|
||||
|
||||
Status: proposed follow-up to PromptKit v0.22.0. This document describes
|
||||
consumer requirements; public API design and implementation remain upstream
|
||||
decisions.
|
||||
|
||||
## Purpose
|
||||
|
||||
PromptKit v0.22.0 supplies the conversation capture, continuation, and execution
|
||||
groups Audita requested. Audita can adopt those capabilities through its existing
|
||||
execution adapter. To also adopt `Engine.Coordinate` and remove its local turn
|
||||
loop, Audita needs two capabilities during coordinator-owned execution:
|
||||
|
||||
1. A fresh, application-selected timeout for each continuation operation.
|
||||
2. Access to the exact prepared request before generation, including for turns
|
||||
that subsequently fail without returning a result.
|
||||
|
||||
Both capabilities apply to applications that run bounded model operations and
|
||||
retain optional diagnostic evidence. Neither requires PromptKit to understand
|
||||
transcripts, correction proposals, or domain validation.
|
||||
|
||||
## Current workflow and integration gap
|
||||
|
||||
Audita's role-bound adapter currently derives a timeout context from its caller,
|
||||
calls `PrepareExecution`, retains `PreparedExecution.Details()`, and executes
|
||||
that same handle with `RunPrepared`. The operation deadline covers preparation,
|
||||
capacity waiting, transient retries, generation, and structural repair. Each
|
||||
model operation receives its own budget; application validation may make
|
||||
additional, separately bounded model calls.
|
||||
|
||||
When debug evidence is enabled, Audita retains the prepared messages and
|
||||
effective settings alongside the available response or failure information.
|
||||
It owns artifact destinations, application-specific redaction, and handling of
|
||||
optional diagnostic write failures. The same adapter supplies ordinary execution
|
||||
observers and classifies operation errors for the application.
|
||||
|
||||
In v0.22.0, the [coordinator](../../coordinate.go) accepts a callback returning a
|
||||
`RunRequest`, then directly calls `Engine.Run` with the coordinator's context.
|
||||
The callback cannot supply a fresh execution context or a prepared handle.
|
||||
There is no callback exposing the actual prepared request before that run.
|
||||
|
||||
A timeout around the entire coordinator bounds the conversation and application
|
||||
callbacks together; it does not preserve a separate timeout for each operation.
|
||||
Provider generation timeouts do not cover the whole operation either.
|
||||
Separately calling `Prepare` before returning a request duplicates preparation
|
||||
and does not bind the inspected snapshot to the execution. Successful exchange
|
||||
inspection cannot supply a failed turn's prepared request when no new exchange
|
||||
is returned. Existing metadata-only observations do not contain that content.
|
||||
|
||||
## Requested behavior
|
||||
|
||||
### Per-operation deadlines for coordinated turns
|
||||
|
||||
Allow the application to select a timeout for each requested continuation, with
|
||||
these semantics:
|
||||
|
||||
- Establish a fresh deadline when that continuation starts, before its
|
||||
preparation. Cover preparation, admission waiting, retries, generation, and
|
||||
structural repair under the same operation budget.
|
||||
- Respect caller cancellation and any earlier parent deadline. Expiry of a
|
||||
local operation budget must not cancel the parent context or independent
|
||||
branches.
|
||||
- End the operation's timeout scope before invoking the application's next-turn
|
||||
callback. Assessment calls made there can retain their own operation budgets.
|
||||
- Preserve useful error identity so Audita can distinguish a local deadline
|
||||
from cancellation of the caller. An execution timeout can end coordination
|
||||
under the existing operational-error rule; no automatic turn replay or new
|
||||
recovery policy is requested.
|
||||
|
||||
The existing additional-turn budget, transport retry budget, structural-repair
|
||||
budget, and provider generation timeout retain their separate meanings.
|
||||
|
||||
### Exact prepared-request access
|
||||
|
||||
Provide an opt-in way to obtain caller-owned details of the actual prepared
|
||||
continuation before its first generation attempt:
|
||||
|
||||
- Include the assembled captured history, follow-up definition messages,
|
||||
appended messages, and ordinary resolved execution metadata, using the
|
||||
existing [prepared-details contract](../../prepared_execution.go) where useful.
|
||||
- Expose the snapshot that execution will use, without a second rendering or
|
||||
input-reading pass. Access to details must not mutate that execution.
|
||||
- Let the application retain those details even if generation subsequently
|
||||
fails or is canceled and returns no ordinary result or captured exchange.
|
||||
If preparation itself fails, a complete prepared snapshot is not required.
|
||||
- Preserve existing credential-redaction semantics. Keep content-bearing access
|
||||
explicitly selected and separate from ordinary metadata-only observations.
|
||||
|
||||
Audita needs the logical prepared request, not raw HTTP envelopes or an archive
|
||||
of intermediate structural-repair requests. It can handle optional debug writes
|
||||
itself without requiring PromptKit to own artifact storage or reporting policy.
|
||||
|
||||
## API direction and ownership
|
||||
|
||||
Prefer small execution-level controls that the coordinator honors through its
|
||||
ordinary execution path. An optional operation timeout and a prepared-details
|
||||
callback are one possible shape, not proposed exported names or signatures.
|
||||
Their applicability to ordinary and prepared execution should follow a coherent
|
||||
lifecycle rather than introducing a second coordinator-specific executor.
|
||||
|
||||
An alternative is allowing a coordinator turn to delegate execution through a
|
||||
caller-controlled prepared-execution path. That could reuse Audita's adapter,
|
||||
but exposes more execution responsibility and requires clear ownership of
|
||||
capture, observers, admission release, and prepared-handle cleanup. Evaluate
|
||||
that option only if it is simpler than the narrow controls above.
|
||||
|
||||
PromptKit should continue owning history assembly, turn progression, execution,
|
||||
and observations. Audita retains response interpretation, validator calls,
|
||||
branch policy, diagnostic storage, and application error classification. Existing
|
||||
request observers, coordinator observers, and execution groups must remain
|
||||
usable together. This request does not reopen the accepted conversation history,
|
||||
final-turn delivery, or grouping contracts.
|
||||
|
||||
## Acceptance scenarios
|
||||
|
||||
1. Two successful continuation operations each finish within their own timeout,
|
||||
while their combined duration exceeds one operation's budget. Both succeed
|
||||
under an active parent context.
|
||||
2. Preparation or admission waiting consumes the same budget as generation and
|
||||
structural repair. Local expiry returns a recognizable deadline error and
|
||||
stops that coordinator without canceling its parent or a sibling branch.
|
||||
3. An earlier parent deadline remains effective. A validator called from the
|
||||
next-turn callback runs after admission is released, including with backend
|
||||
capacity one, under its own application-selected deadline.
|
||||
4. The application receives the prepared continuation details before generation.
|
||||
A later provider failure returns no result, but the retained details still
|
||||
describe that failed operation. Mutating a details copy does not change the
|
||||
messages executed.
|
||||
5. Diagnostic access uses the same prepared snapshot that executes, including
|
||||
when source inputs change afterward. It does not trigger another preparation
|
||||
or provider call.
|
||||
6. Existing observers and selected execution groups continue receiving ordinary
|
||||
execution facts once per their existing contracts, including failure paths.
|
||||
Applications that select neither new capability retain existing behavior.
|
||||
|
||||
The consumer implementation motivating this request is Audita's
|
||||
`internal/framework/llm/adapter.go`, especially `RoleClient.CompleteStructured`.
|
||||
This gap affects full coordinator adoption; it does not block adopting v0.22.0
|
||||
capture, continuation, or execution groups independently.
|
||||
Reference in New Issue
Block a user