170 lines
9.2 KiB
Markdown
170 lines
9.2 KiB
Markdown
# Audita request: conversation continuation and application-directed repair
|
|
|
|
Status: proposed upstream features, requested by Audita. Public API design and
|
|
implementation remain for PromptKit to determine.
|
|
|
|
This request is based on Audita's use of PromptKit v0.21.0. It proposes two
|
|
related capabilities: captured conversation continuations, followed by bounded
|
|
repair driven by application-supplied validation feedback. The goal is to move
|
|
reusable execution machinery into PromptKit and remove its local equivalents
|
|
from Audita.
|
|
|
|
## Audita's workflow
|
|
|
|
Audita polishes transcripts using module-specific proposal, validation, and
|
|
repair prompts:
|
|
|
|
1. A proposal request supplies transcript segments and receives a structured
|
|
collection of proposed corrections.
|
|
2. Audita evaluates each correction using deterministic and LLM-backed
|
|
validators. Accepted corrections are retained for later application.
|
|
3. Each actionable rejection starts an independent repair branch containing the
|
|
original rendered request, the full final assistant response, and a follow-up
|
|
instruction rendered from that module's repair prompt definition.
|
|
4. The follow-up identifies one correction and supplies its current content,
|
|
available before-and-after transcript context, and validation findings. The
|
|
model can revise or withdraw that correction. This response uses a different
|
|
schema from the original collection of proposals.
|
|
5. A revision passes through the validators again. Further attempts retain that
|
|
branch's history and receive updated feedback. Acceptance, withdrawal,
|
|
exhaustion, or failure ends the branch; independent approvals are preserved.
|
|
|
|
Audita owns the meaning of a correction, the validation rules, repair
|
|
eligibility, and eventual application to the transcript. Conversation handling
|
|
and the bounded revision lifecycle are useful beyond transcript processing.
|
|
|
|
## Existing support and the remaining gap
|
|
|
|
PromptKit already supports [structural output repair](../consumers/pkg-promptkit.md#repair-a-structured-result),
|
|
[appended messages](../consumers/pkg-promptkit.md#append-already-rendered-messages),
|
|
prepared execution, profiles, execution observations, and backend capacity
|
|
management. These allow Audita to implement the workflow today.
|
|
|
|
Audita still captures exchanges, reconstructs the original prompt, verifies its
|
|
rendered prefix, assembles branch histories, renders follow-up instructions,
|
|
and coordinates repeated application validation and revision. The requested
|
|
APIs would make those responsibilities supported PromptKit capabilities.
|
|
|
|
## 1. Captured conversations and independent continuations
|
|
|
|
Provide a public way to capture a rendered request and its full final assistant
|
|
response, then continue that exchange without making the consumer reconstruct
|
|
the original prompt and check that its messages are unchanged.
|
|
|
|
Audita needs to:
|
|
|
|
- Preserve the original rendered message content, roles, and cache metadata
|
|
when continuing an exchange.
|
|
- Start multiple independent branches from the same exchange. A branch's later
|
|
instructions and responses must not alter the original exchange or another
|
|
branch's history.
|
|
- Render follow-up instructions using ordinary PromptKit prompt definitions and
|
|
variables, and append them to the selected branch. Audita's repair prompts
|
|
remain operator-replaceable through its existing PromptKit configuration.
|
|
- Select a follow-up profile and output contract independently of the initial
|
|
request. The first response may contain many items while a continuation
|
|
returns one revised item or a withdrawal.
|
|
- Retain the relationship between the original exchange, the follow-up prompt
|
|
definition, and each execution in inspection and observation data.
|
|
|
|
An in-process capability is sufficient for Audita. Persistence, restartable
|
|
sessions, and a conversation database are not required by this request. The
|
|
existing [prepared execution handle](../../prepared_execution.go) is single-use;
|
|
the request does not assume that copying or reusing it creates a continuation.
|
|
|
|
## 2. Bounded application-directed repair
|
|
|
|
Build a reusable revision coordinator on the continuation capability. Audita
|
|
should supply its assessment and request-building behavior while PromptKit
|
|
handles the repeated execution lifecycle.
|
|
|
|
The coordinator needs to support:
|
|
|
|
- Starting with an already-assessed candidate and its rejection findings, so
|
|
Audita need not repeat the initial batch validation simply to enter repair.
|
|
- Application-supplied assessment of each revision, including deterministic
|
|
checks and checks that make further PromptKit calls. Findings may come from
|
|
multiple validators; their meaning and explanatory content remain with the
|
|
application.
|
|
- Application-selected follow-up prompts, variables, profiles, and response
|
|
contracts. Audita supplies the candidate and context, and interprets the
|
|
response as a revision or withdrawal. PromptKit need not prescribe an Audita
|
|
correction schema or a universal feedback format.
|
|
- Reassessment before a revised candidate can be accepted, with fresh feedback
|
|
for the next attempt and accumulated history confined to that branch.
|
|
- A caller-selected additional-revision budget and caller context, with clear
|
|
terminal outcomes covering acceptance, withdrawal, exhaustion, unavailable
|
|
results, and interruption. Callers retain decisions about repair eligibility
|
|
and whether an operational failure should terminate other work.
|
|
- Attempt observations, execution relationships, and available usage accounting
|
|
sufficient to report a repair sequence without recreating its lifecycle
|
|
tracking in Audita.
|
|
|
|
The revision budget must remain distinct from transport retry and structural
|
|
repair budgets. A structurally valid response can still fail Audita's checks;
|
|
each revision generation can itself need structural repair. Accounting should
|
|
distinguish those layers and avoid double-counting shared initial work.
|
|
|
|
Application validators must be able to call the same engine and backend without
|
|
being blocked by capacity retained by their parent operation. Coordinating
|
|
ordinary completed runs is one possible approach. PromptKit should determine
|
|
the implementation in light of its [execution and admission lifecycle](../internal/runner.md#run-flow).
|
|
|
|
This is broader than exposing the current structural-repair hook: the response
|
|
schema can change between proposal and repair, validation may perform model
|
|
calls, and each rejected item has its own history and disposition.
|
|
|
|
## Ownership boundary
|
|
|
|
PromptKit would own captured exchanges, branch continuation, the bounded repair
|
|
lifecycle, and associated execution observations. Audita would retain:
|
|
|
|
- Transcript segments, proposal identities, glossary and protected-term rules,
|
|
editorial judgments, and meaning checks.
|
|
- Validator selection and ordering, actionable-rejection policy, feedback
|
|
content, and module-specific proposal, validation, and repair prompts.
|
|
- Before-and-after transcript previews, preservation of accepted sibling
|
|
proposals, application ordering, and final transcript mutation.
|
|
- Application reports, debug artifact destinations, and CLI behavior.
|
|
|
|
The separate idea of reconciling missing or duplicate items in batch responses
|
|
is outside this request. The two requested capabilities should be usable
|
|
through the supported public facade without an Audita dependency or a general
|
|
workflow framework. Exact types, callbacks, and configuration syntax are left
|
|
to PromptKit's API design.
|
|
|
|
## Suggested delivery and acceptance scenarios
|
|
|
|
Deliver conversation continuation first, allowing Audita to replace its capture
|
|
and replay machinery. Then add application-directed repair and replace the
|
|
corresponding local coordination. Existing prompt definitions and domain
|
|
validators should remain usable through the new APIs.
|
|
|
|
The following scenarios would demonstrate that the capabilities meet Audita's
|
|
needs:
|
|
|
|
1. An initial response contains three proposals. One is accepted; the other two
|
|
receive independent repair conversations containing the same original
|
|
exchange. Revising either branch leaves the accepted proposal and the other
|
|
branch unchanged.
|
|
2. A follow-up uses a separate prompt definition, profile, and single-item
|
|
response schema. Inspection identifies the follow-up and its originating
|
|
exchange.
|
|
3. A revision fails multiple application checks, receives their updated
|
|
feedback, and passes on a later attempt. An LLM-backed check can use the same
|
|
engine and backend even when backend capacity is limited to one.
|
|
4. Withdrawal, exhausted revision budget, unavailable validation, and
|
|
cancellation produce distinguishable outcomes. An unaccepted revision is
|
|
never returned as an accepted candidate, and a recoverable branch failure
|
|
leaves independent approvals available to the application.
|
|
5. A revision generation needs both a transport retry and structural repair.
|
|
Observations distinguish these from application-directed revision attempts
|
|
and expose the available usage without counting the original exchange once
|
|
per branch.
|
|
|
|
For implementation reference, Audita currently owns these mechanics in
|
|
`internal/framework/llm/adapter.go`,
|
|
`internal/framework/structuredoutput/call.go`, and
|
|
`internal/framework/runner/repair.go`. These are examples of the consumer need,
|
|
not proposed PromptKit package boundaries or public API shapes.
|