Prepare v0.22.0 release notes and retire completed roadmaps
This commit is contained in:
@@ -1,169 +0,0 @@
|
||||
# Audita request: conversation continuation and application-directed repair
|
||||
|
||||
Status: proposed upstream features, requested by Audita. Public API design and
|
||||
implementation remain for PromptKit to determine.
|
||||
|
||||
This request is based on Audita's use of PromptKit v0.21.0. It proposes two
|
||||
related capabilities: captured conversation continuations, followed by bounded
|
||||
repair driven by application-supplied validation feedback. The goal is to move
|
||||
reusable execution machinery into PromptKit and remove its local equivalents
|
||||
from Audita.
|
||||
|
||||
## Audita's workflow
|
||||
|
||||
Audita polishes transcripts using module-specific proposal, validation, and
|
||||
repair prompts:
|
||||
|
||||
1. A proposal request supplies transcript segments and receives a structured
|
||||
collection of proposed corrections.
|
||||
2. Audita evaluates each correction using deterministic and LLM-backed
|
||||
validators. Accepted corrections are retained for later application.
|
||||
3. Each actionable rejection starts an independent repair branch containing the
|
||||
original rendered request, the full final assistant response, and a follow-up
|
||||
instruction rendered from that module's repair prompt definition.
|
||||
4. The follow-up identifies one correction and supplies its current content,
|
||||
available before-and-after transcript context, and validation findings. The
|
||||
model can revise or withdraw that correction. This response uses a different
|
||||
schema from the original collection of proposals.
|
||||
5. A revision passes through the validators again. Further attempts retain that
|
||||
branch's history and receive updated feedback. Acceptance, withdrawal,
|
||||
exhaustion, or failure ends the branch; independent approvals are preserved.
|
||||
|
||||
Audita owns the meaning of a correction, the validation rules, repair
|
||||
eligibility, and eventual application to the transcript. Conversation handling
|
||||
and the bounded revision lifecycle are useful beyond transcript processing.
|
||||
|
||||
## Existing support and the remaining gap
|
||||
|
||||
PromptKit already supports [structural output repair](../consumers/pkg-promptkit.md#repair-a-structured-result),
|
||||
[appended messages](../consumers/pkg-promptkit.md#append-already-rendered-messages),
|
||||
prepared execution, profiles, execution observations, and backend capacity
|
||||
management. These allow Audita to implement the workflow today.
|
||||
|
||||
Audita still captures exchanges, reconstructs the original prompt, verifies its
|
||||
rendered prefix, assembles branch histories, renders follow-up instructions,
|
||||
and coordinates repeated application validation and revision. The requested
|
||||
APIs would make those responsibilities supported PromptKit capabilities.
|
||||
|
||||
## 1. Captured conversations and independent continuations
|
||||
|
||||
Provide a public way to capture a rendered request and its full final assistant
|
||||
response, then continue that exchange without making the consumer reconstruct
|
||||
the original prompt and check that its messages are unchanged.
|
||||
|
||||
Audita needs to:
|
||||
|
||||
- Preserve the original rendered message content, roles, and cache metadata
|
||||
when continuing an exchange.
|
||||
- Start multiple independent branches from the same exchange. A branch's later
|
||||
instructions and responses must not alter the original exchange or another
|
||||
branch's history.
|
||||
- Render follow-up instructions using ordinary PromptKit prompt definitions and
|
||||
variables, and append them to the selected branch. Audita's repair prompts
|
||||
remain operator-replaceable through its existing PromptKit configuration.
|
||||
- Select a follow-up profile and output contract independently of the initial
|
||||
request. The first response may contain many items while a continuation
|
||||
returns one revised item or a withdrawal.
|
||||
- Retain the relationship between the original exchange, the follow-up prompt
|
||||
definition, and each execution in inspection and observation data.
|
||||
|
||||
An in-process capability is sufficient for Audita. Persistence, restartable
|
||||
sessions, and a conversation database are not required by this request. The
|
||||
existing [prepared execution handle](../../prepared_execution.go) is single-use;
|
||||
the request does not assume that copying or reusing it creates a continuation.
|
||||
|
||||
## 2. Bounded application-directed repair
|
||||
|
||||
Build a reusable revision coordinator on the continuation capability. Audita
|
||||
should supply its assessment and request-building behavior while PromptKit
|
||||
handles the repeated execution lifecycle.
|
||||
|
||||
The coordinator needs to support:
|
||||
|
||||
- Starting with an already-assessed candidate and its rejection findings, so
|
||||
Audita need not repeat the initial batch validation simply to enter repair.
|
||||
- Application-supplied assessment of each revision, including deterministic
|
||||
checks and checks that make further PromptKit calls. Findings may come from
|
||||
multiple validators; their meaning and explanatory content remain with the
|
||||
application.
|
||||
- Application-selected follow-up prompts, variables, profiles, and response
|
||||
contracts. Audita supplies the candidate and context, and interprets the
|
||||
response as a revision or withdrawal. PromptKit need not prescribe an Audita
|
||||
correction schema or a universal feedback format.
|
||||
- Reassessment before a revised candidate can be accepted, with fresh feedback
|
||||
for the next attempt and accumulated history confined to that branch.
|
||||
- A caller-selected additional-revision budget and caller context, with clear
|
||||
terminal outcomes covering acceptance, withdrawal, exhaustion, unavailable
|
||||
results, and interruption. Callers retain decisions about repair eligibility
|
||||
and whether an operational failure should terminate other work.
|
||||
- Attempt observations, execution relationships, and available usage accounting
|
||||
sufficient to report a repair sequence without recreating its lifecycle
|
||||
tracking in Audita.
|
||||
|
||||
The revision budget must remain distinct from transport retry and structural
|
||||
repair budgets. A structurally valid response can still fail Audita's checks;
|
||||
each revision generation can itself need structural repair. Accounting should
|
||||
distinguish those layers and avoid double-counting shared initial work.
|
||||
|
||||
Application validators must be able to call the same engine and backend without
|
||||
being blocked by capacity retained by their parent operation. Coordinating
|
||||
ordinary completed runs is one possible approach. PromptKit should determine
|
||||
the implementation in light of its [execution and admission lifecycle](../internal/runner.md#run-flow).
|
||||
|
||||
This is broader than exposing the current structural-repair hook: the response
|
||||
schema can change between proposal and repair, validation may perform model
|
||||
calls, and each rejected item has its own history and disposition.
|
||||
|
||||
## Ownership boundary
|
||||
|
||||
PromptKit would own captured exchanges, branch continuation, the bounded repair
|
||||
lifecycle, and associated execution observations. Audita would retain:
|
||||
|
||||
- Transcript segments, proposal identities, glossary and protected-term rules,
|
||||
editorial judgments, and meaning checks.
|
||||
- Validator selection and ordering, actionable-rejection policy, feedback
|
||||
content, and module-specific proposal, validation, and repair prompts.
|
||||
- Before-and-after transcript previews, preservation of accepted sibling
|
||||
proposals, application ordering, and final transcript mutation.
|
||||
- Application reports, debug artifact destinations, and CLI behavior.
|
||||
|
||||
The separate idea of reconciling missing or duplicate items in batch responses
|
||||
is outside this request. The two requested capabilities should be usable
|
||||
through the supported public facade without an Audita dependency or a general
|
||||
workflow framework. Exact types, callbacks, and configuration syntax are left
|
||||
to PromptKit's API design.
|
||||
|
||||
## Suggested delivery and acceptance scenarios
|
||||
|
||||
Deliver conversation continuation first, allowing Audita to replace its capture
|
||||
and replay machinery. Then add application-directed repair and replace the
|
||||
corresponding local coordination. Existing prompt definitions and domain
|
||||
validators should remain usable through the new APIs.
|
||||
|
||||
The following scenarios would demonstrate that the capabilities meet Audita's
|
||||
needs:
|
||||
|
||||
1. An initial response contains three proposals. One is accepted; the other two
|
||||
receive independent repair conversations containing the same original
|
||||
exchange. Revising either branch leaves the accepted proposal and the other
|
||||
branch unchanged.
|
||||
2. A follow-up uses a separate prompt definition, profile, and single-item
|
||||
response schema. Inspection identifies the follow-up and its originating
|
||||
exchange.
|
||||
3. A revision fails multiple application checks, receives their updated
|
||||
feedback, and passes on a later attempt. An LLM-backed check can use the same
|
||||
engine and backend even when backend capacity is limited to one.
|
||||
4. Withdrawal, exhausted revision budget, unavailable validation, and
|
||||
cancellation produce distinguishable outcomes. An unaccepted revision is
|
||||
never returned as an accepted candidate, and a recoverable branch failure
|
||||
leaves independent approvals available to the application.
|
||||
5. A revision generation needs both a transport retry and structural repair.
|
||||
Observations distinguish these from application-directed revision attempts
|
||||
and expose the available usage without counting the original exchange once
|
||||
per branch.
|
||||
|
||||
For implementation reference, Audita currently owns these mechanics in
|
||||
`internal/framework/llm/adapter.go`,
|
||||
`internal/framework/structuredoutput/call.go`, and
|
||||
`internal/framework/runner/repair.go`. These are examples of the consumer need,
|
||||
not proposed PromptKit package boundaries or public API shapes.
|
||||
Reference in New Issue
Block a user