promptkit/docs/roadmap/audita-request.md

Audita request: conversation continuation and application-directed repair

Status: proposed upstream features, requested by Audita. Public API design and implementation remain for PromptKit to determine.

This request is based on Audita's use of PromptKit v0.21.0. It proposes two related capabilities: captured conversation continuations, followed by bounded repair driven by application-supplied validation feedback. The goal is to move reusable execution machinery into PromptKit and remove its local equivalents from Audita.

Audita's workflow

Audita polishes transcripts using module-specific proposal, validation, and repair prompts:

1. A proposal request supplies transcript segments and receives a structured collection of proposed corrections.
2. Audita evaluates each correction using deterministic and LLM-backed validators. Accepted corrections are retained for later application.
3. Each actionable rejection starts an independent repair branch containing the original rendered request, the full final assistant response, and a follow-up instruction rendered from that module's repair prompt definition.
4. The follow-up identifies one correction and supplies its current content, available before-and-after transcript context, and validation findings. The model can revise or withdraw that correction. This response uses a different schema from the original collection of proposals.
5. A revision passes through the validators again. Further attempts retain that branch's history and receive updated feedback. Acceptance, withdrawal, exhaustion, or failure ends the branch; independent approvals are preserved.

Audita owns the meaning of a correction, the validation rules, repair eligibility, and eventual application to the transcript. Conversation handling and the bounded revision lifecycle are useful beyond transcript processing.

Existing support and the remaining gap

PromptKit already supports structural output repair, appended messages, prepared execution, profiles, execution observations, and backend capacity management. These allow Audita to implement the workflow today.

Audita still captures exchanges, reconstructs the original prompt, verifies its rendered prefix, assembles branch histories, renders follow-up instructions, and coordinates repeated application validation and revision. The requested APIs would make those responsibilities supported PromptKit capabilities.

1. Captured conversations and independent continuations

Provide a public way to capture a rendered request and its full final assistant response, then continue that exchange without making the consumer reconstruct the original prompt and check that its messages are unchanged.

Audita needs to:

• Preserve the original rendered message content, roles, and cache metadata when continuing an exchange.
• Start multiple independent branches from the same exchange. A branch's later instructions and responses must not alter the original exchange or another branch's history.
• Render follow-up instructions using ordinary PromptKit prompt definitions and variables, and append them to the selected branch. Audita's repair prompts remain operator-replaceable through its existing PromptKit configuration.
• Select a follow-up profile and output contract independently of the initial request. The first response may contain many items while a continuation returns one revised item or a withdrawal.
• Retain the relationship between the original exchange, the follow-up prompt definition, and each execution in inspection and observation data.

An in-process capability is sufficient for Audita. Persistence, restartable sessions, and a conversation database are not required by this request. The existing prepared execution handle is single-use; the request does not assume that copying or reusing it creates a continuation.

2. Bounded application-directed repair

Build a reusable revision coordinator on the continuation capability. Audita should supply its assessment and request-building behavior while PromptKit handles the repeated execution lifecycle.

The coordinator needs to support:

• Starting with an already-assessed candidate and its rejection findings, so Audita need not repeat the initial batch validation simply to enter repair.
• Application-supplied assessment of each revision, including deterministic checks and checks that make further PromptKit calls. Findings may come from multiple validators; their meaning and explanatory content remain with the application.
• Application-selected follow-up prompts, variables, profiles, and response contracts. Audita supplies the candidate and context, and interprets the response as a revision or withdrawal. PromptKit need not prescribe an Audita correction schema or a universal feedback format.
• Reassessment before a revised candidate can be accepted, with fresh feedback for the next attempt and accumulated history confined to that branch.
• A caller-selected additional-revision budget and caller context, with clear terminal outcomes covering acceptance, withdrawal, exhaustion, unavailable results, and interruption. Callers retain decisions about repair eligibility and whether an operational failure should terminate other work.
• Attempt observations, execution relationships, and available usage accounting sufficient to report a repair sequence without recreating its lifecycle tracking in Audita.

The revision budget must remain distinct from transport retry and structural repair budgets. A structurally valid response can still fail Audita's checks; each revision generation can itself need structural repair. Accounting should distinguish those layers and avoid double-counting shared initial work.

Application validators must be able to call the same engine and backend without being blocked by capacity retained by their parent operation. Coordinating ordinary completed runs is one possible approach. PromptKit should determine the implementation in light of its execution and admission lifecycle.

This is broader than exposing the current structural-repair hook: the response schema can change between proposal and repair, validation may perform model calls, and each rejected item has its own history and disposition.

Ownership boundary

PromptKit would own captured exchanges, branch continuation, the bounded repair lifecycle, and associated execution observations. Audita would retain:

• Transcript segments, proposal identities, glossary and protected-term rules, editorial judgments, and meaning checks.
• Validator selection and ordering, actionable-rejection policy, feedback content, and module-specific proposal, validation, and repair prompts.
• Before-and-after transcript previews, preservation of accepted sibling proposals, application ordering, and final transcript mutation.
• Application reports, debug artifact destinations, and CLI behavior.

The separate idea of reconciling missing or duplicate items in batch responses is outside this request. The two requested capabilities should be usable through the supported public facade without an Audita dependency or a general workflow framework. Exact types, callbacks, and configuration syntax are left to PromptKit's API design.

Suggested delivery and acceptance scenarios

Deliver conversation continuation first, allowing Audita to replace its capture and replay machinery. Then add application-directed repair and replace the corresponding local coordination. Existing prompt definitions and domain validators should remain usable through the new APIs.

The following scenarios would demonstrate that the capabilities meet Audita's needs:

1. An initial response contains three proposals. One is accepted; the other two receive independent repair conversations containing the same original exchange. Revising either branch leaves the accepted proposal and the other branch unchanged.
2. A follow-up uses a separate prompt definition, profile, and single-item response schema. Inspection identifies the follow-up and its originating exchange.
3. A revision fails multiple application checks, receives their updated feedback, and passes on a later attempt. An LLM-backed check can use the same engine and backend even when backend capacity is limited to one.
4. Withdrawal, exhausted revision budget, unavailable validation, and cancellation produce distinguishable outcomes. An unaccepted revision is never returned as an accepted candidate, and a recoverable branch failure leaves independent approvals available to the application.
5. A revision generation needs both a transport retry and structural repair. Observations distinguish these from application-directed revision attempts and expose the available usage without counting the original exchange once per branch.

For implementation reference, Audita currently owns these mechanics in internal/framework/llm/adapter.go, internal/framework/structuredoutput/call.go, and internal/framework/runner/repair.go. These are examples of the consumer need, not proposed PromptKit package boundaries or public API shapes.