From d28b56026570fac6e18e91060b66032bba0a355e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 15 Nov 2026 01:11:58 +0000 Subject: [PATCH] Plan full PromptKit v0.23.0 adoption --- docs/roadmap/promptkit-implementation.md | 168 +++++++++++++++++++++++ docs/roadmap/promptkit.md | 136 ++++++++++++++++++ 2 files changed, 304 insertions(+) create mode 100644 docs/roadmap/promptkit-implementation.md create mode 100644 docs/roadmap/promptkit.md diff --git a/docs/roadmap/promptkit-implementation.md b/docs/roadmap/promptkit-implementation.md new file mode 100644 index 0000000..0f6bae5 --- /dev/null +++ b/docs/roadmap/promptkit-implementation.md @@ -0,0 +1,168 @@ +# PromptKit v0.23.0 Implementation Plan + +**Status:** Ready for implementation; all tasks below are pending. + +The [adoption roadmap](promptkit.md) owns the target behavior and boundaries. +This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed +to establish full adoption. It does not authorize application code changes during +the current documentation pass. + +## 1. Upgrade and unify execution + +- [ ] Update `go.mod` and `go.sum` directly to + `go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove + temporary replacements/workspace files. Check the selected module graph and + compatibility with the existing CommandKit/ckprompt integration. +- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around + shared request construction, response interpretation, and error mapping that + can serve both ordinary calls and coordinator-owned turns. Keep those helpers + narrow; do not wrap `Coordinate` in another turn loop. +- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary + `Run` for calls whose prepare/run split exists only to obtain diagnostics. + Avoid retaining a redundant child timer around the same ordinary operation. +- [ ] Supply `OnPrepared` to collect effective metadata and selected credential + source names, and retain prepared content for debug when enabled. Use the + delivered snapshot without a separate `Prepare` call. Clear per-call state + so a failure before notification cannot reuse a prior turn's evidence. +- [ ] Preserve per-role profile/schema/variable selection, transient and + structural-repair budgets, ordinary observers, output decoding, and error + disposition. Map `ErrOperationTimeout` explicitly, including timeouts during + preparation. Distinguish parent cancellation and provider-local deadlines. + Retain the public processing reason codes where their meanings still apply. +- [ ] Ensure call finalization runs on both results and errors; coordinator + execution errors do not arrive as ordinary response callbacks. Preserve the + last attempted operation's diagnostic and failure context for final reporting. + +**Verification:** focused adapter tests for exact prepared evidence after a +provider failure, absent evidence before notification, independent operation +budgets, timeout classification, metadata redaction, and ordinary observer +composition. Use controlled model clients and cancellation. Retain explicit +prepared execution only where an identified consumer actually requires it. + +## 2. Replace history and repair-loop mechanics + +- [ ] Replace the local `Exchange`/`Replay` model in + [structured completion contracts](../../internal/framework/structuredoutput/call.go) + with a minimal opaque bridge to native captured exchanges. The adapter owns + PromptKit execution; modules and validators must not reconstruct messages or + depend on provider-specific state. +- [ ] Capture the original proposal run through `CaptureExchange`. Select the + repair prompt directly for a continuation; remove original-prompt rerendering, + manual assistant/suffix assembly, prefix comparison, and `RepairInstruction` + history plumbing. Preserve prepared snapshots used only for diagnostic output. +- [ ] Replace the attempt loop in + [repair coordination](../../internal/framework/runner/repair.go) with one + adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal. + Keep scheduling across proposals in Audita. Pass the resolved correction repair + limit as `AdditionalTurns`, and bypass coordination when repair is disabled. +- [ ] Keep domain branch state in the callback: original proposal identity, + current revision, local segment mapping, current before/after preview, safe + rejection findings, and domain outcome. The initial callback uses existing + findings without rerunning the initial batch validation. +- [ ] Build each follow-up request with the repair definition, role profile, + repair schema, variables, operation timeout, prepared notification, and selected + observers. Let the coordinator attach `ContinueFrom` and manage history. +- [ ] Process `ConversationTurn.Result` only when present. Check structural + validation before schema decoding, interpret withdrawal, and revalidate every + revision from the beginning using the parent callback context. Validator calls + select their own profiles, timeouts, and observers through the shared adapter. +- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain + outcomes. If rejection remains actionable, request a further turn even at the + budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate + an independent exhaustion loop. The final permitted revision is still assessed. +- [ ] Keep diagnostic attempt coordinates and operation records, but allocate + execution evidence only for runs actually started. The callback may construct + a request that is never executed because the turn budget is exhausted. +- [ ] Map execution errors, callback errors, stopping, exhaustion, and + `Undelivered` separately. Unassessed or structurally invalid content never + reaches the approved set. Preserve accepted sibling proposals and apply-time + checks; fatal/canceled work still cancels and joins peers. + +**Verification:** a controlled workflow with one initial approval and two rejected +proposals demonstrates independent histories, withdrawal, revised acceptance, +full revalidation, and deterministic application. Cover final-turn acceptance, +exhaustion, zero repair, structural invalidity, recoverable failures, parent +cancellation before delivery, and capacity-one validator calls. Assertions should +exercise Audita outcomes through the real upstream coordinator with a fake model +boundary, rather than test a fake coordinator's loop behavior. + +## 3. Adopt native execution accounting + +- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe` + callback with existing diagnostic observers using `ComposeObservers`. Ensure + observation remains enabled wherever required accounting is consumed, + independently of whether debug artifacts are enabled. +- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use + `CoordinateRequest.Observer` for continuation membership and select membership + separately for nested validation requests. Keep conversation parent identity + distinct from Audita's stage/section/proposal/validator labels. +- [ ] Replace the terminal-map deduplication and usage summation in + [observation summaries](../../internal/framework/llm/observations.go) with + `ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation + boundaries. Retain logical call metadata, safe event evidence, statuses, and + deterministic ordering that the upstream group does not supply. +- [ ] Carry terminal coverage separately from usage known/completeness into the + internal summary and its diagnostic representation. Do not present missing + terminals as complete zero accounting. Keep the established ordinary report + scope; adding token payloads to the ordinary errors/recoveries report is not + part of this upgrade. +- [ ] Count shared initial work once in the invocation group; count each new + continuation and validator run separately. Do not sum overlapping subgroup + totals or attempt events on top of their terminal totals. Sort any presented + member lists deterministically rather than relying on native snapshot ordering. +- [ ] Snapshot final totals after all selected work has joined. Groups add no + separate close/wait lifecycle. Audit and delete local aggregation that became + redundant; document the purpose of any remaining application-specific counters. + +**Verification:** integration tests demonstrate composition with existing +observers, explicit validator membership, shared initial work, failure terminal +facts, unknown versus zero usage, missing terminal coverage, and concurrent calls. +Reuse upstream group semantics; test Audita's enrollment and presentation instead +of duplicating the library's unit suite. + +## 4. Cleanup, documentation, and acceptance + +- [ ] Audit references to the old exchange/replay types, prefix checks, + `RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and + terminal aggregation. Delete dead production helpers and replace tests tied to + those implementations with focused behavior coverage. A thin local adapter is + justified; a parallel conversation/coordinator/accounting implementation is not. +- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md), + [pipeline](../internal/pipeline.md), + [diagnostics/reporting internals](../internal/diagnostics-reporting.md), and + [LLM integration contract](../integrations/openai-compatible-llm.md) as behavior + lands. Update pinned upstream references and dependency attribution as needed. + Document the selected notification boundary and timeout mapping without claiming + that preparation or admission failures always have a prepared snapshot. +- [ ] Update configuration/operations references only for actual externally + visible changes. No prompt rewrite, validator tuning, new configuration format, + compatibility shim, new scheduler, or unavailable-decision regeneration is + required by this migration. +- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect + domain and integration behavior, remove obsolete implementation-coupled tests, + and avoid prompt word/phrase change detectors. Review final tests for redundancy. +- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled + check of changed concurrent adapter/runner/accounting paths. Build the CLI and + exercise existing offline processing and inspection coverage. No paid provider + calls are required for acceptance. +- [ ] Review the final diff, verify local Markdown links and `git diff --check`, + and confirm no local dependency replacements or workspace artifacts remain. + Verify the selected PromptKit version is v0.23.0. +- [ ] Confirm all three upstream capability families are used in production + processing and the agreed local machinery is retired. Record completion only + after this end-state audit; move useful current contracts into their canonical + owners and retire these temporary planning documents under repository policy. + +Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain: + +```sh +go list -m go.maximumdirect.net/promptkit +go test ./... +go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport +go build -o ./bin/audita ./cmd/audita +git diff --check +``` + +Adjust focused package checks if code ownership moves. A green dependency-only +suite is not evidence that the new workflow has been adopted; acceptance also +requires the coordinator, conversation, and group integration scenarios above. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md new file mode 100644 index 0000000..0cb8969 --- /dev/null +++ b/docs/roadmap/promptkit.md @@ -0,0 +1,136 @@ +# PromptKit v0.23.0 Adoption Roadmap + +**Status:** Accepted target; implementation pending. This pass changes planning +documents only. The [implementation plan](promptkit-implementation.md) owns the +ordered tasks and verification. + +## Objective and Baseline + +Upgrade Audita directly from PromptKit v0.21.0 to v0.23.0 and fully adopt its +conversation, coordinator, and execution-group APIs. Removing redundant local +machinery is an explicit completion requirement. A dependency-only upgrade or +continuation adoption with the old repair loop retained is an intermediate state. + +The current [dependency manifest](../../go.mod) pins v0.21.0. Audita captures its +own exchange and replay values, reconstructs and compares conversation prefixes, +appends repair history, manages the revision loop, and aggregates terminal +observations locally. Each role client wraps explicit preparation and execution +in a local deadline. See the [LLM runtime](../internal/llm-runtime.md), +[pipeline](../internal/pipeline.md), and +[diagnostic implementation](../internal/diagnostics-reporting.md). + +[PromptKit v0.23.0](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/releases/v0.23.0.md) +includes the v0.22.0 conversation and accounting capabilities plus the execution +controls needed for full coordinator adoption. The former upstream timeout and +prepared-request gaps are closed; no intermediate v0.22.0 rollout or further +upstream feature gate is required. The Go minimum and dependency versions are +unchanged by these upstream releases. + +The release review verified Audita's existing suite against the tagged v0.23.0 +source through a temporary module override and selected upstream packages with +race detection. That is compatibility evidence, not verification of the migration +described here. Implementation must use the released dependency, without a local +replacement or committed workspace file. + +## Target Ownership + +| Responsibility | Target owner | +| --- | --- | +| Immutable captured history, continuation assembly, and parent run identity | PromptKit `Exchange`, `CaptureExchange`, and `ContinueFrom` | +| Sequential turns, additional-turn budget, callback delivery, and coordination termination | PromptKit `Engine.Coordinate` | +| Whole-operation deadlines and timeout attribution | PromptKit `OperationTimeout` and `ErrOperationTimeout` | +| Exact prepared-request notification before generation | PromptKit `OnPrepared` | +| Selected run membership, deduplication, and terminal count/usage aggregation | PromptKit `ExecutionGroup` and `ComposeObservers` | +| Candidate identity, actionable rejection, feedback, revision decoding, withdrawal, and full revalidation | Audita | +| Branch scheduling, accepted sibling proposals, application order, and transcript mutation | Audita | +| Domain operation labels, failure disposition, reports, redaction, and debug artifact storage | Audita | + +Use the supported public API through Audita's framework adapter boundaries. +Keep module and validator contracts domain-oriented. Any bridge needed by the +runner must delegate to the real coordinator and retain opaque conversation +state, rather than implementing another conversation engine or general workflow +framework. + +## Selected Behavior + +### Execution and diagnostics + +Use ordinary `Engine.Run` with per-request `OperationTimeout` and `OnPrepared` +where explicit preparation currently serves only to inspect and execute the same +request. Map the resolved Audita operation timeout onto each proposal, validation, +and repair operation. Keep provider, transport-retry, structural-repair, and +correction-revision budgets distinct. + +Use `ErrOperationTimeout` to identify the selected local budget, while retaining +Audita's parent-cancellation and recoverable-provider-failure handling. Do not +classify every wrapped preparation error as fatal before considering its timeout +identity. Ordinary configuration failures remain fatal. If split preparation is +still needed for a concrete consumer, one caller context must cover both calls +when a total preparation-plus-execution deadline is required; the upstream +prepared-handle duration starts only at execution. + +`OnPrepared` receives the actual detached snapshot. Retain the metadata needed +for attribution and credential-source redaction, and capture content for optional +debug evidence. Keep the callback short; application validation belongs in the +next-turn callback after admission is released. Notification may be absent on +preparation/admission failure or cancellation before delivery. Reports must not +invent missing snapshots. Optional debug write failures remain nonfatal. + +### Conversations and correction repair + +Capture the proposal exchange when repair needs it. Every rejected candidate's +branch starts from that same immutable exchange; subsequent turns remain private +to that branch. Follow-ups select the module's repair prompt, role profile, +variables, and repair output contract through ordinary PromptKit requests. +Retain native operator prompt overrides and the existing domain prompt content. + +Use one coordinator per actionable correction. Its initial callback uses existing +rejection findings. Each newly delivered response must pass structural validation +before decoding; captured content alone does not imply validity. A decoded +revision is fully revalidated before acceptance. Withdrawal, unavailable results, +and accepted revisions remain Audita outcomes; a normal coordinator stop is not +itself an approval. + +The last permitted response follows the same assessment path. Further requested +work beyond the budget maps to exhaustion. Skip coordination when Audita's +correction-repair setting disables it: upstream's zero budget still invokes an +initial callback. Cancellation's `Undelivered` response is diagnostic evidence, +not an approved correction, and must not trigger model work after cancellation. +Recoverable branch failures preserve independent approvals; fatal failures and +caller cancellation retain Audita's cancel-and-join behavior. + +### Execution groups + +Use native groups as the source of aggregate provider-attempt, retry, repair, +and available token totals. Explicitly select membership for proposal calls, +coordinated continuations, and validator calls. Coordinator observers cover its +own turns; nested validators require their own observer selection. + +Maintain an invocation group for combined execution totals. Add narrower groups +only for accounting views Audita consumes. A shared initial proposal run belongs +once to the invocation total; replaying its history does not enroll it again. +Each new call's reported history tokens still count. Overlapping group totals +must not be summed as disjoint work. + +Preserve upstream usage presence/completeness and terminal coverage separately. +Keep domain logical-call records and deterministic presentation where needed; +their count need not equal observed run membership. Group summaries do not replace +Audita's errors/recoveries, validation decisions, or per-operation failure facts. + +## Delivery and Completion + +Implement in ordered, reviewable stages: dependency and execution adapter; +captured conversations and coordinator; execution groups; cleanup, integration +verification, and canonical documentation. All stages belong to this adoption. + +Completion requires removal of local exchange/replay history, prefix checks, +repair turn-budget loops, and duplicate terminal aggregation where replaced by +PromptKit. Retain local behavior only where it has a concrete Audita domain or +presentation responsibility. Preserve transcript correctness, repair prompts, +validation semantics, deterministic reports, standard inspection commands, and +operator configuration unless a deliberate change is documented and tested. + +No product-level open decisions remain. Exact internal bridge signatures are +implementation choices governed by this ownership split. Future validator +decision recovery and new scheduling behavior remain in the +[deferred work catalog](future.md).