169 lines
11 KiB
Markdown
169 lines
11 KiB
Markdown
|
|
# PromptKit v0.23.0 Implementation Plan
|
||
|
|
|
||
|
|
**Status:** Ready for implementation; all tasks below are pending.
|
||
|
|
|
||
|
|
The [adoption roadmap](promptkit.md) owns the target behavior and boundaries.
|
||
|
|
This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed
|
||
|
|
to establish full adoption. It does not authorize application code changes during
|
||
|
|
the current documentation pass.
|
||
|
|
|
||
|
|
## 1. Upgrade and unify execution
|
||
|
|
|
||
|
|
- [ ] Update `go.mod` and `go.sum` directly to
|
||
|
|
`go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove
|
||
|
|
temporary replacements/workspace files. Check the selected module graph and
|
||
|
|
compatibility with the existing CommandKit/ckprompt integration.
|
||
|
|
- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around
|
||
|
|
shared request construction, response interpretation, and error mapping that
|
||
|
|
can serve both ordinary calls and coordinator-owned turns. Keep those helpers
|
||
|
|
narrow; do not wrap `Coordinate` in another turn loop.
|
||
|
|
- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary
|
||
|
|
`Run` for calls whose prepare/run split exists only to obtain diagnostics.
|
||
|
|
Avoid retaining a redundant child timer around the same ordinary operation.
|
||
|
|
- [ ] Supply `OnPrepared` to collect effective metadata and selected credential
|
||
|
|
source names, and retain prepared content for debug when enabled. Use the
|
||
|
|
delivered snapshot without a separate `Prepare` call. Clear per-call state
|
||
|
|
so a failure before notification cannot reuse a prior turn's evidence.
|
||
|
|
- [ ] Preserve per-role profile/schema/variable selection, transient and
|
||
|
|
structural-repair budgets, ordinary observers, output decoding, and error
|
||
|
|
disposition. Map `ErrOperationTimeout` explicitly, including timeouts during
|
||
|
|
preparation. Distinguish parent cancellation and provider-local deadlines.
|
||
|
|
Retain the public processing reason codes where their meanings still apply.
|
||
|
|
- [ ] Ensure call finalization runs on both results and errors; coordinator
|
||
|
|
execution errors do not arrive as ordinary response callbacks. Preserve the
|
||
|
|
last attempted operation's diagnostic and failure context for final reporting.
|
||
|
|
|
||
|
|
**Verification:** focused adapter tests for exact prepared evidence after a
|
||
|
|
provider failure, absent evidence before notification, independent operation
|
||
|
|
budgets, timeout classification, metadata redaction, and ordinary observer
|
||
|
|
composition. Use controlled model clients and cancellation. Retain explicit
|
||
|
|
prepared execution only where an identified consumer actually requires it.
|
||
|
|
|
||
|
|
## 2. Replace history and repair-loop mechanics
|
||
|
|
|
||
|
|
- [ ] Replace the local `Exchange`/`Replay` model in
|
||
|
|
[structured completion contracts](../../internal/framework/structuredoutput/call.go)
|
||
|
|
with a minimal opaque bridge to native captured exchanges. The adapter owns
|
||
|
|
PromptKit execution; modules and validators must not reconstruct messages or
|
||
|
|
depend on provider-specific state.
|
||
|
|
- [ ] Capture the original proposal run through `CaptureExchange`. Select the
|
||
|
|
repair prompt directly for a continuation; remove original-prompt rerendering,
|
||
|
|
manual assistant/suffix assembly, prefix comparison, and `RepairInstruction`
|
||
|
|
history plumbing. Preserve prepared snapshots used only for diagnostic output.
|
||
|
|
- [ ] Replace the attempt loop in
|
||
|
|
[repair coordination](../../internal/framework/runner/repair.go) with one
|
||
|
|
adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal.
|
||
|
|
Keep scheduling across proposals in Audita. Pass the resolved correction repair
|
||
|
|
limit as `AdditionalTurns`, and bypass coordination when repair is disabled.
|
||
|
|
- [ ] Keep domain branch state in the callback: original proposal identity,
|
||
|
|
current revision, local segment mapping, current before/after preview, safe
|
||
|
|
rejection findings, and domain outcome. The initial callback uses existing
|
||
|
|
findings without rerunning the initial batch validation.
|
||
|
|
- [ ] Build each follow-up request with the repair definition, role profile,
|
||
|
|
repair schema, variables, operation timeout, prepared notification, and selected
|
||
|
|
observers. Let the coordinator attach `ContinueFrom` and manage history.
|
||
|
|
- [ ] Process `ConversationTurn.Result` only when present. Check structural
|
||
|
|
validation before schema decoding, interpret withdrawal, and revalidate every
|
||
|
|
revision from the beginning using the parent callback context. Validator calls
|
||
|
|
select their own profiles, timeouts, and observers through the shared adapter.
|
||
|
|
- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain
|
||
|
|
outcomes. If rejection remains actionable, request a further turn even at the
|
||
|
|
budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate
|
||
|
|
an independent exhaustion loop. The final permitted revision is still assessed.
|
||
|
|
- [ ] Keep diagnostic attempt coordinates and operation records, but allocate
|
||
|
|
execution evidence only for runs actually started. The callback may construct
|
||
|
|
a request that is never executed because the turn budget is exhausted.
|
||
|
|
- [ ] Map execution errors, callback errors, stopping, exhaustion, and
|
||
|
|
`Undelivered` separately. Unassessed or structurally invalid content never
|
||
|
|
reaches the approved set. Preserve accepted sibling proposals and apply-time
|
||
|
|
checks; fatal/canceled work still cancels and joins peers.
|
||
|
|
|
||
|
|
**Verification:** a controlled workflow with one initial approval and two rejected
|
||
|
|
proposals demonstrates independent histories, withdrawal, revised acceptance,
|
||
|
|
full revalidation, and deterministic application. Cover final-turn acceptance,
|
||
|
|
exhaustion, zero repair, structural invalidity, recoverable failures, parent
|
||
|
|
cancellation before delivery, and capacity-one validator calls. Assertions should
|
||
|
|
exercise Audita outcomes through the real upstream coordinator with a fake model
|
||
|
|
boundary, rather than test a fake coordinator's loop behavior.
|
||
|
|
|
||
|
|
## 3. Adopt native execution accounting
|
||
|
|
|
||
|
|
- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe`
|
||
|
|
callback with existing diagnostic observers using `ComposeObservers`. Ensure
|
||
|
|
observation remains enabled wherever required accounting is consumed,
|
||
|
|
independently of whether debug artifacts are enabled.
|
||
|
|
- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use
|
||
|
|
`CoordinateRequest.Observer` for continuation membership and select membership
|
||
|
|
separately for nested validation requests. Keep conversation parent identity
|
||
|
|
distinct from Audita's stage/section/proposal/validator labels.
|
||
|
|
- [ ] Replace the terminal-map deduplication and usage summation in
|
||
|
|
[observation summaries](../../internal/framework/llm/observations.go) with
|
||
|
|
`ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation
|
||
|
|
boundaries. Retain logical call metadata, safe event evidence, statuses, and
|
||
|
|
deterministic ordering that the upstream group does not supply.
|
||
|
|
- [ ] Carry terminal coverage separately from usage known/completeness into the
|
||
|
|
internal summary and its diagnostic representation. Do not present missing
|
||
|
|
terminals as complete zero accounting. Keep the established ordinary report
|
||
|
|
scope; adding token payloads to the ordinary errors/recoveries report is not
|
||
|
|
part of this upgrade.
|
||
|
|
- [ ] Count shared initial work once in the invocation group; count each new
|
||
|
|
continuation and validator run separately. Do not sum overlapping subgroup
|
||
|
|
totals or attempt events on top of their terminal totals. Sort any presented
|
||
|
|
member lists deterministically rather than relying on native snapshot ordering.
|
||
|
|
- [ ] Snapshot final totals after all selected work has joined. Groups add no
|
||
|
|
separate close/wait lifecycle. Audit and delete local aggregation that became
|
||
|
|
redundant; document the purpose of any remaining application-specific counters.
|
||
|
|
|
||
|
|
**Verification:** integration tests demonstrate composition with existing
|
||
|
|
observers, explicit validator membership, shared initial work, failure terminal
|
||
|
|
facts, unknown versus zero usage, missing terminal coverage, and concurrent calls.
|
||
|
|
Reuse upstream group semantics; test Audita's enrollment and presentation instead
|
||
|
|
of duplicating the library's unit suite.
|
||
|
|
|
||
|
|
## 4. Cleanup, documentation, and acceptance
|
||
|
|
|
||
|
|
- [ ] Audit references to the old exchange/replay types, prefix checks,
|
||
|
|
`RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and
|
||
|
|
terminal aggregation. Delete dead production helpers and replace tests tied to
|
||
|
|
those implementations with focused behavior coverage. A thin local adapter is
|
||
|
|
justified; a parallel conversation/coordinator/accounting implementation is not.
|
||
|
|
- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md),
|
||
|
|
[pipeline](../internal/pipeline.md),
|
||
|
|
[diagnostics/reporting internals](../internal/diagnostics-reporting.md), and
|
||
|
|
[LLM integration contract](../integrations/openai-compatible-llm.md) as behavior
|
||
|
|
lands. Update pinned upstream references and dependency attribution as needed.
|
||
|
|
Document the selected notification boundary and timeout mapping without claiming
|
||
|
|
that preparation or admission failures always have a prepared snapshot.
|
||
|
|
- [ ] Update configuration/operations references only for actual externally
|
||
|
|
visible changes. No prompt rewrite, validator tuning, new configuration format,
|
||
|
|
compatibility shim, new scheduler, or unavailable-decision regeneration is
|
||
|
|
required by this migration.
|
||
|
|
- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect
|
||
|
|
domain and integration behavior, remove obsolete implementation-coupled tests,
|
||
|
|
and avoid prompt word/phrase change detectors. Review final tests for redundancy.
|
||
|
|
- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled
|
||
|
|
check of changed concurrent adapter/runner/accounting paths. Build the CLI and
|
||
|
|
exercise existing offline processing and inspection coverage. No paid provider
|
||
|
|
calls are required for acceptance.
|
||
|
|
- [ ] Review the final diff, verify local Markdown links and `git diff --check`,
|
||
|
|
and confirm no local dependency replacements or workspace artifacts remain.
|
||
|
|
Verify the selected PromptKit version is v0.23.0.
|
||
|
|
- [ ] Confirm all three upstream capability families are used in production
|
||
|
|
processing and the agreed local machinery is retired. Record completion only
|
||
|
|
after this end-state audit; move useful current contracts into their canonical
|
||
|
|
owners and retire these temporary planning documents under repository policy.
|
||
|
|
|
||
|
|
Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
go list -m go.maximumdirect.net/promptkit
|
||
|
|
go test ./...
|
||
|
|
go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport
|
||
|
|
go build -o ./bin/audita ./cmd/audita
|
||
|
|
git diff --check
|
||
|
|
```
|
||
|
|
|
||
|
|
Adjust focused package checks if code ownership moves. A green dependency-only
|
||
|
|
suite is not evidence that the new workflow has been adopted; acceptance also
|
||
|
|
requires the coordinator, conversation, and group integration scenarios above.
|