Plan full PromptKit v0.23.0 adoption
This commit is contained in:
168
docs/roadmap/promptkit-implementation.md
Normal file
168
docs/roadmap/promptkit-implementation.md
Normal file
@@ -0,0 +1,168 @@
|
||||
# PromptKit v0.23.0 Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation; all tasks below are pending.
|
||||
|
||||
The [adoption roadmap](promptkit.md) owns the target behavior and boundaries.
|
||||
This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed
|
||||
to establish full adoption. It does not authorize application code changes during
|
||||
the current documentation pass.
|
||||
|
||||
## 1. Upgrade and unify execution
|
||||
|
||||
- [ ] Update `go.mod` and `go.sum` directly to
|
||||
`go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove
|
||||
temporary replacements/workspace files. Check the selected module graph and
|
||||
compatibility with the existing CommandKit/ckprompt integration.
|
||||
- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around
|
||||
shared request construction, response interpretation, and error mapping that
|
||||
can serve both ordinary calls and coordinator-owned turns. Keep those helpers
|
||||
narrow; do not wrap `Coordinate` in another turn loop.
|
||||
- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary
|
||||
`Run` for calls whose prepare/run split exists only to obtain diagnostics.
|
||||
Avoid retaining a redundant child timer around the same ordinary operation.
|
||||
- [ ] Supply `OnPrepared` to collect effective metadata and selected credential
|
||||
source names, and retain prepared content for debug when enabled. Use the
|
||||
delivered snapshot without a separate `Prepare` call. Clear per-call state
|
||||
so a failure before notification cannot reuse a prior turn's evidence.
|
||||
- [ ] Preserve per-role profile/schema/variable selection, transient and
|
||||
structural-repair budgets, ordinary observers, output decoding, and error
|
||||
disposition. Map `ErrOperationTimeout` explicitly, including timeouts during
|
||||
preparation. Distinguish parent cancellation and provider-local deadlines.
|
||||
Retain the public processing reason codes where their meanings still apply.
|
||||
- [ ] Ensure call finalization runs on both results and errors; coordinator
|
||||
execution errors do not arrive as ordinary response callbacks. Preserve the
|
||||
last attempted operation's diagnostic and failure context for final reporting.
|
||||
|
||||
**Verification:** focused adapter tests for exact prepared evidence after a
|
||||
provider failure, absent evidence before notification, independent operation
|
||||
budgets, timeout classification, metadata redaction, and ordinary observer
|
||||
composition. Use controlled model clients and cancellation. Retain explicit
|
||||
prepared execution only where an identified consumer actually requires it.
|
||||
|
||||
## 2. Replace history and repair-loop mechanics
|
||||
|
||||
- [ ] Replace the local `Exchange`/`Replay` model in
|
||||
[structured completion contracts](../../internal/framework/structuredoutput/call.go)
|
||||
with a minimal opaque bridge to native captured exchanges. The adapter owns
|
||||
PromptKit execution; modules and validators must not reconstruct messages or
|
||||
depend on provider-specific state.
|
||||
- [ ] Capture the original proposal run through `CaptureExchange`. Select the
|
||||
repair prompt directly for a continuation; remove original-prompt rerendering,
|
||||
manual assistant/suffix assembly, prefix comparison, and `RepairInstruction`
|
||||
history plumbing. Preserve prepared snapshots used only for diagnostic output.
|
||||
- [ ] Replace the attempt loop in
|
||||
[repair coordination](../../internal/framework/runner/repair.go) with one
|
||||
adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal.
|
||||
Keep scheduling across proposals in Audita. Pass the resolved correction repair
|
||||
limit as `AdditionalTurns`, and bypass coordination when repair is disabled.
|
||||
- [ ] Keep domain branch state in the callback: original proposal identity,
|
||||
current revision, local segment mapping, current before/after preview, safe
|
||||
rejection findings, and domain outcome. The initial callback uses existing
|
||||
findings without rerunning the initial batch validation.
|
||||
- [ ] Build each follow-up request with the repair definition, role profile,
|
||||
repair schema, variables, operation timeout, prepared notification, and selected
|
||||
observers. Let the coordinator attach `ContinueFrom` and manage history.
|
||||
- [ ] Process `ConversationTurn.Result` only when present. Check structural
|
||||
validation before schema decoding, interpret withdrawal, and revalidate every
|
||||
revision from the beginning using the parent callback context. Validator calls
|
||||
select their own profiles, timeouts, and observers through the shared adapter.
|
||||
- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain
|
||||
outcomes. If rejection remains actionable, request a further turn even at the
|
||||
budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate
|
||||
an independent exhaustion loop. The final permitted revision is still assessed.
|
||||
- [ ] Keep diagnostic attempt coordinates and operation records, but allocate
|
||||
execution evidence only for runs actually started. The callback may construct
|
||||
a request that is never executed because the turn budget is exhausted.
|
||||
- [ ] Map execution errors, callback errors, stopping, exhaustion, and
|
||||
`Undelivered` separately. Unassessed or structurally invalid content never
|
||||
reaches the approved set. Preserve accepted sibling proposals and apply-time
|
||||
checks; fatal/canceled work still cancels and joins peers.
|
||||
|
||||
**Verification:** a controlled workflow with one initial approval and two rejected
|
||||
proposals demonstrates independent histories, withdrawal, revised acceptance,
|
||||
full revalidation, and deterministic application. Cover final-turn acceptance,
|
||||
exhaustion, zero repair, structural invalidity, recoverable failures, parent
|
||||
cancellation before delivery, and capacity-one validator calls. Assertions should
|
||||
exercise Audita outcomes through the real upstream coordinator with a fake model
|
||||
boundary, rather than test a fake coordinator's loop behavior.
|
||||
|
||||
## 3. Adopt native execution accounting
|
||||
|
||||
- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe`
|
||||
callback with existing diagnostic observers using `ComposeObservers`. Ensure
|
||||
observation remains enabled wherever required accounting is consumed,
|
||||
independently of whether debug artifacts are enabled.
|
||||
- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use
|
||||
`CoordinateRequest.Observer` for continuation membership and select membership
|
||||
separately for nested validation requests. Keep conversation parent identity
|
||||
distinct from Audita's stage/section/proposal/validator labels.
|
||||
- [ ] Replace the terminal-map deduplication and usage summation in
|
||||
[observation summaries](../../internal/framework/llm/observations.go) with
|
||||
`ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation
|
||||
boundaries. Retain logical call metadata, safe event evidence, statuses, and
|
||||
deterministic ordering that the upstream group does not supply.
|
||||
- [ ] Carry terminal coverage separately from usage known/completeness into the
|
||||
internal summary and its diagnostic representation. Do not present missing
|
||||
terminals as complete zero accounting. Keep the established ordinary report
|
||||
scope; adding token payloads to the ordinary errors/recoveries report is not
|
||||
part of this upgrade.
|
||||
- [ ] Count shared initial work once in the invocation group; count each new
|
||||
continuation and validator run separately. Do not sum overlapping subgroup
|
||||
totals or attempt events on top of their terminal totals. Sort any presented
|
||||
member lists deterministically rather than relying on native snapshot ordering.
|
||||
- [ ] Snapshot final totals after all selected work has joined. Groups add no
|
||||
separate close/wait lifecycle. Audit and delete local aggregation that became
|
||||
redundant; document the purpose of any remaining application-specific counters.
|
||||
|
||||
**Verification:** integration tests demonstrate composition with existing
|
||||
observers, explicit validator membership, shared initial work, failure terminal
|
||||
facts, unknown versus zero usage, missing terminal coverage, and concurrent calls.
|
||||
Reuse upstream group semantics; test Audita's enrollment and presentation instead
|
||||
of duplicating the library's unit suite.
|
||||
|
||||
## 4. Cleanup, documentation, and acceptance
|
||||
|
||||
- [ ] Audit references to the old exchange/replay types, prefix checks,
|
||||
`RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and
|
||||
terminal aggregation. Delete dead production helpers and replace tests tied to
|
||||
those implementations with focused behavior coverage. A thin local adapter is
|
||||
justified; a parallel conversation/coordinator/accounting implementation is not.
|
||||
- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md),
|
||||
[diagnostics/reporting internals](../internal/diagnostics-reporting.md), and
|
||||
[LLM integration contract](../integrations/openai-compatible-llm.md) as behavior
|
||||
lands. Update pinned upstream references and dependency attribution as needed.
|
||||
Document the selected notification boundary and timeout mapping without claiming
|
||||
that preparation or admission failures always have a prepared snapshot.
|
||||
- [ ] Update configuration/operations references only for actual externally
|
||||
visible changes. No prompt rewrite, validator tuning, new configuration format,
|
||||
compatibility shim, new scheduler, or unavailable-decision regeneration is
|
||||
required by this migration.
|
||||
- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect
|
||||
domain and integration behavior, remove obsolete implementation-coupled tests,
|
||||
and avoid prompt word/phrase change detectors. Review final tests for redundancy.
|
||||
- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled
|
||||
check of changed concurrent adapter/runner/accounting paths. Build the CLI and
|
||||
exercise existing offline processing and inspection coverage. No paid provider
|
||||
calls are required for acceptance.
|
||||
- [ ] Review the final diff, verify local Markdown links and `git diff --check`,
|
||||
and confirm no local dependency replacements or workspace artifacts remain.
|
||||
Verify the selected PromptKit version is v0.23.0.
|
||||
- [ ] Confirm all three upstream capability families are used in production
|
||||
processing and the agreed local machinery is retired. Record completion only
|
||||
after this end-state audit; move useful current contracts into their canonical
|
||||
owners and retire these temporary planning documents under repository policy.
|
||||
|
||||
Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain:
|
||||
|
||||
```sh
|
||||
go list -m go.maximumdirect.net/promptkit
|
||||
go test ./...
|
||||
go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Adjust focused package checks if code ownership moves. A green dependency-only
|
||||
suite is not evidence that the new workflow has been adopted; acceptance also
|
||||
requires the coordinator, conversation, and group integration scenarios above.
|
||||
136
docs/roadmap/promptkit.md
Normal file
136
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# PromptKit v0.23.0 Adoption Roadmap
|
||||
|
||||
**Status:** Accepted target; implementation pending. This pass changes planning
|
||||
documents only. The [implementation plan](promptkit-implementation.md) owns the
|
||||
ordered tasks and verification.
|
||||
|
||||
## Objective and Baseline
|
||||
|
||||
Upgrade Audita directly from PromptKit v0.21.0 to v0.23.0 and fully adopt its
|
||||
conversation, coordinator, and execution-group APIs. Removing redundant local
|
||||
machinery is an explicit completion requirement. A dependency-only upgrade or
|
||||
continuation adoption with the old repair loop retained is an intermediate state.
|
||||
|
||||
The current [dependency manifest](../../go.mod) pins v0.21.0. Audita captures its
|
||||
own exchange and replay values, reconstructs and compares conversation prefixes,
|
||||
appends repair history, manages the revision loop, and aggregates terminal
|
||||
observations locally. Each role client wraps explicit preparation and execution
|
||||
in a local deadline. See the [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md), and
|
||||
[diagnostic implementation](../internal/diagnostics-reporting.md).
|
||||
|
||||
[PromptKit v0.23.0](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/releases/v0.23.0.md)
|
||||
includes the v0.22.0 conversation and accounting capabilities plus the execution
|
||||
controls needed for full coordinator adoption. The former upstream timeout and
|
||||
prepared-request gaps are closed; no intermediate v0.22.0 rollout or further
|
||||
upstream feature gate is required. The Go minimum and dependency versions are
|
||||
unchanged by these upstream releases.
|
||||
|
||||
The release review verified Audita's existing suite against the tagged v0.23.0
|
||||
source through a temporary module override and selected upstream packages with
|
||||
race detection. That is compatibility evidence, not verification of the migration
|
||||
described here. Implementation must use the released dependency, without a local
|
||||
replacement or committed workspace file.
|
||||
|
||||
## Target Ownership
|
||||
|
||||
| Responsibility | Target owner |
|
||||
| --- | --- |
|
||||
| Immutable captured history, continuation assembly, and parent run identity | PromptKit `Exchange`, `CaptureExchange`, and `ContinueFrom` |
|
||||
| Sequential turns, additional-turn budget, callback delivery, and coordination termination | PromptKit `Engine.Coordinate` |
|
||||
| Whole-operation deadlines and timeout attribution | PromptKit `OperationTimeout` and `ErrOperationTimeout` |
|
||||
| Exact prepared-request notification before generation | PromptKit `OnPrepared` |
|
||||
| Selected run membership, deduplication, and terminal count/usage aggregation | PromptKit `ExecutionGroup` and `ComposeObservers` |
|
||||
| Candidate identity, actionable rejection, feedback, revision decoding, withdrawal, and full revalidation | Audita |
|
||||
| Branch scheduling, accepted sibling proposals, application order, and transcript mutation | Audita |
|
||||
| Domain operation labels, failure disposition, reports, redaction, and debug artifact storage | Audita |
|
||||
|
||||
Use the supported public API through Audita's framework adapter boundaries.
|
||||
Keep module and validator contracts domain-oriented. Any bridge needed by the
|
||||
runner must delegate to the real coordinator and retain opaque conversation
|
||||
state, rather than implementing another conversation engine or general workflow
|
||||
framework.
|
||||
|
||||
## Selected Behavior
|
||||
|
||||
### Execution and diagnostics
|
||||
|
||||
Use ordinary `Engine.Run` with per-request `OperationTimeout` and `OnPrepared`
|
||||
where explicit preparation currently serves only to inspect and execute the same
|
||||
request. Map the resolved Audita operation timeout onto each proposal, validation,
|
||||
and repair operation. Keep provider, transport-retry, structural-repair, and
|
||||
correction-revision budgets distinct.
|
||||
|
||||
Use `ErrOperationTimeout` to identify the selected local budget, while retaining
|
||||
Audita's parent-cancellation and recoverable-provider-failure handling. Do not
|
||||
classify every wrapped preparation error as fatal before considering its timeout
|
||||
identity. Ordinary configuration failures remain fatal. If split preparation is
|
||||
still needed for a concrete consumer, one caller context must cover both calls
|
||||
when a total preparation-plus-execution deadline is required; the upstream
|
||||
prepared-handle duration starts only at execution.
|
||||
|
||||
`OnPrepared` receives the actual detached snapshot. Retain the metadata needed
|
||||
for attribution and credential-source redaction, and capture content for optional
|
||||
debug evidence. Keep the callback short; application validation belongs in the
|
||||
next-turn callback after admission is released. Notification may be absent on
|
||||
preparation/admission failure or cancellation before delivery. Reports must not
|
||||
invent missing snapshots. Optional debug write failures remain nonfatal.
|
||||
|
||||
### Conversations and correction repair
|
||||
|
||||
Capture the proposal exchange when repair needs it. Every rejected candidate's
|
||||
branch starts from that same immutable exchange; subsequent turns remain private
|
||||
to that branch. Follow-ups select the module's repair prompt, role profile,
|
||||
variables, and repair output contract through ordinary PromptKit requests.
|
||||
Retain native operator prompt overrides and the existing domain prompt content.
|
||||
|
||||
Use one coordinator per actionable correction. Its initial callback uses existing
|
||||
rejection findings. Each newly delivered response must pass structural validation
|
||||
before decoding; captured content alone does not imply validity. A decoded
|
||||
revision is fully revalidated before acceptance. Withdrawal, unavailable results,
|
||||
and accepted revisions remain Audita outcomes; a normal coordinator stop is not
|
||||
itself an approval.
|
||||
|
||||
The last permitted response follows the same assessment path. Further requested
|
||||
work beyond the budget maps to exhaustion. Skip coordination when Audita's
|
||||
correction-repair setting disables it: upstream's zero budget still invokes an
|
||||
initial callback. Cancellation's `Undelivered` response is diagnostic evidence,
|
||||
not an approved correction, and must not trigger model work after cancellation.
|
||||
Recoverable branch failures preserve independent approvals; fatal failures and
|
||||
caller cancellation retain Audita's cancel-and-join behavior.
|
||||
|
||||
### Execution groups
|
||||
|
||||
Use native groups as the source of aggregate provider-attempt, retry, repair,
|
||||
and available token totals. Explicitly select membership for proposal calls,
|
||||
coordinated continuations, and validator calls. Coordinator observers cover its
|
||||
own turns; nested validators require their own observer selection.
|
||||
|
||||
Maintain an invocation group for combined execution totals. Add narrower groups
|
||||
only for accounting views Audita consumes. A shared initial proposal run belongs
|
||||
once to the invocation total; replaying its history does not enroll it again.
|
||||
Each new call's reported history tokens still count. Overlapping group totals
|
||||
must not be summed as disjoint work.
|
||||
|
||||
Preserve upstream usage presence/completeness and terminal coverage separately.
|
||||
Keep domain logical-call records and deterministic presentation where needed;
|
||||
their count need not equal observed run membership. Group summaries do not replace
|
||||
Audita's errors/recoveries, validation decisions, or per-operation failure facts.
|
||||
|
||||
## Delivery and Completion
|
||||
|
||||
Implement in ordered, reviewable stages: dependency and execution adapter;
|
||||
captured conversations and coordinator; execution groups; cleanup, integration
|
||||
verification, and canonical documentation. All stages belong to this adoption.
|
||||
|
||||
Completion requires removal of local exchange/replay history, prefix checks,
|
||||
repair turn-budget loops, and duplicate terminal aggregation where replaced by
|
||||
PromptKit. Retain local behavior only where it has a concrete Audita domain or
|
||||
presentation responsibility. Preserve transcript correctness, repair prompts,
|
||||
validation semantics, deterministic reports, standard inspection commands, and
|
||||
operator configuration unless a deliberate change is documented and tested.
|
||||
|
||||
No product-level open decisions remain. Exact internal bridge signatures are
|
||||
implementation choices governed by this ownership split. Future validator
|
||||
decision recovery and new scheduling behavior remain in the
|
||||
[deferred work catalog](future.md).
|
||||
Reference in New Issue
Block a user