Adopt PromptKit v0.23.0 conversations and execution accounting
This commit is contained in:
@@ -1,229 +1,106 @@
|
||||
# Prompt Quality Roadmap
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
**Status:** The earlier six prompt improvements are implemented. This follow-up
|
||||
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
|
||||
validation, and repair prompts through the same execution path used in production.
|
||||
Live-model comparisons and broader context expansion remain deferred.
|
||||
|
||||
**Status:** All six stages implemented and verified. Broader context changes and
|
||||
live-model editorial evaluation remain deferred as described below. The plan
|
||||
followed the review of revision `d155bf4`, which covered all eleven prompt
|
||||
definitions, shared instructions, response schemas, and request assembly.
|
||||
That review identified design concerns, not measured live-model regressions.
|
||||
## Baseline and Boundaries
|
||||
|
||||
## Target State and Boundaries
|
||||
The [prompt registry](../internal/prompts.md),
|
||||
[validator guide](../internal/validators.md), and
|
||||
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
|
||||
prompts separate durable system principles from initial proposal assignments,
|
||||
guide independent and overlapping edits, supply current before/after repair
|
||||
previews, include the target speaker in validation, and illustrate editorial
|
||||
judgments, revision, and withdrawal.
|
||||
|
||||
Audita's proposal, validation, and repair prompts will give each model call a
|
||||
coherent assignment and the evidence needed to carry it out. Proposals will
|
||||
better account for interacting edits, repair will receive an explicit view of
|
||||
its current proposed result, and validation will identify the target speaker.
|
||||
Focused examples will clarify difficult editorial judgments.
|
||||
Preserve these selected decisions:
|
||||
|
||||
Preserve the established design:
|
||||
- Module focus guides generation; otherwise acceptable minor editorial cleanup
|
||||
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
||||
- Keep the carefully tuned meaning-reversal criterion and distinct validator
|
||||
responsibilities. Approvals require no explanation; confidence scores remain absent.
|
||||
- Preserve literal spans, deletion, replacement policies, glossary context,
|
||||
protected spellings, full revalidation, and deterministic application.
|
||||
- Retain English defaults and native operator prompt replacement. Module sequence
|
||||
reevaluation follows the [module guidance](../internal/modules.md).
|
||||
|
||||
- Module focus guides proposal generation while otherwise acceptable minor
|
||||
editorial cleanup remains permissible across modules, as selected in
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
||||
- Validators retain distinct responsibilities. Preserve the carefully tuned
|
||||
meaning-reversal criterion and its focus on reversal of the full assertion.
|
||||
- Keep literal-span matching, deletion semantics, separate glossary context and
|
||||
protected spellings, and full before/after validation.
|
||||
- Keep independent repair conversations with full history, successful withdrawal,
|
||||
and full revalidation of revisions. Validators explain defects; repair owns
|
||||
revision or withdrawal. Approvals continue to need no explanation.
|
||||
- Retain confidence-free responses, English defaults, existing module identities,
|
||||
and native operator prompt replacement. The current module sequence is outside
|
||||
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
|
||||
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
|
||||
coordination, per-operation deadlines, prepared-request notification, and selected
|
||||
run accounting. Audita owns candidate identity, findings, previews, acceptance,
|
||||
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
|
||||
it must not recreate message replay, turn loops, or token aggregation locally.
|
||||
|
||||
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
|
||||
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
|
||||
Update these canonical owners when the corresponding changes land.
|
||||
## 1. Review Complete Native Conversations
|
||||
|
||||
## Implementation Sequence
|
||||
Use captured proposal exchanges and independent coordinator branches to inspect
|
||||
representative proposal-to-repair conversations, including a second revision and
|
||||
withdrawal. Review actual `OnPrepared` snapshots for instruction authority,
|
||||
response-format transitions, glossary/protected-term context, and fresh previews.
|
||||
A follow-up selects its repair definition, role profile, and schema directly.
|
||||
|
||||
### 1. Align Proposal and Repair Instruction Authority
|
||||
Exercise operator replacements through native prompt discovery and precedence.
|
||||
Compare the request that actually executes rather than separately rendering an
|
||||
approximation. Reusing captured history must preserve the proposal context while
|
||||
keeping each correction's repair history private.
|
||||
|
||||
Proposal system messages currently include categorical task restrictions, such
|
||||
as prohibiting punctuation or grammar edits in the homophone module. Repair
|
||||
inherits those messages and then permits acceptable minor editorial cleanup in a
|
||||
later user message. These instructions can compete across authority levels.
|
||||
Record concrete editorial concerns and revise affected prompts when warranted.
|
||||
Do not change validator acceptance or narrow allowable cleanup merely to make
|
||||
proposal and repair assignments identical.
|
||||
|
||||
- Put durable correction principles in the system messages: evidence-supported
|
||||
work, established protection and meaning constraints, and the distinction
|
||||
between requested focus and acceptable editorial corrections.
|
||||
- Put the initial proposal's focused assignment and task-specific exclusions in
|
||||
its user message. Preserve the focused tasks rather than asking every module
|
||||
to perform every kind of cleanup.
|
||||
- Let each repair instruction give the subsequent reassessment assignment,
|
||||
including the selected editorial allowance, without conflicting with an
|
||||
inherited system prohibition.
|
||||
- Review each complete proposal-to-repair conversation, including later repair
|
||||
attempts, for coherent instructions and response-format transitions.
|
||||
## 2. Assess Repair Feedback and Outcomes
|
||||
|
||||
This implements the existing acceptance decision; it does not tighten validators
|
||||
or change the repair lifecycle. Keep the validator prompts' separate remits,
|
||||
especially meaning-reversal review, intact.
|
||||
Use the real coordinator with controlled model responses for deterministic
|
||||
integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion,
|
||||
unavailable responses, and independent accepted siblings. Revalidate each revision
|
||||
from the beginning. A normal coordination stop is not evidence of acceptance;
|
||||
Audita's domain outcome determines that. Canceled undelivered content is evidence
|
||||
only and must never enter the approved set.
|
||||
|
||||
### 2. Guide Independently Useful and Overlapping Corrections
|
||||
Use current target previews and validator-owned findings to assess whether a
|
||||
repair request explains the defect sufficiently. Keep presentation changes in
|
||||
prompt assets and domain feedback construction; keep history and budget mechanics
|
||||
upstream. Retain separate transient retry, structural repair, and correction
|
||||
revision budgets.
|
||||
|
||||
The shared proposal contract already makes each edit refer to the supplied
|
||||
segment rather than the imagined result of another proposal. Add guidance for
|
||||
edits that overlap or depend on one another:
|
||||
## 3. Diagnose Evidence Gaps Using Actual Requests
|
||||
|
||||
- Prefer independently useful corrections with literal spans that identify the
|
||||
intended changes.
|
||||
- Combine changes into one proposal when their spans overlap or when one edit
|
||||
requires the other. Preserve unchanged context needed for unique matching or
|
||||
for selecting an occurrence under the glossary replacement policy.
|
||||
- Continue to allow multiple independent corrections to the same segment.
|
||||
For a concrete rejected correction, compare the captured proposal evidence with
|
||||
what the validator actually received. The target speaker is already included.
|
||||
Broader validation context remains evidence-dependent: propose a focused change
|
||||
only when a captured case demonstrates relevant evidence missing from validation.
|
||||
|
||||
For example, separate proposals to capitalize `hello` and replace
|
||||
`hello world` with `hello world` may both pass validation against the original,
|
||||
but applying the capitalization first prevents the second span from matching.
|
||||
A combined replacement can express both edits against the original text.
|
||||
Use native execution-group totals to distinguish additional model work from
|
||||
shared initial history. Select validator membership explicitly, count the initial
|
||||
proposal once, and do not add overlapping group totals. Preserve unknown usage
|
||||
and terminal coverage separately. These measurements describe execution and cost;
|
||||
they do not establish editorial quality.
|
||||
|
||||
These are prompt instructions. Add no new runtime overlap rejection, proposal
|
||||
limit, or application-order rule. Assess combined semantic effects separately
|
||||
if practical examples establish a need for additional processing changes.
|
||||
Existing local examples have not established a need for broader validation windows
|
||||
or combined semantic validation. Keep context fitting and apply-time conflict
|
||||
handling unchanged until concrete cases justify further work.
|
||||
|
||||
### 3. Supply Repair with the Current Before/After Segment
|
||||
## 4. Verification and Documentation
|
||||
|
||||
The repair request currently identifies the proposal, its current correction,
|
||||
and validation findings. Although the original section remains available in
|
||||
conversation history, repair must locate it and reconstruct the proposed result.
|
||||
Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter
|
||||
and coordinator with deterministic model-boundary fixtures. Protect supplied-data
|
||||
rendering, schemas, profile selection, current previews, branching, revalidation,
|
||||
withdrawal, and operator replacement. Inspect representative rendered prose as
|
||||
part of review; avoid phrase checks, prose snapshots, and tests that duplicate
|
||||
upstream history or accounting algorithms.
|
||||
|
||||
- Add the original target segment and its proposed resulting text to repair
|
||||
context when the current proposal is applicable.
|
||||
- Reuse the same domain preview semantics used by validation, including literal
|
||||
deletion, whitespace, and replacement of every matching occurrence where
|
||||
applicable. Avoid a second implementation of replacement behavior.
|
||||
- Refresh this view for each repair attempt, using the current proposal version
|
||||
and target rather than a stale view of the initial proposal.
|
||||
- When a preview cannot be produced, retain the relevant findings and available
|
||||
source context without fabricating a resulting segment or implying application
|
||||
succeeded.
|
||||
- Keep the full conversation history, proposal references, and revision or
|
||||
withdrawal response contract. Explain the added view as evidence for assessing
|
||||
the current correction, not an already-applied transcript change.
|
||||
Update canonical prompt and validator documentation when behavior changes.
|
||||
Keep examples valid against their response schemas and verify affected links.
|
||||
Run focused tests and the full offline suite for runtime changes. Pure wording
|
||||
changes need proportionate review, not new change-detector tests.
|
||||
|
||||
This should make spacing, deletion, and replacement consequences easier to
|
||||
judge without requiring the model to reproduce Audita's application mechanics.
|
||||
## Deferred Evaluation
|
||||
|
||||
### 4. Improve Validation Evidence
|
||||
Formal corpus preparation and live-model comparisons remain optional follow-up
|
||||
work. Keep live calls opt-in and outside the offline suite. When undertaken,
|
||||
compare needed and unnecessary edits, missed corrections, validator disagreement,
|
||||
and repair outcomes using independent editorial judgment. Native groups can
|
||||
supply execution measurements without introducing a second accounting system.
|
||||
|
||||
Add the target segment's speaker to each validation item. Neighboring segments
|
||||
already carry their speakers; the candidate's identity completes that context
|
||||
and helps distinguish continued speech, interruptions, and another speaker's
|
||||
response. Carry the supplied value through the existing payload construction
|
||||
and token accounting without inferring a missing speaker.
|
||||
|
||||
Investigate the broader evidence imbalance using available practical cases:
|
||||
proposal sees a section, while validation sees immediate neighbors and selected
|
||||
glossary entries. A correction supported several turns earlier may appear
|
||||
unsupported to a validator. Compare the evidence actually supplied to both
|
||||
calls before attributing a rejection to prompt wording.
|
||||
|
||||
Adding the target speaker is agreed implementation work. Substantial expansion
|
||||
of validation context remains evidence-dependent: record a concrete case and a
|
||||
focused follow-up recommendation if needed. Retain the current context selection
|
||||
and fitting behavior otherwise. Stronger instructions cannot supply missing
|
||||
evidence, and this roadmap does not require a broader context redesign.
|
||||
|
||||
### 5. Add Focused Editorial and Repair Examples
|
||||
|
||||
Add a small number of examples where they clarify judgments that current
|
||||
instructions leave implicit:
|
||||
|
||||
- Apparent filler that contributes meaning, such as `like` expressing
|
||||
approximation or `I mean` introducing a substantive self-correction. Use
|
||||
surrounding context to contrast meaningful usage with removable dysfluency.
|
||||
- One compact repair revision and one withdrawal, using the existing response
|
||||
schema and making withdrawal a successful preservation of the original text.
|
||||
- A combined overlapping edit illustrating step 2.
|
||||
|
||||
Place examples in the relevant proposal, validation, or shared repair guidance.
|
||||
Keep existing useful contextual contrasts. Examples should illustrate reasoning
|
||||
without becoming unconditional token-based editing rules or an extensive
|
||||
rulebook. Maintain the narrow meaning-reversal instructions.
|
||||
|
||||
### 6. Verify Integration, Documentation, and Test-Policy Compliance
|
||||
|
||||
Follow the [testing policy](../policy/testing.md) throughout implementation and
|
||||
finish with a focused review of affected tests:
|
||||
|
||||
- Verify repair context reflects the current proposal and the shared preview
|
||||
behavior, including representative deletion, replacement-policy, subsequent
|
||||
revision, and unavailable-preview cases where coverage is warranted.
|
||||
- Verify target-speaker data reaches validation and participates in existing
|
||||
payload accounting. Preserve coverage of response handling, revision,
|
||||
withdrawal, revalidation, and operator prompt replacement.
|
||||
- Review rendered messages and representative complete repair conversations
|
||||
directly for instruction authority, editorial intent, and format consistency.
|
||||
Check response examples against their schemas.
|
||||
- Remove any affected tests that freeze bundled prompt wording through phrase
|
||||
checks, exact prose comparisons, or snapshots. Do not replace them with checks
|
||||
for the new wording. Retain tests of supplied data and meaningful machine
|
||||
contracts where their protective value justifies their maintenance cost.
|
||||
- Update the relevant canonical prompt, module, validator, and runtime documents
|
||||
as behavior lands. Update maintained examples where affected; check links and
|
||||
whitespace.
|
||||
|
||||
Run the appropriate package checks, build, and full offline suite for the
|
||||
implementation under the [development guidance](../development.md#baseline-validation).
|
||||
Report the test-policy review's meaningful removals, revisions, and retained
|
||||
coverage. Offline tests establish data and execution contracts; they do not
|
||||
establish live-model editorial quality.
|
||||
|
||||
## Deferred Evidence and Follow-Up
|
||||
|
||||
The evidence review examined the maintained local, campaign, and prompt-edit
|
||||
transcript examples, adjacent-context validation fixtures, and proposal
|
||||
application fixtures. The glossary example is a single-segment correction;
|
||||
the validation fixture supplies the troupe definition and adjacent evidence
|
||||
needed to assess its proposed name change. These sources contain no recorded
|
||||
live-model rejection demonstrating a need for more distant context. The target
|
||||
speaker is now included; broader selection and context fitting remain unchanged.
|
||||
Revisit the window when a captured case shows material evidence in the proposal
|
||||
request that was absent from validation.
|
||||
|
||||
The existing stale-span application fixture demonstrates the mechanical overlap
|
||||
addressed by the new guidance. It does not establish a need for new combined
|
||||
semantic validation, so application ordering and rejection rules remain unchanged.
|
||||
|
||||
Formal evaluation corpus preparation and live-model comparisons remain deferred
|
||||
and are not prerequisites for the agreed changes. Broader validation context and
|
||||
additional handling of combined semantic effects should follow concrete evidence
|
||||
rather than hypothetical restrictions. Any later evaluation should examine
|
||||
needed and unnecessary edits, missed corrections, validator disagreement, and
|
||||
repair outcomes using independent editorial judgment. Keep live calls opt-in
|
||||
and separate from the default offline suite.
|
||||
|
||||
No blocking product decisions remain for the implementation scope above.
|
||||
|
||||
## Completion and Verification
|
||||
|
||||
- The four proposal definitions now separate durable system principles from
|
||||
their initial user assignments. The shared proposal guidance covers overlapping
|
||||
and dependent edits; the grammar prompt illustrates a combined edit.
|
||||
- Every repair attempt receives the current target's original text when
|
||||
available and its previewed result when applicable, using the same domain
|
||||
replacement function as validation. Revision and withdrawal examples accompany
|
||||
the shared repair guidance.
|
||||
- Validation includes the target speaker in rendered input and token accounting.
|
||||
Spoken-word proposal and editorial review examples distinguish meaningful
|
||||
apparent filler from removable hesitation. The meaning-reversal definition and
|
||||
all response schemas retain their existing contracts.
|
||||
- Behavioral tests cover repair deletion and replace-all previews, ambiguous or
|
||||
missing spans, blank results, missing targets, and refresh after retargeting.
|
||||
Existing validation tests now cover target-speaker propagation, absence, and
|
||||
budget fitting, including parity with serialized payload estimates.
|
||||
- The affected prompt, module, request-assembly, and conversation tests were
|
||||
reviewed for wording checks. No further bundled-prose change detectors were
|
||||
found to remove; the earlier cleanup remains effective. Retained assertions
|
||||
protect supplied-data rendering, schema selection, operator replacement,
|
||||
conversation history, revalidation, withdrawal, and outcomes. New assertions
|
||||
check request data and behavior rather than prompt phrases.
|
||||
- All eleven definitions prepared through the CLI. Four illustrative complete
|
||||
conversations, including subsequent repair requests, were reviewed, and the
|
||||
sixteen embedded JSON response examples passed schema checks. The application
|
||||
build and full offline Go suite passed. Canonical documentation and maintained
|
||||
preview examples were updated; links and whitespace were checked.
|
||||
|
||||
These checks verify implementation and editorial intent, not live-model quality.
|
||||
No blocking product decisions remain for this follow-up scope.
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
# PromptKit v0.23.0 Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation; all tasks below are pending.
|
||||
|
||||
The [adoption roadmap](promptkit.md) owns the target behavior and boundaries.
|
||||
This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed
|
||||
to establish full adoption. It does not authorize application code changes during
|
||||
the current documentation pass.
|
||||
|
||||
## 1. Upgrade and unify execution
|
||||
|
||||
- [ ] Update `go.mod` and `go.sum` directly to
|
||||
`go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove
|
||||
temporary replacements/workspace files. Check the selected module graph and
|
||||
compatibility with the existing CommandKit/ckprompt integration.
|
||||
- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around
|
||||
shared request construction, response interpretation, and error mapping that
|
||||
can serve both ordinary calls and coordinator-owned turns. Keep those helpers
|
||||
narrow; do not wrap `Coordinate` in another turn loop.
|
||||
- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary
|
||||
`Run` for calls whose prepare/run split exists only to obtain diagnostics.
|
||||
Avoid retaining a redundant child timer around the same ordinary operation.
|
||||
- [ ] Supply `OnPrepared` to collect effective metadata and selected credential
|
||||
source names, and retain prepared content for debug when enabled. Use the
|
||||
delivered snapshot without a separate `Prepare` call. Clear per-call state
|
||||
so a failure before notification cannot reuse a prior turn's evidence.
|
||||
- [ ] Preserve per-role profile/schema/variable selection, transient and
|
||||
structural-repair budgets, ordinary observers, output decoding, and error
|
||||
disposition. Map `ErrOperationTimeout` explicitly, including timeouts during
|
||||
preparation. Distinguish parent cancellation and provider-local deadlines.
|
||||
Retain the public processing reason codes where their meanings still apply.
|
||||
- [ ] Ensure call finalization runs on both results and errors; coordinator
|
||||
execution errors do not arrive as ordinary response callbacks. Preserve the
|
||||
last attempted operation's diagnostic and failure context for final reporting.
|
||||
|
||||
**Verification:** focused adapter tests for exact prepared evidence after a
|
||||
provider failure, absent evidence before notification, independent operation
|
||||
budgets, timeout classification, metadata redaction, and ordinary observer
|
||||
composition. Use controlled model clients and cancellation. Retain explicit
|
||||
prepared execution only where an identified consumer actually requires it.
|
||||
|
||||
## 2. Replace history and repair-loop mechanics
|
||||
|
||||
- [ ] Replace the local `Exchange`/`Replay` model in
|
||||
[structured completion contracts](../../internal/framework/structuredoutput/call.go)
|
||||
with a minimal opaque bridge to native captured exchanges. The adapter owns
|
||||
PromptKit execution; modules and validators must not reconstruct messages or
|
||||
depend on provider-specific state.
|
||||
- [ ] Capture the original proposal run through `CaptureExchange`. Select the
|
||||
repair prompt directly for a continuation; remove original-prompt rerendering,
|
||||
manual assistant/suffix assembly, prefix comparison, and `RepairInstruction`
|
||||
history plumbing. Preserve prepared snapshots used only for diagnostic output.
|
||||
- [ ] Replace the attempt loop in
|
||||
[repair coordination](../../internal/framework/runner/repair.go) with one
|
||||
adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal.
|
||||
Keep scheduling across proposals in Audita. Pass the resolved correction repair
|
||||
limit as `AdditionalTurns`, and bypass coordination when repair is disabled.
|
||||
- [ ] Keep domain branch state in the callback: original proposal identity,
|
||||
current revision, local segment mapping, current before/after preview, safe
|
||||
rejection findings, and domain outcome. The initial callback uses existing
|
||||
findings without rerunning the initial batch validation.
|
||||
- [ ] Build each follow-up request with the repair definition, role profile,
|
||||
repair schema, variables, operation timeout, prepared notification, and selected
|
||||
observers. Let the coordinator attach `ContinueFrom` and manage history.
|
||||
- [ ] Process `ConversationTurn.Result` only when present. Check structural
|
||||
validation before schema decoding, interpret withdrawal, and revalidate every
|
||||
revision from the beginning using the parent callback context. Validator calls
|
||||
select their own profiles, timeouts, and observers through the shared adapter.
|
||||
- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain
|
||||
outcomes. If rejection remains actionable, request a further turn even at the
|
||||
budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate
|
||||
an independent exhaustion loop. The final permitted revision is still assessed.
|
||||
- [ ] Keep diagnostic attempt coordinates and operation records, but allocate
|
||||
execution evidence only for runs actually started. The callback may construct
|
||||
a request that is never executed because the turn budget is exhausted.
|
||||
- [ ] Map execution errors, callback errors, stopping, exhaustion, and
|
||||
`Undelivered` separately. Unassessed or structurally invalid content never
|
||||
reaches the approved set. Preserve accepted sibling proposals and apply-time
|
||||
checks; fatal/canceled work still cancels and joins peers.
|
||||
|
||||
**Verification:** a controlled workflow with one initial approval and two rejected
|
||||
proposals demonstrates independent histories, withdrawal, revised acceptance,
|
||||
full revalidation, and deterministic application. Cover final-turn acceptance,
|
||||
exhaustion, zero repair, structural invalidity, recoverable failures, parent
|
||||
cancellation before delivery, and capacity-one validator calls. Assertions should
|
||||
exercise Audita outcomes through the real upstream coordinator with a fake model
|
||||
boundary, rather than test a fake coordinator's loop behavior.
|
||||
|
||||
## 3. Adopt native execution accounting
|
||||
|
||||
- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe`
|
||||
callback with existing diagnostic observers using `ComposeObservers`. Ensure
|
||||
observation remains enabled wherever required accounting is consumed,
|
||||
independently of whether debug artifacts are enabled.
|
||||
- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use
|
||||
`CoordinateRequest.Observer` for continuation membership and select membership
|
||||
separately for nested validation requests. Keep conversation parent identity
|
||||
distinct from Audita's stage/section/proposal/validator labels.
|
||||
- [ ] Replace the terminal-map deduplication and usage summation in
|
||||
[observation summaries](../../internal/framework/llm/observations.go) with
|
||||
`ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation
|
||||
boundaries. Retain logical call metadata, safe event evidence, statuses, and
|
||||
deterministic ordering that the upstream group does not supply.
|
||||
- [ ] Carry terminal coverage separately from usage known/completeness into the
|
||||
internal summary and its diagnostic representation. Do not present missing
|
||||
terminals as complete zero accounting. Keep the established ordinary report
|
||||
scope; adding token payloads to the ordinary errors/recoveries report is not
|
||||
part of this upgrade.
|
||||
- [ ] Count shared initial work once in the invocation group; count each new
|
||||
continuation and validator run separately. Do not sum overlapping subgroup
|
||||
totals or attempt events on top of their terminal totals. Sort any presented
|
||||
member lists deterministically rather than relying on native snapshot ordering.
|
||||
- [ ] Snapshot final totals after all selected work has joined. Groups add no
|
||||
separate close/wait lifecycle. Audit and delete local aggregation that became
|
||||
redundant; document the purpose of any remaining application-specific counters.
|
||||
|
||||
**Verification:** integration tests demonstrate composition with existing
|
||||
observers, explicit validator membership, shared initial work, failure terminal
|
||||
facts, unknown versus zero usage, missing terminal coverage, and concurrent calls.
|
||||
Reuse upstream group semantics; test Audita's enrollment and presentation instead
|
||||
of duplicating the library's unit suite.
|
||||
|
||||
## 4. Cleanup, documentation, and acceptance
|
||||
|
||||
- [ ] Audit references to the old exchange/replay types, prefix checks,
|
||||
`RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and
|
||||
terminal aggregation. Delete dead production helpers and replace tests tied to
|
||||
those implementations with focused behavior coverage. A thin local adapter is
|
||||
justified; a parallel conversation/coordinator/accounting implementation is not.
|
||||
- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md),
|
||||
[diagnostics/reporting internals](../internal/diagnostics-reporting.md), and
|
||||
[LLM integration contract](../integrations/openai-compatible-llm.md) as behavior
|
||||
lands. Update pinned upstream references and dependency attribution as needed.
|
||||
Document the selected notification boundary and timeout mapping without claiming
|
||||
that preparation or admission failures always have a prepared snapshot.
|
||||
- [ ] Update configuration/operations references only for actual externally
|
||||
visible changes. No prompt rewrite, validator tuning, new configuration format,
|
||||
compatibility shim, new scheduler, or unavailable-decision regeneration is
|
||||
required by this migration.
|
||||
- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect
|
||||
domain and integration behavior, remove obsolete implementation-coupled tests,
|
||||
and avoid prompt word/phrase change detectors. Review final tests for redundancy.
|
||||
- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled
|
||||
check of changed concurrent adapter/runner/accounting paths. Build the CLI and
|
||||
exercise existing offline processing and inspection coverage. No paid provider
|
||||
calls are required for acceptance.
|
||||
- [ ] Review the final diff, verify local Markdown links and `git diff --check`,
|
||||
and confirm no local dependency replacements or workspace artifacts remain.
|
||||
Verify the selected PromptKit version is v0.23.0.
|
||||
- [ ] Confirm all three upstream capability families are used in production
|
||||
processing and the agreed local machinery is retired. Record completion only
|
||||
after this end-state audit; move useful current contracts into their canonical
|
||||
owners and retire these temporary planning documents under repository policy.
|
||||
|
||||
Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain:
|
||||
|
||||
```sh
|
||||
go list -m go.maximumdirect.net/promptkit
|
||||
go test ./...
|
||||
go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Adjust focused package checks if code ownership moves. A green dependency-only
|
||||
suite is not evidence that the new workflow has been adopted; acceptance also
|
||||
requires the coordinator, conversation, and group integration scenarios above.
|
||||
@@ -1,136 +0,0 @@
|
||||
# PromptKit v0.23.0 Adoption Roadmap
|
||||
|
||||
**Status:** Accepted target; implementation pending. This pass changes planning
|
||||
documents only. The [implementation plan](promptkit-implementation.md) owns the
|
||||
ordered tasks and verification.
|
||||
|
||||
## Objective and Baseline
|
||||
|
||||
Upgrade Audita directly from PromptKit v0.21.0 to v0.23.0 and fully adopt its
|
||||
conversation, coordinator, and execution-group APIs. Removing redundant local
|
||||
machinery is an explicit completion requirement. A dependency-only upgrade or
|
||||
continuation adoption with the old repair loop retained is an intermediate state.
|
||||
|
||||
The current [dependency manifest](../../go.mod) pins v0.21.0. Audita captures its
|
||||
own exchange and replay values, reconstructs and compares conversation prefixes,
|
||||
appends repair history, manages the revision loop, and aggregates terminal
|
||||
observations locally. Each role client wraps explicit preparation and execution
|
||||
in a local deadline. See the [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md), and
|
||||
[diagnostic implementation](../internal/diagnostics-reporting.md).
|
||||
|
||||
[PromptKit v0.23.0](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/releases/v0.23.0.md)
|
||||
includes the v0.22.0 conversation and accounting capabilities plus the execution
|
||||
controls needed for full coordinator adoption. The former upstream timeout and
|
||||
prepared-request gaps are closed; no intermediate v0.22.0 rollout or further
|
||||
upstream feature gate is required. The Go minimum and dependency versions are
|
||||
unchanged by these upstream releases.
|
||||
|
||||
The release review verified Audita's existing suite against the tagged v0.23.0
|
||||
source through a temporary module override and selected upstream packages with
|
||||
race detection. That is compatibility evidence, not verification of the migration
|
||||
described here. Implementation must use the released dependency, without a local
|
||||
replacement or committed workspace file.
|
||||
|
||||
## Target Ownership
|
||||
|
||||
| Responsibility | Target owner |
|
||||
| --- | --- |
|
||||
| Immutable captured history, continuation assembly, and parent run identity | PromptKit `Exchange`, `CaptureExchange`, and `ContinueFrom` |
|
||||
| Sequential turns, additional-turn budget, callback delivery, and coordination termination | PromptKit `Engine.Coordinate` |
|
||||
| Whole-operation deadlines and timeout attribution | PromptKit `OperationTimeout` and `ErrOperationTimeout` |
|
||||
| Exact prepared-request notification before generation | PromptKit `OnPrepared` |
|
||||
| Selected run membership, deduplication, and terminal count/usage aggregation | PromptKit `ExecutionGroup` and `ComposeObservers` |
|
||||
| Candidate identity, actionable rejection, feedback, revision decoding, withdrawal, and full revalidation | Audita |
|
||||
| Branch scheduling, accepted sibling proposals, application order, and transcript mutation | Audita |
|
||||
| Domain operation labels, failure disposition, reports, redaction, and debug artifact storage | Audita |
|
||||
|
||||
Use the supported public API through Audita's framework adapter boundaries.
|
||||
Keep module and validator contracts domain-oriented. Any bridge needed by the
|
||||
runner must delegate to the real coordinator and retain opaque conversation
|
||||
state, rather than implementing another conversation engine or general workflow
|
||||
framework.
|
||||
|
||||
## Selected Behavior
|
||||
|
||||
### Execution and diagnostics
|
||||
|
||||
Use ordinary `Engine.Run` with per-request `OperationTimeout` and `OnPrepared`
|
||||
where explicit preparation currently serves only to inspect and execute the same
|
||||
request. Map the resolved Audita operation timeout onto each proposal, validation,
|
||||
and repair operation. Keep provider, transport-retry, structural-repair, and
|
||||
correction-revision budgets distinct.
|
||||
|
||||
Use `ErrOperationTimeout` to identify the selected local budget, while retaining
|
||||
Audita's parent-cancellation and recoverable-provider-failure handling. Do not
|
||||
classify every wrapped preparation error as fatal before considering its timeout
|
||||
identity. Ordinary configuration failures remain fatal. If split preparation is
|
||||
still needed for a concrete consumer, one caller context must cover both calls
|
||||
when a total preparation-plus-execution deadline is required; the upstream
|
||||
prepared-handle duration starts only at execution.
|
||||
|
||||
`OnPrepared` receives the actual detached snapshot. Retain the metadata needed
|
||||
for attribution and credential-source redaction, and capture content for optional
|
||||
debug evidence. Keep the callback short; application validation belongs in the
|
||||
next-turn callback after admission is released. Notification may be absent on
|
||||
preparation/admission failure or cancellation before delivery. Reports must not
|
||||
invent missing snapshots. Optional debug write failures remain nonfatal.
|
||||
|
||||
### Conversations and correction repair
|
||||
|
||||
Capture the proposal exchange when repair needs it. Every rejected candidate's
|
||||
branch starts from that same immutable exchange; subsequent turns remain private
|
||||
to that branch. Follow-ups select the module's repair prompt, role profile,
|
||||
variables, and repair output contract through ordinary PromptKit requests.
|
||||
Retain native operator prompt overrides and the existing domain prompt content.
|
||||
|
||||
Use one coordinator per actionable correction. Its initial callback uses existing
|
||||
rejection findings. Each newly delivered response must pass structural validation
|
||||
before decoding; captured content alone does not imply validity. A decoded
|
||||
revision is fully revalidated before acceptance. Withdrawal, unavailable results,
|
||||
and accepted revisions remain Audita outcomes; a normal coordinator stop is not
|
||||
itself an approval.
|
||||
|
||||
The last permitted response follows the same assessment path. Further requested
|
||||
work beyond the budget maps to exhaustion. Skip coordination when Audita's
|
||||
correction-repair setting disables it: upstream's zero budget still invokes an
|
||||
initial callback. Cancellation's `Undelivered` response is diagnostic evidence,
|
||||
not an approved correction, and must not trigger model work after cancellation.
|
||||
Recoverable branch failures preserve independent approvals; fatal failures and
|
||||
caller cancellation retain Audita's cancel-and-join behavior.
|
||||
|
||||
### Execution groups
|
||||
|
||||
Use native groups as the source of aggregate provider-attempt, retry, repair,
|
||||
and available token totals. Explicitly select membership for proposal calls,
|
||||
coordinated continuations, and validator calls. Coordinator observers cover its
|
||||
own turns; nested validators require their own observer selection.
|
||||
|
||||
Maintain an invocation group for combined execution totals. Add narrower groups
|
||||
only for accounting views Audita consumes. A shared initial proposal run belongs
|
||||
once to the invocation total; replaying its history does not enroll it again.
|
||||
Each new call's reported history tokens still count. Overlapping group totals
|
||||
must not be summed as disjoint work.
|
||||
|
||||
Preserve upstream usage presence/completeness and terminal coverage separately.
|
||||
Keep domain logical-call records and deterministic presentation where needed;
|
||||
their count need not equal observed run membership. Group summaries do not replace
|
||||
Audita's errors/recoveries, validation decisions, or per-operation failure facts.
|
||||
|
||||
## Delivery and Completion
|
||||
|
||||
Implement in ordered, reviewable stages: dependency and execution adapter;
|
||||
captured conversations and coordinator; execution groups; cleanup, integration
|
||||
verification, and canonical documentation. All stages belong to this adoption.
|
||||
|
||||
Completion requires removal of local exchange/replay history, prefix checks,
|
||||
repair turn-budget loops, and duplicate terminal aggregation where replaced by
|
||||
PromptKit. Retain local behavior only where it has a concrete Audita domain or
|
||||
presentation responsibility. Preserve transcript correctness, repair prompts,
|
||||
validation semantics, deterministic reports, standard inspection commands, and
|
||||
operator configuration unless a deliberate change is documented and tested.
|
||||
|
||||
No product-level open decisions remain. Exact internal bridge signatures are
|
||||
implementation choices governed by this ownership split. Future validator
|
||||
decision recovery and new scheduling behavior remain in the
|
||||
[deferred work catalog](future.md).
|
||||
Reference in New Issue
Block a user