diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..d1ecf02 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,104 @@ +# Prompt Quality Implementation Plan + +Follow the stages in order within one sprint. Intermediate stages may leave +dependent edits in progress. Verify the integrated result at the end of the +sprint. Read the [feature roadmap](prompt-quality.md) for the selected behavior +and [developer guide](../development.md) for the applicable policies. + +## Stage 1 — Finish the PromptKit Adoption Follow-ups + +1. In the debug case of `TestSubprocessCorrectionRepairConversation`, read the + run's `llm-execution.json` and compare its top-level + `summary.provider_attempts` with the number of requests captured by the + provider fixture. Use the existing fixture with an initial proposal, repair + branches, and nested validation. This checks that the shared proposal is + counted once and that repair and validation calls join the invocation total. + [Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go) +2. In `repairCandidates`, remove the extra JSON unmarshal and use the typed + response from one decode for the domain decision and the interaction + artifact. Reuse `DecodeRepairResponse` and its existing proposal-reference + and action checks. Where a decoded response fails a correspondence check, + retain it as diagnostic evidence alongside the raw response and failure. + Assess only delivered, structurally valid content. Keep the existing + invalid-output disposition, failure attribution, and revalidation behavior. + [Repair runner](../../internal/framework/runner/repair.go) + [Repair decoder](../../internal/framework/proposal_generation/repair.go) +3. In `BuildLLMExecution`, call `llm.Summarize` once per call and use the result + for both terminal coverage and the serialized summary. Keep the invocation + summary sourced from the native group. Preserve field names, ordering, and + the distinction between unknown usage and zero usage. + [Execution report](../../internal/framework/processreport/llm.go) + +## Stage 2 — Review Proposal and Repair Conversations + +1. Use the existing subprocess fixture to exercise the production adapter and + coordinator with scripted responses. Inspect its captured request messages + and response schemas for the initial proposal and the two-turn repair. + Inspect the existing withdrawal case in the runner fixture as well. Use + prepared request artifacts for the rendered instructions and wire requests + for the effective schema. Inspect the shipped definition and an operator + replacement from the existing fixture. Use temporary test logging if + captured requests or artifacts are otherwise discarded after a passing run. + [Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go) + [Runner fixtures](../../internal/framework/runner/repair_test.go) +2. Check whether the original system instructions, the initial assignment, + and the subsequent repair assignment give a coherent task. Check that the + continuation retains the relevant glossary and protected spellings when + supplied, selects the repair definition, role profile, and one-result + schema, and includes each branch's own revision history. Compare the latest + target and before/after preview with the candidate under review, including + a retargeted edit and the existing omission of an inapplicable after-preview. + Use `TestRoleClientContinuesIndependentRepairConversations` for distinct + proposal and repair profiles. + [Adapter conversation fixture](../../internal/framework/llm/conversation_test.go) +3. Check whether the findings give the model a specific defect to address, + whether the current preview identifies the edit being assessed, and whether + the instruction makes revision and withdrawal clear. Fix concrete + ambiguities in the affected bundled prompt or feedback presentation. Keep + module focus, editorial acceptance, and required evidence as specified by + the feature roadmap. If the requests are coherent, record the reviewed + cases and the outcome in the sprint summary. [Prompt registry](../internal/prompts.md) + +## Stage 3 — Review Evidence at the Validator Boundary + +1. Follow a rejected candidate and, where available, its revision through the + captured proposal, validator request, decision, and next repair request. + Compare the target speaker, literal edit, before/after text, supplied + neighbors, matching glossary entries, and validator's stated reason. + Account for optional context omitted by the existing payload fitting. + [Validator payload and fitting](../internal/validators.md#supplemental-validation-context) +2. If evidence is already present but the instruction obscures how to use it, + revise the relevant prompt guidance or example. Preserve each validator's + remit, including the full-assertion reversal criterion. If the case needs + unavailable evidence, describe the missing evidence and the smallest + candidate payload change in the sprint summary. Broader context expansion + and combined semantic validation remain deferred. If no concrete gap + appears, the captured comparison completes this stage. +3. Use the native invocation summary to describe any additional requests + attributable to repair and revalidation. Use per-call evidence to identify + their roles. Treat group totals as encompassing potentially overlapping + work and account for terminal coverage separately from known or complete + usage. Report editorial findings from the requests themselves. + [Diagnostics and reporting](../internal/diagnostics-reporting.md) + +## Stage 4 — Verify the Integrated Result + +1. Review the existing runner, adapter, and subprocess evidence together for + independent branches, full revalidation, final-turn acceptance, withdrawal, + exhaustion, recoverable unavailability, and cancellation before delivery. + Confirm that accepted siblings survive recoverable failures and that + undelivered content cannot become an approved correction. Extend the nearest + existing fixture only if a changed behavior needs coverage. Use the real + adapter and coordinator with controlled model responses for execution tests. +2. For any prompt edits, inspect representative rendered requests and verify + supplied-data rendering and affected examples against the applicable + response schemas. Use durable assertions for changed behavior and inspect + the prose as part of review. Follow the [testing policy](../policy/testing.md). + Update the [prompt registry](../internal/prompts.md), + [module guide](../internal/modules.md), or [validator guide](../internal/validators.md) + where their owned behavior changes. +3. Use focused tests for iteration in the touched packages. At sprint + completion, run `go test ./...` with the repository's Go toolchain and + offline fixtures. Verify affected links and run `git diff --check`. + Summarize the cases inspected, any evidence-backed edits, and the results + of verification. Live provider comparisons remain deferred. diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index 0eafe9b..d62afe3 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -3,7 +3,8 @@ **Status:** The earlier six prompt improvements are implemented. This follow-up uses Audita's PromptKit v0.23.0 integration to review and develop proposal, validation, and repair prompts through the same execution path used in production. -Live-model comparisons and broader context expansion remain deferred. +Live-model comparisons and broader context expansion remain deferred. See the +[implementation plan](implementation.md) for the sprint sequence. ## Baseline and Boundaries @@ -32,83 +33,35 @@ run accounting. Audita owns candidate identity, findings, previews, acceptance, withdrawal, and transcript mutation. Follow-up prompt work must use these APIs; it must not recreate message replay, turn loops, or token aggregation locally. -## 1. Review Complete Native Conversations +## Target State -Complete the PromptKit adoption review follow-ups before revising prompts. +Proposal and repair instructions should remain coherent across a complete +conversation. The original context and supplied glossary and protected +spellings remain available, each repair branch keeps its own history, and +continuations select the repair definition, profile, and output schema. +Both shipped and operator-replaced definitions use native prompt selection. +Repair feedback identifies the current edit and the validator's findings so +that the model can revise the proposal or withdraw it. -- Extend the debug case of `TestSubprocessCorrectionRepairConversation` to - compare the top-level `llm-execution.json` provider-attempt total with the - provider fixture's request count. The total should include the initial proposal - once and count each repair and validator request. - [Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go) -- Decode each repair response once in the runner. Reuse the decoded value for - proposal-reference and action checks and for debug output. Preserve the existing - invalid-output disposition and raw response evidence. - [Repair runner](../../internal/framework/runner/repair.go) -- Compute each per-call execution summary once and reuse it for terminal coverage - and usage in `llm-execution.json`. - [Execution report](../../internal/framework/processreport/llm.go) +Audita assesses every delivered revision, including the last allowed response. +Coordination stopping and correction acceptance are distinct outcomes. +Undelivered content is diagnostic evidence. Independent approvals survive +recoverable failures. Transient retries, structural repair, and correction +revision retain their separate budgets. -Use captured proposal exchanges and independent coordinator branches to inspect -representative proposal-to-repair conversations, including a second revision and -withdrawal. Review actual `OnPrepared` snapshots for instruction authority, -response-format transitions, glossary/protected-term context, and fresh previews. -A follow-up selects its repair definition, role profile, and schema directly. +Prompt and feedback changes should address concrete issues in captured +requests while preserving validator acceptance and the allowance for minor +editorial cleanup. Broader context and combined semantic validation remain +deferred. Any proposal to expand validator context should identify a concrete +case and the evidence missing from its request. This follow-up retains current +context fitting and apply-time conflict handling. -Exercise operator replacements through native prompt discovery and precedence. -Compare the request that actually executes rather than separately rendering an -approximation. Reusing captured history must preserve the proposal context while -keeping each correction's repair history private. - -Record concrete editorial concerns and revise affected prompts when warranted. -Do not change validator acceptance or narrow allowable cleanup merely to make -proposal and repair assignments identical. - -## 2. Assess Repair Feedback and Outcomes - -Use the real coordinator with controlled model responses for deterministic -integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion, -unavailable responses, and independent accepted siblings. Revalidate each revision -from the beginning. A normal coordination stop is not evidence of acceptance; -Audita's domain outcome determines that. Canceled undelivered content is evidence -only and must never enter the approved set. - -Use current target previews and validator-owned findings to assess whether a -repair request explains the defect sufficiently. Keep presentation changes in -prompt assets and domain feedback construction; keep history and budget mechanics -upstream. Retain separate transient retry, structural repair, and correction -revision budgets. - -## 3. Diagnose Evidence Gaps Using Actual Requests - -For a concrete rejected correction, compare the captured proposal evidence with -what the validator actually received. The target speaker is already included. -Broader validation context remains evidence-dependent: propose a focused change -only when a captured case demonstrates relevant evidence missing from validation. - -Use native execution-group totals to distinguish additional model work from -shared initial history. Select validator membership explicitly, count the initial -proposal once, and do not add overlapping group totals. Preserve unknown usage -and terminal coverage separately. These measurements describe execution and cost; -they do not establish editorial quality. - -Existing local examples have not established a need for broader validation windows -or combined semantic validation. Keep context fitting and apply-time conflict -handling unchanged until concrete cases justify further work. - -## 4. Verification and Documentation - -Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter -and coordinator with deterministic model-boundary fixtures. Protect supplied-data -rendering, schemas, profile selection, current previews, branching, revalidation, -withdrawal, and operator replacement. Inspect representative rendered prose as -part of review; avoid phrase checks, prose snapshots, and tests that duplicate -upstream history or accounting algorithms. - -Update canonical prompt and validator documentation when behavior changes. -Keep examples valid against their response schemas and verify affected links. -Run focused tests and the full offline suite for runtime changes. Pure wording -changes need proportionate review, not new change-detector tests. +Native execution diagnostics count the shared proposal once and include each +new repair and validator run. Terminal coverage and usage availability +remain distinct. Execution measurements describe work and cost. Review +prompt clarity using the supplied evidence and rendered instructions. +The adoption follow-up also removes duplicate repair decoding and per-call +summarization while preserving the existing diagnostic evidence. ## Deferred Evaluation