Plan the prompt quality sprint
This commit is contained in:
104
docs/roadmap/implementation.md
Normal file
104
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Prompt Quality Implementation Plan
|
||||
|
||||
Follow the stages in order within one sprint. Intermediate stages may leave
|
||||
dependent edits in progress. Verify the integrated result at the end of the
|
||||
sprint. Read the [feature roadmap](prompt-quality.md) for the selected behavior
|
||||
and [developer guide](../development.md) for the applicable policies.
|
||||
|
||||
## Stage 1 — Finish the PromptKit Adoption Follow-ups
|
||||
|
||||
1. In the debug case of `TestSubprocessCorrectionRepairConversation`, read the
|
||||
run's `llm-execution.json` and compare its top-level
|
||||
`summary.provider_attempts` with the number of requests captured by the
|
||||
provider fixture. Use the existing fixture with an initial proposal, repair
|
||||
branches, and nested validation. This checks that the shared proposal is
|
||||
counted once and that repair and validation calls join the invocation total.
|
||||
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
||||
2. In `repairCandidates`, remove the extra JSON unmarshal and use the typed
|
||||
response from one decode for the domain decision and the interaction
|
||||
artifact. Reuse `DecodeRepairResponse` and its existing proposal-reference
|
||||
and action checks. Where a decoded response fails a correspondence check,
|
||||
retain it as diagnostic evidence alongside the raw response and failure.
|
||||
Assess only delivered, structurally valid content. Keep the existing
|
||||
invalid-output disposition, failure attribution, and revalidation behavior.
|
||||
[Repair runner](../../internal/framework/runner/repair.go)
|
||||
[Repair decoder](../../internal/framework/proposal_generation/repair.go)
|
||||
3. In `BuildLLMExecution`, call `llm.Summarize` once per call and use the result
|
||||
for both terminal coverage and the serialized summary. Keep the invocation
|
||||
summary sourced from the native group. Preserve field names, ordering, and
|
||||
the distinction between unknown usage and zero usage.
|
||||
[Execution report](../../internal/framework/processreport/llm.go)
|
||||
|
||||
## Stage 2 — Review Proposal and Repair Conversations
|
||||
|
||||
1. Use the existing subprocess fixture to exercise the production adapter and
|
||||
coordinator with scripted responses. Inspect its captured request messages
|
||||
and response schemas for the initial proposal and the two-turn repair.
|
||||
Inspect the existing withdrawal case in the runner fixture as well. Use
|
||||
prepared request artifacts for the rendered instructions and wire requests
|
||||
for the effective schema. Inspect the shipped definition and an operator
|
||||
replacement from the existing fixture. Use temporary test logging if
|
||||
captured requests or artifacts are otherwise discarded after a passing run.
|
||||
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
||||
[Runner fixtures](../../internal/framework/runner/repair_test.go)
|
||||
2. Check whether the original system instructions, the initial assignment,
|
||||
and the subsequent repair assignment give a coherent task. Check that the
|
||||
continuation retains the relevant glossary and protected spellings when
|
||||
supplied, selects the repair definition, role profile, and one-result
|
||||
schema, and includes each branch's own revision history. Compare the latest
|
||||
target and before/after preview with the candidate under review, including
|
||||
a retargeted edit and the existing omission of an inapplicable after-preview.
|
||||
Use `TestRoleClientContinuesIndependentRepairConversations` for distinct
|
||||
proposal and repair profiles.
|
||||
[Adapter conversation fixture](../../internal/framework/llm/conversation_test.go)
|
||||
3. Check whether the findings give the model a specific defect to address,
|
||||
whether the current preview identifies the edit being assessed, and whether
|
||||
the instruction makes revision and withdrawal clear. Fix concrete
|
||||
ambiguities in the affected bundled prompt or feedback presentation. Keep
|
||||
module focus, editorial acceptance, and required evidence as specified by
|
||||
the feature roadmap. If the requests are coherent, record the reviewed
|
||||
cases and the outcome in the sprint summary. [Prompt registry](../internal/prompts.md)
|
||||
|
||||
## Stage 3 — Review Evidence at the Validator Boundary
|
||||
|
||||
1. Follow a rejected candidate and, where available, its revision through the
|
||||
captured proposal, validator request, decision, and next repair request.
|
||||
Compare the target speaker, literal edit, before/after text, supplied
|
||||
neighbors, matching glossary entries, and validator's stated reason.
|
||||
Account for optional context omitted by the existing payload fitting.
|
||||
[Validator payload and fitting](../internal/validators.md#supplemental-validation-context)
|
||||
2. If evidence is already present but the instruction obscures how to use it,
|
||||
revise the relevant prompt guidance or example. Preserve each validator's
|
||||
remit, including the full-assertion reversal criterion. If the case needs
|
||||
unavailable evidence, describe the missing evidence and the smallest
|
||||
candidate payload change in the sprint summary. Broader context expansion
|
||||
and combined semantic validation remain deferred. If no concrete gap
|
||||
appears, the captured comparison completes this stage.
|
||||
3. Use the native invocation summary to describe any additional requests
|
||||
attributable to repair and revalidation. Use per-call evidence to identify
|
||||
their roles. Treat group totals as encompassing potentially overlapping
|
||||
work and account for terminal coverage separately from known or complete
|
||||
usage. Report editorial findings from the requests themselves.
|
||||
[Diagnostics and reporting](../internal/diagnostics-reporting.md)
|
||||
|
||||
## Stage 4 — Verify the Integrated Result
|
||||
|
||||
1. Review the existing runner, adapter, and subprocess evidence together for
|
||||
independent branches, full revalidation, final-turn acceptance, withdrawal,
|
||||
exhaustion, recoverable unavailability, and cancellation before delivery.
|
||||
Confirm that accepted siblings survive recoverable failures and that
|
||||
undelivered content cannot become an approved correction. Extend the nearest
|
||||
existing fixture only if a changed behavior needs coverage. Use the real
|
||||
adapter and coordinator with controlled model responses for execution tests.
|
||||
2. For any prompt edits, inspect representative rendered requests and verify
|
||||
supplied-data rendering and affected examples against the applicable
|
||||
response schemas. Use durable assertions for changed behavior and inspect
|
||||
the prose as part of review. Follow the [testing policy](../policy/testing.md).
|
||||
Update the [prompt registry](../internal/prompts.md),
|
||||
[module guide](../internal/modules.md), or [validator guide](../internal/validators.md)
|
||||
where their owned behavior changes.
|
||||
3. Use focused tests for iteration in the touched packages. At sprint
|
||||
completion, run `go test ./...` with the repository's Go toolchain and
|
||||
offline fixtures. Verify affected links and run `git diff --check`.
|
||||
Summarize the cases inspected, any evidence-backed edits, and the results
|
||||
of verification. Live provider comparisons remain deferred.
|
||||
@@ -3,7 +3,8 @@
|
||||
**Status:** The earlier six prompt improvements are implemented. This follow-up
|
||||
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
|
||||
validation, and repair prompts through the same execution path used in production.
|
||||
Live-model comparisons and broader context expansion remain deferred.
|
||||
Live-model comparisons and broader context expansion remain deferred. See the
|
||||
[implementation plan](implementation.md) for the sprint sequence.
|
||||
|
||||
## Baseline and Boundaries
|
||||
|
||||
@@ -32,83 +33,35 @@ run accounting. Audita owns candidate identity, findings, previews, acceptance,
|
||||
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
|
||||
it must not recreate message replay, turn loops, or token aggregation locally.
|
||||
|
||||
## 1. Review Complete Native Conversations
|
||||
## Target State
|
||||
|
||||
Complete the PromptKit adoption review follow-ups before revising prompts.
|
||||
Proposal and repair instructions should remain coherent across a complete
|
||||
conversation. The original context and supplied glossary and protected
|
||||
spellings remain available, each repair branch keeps its own history, and
|
||||
continuations select the repair definition, profile, and output schema.
|
||||
Both shipped and operator-replaced definitions use native prompt selection.
|
||||
Repair feedback identifies the current edit and the validator's findings so
|
||||
that the model can revise the proposal or withdraw it.
|
||||
|
||||
- Extend the debug case of `TestSubprocessCorrectionRepairConversation` to
|
||||
compare the top-level `llm-execution.json` provider-attempt total with the
|
||||
provider fixture's request count. The total should include the initial proposal
|
||||
once and count each repair and validator request.
|
||||
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
||||
- Decode each repair response once in the runner. Reuse the decoded value for
|
||||
proposal-reference and action checks and for debug output. Preserve the existing
|
||||
invalid-output disposition and raw response evidence.
|
||||
[Repair runner](../../internal/framework/runner/repair.go)
|
||||
- Compute each per-call execution summary once and reuse it for terminal coverage
|
||||
and usage in `llm-execution.json`.
|
||||
[Execution report](../../internal/framework/processreport/llm.go)
|
||||
Audita assesses every delivered revision, including the last allowed response.
|
||||
Coordination stopping and correction acceptance are distinct outcomes.
|
||||
Undelivered content is diagnostic evidence. Independent approvals survive
|
||||
recoverable failures. Transient retries, structural repair, and correction
|
||||
revision retain their separate budgets.
|
||||
|
||||
Use captured proposal exchanges and independent coordinator branches to inspect
|
||||
representative proposal-to-repair conversations, including a second revision and
|
||||
withdrawal. Review actual `OnPrepared` snapshots for instruction authority,
|
||||
response-format transitions, glossary/protected-term context, and fresh previews.
|
||||
A follow-up selects its repair definition, role profile, and schema directly.
|
||||
Prompt and feedback changes should address concrete issues in captured
|
||||
requests while preserving validator acceptance and the allowance for minor
|
||||
editorial cleanup. Broader context and combined semantic validation remain
|
||||
deferred. Any proposal to expand validator context should identify a concrete
|
||||
case and the evidence missing from its request. This follow-up retains current
|
||||
context fitting and apply-time conflict handling.
|
||||
|
||||
Exercise operator replacements through native prompt discovery and precedence.
|
||||
Compare the request that actually executes rather than separately rendering an
|
||||
approximation. Reusing captured history must preserve the proposal context while
|
||||
keeping each correction's repair history private.
|
||||
|
||||
Record concrete editorial concerns and revise affected prompts when warranted.
|
||||
Do not change validator acceptance or narrow allowable cleanup merely to make
|
||||
proposal and repair assignments identical.
|
||||
|
||||
## 2. Assess Repair Feedback and Outcomes
|
||||
|
||||
Use the real coordinator with controlled model responses for deterministic
|
||||
integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion,
|
||||
unavailable responses, and independent accepted siblings. Revalidate each revision
|
||||
from the beginning. A normal coordination stop is not evidence of acceptance;
|
||||
Audita's domain outcome determines that. Canceled undelivered content is evidence
|
||||
only and must never enter the approved set.
|
||||
|
||||
Use current target previews and validator-owned findings to assess whether a
|
||||
repair request explains the defect sufficiently. Keep presentation changes in
|
||||
prompt assets and domain feedback construction; keep history and budget mechanics
|
||||
upstream. Retain separate transient retry, structural repair, and correction
|
||||
revision budgets.
|
||||
|
||||
## 3. Diagnose Evidence Gaps Using Actual Requests
|
||||
|
||||
For a concrete rejected correction, compare the captured proposal evidence with
|
||||
what the validator actually received. The target speaker is already included.
|
||||
Broader validation context remains evidence-dependent: propose a focused change
|
||||
only when a captured case demonstrates relevant evidence missing from validation.
|
||||
|
||||
Use native execution-group totals to distinguish additional model work from
|
||||
shared initial history. Select validator membership explicitly, count the initial
|
||||
proposal once, and do not add overlapping group totals. Preserve unknown usage
|
||||
and terminal coverage separately. These measurements describe execution and cost;
|
||||
they do not establish editorial quality.
|
||||
|
||||
Existing local examples have not established a need for broader validation windows
|
||||
or combined semantic validation. Keep context fitting and apply-time conflict
|
||||
handling unchanged until concrete cases justify further work.
|
||||
|
||||
## 4. Verification and Documentation
|
||||
|
||||
Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter
|
||||
and coordinator with deterministic model-boundary fixtures. Protect supplied-data
|
||||
rendering, schemas, profile selection, current previews, branching, revalidation,
|
||||
withdrawal, and operator replacement. Inspect representative rendered prose as
|
||||
part of review; avoid phrase checks, prose snapshots, and tests that duplicate
|
||||
upstream history or accounting algorithms.
|
||||
|
||||
Update canonical prompt and validator documentation when behavior changes.
|
||||
Keep examples valid against their response schemas and verify affected links.
|
||||
Run focused tests and the full offline suite for runtime changes. Pure wording
|
||||
changes need proportionate review, not new change-detector tests.
|
||||
Native execution diagnostics count the shared proposal once and include each
|
||||
new repair and validator run. Terminal coverage and usage availability
|
||||
remain distinct. Execution measurements describe work and cost. Review
|
||||
prompt clarity using the supplied evidence and rendered instructions.
|
||||
The adoption follow-up also removes duplicate repair decoding and per-call
|
||||
summarization while preserving the existing diagnostic evidence.
|
||||
|
||||
## Deferred Evaluation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user