137 lines
8.8 KiB
Markdown
137 lines
8.8 KiB
Markdown
# Prompt Quality Implementation Plan
|
|
|
|
Follow the stages in order within one sprint. Intermediate stages may leave
|
|
dependent edits in progress. Verify the integrated result at the end of the
|
|
sprint. Read the [feature roadmap](prompt-quality.md) for the selected behavior
|
|
and [developer guide](../development.md) for the applicable policies.
|
|
|
|
## Stage 1 — Finish the PromptKit Adoption Follow-ups
|
|
|
|
**Complete.** Focused decoder, runner, reporting, and subprocess checks passed.
|
|
|
|
1. In the debug case of `TestSubprocessCorrectionRepairConversation`, read the
|
|
run's `llm-execution.json` and compare its top-level
|
|
`summary.provider_attempts` with the number of requests captured by the
|
|
provider fixture. Use the existing fixture with an initial proposal, repair
|
|
branches, and nested validation. This checks that the shared proposal is
|
|
counted once and that repair and validation calls join the invocation total.
|
|
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
|
2. In `repairCandidates`, remove the extra JSON unmarshal and use the typed
|
|
response from one decode for the domain decision and the interaction
|
|
artifact. Reuse `DecodeRepairResponse` and its existing proposal-reference
|
|
and action checks. Where a decoded response fails a correspondence check,
|
|
retain it as diagnostic evidence alongside the raw response and failure.
|
|
Assess only delivered, structurally valid content. Keep the existing
|
|
invalid-output disposition, failure attribution, and revalidation behavior.
|
|
[Repair runner](../../internal/framework/runner/repair.go)
|
|
[Repair decoder](../../internal/framework/proposal_generation/repair.go)
|
|
3. In `BuildLLMExecution`, call `llm.Summarize` once per call and use the result
|
|
for both terminal coverage and the serialized summary. Keep the invocation
|
|
summary sourced from the native group. Preserve field names, ordering, and
|
|
the distinction between unknown usage and zero usage.
|
|
[Execution report](../../internal/framework/processreport/llm.go)
|
|
|
|
## Stage 2 — Review Proposal and Repair Conversations
|
|
|
|
**Complete.** The shipped and operator-replaced grammar requests retained the
|
|
proposal context, including a synthetic glossary entry and protected spelling.
|
|
The wire schema changed from a correction set to one repair result. The second
|
|
turn retained its branch's prior revision, and the other branch started from
|
|
the original proposal exchange. The adapter fixture confirmed separate proposal
|
|
and repair profiles. Runner fixtures confirmed updated previews after retargeting,
|
|
omission of an inapplicable after-preview, and withdrawal.
|
|
|
|
The shipped grammar repair wording could read as a blanket restriction on
|
|
changes to other words. The revised text describes the grammar focus and leaves
|
|
the shared editorial allowance explicit in the rendered request.
|
|
|
|
1. Use the existing subprocess fixture to exercise the production adapter and
|
|
coordinator with scripted responses. Inspect its captured request messages
|
|
and response schemas for the initial proposal and the two-turn repair.
|
|
Inspect the existing withdrawal case in the runner fixture as well. Use
|
|
prepared request artifacts for the rendered instructions and wire requests
|
|
for the effective schema. Inspect the shipped definition and an operator
|
|
replacement from the existing fixture. Use temporary test logging if
|
|
captured requests or artifacts are otherwise discarded after a passing run.
|
|
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
|
[Runner fixtures](../../internal/framework/runner/repair_test.go)
|
|
2. Check whether the original system instructions, the initial assignment,
|
|
and the subsequent repair assignment give a coherent task. Check that the
|
|
continuation retains the relevant glossary and protected spellings when
|
|
supplied, selects the repair definition, role profile, and one-result
|
|
schema, and includes each branch's own revision history. Compare the latest
|
|
target and before/after preview with the candidate under review, including
|
|
a retargeted edit and the existing omission of an inapplicable after-preview.
|
|
Use `TestRoleClientContinuesIndependentRepairConversations` for distinct
|
|
proposal and repair profiles.
|
|
[Adapter conversation fixture](../../internal/framework/llm/conversation_test.go)
|
|
3. Check whether the findings give the model a specific defect to address,
|
|
whether the current preview identifies the edit being assessed, and whether
|
|
the instruction makes revision and withdrawal clear. Fix concrete
|
|
ambiguities in the affected bundled prompt or feedback presentation. Keep
|
|
module focus, editorial acceptance, and required evidence as specified by
|
|
the feature roadmap. If the requests are coherent, record the reviewed
|
|
cases and the outcome in the sprint summary. [Prompt registry](../internal/prompts.md)
|
|
|
|
## Stage 3 — Review Evidence at the Validator Boundary
|
|
|
|
**Complete.** Both model validators received speaker `A`, the literal
|
|
`second` → `Second` edit, full `second text` → `Second text` previews,
|
|
the preceding and following segments, and the matching `Second text` glossary
|
|
entry for the original and revised candidate. The capitalization objection
|
|
reached the next repair request with each validator's identity. The fixture
|
|
scripts that same objection for both validators, so it confirms feedback routing.
|
|
The rendered meaning-reversal instructions still require opposing full
|
|
assertions. This case revealed no missing evidence or unclear validator
|
|
instruction. The captured items retained their supplemental context, and
|
|
the fitting tests exercised omission of optional context under a smaller budget.
|
|
|
|
The native invocation reported 10 provider attempts with complete terminal
|
|
coverage and unknown token usage. Per-call roles accounted for one proposal,
|
|
two initial validator requests, three repair requests, and four validator
|
|
requests across two revisions. The seven repair-related requests are already
|
|
included in the invocation total. The fixture's last revision repeats the
|
|
same edit and receives a scripted approval, so the counts measure execution.
|
|
|
|
1. Follow a rejected candidate and, where available, its revision through the
|
|
captured proposal, validator request, decision, and next repair request.
|
|
Compare the target speaker, literal edit, before/after text, supplied
|
|
neighbors, matching glossary entries, and validator's stated reason.
|
|
Account for optional context omitted by the existing payload fitting.
|
|
[Validator payload and fitting](../internal/validators.md#supplemental-validation-context)
|
|
2. If evidence is already present but the instruction obscures how to use it,
|
|
revise the relevant prompt guidance or example. Preserve each validator's
|
|
remit, including the full-assertion reversal criterion. If the case needs
|
|
unavailable evidence, describe the missing evidence and the smallest
|
|
candidate payload change in the sprint summary. Broader context expansion
|
|
and combined semantic validation remain deferred. If no concrete gap
|
|
appears, the captured comparison completes this stage.
|
|
3. Use the native invocation summary to describe any additional requests
|
|
attributable to repair and revalidation. Use per-call evidence to identify
|
|
their roles. Treat group totals as encompassing potentially overlapping
|
|
work and account for terminal coverage separately from known or complete
|
|
usage. Report editorial findings from the requests themselves.
|
|
[Diagnostics and reporting](../internal/diagnostics-reporting.md)
|
|
|
|
## Stage 4 — Verify the Integrated Result
|
|
|
|
1. Review the existing runner, adapter, and subprocess evidence together for
|
|
independent branches, full revalidation, final-turn acceptance, withdrawal,
|
|
exhaustion, recoverable unavailability, and cancellation before delivery.
|
|
Confirm that accepted siblings survive recoverable failures and that
|
|
undelivered content cannot become an approved correction. Extend the nearest
|
|
existing fixture only if a changed behavior needs coverage. Use the real
|
|
adapter and coordinator with controlled model responses for execution tests.
|
|
2. For any prompt edits, inspect representative rendered requests and verify
|
|
supplied-data rendering and affected examples against the applicable
|
|
response schemas. Use durable assertions for changed behavior and inspect
|
|
the prose as part of review. Follow the [testing policy](../policy/testing.md).
|
|
Update the [prompt registry](../internal/prompts.md),
|
|
[module guide](../internal/modules.md), or [validator guide](../internal/validators.md)
|
|
where their owned behavior changes.
|
|
3. Use focused tests for iteration in the touched packages. At sprint
|
|
completion, run `go test ./...` with the repository's Go toolchain and
|
|
offline fixtures. Verify affected links and run `git diff --check`.
|
|
Summarize the cases inspected, any evidence-backed edits, and the results
|
|
of verification. Live provider comparisons remain deferred.
|