9.7 KiB
Prompt Quality Implementation Plan
Follow the stages in order within one sprint. Intermediate stages may leave dependent edits in progress. Verify the integrated result at the end of the sprint. Read the feature roadmap for the selected behavior and developer guide for the applicable policies.
Stage 1 — Finish the PromptKit Adoption Follow-ups
Complete. Focused decoder, runner, reporting, and subprocess checks passed.
- In the debug case of
TestSubprocessCorrectionRepairConversation, read the run'sllm-execution.jsonand compare its top-levelsummary.provider_attemptswith the number of requests captured by the provider fixture. Use the existing fixture with an initial proposal, repair branches, and nested validation. This checks that the shared proposal is counted once and that repair and validation calls join the invocation total. Subprocess fixture - In
repairCandidates, remove the extra JSON unmarshal and use the typed response from one decode for the domain decision and the interaction artifact. ReuseDecodeRepairResponseand its existing proposal-reference and action checks. Where a decoded response fails a correspondence check, retain it as diagnostic evidence alongside the raw response and failure. Assess only delivered, structurally valid content. Keep the existing invalid-output disposition, failure attribution, and revalidation behavior. Repair runner Repair decoder - In
BuildLLMExecution, callllm.Summarizeonce per call and use the result for both terminal coverage and the serialized summary. Keep the invocation summary sourced from the native group. Preserve field names, ordering, and the distinction between unknown usage and zero usage. Execution report
Stage 2 — Review Proposal and Repair Conversations
Complete. The shipped and operator-replaced grammar requests retained the proposal context, including a synthetic glossary entry and protected spelling. The wire schema changed from a correction set to one repair result. The second turn retained its branch's prior revision, and the other branch started from the original proposal exchange. The adapter fixture confirmed separate proposal and repair profiles. Runner fixtures confirmed updated previews after retargeting, omission of an inapplicable after-preview, and withdrawal.
The shipped grammar repair wording could read as a blanket restriction on changes to other words. The revised text describes the grammar focus and leaves the shared editorial allowance explicit in the rendered request.
- Use the existing subprocess fixture to exercise the production adapter and coordinator with scripted responses. Inspect its captured request messages and response schemas for the initial proposal and the two-turn repair. Inspect the existing withdrawal case in the runner fixture as well. Use prepared request artifacts for the rendered instructions and wire requests for the effective schema. Inspect the shipped definition and an operator replacement from the existing fixture. Use temporary test logging if captured requests or artifacts are otherwise discarded after a passing run. Subprocess fixture Runner fixtures
- Check whether the original system instructions, the initial assignment,
and the subsequent repair assignment give a coherent task. Check that the
continuation retains the relevant glossary and protected spellings when
supplied, selects the repair definition, role profile, and one-result
schema, and includes each branch's own revision history. Compare the latest
target and before/after preview with the candidate under review, including
a retargeted edit and the existing omission of an inapplicable after-preview.
Use
TestRoleClientContinuesIndependentRepairConversationsfor distinct proposal and repair profiles. Adapter conversation fixture - Check whether the findings give the model a specific defect to address, whether the current preview identifies the edit being assessed, and whether the instruction makes revision and withdrawal clear. Fix concrete ambiguities in the affected bundled prompt or feedback presentation. Keep module focus, editorial acceptance, and required evidence as specified by the feature roadmap. If the requests are coherent, record the reviewed cases and the outcome in the sprint summary. Prompt registry
Stage 3 — Review Evidence at the Validator Boundary
Complete. Both model validators received speaker A, the literal
second → Second edit, full second text → Second text previews,
the preceding and following segments, and the matching Second text glossary
entry for the original and revised candidate. The capitalization objection
reached the next repair request with each validator's identity. The fixture
scripts that same objection for both validators, so it confirms feedback routing.
The rendered meaning-reversal instructions still require opposing full
assertions. This case revealed no missing evidence or unclear validator
instruction. The captured items retained their supplemental context, and
the fitting tests exercised omission of optional context under a smaller budget.
The native invocation reported 10 provider attempts with complete terminal coverage and unknown token usage. Per-call roles accounted for one proposal, two initial validator requests, three repair requests, and four validator requests across two revisions. The seven repair-related requests are already included in the invocation total. The fixture's last revision repeats the same edit and receives a scripted approval, so the counts measure execution.
- Follow a rejected candidate and, where available, its revision through the captured proposal, validator request, decision, and next repair request. Compare the target speaker, literal edit, before/after text, supplied neighbors, matching glossary entries, and validator's stated reason. Account for optional context omitted by the existing payload fitting. Validator payload and fitting
- If evidence is already present but the instruction obscures how to use it, revise the relevant prompt guidance or example. Preserve each validator's remit, including the full-assertion reversal criterion. If the case needs unavailable evidence, describe the missing evidence and the smallest candidate payload change in the sprint summary. Broader context expansion and combined semantic validation remain deferred. If no concrete gap appears, the captured comparison completes this stage.
- Use the native invocation summary to describe any additional requests attributable to repair and revalidation. Use per-call evidence to identify their roles. Treat group totals as encompassing potentially overlapping work and account for terminal coverage separately from known or complete usage. Report editorial findings from the requests themselves. Diagnostics and reporting
Stage 4 — Verify the Integrated Result
Complete. Subprocess, adapter, and runner fixtures together verified independent repair histories, revalidation from the beginning, acceptance on the last allowed turn, withdrawal, exhaustion, recoverable provider failure, and preservation of independently approved edits. The adapter's cancellation fixture retained the prepared request and raw response with no assessable content. The runner records the cancellation before attempting to decode, and the subprocess cancellation fixture verified that no transcript was published.
Reviewed the revised shipped grammar request and the operator replacement as
sent by PromptKit. Supplied-data rendering and schema checks passed, including
the shared revision and withdrawal response examples. The focused tests and
full offline go test ./... suite passed with Go 1.27.3. Roadmap links and
anchors resolved, and git diff --check passed.
- Review the existing runner, adapter, and subprocess evidence together for independent branches, full revalidation, final-turn acceptance, withdrawal, exhaustion, recoverable unavailability, and cancellation before delivery. Confirm that accepted siblings survive recoverable failures and that undelivered content cannot become an approved correction. Extend the nearest existing fixture only if a changed behavior needs coverage. Use the real adapter and coordinator with controlled model responses for execution tests.
- For any prompt edits, inspect representative rendered requests and verify supplied-data rendering and affected examples against the applicable response schemas. Use durable assertions for changed behavior and inspect the prose as part of review. Follow the testing policy. Update the prompt registry, module guide, or validator guide where their owned behavior changes.
- Use focused tests for iteration in the touched packages. At sprint
completion, run
go test ./...with the repository's Go toolchain and offline fixtures. Verify affected links and rungit diff --check. Summarize the cases inspected, any evidence-backed edits, and the results of verification. Live provider comparisons remain deferred.