5.7 KiB
Prompt Quality Roadmap
Status: The earlier six prompt improvements are implemented. This follow-up uses Audita's PromptKit v0.23.0 integration to review and develop proposal, validation, and repair prompts through the same execution path used in production. Live-model comparisons and broader context expansion remain deferred.
Baseline and Boundaries
The prompt registry, validator guide, and LLM runtime own implemented behavior. The existing prompts separate durable system principles from initial proposal assignments, guide independent and overlapping edits, supply current before/after repair previews, include the target speaker in validation, and illustrate editorial judgments, revision, and withdrawal.
Preserve these selected decisions:
- Module focus guides generation; otherwise acceptable minor editorial cleanup remains permissible across modules under ADR 0003.
- Keep the carefully tuned meaning-reversal criterion and distinct validator responsibilities. Approvals require no explanation; confidence scores remain absent.
- Preserve literal spans, deletion, replacement policies, glossary context, protected spellings, full revalidation, and deterministic application.
- Retain English defaults and native operator prompt replacement. Module sequence reevaluation follows the module guidance.
PromptKit now owns immutable captured exchanges, continuation assembly, bounded coordination, per-operation deadlines, prepared-request notification, and selected run accounting. Audita owns candidate identity, findings, previews, acceptance, withdrawal, and transcript mutation. Follow-up prompt work must use these APIs; it must not recreate message replay, turn loops, or token aggregation locally.
1. Review Complete Native Conversations
Use captured proposal exchanges and independent coordinator branches to inspect
representative proposal-to-repair conversations, including a second revision and
withdrawal. Review actual OnPrepared snapshots for instruction authority,
response-format transitions, glossary/protected-term context, and fresh previews.
A follow-up selects its repair definition, role profile, and schema directly.
Exercise operator replacements through native prompt discovery and precedence. Compare the request that actually executes rather than separately rendering an approximation. Reusing captured history must preserve the proposal context while keeping each correction's repair history private.
Record concrete editorial concerns and revise affected prompts when warranted. Do not change validator acceptance or narrow allowable cleanup merely to make proposal and repair assignments identical.
2. Assess Repair Feedback and Outcomes
Use the real coordinator with controlled model responses for deterministic integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion, unavailable responses, and independent accepted siblings. Revalidate each revision from the beginning. A normal coordination stop is not evidence of acceptance; Audita's domain outcome determines that. Canceled undelivered content is evidence only and must never enter the approved set.
Use current target previews and validator-owned findings to assess whether a repair request explains the defect sufficiently. Keep presentation changes in prompt assets and domain feedback construction; keep history and budget mechanics upstream. Retain separate transient retry, structural repair, and correction revision budgets.
3. Diagnose Evidence Gaps Using Actual Requests
For a concrete rejected correction, compare the captured proposal evidence with what the validator actually received. The target speaker is already included. Broader validation context remains evidence-dependent: propose a focused change only when a captured case demonstrates relevant evidence missing from validation.
Use native execution-group totals to distinguish additional model work from shared initial history. Select validator membership explicitly, count the initial proposal once, and do not add overlapping group totals. Preserve unknown usage and terminal coverage separately. These measurements describe execution and cost; they do not establish editorial quality.
Existing local examples have not established a need for broader validation windows or combined semantic validation. Keep context fitting and apply-time conflict handling unchanged until concrete cases justify further work.
4. Verification and Documentation
Follow the testing policy. Use the real PromptKit adapter and coordinator with deterministic model-boundary fixtures. Protect supplied-data rendering, schemas, profile selection, current previews, branching, revalidation, withdrawal, and operator replacement. Inspect representative rendered prose as part of review; avoid phrase checks, prose snapshots, and tests that duplicate upstream history or accounting algorithms.
Update canonical prompt and validator documentation when behavior changes. Keep examples valid against their response schemas and verify affected links. Run focused tests and the full offline suite for runtime changes. Pure wording changes need proportionate review, not new change-detector tests.
Deferred Evaluation
Formal corpus preparation and live-model comparisons remain optional follow-up work. Keep live calls opt-in and outside the offline suite. When undertaken, compare needed and unnecessary edits, missed corrections, validator disagreement, and repair outcomes using independent editorial judgment. Native groups can supply execution measurements without introducing a second accounting system.
No blocking product decisions remain for this follow-up scope.