Files
audita/docs/roadmap/prompt-quality.md

75 lines
4.1 KiB
Markdown

# Prompt Quality Roadmap
**Status:** The earlier six prompt improvements are implemented. This follow-up
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
validation, and repair prompts through the same execution path used in production.
Live-model comparisons and broader context expansion remain deferred. See the
[implementation plan](implementation.md) for the sprint sequence.
## Baseline and Boundaries
The [prompt registry](../internal/prompts.md),
[validator guide](../internal/validators.md), and
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
prompts separate durable system principles from initial proposal assignments,
guide independent and overlapping edits, supply current before/after repair
previews, include the target speaker in validation, and illustrate editorial
judgments, revision, and withdrawal.
Preserve these selected decisions:
- Module focus guides generation; otherwise acceptable minor editorial cleanup
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
- Keep the carefully tuned meaning-reversal criterion and distinct validator
responsibilities. Approvals require no explanation; confidence scores remain absent.
- Preserve literal spans, deletion, replacement policies, glossary context,
protected spellings, full revalidation, and deterministic application.
- Retain English defaults and native operator prompt replacement. Module sequence
reevaluation follows the [module guidance](../internal/modules.md).
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
coordination, per-operation deadlines, prepared-request notification, and selected
run accounting. Audita owns candidate identity, findings, previews, acceptance,
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
it must not recreate message replay, turn loops, or token aggregation locally.
## Target State
Proposal and repair instructions should remain coherent across a complete
conversation. The original context and supplied glossary and protected
spellings remain available, each repair branch keeps its own history, and
continuations select the repair definition, profile, and output schema.
Both shipped and operator-replaced definitions use native prompt selection.
Repair feedback identifies the current edit and the validator's findings so
that the model can revise the proposal or withdraw it.
Audita assesses every delivered revision, including the last allowed response.
Coordination stopping and correction acceptance are distinct outcomes.
Undelivered content is diagnostic evidence. Independent approvals survive
recoverable failures. Transient retries, structural repair, and correction
revision retain their separate budgets.
Prompt and feedback changes should address concrete issues in captured
requests while preserving validator acceptance and the allowance for minor
editorial cleanup. Broader context and combined semantic validation remain
deferred. Any proposal to expand validator context should identify a concrete
case and the evidence missing from its request. This follow-up retains current
context fitting and apply-time conflict handling.
Native execution diagnostics count the shared proposal once and include each
new repair and validator run. Terminal coverage and usage availability
remain distinct. Execution measurements describe work and cost. Review
prompt clarity using the supplied evidence and rendered instructions.
The adoption follow-up also removes duplicate repair decoding and per-call
summarization while preserving the existing diagnostic evidence.
## Deferred Evaluation
Formal corpus preparation and live-model comparisons remain optional follow-up
work. Keep live calls opt-in and outside the offline suite. When undertaken,
compare needed and unnecessary edits, missed corrections, validator disagreement,
and repair outcomes using independent editorial judgment. Native groups can
supply execution measurements without introducing a second accounting system.
No blocking product decisions remain for this follow-up scope.