75 lines
4.1 KiB
Markdown
75 lines
4.1 KiB
Markdown
# Prompt Quality Roadmap
|
|
|
|
**Status:** The earlier six prompt improvements are implemented. This follow-up
|
|
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
|
|
validation, and repair prompts through the same execution path used in production.
|
|
Live-model comparisons and broader context expansion remain deferred. See the
|
|
[implementation plan](implementation.md) for the sprint sequence.
|
|
|
|
## Baseline and Boundaries
|
|
|
|
The [prompt registry](../internal/prompts.md),
|
|
[validator guide](../internal/validators.md), and
|
|
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
|
|
prompts separate durable system principles from initial proposal assignments,
|
|
guide independent and overlapping edits, supply current before/after repair
|
|
previews, include the target speaker in validation, and illustrate editorial
|
|
judgments, revision, and withdrawal.
|
|
|
|
Preserve these selected decisions:
|
|
|
|
- Module focus guides generation; otherwise acceptable minor editorial cleanup
|
|
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
|
- Keep the carefully tuned meaning-reversal criterion and distinct validator
|
|
responsibilities. Approvals require no explanation; confidence scores remain absent.
|
|
- Preserve literal spans, deletion, replacement policies, glossary context,
|
|
protected spellings, full revalidation, and deterministic application.
|
|
- Retain English defaults and native operator prompt replacement. Module sequence
|
|
reevaluation follows the [module guidance](../internal/modules.md).
|
|
|
|
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
|
|
coordination, per-operation deadlines, prepared-request notification, and selected
|
|
run accounting. Audita owns candidate identity, findings, previews, acceptance,
|
|
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
|
|
it must not recreate message replay, turn loops, or token aggregation locally.
|
|
|
|
## Target State
|
|
|
|
Proposal and repair instructions should remain coherent across a complete
|
|
conversation. The original context and supplied glossary and protected
|
|
spellings remain available, each repair branch keeps its own history, and
|
|
continuations select the repair definition, profile, and output schema.
|
|
Both shipped and operator-replaced definitions use native prompt selection.
|
|
Repair feedback identifies the current edit and the validator's findings so
|
|
that the model can revise the proposal or withdraw it.
|
|
|
|
Audita assesses every delivered revision, including the last allowed response.
|
|
Coordination stopping and correction acceptance are distinct outcomes.
|
|
Undelivered content is diagnostic evidence. Independent approvals survive
|
|
recoverable failures. Transient retries, structural repair, and correction
|
|
revision retain their separate budgets.
|
|
|
|
Prompt and feedback changes should address concrete issues in captured
|
|
requests while preserving validator acceptance and the allowance for minor
|
|
editorial cleanup. Broader context and combined semantic validation remain
|
|
deferred. Any proposal to expand validator context should identify a concrete
|
|
case and the evidence missing from its request. This follow-up retains current
|
|
context fitting and apply-time conflict handling.
|
|
|
|
Native execution diagnostics count the shared proposal once and include each
|
|
new repair and validator run. Terminal coverage and usage availability
|
|
remain distinct. Execution measurements describe work and cost. Review
|
|
prompt clarity using the supplied evidence and rendered instructions.
|
|
The adoption follow-up also removes duplicate repair decoding and per-call
|
|
summarization while preserving the existing diagnostic evidence.
|
|
|
|
## Deferred Evaluation
|
|
|
|
Formal corpus preparation and live-model comparisons remain optional follow-up
|
|
work. Keep live calls opt-in and outside the offline suite. When undertaken,
|
|
compare needed and unnecessary edits, missed corrections, validator disagreement,
|
|
and repair outcomes using independent editorial judgment. Native groups can
|
|
supply execution measurements without introducing a second accounting system.
|
|
|
|
No blocking product decisions remain for this follow-up scope.
|