Files
audita/docs/roadmap/prompt-quality.md

122 lines
6.6 KiB
Markdown
Raw Normal View History

# Prompt Quality Roadmap
**Status:** The earlier six prompt improvements are implemented. This follow-up
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
validation, and repair prompts through the same execution path used in production.
Live-model comparisons and broader context expansion remain deferred.
## Baseline and Boundaries
The [prompt registry](../internal/prompts.md),
[validator guide](../internal/validators.md), and
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
prompts separate durable system principles from initial proposal assignments,
guide independent and overlapping edits, supply current before/after repair
previews, include the target speaker in validation, and illustrate editorial
judgments, revision, and withdrawal.
Preserve these selected decisions:
- Module focus guides generation; otherwise acceptable minor editorial cleanup
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
- Keep the carefully tuned meaning-reversal criterion and distinct validator
responsibilities. Approvals require no explanation; confidence scores remain absent.
- Preserve literal spans, deletion, replacement policies, glossary context,
protected spellings, full revalidation, and deterministic application.
- Retain English defaults and native operator prompt replacement. Module sequence
reevaluation follows the [module guidance](../internal/modules.md).
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
coordination, per-operation deadlines, prepared-request notification, and selected
run accounting. Audita owns candidate identity, findings, previews, acceptance,
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
it must not recreate message replay, turn loops, or token aggregation locally.
## 1. Review Complete Native Conversations
Complete the PromptKit adoption review follow-ups before revising prompts.
- Extend the debug case of `TestSubprocessCorrectionRepairConversation` to
compare the top-level `llm-execution.json` provider-attempt total with the
provider fixture's request count. The total should include the initial proposal
once and count each repair and validator request.
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
- Decode each repair response once in the runner. Reuse the decoded value for
proposal-reference and action checks and for debug output. Preserve the existing
invalid-output disposition and raw response evidence.
[Repair runner](../../internal/framework/runner/repair.go)
- Compute each per-call execution summary once and reuse it for terminal coverage
and usage in `llm-execution.json`.
[Execution report](../../internal/framework/processreport/llm.go)
Use captured proposal exchanges and independent coordinator branches to inspect
representative proposal-to-repair conversations, including a second revision and
withdrawal. Review actual `OnPrepared` snapshots for instruction authority,
response-format transitions, glossary/protected-term context, and fresh previews.
A follow-up selects its repair definition, role profile, and schema directly.
Exercise operator replacements through native prompt discovery and precedence.
Compare the request that actually executes rather than separately rendering an
approximation. Reusing captured history must preserve the proposal context while
keeping each correction's repair history private.
Record concrete editorial concerns and revise affected prompts when warranted.
Do not change validator acceptance or narrow allowable cleanup merely to make
proposal and repair assignments identical.
## 2. Assess Repair Feedback and Outcomes
Use the real coordinator with controlled model responses for deterministic
integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion,
unavailable responses, and independent accepted siblings. Revalidate each revision
from the beginning. A normal coordination stop is not evidence of acceptance;
Audita's domain outcome determines that. Canceled undelivered content is evidence
only and must never enter the approved set.
Use current target previews and validator-owned findings to assess whether a
repair request explains the defect sufficiently. Keep presentation changes in
prompt assets and domain feedback construction; keep history and budget mechanics
upstream. Retain separate transient retry, structural repair, and correction
revision budgets.
## 3. Diagnose Evidence Gaps Using Actual Requests
For a concrete rejected correction, compare the captured proposal evidence with
what the validator actually received. The target speaker is already included.
Broader validation context remains evidence-dependent: propose a focused change
only when a captured case demonstrates relevant evidence missing from validation.
Use native execution-group totals to distinguish additional model work from
shared initial history. Select validator membership explicitly, count the initial
proposal once, and do not add overlapping group totals. Preserve unknown usage
and terminal coverage separately. These measurements describe execution and cost;
they do not establish editorial quality.
Existing local examples have not established a need for broader validation windows
or combined semantic validation. Keep context fitting and apply-time conflict
handling unchanged until concrete cases justify further work.
## 4. Verification and Documentation
Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter
and coordinator with deterministic model-boundary fixtures. Protect supplied-data
rendering, schemas, profile selection, current previews, branching, revalidation,
withdrawal, and operator replacement. Inspect representative rendered prose as
part of review; avoid phrase checks, prose snapshots, and tests that duplicate
upstream history or accounting algorithms.
Update canonical prompt and validator documentation when behavior changes.
Keep examples valid against their response schemas and verify affected links.
Run focused tests and the full offline suite for runtime changes. Pure wording
changes need proportionate review, not new change-detector tests.
## Deferred Evaluation
Formal corpus preparation and live-model comparisons remain optional follow-up
work. Keep live calls opt-in and outside the offline suite. When undertaken,
compare needed and unnecessary edits, missed corrections, validator disagreement,
and repair outcomes using independent editorial judgment. Native groups can
supply execution measurements without introducing a second accounting system.
No blocking product decisions remain for this follow-up scope.