122 lines
6.6 KiB
Markdown
122 lines
6.6 KiB
Markdown
# Prompt Quality Roadmap
|
|
|
|
**Status:** The earlier six prompt improvements are implemented. This follow-up
|
|
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
|
|
validation, and repair prompts through the same execution path used in production.
|
|
Live-model comparisons and broader context expansion remain deferred.
|
|
|
|
## Baseline and Boundaries
|
|
|
|
The [prompt registry](../internal/prompts.md),
|
|
[validator guide](../internal/validators.md), and
|
|
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
|
|
prompts separate durable system principles from initial proposal assignments,
|
|
guide independent and overlapping edits, supply current before/after repair
|
|
previews, include the target speaker in validation, and illustrate editorial
|
|
judgments, revision, and withdrawal.
|
|
|
|
Preserve these selected decisions:
|
|
|
|
- Module focus guides generation; otherwise acceptable minor editorial cleanup
|
|
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
|
- Keep the carefully tuned meaning-reversal criterion and distinct validator
|
|
responsibilities. Approvals require no explanation; confidence scores remain absent.
|
|
- Preserve literal spans, deletion, replacement policies, glossary context,
|
|
protected spellings, full revalidation, and deterministic application.
|
|
- Retain English defaults and native operator prompt replacement. Module sequence
|
|
reevaluation follows the [module guidance](../internal/modules.md).
|
|
|
|
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
|
|
coordination, per-operation deadlines, prepared-request notification, and selected
|
|
run accounting. Audita owns candidate identity, findings, previews, acceptance,
|
|
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
|
|
it must not recreate message replay, turn loops, or token aggregation locally.
|
|
|
|
## 1. Review Complete Native Conversations
|
|
|
|
Complete the PromptKit adoption review follow-ups before revising prompts.
|
|
|
|
- Extend the debug case of `TestSubprocessCorrectionRepairConversation` to
|
|
compare the top-level `llm-execution.json` provider-attempt total with the
|
|
provider fixture's request count. The total should include the initial proposal
|
|
once and count each repair and validator request.
|
|
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
|
|
- Decode each repair response once in the runner. Reuse the decoded value for
|
|
proposal-reference and action checks and for debug output. Preserve the existing
|
|
invalid-output disposition and raw response evidence.
|
|
[Repair runner](../../internal/framework/runner/repair.go)
|
|
- Compute each per-call execution summary once and reuse it for terminal coverage
|
|
and usage in `llm-execution.json`.
|
|
[Execution report](../../internal/framework/processreport/llm.go)
|
|
|
|
Use captured proposal exchanges and independent coordinator branches to inspect
|
|
representative proposal-to-repair conversations, including a second revision and
|
|
withdrawal. Review actual `OnPrepared` snapshots for instruction authority,
|
|
response-format transitions, glossary/protected-term context, and fresh previews.
|
|
A follow-up selects its repair definition, role profile, and schema directly.
|
|
|
|
Exercise operator replacements through native prompt discovery and precedence.
|
|
Compare the request that actually executes rather than separately rendering an
|
|
approximation. Reusing captured history must preserve the proposal context while
|
|
keeping each correction's repair history private.
|
|
|
|
Record concrete editorial concerns and revise affected prompts when warranted.
|
|
Do not change validator acceptance or narrow allowable cleanup merely to make
|
|
proposal and repair assignments identical.
|
|
|
|
## 2. Assess Repair Feedback and Outcomes
|
|
|
|
Use the real coordinator with controlled model responses for deterministic
|
|
integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion,
|
|
unavailable responses, and independent accepted siblings. Revalidate each revision
|
|
from the beginning. A normal coordination stop is not evidence of acceptance;
|
|
Audita's domain outcome determines that. Canceled undelivered content is evidence
|
|
only and must never enter the approved set.
|
|
|
|
Use current target previews and validator-owned findings to assess whether a
|
|
repair request explains the defect sufficiently. Keep presentation changes in
|
|
prompt assets and domain feedback construction; keep history and budget mechanics
|
|
upstream. Retain separate transient retry, structural repair, and correction
|
|
revision budgets.
|
|
|
|
## 3. Diagnose Evidence Gaps Using Actual Requests
|
|
|
|
For a concrete rejected correction, compare the captured proposal evidence with
|
|
what the validator actually received. The target speaker is already included.
|
|
Broader validation context remains evidence-dependent: propose a focused change
|
|
only when a captured case demonstrates relevant evidence missing from validation.
|
|
|
|
Use native execution-group totals to distinguish additional model work from
|
|
shared initial history. Select validator membership explicitly, count the initial
|
|
proposal once, and do not add overlapping group totals. Preserve unknown usage
|
|
and terminal coverage separately. These measurements describe execution and cost;
|
|
they do not establish editorial quality.
|
|
|
|
Existing local examples have not established a need for broader validation windows
|
|
or combined semantic validation. Keep context fitting and apply-time conflict
|
|
handling unchanged until concrete cases justify further work.
|
|
|
|
## 4. Verification and Documentation
|
|
|
|
Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter
|
|
and coordinator with deterministic model-boundary fixtures. Protect supplied-data
|
|
rendering, schemas, profile selection, current previews, branching, revalidation,
|
|
withdrawal, and operator replacement. Inspect representative rendered prose as
|
|
part of review; avoid phrase checks, prose snapshots, and tests that duplicate
|
|
upstream history or accounting algorithms.
|
|
|
|
Update canonical prompt and validator documentation when behavior changes.
|
|
Keep examples valid against their response schemas and verify affected links.
|
|
Run focused tests and the full offline suite for runtime changes. Pure wording
|
|
changes need proportionate review, not new change-detector tests.
|
|
|
|
## Deferred Evaluation
|
|
|
|
Formal corpus preparation and live-model comparisons remain optional follow-up
|
|
work. Keep live calls opt-in and outside the offline suite. When undertaken,
|
|
compare needed and unnecessary edits, missed corrections, validator disagreement,
|
|
and repair outcomes using independent editorial judgment. Native groups can
|
|
supply execution measurements without introducing a second accounting system.
|
|
|
|
No blocking product decisions remain for this follow-up scope.
|