Files
audita/docs/roadmap/prompt-quality.md

153 lines
8.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Prompt Quality Roadmap
**Updated:** 2026-11-11
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
pending; topics 4–6 remain for discussion.
Work through the remaining topics in order. Their recommendations are starting
points for discussion, not newly adopted restrictions or guarantees.
## Baseline and Intended Direction
The review of revision `6d34602` covered all eleven
[prompt definitions](../internal/prompts.md), their shared instructions, and
the context and response handling for proposals, validation, and repair.
It was a design review, not a measurement of live-model quality.
Retain the strengths: conservative transcript editing, explicit literal-span and
deletion semantics, distinct glossary context and protection, full before/after
validation, independent repair conversations with withdrawal, and native
PromptKit/ConfigKit prompt replacement. Improve consistency across the three
roles, useful rejection feedback, and maintainability. Remove model-reported
confidence scores and confidence thresholds throughout the correction workflow.
## Discussion and Work Order
### 1. Module Focus and Editorial Acceptance — Resolved
The intentional distinction between requested module focus and acceptance of
minor editorial cleanup is recorded in
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). The maintainer's
testing found that strict scope instructions also suppress desirable edits.
Apply the accepted distinction throughout the remaining prompt work.
### 2. Make Rejection Reasons Useful for Repair — Decided
Validator explanations now become repair feedback, but current instructions give
little guidance about producing actionable reasons.
**Decision:** retain the existing `reason` field. Instruct validators to explain
rejections with a concise description of the specific problematic edit, the
applicable criterion, and relevant supplied evidence, within their own remit.
For example: removing "might" turns a tentative statement into a definite one.
Validators identify the defect; the repair module owns revision or withdrawal.
Respect the accepted minor editorial allowance, distinguish insufficient
evidence from an established defect, and include distinct material issues when
found without requiring an exhaustive analysis.
Approvals require no explanation: instruct the model to return `reason: ""`.
The existing schema requires the field but permits an empty string, which the
response handler already accepts. This reason convention itself needs no schema
or response-handling changes; topic 3 separately removes confidence from those
contracts. The rejection guidance is a prompt instruction;
this decision adds no runtime rule rejecting blank explanations.
**Remaining work:** update the shared validation guidance and all three validator
definitions with representative rejection examples and the empty-reason approval
convention. Check rendered prompts and representative repair payloads to confirm
that rejection feedback identifies defects without prescribing replacement
corrections. Consider schema expansion only if a concrete need emerges.
### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided
**Decision:** remove confidence entirely from proposals, validation, and repair,
including the threshold validator and its configuration. Audita has no established
calibration that gives its model-reported scores a reliable numerical meaning
across transcripts and models. Proposal scores currently affect acceptance;
validator scores are range-checked but do not determine approval. Retaining these
numbers adds complexity and encourages repairs aimed at meeting a threshold.
The target workflow relies on substantive validation criteria and supplied
evidence, without a replacement numerical or categorical confidence scale.
Repair instructions should ask the model to reassess a correction using the
original transcript, supplied context, and rejection feedback. Revise when the
evidence supports a correction that addresses the identified problems; withdraw
when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal
is a successful resolution that preserves the original text. Explain concrete
uncertainty in rejection feedback where relevant, such as context that cannot
distinguish two possible names. Preserve the module's requested focus, the
accepted minor editorial allowance, the full-history conversation model, and
the existing repair lifecycle and revalidation.
**Remaining work:**
- Remove confidence fields, instructions, and examples from all proposal,
validation, and repair prompts and response schemas, and from their Go models,
decoding, and validation paths.
- Remove the confidence-threshold validator, its registration and chain entries,
root and stage threshold settings, and any corresponding CLI exposure.
- Remove confidence-specific rejection feedback, reason codes, report/debug
fields, and other plumbing wherever present. Update affected tests, fixtures,
configuration examples, and canonical documentation when implementation lands.
- Treat this as a deliberate contract change for configuration, custom prompts,
and affected serialized outputs; provide migration guidance to remove obsolete
fields and settings. Do not retain inert confidence fields or threshold knobs.
- Verify that confidence-free proposals, validation decisions, and repair
responses work end to end, while substantive validators, revision, withdrawal,
and revalidation continue to operate as intended.
Removing the proposal threshold can admit corrections it previously rejected.
Use the evaluation work in topic 5 to assess that effect; removal is the selected
direction, not conditional on a further confidence-policy decision.
### 4. Simplify Instructions and Improve Examples
Restrictions and examples recur across proposal, validation, and repair assets;
the grammar prompt repeats its word-substitution boundary several ways.
**Recommendation:** distinguish shared editing principles, requested module focus,
validator acceptance criteria, and task-specific response contracts. Use small
shared fragments where they reduce drift. Replace some repetitive example lists
with contextual contrasts, such as
accidental repetition versus deliberate emphasis. Preserve useful distinctions;
shorter prompts alone are not the objective. Discuss what belongs in shared
fragments and what remains local to each definition.
### 5. Evaluate Editorial Quality
Rendering, schema, and controlled-response tests establish integration behavior,
but do not establish the quality of model editorial judgments.
**Recommendation:** assemble a modest evaluation set containing both needed edits
and text that should remain unchanged. Include ambiguous glossary matches,
intentional repetition, meaningful uses of apparent filler, punctuation that
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
missed corrections, validator disagreement, and repair outcomes using independent
editorial assessment; validator approval alone is insufficient evidence.
Include a comparison against the prior confidence-threshold behavior to identify
changes in both useful corrections and erroneous edits after confidence removal.
Capture candidate examples during topics 1–4 and establish a baseline before
changing prompts. Discuss representative models, expected outcomes, and how to
run evaluations. Keep model-dependent evaluation separate from the default
offline test suite; no numerical quality gate is selected here.
### 6. Revisit Broader Product Choices
Use the preceding decisions and evaluation evidence to discuss these in order:
1. **Grammar naming and scope:** the current module is close to copy editing,
with an isolated article exception. Decide whether its name and scope convey
the intended requested work, preserving the accepted editorial allowance.
2. **Glossary/homophone overlap:** assess the benefit of their overlapping remit
and repeated glossary passes before changing the default sequence.
3. **Language and examples:** decide whether spoken English and campaign-specific
examples are the right bundled defaults, and how broadly those defaults
should apply.
4. **Meaning-reversal policy:** the guard can also exclude a genuine correction
of a mistranscription. Initially retain the conservative default while
explicitly evaluating that tradeoff.
These are product discussions, not selected renames, broader editing permissions,
language-support promises, or changes to validation policy.