108 lines
5.5 KiB
Markdown
108 lines
5.5 KiB
Markdown
# Prompt Quality Roadmap
|
||
|
||
**Updated:** 2026-11-11
|
||
|
||
**Status:** Proposed discussion roadmap; decisions and implementation pending.
|
||
Work through the topics below in order. Recommendations are starting points for
|
||
discussion, not newly adopted restrictions or guarantees.
|
||
|
||
## Baseline and Intended Direction
|
||
|
||
The review of revision `6d34602` covered all eleven
|
||
[prompt definitions](../internal/prompts.md), their shared instructions, and
|
||
the context and response handling for proposals, validation, and repair.
|
||
It was a design review, not a measurement of live-model quality.
|
||
|
||
Retain the strengths: conservative transcript editing, explicit literal-span and
|
||
deletion semantics, distinct glossary context and protection, full before/after
|
||
validation, independent repair conversations with withdrawal, and native
|
||
PromptKit/ConfigKit prompt replacement. Improve consistency across the three
|
||
roles, useful rejection feedback, and maintainability.
|
||
|
||
## Discussion and Work Order
|
||
|
||
### 1. Align Validation with Module Scope
|
||
|
||
The grammar proposal permits formatting and the narrow `a`/`an` exception,
|
||
while its editorial validator also permits homophone corrections, dysfluency
|
||
removal, and combinations of those changes. An edit can satisfy that validator
|
||
while violating the proposing module's instructions.
|
||
|
||
**Recommendation:** supply the originating module's allowed edits to editorial
|
||
validation and assess them alongside meaning preservation. Prefer reusing the
|
||
shared validator where practical. This needs request-context wiring as well as
|
||
prompt changes; a prompt rewrite alone cannot supply missing scope information.
|
||
Discuss the scope contract and shared versus specialized definitions first.
|
||
See [modules](../internal/modules.md) and [validators](../internal/validators.md).
|
||
|
||
### 2. Make Rejection Reasons Useful for Repair
|
||
|
||
Validator explanations now become repair feedback, but current instructions give
|
||
little guidance about producing actionable reasons.
|
||
|
||
**Recommendation:** ask for a concise description of the specific change, the
|
||
applicable criterion, and relevant evidence, within the validator's remit.
|
||
For example: removing "might" turns a tentative statement into a definite one;
|
||
preserve the speaker's uncertainty. Start with the existing `reason` field.
|
||
Discuss examples and sufficient detail before considering schema expansion.
|
||
|
||
### 3. Make Repair a Reconsideration of the Edit's Merits
|
||
|
||
Withdrawal is already explicit. Clarify when the model should use it and how it
|
||
should respond when the original evidence remains insufficient.
|
||
|
||
**Recommendation:** ask the model to reassess the correction from the supplied
|
||
evidence, preserve module scope, and withdraw unsupported proposals. Address the
|
||
risk that numerical confidence feedback merely encourages raising the score.
|
||
Do not automatically prohibit confidence-only revisions: legitimate reassessment
|
||
is possible. Discuss and evaluate that behavior before adding a business rule.
|
||
Retain the agreed full-history conversation model and existing repair lifecycle.
|
||
|
||
### 4. Simplify Instructions and Improve Examples
|
||
|
||
Restrictions and examples recur across proposal, validation, and repair assets;
|
||
the grammar prompt repeats its word-substitution boundary several ways.
|
||
|
||
**Recommendation:** distinguish shared editing principles, module scope, and
|
||
task-specific response contracts. Use small shared fragments where they reduce
|
||
drift. Replace some repetitive example lists with contextual contrasts, such as
|
||
accidental repetition versus deliberate emphasis. Preserve useful distinctions;
|
||
shorter prompts alone are not the objective. Discuss what belongs in shared
|
||
fragments and what remains local to each definition.
|
||
|
||
### 5. Evaluate Editorial Quality
|
||
|
||
Rendering, schema, and controlled-response tests establish integration behavior,
|
||
but do not establish the quality of model editorial judgments.
|
||
|
||
**Recommendation:** assemble a modest evaluation set containing both needed edits
|
||
and text that should remain unchanged. Include ambiguous glossary matches,
|
||
intentional repetition, meaningful uses of apparent filler, punctuation that
|
||
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
|
||
missed corrections, validator disagreement, and repair outcomes using independent
|
||
editorial assessment; validator approval alone is insufficient evidence.
|
||
|
||
Capture candidate examples during topics 1–4 and establish a baseline before
|
||
changing prompts. Discuss representative models, expected outcomes, and how to
|
||
run evaluations. Keep model-dependent evaluation separate from the default
|
||
offline test suite; no numerical quality gate is selected here.
|
||
|
||
### 6. Revisit Broader Product Choices
|
||
|
||
Use the preceding decisions and evaluation evidence to discuss these in order:
|
||
|
||
1. **Grammar naming and scope:** the current module is close to copy editing,
|
||
with an isolated article exception. Decide whether its name and scope convey
|
||
the intended product behavior.
|
||
2. **Glossary/homophone overlap:** assess the benefit of their overlapping remit
|
||
and repeated glossary passes before changing the default sequence.
|
||
3. **Language and examples:** decide whether spoken English and campaign-specific
|
||
examples are the right bundled defaults, and how broadly those defaults
|
||
should apply.
|
||
4. **Meaning-reversal policy:** the guard can also exclude a genuine correction
|
||
of a mistranscription. Initially retain the conservative default while
|
||
explicitly evaluating that tradeoff.
|
||
|
||
These are product discussions, not selected renames, broader editing permissions,
|
||
language-support promises, or changes to validation policy.
|