5.5 KiB
Prompt Quality Roadmap
Updated: 2026-11-11
Status: Proposed discussion roadmap; decisions and implementation pending. Work through the topics below in order. Recommendations are starting points for discussion, not newly adopted restrictions or guarantees.
Baseline and Intended Direction
The review of revision 6d34602 covered all eleven
prompt definitions, their shared instructions, and
the context and response handling for proposals, validation, and repair.
It was a design review, not a measurement of live-model quality.
Retain the strengths: conservative transcript editing, explicit literal-span and deletion semantics, distinct glossary context and protection, full before/after validation, independent repair conversations with withdrawal, and native PromptKit/ConfigKit prompt replacement. Improve consistency across the three roles, useful rejection feedback, and maintainability.
Discussion and Work Order
1. Align Validation with Module Scope
The grammar proposal permits formatting and the narrow a/an exception,
while its editorial validator also permits homophone corrections, dysfluency
removal, and combinations of those changes. An edit can satisfy that validator
while violating the proposing module's instructions.
Recommendation: supply the originating module's allowed edits to editorial validation and assess them alongside meaning preservation. Prefer reusing the shared validator where practical. This needs request-context wiring as well as prompt changes; a prompt rewrite alone cannot supply missing scope information. Discuss the scope contract and shared versus specialized definitions first. See modules and validators.
2. Make Rejection Reasons Useful for Repair
Validator explanations now become repair feedback, but current instructions give little guidance about producing actionable reasons.
Recommendation: ask for a concise description of the specific change, the
applicable criterion, and relevant evidence, within the validator's remit.
For example: removing "might" turns a tentative statement into a definite one;
preserve the speaker's uncertainty. Start with the existing reason field.
Discuss examples and sufficient detail before considering schema expansion.
3. Make Repair a Reconsideration of the Edit's Merits
Withdrawal is already explicit. Clarify when the model should use it and how it should respond when the original evidence remains insufficient.
Recommendation: ask the model to reassess the correction from the supplied evidence, preserve module scope, and withdraw unsupported proposals. Address the risk that numerical confidence feedback merely encourages raising the score. Do not automatically prohibit confidence-only revisions: legitimate reassessment is possible. Discuss and evaluate that behavior before adding a business rule. Retain the agreed full-history conversation model and existing repair lifecycle.
4. Simplify Instructions and Improve Examples
Restrictions and examples recur across proposal, validation, and repair assets; the grammar prompt repeats its word-substitution boundary several ways.
Recommendation: distinguish shared editing principles, module scope, and task-specific response contracts. Use small shared fragments where they reduce drift. Replace some repetitive example lists with contextual contrasts, such as accidental repetition versus deliberate emphasis. Preserve useful distinctions; shorter prompts alone are not the objective. Discuss what belongs in shared fragments and what remains local to each definition.
5. Evaluate Editorial Quality
Rendering, schema, and controlled-response tests establish integration behavior, but do not establish the quality of model editorial judgments.
Recommendation: assemble a modest evaluation set containing both needed edits and text that should remain unchanged. Include ambiguous glossary matches, intentional repetition, meaningful uses of apparent filler, punctuation that changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, missed corrections, validator disagreement, and repair outcomes using independent editorial assessment; validator approval alone is insufficient evidence.
Capture candidate examples during topics 1–4 and establish a baseline before changing prompts. Discuss representative models, expected outcomes, and how to run evaluations. Keep model-dependent evaluation separate from the default offline test suite; no numerical quality gate is selected here.
6. Revisit Broader Product Choices
Use the preceding decisions and evaluation evidence to discuss these in order:
- Grammar naming and scope: the current module is close to copy editing, with an isolated article exception. Decide whether its name and scope convey the intended product behavior.
- Glossary/homophone overlap: assess the benefit of their overlapping remit and repeated glossary passes before changing the default sequence.
- Language and examples: decide whether spoken English and campaign-specific examples are the right bundled defaults, and how broadly those defaults should apply.
- Meaning-reversal policy: the guard can also exclude a genuine correction of a mistranscription. Initially retain the conservative default while explicitly evaluating that tradeoff.
These are product discussions, not selected renames, broader editing permissions, language-support promises, or changes to validation policy.