From ca36c3e31964270b0d22751418cbf83008408722 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 14:05:20 +0000 Subject: [PATCH] Outline prompt quality discussion roadmap --- docs/roadmap/prompt-quality.md | 107 +++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/roadmap/prompt-quality.md diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md new file mode 100644 index 0000000..58b0646 --- /dev/null +++ b/docs/roadmap/prompt-quality.md @@ -0,0 +1,107 @@ +# Prompt Quality Roadmap + +**Updated:** 2026-11-11 + +**Status:** Proposed discussion roadmap; decisions and implementation pending. +Work through the topics below in order. Recommendations are starting points for +discussion, not newly adopted restrictions or guarantees. + +## Baseline and Intended Direction + +The review of revision `6d34602` covered all eleven +[prompt definitions](../internal/prompts.md), their shared instructions, and +the context and response handling for proposals, validation, and repair. +It was a design review, not a measurement of live-model quality. + +Retain the strengths: conservative transcript editing, explicit literal-span and +deletion semantics, distinct glossary context and protection, full before/after +validation, independent repair conversations with withdrawal, and native +PromptKit/ConfigKit prompt replacement. Improve consistency across the three +roles, useful rejection feedback, and maintainability. + +## Discussion and Work Order + +### 1. Align Validation with Module Scope + +The grammar proposal permits formatting and the narrow `a`/`an` exception, +while its editorial validator also permits homophone corrections, dysfluency +removal, and combinations of those changes. An edit can satisfy that validator +while violating the proposing module's instructions. + +**Recommendation:** supply the originating module's allowed edits to editorial +validation and assess them alongside meaning preservation. Prefer reusing the +shared validator where practical. This needs request-context wiring as well as +prompt changes; a prompt rewrite alone cannot supply missing scope information. +Discuss the scope contract and shared versus specialized definitions first. +See [modules](../internal/modules.md) and [validators](../internal/validators.md). + +### 2. Make Rejection Reasons Useful for Repair + +Validator explanations now become repair feedback, but current instructions give +little guidance about producing actionable reasons. + +**Recommendation:** ask for a concise description of the specific change, the +applicable criterion, and relevant evidence, within the validator's remit. +For example: removing "might" turns a tentative statement into a definite one; +preserve the speaker's uncertainty. Start with the existing `reason` field. +Discuss examples and sufficient detail before considering schema expansion. + +### 3. Make Repair a Reconsideration of the Edit's Merits + +Withdrawal is already explicit. Clarify when the model should use it and how it +should respond when the original evidence remains insufficient. + +**Recommendation:** ask the model to reassess the correction from the supplied +evidence, preserve module scope, and withdraw unsupported proposals. Address the +risk that numerical confidence feedback merely encourages raising the score. +Do not automatically prohibit confidence-only revisions: legitimate reassessment +is possible. Discuss and evaluate that behavior before adding a business rule. +Retain the agreed full-history conversation model and existing repair lifecycle. + +### 4. Simplify Instructions and Improve Examples + +Restrictions and examples recur across proposal, validation, and repair assets; +the grammar prompt repeats its word-substitution boundary several ways. + +**Recommendation:** distinguish shared editing principles, module scope, and +task-specific response contracts. Use small shared fragments where they reduce +drift. Replace some repetitive example lists with contextual contrasts, such as +accidental repetition versus deliberate emphasis. Preserve useful distinctions; +shorter prompts alone are not the objective. Discuss what belongs in shared +fragments and what remains local to each definition. + +### 5. Evaluate Editorial Quality + +Rendering, schema, and controlled-response tests establish integration behavior, +but do not establish the quality of model editorial judgments. + +**Recommendation:** assemble a modest evaluation set containing both needed edits +and text that should remain unchanged. Include ambiguous glossary matches, +intentional repetition, meaningful uses of apparent filler, punctuation that +changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, +missed corrections, validator disagreement, and repair outcomes using independent +editorial assessment; validator approval alone is insufficient evidence. + +Capture candidate examples during topics 1–4 and establish a baseline before +changing prompts. Discuss representative models, expected outcomes, and how to +run evaluations. Keep model-dependent evaluation separate from the default +offline test suite; no numerical quality gate is selected here. + +### 6. Revisit Broader Product Choices + +Use the preceding decisions and evaluation evidence to discuss these in order: + +1. **Grammar naming and scope:** the current module is close to copy editing, + with an isolated article exception. Decide whether its name and scope convey + the intended product behavior. +2. **Glossary/homophone overlap:** assess the benefit of their overlapping remit + and repeated glossary passes before changing the default sequence. +3. **Language and examples:** decide whether spoken English and campaign-specific + examples are the right bundled defaults, and how broadly those defaults + should apply. +4. **Meaning-reversal policy:** the guard can also exclude a genuine correction + of a mistranscription. Initially retain the conservative default while + explicitly evaluating that tradeoff. + +These are product discussions, not selected renames, broader editing permissions, +language-support promises, or changes to validation policy.