# Prompt Quality Roadmap **Updated:** 2026-11-11 **Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation pending; topic 5 deferred; grammar naming/scope, glossary/homophone overlap, and language/examples in topic 6 decided, with description and example updates pending; meaning-reversal policy awaits discussion. Work through the remaining topics in order. Their recommendations are starting points for discussion, not newly adopted restrictions or guarantees. ## Baseline and Intended Direction The review of revision `6d34602` covered all eleven [prompt definitions](../internal/prompts.md), their shared instructions, and the context and response handling for proposals, validation, and repair. It was a design review, not a measurement of live-model quality. The maintainer reports that nearly all module behavior was selected through practical experimentation. Treat that experience as evidence when reviewing the design: overlapping responsibilities and repeated passes are not inherently redundant. The deferred evaluation work does not imply an absence of empirical support for the existing module design. This experience informs ongoing choices; it does not lock in the current default. Frequently reevaluate the default module combination and sequence, selecting what has been observed to work best for most workloads. Retain the strengths: conservative transcript editing, explicit literal-span and deletion semantics, distinct glossary context and protection, full before/after validation, independent repair conversations with withdrawal, and native PromptKit/ConfigKit prompt replacement. Improve consistency across the three roles, useful rejection feedback, and maintainability. Remove model-reported confidence scores and confidence thresholds throughout the correction workflow. ## Discussion and Work Order ### 1. Module Focus and Editorial Acceptance — Resolved The intentional distinction between requested module focus and acceptance of minor editorial cleanup is recorded in [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). The maintainer's testing found that strict scope instructions also suppress desirable edits. Apply the accepted distinction throughout the remaining prompt work. ### 2. Make Rejection Reasons Useful for Repair — Decided Validator explanations now become repair feedback, but current instructions give little guidance about producing actionable reasons. **Decision:** retain the existing `reason` field. Instruct validators to explain rejections with a concise description of the specific problematic edit, the applicable criterion, and relevant supplied evidence, within their own remit. For example: removing "might" turns a tentative statement into a definite one. Validators identify the defect; the repair module owns revision or withdrawal. Respect the accepted minor editorial allowance, distinguish insufficient evidence from an established defect, and include distinct material issues when found without requiring an exhaustive analysis. Approvals require no explanation: instruct the model to return `reason: ""`. The existing schema requires the field but permits an empty string, which the response handler already accepts. This reason convention itself needs no schema or response-handling changes; topic 3 separately removes confidence from those contracts. The rejection guidance is a prompt instruction; this decision adds no runtime rule rejecting blank explanations. **Remaining work:** update the shared validation guidance and all three validator definitions with representative rejection examples and the empty-reason approval convention. Check rendered prompts and representative repair payloads to confirm that rejection feedback identifies defects without prescribing replacement corrections. Consider schema expansion only if a concrete need emerges. ### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided **Decision:** remove confidence entirely from proposals, validation, and repair, including the threshold validator and its configuration. Audita has no established calibration that gives its model-reported scores a reliable numerical meaning across transcripts and models. Proposal scores currently affect acceptance; validator scores are range-checked but do not determine approval. Retaining these numbers adds complexity and encourages repairs aimed at meeting a threshold. The target workflow relies on substantive validation criteria and supplied evidence, without a replacement numerical or categorical confidence scale. Repair instructions should ask the model to reassess a correction using the original transcript, supplied context, and rejection feedback. Revise when the evidence supports a correction that addresses the identified problems; withdraw when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal is a successful resolution that preserves the original text. Explain concrete uncertainty in rejection feedback where relevant, such as context that cannot distinguish two possible names. Preserve the module's requested focus, the accepted minor editorial allowance, the full-history conversation model, and the existing repair lifecycle and revalidation. **Remaining work:** - Remove confidence fields, instructions, and examples from all proposal, validation, and repair prompts and response schemas, and from their Go models, decoding, and validation paths. - Remove the confidence-threshold validator, its registration and chain entries, root and stage threshold settings, and any corresponding CLI exposure. - Remove confidence-specific rejection feedback, reason codes, report/debug fields, and other plumbing wherever present. Update affected tests, fixtures, configuration examples, and canonical documentation when implementation lands. - Treat this as a deliberate contract change for configuration, custom prompts, and affected serialized outputs; provide migration guidance to remove obsolete fields and settings. Do not retain inert confidence fields or threshold knobs. - Verify that confidence-free proposals, validation decisions, and repair responses work end to end, while substantive validators, revision, withdrawal, and revalidation continue to operate as intended. Removing the proposal threshold can admit corrections it previously rejected. The deferred evaluation work in topic 5 can assess that effect when resumed; removal is the selected direction and does not depend on that work. ### 4. Simplify Instructions and Improve Examples — Decided Restrictions and examples recur across proposal, validation, and repair assets; the grammar prompt repeats its word-substitution boundary several ways. **Decision:** make a focused editing pass using the existing module and prompt fragment structure. Organize instructions around shared contracts (including response format, literal matching, and deletion semantics), module-specific requested work, and the distinct responsibilities of validation and repair. Consolidate duplicated rules where small shared fragments prevent drift, remove contradictions and unnecessary repetition, and keep each rendered prompt understandable on its own. Retain repetition across separate model calls when each call needs the instruction. Use short contextual contrasts to clarify difficult judgments, such as an accidental restart versus deliberate emphasis, or a glossary match supported by context versus a merely similar spelling. Examples should illustrate how context informs the decision, not create unconditional editing rules. Preserve the distinction between requested module focus and acceptance of otherwise appropriate minor editorial cleanup. Shorter prompts alone are not the objective. **Remaining work:** revise the proposal, validation, and repair assets and shared fragments consistently with topics 1–3, retaining the existing module structure. Check rendering, response-schema compatibility, and integration behavior as appropriate. Proceed with these revisions before waiting for live-model evaluation evidence or a captured baseline. Topic 5 is deferred and is not a prerequisite for this editing pass. ### 5. Evaluate Editorial Quality — Deferred **Decision:** defer this work for now. Evaluation corpus preparation, live-model comparisons, and an evaluation workflow are outside the current implementation scope. This deferral does not block topics 2–4 or the remaining product discussions; ordinary implementation checks still apply. The following direction is retained for reconsideration when evaluation work resumes. Rendering, schema, and controlled-response tests establish integration behavior, but do not establish the quality of model editorial judgments. **Future recommendation:** assemble a modest evaluation set containing both needed edits and text that should remain unchanged. Include ambiguous glossary matches, intentional repetition, meaningful uses of apparent filler, punctuation that changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, missed corrections, validator disagreement, and repair outcomes using independent editorial assessment; validator approval alone is insufficient evidence. Include a comparison against the prior confidence-threshold behavior to identify changes in both useful corrections and erroneous edits after confidence removal. When this work resumes, discuss representative models, expected outcomes, and how to run evaluations. Use the prior repository revisions for before/after comparisons when that work proceeds; a pre-change evaluation run is not required. Keep model-dependent evaluation separate from the default offline test suite; no numerical quality gate is selected here. ### 6. Revisit Broader Product Choices Use the preceding decisions and any available evidence to discuss these in order, without waiting for the deferred evaluation work: 1. **Grammar naming and scope — Decided:** retain the `grammar` name, module key, and corresponding prompt IDs. Clarify its description as conservative punctuation, capitalization, spacing, and whole-word `a`/`an` article cleanup. Preserve its current requested scope and the accepted cross-module minor editorial allowance. Update the relevant module descriptions and canonical documentation during implementation; no rename or broader grammar-correction remit is planned. 2. **Glossary/homophone overlap and defaults — Decided:** retain both modules and their useful overlap. Keep the existing default sequence documented in [configuration](../config.md#pipeline-and-glossary-selections) for this pass. Frequently reevaluate the default and set it to the combination and sequence of modules observed to work best for most workloads. The maintainer has observed that two glossary passes help and some workloads benefit from three or more; these observations inform the current choice without fixing a permanent number or order of passes. Preserve operators' ability to configure additional passes. Clarify the complementary requested roles: glossary stages focus on known terms supported by selected glossaries; homophone stages look more broadly for plausible spoken-word mistranscriptions and may also use glossary evidence. Do not exclude an otherwise supported homophone correction merely because it targets a glossary term. No module consolidation is planned in this pass. Reevaluating defaults does not depend on the deferred formal evaluation work in topic 5. 3. **Language and examples — Decided:** retain English as the bundled default. Keep useful existing campaign examples and add a few more general examples during the prompt editing pass. Use broadly applicable wording for names, terminology, and glossary entries, without asserting D&D specificity or adding a disclaimer about it. Broader language support is outside this pass. 4. **Meaning-reversal policy:** the guard can also exclude a genuine correction of a mistranscription. Initially retain the conservative default while explicitly evaluating that tradeoff. The remaining meaning-reversal item is a product discussion, not a selected change to validation policy.