From d15d80c79f72982cd70ab460c4702d6637ad29a9 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 16:42:28 +0000 Subject: [PATCH] Preserve narrowly tuned meaning reversal validation --- docs/roadmap/prompt-quality.md | 44 ++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index 088db5a..a3fff9b 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -2,12 +2,9 @@ **Updated:** 2026-11-11 -**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation -pending; topic 5 deferred; grammar naming/scope, glossary/homophone overlap, -and language/examples in topic 6 decided, with description and example updates -pending; meaning-reversal policy awaits discussion. -Work through the remaining topics in order. Their recommendations are starting -points for discussion, not newly adopted restrictions or guarantees. +**Status:** Topic 1 resolved in ADR 0003; topics 2–4 and 6 decided, with +implementation pending; topic 5 deferred. No open decisions remain in the +current scope. Topic 5 retains recommendations for future discussion. ## Baseline and Intended Direction @@ -50,7 +47,8 @@ little guidance about producing actionable reasons. **Decision:** retain the existing `reason` field. Instruct validators to explain rejections with a concise description of the specific problematic edit, the applicable criterion, and relevant supplied evidence, within their own remit. -For example: removing "might" turns a tentative statement into a definite one. +For editorial review, for example: removing "might" turns a tentative statement +into a definite one. Validators identify the defect; the repair module owns revision or withdrawal. Respect the accepted minor editorial allowance, distinguish insufficient evidence from an established defect, and include distinct material issues when @@ -125,6 +123,11 @@ contradictions and unnecessary repetition, and keep each rendered prompt understandable on its own. Retain repetition across separate model calls when each call needs the instruction. +Apply the meaning-reversal decision in topic 6 as an exception to broad prompt +simplification: preserve that validator's tuned core instructions and make only +small, targeted changes to feedback and examples, alongside the agreed confidence +removal and approval-reason convention. + Use short contextual contrasts to clarify difficult judgments, such as an accidental restart versus deliberate emphasis, or a glossary match supported by context versus a merely similar spelling. Examples should illustrate how context @@ -167,8 +170,8 @@ test suite; no numerical quality gate is selected here. ### 6. Revisit Broader Product Choices -Use the preceding decisions and any available evidence to discuss these in order, -without waiting for the deferred evaluation work: +The following decisions guide this pass without waiting for the deferred +evaluation work: 1. **Grammar naming and scope — Decided:** retain the `grammar` name, module key, and corresponding prompt IDs. Clarify its description as conservative @@ -197,9 +200,20 @@ without waiting for the deferred evaluation work: during the prompt editing pass. Use broadly applicable wording for names, terminology, and glossary entries, without asserting D&D specificity or adding a disclaimer about it. Broader language support is outside this pass. -4. **Meaning-reversal policy:** the guard can also exclude a genuine correction - of a mistranscription. Initially retain the conservative default while - explicitly evaluating that tradeoff. - -The remaining meaning-reversal item is a product discussion, not a selected -change to validation policy. +4. **Meaning-reversal policy — Decided:** retain the narrow guard against reversal + of the full segment's assertion, rather than rejection of any change in + meaning. The maintainer has observed that overly broad instructions cause this + validator to reject virtually all corrections; its wording requires unusually + precise tuning. Preserve the existing core instructions and make small, + targeted changes, principally improving rejection feedback and adding a + contrasting example. Changes to names, referents, or other information are + not grounds for rejection by this validator unless they reverse the assertion: + "We met Alice" becoming "We met Alex" changes meaning but is not a reversal. + Assess the full before/after segment; the presence of an antonym in a + replacement does not by itself establish reversal. Instruct rejection feedback + to identify the specific opposition in the assertions; "this changes the + meaning" is insufficient. Approve corrections outside this narrow rejection + criterion, leaving other validators to assess their own criteria. These are + prompt instructions, not new runtime checks on explanation text. Preserve the + safeguard even though it can exclude a genuine mistranscription correction + that reverses the assertion; no contextual exception is selected in this pass.