From 5aca3ebccf69a5b5d767a089a12c4251f1057ea9 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 15:33:08 +0000 Subject: [PATCH] Plan prompt revisions ahead of evaluation evidence --- docs/roadmap/prompt-quality.md | 45 +++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index f564bb8..ace269b 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -2,8 +2,8 @@ **Updated:** 2026-11-11 -**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation -pending; topics 4–6 remain for discussion. +**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation +pending; topics 5–6 remain for discussion. Work through the remaining topics in order. Their recommendations are starting points for discussion, not newly adopted restrictions or guarantees. @@ -100,18 +100,33 @@ Removing the proposal threshold can admit corrections it previously rejected. Use the evaluation work in topic 5 to assess that effect; removal is the selected direction, not conditional on a further confidence-policy decision. -### 4. Simplify Instructions and Improve Examples +### 4. Simplify Instructions and Improve Examples — Decided Restrictions and examples recur across proposal, validation, and repair assets; the grammar prompt repeats its word-substitution boundary several ways. -**Recommendation:** distinguish shared editing principles, requested module focus, -validator acceptance criteria, and task-specific response contracts. Use small -shared fragments where they reduce drift. Replace some repetitive example lists -with contextual contrasts, such as -accidental repetition versus deliberate emphasis. Preserve useful distinctions; -shorter prompts alone are not the objective. Discuss what belongs in shared -fragments and what remains local to each definition. +**Decision:** make a focused editing pass using the existing module and prompt +fragment structure. Organize instructions around shared contracts (including +response format, literal matching, and deletion semantics), module-specific +requested work, and the distinct responsibilities of validation and repair. +Consolidate duplicated rules where small shared fragments prevent drift, remove +contradictions and unnecessary repetition, and keep each rendered prompt +understandable on its own. Retain repetition across separate model calls when +each call needs the instruction. + +Use short contextual contrasts to clarify difficult judgments, such as an +accidental restart versus deliberate emphasis, or a glossary match supported by +context versus a merely similar spelling. Examples should illustrate how context +informs the decision, not create unconditional editing rules. Preserve the +distinction between requested module focus and acceptance of otherwise appropriate +minor editorial cleanup. Shorter prompts alone are not the objective. + +**Remaining work:** revise the proposal, validation, and repair assets and shared +fragments consistently with topics 1–3, retaining the existing module structure. +Check rendering, response-schema compatibility, and integration behavior as +appropriate. Proceed with these revisions before waiting for live-model +evaluation evidence or a captured baseline. Topic 5 provides subsequent quality +assessment and refinement; it is not a prerequisite for this editing pass. ### 5. Evaluate Editorial Quality @@ -127,10 +142,12 @@ editorial assessment; validator approval alone is insufficient evidence. Include a comparison against the prior confidence-threshold behavior to identify changes in both useful corrections and erroneous edits after confidence removal. -Capture candidate examples during topics 1–4 and establish a baseline before -changing prompts. Discuss representative models, expected outcomes, and how to -run evaluations. Keep model-dependent evaluation separate from the default -offline test suite; no numerical quality gate is selected here. +Capture candidate examples during topics 1–4 where useful, without delaying the +agreed prompt revisions. Discuss representative models, expected outcomes, and +how to run evaluations as follow-up work. Use the prior repository revisions for +before/after comparisons when that work proceeds; a pre-change evaluation run is +not required. Keep model-dependent evaluation separate from the default offline +test suite; no numerical quality gate is selected here. ### 6. Revisit Broader Product Choices