diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index ace269b..7016ad3 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -3,7 +3,7 @@ **Updated:** 2026-11-11 **Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation -pending; topics 5–6 remain for discussion. +pending; topic 5 deferred; topic 6 remains for discussion. Work through the remaining topics in order. Their recommendations are starting points for discussion, not newly adopted restrictions or guarantees. @@ -97,8 +97,8 @@ the existing repair lifecycle and revalidation. and revalidation continue to operate as intended. Removing the proposal threshold can admit corrections it previously rejected. -Use the evaluation work in topic 5 to assess that effect; removal is the selected -direction, not conditional on a further confidence-policy decision. +The deferred evaluation work in topic 5 can assess that effect when resumed; +removal is the selected direction and does not depend on that work. ### 4. Simplify Instructions and Improve Examples — Decided @@ -125,15 +125,21 @@ minor editorial cleanup. Shorter prompts alone are not the objective. fragments consistently with topics 1–3, retaining the existing module structure. Check rendering, response-schema compatibility, and integration behavior as appropriate. Proceed with these revisions before waiting for live-model -evaluation evidence or a captured baseline. Topic 5 provides subsequent quality -assessment and refinement; it is not a prerequisite for this editing pass. +evaluation evidence or a captured baseline. Topic 5 is deferred and is not a +prerequisite for this editing pass. -### 5. Evaluate Editorial Quality +### 5. Evaluate Editorial Quality — Deferred + +**Decision:** defer this work for now. Evaluation corpus preparation, live-model +comparisons, and an evaluation workflow are outside the current implementation +scope. This deferral does not block topics 2–4 or the remaining product +discussions; ordinary implementation checks still apply. The following direction +is retained for reconsideration when evaluation work resumes. Rendering, schema, and controlled-response tests establish integration behavior, but do not establish the quality of model editorial judgments. -**Recommendation:** assemble a modest evaluation set containing both needed edits +**Future recommendation:** assemble a modest evaluation set containing both needed edits and text that should remain unchanged. Include ambiguous glossary matches, intentional repetition, meaningful uses of apparent filler, punctuation that changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, @@ -142,16 +148,16 @@ editorial assessment; validator approval alone is insufficient evidence. Include a comparison against the prior confidence-threshold behavior to identify changes in both useful corrections and erroneous edits after confidence removal. -Capture candidate examples during topics 1–4 where useful, without delaying the -agreed prompt revisions. Discuss representative models, expected outcomes, and -how to run evaluations as follow-up work. Use the prior repository revisions for +When this work resumes, discuss representative models, expected outcomes, and +how to run evaluations. Use the prior repository revisions for before/after comparisons when that work proceeds; a pre-change evaluation run is not required. Keep model-dependent evaluation separate from the default offline test suite; no numerical quality gate is selected here. ### 6. Revisit Broader Product Choices -Use the preceding decisions and evaluation evidence to discuss these in order: +Use the preceding decisions and any available evidence to discuss these in order, +without waiting for the deferred evaluation work: 1. **Grammar naming and scope:** the current module is close to copy editing, with an isolated article exception. Decide whether its name and scope convey