Plan prompt revisions ahead of evaluation evidence

This commit is contained in:
2026-11-11 15:33:08 +00:00
parent 6a843481d2
commit 5aca3ebccf

View File

@@ -2,8 +2,8 @@
**Updated:** 2026-11-11 **Updated:** 2026-11-11
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation **Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation
pending; topics 4–6 remain for discussion. pending; topics 5–6 remain for discussion.
Work through the remaining topics in order. Their recommendations are starting Work through the remaining topics in order. Their recommendations are starting
points for discussion, not newly adopted restrictions or guarantees. points for discussion, not newly adopted restrictions or guarantees.
@@ -100,18 +100,33 @@ Removing the proposal threshold can admit corrections it previously rejected.
Use the evaluation work in topic 5 to assess that effect; removal is the selected Use the evaluation work in topic 5 to assess that effect; removal is the selected
direction, not conditional on a further confidence-policy decision. direction, not conditional on a further confidence-policy decision.
### 4. Simplify Instructions and Improve Examples ### 4. Simplify Instructions and Improve Examples — Decided
Restrictions and examples recur across proposal, validation, and repair assets; Restrictions and examples recur across proposal, validation, and repair assets;
the grammar prompt repeats its word-substitution boundary several ways. the grammar prompt repeats its word-substitution boundary several ways.
**Recommendation:** distinguish shared editing principles, requested module focus, **Decision:** make a focused editing pass using the existing module and prompt
validator acceptance criteria, and task-specific response contracts. Use small fragment structure. Organize instructions around shared contracts (including
shared fragments where they reduce drift. Replace some repetitive example lists response format, literal matching, and deletion semantics), module-specific
with contextual contrasts, such as requested work, and the distinct responsibilities of validation and repair.
accidental repetition versus deliberate emphasis. Preserve useful distinctions; Consolidate duplicated rules where small shared fragments prevent drift, remove
shorter prompts alone are not the objective. Discuss what belongs in shared contradictions and unnecessary repetition, and keep each rendered prompt
fragments and what remains local to each definition. understandable on its own. Retain repetition across separate model calls when
each call needs the instruction.
Use short contextual contrasts to clarify difficult judgments, such as an
accidental restart versus deliberate emphasis, or a glossary match supported by
context versus a merely similar spelling. Examples should illustrate how context
informs the decision, not create unconditional editing rules. Preserve the
distinction between requested module focus and acceptance of otherwise appropriate
minor editorial cleanup. Shorter prompts alone are not the objective.
**Remaining work:** revise the proposal, validation, and repair assets and shared
fragments consistently with topics 1–3, retaining the existing module structure.
Check rendering, response-schema compatibility, and integration behavior as
appropriate. Proceed with these revisions before waiting for live-model
evaluation evidence or a captured baseline. Topic 5 provides subsequent quality
assessment and refinement; it is not a prerequisite for this editing pass.
### 5. Evaluate Editorial Quality ### 5. Evaluate Editorial Quality
@@ -127,10 +142,12 @@ editorial assessment; validator approval alone is insufficient evidence.
Include a comparison against the prior confidence-threshold behavior to identify Include a comparison against the prior confidence-threshold behavior to identify
changes in both useful corrections and erroneous edits after confidence removal. changes in both useful corrections and erroneous edits after confidence removal.
Capture candidate examples during topics 1–4 and establish a baseline before Capture candidate examples during topics 1–4 where useful, without delaying the
changing prompts. Discuss representative models, expected outcomes, and how to agreed prompt revisions. Discuss representative models, expected outcomes, and
run evaluations. Keep model-dependent evaluation separate from the default how to run evaluations as follow-up work. Use the prior repository revisions for
offline test suite; no numerical quality gate is selected here. before/after comparisons when that work proceeds; a pre-change evaluation run is
not required. Keep model-dependent evaluation separate from the default offline
test suite; no numerical quality gate is selected here.
### 6. Revisit Broader Product Choices ### 6. Revisit Broader Product Choices