Plan prompt revisions ahead of evaluation evidence

This commit is contained in:
2026-11-11 15:33:08 +00:00
parent 6a843481d2
commit 5aca3ebccf

View File

@@ -2,8 +2,8 @@
**Updated:** 2026-11-11
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
pending; topics 4–6 remain for discussion.
**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation
pending; topics 5–6 remain for discussion.
Work through the remaining topics in order. Their recommendations are starting
points for discussion, not newly adopted restrictions or guarantees.
@@ -100,18 +100,33 @@ Removing the proposal threshold can admit corrections it previously rejected.
Use the evaluation work in topic 5 to assess that effect; removal is the selected
direction, not conditional on a further confidence-policy decision.
### 4. Simplify Instructions and Improve Examples
### 4. Simplify Instructions and Improve Examples — Decided
Restrictions and examples recur across proposal, validation, and repair assets;
the grammar prompt repeats its word-substitution boundary several ways.
**Recommendation:** distinguish shared editing principles, requested module focus,
validator acceptance criteria, and task-specific response contracts. Use small
shared fragments where they reduce drift. Replace some repetitive example lists
with contextual contrasts, such as
accidental repetition versus deliberate emphasis. Preserve useful distinctions;
shorter prompts alone are not the objective. Discuss what belongs in shared
fragments and what remains local to each definition.
**Decision:** make a focused editing pass using the existing module and prompt
fragment structure. Organize instructions around shared contracts (including
response format, literal matching, and deletion semantics), module-specific
requested work, and the distinct responsibilities of validation and repair.
Consolidate duplicated rules where small shared fragments prevent drift, remove
contradictions and unnecessary repetition, and keep each rendered prompt
understandable on its own. Retain repetition across separate model calls when
each call needs the instruction.
Use short contextual contrasts to clarify difficult judgments, such as an
accidental restart versus deliberate emphasis, or a glossary match supported by
context versus a merely similar spelling. Examples should illustrate how context
informs the decision, not create unconditional editing rules. Preserve the
distinction between requested module focus and acceptance of otherwise appropriate
minor editorial cleanup. Shorter prompts alone are not the objective.
**Remaining work:** revise the proposal, validation, and repair assets and shared
fragments consistently with topics 1–3, retaining the existing module structure.
Check rendering, response-schema compatibility, and integration behavior as
appropriate. Proceed with these revisions before waiting for live-model
evaluation evidence or a captured baseline. Topic 5 provides subsequent quality
assessment and refinement; it is not a prerequisite for this editing pass.
### 5. Evaluate Editorial Quality
@@ -127,10 +142,12 @@ editorial assessment; validator approval alone is insufficient evidence.
Include a comparison against the prior confidence-threshold behavior to identify
changes in both useful corrections and erroneous edits after confidence removal.
Capture candidate examples during topics 1–4 and establish a baseline before
changing prompts. Discuss representative models, expected outcomes, and how to
run evaluations. Keep model-dependent evaluation separate from the default
offline test suite; no numerical quality gate is selected here.
Capture candidate examples during topics 1–4 where useful, without delaying the
agreed prompt revisions. Discuss representative models, expected outcomes, and
how to run evaluations as follow-up work. Use the prior repository revisions for
before/after comparisons when that work proceeds; a pre-change evaluation run is
not required. Keep model-dependent evaluation separate from the default offline
test suite; no numerical quality gate is selected here.
### 6. Revisit Broader Product Choices