Plan prompt revisions ahead of evaluation evidence
This commit is contained in:
@@ -2,8 +2,8 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
|
||||
pending; topics 4–6 remain for discussion.
|
||||
**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation
|
||||
pending; topics 5–6 remain for discussion.
|
||||
Work through the remaining topics in order. Their recommendations are starting
|
||||
points for discussion, not newly adopted restrictions or guarantees.
|
||||
|
||||
@@ -100,18 +100,33 @@ Removing the proposal threshold can admit corrections it previously rejected.
|
||||
Use the evaluation work in topic 5 to assess that effect; removal is the selected
|
||||
direction, not conditional on a further confidence-policy decision.
|
||||
|
||||
### 4. Simplify Instructions and Improve Examples
|
||||
### 4. Simplify Instructions and Improve Examples — Decided
|
||||
|
||||
Restrictions and examples recur across proposal, validation, and repair assets;
|
||||
the grammar prompt repeats its word-substitution boundary several ways.
|
||||
|
||||
**Recommendation:** distinguish shared editing principles, requested module focus,
|
||||
validator acceptance criteria, and task-specific response contracts. Use small
|
||||
shared fragments where they reduce drift. Replace some repetitive example lists
|
||||
with contextual contrasts, such as
|
||||
accidental repetition versus deliberate emphasis. Preserve useful distinctions;
|
||||
shorter prompts alone are not the objective. Discuss what belongs in shared
|
||||
fragments and what remains local to each definition.
|
||||
**Decision:** make a focused editing pass using the existing module and prompt
|
||||
fragment structure. Organize instructions around shared contracts (including
|
||||
response format, literal matching, and deletion semantics), module-specific
|
||||
requested work, and the distinct responsibilities of validation and repair.
|
||||
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
||||
contradictions and unnecessary repetition, and keep each rendered prompt
|
||||
understandable on its own. Retain repetition across separate model calls when
|
||||
each call needs the instruction.
|
||||
|
||||
Use short contextual contrasts to clarify difficult judgments, such as an
|
||||
accidental restart versus deliberate emphasis, or a glossary match supported by
|
||||
context versus a merely similar spelling. Examples should illustrate how context
|
||||
informs the decision, not create unconditional editing rules. Preserve the
|
||||
distinction between requested module focus and acceptance of otherwise appropriate
|
||||
minor editorial cleanup. Shorter prompts alone are not the objective.
|
||||
|
||||
**Remaining work:** revise the proposal, validation, and repair assets and shared
|
||||
fragments consistently with topics 1–3, retaining the existing module structure.
|
||||
Check rendering, response-schema compatibility, and integration behavior as
|
||||
appropriate. Proceed with these revisions before waiting for live-model
|
||||
evaluation evidence or a captured baseline. Topic 5 provides subsequent quality
|
||||
assessment and refinement; it is not a prerequisite for this editing pass.
|
||||
|
||||
### 5. Evaluate Editorial Quality
|
||||
|
||||
@@ -127,10 +142,12 @@ editorial assessment; validator approval alone is insufficient evidence.
|
||||
Include a comparison against the prior confidence-threshold behavior to identify
|
||||
changes in both useful corrections and erroneous edits after confidence removal.
|
||||
|
||||
Capture candidate examples during topics 1–4 and establish a baseline before
|
||||
changing prompts. Discuss representative models, expected outcomes, and how to
|
||||
run evaluations. Keep model-dependent evaluation separate from the default
|
||||
offline test suite; no numerical quality gate is selected here.
|
||||
Capture candidate examples during topics 1–4 where useful, without delaying the
|
||||
agreed prompt revisions. Discuss representative models, expected outcomes, and
|
||||
how to run evaluations as follow-up work. Use the prior repository revisions for
|
||||
before/after comparisons when that work proceeds; a pre-change evaluation run is
|
||||
not required. Keep model-dependent evaluation separate from the default offline
|
||||
test suite; no numerical quality gate is selected here.
|
||||
|
||||
### 6. Revisit Broader Product Choices
|
||||
|
||||
|
||||
Reference in New Issue
Block a user