Plan prompt revisions ahead of evaluation evidence
This commit is contained in:
@@ -2,8 +2,8 @@
|
|||||||
|
|
||||||
**Updated:** 2026-11-11
|
**Updated:** 2026-11-11
|
||||||
|
|
||||||
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
|
**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation
|
||||||
pending; topics 4–6 remain for discussion.
|
pending; topics 5–6 remain for discussion.
|
||||||
Work through the remaining topics in order. Their recommendations are starting
|
Work through the remaining topics in order. Their recommendations are starting
|
||||||
points for discussion, not newly adopted restrictions or guarantees.
|
points for discussion, not newly adopted restrictions or guarantees.
|
||||||
|
|
||||||
@@ -100,18 +100,33 @@ Removing the proposal threshold can admit corrections it previously rejected.
|
|||||||
Use the evaluation work in topic 5 to assess that effect; removal is the selected
|
Use the evaluation work in topic 5 to assess that effect; removal is the selected
|
||||||
direction, not conditional on a further confidence-policy decision.
|
direction, not conditional on a further confidence-policy decision.
|
||||||
|
|
||||||
### 4. Simplify Instructions and Improve Examples
|
### 4. Simplify Instructions and Improve Examples — Decided
|
||||||
|
|
||||||
Restrictions and examples recur across proposal, validation, and repair assets;
|
Restrictions and examples recur across proposal, validation, and repair assets;
|
||||||
the grammar prompt repeats its word-substitution boundary several ways.
|
the grammar prompt repeats its word-substitution boundary several ways.
|
||||||
|
|
||||||
**Recommendation:** distinguish shared editing principles, requested module focus,
|
**Decision:** make a focused editing pass using the existing module and prompt
|
||||||
validator acceptance criteria, and task-specific response contracts. Use small
|
fragment structure. Organize instructions around shared contracts (including
|
||||||
shared fragments where they reduce drift. Replace some repetitive example lists
|
response format, literal matching, and deletion semantics), module-specific
|
||||||
with contextual contrasts, such as
|
requested work, and the distinct responsibilities of validation and repair.
|
||||||
accidental repetition versus deliberate emphasis. Preserve useful distinctions;
|
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
||||||
shorter prompts alone are not the objective. Discuss what belongs in shared
|
contradictions and unnecessary repetition, and keep each rendered prompt
|
||||||
fragments and what remains local to each definition.
|
understandable on its own. Retain repetition across separate model calls when
|
||||||
|
each call needs the instruction.
|
||||||
|
|
||||||
|
Use short contextual contrasts to clarify difficult judgments, such as an
|
||||||
|
accidental restart versus deliberate emphasis, or a glossary match supported by
|
||||||
|
context versus a merely similar spelling. Examples should illustrate how context
|
||||||
|
informs the decision, not create unconditional editing rules. Preserve the
|
||||||
|
distinction between requested module focus and acceptance of otherwise appropriate
|
||||||
|
minor editorial cleanup. Shorter prompts alone are not the objective.
|
||||||
|
|
||||||
|
**Remaining work:** revise the proposal, validation, and repair assets and shared
|
||||||
|
fragments consistently with topics 1–3, retaining the existing module structure.
|
||||||
|
Check rendering, response-schema compatibility, and integration behavior as
|
||||||
|
appropriate. Proceed with these revisions before waiting for live-model
|
||||||
|
evaluation evidence or a captured baseline. Topic 5 provides subsequent quality
|
||||||
|
assessment and refinement; it is not a prerequisite for this editing pass.
|
||||||
|
|
||||||
### 5. Evaluate Editorial Quality
|
### 5. Evaluate Editorial Quality
|
||||||
|
|
||||||
@@ -127,10 +142,12 @@ editorial assessment; validator approval alone is insufficient evidence.
|
|||||||
Include a comparison against the prior confidence-threshold behavior to identify
|
Include a comparison against the prior confidence-threshold behavior to identify
|
||||||
changes in both useful corrections and erroneous edits after confidence removal.
|
changes in both useful corrections and erroneous edits after confidence removal.
|
||||||
|
|
||||||
Capture candidate examples during topics 1–4 and establish a baseline before
|
Capture candidate examples during topics 1–4 where useful, without delaying the
|
||||||
changing prompts. Discuss representative models, expected outcomes, and how to
|
agreed prompt revisions. Discuss representative models, expected outcomes, and
|
||||||
run evaluations. Keep model-dependent evaluation separate from the default
|
how to run evaluations as follow-up work. Use the prior repository revisions for
|
||||||
offline test suite; no numerical quality gate is selected here.
|
before/after comparisons when that work proceeds; a pre-change evaluation run is
|
||||||
|
not required. Keep model-dependent evaluation separate from the default offline
|
||||||
|
test suite; no numerical quality gate is selected here.
|
||||||
|
|
||||||
### 6. Revisit Broader Product Choices
|
### 6. Revisit Broader Product Choices
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user