Add final test policy review to prompt quality roadmap
This commit is contained in:
@@ -46,9 +46,10 @@ This ongoing maintenance does not depend on the deferred formal evaluation work.
|
|||||||
|
|
||||||
Work through the following steps in order. Keep schemas, consumers, prompt
|
Work through the following steps in order. Keep schemas, consumers, prompt
|
||||||
examples, and affected tests consistent as each change lands, and update the
|
examples, and affected tests consistent as each change lands, and update the
|
||||||
canonical documentation with the corresponding behavior. The final step checks
|
canonical documentation with the corresponding behavior. Step 5 checks the
|
||||||
the assembled result. Proceed without waiting for live-model evaluation evidence
|
assembled result; step 6 reviews test-policy compliance. Apply that policy during
|
||||||
or a captured baseline.
|
the earlier steps as well. Proceed without waiting for live-model evaluation
|
||||||
|
evidence or a captured baseline.
|
||||||
|
|
||||||
### 1. Remove Confidence Throughout the Workflow
|
### 1. Remove Confidence Throughout the Workflow
|
||||||
|
|
||||||
@@ -185,6 +186,35 @@ settings and fields from configuration, custom prompts, and affected output
|
|||||||
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
||||||
of confidence-based behavior.
|
of confidence-based behavior.
|
||||||
|
|
||||||
|
### 6. Review Tests for Policy Compliance
|
||||||
|
|
||||||
|
Review existing and newly changed tests supporting this work against the
|
||||||
|
[testing policy](../policy/testing.md), with particular attention to prompt tests.
|
||||||
|
Each retained test should protect a meaningful contract or behavior against a
|
||||||
|
realistic defect at a maintenance cost justified by that protection.
|
||||||
|
|
||||||
|
Prompt wording checks carry an unusually high cost: assertions that a particular
|
||||||
|
word or phrase appears in a prompt body obstruct development and experimentation
|
||||||
|
with alternative language without establishing that a model follows the intended
|
||||||
|
instruction. Remove these change-detector tests rather than updating their
|
||||||
|
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
|
||||||
|
body snapshots, exact prose comparisons, and equivalent checks hidden behind
|
||||||
|
helpers. Do not add new tests that freeze incidental prompt wording, including
|
||||||
|
the carefully tuned meaning-reversal instructions.
|
||||||
|
|
||||||
|
Retain or replace tests where they protect meaningful behavior, such as template
|
||||||
|
rendering with supplied data, prompt selection and operator replacement, response
|
||||||
|
schema compatibility, repair context assembly, and processing of validation and
|
||||||
|
repair responses. Exact comparisons of test-controlled data or explicit machine
|
||||||
|
contracts can be appropriate; distinguish those from assertions about bundled
|
||||||
|
instruction prose. Replace a removed test only when a genuine coverage gap
|
||||||
|
justifies it, and consolidate redundant coverage.
|
||||||
|
|
||||||
|
Review prompt language directly for the agreed editorial intent. Keyword checks
|
||||||
|
are not a substitute for that review or for the deferred live-model evaluation.
|
||||||
|
Summarize the tests removed, revised, and retained with their behavioral rationale,
|
||||||
|
and run the checks appropriate to any test changes made in this final stage.
|
||||||
|
|
||||||
## Deferred Editorial Evaluation
|
## Deferred Editorial Evaluation
|
||||||
|
|
||||||
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
||||||
|
|||||||
Reference in New Issue
Block a user