Add final test policy review to prompt quality roadmap
This commit is contained in:
@@ -46,9 +46,10 @@ This ongoing maintenance does not depend on the deferred formal evaluation work.
|
||||
|
||||
Work through the following steps in order. Keep schemas, consumers, prompt
|
||||
examples, and affected tests consistent as each change lands, and update the
|
||||
canonical documentation with the corresponding behavior. The final step checks
|
||||
the assembled result. Proceed without waiting for live-model evaluation evidence
|
||||
or a captured baseline.
|
||||
canonical documentation with the corresponding behavior. Step 5 checks the
|
||||
assembled result; step 6 reviews test-policy compliance. Apply that policy during
|
||||
the earlier steps as well. Proceed without waiting for live-model evaluation
|
||||
evidence or a captured baseline.
|
||||
|
||||
### 1. Remove Confidence Throughout the Workflow
|
||||
|
||||
@@ -185,6 +186,35 @@ settings and fields from configuration, custom prompts, and affected output
|
||||
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
||||
of confidence-based behavior.
|
||||
|
||||
### 6. Review Tests for Policy Compliance
|
||||
|
||||
Review existing and newly changed tests supporting this work against the
|
||||
[testing policy](../policy/testing.md), with particular attention to prompt tests.
|
||||
Each retained test should protect a meaningful contract or behavior against a
|
||||
realistic defect at a maintenance cost justified by that protection.
|
||||
|
||||
Prompt wording checks carry an unusually high cost: assertions that a particular
|
||||
word or phrase appears in a prompt body obstruct development and experimentation
|
||||
with alternative language without establishing that a model follows the intended
|
||||
instruction. Remove these change-detector tests rather than updating their
|
||||
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
|
||||
body snapshots, exact prose comparisons, and equivalent checks hidden behind
|
||||
helpers. Do not add new tests that freeze incidental prompt wording, including
|
||||
the carefully tuned meaning-reversal instructions.
|
||||
|
||||
Retain or replace tests where they protect meaningful behavior, such as template
|
||||
rendering with supplied data, prompt selection and operator replacement, response
|
||||
schema compatibility, repair context assembly, and processing of validation and
|
||||
repair responses. Exact comparisons of test-controlled data or explicit machine
|
||||
contracts can be appropriate; distinguish those from assertions about bundled
|
||||
instruction prose. Replace a removed test only when a genuine coverage gap
|
||||
justifies it, and consolidate redundant coverage.
|
||||
|
||||
Review prompt language directly for the agreed editorial intent. Keyword checks
|
||||
are not a substitute for that review or for the deferred live-model evaluation.
|
||||
Summarize the tests removed, revised, and retained with their behavioral rationale,
|
||||
and run the checks appropriate to any test changes made in this final stage.
|
||||
|
||||
## Deferred Editorial Evaluation
|
||||
|
||||
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
||||
|
||||
Reference in New Issue
Block a user