Add final test policy review to prompt quality roadmap

This commit is contained in:
2026-11-11 16:52:56 +00:00
parent 7c567d8d8b
commit bbf89bb4f5

View File

@@ -46,9 +46,10 @@ This ongoing maintenance does not depend on the deferred formal evaluation work.
Work through the following steps in order. Keep schemas, consumers, prompt Work through the following steps in order. Keep schemas, consumers, prompt
examples, and affected tests consistent as each change lands, and update the examples, and affected tests consistent as each change lands, and update the
canonical documentation with the corresponding behavior. The final step checks canonical documentation with the corresponding behavior. Step 5 checks the
the assembled result. Proceed without waiting for live-model evaluation evidence assembled result; step 6 reviews test-policy compliance. Apply that policy during
or a captured baseline. the earlier steps as well. Proceed without waiting for live-model evaluation
evidence or a captured baseline.
### 1. Remove Confidence Throughout the Workflow ### 1. Remove Confidence Throughout the Workflow
@@ -185,6 +186,35 @@ settings and fields from configuration, custom prompts, and affected output
consumers. Check links, examples, and whitespace, and remove stale descriptions consumers. Check links, examples, and whitespace, and remove stale descriptions
of confidence-based behavior. of confidence-based behavior.
### 6. Review Tests for Policy Compliance
Review existing and newly changed tests supporting this work against the
[testing policy](../policy/testing.md), with particular attention to prompt tests.
Each retained test should protect a meaningful contract or behavior against a
realistic defect at a maintenance cost justified by that protection.
Prompt wording checks carry an unusually high cost: assertions that a particular
word or phrase appears in a prompt body obstruct development and experimentation
with alternative language without establishing that a model follows the intended
instruction. Remove these change-detector tests rather than updating their
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
body snapshots, exact prose comparisons, and equivalent checks hidden behind
helpers. Do not add new tests that freeze incidental prompt wording, including
the carefully tuned meaning-reversal instructions.
Retain or replace tests where they protect meaningful behavior, such as template
rendering with supplied data, prompt selection and operator replacement, response
schema compatibility, repair context assembly, and processing of validation and
repair responses. Exact comparisons of test-controlled data or explicit machine
contracts can be appropriate; distinguish those from assertions about bundled
instruction prose. Replace a removed test only when a genuine coverage gap
justifies it, and consolidate redundant coverage.
Review prompt language directly for the agreed editorial intent. Keyword checks
are not a substitute for that review or for the deferred live-model evaluation.
Summarize the tests removed, revised, and retained with their behavioral rationale,
and run the checks appropriate to any test changes made in this final stage.
## Deferred Editorial Evaluation ## Deferred Editorial Evaluation
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow Evaluation corpus preparation, live-model comparisons, and an evaluation workflow