diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index da7090e..a3f5afd 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -46,9 +46,10 @@ This ongoing maintenance does not depend on the deferred formal evaluation work. Work through the following steps in order. Keep schemas, consumers, prompt examples, and affected tests consistent as each change lands, and update the -canonical documentation with the corresponding behavior. The final step checks -the assembled result. Proceed without waiting for live-model evaluation evidence -or a captured baseline. +canonical documentation with the corresponding behavior. Step 5 checks the +assembled result; step 6 reviews test-policy compliance. Apply that policy during +the earlier steps as well. Proceed without waiting for live-model evaluation +evidence or a captured baseline. ### 1. Remove Confidence Throughout the Workflow @@ -185,6 +186,35 @@ settings and fields from configuration, custom prompts, and affected output consumers. Check links, examples, and whitespace, and remove stale descriptions of confidence-based behavior. +### 6. Review Tests for Policy Compliance + +Review existing and newly changed tests supporting this work against the +[testing policy](../policy/testing.md), with particular attention to prompt tests. +Each retained test should protect a meaningful contract or behavior against a +realistic defect at a maintenance cost justified by that protection. + +Prompt wording checks carry an unusually high cost: assertions that a particular +word or phrase appears in a prompt body obstruct development and experimentation +with alternative language without establishing that a model follows the intended +instruction. Remove these change-detector tests rather than updating their +expected phrases to match the revised prompts. Apply the same scrutiny to prompt +body snapshots, exact prose comparisons, and equivalent checks hidden behind +helpers. Do not add new tests that freeze incidental prompt wording, including +the carefully tuned meaning-reversal instructions. + +Retain or replace tests where they protect meaningful behavior, such as template +rendering with supplied data, prompt selection and operator replacement, response +schema compatibility, repair context assembly, and processing of validation and +repair responses. Exact comparisons of test-controlled data or explicit machine +contracts can be appropriate; distinguish those from assertions about bundled +instruction prose. Replace a removed test only when a genuine coverage gap +justifies it, and consolidate redundant coverage. + +Review prompt language directly for the agreed editorial intent. Keyword checks +are not a substitute for that review or for the deferred live-model evaluation. +Summarize the tests removed, revised, and retained with their behavioral rationale, +and run the checks appropriate to any test changes made in this final stage. + ## Deferred Editorial Evaluation Evaluation corpus preparation, live-model comparisons, and an evaluation workflow