Files
audita/docs/roadmap/prompt-quality.md

13 KiB

Prompt Quality Roadmap

Updated: 2026-11-11

Status: Decisions settled; implementation pending. Editorial evaluation is deferred. No open decisions remain in the current scope.

Target State and Constraints

Audita will use clearer proposal, validation, and repair prompts, actionable rejection feedback, and no model-reported confidence scores or thresholds. Repair will reconsider corrections on their merits and revise or withdraw them based on the supplied evidence. The initial review covered the eleven prompt definitions and their consumers at revision 6d34602; it was a design review, not a live-model quality measurement.

Preserve the following throughout implementation:

  • Editorial acceptance: module focus guides requested work, while validators permit otherwise acceptable minor editorial cleanup across modules. Follow ADR 0003, which records the maintainer's finding that strict scope instructions also suppress desirable edits. Applicability, protection, and other substantive checks still apply.
  • Correction mechanics: retain literal-span and deletion semantics, distinct glossary context and protection, full before/after validation, and independent repair conversations with full history, withdrawal, and revalidation. Preserve native PromptKit/ConfigKit prompt replacement and the existing module structure.
  • Module behavior: keep grammar and its current requested scope; retain glossary and homophone modules with their useful overlap. Keep the current default sequence for this pass and preserve configurable additional passes.
  • Language: retain English as the bundled default and useful existing campaign examples, supplemented with a few general examples. Broader language support is outside this pass.
  • Meaning reversal: preserve the narrowly tuned guard against reversal of the full segment's assertion. Its targeted prompt changes are specified below.

Nearly all module behavior reflects the maintainer's practical experimentation. Two glossary passes have proved useful, and some workloads benefit from three or more. These observations inform the current choice without fixing a permanent number or order of passes. Frequently reevaluate the default module combination and sequence and select what has been observed to work best for most workloads. This ongoing maintenance does not depend on the deferred formal evaluation work.

Implementation Sequence

Work through the following steps in order. Keep schemas, consumers, prompt examples, and affected tests consistent as each change lands, and update the canonical documentation with the corresponding behavior. Step 5 checks the assembled result; step 6 reviews test-policy compliance. Apply that policy during the earlier steps as well. Proceed without waiting for live-model evaluation evidence or a captured baseline.

1. Remove Confidence Throughout the Workflow

Remove confidence entirely from proposals, validation, and repair. Audita has no established calibration that gives its model-reported scores reliable numerical meaning across transcripts and models. Proposal scores currently affect acceptance; validator scores are range-checked but do not determine approval. Retaining the numbers adds complexity and encourages repair toward a threshold.

  • Remove confidence fields from response schemas, Go models, decoding, and validation paths, together with corresponding prompt instructions and examples.
  • Remove the confidence-threshold validator, its registration and chain entries, root and stage threshold settings, and any corresponding CLI exposure.
  • Remove confidence-specific rejection feedback, reason codes, report/debug fields, and other supporting code wherever present. Update affected tests, fixtures, and configuration examples.
  • Treat removal as a deliberate contract change for configuration, custom prompts, and affected serialized outputs. Retain no inert confidence fields or threshold knobs, and introduce no replacement numerical or categorical confidence scale.

Removing the proposal threshold can admit corrections it previously rejected. That effect can be assessed when editorial evaluation resumes; it does not block the selected removal.

2. Improve Validation Feedback

Retain the existing reason field. Update shared validation guidance and all three validator definitions to request concise rejection explanations identifying the specific problematic edit, applicable criterion, and relevant supplied evidence within each validator's remit. For editorial review, for example: removing "might" turns a tentative statement into a definite one.

Validators identify defects; the repair module owns revision or withdrawal. Distinguish insufficient evidence from an established defect, respect the accepted editorial allowance, and include distinct material issues when found without requiring exhaustive analysis. Provide representative rejection examples.

Approvals need no explanation: instruct the model to return reason: "" and use that convention in approval examples. The field remains required and permits an empty string; this reason convention itself needs no schema or response-handling change. Explanation guidance is expressed in prompts, with no new runtime rule rejecting blank explanations. Consider schema expansion only if a concrete need emerges.

The meaning-reversal validator requires particular care. The maintainer has observed overly broad instructions causing it to reject virtually all corrections. Preserve its core instructions and make only small, targeted feedback and example changes, alongside confidence removal and the approval-reason convention:

  • Detect reversal of the full segment's assertion, not every change in meaning. Changes to names, referents, or other information alone are not grounds for rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change that is not a reversal.
  • Assess the full before/after segment. An antonym appearing in the replacement does not by itself establish reversal.
  • Instruct rejection feedback to identify the specific opposition in the assertions; "this changes the meaning" is insufficient.
  • Approve corrections outside this narrow rejection criterion, leaving other validators to assess their own criteria. Preserve the guard even though it can exclude a genuine mistranscription correction that reverses the assertion; no contextual exception is selected in this pass.

3. Refine Repair Guidance

Ask the model to reassess the correction using the original transcript, supplied context, and rejection feedback:

  • Revise when the evidence supports a correction that addresses the identified problems.
  • Withdraw when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal is a successful resolution that preserves the original text.

Use concrete explanations of uncertainty where relevant, such as context that cannot distinguish two possible names. Preserve requested module focus and the accepted editorial allowance. Apply these instructions within the existing full-history repair lifecycle and revalidation, without adding new retry rules.

4. Consolidate Prompts, Descriptions, and Examples

Make a focused editing pass using the existing prompt fragment and module structure. Organize instructions around shared contracts (including response format, literal matching, and deletion semantics), module-specific requested work, and the distinct responsibilities of validation and repair.

Consolidate duplicated rules where small shared fragments prevent drift, remove contradictions and unnecessary repetition, and keep each rendered prompt understandable on its own. Retain repetition across separate model calls when each call needs the instruction. Preserve the meaning-reversal validator's tuned core wording as specified in step 2; broad simplification for consistency is not appropriate there. Shorter prompts alone are not the objective.

Clarify descriptions and examples as follows:

  • Keep the grammar name, module key, and prompt IDs. Describe its requested work as conservative punctuation, capitalization, spacing, and whole-word a/an article cleanup. Preserve the current scope and cross-module editorial allowance.
  • Describe glossary stages as focusing on known terms supported by selected glossaries, and homophone stages as looking more broadly for plausible spoken-word mistranscriptions, including glossary-supported corrections. Do not exclude an otherwise supported homophone correction because it targets a glossary term.
  • Use short contextual contrasts for difficult judgments, such as accidental restarts versus deliberate emphasis and context-supported glossary matches versus merely similar spellings. Examples illustrate judgments rather than establish unconditional editing rules.
  • Retain useful campaign examples and add a few general examples. Use broadly applicable wording for names, terminology, and glossary entries without asserting D&D specificity or adding a disclaimer about it.

5. Complete Integration Checks and Documentation

Follow the testing policy and the development validation guidance. Verify the assembled workflow, including:

  • Confidence-free proposal, validation, and repair responses work through their schemas and consumers; substantive validators remain effective in the workflow.
  • Rendered prompts and examples match the revised contracts, including empty approval reasons and rejection feedback carried into repair requests.
  • Revision, withdrawal, and full revalidation retain their existing mechanics.

Review the actual prompt wording and representative repair payloads to check the selected responsibilities and the narrow meaning-reversal criterion. Offline tests verify contracts and integration; they do not establish live-model editorial quality. Complete the appropriate package checks, build, and full test suite for the changes, without introducing paid calls into the default suite.

Update canonical configuration, CLI, prompt, module, validator, and output documentation where affected, together with maintained examples. Provide migration guidance to remove obsolete confidence settings and fields from configuration, custom prompts, and affected output consumers. Check links, examples, and whitespace, and remove stale descriptions of confidence-based behavior.

6. Review Tests for Policy Compliance

Review existing and newly changed tests supporting this work against the testing policy, with particular attention to prompt tests. Each retained test should protect a meaningful contract or behavior against a realistic defect at a maintenance cost justified by that protection.

Prompt wording checks carry an unusually high cost: assertions that a particular word or phrase appears in a prompt body obstruct development and experimentation with alternative language without establishing that a model follows the intended instruction. Remove these change-detector tests rather than updating their expected phrases to match the revised prompts. Apply the same scrutiny to prompt body snapshots, exact prose comparisons, and equivalent checks hidden behind helpers. Do not add new tests that freeze incidental prompt wording, including the carefully tuned meaning-reversal instructions.

Retain or replace tests where they protect meaningful behavior, such as template rendering with supplied data, prompt selection and operator replacement, response schema compatibility, repair context assembly, and processing of validation and repair responses. Exact comparisons of test-controlled data or explicit machine contracts can be appropriate; distinguish those from assertions about bundled instruction prose. Replace a removed test only when a genuine coverage gap justifies it, and consolidate redundant coverage.

Review prompt language directly for the agreed editorial intent. Keyword checks are not a substitute for that review or for the deferred live-model evaluation. Summarize the tests removed, revised, and retained with their behavioral rationale, and run the checks appropriate to any test changes made in this final stage.

Deferred Editorial Evaluation

Evaluation corpus preparation, live-model comparisons, and an evaluation workflow are outside this implementation scope. No baseline run or numerical quality gate is required before the agreed changes proceed.

When this work resumes, consider a modest collection of needed corrections and text that should remain unchanged: ambiguous glossary matches, intentional repetition, meaningful apparent filler, punctuation affecting meaning, and repairs that should be withdrawn. Compare unnecessary edits, missed corrections, validator disagreement, and repair outcomes using independent editorial judgment; validator approval alone is insufficient evidence.

Discuss representative models, expected outcomes, and an evaluation workflow at that time. Prior repository revisions can support before/after comparisons, including the effects of removing confidence thresholds. Keep model-dependent evaluation separate from the default offline test suite.