235 lines
13 KiB
Markdown
235 lines
13 KiB
Markdown
# Prompt Quality Roadmap
|
|
|
|
**Updated:** 2026-11-11
|
|
|
|
**Status:** Decisions settled; implementation pending. Editorial evaluation is
|
|
deferred. No open decisions remain in the current scope.
|
|
|
|
## Target State and Constraints
|
|
|
|
Audita will use clearer proposal, validation, and repair prompts, actionable
|
|
rejection feedback, and no model-reported confidence scores or thresholds.
|
|
Repair will reconsider corrections on their merits and revise or withdraw them
|
|
based on the supplied evidence. The initial review covered the eleven
|
|
[prompt definitions](../internal/prompts.md) and their consumers at revision
|
|
`6d34602`; it was a design review, not a live-model quality measurement.
|
|
|
|
Preserve the following throughout implementation:
|
|
|
|
- **Editorial acceptance:** module focus guides requested work, while validators
|
|
permit otherwise acceptable minor editorial cleanup across modules. Follow
|
|
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records
|
|
the maintainer's finding that strict scope instructions also suppress desirable
|
|
edits. Applicability, protection, and other substantive checks still apply.
|
|
- **Correction mechanics:** retain literal-span and deletion semantics, distinct
|
|
glossary context and protection, full before/after validation, and independent
|
|
repair conversations with full history, withdrawal, and revalidation. Preserve
|
|
native PromptKit/ConfigKit prompt replacement and the existing module structure.
|
|
- **Module behavior:** keep `grammar` and its current requested scope; retain
|
|
glossary and homophone modules with their useful overlap. Keep the current
|
|
[default sequence](../config.md#pipeline-and-glossary-selections) for this pass
|
|
and preserve configurable additional passes.
|
|
- **Language:** retain English as the bundled default and useful existing campaign
|
|
examples, supplemented with a few general examples. Broader language support is
|
|
outside this pass.
|
|
- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the
|
|
full segment's assertion. Its targeted prompt changes are specified below.
|
|
|
|
Nearly all module behavior reflects the maintainer's practical experimentation.
|
|
Two glossary passes have proved useful, and some workloads benefit from three or
|
|
more. These observations inform the current choice without fixing a permanent
|
|
number or order of passes. Frequently reevaluate the default module combination
|
|
and sequence and select what has been observed to work best for most workloads.
|
|
This ongoing maintenance does not depend on the deferred formal evaluation work.
|
|
|
|
## Implementation Sequence
|
|
|
|
Work through the following steps in order. Keep schemas, consumers, prompt
|
|
examples, and affected tests consistent as each change lands, and update the
|
|
canonical documentation with the corresponding behavior. Step 5 checks the
|
|
assembled result; step 6 reviews test-policy compliance. Apply that policy during
|
|
the earlier steps as well. Proceed without waiting for live-model evaluation
|
|
evidence or a captured baseline.
|
|
|
|
### 1. Remove Confidence Throughout the Workflow
|
|
|
|
Remove confidence entirely from proposals, validation, and repair. Audita has no
|
|
established calibration that gives its model-reported scores reliable numerical
|
|
meaning across transcripts and models. Proposal scores currently affect
|
|
acceptance; validator scores are range-checked but do not determine approval.
|
|
Retaining the numbers adds complexity and encourages repair toward a threshold.
|
|
|
|
- Remove confidence fields from response schemas, Go models, decoding, and
|
|
validation paths, together with corresponding prompt instructions and examples.
|
|
- Remove the confidence-threshold validator, its registration and chain entries,
|
|
root and stage threshold settings, and any corresponding CLI exposure.
|
|
- Remove confidence-specific rejection feedback, reason codes, report/debug
|
|
fields, and other supporting code wherever present. Update affected tests,
|
|
fixtures, and configuration examples.
|
|
- Treat removal as a deliberate contract change for configuration, custom prompts,
|
|
and affected serialized outputs. Retain no inert confidence fields or threshold
|
|
knobs, and introduce no replacement numerical or categorical confidence scale.
|
|
|
|
Removing the proposal threshold can admit corrections it previously rejected.
|
|
That effect can be assessed when editorial evaluation resumes; it does not block
|
|
the selected removal.
|
|
|
|
### 2. Improve Validation Feedback
|
|
|
|
Retain the existing `reason` field. Update shared validation guidance and all
|
|
three validator definitions to request concise rejection explanations identifying
|
|
the specific problematic edit, applicable criterion, and relevant supplied
|
|
evidence within each validator's remit. For editorial review, for example:
|
|
removing "might" turns a tentative statement into a definite one.
|
|
|
|
Validators identify defects; the repair module owns revision or withdrawal.
|
|
Distinguish insufficient evidence from an established defect, respect the accepted
|
|
editorial allowance, and include distinct material issues when found without
|
|
requiring exhaustive analysis. Provide representative rejection examples.
|
|
|
|
Approvals need no explanation: instruct the model to return `reason: ""` and use
|
|
that convention in approval examples. The field remains required and permits an
|
|
empty string; this reason convention itself needs no schema or response-handling
|
|
change. Explanation guidance is expressed in prompts, with no new runtime
|
|
rule rejecting blank explanations. Consider schema expansion only if a concrete
|
|
need emerges.
|
|
|
|
The meaning-reversal validator requires particular care. The maintainer has
|
|
observed overly broad instructions causing it to reject virtually all corrections.
|
|
Preserve its core instructions and make only small, targeted feedback and example
|
|
changes, alongside confidence removal and the approval-reason convention:
|
|
|
|
- Detect reversal of the full segment's assertion, not every change in meaning.
|
|
Changes to names, referents, or other information alone are not grounds for
|
|
rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change
|
|
that is not a reversal.
|
|
- Assess the full before/after segment. An antonym appearing in the replacement
|
|
does not by itself establish reversal.
|
|
- Instruct rejection feedback to identify the specific opposition in the
|
|
assertions; "this changes the meaning" is insufficient.
|
|
- Approve corrections outside this narrow rejection criterion, leaving other
|
|
validators to assess their own criteria. Preserve the guard even though it can
|
|
exclude a genuine mistranscription correction that reverses the assertion;
|
|
no contextual exception is selected in this pass.
|
|
|
|
### 3. Refine Repair Guidance
|
|
|
|
Ask the model to reassess the correction using the original transcript, supplied
|
|
context, and rejection feedback:
|
|
|
|
- Revise when the evidence supports a correction that addresses the identified
|
|
problems.
|
|
- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain.
|
|
Withdrawal is a successful resolution that preserves the original text.
|
|
|
|
Use concrete explanations of uncertainty where relevant, such as context that
|
|
cannot distinguish two possible names. Preserve requested module focus and the
|
|
accepted editorial allowance. Apply these instructions within the existing
|
|
full-history repair lifecycle and revalidation, without adding new retry rules.
|
|
|
|
### 4. Consolidate Prompts, Descriptions, and Examples
|
|
|
|
Make a focused editing pass using the existing prompt fragment and module
|
|
structure. Organize instructions around shared contracts (including response
|
|
format, literal matching, and deletion semantics), module-specific requested work,
|
|
and the distinct responsibilities of validation and repair.
|
|
|
|
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
|
contradictions and unnecessary repetition, and keep each rendered prompt
|
|
understandable on its own. Retain repetition across separate model calls when
|
|
each call needs the instruction. Preserve the meaning-reversal validator's tuned
|
|
core wording as specified in step 2; broad simplification for consistency is not
|
|
appropriate there. Shorter prompts alone are not the objective.
|
|
|
|
Clarify descriptions and examples as follows:
|
|
|
|
- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work
|
|
as conservative punctuation, capitalization, spacing, and whole-word `a`/`an`
|
|
article cleanup. Preserve the current scope and cross-module editorial allowance.
|
|
- Describe glossary stages as focusing on known terms supported by selected
|
|
glossaries, and homophone stages as looking more broadly for plausible
|
|
spoken-word mistranscriptions, including glossary-supported corrections. Do not
|
|
exclude an otherwise supported homophone correction because it targets a
|
|
glossary term.
|
|
- Use short contextual contrasts for difficult judgments, such as accidental
|
|
restarts versus deliberate emphasis and context-supported glossary matches
|
|
versus merely similar spellings. Examples illustrate judgments rather than
|
|
establish unconditional editing rules.
|
|
- Retain useful campaign examples and add a few general examples. Use broadly
|
|
applicable wording for names, terminology, and glossary entries without asserting
|
|
D&D specificity or adding a disclaimer about it.
|
|
|
|
### 5. Complete Integration Checks and Documentation
|
|
|
|
Follow the [testing policy](../policy/testing.md) and the
|
|
[development validation guidance](../development.md#baseline-validation). Verify
|
|
the assembled workflow, including:
|
|
|
|
- Confidence-free proposal, validation, and repair responses work through their
|
|
schemas and consumers; substantive validators remain effective in the workflow.
|
|
- Rendered prompts and examples match the revised contracts, including empty
|
|
approval reasons and rejection feedback carried into repair requests.
|
|
- Revision, withdrawal, and full revalidation retain their existing mechanics.
|
|
|
|
Review the actual prompt wording and representative repair payloads to check the
|
|
selected responsibilities and the narrow meaning-reversal criterion. Offline
|
|
tests verify contracts and integration; they do not establish live-model
|
|
editorial quality. Complete the appropriate package checks, build, and full test
|
|
suite for the changes, without introducing paid calls into the default suite.
|
|
|
|
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
|
[prompt](../internal/prompts.md), [module](../internal/modules.md),
|
|
[validator](../internal/validators.md), and
|
|
[output](../integrations/output-files.md) documentation where affected, together
|
|
with maintained examples. Provide migration guidance to remove obsolete confidence
|
|
settings and fields from configuration, custom prompts, and affected output
|
|
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
|
of confidence-based behavior.
|
|
|
|
### 6. Review Tests for Policy Compliance
|
|
|
|
Review existing and newly changed tests supporting this work against the
|
|
[testing policy](../policy/testing.md), with particular attention to prompt tests.
|
|
Each retained test should protect a meaningful contract or behavior against a
|
|
realistic defect at a maintenance cost justified by that protection.
|
|
|
|
Prompt wording checks carry an unusually high cost: assertions that a particular
|
|
word or phrase appears in a prompt body obstruct development and experimentation
|
|
with alternative language without establishing that a model follows the intended
|
|
instruction. Remove these change-detector tests rather than updating their
|
|
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
|
|
body snapshots, exact prose comparisons, and equivalent checks hidden behind
|
|
helpers. Do not add new tests that freeze incidental prompt wording, including
|
|
the carefully tuned meaning-reversal instructions.
|
|
|
|
Retain or replace tests where they protect meaningful behavior, such as template
|
|
rendering with supplied data, prompt selection and operator replacement, response
|
|
schema compatibility, repair context assembly, and processing of validation and
|
|
repair responses. Exact comparisons of test-controlled data or explicit machine
|
|
contracts can be appropriate; distinguish those from assertions about bundled
|
|
instruction prose. Replace a removed test only when a genuine coverage gap
|
|
justifies it, and consolidate redundant coverage.
|
|
|
|
Review prompt language directly for the agreed editorial intent. Keyword checks
|
|
are not a substitute for that review or for the deferred live-model evaluation.
|
|
Summarize the tests removed, revised, and retained with their behavioral rationale,
|
|
and run the checks appropriate to any test changes made in this final stage.
|
|
|
|
## Deferred Editorial Evaluation
|
|
|
|
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
|
are outside this implementation scope. No baseline run or numerical quality gate
|
|
is required before the agreed changes proceed.
|
|
|
|
When this work resumes, consider a modest collection of needed corrections and
|
|
text that should remain unchanged: ambiguous glossary matches, intentional
|
|
repetition, meaningful apparent filler, punctuation affecting meaning, and repairs
|
|
that should be withdrawn. Compare unnecessary edits, missed corrections,
|
|
validator disagreement, and repair outcomes using independent editorial judgment;
|
|
validator approval alone is insufficient evidence.
|
|
|
|
Discuss representative models, expected outcomes, and an evaluation workflow at
|
|
that time. Prior repository revisions can support before/after comparisons,
|
|
including the effects of removing confidence thresholds. Keep model-dependent
|
|
evaluation separate from the default offline test suite.
|