Replace prompt quality roadmap with next improvements

This commit is contained in:
2026-11-11 17:26:22 +00:00
parent d155bf4cb3
commit cf1fbe5748

View File

@@ -2,263 +2,183 @@
**Updated:** 2026-11-11
**Status:** All six implementation stages completed. Editorial evaluation remains
deferred. No open decisions remain in the current scope.
**Status:** Accepted; implementation has not begun. This roadmap replaces the
completed prompt-quality plan with the recommendations from the fresh review of
revision `d155bf4`. The review covered all eleven prompt definitions, shared
instructions, response schemas, and request assembly. It identified design
concerns, not measured live-model regressions.
## Target State and Constraints
## Target State and Boundaries
Audita will use clearer proposal, validation, and repair prompts, actionable
rejection feedback, and no model-reported confidence scores or thresholds.
Repair will reconsider corrections on their merits and revise or withdraw them
based on the supplied evidence. The initial review covered the eleven
[prompt definitions](../internal/prompts.md) and their consumers at revision
`6d34602`; it was a design review, not a live-model quality measurement.
Audita's proposal, validation, and repair prompts will give each model call a
coherent assignment and the evidence needed to carry it out. Proposals will
better account for interacting edits, repair will receive an explicit view of
its current proposed result, and validation will identify the target speaker.
Focused examples will clarify difficult editorial judgments.
Preserve the following throughout implementation:
Preserve the established design:
- **Editorial acceptance:** module focus guides requested work, while validators
permit otherwise acceptable minor editorial cleanup across modules. Follow
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records
the maintainer's finding that strict scope instructions also suppress desirable
edits. Applicability, protection, and other substantive checks still apply.
- **Correction mechanics:** retain literal-span and deletion semantics, distinct
glossary context and protection, full before/after validation, and independent
repair conversations with full history, withdrawal, and revalidation. Preserve
native PromptKit/ConfigKit prompt replacement and the existing module structure.
- **Module behavior:** keep `grammar` and its current requested scope; retain
glossary and homophone modules with their useful overlap. Keep the current
[default sequence](../config.md#pipeline-and-glossary-selections) for this pass
and preserve configurable additional passes.
- **Language:** retain English as the bundled default and useful existing campaign
examples, supplemented with a few general examples. Broader language support is
outside this pass.
- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the
full segment's assertion. Its targeted prompt changes are specified below.
- Module focus guides proposal generation while otherwise acceptable minor
editorial cleanup remains permissible across modules, as selected in
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
- Validators retain distinct responsibilities. Preserve the carefully tuned
meaning-reversal criterion and its focus on reversal of the full assertion.
- Keep literal-span matching, deletion semantics, separate glossary context and
protected spellings, and full before/after validation.
- Keep independent repair conversations with full history, successful withdrawal,
and full revalidation of revisions. Validators explain defects; repair owns
revision or withdrawal. Approvals continue to need no explanation.
- Retain confidence-free responses, English defaults, existing module identities,
and native operator prompt replacement. The current module sequence is outside
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
Nearly all module behavior reflects the maintainer's practical experimentation.
Two glossary passes have proved useful, and some workloads benefit from three or
more. These observations inform the current choice without fixing a permanent
number or order of passes. Frequently reevaluate the default module combination
and sequence and select what has been observed to work best for most workloads.
This ongoing maintenance does not depend on the deferred formal evaluation work.
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
Update these canonical owners when the corresponding changes land.
## Implementation Sequence
Work through the following steps in order. Keep schemas, consumers, prompt
examples, and affected tests consistent as each change lands, and update the
canonical documentation with the corresponding behavior. Step 5 checks the
assembled result; step 6 reviews test-policy compliance. Apply that policy during
the earlier steps as well. Proceed without waiting for live-model evaluation
evidence or a captured baseline.
### 1. Align Proposal and Repair Instruction Authority
### 1. Remove Confidence Throughout the Workflow
Proposal system messages currently include categorical task restrictions, such
as prohibiting punctuation or grammar edits in the homophone module. Repair
inherits those messages and then permits acceptable minor editorial cleanup in a
later user message. These instructions can compete across authority levels.
Remove confidence entirely from proposals, validation, and repair. Audita has no
established calibration that gives its model-reported scores reliable numerical
meaning across transcripts and models. Proposal scores currently affect
acceptance; validator scores are range-checked but do not determine approval.
Retaining the numbers adds complexity and encourages repair toward a threshold.
- Put durable correction principles in the system messages: evidence-supported
work, established protection and meaning constraints, and the distinction
between requested focus and acceptable editorial corrections.
- Put the initial proposal's focused assignment and task-specific exclusions in
its user message. Preserve the focused tasks rather than asking every module
to perform every kind of cleanup.
- Let each repair instruction give the subsequent reassessment assignment,
including the selected editorial allowance, without conflicting with an
inherited system prohibition.
- Review each complete proposal-to-repair conversation, including later repair
attempts, for coherent instructions and response-format transitions.
- Remove confidence fields from response schemas, Go models, decoding, and
validation paths, together with corresponding prompt instructions and examples.
- Remove the confidence-threshold validator, its registration and chain entries,
root and stage threshold settings, and any corresponding CLI exposure.
- Remove confidence-specific rejection feedback, reason codes, report/debug
fields, and other supporting code wherever present. Update affected tests,
fixtures, and configuration examples.
- Treat removal as a deliberate contract change for configuration, custom prompts,
and affected serialized outputs. Retain no inert confidence fields or threshold
knobs, and introduce no replacement numerical or categorical confidence scale.
This implements the existing acceptance decision; it does not tighten validators
or change the repair lifecycle. Keep the validator prompts' separate remits,
especially meaning-reversal review, intact.
Removing the proposal threshold can admit corrections it previously rejected.
That effect can be assessed when editorial evaluation resumes; it does not block
the selected removal.
### 2. Guide Independently Useful and Overlapping Corrections
### 2. Improve Validation Feedback
The shared proposal contract already makes each edit refer to the supplied
segment rather than the imagined result of another proposal. Add guidance for
edits that overlap or depend on one another:
Retain the existing `reason` field. Update shared validation guidance and all
three validator definitions to request concise rejection explanations identifying
the specific problematic edit, applicable criterion, and relevant supplied
evidence within each validator's remit. For editorial review, for example:
removing "might" turns a tentative statement into a definite one.
- Prefer independently useful corrections with literal spans that identify the
intended changes.
- Combine changes into one proposal when their spans overlap or when one edit
requires the other. Preserve unchanged context needed for unique matching or
for selecting an occurrence under the glossary replacement policy.
- Continue to allow multiple independent corrections to the same segment.
Validators identify defects; the repair module owns revision or withdrawal.
Distinguish insufficient evidence from an established defect, respect the accepted
editorial allowance, and include distinct material issues when found without
requiring exhaustive analysis. Provide representative rejection examples.
For example, separate proposals to capitalize `hello` and replace
`hello world` with `hello world` may both pass validation against the original,
but applying the capitalization first prevents the second span from matching.
A combined replacement can express both edits against the original text.
Approvals need no explanation: instruct the model to return `reason: ""` and use
that convention in approval examples. The field remains required and permits an
empty string; this reason convention itself needs no schema or response-handling
change. Explanation guidance is expressed in prompts, with no new runtime
rule rejecting blank explanations. Consider schema expansion only if a concrete
need emerges.
These are prompt instructions. Add no new runtime overlap rejection, proposal
limit, or application-order rule. Assess combined semantic effects separately
if practical examples establish a need for additional processing changes.
The meaning-reversal validator requires particular care. The maintainer has
observed overly broad instructions causing it to reject virtually all corrections.
Preserve its core instructions and make only small, targeted feedback and example
changes, alongside confidence removal and the approval-reason convention:
### 3. Supply Repair with the Current Before/After Segment
- Detect reversal of the full segment's assertion, not every change in meaning.
Changes to names, referents, or other information alone are not grounds for
rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change
that is not a reversal.
- Assess the full before/after segment. An antonym appearing in the replacement
does not by itself establish reversal.
- Instruct rejection feedback to identify the specific opposition in the
assertions; "this changes the meaning" is insufficient.
- Approve corrections outside this narrow rejection criterion, leaving other
validators to assess their own criteria. Preserve the guard even though it can
exclude a genuine mistranscription correction that reverses the assertion;
no contextual exception is selected in this pass.
The repair request currently identifies the proposal, its current correction,
and validation findings. Although the original section remains available in
conversation history, repair must locate it and reconstruct the proposed result.
### 3. Refine Repair Guidance
- Add the original target segment and its proposed resulting text to repair
context when the current proposal is applicable.
- Reuse the same domain preview semantics used by validation, including literal
deletion, whitespace, and replacement of every matching occurrence where
applicable. Avoid a second implementation of replacement behavior.
- Refresh this view for each repair attempt, using the current proposal version
and target rather than a stale view of the initial proposal.
- When a preview cannot be produced, retain the relevant findings and available
source context without fabricating a resulting segment or implying application
succeeded.
- Keep the full conversation history, proposal references, and revision or
withdrawal response contract. Explain the added view as evidence for assessing
the current correction, not an already-applied transcript change.
Ask the model to reassess the correction using the original transcript, supplied
context, and rejection feedback:
This should make spacing, deletion, and replacement consequences easier to
judge without requiring the model to reproduce Audita's application mechanics.
- Revise when the evidence supports a correction that addresses the identified
problems.
- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain.
Withdrawal is a successful resolution that preserves the original text.
### 4. Improve Validation Evidence
Use concrete explanations of uncertainty where relevant, such as context that
cannot distinguish two possible names. Preserve requested module focus and the
accepted editorial allowance. Apply these instructions within the existing
full-history repair lifecycle and revalidation, without adding new retry rules.
Add the target segment's speaker to each validation item. Neighboring segments
already carry their speakers; the candidate's identity completes that context
and helps distinguish continued speech, interruptions, and another speaker's
response. Carry the supplied value through the existing payload construction
and token accounting without inferring a missing speaker.
### 4. Consolidate Prompts, Descriptions, and Examples
Investigate the broader evidence imbalance using available practical cases:
proposal sees a section, while validation sees immediate neighbors and selected
glossary entries. A correction supported several turns earlier may appear
unsupported to a validator. Compare the evidence actually supplied to both
calls before attributing a rejection to prompt wording.
Make a focused editing pass using the existing prompt fragment and module
structure. Organize instructions around shared contracts (including response
format, literal matching, and deletion semantics), module-specific requested work,
and the distinct responsibilities of validation and repair.
Adding the target speaker is agreed implementation work. Substantial expansion
of validation context remains evidence-dependent: record a concrete case and a
focused follow-up recommendation if needed. Retain the current context selection
and fitting behavior otherwise. Stronger instructions cannot supply missing
evidence, and this roadmap does not require a broader context redesign.
Consolidate duplicated rules where small shared fragments prevent drift, remove
contradictions and unnecessary repetition, and keep each rendered prompt
understandable on its own. Retain repetition across separate model calls when
each call needs the instruction. Preserve the meaning-reversal validator's tuned
core wording as specified in step 2; broad simplification for consistency is not
appropriate there. Shorter prompts alone are not the objective.
### 5. Add Focused Editorial and Repair Examples
Clarify descriptions and examples as follows:
Add a small number of examples where they clarify judgments that current
instructions leave implicit:
- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work
as conservative punctuation, capitalization, spacing, and whole-word `a`/`an`
article cleanup. Preserve the current scope and cross-module editorial allowance.
- Describe glossary stages as focusing on known terms supported by selected
glossaries, and homophone stages as looking more broadly for plausible
spoken-word mistranscriptions, including glossary-supported corrections. Do not
exclude an otherwise supported homophone correction because it targets a
glossary term.
- Use short contextual contrasts for difficult judgments, such as accidental
restarts versus deliberate emphasis and context-supported glossary matches
versus merely similar spellings. Examples illustrate judgments rather than
establish unconditional editing rules.
- Retain useful campaign examples and add a few general examples. Use broadly
applicable wording for names, terminology, and glossary entries without asserting
D&D specificity or adding a disclaimer about it.
- Apparent filler that contributes meaning, such as `like` expressing
approximation or `I mean` introducing a substantive self-correction. Use
surrounding context to contrast meaningful usage with removable dysfluency.
- One compact repair revision and one withdrawal, using the existing response
schema and making withdrawal a successful preservation of the original text.
- A combined overlapping edit illustrating step 2.
### 5. Complete Integration Checks and Documentation
Place examples in the relevant proposal, validation, or shared repair guidance.
Keep existing useful contextual contrasts. Examples should illustrate reasoning
without becoming unconditional token-based editing rules or an extensive
rulebook. Maintain the narrow meaning-reversal instructions.
Follow the [testing policy](../policy/testing.md) and the
[development validation guidance](../development.md#baseline-validation). Verify
the assembled workflow, including:
### 6. Verify Integration, Documentation, and Test-Policy Compliance
- Confidence-free proposal, validation, and repair responses work through their
schemas and consumers; substantive validators remain effective in the workflow.
- Rendered prompts and examples match the revised contracts, including empty
approval reasons and rejection feedback carried into repair requests.
- Revision, withdrawal, and full revalidation retain their existing mechanics.
Follow the [testing policy](../policy/testing.md) throughout implementation and
finish with a focused review of affected tests:
Review the actual prompt wording and representative repair payloads to check the
selected responsibilities and the narrow meaning-reversal criterion. Offline
tests verify contracts and integration; they do not establish live-model
editorial quality. Complete the appropriate package checks, build, and full test
suite for the changes, without introducing paid calls into the default suite.
- Verify repair context reflects the current proposal and the shared preview
behavior, including representative deletion, replacement-policy, subsequent
revision, and unavailable-preview cases where coverage is warranted.
- Verify target-speaker data reaches validation and participates in existing
payload accounting. Preserve coverage of response handling, revision,
withdrawal, revalidation, and operator prompt replacement.
- Review rendered messages and representative complete repair conversations
directly for instruction authority, editorial intent, and format consistency.
Check response examples against their schemas.
- Remove any affected tests that freeze bundled prompt wording through phrase
checks, exact prose comparisons, or snapshots. Do not replace them with checks
for the new wording. Retain tests of supplied data and meaningful machine
contracts where their protective value justifies their maintenance cost.
- Update the relevant canonical prompt, module, validator, and runtime documents
as behavior lands. Update maintained examples where affected; check links and
whitespace.
Update canonical [configuration](../config.md), [CLI](../cli.md),
[prompt](../internal/prompts.md), [module](../internal/modules.md),
[validator](../internal/validators.md), and
[output](../integrations/output-files.md) documentation where affected, together
with maintained examples. Provide migration guidance to remove obsolete confidence
settings and fields from configuration, custom prompts, and affected output
consumers. Check links, examples, and whitespace, and remove stale descriptions
of confidence-based behavior.
Run the appropriate package checks, build, and full offline suite for the
implementation under the [development guidance](../development.md#baseline-validation).
Report the test-policy review's meaningful removals, revisions, and retained
coverage. Offline tests establish data and execution contracts; they do not
establish live-model editorial quality.
### 6. Review Tests for Policy Compliance
## Deferred Evidence and Follow-Up
Review existing and newly changed tests supporting this work against the
[testing policy](../policy/testing.md), with particular attention to prompt tests.
Each retained test should protect a meaningful contract or behavior against a
realistic defect at a maintenance cost justified by that protection.
Formal evaluation corpus preparation and live-model comparisons remain deferred
and are not prerequisites for the agreed changes. Broader validation context and
additional handling of combined semantic effects should follow concrete evidence
rather than hypothetical restrictions. Any later evaluation should examine
needed and unnecessary edits, missed corrections, validator disagreement, and
repair outcomes using independent editorial judgment. Keep live calls opt-in
and separate from the default offline suite.
Prompt wording checks carry an unusually high cost: assertions that a particular
word or phrase appears in a prompt body obstruct development and experimentation
with alternative language without establishing that a model follows the intended
instruction. Remove these change-detector tests rather than updating their
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
body snapshots, exact prose comparisons, and equivalent checks hidden behind
helpers. Do not add new tests that freeze incidental prompt wording, including
the carefully tuned meaning-reversal instructions.
Retain or replace tests where they protect meaningful behavior, such as template
rendering with supplied data, prompt selection and operator replacement, response
schema compatibility, repair context assembly, and processing of validation and
repair responses. Exact comparisons of test-controlled data or explicit machine
contracts can be appropriate; distinguish those from assertions about bundled
instruction prose. Replace a removed test only when a genuine coverage gap
justifies it, and consolidate redundant coverage.
Review prompt language directly for the agreed editorial intent. Keyword checks
are not a substitute for that review or for the deferred live-model evaluation.
Summarize the tests removed, revised, and retained with their behavioral rationale,
and run the checks appropriate to any test changes made in this final stage.
## Completion and Test Review
Confidence has been removed from schemas, models, validators, configuration,
CLI flags, feedback, and fixtures. All eleven prompt definitions and their shared
contracts have been revised. Current behavior and cutover instructions are
recorded in the canonical documentation linked above and the
[v3 migration guide](../migrations/v3.md#confidence-removal).
The final test-policy review made the following changes:
- Removed assertions about bundled instruction phrases, module-guidance wording,
and exact shared-instruction repetition. These detected prose changes without
demonstrating model behavior and obstructed prompt experimentation.
- Removed confidence-specific tests with the retired contract. Repair and report
scenarios now use substantive rejection cases, preserving their coverage of
revision, withdrawal, exhaustion, partial success, and failure handling.
- Revised subprocess model fixtures to identify structured validation input
without depending on a prose heading; removed an incidental schema-formatting
assertion.
- Retained supplied-data interpolation, operator prompt replacement, schema and
response handling, repair history, protected-term behavior, and end-to-end
outcomes. Test-controlled strings and machine-contract comparisons remain where
they detect meaningful defects.
The full offline Go test suite and application build passed. Prompt rendering,
response examples, documentation links, and whitespace were checked. Prompt
language was reviewed directly for the selected responsibilities and narrow
meaning-reversal criterion; these checks do not establish live-model editorial
quality.
## Deferred Editorial Evaluation
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
are outside this implementation scope. No baseline run or numerical quality gate
is required before the agreed changes proceed.
When this work resumes, consider a modest collection of needed corrections and
text that should remain unchanged: ambiguous glossary matches, intentional
repetition, meaningful apparent filler, punctuation affecting meaning, and repairs
that should be withdrawn. Compare unnecessary edits, missed corrections,
validator disagreement, and repair outcomes using independent editorial judgment;
validator approval alone is insufficient evidence.
Discuss representative models, expected outcomes, and an evaluation workflow at
that time. Prior repository revisions can support before/after comparisons,
including the effects of removing confidence thresholds. Keep model-dependent
evaluation separate from the default offline test suite.
No blocking product decisions remain for the implementation scope above.