Replace prompt quality roadmap with next improvements
This commit is contained in:
@@ -2,263 +2,183 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** All six implementation stages completed. Editorial evaluation remains
|
||||
deferred. No open decisions remain in the current scope.
|
||||
**Status:** Accepted; implementation has not begun. This roadmap replaces the
|
||||
completed prompt-quality plan with the recommendations from the fresh review of
|
||||
revision `d155bf4`. The review covered all eleven prompt definitions, shared
|
||||
instructions, response schemas, and request assembly. It identified design
|
||||
concerns, not measured live-model regressions.
|
||||
|
||||
## Target State and Constraints
|
||||
## Target State and Boundaries
|
||||
|
||||
Audita will use clearer proposal, validation, and repair prompts, actionable
|
||||
rejection feedback, and no model-reported confidence scores or thresholds.
|
||||
Repair will reconsider corrections on their merits and revise or withdraw them
|
||||
based on the supplied evidence. The initial review covered the eleven
|
||||
[prompt definitions](../internal/prompts.md) and their consumers at revision
|
||||
`6d34602`; it was a design review, not a live-model quality measurement.
|
||||
Audita's proposal, validation, and repair prompts will give each model call a
|
||||
coherent assignment and the evidence needed to carry it out. Proposals will
|
||||
better account for interacting edits, repair will receive an explicit view of
|
||||
its current proposed result, and validation will identify the target speaker.
|
||||
Focused examples will clarify difficult editorial judgments.
|
||||
|
||||
Preserve the following throughout implementation:
|
||||
Preserve the established design:
|
||||
|
||||
- **Editorial acceptance:** module focus guides requested work, while validators
|
||||
permit otherwise acceptable minor editorial cleanup across modules. Follow
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records
|
||||
the maintainer's finding that strict scope instructions also suppress desirable
|
||||
edits. Applicability, protection, and other substantive checks still apply.
|
||||
- **Correction mechanics:** retain literal-span and deletion semantics, distinct
|
||||
glossary context and protection, full before/after validation, and independent
|
||||
repair conversations with full history, withdrawal, and revalidation. Preserve
|
||||
native PromptKit/ConfigKit prompt replacement and the existing module structure.
|
||||
- **Module behavior:** keep `grammar` and its current requested scope; retain
|
||||
glossary and homophone modules with their useful overlap. Keep the current
|
||||
[default sequence](../config.md#pipeline-and-glossary-selections) for this pass
|
||||
and preserve configurable additional passes.
|
||||
- **Language:** retain English as the bundled default and useful existing campaign
|
||||
examples, supplemented with a few general examples. Broader language support is
|
||||
outside this pass.
|
||||
- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the
|
||||
full segment's assertion. Its targeted prompt changes are specified below.
|
||||
- Module focus guides proposal generation while otherwise acceptable minor
|
||||
editorial cleanup remains permissible across modules, as selected in
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
||||
- Validators retain distinct responsibilities. Preserve the carefully tuned
|
||||
meaning-reversal criterion and its focus on reversal of the full assertion.
|
||||
- Keep literal-span matching, deletion semantics, separate glossary context and
|
||||
protected spellings, and full before/after validation.
|
||||
- Keep independent repair conversations with full history, successful withdrawal,
|
||||
and full revalidation of revisions. Validators explain defects; repair owns
|
||||
revision or withdrawal. Approvals continue to need no explanation.
|
||||
- Retain confidence-free responses, English defaults, existing module identities,
|
||||
and native operator prompt replacement. The current module sequence is outside
|
||||
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
|
||||
|
||||
Nearly all module behavior reflects the maintainer's practical experimentation.
|
||||
Two glossary passes have proved useful, and some workloads benefit from three or
|
||||
more. These observations inform the current choice without fixing a permanent
|
||||
number or order of passes. Frequently reevaluate the default module combination
|
||||
and sequence and select what has been observed to work best for most workloads.
|
||||
This ongoing maintenance does not depend on the deferred formal evaluation work.
|
||||
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
|
||||
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
|
||||
Update these canonical owners when the corresponding changes land.
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
Work through the following steps in order. Keep schemas, consumers, prompt
|
||||
examples, and affected tests consistent as each change lands, and update the
|
||||
canonical documentation with the corresponding behavior. Step 5 checks the
|
||||
assembled result; step 6 reviews test-policy compliance. Apply that policy during
|
||||
the earlier steps as well. Proceed without waiting for live-model evaluation
|
||||
evidence or a captured baseline.
|
||||
### 1. Align Proposal and Repair Instruction Authority
|
||||
|
||||
### 1. Remove Confidence Throughout the Workflow
|
||||
Proposal system messages currently include categorical task restrictions, such
|
||||
as prohibiting punctuation or grammar edits in the homophone module. Repair
|
||||
inherits those messages and then permits acceptable minor editorial cleanup in a
|
||||
later user message. These instructions can compete across authority levels.
|
||||
|
||||
Remove confidence entirely from proposals, validation, and repair. Audita has no
|
||||
established calibration that gives its model-reported scores reliable numerical
|
||||
meaning across transcripts and models. Proposal scores currently affect
|
||||
acceptance; validator scores are range-checked but do not determine approval.
|
||||
Retaining the numbers adds complexity and encourages repair toward a threshold.
|
||||
- Put durable correction principles in the system messages: evidence-supported
|
||||
work, established protection and meaning constraints, and the distinction
|
||||
between requested focus and acceptable editorial corrections.
|
||||
- Put the initial proposal's focused assignment and task-specific exclusions in
|
||||
its user message. Preserve the focused tasks rather than asking every module
|
||||
to perform every kind of cleanup.
|
||||
- Let each repair instruction give the subsequent reassessment assignment,
|
||||
including the selected editorial allowance, without conflicting with an
|
||||
inherited system prohibition.
|
||||
- Review each complete proposal-to-repair conversation, including later repair
|
||||
attempts, for coherent instructions and response-format transitions.
|
||||
|
||||
- Remove confidence fields from response schemas, Go models, decoding, and
|
||||
validation paths, together with corresponding prompt instructions and examples.
|
||||
- Remove the confidence-threshold validator, its registration and chain entries,
|
||||
root and stage threshold settings, and any corresponding CLI exposure.
|
||||
- Remove confidence-specific rejection feedback, reason codes, report/debug
|
||||
fields, and other supporting code wherever present. Update affected tests,
|
||||
fixtures, and configuration examples.
|
||||
- Treat removal as a deliberate contract change for configuration, custom prompts,
|
||||
and affected serialized outputs. Retain no inert confidence fields or threshold
|
||||
knobs, and introduce no replacement numerical or categorical confidence scale.
|
||||
This implements the existing acceptance decision; it does not tighten validators
|
||||
or change the repair lifecycle. Keep the validator prompts' separate remits,
|
||||
especially meaning-reversal review, intact.
|
||||
|
||||
Removing the proposal threshold can admit corrections it previously rejected.
|
||||
That effect can be assessed when editorial evaluation resumes; it does not block
|
||||
the selected removal.
|
||||
### 2. Guide Independently Useful and Overlapping Corrections
|
||||
|
||||
### 2. Improve Validation Feedback
|
||||
The shared proposal contract already makes each edit refer to the supplied
|
||||
segment rather than the imagined result of another proposal. Add guidance for
|
||||
edits that overlap or depend on one another:
|
||||
|
||||
Retain the existing `reason` field. Update shared validation guidance and all
|
||||
three validator definitions to request concise rejection explanations identifying
|
||||
the specific problematic edit, applicable criterion, and relevant supplied
|
||||
evidence within each validator's remit. For editorial review, for example:
|
||||
removing "might" turns a tentative statement into a definite one.
|
||||
- Prefer independently useful corrections with literal spans that identify the
|
||||
intended changes.
|
||||
- Combine changes into one proposal when their spans overlap or when one edit
|
||||
requires the other. Preserve unchanged context needed for unique matching or
|
||||
for selecting an occurrence under the glossary replacement policy.
|
||||
- Continue to allow multiple independent corrections to the same segment.
|
||||
|
||||
Validators identify defects; the repair module owns revision or withdrawal.
|
||||
Distinguish insufficient evidence from an established defect, respect the accepted
|
||||
editorial allowance, and include distinct material issues when found without
|
||||
requiring exhaustive analysis. Provide representative rejection examples.
|
||||
For example, separate proposals to capitalize `hello` and replace
|
||||
`hello world` with `hello world` may both pass validation against the original,
|
||||
but applying the capitalization first prevents the second span from matching.
|
||||
A combined replacement can express both edits against the original text.
|
||||
|
||||
Approvals need no explanation: instruct the model to return `reason: ""` and use
|
||||
that convention in approval examples. The field remains required and permits an
|
||||
empty string; this reason convention itself needs no schema or response-handling
|
||||
change. Explanation guidance is expressed in prompts, with no new runtime
|
||||
rule rejecting blank explanations. Consider schema expansion only if a concrete
|
||||
need emerges.
|
||||
These are prompt instructions. Add no new runtime overlap rejection, proposal
|
||||
limit, or application-order rule. Assess combined semantic effects separately
|
||||
if practical examples establish a need for additional processing changes.
|
||||
|
||||
The meaning-reversal validator requires particular care. The maintainer has
|
||||
observed overly broad instructions causing it to reject virtually all corrections.
|
||||
Preserve its core instructions and make only small, targeted feedback and example
|
||||
changes, alongside confidence removal and the approval-reason convention:
|
||||
### 3. Supply Repair with the Current Before/After Segment
|
||||
|
||||
- Detect reversal of the full segment's assertion, not every change in meaning.
|
||||
Changes to names, referents, or other information alone are not grounds for
|
||||
rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change
|
||||
that is not a reversal.
|
||||
- Assess the full before/after segment. An antonym appearing in the replacement
|
||||
does not by itself establish reversal.
|
||||
- Instruct rejection feedback to identify the specific opposition in the
|
||||
assertions; "this changes the meaning" is insufficient.
|
||||
- Approve corrections outside this narrow rejection criterion, leaving other
|
||||
validators to assess their own criteria. Preserve the guard even though it can
|
||||
exclude a genuine mistranscription correction that reverses the assertion;
|
||||
no contextual exception is selected in this pass.
|
||||
The repair request currently identifies the proposal, its current correction,
|
||||
and validation findings. Although the original section remains available in
|
||||
conversation history, repair must locate it and reconstruct the proposed result.
|
||||
|
||||
### 3. Refine Repair Guidance
|
||||
- Add the original target segment and its proposed resulting text to repair
|
||||
context when the current proposal is applicable.
|
||||
- Reuse the same domain preview semantics used by validation, including literal
|
||||
deletion, whitespace, and replacement of every matching occurrence where
|
||||
applicable. Avoid a second implementation of replacement behavior.
|
||||
- Refresh this view for each repair attempt, using the current proposal version
|
||||
and target rather than a stale view of the initial proposal.
|
||||
- When a preview cannot be produced, retain the relevant findings and available
|
||||
source context without fabricating a resulting segment or implying application
|
||||
succeeded.
|
||||
- Keep the full conversation history, proposal references, and revision or
|
||||
withdrawal response contract. Explain the added view as evidence for assessing
|
||||
the current correction, not an already-applied transcript change.
|
||||
|
||||
Ask the model to reassess the correction using the original transcript, supplied
|
||||
context, and rejection feedback:
|
||||
This should make spacing, deletion, and replacement consequences easier to
|
||||
judge without requiring the model to reproduce Audita's application mechanics.
|
||||
|
||||
- Revise when the evidence supports a correction that addresses the identified
|
||||
problems.
|
||||
- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain.
|
||||
Withdrawal is a successful resolution that preserves the original text.
|
||||
### 4. Improve Validation Evidence
|
||||
|
||||
Use concrete explanations of uncertainty where relevant, such as context that
|
||||
cannot distinguish two possible names. Preserve requested module focus and the
|
||||
accepted editorial allowance. Apply these instructions within the existing
|
||||
full-history repair lifecycle and revalidation, without adding new retry rules.
|
||||
Add the target segment's speaker to each validation item. Neighboring segments
|
||||
already carry their speakers; the candidate's identity completes that context
|
||||
and helps distinguish continued speech, interruptions, and another speaker's
|
||||
response. Carry the supplied value through the existing payload construction
|
||||
and token accounting without inferring a missing speaker.
|
||||
|
||||
### 4. Consolidate Prompts, Descriptions, and Examples
|
||||
Investigate the broader evidence imbalance using available practical cases:
|
||||
proposal sees a section, while validation sees immediate neighbors and selected
|
||||
glossary entries. A correction supported several turns earlier may appear
|
||||
unsupported to a validator. Compare the evidence actually supplied to both
|
||||
calls before attributing a rejection to prompt wording.
|
||||
|
||||
Make a focused editing pass using the existing prompt fragment and module
|
||||
structure. Organize instructions around shared contracts (including response
|
||||
format, literal matching, and deletion semantics), module-specific requested work,
|
||||
and the distinct responsibilities of validation and repair.
|
||||
Adding the target speaker is agreed implementation work. Substantial expansion
|
||||
of validation context remains evidence-dependent: record a concrete case and a
|
||||
focused follow-up recommendation if needed. Retain the current context selection
|
||||
and fitting behavior otherwise. Stronger instructions cannot supply missing
|
||||
evidence, and this roadmap does not require a broader context redesign.
|
||||
|
||||
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
||||
contradictions and unnecessary repetition, and keep each rendered prompt
|
||||
understandable on its own. Retain repetition across separate model calls when
|
||||
each call needs the instruction. Preserve the meaning-reversal validator's tuned
|
||||
core wording as specified in step 2; broad simplification for consistency is not
|
||||
appropriate there. Shorter prompts alone are not the objective.
|
||||
### 5. Add Focused Editorial and Repair Examples
|
||||
|
||||
Clarify descriptions and examples as follows:
|
||||
Add a small number of examples where they clarify judgments that current
|
||||
instructions leave implicit:
|
||||
|
||||
- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work
|
||||
as conservative punctuation, capitalization, spacing, and whole-word `a`/`an`
|
||||
article cleanup. Preserve the current scope and cross-module editorial allowance.
|
||||
- Describe glossary stages as focusing on known terms supported by selected
|
||||
glossaries, and homophone stages as looking more broadly for plausible
|
||||
spoken-word mistranscriptions, including glossary-supported corrections. Do not
|
||||
exclude an otherwise supported homophone correction because it targets a
|
||||
glossary term.
|
||||
- Use short contextual contrasts for difficult judgments, such as accidental
|
||||
restarts versus deliberate emphasis and context-supported glossary matches
|
||||
versus merely similar spellings. Examples illustrate judgments rather than
|
||||
establish unconditional editing rules.
|
||||
- Retain useful campaign examples and add a few general examples. Use broadly
|
||||
applicable wording for names, terminology, and glossary entries without asserting
|
||||
D&D specificity or adding a disclaimer about it.
|
||||
- Apparent filler that contributes meaning, such as `like` expressing
|
||||
approximation or `I mean` introducing a substantive self-correction. Use
|
||||
surrounding context to contrast meaningful usage with removable dysfluency.
|
||||
- One compact repair revision and one withdrawal, using the existing response
|
||||
schema and making withdrawal a successful preservation of the original text.
|
||||
- A combined overlapping edit illustrating step 2.
|
||||
|
||||
### 5. Complete Integration Checks and Documentation
|
||||
Place examples in the relevant proposal, validation, or shared repair guidance.
|
||||
Keep existing useful contextual contrasts. Examples should illustrate reasoning
|
||||
without becoming unconditional token-based editing rules or an extensive
|
||||
rulebook. Maintain the narrow meaning-reversal instructions.
|
||||
|
||||
Follow the [testing policy](../policy/testing.md) and the
|
||||
[development validation guidance](../development.md#baseline-validation). Verify
|
||||
the assembled workflow, including:
|
||||
### 6. Verify Integration, Documentation, and Test-Policy Compliance
|
||||
|
||||
- Confidence-free proposal, validation, and repair responses work through their
|
||||
schemas and consumers; substantive validators remain effective in the workflow.
|
||||
- Rendered prompts and examples match the revised contracts, including empty
|
||||
approval reasons and rejection feedback carried into repair requests.
|
||||
- Revision, withdrawal, and full revalidation retain their existing mechanics.
|
||||
Follow the [testing policy](../policy/testing.md) throughout implementation and
|
||||
finish with a focused review of affected tests:
|
||||
|
||||
Review the actual prompt wording and representative repair payloads to check the
|
||||
selected responsibilities and the narrow meaning-reversal criterion. Offline
|
||||
tests verify contracts and integration; they do not establish live-model
|
||||
editorial quality. Complete the appropriate package checks, build, and full test
|
||||
suite for the changes, without introducing paid calls into the default suite.
|
||||
- Verify repair context reflects the current proposal and the shared preview
|
||||
behavior, including representative deletion, replacement-policy, subsequent
|
||||
revision, and unavailable-preview cases where coverage is warranted.
|
||||
- Verify target-speaker data reaches validation and participates in existing
|
||||
payload accounting. Preserve coverage of response handling, revision,
|
||||
withdrawal, revalidation, and operator prompt replacement.
|
||||
- Review rendered messages and representative complete repair conversations
|
||||
directly for instruction authority, editorial intent, and format consistency.
|
||||
Check response examples against their schemas.
|
||||
- Remove any affected tests that freeze bundled prompt wording through phrase
|
||||
checks, exact prose comparisons, or snapshots. Do not replace them with checks
|
||||
for the new wording. Retain tests of supplied data and meaningful machine
|
||||
contracts where their protective value justifies their maintenance cost.
|
||||
- Update the relevant canonical prompt, module, validator, and runtime documents
|
||||
as behavior lands. Update maintained examples where affected; check links and
|
||||
whitespace.
|
||||
|
||||
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
||||
[prompt](../internal/prompts.md), [module](../internal/modules.md),
|
||||
[validator](../internal/validators.md), and
|
||||
[output](../integrations/output-files.md) documentation where affected, together
|
||||
with maintained examples. Provide migration guidance to remove obsolete confidence
|
||||
settings and fields from configuration, custom prompts, and affected output
|
||||
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
||||
of confidence-based behavior.
|
||||
Run the appropriate package checks, build, and full offline suite for the
|
||||
implementation under the [development guidance](../development.md#baseline-validation).
|
||||
Report the test-policy review's meaningful removals, revisions, and retained
|
||||
coverage. Offline tests establish data and execution contracts; they do not
|
||||
establish live-model editorial quality.
|
||||
|
||||
### 6. Review Tests for Policy Compliance
|
||||
## Deferred Evidence and Follow-Up
|
||||
|
||||
Review existing and newly changed tests supporting this work against the
|
||||
[testing policy](../policy/testing.md), with particular attention to prompt tests.
|
||||
Each retained test should protect a meaningful contract or behavior against a
|
||||
realistic defect at a maintenance cost justified by that protection.
|
||||
Formal evaluation corpus preparation and live-model comparisons remain deferred
|
||||
and are not prerequisites for the agreed changes. Broader validation context and
|
||||
additional handling of combined semantic effects should follow concrete evidence
|
||||
rather than hypothetical restrictions. Any later evaluation should examine
|
||||
needed and unnecessary edits, missed corrections, validator disagreement, and
|
||||
repair outcomes using independent editorial judgment. Keep live calls opt-in
|
||||
and separate from the default offline suite.
|
||||
|
||||
Prompt wording checks carry an unusually high cost: assertions that a particular
|
||||
word or phrase appears in a prompt body obstruct development and experimentation
|
||||
with alternative language without establishing that a model follows the intended
|
||||
instruction. Remove these change-detector tests rather than updating their
|
||||
expected phrases to match the revised prompts. Apply the same scrutiny to prompt
|
||||
body snapshots, exact prose comparisons, and equivalent checks hidden behind
|
||||
helpers. Do not add new tests that freeze incidental prompt wording, including
|
||||
the carefully tuned meaning-reversal instructions.
|
||||
|
||||
Retain or replace tests where they protect meaningful behavior, such as template
|
||||
rendering with supplied data, prompt selection and operator replacement, response
|
||||
schema compatibility, repair context assembly, and processing of validation and
|
||||
repair responses. Exact comparisons of test-controlled data or explicit machine
|
||||
contracts can be appropriate; distinguish those from assertions about bundled
|
||||
instruction prose. Replace a removed test only when a genuine coverage gap
|
||||
justifies it, and consolidate redundant coverage.
|
||||
|
||||
Review prompt language directly for the agreed editorial intent. Keyword checks
|
||||
are not a substitute for that review or for the deferred live-model evaluation.
|
||||
Summarize the tests removed, revised, and retained with their behavioral rationale,
|
||||
and run the checks appropriate to any test changes made in this final stage.
|
||||
|
||||
## Completion and Test Review
|
||||
|
||||
Confidence has been removed from schemas, models, validators, configuration,
|
||||
CLI flags, feedback, and fixtures. All eleven prompt definitions and their shared
|
||||
contracts have been revised. Current behavior and cutover instructions are
|
||||
recorded in the canonical documentation linked above and the
|
||||
[v3 migration guide](../migrations/v3.md#confidence-removal).
|
||||
|
||||
The final test-policy review made the following changes:
|
||||
|
||||
- Removed assertions about bundled instruction phrases, module-guidance wording,
|
||||
and exact shared-instruction repetition. These detected prose changes without
|
||||
demonstrating model behavior and obstructed prompt experimentation.
|
||||
- Removed confidence-specific tests with the retired contract. Repair and report
|
||||
scenarios now use substantive rejection cases, preserving their coverage of
|
||||
revision, withdrawal, exhaustion, partial success, and failure handling.
|
||||
- Revised subprocess model fixtures to identify structured validation input
|
||||
without depending on a prose heading; removed an incidental schema-formatting
|
||||
assertion.
|
||||
- Retained supplied-data interpolation, operator prompt replacement, schema and
|
||||
response handling, repair history, protected-term behavior, and end-to-end
|
||||
outcomes. Test-controlled strings and machine-contract comparisons remain where
|
||||
they detect meaningful defects.
|
||||
|
||||
The full offline Go test suite and application build passed. Prompt rendering,
|
||||
response examples, documentation links, and whitespace were checked. Prompt
|
||||
language was reviewed directly for the selected responsibilities and narrow
|
||||
meaning-reversal criterion; these checks do not establish live-model editorial
|
||||
quality.
|
||||
|
||||
## Deferred Editorial Evaluation
|
||||
|
||||
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
||||
are outside this implementation scope. No baseline run or numerical quality gate
|
||||
is required before the agreed changes proceed.
|
||||
|
||||
When this work resumes, consider a modest collection of needed corrections and
|
||||
text that should remain unchanged: ambiguous glossary matches, intentional
|
||||
repetition, meaningful apparent filler, punctuation affecting meaning, and repairs
|
||||
that should be withdrawn. Compare unnecessary edits, missed corrections,
|
||||
validator disagreement, and repair outcomes using independent editorial judgment;
|
||||
validator approval alone is insufficient evidence.
|
||||
|
||||
Discuss representative models, expected outcomes, and an evaluation workflow at
|
||||
that time. Prior repository revisions can support before/after comparisons,
|
||||
including the effects of removing confidence thresholds. Keep model-dependent
|
||||
evaluation separate from the default offline test suite.
|
||||
No blocking product decisions remain for the implementation scope above.
|
||||
|
||||
Reference in New Issue
Block a user