Consolidate prompt quality roadmap into implementation sequence
This commit is contained in:
@@ -2,218 +2,203 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Topic 1 resolved in ADR 0003; topics 2–4 and 6 decided, with
|
||||
implementation pending; topic 5 deferred. No open decisions remain in the
|
||||
current scope. Topic 5 retains recommendations for future discussion.
|
||||
**Status:** Decisions settled; implementation pending. Editorial evaluation is
|
||||
deferred. No open decisions remain in the current scope.
|
||||
|
||||
## Baseline and Intended Direction
|
||||
## Target State and Constraints
|
||||
|
||||
The review of revision `6d34602` covered all eleven
|
||||
[prompt definitions](../internal/prompts.md), their shared instructions, and
|
||||
the context and response handling for proposals, validation, and repair.
|
||||
It was a design review, not a measurement of live-model quality.
|
||||
Audita will use clearer proposal, validation, and repair prompts, actionable
|
||||
rejection feedback, and no model-reported confidence scores or thresholds.
|
||||
Repair will reconsider corrections on their merits and revise or withdraw them
|
||||
based on the supplied evidence. The initial review covered the eleven
|
||||
[prompt definitions](../internal/prompts.md) and their consumers at revision
|
||||
`6d34602`; it was a design review, not a live-model quality measurement.
|
||||
|
||||
The maintainer reports that nearly all module behavior was selected through
|
||||
practical experimentation. Treat that experience as evidence when reviewing the
|
||||
design: overlapping responsibilities and repeated passes are not inherently
|
||||
redundant. The deferred evaluation work does not imply an absence of empirical
|
||||
support for the existing module design. This experience informs ongoing choices;
|
||||
it does not lock in the current default. Frequently reevaluate the default module
|
||||
combination and sequence, selecting what has been observed to work best for most
|
||||
workloads.
|
||||
Preserve the following throughout implementation:
|
||||
|
||||
Retain the strengths: conservative transcript editing, explicit literal-span and
|
||||
deletion semantics, distinct glossary context and protection, full before/after
|
||||
validation, independent repair conversations with withdrawal, and native
|
||||
PromptKit/ConfigKit prompt replacement. Improve consistency across the three
|
||||
roles, useful rejection feedback, and maintainability. Remove model-reported
|
||||
confidence scores and confidence thresholds throughout the correction workflow.
|
||||
- **Editorial acceptance:** module focus guides requested work, while validators
|
||||
permit otherwise acceptable minor editorial cleanup across modules. Follow
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records
|
||||
the maintainer's finding that strict scope instructions also suppress desirable
|
||||
edits. Applicability, protection, and other substantive checks still apply.
|
||||
- **Correction mechanics:** retain literal-span and deletion semantics, distinct
|
||||
glossary context and protection, full before/after validation, and independent
|
||||
repair conversations with full history, withdrawal, and revalidation. Preserve
|
||||
native PromptKit/ConfigKit prompt replacement and the existing module structure.
|
||||
- **Module behavior:** keep `grammar` and its current requested scope; retain
|
||||
glossary and homophone modules with their useful overlap. Keep the current
|
||||
[default sequence](../config.md#pipeline-and-glossary-selections) for this pass
|
||||
and preserve configurable additional passes.
|
||||
- **Language:** retain English as the bundled default and useful existing campaign
|
||||
examples, supplemented with a few general examples. Broader language support is
|
||||
outside this pass.
|
||||
- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the
|
||||
full segment's assertion. Its targeted prompt changes are specified below.
|
||||
|
||||
## Discussion and Work Order
|
||||
Nearly all module behavior reflects the maintainer's practical experimentation.
|
||||
Two glossary passes have proved useful, and some workloads benefit from three or
|
||||
more. These observations inform the current choice without fixing a permanent
|
||||
number or order of passes. Frequently reevaluate the default module combination
|
||||
and sequence and select what has been observed to work best for most workloads.
|
||||
This ongoing maintenance does not depend on the deferred formal evaluation work.
|
||||
|
||||
### 1. Module Focus and Editorial Acceptance — Resolved
|
||||
## Implementation Sequence
|
||||
|
||||
The intentional distinction between requested module focus and acceptance of
|
||||
minor editorial cleanup is recorded in
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). The maintainer's
|
||||
testing found that strict scope instructions also suppress desirable edits.
|
||||
Apply the accepted distinction throughout the remaining prompt work.
|
||||
Work through the following steps in order. Keep schemas, consumers, prompt
|
||||
examples, and affected tests consistent as each change lands, and update the
|
||||
canonical documentation with the corresponding behavior. The final step checks
|
||||
the assembled result. Proceed without waiting for live-model evaluation evidence
|
||||
or a captured baseline.
|
||||
|
||||
### 2. Make Rejection Reasons Useful for Repair — Decided
|
||||
### 1. Remove Confidence Throughout the Workflow
|
||||
|
||||
Validator explanations now become repair feedback, but current instructions give
|
||||
little guidance about producing actionable reasons.
|
||||
Remove confidence entirely from proposals, validation, and repair. Audita has no
|
||||
established calibration that gives its model-reported scores reliable numerical
|
||||
meaning across transcripts and models. Proposal scores currently affect
|
||||
acceptance; validator scores are range-checked but do not determine approval.
|
||||
Retaining the numbers adds complexity and encourages repair toward a threshold.
|
||||
|
||||
**Decision:** retain the existing `reason` field. Instruct validators to explain
|
||||
rejections with a concise description of the specific problematic edit, the
|
||||
applicable criterion, and relevant supplied evidence, within their own remit.
|
||||
For editorial review, for example: removing "might" turns a tentative statement
|
||||
into a definite one.
|
||||
Validators identify the defect; the repair module owns revision or withdrawal.
|
||||
Respect the accepted minor editorial allowance, distinguish insufficient
|
||||
evidence from an established defect, and include distinct material issues when
|
||||
found without requiring an exhaustive analysis.
|
||||
|
||||
Approvals require no explanation: instruct the model to return `reason: ""`.
|
||||
The existing schema requires the field but permits an empty string, which the
|
||||
response handler already accepts. This reason convention itself needs no schema
|
||||
or response-handling changes; topic 3 separately removes confidence from those
|
||||
contracts. The rejection guidance is a prompt instruction;
|
||||
this decision adds no runtime rule rejecting blank explanations.
|
||||
|
||||
**Remaining work:** update the shared validation guidance and all three validator
|
||||
definitions with representative rejection examples and the empty-reason approval
|
||||
convention. Check rendered prompts and representative repair payloads to confirm
|
||||
that rejection feedback identifies defects without prescribing replacement
|
||||
corrections. Consider schema expansion only if a concrete need emerges.
|
||||
|
||||
### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided
|
||||
|
||||
**Decision:** remove confidence entirely from proposals, validation, and repair,
|
||||
including the threshold validator and its configuration. Audita has no established
|
||||
calibration that gives its model-reported scores a reliable numerical meaning
|
||||
across transcripts and models. Proposal scores currently affect acceptance;
|
||||
validator scores are range-checked but do not determine approval. Retaining these
|
||||
numbers adds complexity and encourages repairs aimed at meeting a threshold.
|
||||
The target workflow relies on substantive validation criteria and supplied
|
||||
evidence, without a replacement numerical or categorical confidence scale.
|
||||
|
||||
Repair instructions should ask the model to reassess a correction using the
|
||||
original transcript, supplied context, and rejection feedback. Revise when the
|
||||
evidence supports a correction that addresses the identified problems; withdraw
|
||||
when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal
|
||||
is a successful resolution that preserves the original text. Explain concrete
|
||||
uncertainty in rejection feedback where relevant, such as context that cannot
|
||||
distinguish two possible names. Preserve the module's requested focus, the
|
||||
accepted minor editorial allowance, the full-history conversation model, and
|
||||
the existing repair lifecycle and revalidation.
|
||||
|
||||
**Remaining work:**
|
||||
|
||||
- Remove confidence fields, instructions, and examples from all proposal,
|
||||
validation, and repair prompts and response schemas, and from their Go models,
|
||||
decoding, and validation paths.
|
||||
- Remove confidence fields from response schemas, Go models, decoding, and
|
||||
validation paths, together with corresponding prompt instructions and examples.
|
||||
- Remove the confidence-threshold validator, its registration and chain entries,
|
||||
root and stage threshold settings, and any corresponding CLI exposure.
|
||||
- Remove confidence-specific rejection feedback, reason codes, report/debug
|
||||
fields, and other plumbing wherever present. Update affected tests, fixtures,
|
||||
configuration examples, and canonical documentation when implementation lands.
|
||||
- Treat this as a deliberate contract change for configuration, custom prompts,
|
||||
and affected serialized outputs; provide migration guidance to remove obsolete
|
||||
fields and settings. Do not retain inert confidence fields or threshold knobs.
|
||||
- Verify that confidence-free proposals, validation decisions, and repair
|
||||
responses work end to end, while substantive validators, revision, withdrawal,
|
||||
and revalidation continue to operate as intended.
|
||||
fields, and other supporting code wherever present. Update affected tests,
|
||||
fixtures, and configuration examples.
|
||||
- Treat removal as a deliberate contract change for configuration, custom prompts,
|
||||
and affected serialized outputs. Retain no inert confidence fields or threshold
|
||||
knobs, and introduce no replacement numerical or categorical confidence scale.
|
||||
|
||||
Removing the proposal threshold can admit corrections it previously rejected.
|
||||
The deferred evaluation work in topic 5 can assess that effect when resumed;
|
||||
removal is the selected direction and does not depend on that work.
|
||||
That effect can be assessed when editorial evaluation resumes; it does not block
|
||||
the selected removal.
|
||||
|
||||
### 4. Simplify Instructions and Improve Examples — Decided
|
||||
### 2. Improve Validation Feedback
|
||||
|
||||
Restrictions and examples recur across proposal, validation, and repair assets;
|
||||
the grammar prompt repeats its word-substitution boundary several ways.
|
||||
Retain the existing `reason` field. Update shared validation guidance and all
|
||||
three validator definitions to request concise rejection explanations identifying
|
||||
the specific problematic edit, applicable criterion, and relevant supplied
|
||||
evidence within each validator's remit. For editorial review, for example:
|
||||
removing "might" turns a tentative statement into a definite one.
|
||||
|
||||
Validators identify defects; the repair module owns revision or withdrawal.
|
||||
Distinguish insufficient evidence from an established defect, respect the accepted
|
||||
editorial allowance, and include distinct material issues when found without
|
||||
requiring exhaustive analysis. Provide representative rejection examples.
|
||||
|
||||
Approvals need no explanation: instruct the model to return `reason: ""` and use
|
||||
that convention in approval examples. The field remains required and permits an
|
||||
empty string; this reason convention itself needs no schema or response-handling
|
||||
change. Explanation guidance is expressed in prompts, with no new runtime
|
||||
rule rejecting blank explanations. Consider schema expansion only if a concrete
|
||||
need emerges.
|
||||
|
||||
The meaning-reversal validator requires particular care. The maintainer has
|
||||
observed overly broad instructions causing it to reject virtually all corrections.
|
||||
Preserve its core instructions and make only small, targeted feedback and example
|
||||
changes, alongside confidence removal and the approval-reason convention:
|
||||
|
||||
- Detect reversal of the full segment's assertion, not every change in meaning.
|
||||
Changes to names, referents, or other information alone are not grounds for
|
||||
rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change
|
||||
that is not a reversal.
|
||||
- Assess the full before/after segment. An antonym appearing in the replacement
|
||||
does not by itself establish reversal.
|
||||
- Instruct rejection feedback to identify the specific opposition in the
|
||||
assertions; "this changes the meaning" is insufficient.
|
||||
- Approve corrections outside this narrow rejection criterion, leaving other
|
||||
validators to assess their own criteria. Preserve the guard even though it can
|
||||
exclude a genuine mistranscription correction that reverses the assertion;
|
||||
no contextual exception is selected in this pass.
|
||||
|
||||
### 3. Refine Repair Guidance
|
||||
|
||||
Ask the model to reassess the correction using the original transcript, supplied
|
||||
context, and rejection feedback:
|
||||
|
||||
- Revise when the evidence supports a correction that addresses the identified
|
||||
problems.
|
||||
- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain.
|
||||
Withdrawal is a successful resolution that preserves the original text.
|
||||
|
||||
Use concrete explanations of uncertainty where relevant, such as context that
|
||||
cannot distinguish two possible names. Preserve requested module focus and the
|
||||
accepted editorial allowance. Apply these instructions within the existing
|
||||
full-history repair lifecycle and revalidation, without adding new retry rules.
|
||||
|
||||
### 4. Consolidate Prompts, Descriptions, and Examples
|
||||
|
||||
Make a focused editing pass using the existing prompt fragment and module
|
||||
structure. Organize instructions around shared contracts (including response
|
||||
format, literal matching, and deletion semantics), module-specific requested work,
|
||||
and the distinct responsibilities of validation and repair.
|
||||
|
||||
**Decision:** make a focused editing pass using the existing module and prompt
|
||||
fragment structure. Organize instructions around shared contracts (including
|
||||
response format, literal matching, and deletion semantics), module-specific
|
||||
requested work, and the distinct responsibilities of validation and repair.
|
||||
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
||||
contradictions and unnecessary repetition, and keep each rendered prompt
|
||||
understandable on its own. Retain repetition across separate model calls when
|
||||
each call needs the instruction.
|
||||
each call needs the instruction. Preserve the meaning-reversal validator's tuned
|
||||
core wording as specified in step 2; broad simplification for consistency is not
|
||||
appropriate there. Shorter prompts alone are not the objective.
|
||||
|
||||
Apply the meaning-reversal decision in topic 6 as an exception to broad prompt
|
||||
simplification: preserve that validator's tuned core instructions and make only
|
||||
small, targeted changes to feedback and examples, alongside the agreed confidence
|
||||
removal and approval-reason convention.
|
||||
Clarify descriptions and examples as follows:
|
||||
|
||||
Use short contextual contrasts to clarify difficult judgments, such as an
|
||||
accidental restart versus deliberate emphasis, or a glossary match supported by
|
||||
context versus a merely similar spelling. Examples should illustrate how context
|
||||
informs the decision, not create unconditional editing rules. Preserve the
|
||||
distinction between requested module focus and acceptance of otherwise appropriate
|
||||
minor editorial cleanup. Shorter prompts alone are not the objective.
|
||||
- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work
|
||||
as conservative punctuation, capitalization, spacing, and whole-word `a`/`an`
|
||||
article cleanup. Preserve the current scope and cross-module editorial allowance.
|
||||
- Describe glossary stages as focusing on known terms supported by selected
|
||||
glossaries, and homophone stages as looking more broadly for plausible
|
||||
spoken-word mistranscriptions, including glossary-supported corrections. Do not
|
||||
exclude an otherwise supported homophone correction because it targets a
|
||||
glossary term.
|
||||
- Use short contextual contrasts for difficult judgments, such as accidental
|
||||
restarts versus deliberate emphasis and context-supported glossary matches
|
||||
versus merely similar spellings. Examples illustrate judgments rather than
|
||||
establish unconditional editing rules.
|
||||
- Retain useful campaign examples and add a few general examples. Use broadly
|
||||
applicable wording for names, terminology, and glossary entries without asserting
|
||||
D&D specificity or adding a disclaimer about it.
|
||||
|
||||
**Remaining work:** revise the proposal, validation, and repair assets and shared
|
||||
fragments consistently with topics 1–3, retaining the existing module structure.
|
||||
Check rendering, response-schema compatibility, and integration behavior as
|
||||
appropriate. Proceed with these revisions before waiting for live-model
|
||||
evaluation evidence or a captured baseline. Topic 5 is deferred and is not a
|
||||
prerequisite for this editing pass.
|
||||
### 5. Complete Integration Checks and Documentation
|
||||
|
||||
### 5. Evaluate Editorial Quality — Deferred
|
||||
Follow the [testing policy](../policy/testing.md) and the
|
||||
[development validation guidance](../development.md#baseline-validation). Verify
|
||||
the assembled workflow, including:
|
||||
|
||||
**Decision:** defer this work for now. Evaluation corpus preparation, live-model
|
||||
comparisons, and an evaluation workflow are outside the current implementation
|
||||
scope. This deferral does not block topics 2–4 or the remaining product
|
||||
discussions; ordinary implementation checks still apply. The following direction
|
||||
is retained for reconsideration when evaluation work resumes.
|
||||
- Confidence-free proposal, validation, and repair responses work through their
|
||||
schemas and consumers; substantive validators remain effective in the workflow.
|
||||
- Rendered prompts and examples match the revised contracts, including empty
|
||||
approval reasons and rejection feedback carried into repair requests.
|
||||
- Revision, withdrawal, and full revalidation retain their existing mechanics.
|
||||
|
||||
Rendering, schema, and controlled-response tests establish integration behavior,
|
||||
but do not establish the quality of model editorial judgments.
|
||||
Review the actual prompt wording and representative repair payloads to check the
|
||||
selected responsibilities and the narrow meaning-reversal criterion. Offline
|
||||
tests verify contracts and integration; they do not establish live-model
|
||||
editorial quality. Complete the appropriate package checks, build, and full test
|
||||
suite for the changes, without introducing paid calls into the default suite.
|
||||
|
||||
**Future recommendation:** assemble a modest evaluation set containing both needed edits
|
||||
and text that should remain unchanged. Include ambiguous glossary matches,
|
||||
intentional repetition, meaningful uses of apparent filler, punctuation that
|
||||
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
|
||||
missed corrections, validator disagreement, and repair outcomes using independent
|
||||
editorial assessment; validator approval alone is insufficient evidence.
|
||||
Include a comparison against the prior confidence-threshold behavior to identify
|
||||
changes in both useful corrections and erroneous edits after confidence removal.
|
||||
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
||||
[prompt](../internal/prompts.md), [module](../internal/modules.md),
|
||||
[validator](../internal/validators.md), and
|
||||
[output](../integrations/output-files.md) documentation where affected, together
|
||||
with maintained examples. Provide migration guidance to remove obsolete confidence
|
||||
settings and fields from configuration, custom prompts, and affected output
|
||||
consumers. Check links, examples, and whitespace, and remove stale descriptions
|
||||
of confidence-based behavior.
|
||||
|
||||
When this work resumes, discuss representative models, expected outcomes, and
|
||||
how to run evaluations. Use the prior repository revisions for
|
||||
before/after comparisons when that work proceeds; a pre-change evaluation run is
|
||||
not required. Keep model-dependent evaluation separate from the default offline
|
||||
test suite; no numerical quality gate is selected here.
|
||||
## Deferred Editorial Evaluation
|
||||
|
||||
### 6. Revisit Broader Product Choices
|
||||
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
||||
are outside this implementation scope. No baseline run or numerical quality gate
|
||||
is required before the agreed changes proceed.
|
||||
|
||||
The following decisions guide this pass without waiting for the deferred
|
||||
evaluation work:
|
||||
When this work resumes, consider a modest collection of needed corrections and
|
||||
text that should remain unchanged: ambiguous glossary matches, intentional
|
||||
repetition, meaningful apparent filler, punctuation affecting meaning, and repairs
|
||||
that should be withdrawn. Compare unnecessary edits, missed corrections,
|
||||
validator disagreement, and repair outcomes using independent editorial judgment;
|
||||
validator approval alone is insufficient evidence.
|
||||
|
||||
1. **Grammar naming and scope — Decided:** retain the `grammar` name, module key,
|
||||
and corresponding prompt IDs. Clarify its description as conservative
|
||||
punctuation, capitalization, spacing, and whole-word `a`/`an` article cleanup.
|
||||
Preserve its current requested scope and the accepted cross-module minor
|
||||
editorial allowance. Update the relevant module descriptions and canonical
|
||||
documentation during implementation; no rename or broader grammar-correction
|
||||
remit is planned.
|
||||
2. **Glossary/homophone overlap and defaults — Decided:** retain both modules and
|
||||
their useful overlap. Keep the existing default sequence documented in
|
||||
[configuration](../config.md#pipeline-and-glossary-selections) for this pass.
|
||||
Frequently reevaluate the default and set it to the combination and sequence
|
||||
of modules observed to work best for most workloads. The maintainer has
|
||||
observed that two glossary passes help and some workloads benefit from three
|
||||
or more; these observations inform the current choice without fixing a
|
||||
permanent number or order of passes. Preserve operators' ability to configure
|
||||
additional passes. Clarify the complementary requested roles: glossary stages
|
||||
focus on known terms supported by selected glossaries; homophone stages look
|
||||
more broadly for plausible spoken-word mistranscriptions and may also use
|
||||
glossary evidence. Do not exclude an otherwise supported homophone correction
|
||||
merely because it targets a glossary term. No module consolidation is planned
|
||||
in this pass. Reevaluating defaults does not depend on the deferred formal
|
||||
evaluation work in topic 5.
|
||||
3. **Language and examples — Decided:** retain English as the bundled default.
|
||||
Keep useful existing campaign examples and add a few more general examples
|
||||
during the prompt editing pass. Use broadly applicable wording for names,
|
||||
terminology, and glossary entries, without asserting D&D specificity or
|
||||
adding a disclaimer about it. Broader language support is outside this pass.
|
||||
4. **Meaning-reversal policy — Decided:** retain the narrow guard against reversal
|
||||
of the full segment's assertion, rather than rejection of any change in
|
||||
meaning. The maintainer has observed that overly broad instructions cause this
|
||||
validator to reject virtually all corrections; its wording requires unusually
|
||||
precise tuning. Preserve the existing core instructions and make small,
|
||||
targeted changes, principally improving rejection feedback and adding a
|
||||
contrasting example. Changes to names, referents, or other information are
|
||||
not grounds for rejection by this validator unless they reverse the assertion:
|
||||
"We met Alice" becoming "We met Alex" changes meaning but is not a reversal.
|
||||
Assess the full before/after segment; the presence of an antonym in a
|
||||
replacement does not by itself establish reversal. Instruct rejection feedback
|
||||
to identify the specific opposition in the assertions; "this changes the
|
||||
meaning" is insufficient. Approve corrections outside this narrow rejection
|
||||
criterion, leaving other validators to assess their own criteria. These are
|
||||
prompt instructions, not new runtime checks on explanation text. Preserve the
|
||||
safeguard even though it can exclude a genuine mistranscription correction
|
||||
that reverses the assertion; no contextual exception is selected in this pass.
|
||||
Discuss representative models, expected outcomes, and an evaluation workflow at
|
||||
that time. Prior repository revisions can support before/after comparisons,
|
||||
including the effects of removing confidence thresholds. Keep model-dependent
|
||||
evaluation separate from the default offline test suite.
|
||||
|
||||
Reference in New Issue
Block a user