2026-11-11 14:05:20 +00:00
|
|
|
|
# Prompt Quality Roadmap
|
|
|
|
|
|
|
|
|
|
|
|
**Updated:** 2026-11-11
|
|
|
|
|
|
|
2026-11-11 15:33:08 +00:00
|
|
|
|
**Status:** Topic 1 resolved in ADR 0003; topics 2–4 decided, with implementation
|
2026-11-11 15:43:26 +00:00
|
|
|
|
pending; topic 5 deferred; grammar naming and scope in topic 6 decided, with
|
|
|
|
|
|
description updates pending; the remaining topic 6 choices await discussion.
|
2026-11-11 14:19:06 +00:00
|
|
|
|
Work through the remaining topics in order. Their recommendations are starting
|
|
|
|
|
|
points for discussion, not newly adopted restrictions or guarantees.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
## Baseline and Intended Direction
|
|
|
|
|
|
|
|
|
|
|
|
The review of revision `6d34602` covered all eleven
|
|
|
|
|
|
[prompt definitions](../internal/prompts.md), their shared instructions, and
|
|
|
|
|
|
the context and response handling for proposals, validation, and repair.
|
|
|
|
|
|
It was a design review, not a measurement of live-model quality.
|
|
|
|
|
|
|
|
|
|
|
|
Retain the strengths: conservative transcript editing, explicit literal-span and
|
|
|
|
|
|
deletion semantics, distinct glossary context and protection, full before/after
|
|
|
|
|
|
validation, independent repair conversations with withdrawal, and native
|
|
|
|
|
|
PromptKit/ConfigKit prompt replacement. Improve consistency across the three
|
2026-11-11 15:18:20 +00:00
|
|
|
|
roles, useful rejection feedback, and maintainability. Remove model-reported
|
|
|
|
|
|
confidence scores and confidence thresholds throughout the correction workflow.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
## Discussion and Work Order
|
|
|
|
|
|
|
2026-11-11 14:19:06 +00:00
|
|
|
|
### 1. Module Focus and Editorial Acceptance — Resolved
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 14:19:06 +00:00
|
|
|
|
The intentional distinction between requested module focus and acceptance of
|
|
|
|
|
|
minor editorial cleanup is recorded in
|
|
|
|
|
|
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). The maintainer's
|
|
|
|
|
|
testing found that strict scope instructions also suppress desirable edits.
|
|
|
|
|
|
Apply the accepted distinction throughout the remaining prompt work.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 14:59:10 +00:00
|
|
|
|
### 2. Make Rejection Reasons Useful for Repair — Decided
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
Validator explanations now become repair feedback, but current instructions give
|
|
|
|
|
|
little guidance about producing actionable reasons.
|
|
|
|
|
|
|
2026-11-11 14:59:10 +00:00
|
|
|
|
**Decision:** retain the existing `reason` field. Instruct validators to explain
|
|
|
|
|
|
rejections with a concise description of the specific problematic edit, the
|
|
|
|
|
|
applicable criterion, and relevant supplied evidence, within their own remit.
|
|
|
|
|
|
For example: removing "might" turns a tentative statement into a definite one.
|
|
|
|
|
|
Validators identify the defect; the repair module owns revision or withdrawal.
|
|
|
|
|
|
Respect the accepted minor editorial allowance, distinguish insufficient
|
|
|
|
|
|
evidence from an established defect, and include distinct material issues when
|
|
|
|
|
|
found without requiring an exhaustive analysis.
|
|
|
|
|
|
|
|
|
|
|
|
Approvals require no explanation: instruct the model to return `reason: ""`.
|
|
|
|
|
|
The existing schema requires the field but permits an empty string, which the
|
2026-11-11 15:18:20 +00:00
|
|
|
|
response handler already accepts. This reason convention itself needs no schema
|
|
|
|
|
|
or response-handling changes; topic 3 separately removes confidence from those
|
|
|
|
|
|
contracts. The rejection guidance is a prompt instruction;
|
2026-11-11 14:59:10 +00:00
|
|
|
|
this decision adds no runtime rule rejecting blank explanations.
|
|
|
|
|
|
|
|
|
|
|
|
**Remaining work:** update the shared validation guidance and all three validator
|
|
|
|
|
|
definitions with representative rejection examples and the empty-reason approval
|
|
|
|
|
|
convention. Check rendered prompts and representative repair payloads to confirm
|
|
|
|
|
|
that rejection feedback identifies defects without prescribing replacement
|
|
|
|
|
|
corrections. Consider schema expansion only if a concrete need emerges.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 15:18:20 +00:00
|
|
|
|
### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided
|
|
|
|
|
|
|
|
|
|
|
|
**Decision:** remove confidence entirely from proposals, validation, and repair,
|
|
|
|
|
|
including the threshold validator and its configuration. Audita has no established
|
|
|
|
|
|
calibration that gives its model-reported scores a reliable numerical meaning
|
|
|
|
|
|
across transcripts and models. Proposal scores currently affect acceptance;
|
|
|
|
|
|
validator scores are range-checked but do not determine approval. Retaining these
|
|
|
|
|
|
numbers adds complexity and encourages repairs aimed at meeting a threshold.
|
|
|
|
|
|
The target workflow relies on substantive validation criteria and supplied
|
|
|
|
|
|
evidence, without a replacement numerical or categorical confidence scale.
|
|
|
|
|
|
|
|
|
|
|
|
Repair instructions should ask the model to reassess a correction using the
|
|
|
|
|
|
original transcript, supplied context, and rejection feedback. Revise when the
|
|
|
|
|
|
evidence supports a correction that addresses the identified problems; withdraw
|
|
|
|
|
|
when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal
|
|
|
|
|
|
is a successful resolution that preserves the original text. Explain concrete
|
|
|
|
|
|
uncertainty in rejection feedback where relevant, such as context that cannot
|
|
|
|
|
|
distinguish two possible names. Preserve the module's requested focus, the
|
|
|
|
|
|
accepted minor editorial allowance, the full-history conversation model, and
|
|
|
|
|
|
the existing repair lifecycle and revalidation.
|
|
|
|
|
|
|
|
|
|
|
|
**Remaining work:**
|
|
|
|
|
|
|
|
|
|
|
|
- Remove confidence fields, instructions, and examples from all proposal,
|
|
|
|
|
|
validation, and repair prompts and response schemas, and from their Go models,
|
|
|
|
|
|
decoding, and validation paths.
|
|
|
|
|
|
- Remove the confidence-threshold validator, its registration and chain entries,
|
|
|
|
|
|
root and stage threshold settings, and any corresponding CLI exposure.
|
|
|
|
|
|
- Remove confidence-specific rejection feedback, reason codes, report/debug
|
|
|
|
|
|
fields, and other plumbing wherever present. Update affected tests, fixtures,
|
|
|
|
|
|
configuration examples, and canonical documentation when implementation lands.
|
|
|
|
|
|
- Treat this as a deliberate contract change for configuration, custom prompts,
|
|
|
|
|
|
and affected serialized outputs; provide migration guidance to remove obsolete
|
|
|
|
|
|
fields and settings. Do not retain inert confidence fields or threshold knobs.
|
|
|
|
|
|
- Verify that confidence-free proposals, validation decisions, and repair
|
|
|
|
|
|
responses work end to end, while substantive validators, revision, withdrawal,
|
|
|
|
|
|
and revalidation continue to operate as intended.
|
|
|
|
|
|
|
|
|
|
|
|
Removing the proposal threshold can admit corrections it previously rejected.
|
2026-11-11 15:41:23 +00:00
|
|
|
|
The deferred evaluation work in topic 5 can assess that effect when resumed;
|
|
|
|
|
|
removal is the selected direction and does not depend on that work.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 15:33:08 +00:00
|
|
|
|
### 4. Simplify Instructions and Improve Examples — Decided
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
Restrictions and examples recur across proposal, validation, and repair assets;
|
|
|
|
|
|
the grammar prompt repeats its word-substitution boundary several ways.
|
|
|
|
|
|
|
2026-11-11 15:33:08 +00:00
|
|
|
|
**Decision:** make a focused editing pass using the existing module and prompt
|
|
|
|
|
|
fragment structure. Organize instructions around shared contracts (including
|
|
|
|
|
|
response format, literal matching, and deletion semantics), module-specific
|
|
|
|
|
|
requested work, and the distinct responsibilities of validation and repair.
|
|
|
|
|
|
Consolidate duplicated rules where small shared fragments prevent drift, remove
|
|
|
|
|
|
contradictions and unnecessary repetition, and keep each rendered prompt
|
|
|
|
|
|
understandable on its own. Retain repetition across separate model calls when
|
|
|
|
|
|
each call needs the instruction.
|
|
|
|
|
|
|
|
|
|
|
|
Use short contextual contrasts to clarify difficult judgments, such as an
|
|
|
|
|
|
accidental restart versus deliberate emphasis, or a glossary match supported by
|
|
|
|
|
|
context versus a merely similar spelling. Examples should illustrate how context
|
|
|
|
|
|
informs the decision, not create unconditional editing rules. Preserve the
|
|
|
|
|
|
distinction between requested module focus and acceptance of otherwise appropriate
|
|
|
|
|
|
minor editorial cleanup. Shorter prompts alone are not the objective.
|
|
|
|
|
|
|
|
|
|
|
|
**Remaining work:** revise the proposal, validation, and repair assets and shared
|
|
|
|
|
|
fragments consistently with topics 1–3, retaining the existing module structure.
|
|
|
|
|
|
Check rendering, response-schema compatibility, and integration behavior as
|
|
|
|
|
|
appropriate. Proceed with these revisions before waiting for live-model
|
2026-11-11 15:41:23 +00:00
|
|
|
|
evaluation evidence or a captured baseline. Topic 5 is deferred and is not a
|
|
|
|
|
|
prerequisite for this editing pass.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 15:41:23 +00:00
|
|
|
|
### 5. Evaluate Editorial Quality — Deferred
|
|
|
|
|
|
|
|
|
|
|
|
**Decision:** defer this work for now. Evaluation corpus preparation, live-model
|
|
|
|
|
|
comparisons, and an evaluation workflow are outside the current implementation
|
|
|
|
|
|
scope. This deferral does not block topics 2–4 or the remaining product
|
|
|
|
|
|
discussions; ordinary implementation checks still apply. The following direction
|
|
|
|
|
|
is retained for reconsideration when evaluation work resumes.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
Rendering, schema, and controlled-response tests establish integration behavior,
|
|
|
|
|
|
but do not establish the quality of model editorial judgments.
|
|
|
|
|
|
|
2026-11-11 15:41:23 +00:00
|
|
|
|
**Future recommendation:** assemble a modest evaluation set containing both needed edits
|
2026-11-11 14:05:20 +00:00
|
|
|
|
and text that should remain unchanged. Include ambiguous glossary matches,
|
|
|
|
|
|
intentional repetition, meaningful uses of apparent filler, punctuation that
|
|
|
|
|
|
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
|
|
|
|
|
|
missed corrections, validator disagreement, and repair outcomes using independent
|
|
|
|
|
|
editorial assessment; validator approval alone is insufficient evidence.
|
2026-11-11 15:18:20 +00:00
|
|
|
|
Include a comparison against the prior confidence-threshold behavior to identify
|
|
|
|
|
|
changes in both useful corrections and erroneous edits after confidence removal.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 15:41:23 +00:00
|
|
|
|
When this work resumes, discuss representative models, expected outcomes, and
|
|
|
|
|
|
how to run evaluations. Use the prior repository revisions for
|
2026-11-11 15:33:08 +00:00
|
|
|
|
before/after comparisons when that work proceeds; a pre-change evaluation run is
|
|
|
|
|
|
not required. Keep model-dependent evaluation separate from the default offline
|
|
|
|
|
|
test suite; no numerical quality gate is selected here.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
|
|
|
|
|
### 6. Revisit Broader Product Choices
|
|
|
|
|
|
|
2026-11-11 15:41:23 +00:00
|
|
|
|
Use the preceding decisions and any available evidence to discuss these in order,
|
|
|
|
|
|
without waiting for the deferred evaluation work:
|
2026-11-11 14:05:20 +00:00
|
|
|
|
|
2026-11-11 15:43:26 +00:00
|
|
|
|
1. **Grammar naming and scope — Decided:** retain the `grammar` name, module key,
|
|
|
|
|
|
and corresponding prompt IDs. Clarify its description as conservative
|
|
|
|
|
|
punctuation, capitalization, spacing, and whole-word `a`/`an` article cleanup.
|
|
|
|
|
|
Preserve its current requested scope and the accepted cross-module minor
|
|
|
|
|
|
editorial allowance. Update the relevant module descriptions and canonical
|
|
|
|
|
|
documentation during implementation; no rename or broader grammar-correction
|
|
|
|
|
|
remit is planned.
|
2026-11-11 14:05:20 +00:00
|
|
|
|
2. **Glossary/homophone overlap:** assess the benefit of their overlapping remit
|
|
|
|
|
|
and repeated glossary passes before changing the default sequence.
|
|
|
|
|
|
3. **Language and examples:** decide whether spoken English and campaign-specific
|
|
|
|
|
|
examples are the right bundled defaults, and how broadly those defaults
|
|
|
|
|
|
should apply.
|
|
|
|
|
|
4. **Meaning-reversal policy:** the guard can also exclude a genuine correction
|
|
|
|
|
|
of a mistranscription. Initially retain the conservative default while
|
|
|
|
|
|
explicitly evaluating that tradeoff.
|
|
|
|
|
|
|
2026-11-11 15:43:26 +00:00
|
|
|
|
The remaining items are product discussions, not selected changes to module
|
|
|
|
|
|
sequencing, language support, or validation policy.
|