diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index a3fff9b..da7090e 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -2,218 +2,203 @@ **Updated:** 2026-11-11 -**Status:** Topic 1 resolved in ADR 0003; topics 2–4 and 6 decided, with -implementation pending; topic 5 deferred. No open decisions remain in the -current scope. Topic 5 retains recommendations for future discussion. +**Status:** Decisions settled; implementation pending. Editorial evaluation is +deferred. No open decisions remain in the current scope. -## Baseline and Intended Direction +## Target State and Constraints -The review of revision `6d34602` covered all eleven -[prompt definitions](../internal/prompts.md), their shared instructions, and -the context and response handling for proposals, validation, and repair. -It was a design review, not a measurement of live-model quality. +Audita will use clearer proposal, validation, and repair prompts, actionable +rejection feedback, and no model-reported confidence scores or thresholds. +Repair will reconsider corrections on their merits and revise or withdraw them +based on the supplied evidence. The initial review covered the eleven +[prompt definitions](../internal/prompts.md) and their consumers at revision +`6d34602`; it was a design review, not a live-model quality measurement. -The maintainer reports that nearly all module behavior was selected through -practical experimentation. Treat that experience as evidence when reviewing the -design: overlapping responsibilities and repeated passes are not inherently -redundant. The deferred evaluation work does not imply an absence of empirical -support for the existing module design. This experience informs ongoing choices; -it does not lock in the current default. Frequently reevaluate the default module -combination and sequence, selecting what has been observed to work best for most -workloads. +Preserve the following throughout implementation: -Retain the strengths: conservative transcript editing, explicit literal-span and -deletion semantics, distinct glossary context and protection, full before/after -validation, independent repair conversations with withdrawal, and native -PromptKit/ConfigKit prompt replacement. Improve consistency across the three -roles, useful rejection feedback, and maintainability. Remove model-reported -confidence scores and confidence thresholds throughout the correction workflow. +- **Editorial acceptance:** module focus guides requested work, while validators + permit otherwise acceptable minor editorial cleanup across modules. Follow + [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records + the maintainer's finding that strict scope instructions also suppress desirable + edits. Applicability, protection, and other substantive checks still apply. +- **Correction mechanics:** retain literal-span and deletion semantics, distinct + glossary context and protection, full before/after validation, and independent + repair conversations with full history, withdrawal, and revalidation. Preserve + native PromptKit/ConfigKit prompt replacement and the existing module structure. +- **Module behavior:** keep `grammar` and its current requested scope; retain + glossary and homophone modules with their useful overlap. Keep the current + [default sequence](../config.md#pipeline-and-glossary-selections) for this pass + and preserve configurable additional passes. +- **Language:** retain English as the bundled default and useful existing campaign + examples, supplemented with a few general examples. Broader language support is + outside this pass. +- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the + full segment's assertion. Its targeted prompt changes are specified below. -## Discussion and Work Order +Nearly all module behavior reflects the maintainer's practical experimentation. +Two glossary passes have proved useful, and some workloads benefit from three or +more. These observations inform the current choice without fixing a permanent +number or order of passes. Frequently reevaluate the default module combination +and sequence and select what has been observed to work best for most workloads. +This ongoing maintenance does not depend on the deferred formal evaluation work. -### 1. Module Focus and Editorial Acceptance — Resolved +## Implementation Sequence -The intentional distinction between requested module focus and acceptance of -minor editorial cleanup is recorded in -[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). The maintainer's -testing found that strict scope instructions also suppress desirable edits. -Apply the accepted distinction throughout the remaining prompt work. +Work through the following steps in order. Keep schemas, consumers, prompt +examples, and affected tests consistent as each change lands, and update the +canonical documentation with the corresponding behavior. The final step checks +the assembled result. Proceed without waiting for live-model evaluation evidence +or a captured baseline. -### 2. Make Rejection Reasons Useful for Repair — Decided +### 1. Remove Confidence Throughout the Workflow -Validator explanations now become repair feedback, but current instructions give -little guidance about producing actionable reasons. +Remove confidence entirely from proposals, validation, and repair. Audita has no +established calibration that gives its model-reported scores reliable numerical +meaning across transcripts and models. Proposal scores currently affect +acceptance; validator scores are range-checked but do not determine approval. +Retaining the numbers adds complexity and encourages repair toward a threshold. -**Decision:** retain the existing `reason` field. Instruct validators to explain -rejections with a concise description of the specific problematic edit, the -applicable criterion, and relevant supplied evidence, within their own remit. -For editorial review, for example: removing "might" turns a tentative statement -into a definite one. -Validators identify the defect; the repair module owns revision or withdrawal. -Respect the accepted minor editorial allowance, distinguish insufficient -evidence from an established defect, and include distinct material issues when -found without requiring an exhaustive analysis. - -Approvals require no explanation: instruct the model to return `reason: ""`. -The existing schema requires the field but permits an empty string, which the -response handler already accepts. This reason convention itself needs no schema -or response-handling changes; topic 3 separately removes confidence from those -contracts. The rejection guidance is a prompt instruction; -this decision adds no runtime rule rejecting blank explanations. - -**Remaining work:** update the shared validation guidance and all three validator -definitions with representative rejection examples and the empty-reason approval -convention. Check rendered prompts and representative repair payloads to confirm -that rejection feedback identifies defects without prescribing replacement -corrections. Consider schema expansion only if a concrete need emerges. - -### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided - -**Decision:** remove confidence entirely from proposals, validation, and repair, -including the threshold validator and its configuration. Audita has no established -calibration that gives its model-reported scores a reliable numerical meaning -across transcripts and models. Proposal scores currently affect acceptance; -validator scores are range-checked but do not determine approval. Retaining these -numbers adds complexity and encourages repairs aimed at meeting a threshold. -The target workflow relies on substantive validation criteria and supplied -evidence, without a replacement numerical or categorical confidence scale. - -Repair instructions should ask the model to reassess a correction using the -original transcript, supplied context, and rejection feedback. Revise when the -evidence supports a correction that addresses the identified problems; withdraw -when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal -is a successful resolution that preserves the original text. Explain concrete -uncertainty in rejection feedback where relevant, such as context that cannot -distinguish two possible names. Preserve the module's requested focus, the -accepted minor editorial allowance, the full-history conversation model, and -the existing repair lifecycle and revalidation. - -**Remaining work:** - -- Remove confidence fields, instructions, and examples from all proposal, - validation, and repair prompts and response schemas, and from their Go models, - decoding, and validation paths. +- Remove confidence fields from response schemas, Go models, decoding, and + validation paths, together with corresponding prompt instructions and examples. - Remove the confidence-threshold validator, its registration and chain entries, root and stage threshold settings, and any corresponding CLI exposure. - Remove confidence-specific rejection feedback, reason codes, report/debug - fields, and other plumbing wherever present. Update affected tests, fixtures, - configuration examples, and canonical documentation when implementation lands. -- Treat this as a deliberate contract change for configuration, custom prompts, - and affected serialized outputs; provide migration guidance to remove obsolete - fields and settings. Do not retain inert confidence fields or threshold knobs. -- Verify that confidence-free proposals, validation decisions, and repair - responses work end to end, while substantive validators, revision, withdrawal, - and revalidation continue to operate as intended. + fields, and other supporting code wherever present. Update affected tests, + fixtures, and configuration examples. +- Treat removal as a deliberate contract change for configuration, custom prompts, + and affected serialized outputs. Retain no inert confidence fields or threshold + knobs, and introduce no replacement numerical or categorical confidence scale. Removing the proposal threshold can admit corrections it previously rejected. -The deferred evaluation work in topic 5 can assess that effect when resumed; -removal is the selected direction and does not depend on that work. +That effect can be assessed when editorial evaluation resumes; it does not block +the selected removal. -### 4. Simplify Instructions and Improve Examples — Decided +### 2. Improve Validation Feedback -Restrictions and examples recur across proposal, validation, and repair assets; -the grammar prompt repeats its word-substitution boundary several ways. +Retain the existing `reason` field. Update shared validation guidance and all +three validator definitions to request concise rejection explanations identifying +the specific problematic edit, applicable criterion, and relevant supplied +evidence within each validator's remit. For editorial review, for example: +removing "might" turns a tentative statement into a definite one. + +Validators identify defects; the repair module owns revision or withdrawal. +Distinguish insufficient evidence from an established defect, respect the accepted +editorial allowance, and include distinct material issues when found without +requiring exhaustive analysis. Provide representative rejection examples. + +Approvals need no explanation: instruct the model to return `reason: ""` and use +that convention in approval examples. The field remains required and permits an +empty string; this reason convention itself needs no schema or response-handling +change. Explanation guidance is expressed in prompts, with no new runtime +rule rejecting blank explanations. Consider schema expansion only if a concrete +need emerges. + +The meaning-reversal validator requires particular care. The maintainer has +observed overly broad instructions causing it to reject virtually all corrections. +Preserve its core instructions and make only small, targeted feedback and example +changes, alongside confidence removal and the approval-reason convention: + +- Detect reversal of the full segment's assertion, not every change in meaning. + Changes to names, referents, or other information alone are not grounds for + rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change + that is not a reversal. +- Assess the full before/after segment. An antonym appearing in the replacement + does not by itself establish reversal. +- Instruct rejection feedback to identify the specific opposition in the + assertions; "this changes the meaning" is insufficient. +- Approve corrections outside this narrow rejection criterion, leaving other + validators to assess their own criteria. Preserve the guard even though it can + exclude a genuine mistranscription correction that reverses the assertion; + no contextual exception is selected in this pass. + +### 3. Refine Repair Guidance + +Ask the model to reassess the correction using the original transcript, supplied +context, and rejection feedback: + +- Revise when the evidence supports a correction that addresses the identified + problems. +- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain. + Withdrawal is a successful resolution that preserves the original text. + +Use concrete explanations of uncertainty where relevant, such as context that +cannot distinguish two possible names. Preserve requested module focus and the +accepted editorial allowance. Apply these instructions within the existing +full-history repair lifecycle and revalidation, without adding new retry rules. + +### 4. Consolidate Prompts, Descriptions, and Examples + +Make a focused editing pass using the existing prompt fragment and module +structure. Organize instructions around shared contracts (including response +format, literal matching, and deletion semantics), module-specific requested work, +and the distinct responsibilities of validation and repair. -**Decision:** make a focused editing pass using the existing module and prompt -fragment structure. Organize instructions around shared contracts (including -response format, literal matching, and deletion semantics), module-specific -requested work, and the distinct responsibilities of validation and repair. Consolidate duplicated rules where small shared fragments prevent drift, remove contradictions and unnecessary repetition, and keep each rendered prompt understandable on its own. Retain repetition across separate model calls when -each call needs the instruction. +each call needs the instruction. Preserve the meaning-reversal validator's tuned +core wording as specified in step 2; broad simplification for consistency is not +appropriate there. Shorter prompts alone are not the objective. -Apply the meaning-reversal decision in topic 6 as an exception to broad prompt -simplification: preserve that validator's tuned core instructions and make only -small, targeted changes to feedback and examples, alongside the agreed confidence -removal and approval-reason convention. +Clarify descriptions and examples as follows: -Use short contextual contrasts to clarify difficult judgments, such as an -accidental restart versus deliberate emphasis, or a glossary match supported by -context versus a merely similar spelling. Examples should illustrate how context -informs the decision, not create unconditional editing rules. Preserve the -distinction between requested module focus and acceptance of otherwise appropriate -minor editorial cleanup. Shorter prompts alone are not the objective. +- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work + as conservative punctuation, capitalization, spacing, and whole-word `a`/`an` + article cleanup. Preserve the current scope and cross-module editorial allowance. +- Describe glossary stages as focusing on known terms supported by selected + glossaries, and homophone stages as looking more broadly for plausible + spoken-word mistranscriptions, including glossary-supported corrections. Do not + exclude an otherwise supported homophone correction because it targets a + glossary term. +- Use short contextual contrasts for difficult judgments, such as accidental + restarts versus deliberate emphasis and context-supported glossary matches + versus merely similar spellings. Examples illustrate judgments rather than + establish unconditional editing rules. +- Retain useful campaign examples and add a few general examples. Use broadly + applicable wording for names, terminology, and glossary entries without asserting + D&D specificity or adding a disclaimer about it. -**Remaining work:** revise the proposal, validation, and repair assets and shared -fragments consistently with topics 1–3, retaining the existing module structure. -Check rendering, response-schema compatibility, and integration behavior as -appropriate. Proceed with these revisions before waiting for live-model -evaluation evidence or a captured baseline. Topic 5 is deferred and is not a -prerequisite for this editing pass. +### 5. Complete Integration Checks and Documentation -### 5. Evaluate Editorial Quality — Deferred +Follow the [testing policy](../policy/testing.md) and the +[development validation guidance](../development.md#baseline-validation). Verify +the assembled workflow, including: -**Decision:** defer this work for now. Evaluation corpus preparation, live-model -comparisons, and an evaluation workflow are outside the current implementation -scope. This deferral does not block topics 2–4 or the remaining product -discussions; ordinary implementation checks still apply. The following direction -is retained for reconsideration when evaluation work resumes. +- Confidence-free proposal, validation, and repair responses work through their + schemas and consumers; substantive validators remain effective in the workflow. +- Rendered prompts and examples match the revised contracts, including empty + approval reasons and rejection feedback carried into repair requests. +- Revision, withdrawal, and full revalidation retain their existing mechanics. -Rendering, schema, and controlled-response tests establish integration behavior, -but do not establish the quality of model editorial judgments. +Review the actual prompt wording and representative repair payloads to check the +selected responsibilities and the narrow meaning-reversal criterion. Offline +tests verify contracts and integration; they do not establish live-model +editorial quality. Complete the appropriate package checks, build, and full test +suite for the changes, without introducing paid calls into the default suite. -**Future recommendation:** assemble a modest evaluation set containing both needed edits -and text that should remain unchanged. Include ambiguous glossary matches, -intentional repetition, meaningful uses of apparent filler, punctuation that -changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, -missed corrections, validator disagreement, and repair outcomes using independent -editorial assessment; validator approval alone is insufficient evidence. -Include a comparison against the prior confidence-threshold behavior to identify -changes in both useful corrections and erroneous edits after confidence removal. +Update canonical [configuration](../config.md), [CLI](../cli.md), +[prompt](../internal/prompts.md), [module](../internal/modules.md), +[validator](../internal/validators.md), and +[output](../integrations/output-files.md) documentation where affected, together +with maintained examples. Provide migration guidance to remove obsolete confidence +settings and fields from configuration, custom prompts, and affected output +consumers. Check links, examples, and whitespace, and remove stale descriptions +of confidence-based behavior. -When this work resumes, discuss representative models, expected outcomes, and -how to run evaluations. Use the prior repository revisions for -before/after comparisons when that work proceeds; a pre-change evaluation run is -not required. Keep model-dependent evaluation separate from the default offline -test suite; no numerical quality gate is selected here. +## Deferred Editorial Evaluation -### 6. Revisit Broader Product Choices +Evaluation corpus preparation, live-model comparisons, and an evaluation workflow +are outside this implementation scope. No baseline run or numerical quality gate +is required before the agreed changes proceed. -The following decisions guide this pass without waiting for the deferred -evaluation work: +When this work resumes, consider a modest collection of needed corrections and +text that should remain unchanged: ambiguous glossary matches, intentional +repetition, meaningful apparent filler, punctuation affecting meaning, and repairs +that should be withdrawn. Compare unnecessary edits, missed corrections, +validator disagreement, and repair outcomes using independent editorial judgment; +validator approval alone is insufficient evidence. -1. **Grammar naming and scope — Decided:** retain the `grammar` name, module key, - and corresponding prompt IDs. Clarify its description as conservative - punctuation, capitalization, spacing, and whole-word `a`/`an` article cleanup. - Preserve its current requested scope and the accepted cross-module minor - editorial allowance. Update the relevant module descriptions and canonical - documentation during implementation; no rename or broader grammar-correction - remit is planned. -2. **Glossary/homophone overlap and defaults — Decided:** retain both modules and - their useful overlap. Keep the existing default sequence documented in - [configuration](../config.md#pipeline-and-glossary-selections) for this pass. - Frequently reevaluate the default and set it to the combination and sequence - of modules observed to work best for most workloads. The maintainer has - observed that two glossary passes help and some workloads benefit from three - or more; these observations inform the current choice without fixing a - permanent number or order of passes. Preserve operators' ability to configure - additional passes. Clarify the complementary requested roles: glossary stages - focus on known terms supported by selected glossaries; homophone stages look - more broadly for plausible spoken-word mistranscriptions and may also use - glossary evidence. Do not exclude an otherwise supported homophone correction - merely because it targets a glossary term. No module consolidation is planned - in this pass. Reevaluating defaults does not depend on the deferred formal - evaluation work in topic 5. -3. **Language and examples — Decided:** retain English as the bundled default. - Keep useful existing campaign examples and add a few more general examples - during the prompt editing pass. Use broadly applicable wording for names, - terminology, and glossary entries, without asserting D&D specificity or - adding a disclaimer about it. Broader language support is outside this pass. -4. **Meaning-reversal policy — Decided:** retain the narrow guard against reversal - of the full segment's assertion, rather than rejection of any change in - meaning. The maintainer has observed that overly broad instructions cause this - validator to reject virtually all corrections; its wording requires unusually - precise tuning. Preserve the existing core instructions and make small, - targeted changes, principally improving rejection feedback and adding a - contrasting example. Changes to names, referents, or other information are - not grounds for rejection by this validator unless they reverse the assertion: - "We met Alice" becoming "We met Alex" changes meaning but is not a reversal. - Assess the full before/after segment; the presence of an antonym in a - replacement does not by itself establish reversal. Instruct rejection feedback - to identify the specific opposition in the assertions; "this changes the - meaning" is insufficient. Approve corrections outside this narrow rejection - criterion, leaving other validators to assess their own criteria. These are - prompt instructions, not new runtime checks on explanation text. Preserve the - safeguard even though it can exclude a genuine mistranscription correction - that reverses the assertion; no contextual exception is selected in this pass. +Discuss representative models, expected outcomes, and an evaluation workflow at +that time. Prior repository revisions can support before/after comparisons, +including the effects of removing confidence thresholds. Keep model-dependent +evaluation separate from the default offline test suite.