diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index cb7a9c9..e825799 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -2,263 +2,183 @@ **Updated:** 2026-11-11 -**Status:** All six implementation stages completed. Editorial evaluation remains -deferred. No open decisions remain in the current scope. +**Status:** Accepted; implementation has not begun. This roadmap replaces the +completed prompt-quality plan with the recommendations from the fresh review of +revision `d155bf4`. The review covered all eleven prompt definitions, shared +instructions, response schemas, and request assembly. It identified design +concerns, not measured live-model regressions. -## Target State and Constraints +## Target State and Boundaries -Audita will use clearer proposal, validation, and repair prompts, actionable -rejection feedback, and no model-reported confidence scores or thresholds. -Repair will reconsider corrections on their merits and revise or withdraw them -based on the supplied evidence. The initial review covered the eleven -[prompt definitions](../internal/prompts.md) and their consumers at revision -`6d34602`; it was a design review, not a live-model quality measurement. +Audita's proposal, validation, and repair prompts will give each model call a +coherent assignment and the evidence needed to carry it out. Proposals will +better account for interacting edits, repair will receive an explicit view of +its current proposed result, and validation will identify the target speaker. +Focused examples will clarify difficult editorial judgments. -Preserve the following throughout implementation: +Preserve the established design: -- **Editorial acceptance:** module focus guides requested work, while validators - permit otherwise acceptable minor editorial cleanup across modules. Follow - [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md), which records - the maintainer's finding that strict scope instructions also suppress desirable - edits. Applicability, protection, and other substantive checks still apply. -- **Correction mechanics:** retain literal-span and deletion semantics, distinct - glossary context and protection, full before/after validation, and independent - repair conversations with full history, withdrawal, and revalidation. Preserve - native PromptKit/ConfigKit prompt replacement and the existing module structure. -- **Module behavior:** keep `grammar` and its current requested scope; retain - glossary and homophone modules with their useful overlap. Keep the current - [default sequence](../config.md#pipeline-and-glossary-selections) for this pass - and preserve configurable additional passes. -- **Language:** retain English as the bundled default and useful existing campaign - examples, supplemented with a few general examples. Broader language support is - outside this pass. -- **Meaning reversal:** preserve the narrowly tuned guard against reversal of the - full segment's assertion. Its targeted prompt changes are specified below. +- Module focus guides proposal generation while otherwise acceptable minor + editorial cleanup remains permissible across modules, as selected in + [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). +- Validators retain distinct responsibilities. Preserve the carefully tuned + meaning-reversal criterion and its focus on reversal of the full assertion. +- Keep literal-span matching, deletion semantics, separate glossary context and + protected spellings, and full before/after validation. +- Keep independent repair conversations with full history, successful withdrawal, + and full revalidation of revisions. Validators explain defects; repair owns + revision or withdrawal. Approvals continue to need no explanation. +- Retain confidence-free responses, English defaults, existing module identities, + and native operator prompt replacement. The current module sequence is outside + this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md). -Nearly all module behavior reflects the maintainer's practical experimentation. -Two glossary passes have proved useful, and some workloads benefit from three or -more. These observations inform the current choice without fixing a permanent -number or order of passes. Frequently reevaluate the default module combination -and sequence and select what has been observed to work best for most workloads. -This ongoing maintenance does not depend on the deferred formal evaluation work. +The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md), +and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline. +Update these canonical owners when the corresponding changes land. ## Implementation Sequence -Work through the following steps in order. Keep schemas, consumers, prompt -examples, and affected tests consistent as each change lands, and update the -canonical documentation with the corresponding behavior. Step 5 checks the -assembled result; step 6 reviews test-policy compliance. Apply that policy during -the earlier steps as well. Proceed without waiting for live-model evaluation -evidence or a captured baseline. +### 1. Align Proposal and Repair Instruction Authority -### 1. Remove Confidence Throughout the Workflow +Proposal system messages currently include categorical task restrictions, such +as prohibiting punctuation or grammar edits in the homophone module. Repair +inherits those messages and then permits acceptable minor editorial cleanup in a +later user message. These instructions can compete across authority levels. -Remove confidence entirely from proposals, validation, and repair. Audita has no -established calibration that gives its model-reported scores reliable numerical -meaning across transcripts and models. Proposal scores currently affect -acceptance; validator scores are range-checked but do not determine approval. -Retaining the numbers adds complexity and encourages repair toward a threshold. +- Put durable correction principles in the system messages: evidence-supported + work, established protection and meaning constraints, and the distinction + between requested focus and acceptable editorial corrections. +- Put the initial proposal's focused assignment and task-specific exclusions in + its user message. Preserve the focused tasks rather than asking every module + to perform every kind of cleanup. +- Let each repair instruction give the subsequent reassessment assignment, + including the selected editorial allowance, without conflicting with an + inherited system prohibition. +- Review each complete proposal-to-repair conversation, including later repair + attempts, for coherent instructions and response-format transitions. -- Remove confidence fields from response schemas, Go models, decoding, and - validation paths, together with corresponding prompt instructions and examples. -- Remove the confidence-threshold validator, its registration and chain entries, - root and stage threshold settings, and any corresponding CLI exposure. -- Remove confidence-specific rejection feedback, reason codes, report/debug - fields, and other supporting code wherever present. Update affected tests, - fixtures, and configuration examples. -- Treat removal as a deliberate contract change for configuration, custom prompts, - and affected serialized outputs. Retain no inert confidence fields or threshold - knobs, and introduce no replacement numerical or categorical confidence scale. +This implements the existing acceptance decision; it does not tighten validators +or change the repair lifecycle. Keep the validator prompts' separate remits, +especially meaning-reversal review, intact. -Removing the proposal threshold can admit corrections it previously rejected. -That effect can be assessed when editorial evaluation resumes; it does not block -the selected removal. +### 2. Guide Independently Useful and Overlapping Corrections -### 2. Improve Validation Feedback +The shared proposal contract already makes each edit refer to the supplied +segment rather than the imagined result of another proposal. Add guidance for +edits that overlap or depend on one another: -Retain the existing `reason` field. Update shared validation guidance and all -three validator definitions to request concise rejection explanations identifying -the specific problematic edit, applicable criterion, and relevant supplied -evidence within each validator's remit. For editorial review, for example: -removing "might" turns a tentative statement into a definite one. +- Prefer independently useful corrections with literal spans that identify the + intended changes. +- Combine changes into one proposal when their spans overlap or when one edit + requires the other. Preserve unchanged context needed for unique matching or + for selecting an occurrence under the glossary replacement policy. +- Continue to allow multiple independent corrections to the same segment. -Validators identify defects; the repair module owns revision or withdrawal. -Distinguish insufficient evidence from an established defect, respect the accepted -editorial allowance, and include distinct material issues when found without -requiring exhaustive analysis. Provide representative rejection examples. +For example, separate proposals to capitalize `hello` and replace +`hello world` with `hello world` may both pass validation against the original, +but applying the capitalization first prevents the second span from matching. +A combined replacement can express both edits against the original text. -Approvals need no explanation: instruct the model to return `reason: ""` and use -that convention in approval examples. The field remains required and permits an -empty string; this reason convention itself needs no schema or response-handling -change. Explanation guidance is expressed in prompts, with no new runtime -rule rejecting blank explanations. Consider schema expansion only if a concrete -need emerges. +These are prompt instructions. Add no new runtime overlap rejection, proposal +limit, or application-order rule. Assess combined semantic effects separately +if practical examples establish a need for additional processing changes. -The meaning-reversal validator requires particular care. The maintainer has -observed overly broad instructions causing it to reject virtually all corrections. -Preserve its core instructions and make only small, targeted feedback and example -changes, alongside confidence removal and the approval-reason convention: +### 3. Supply Repair with the Current Before/After Segment -- Detect reversal of the full segment's assertion, not every change in meaning. - Changes to names, referents, or other information alone are not grounds for - rejection. "We met Alice" becoming "We met Alex" illustrates a meaning change - that is not a reversal. -- Assess the full before/after segment. An antonym appearing in the replacement - does not by itself establish reversal. -- Instruct rejection feedback to identify the specific opposition in the - assertions; "this changes the meaning" is insufficient. -- Approve corrections outside this narrow rejection criterion, leaving other - validators to assess their own criteria. Preserve the guard even though it can - exclude a genuine mistranscription correction that reverses the assertion; - no contextual exception is selected in this pass. +The repair request currently identifies the proposal, its current correction, +and validation findings. Although the original section remains available in +conversation history, repair must locate it and reconstruct the proposed result. -### 3. Refine Repair Guidance +- Add the original target segment and its proposed resulting text to repair + context when the current proposal is applicable. +- Reuse the same domain preview semantics used by validation, including literal + deletion, whitespace, and replacement of every matching occurrence where + applicable. Avoid a second implementation of replacement behavior. +- Refresh this view for each repair attempt, using the current proposal version + and target rather than a stale view of the initial proposal. +- When a preview cannot be produced, retain the relevant findings and available + source context without fabricating a resulting segment or implying application + succeeded. +- Keep the full conversation history, proposal references, and revision or + withdrawal response contract. Explain the added view as evidence for assessing + the current correction, not an already-applied transcript change. -Ask the model to reassess the correction using the original transcript, supplied -context, and rejection feedback: +This should make spacing, deletion, and replacement consequences easier to +judge without requiring the model to reproduce Audita's application mechanics. -- Revise when the evidence supports a correction that addresses the identified - problems. -- Withdraw when the proposal remains unsupported, unnecessary, or too uncertain. - Withdrawal is a successful resolution that preserves the original text. +### 4. Improve Validation Evidence -Use concrete explanations of uncertainty where relevant, such as context that -cannot distinguish two possible names. Preserve requested module focus and the -accepted editorial allowance. Apply these instructions within the existing -full-history repair lifecycle and revalidation, without adding new retry rules. +Add the target segment's speaker to each validation item. Neighboring segments +already carry their speakers; the candidate's identity completes that context +and helps distinguish continued speech, interruptions, and another speaker's +response. Carry the supplied value through the existing payload construction +and token accounting without inferring a missing speaker. -### 4. Consolidate Prompts, Descriptions, and Examples +Investigate the broader evidence imbalance using available practical cases: +proposal sees a section, while validation sees immediate neighbors and selected +glossary entries. A correction supported several turns earlier may appear +unsupported to a validator. Compare the evidence actually supplied to both +calls before attributing a rejection to prompt wording. -Make a focused editing pass using the existing prompt fragment and module -structure. Organize instructions around shared contracts (including response -format, literal matching, and deletion semantics), module-specific requested work, -and the distinct responsibilities of validation and repair. +Adding the target speaker is agreed implementation work. Substantial expansion +of validation context remains evidence-dependent: record a concrete case and a +focused follow-up recommendation if needed. Retain the current context selection +and fitting behavior otherwise. Stronger instructions cannot supply missing +evidence, and this roadmap does not require a broader context redesign. -Consolidate duplicated rules where small shared fragments prevent drift, remove -contradictions and unnecessary repetition, and keep each rendered prompt -understandable on its own. Retain repetition across separate model calls when -each call needs the instruction. Preserve the meaning-reversal validator's tuned -core wording as specified in step 2; broad simplification for consistency is not -appropriate there. Shorter prompts alone are not the objective. +### 5. Add Focused Editorial and Repair Examples -Clarify descriptions and examples as follows: +Add a small number of examples where they clarify judgments that current +instructions leave implicit: -- Keep the `grammar` name, module key, and prompt IDs. Describe its requested work - as conservative punctuation, capitalization, spacing, and whole-word `a`/`an` - article cleanup. Preserve the current scope and cross-module editorial allowance. -- Describe glossary stages as focusing on known terms supported by selected - glossaries, and homophone stages as looking more broadly for plausible - spoken-word mistranscriptions, including glossary-supported corrections. Do not - exclude an otherwise supported homophone correction because it targets a - glossary term. -- Use short contextual contrasts for difficult judgments, such as accidental - restarts versus deliberate emphasis and context-supported glossary matches - versus merely similar spellings. Examples illustrate judgments rather than - establish unconditional editing rules. -- Retain useful campaign examples and add a few general examples. Use broadly - applicable wording for names, terminology, and glossary entries without asserting - D&D specificity or adding a disclaimer about it. +- Apparent filler that contributes meaning, such as `like` expressing + approximation or `I mean` introducing a substantive self-correction. Use + surrounding context to contrast meaningful usage with removable dysfluency. +- One compact repair revision and one withdrawal, using the existing response + schema and making withdrawal a successful preservation of the original text. +- A combined overlapping edit illustrating step 2. -### 5. Complete Integration Checks and Documentation +Place examples in the relevant proposal, validation, or shared repair guidance. +Keep existing useful contextual contrasts. Examples should illustrate reasoning +without becoming unconditional token-based editing rules or an extensive +rulebook. Maintain the narrow meaning-reversal instructions. -Follow the [testing policy](../policy/testing.md) and the -[development validation guidance](../development.md#baseline-validation). Verify -the assembled workflow, including: +### 6. Verify Integration, Documentation, and Test-Policy Compliance -- Confidence-free proposal, validation, and repair responses work through their - schemas and consumers; substantive validators remain effective in the workflow. -- Rendered prompts and examples match the revised contracts, including empty - approval reasons and rejection feedback carried into repair requests. -- Revision, withdrawal, and full revalidation retain their existing mechanics. +Follow the [testing policy](../policy/testing.md) throughout implementation and +finish with a focused review of affected tests: -Review the actual prompt wording and representative repair payloads to check the -selected responsibilities and the narrow meaning-reversal criterion. Offline -tests verify contracts and integration; they do not establish live-model -editorial quality. Complete the appropriate package checks, build, and full test -suite for the changes, without introducing paid calls into the default suite. +- Verify repair context reflects the current proposal and the shared preview + behavior, including representative deletion, replacement-policy, subsequent + revision, and unavailable-preview cases where coverage is warranted. +- Verify target-speaker data reaches validation and participates in existing + payload accounting. Preserve coverage of response handling, revision, + withdrawal, revalidation, and operator prompt replacement. +- Review rendered messages and representative complete repair conversations + directly for instruction authority, editorial intent, and format consistency. + Check response examples against their schemas. +- Remove any affected tests that freeze bundled prompt wording through phrase + checks, exact prose comparisons, or snapshots. Do not replace them with checks + for the new wording. Retain tests of supplied data and meaningful machine + contracts where their protective value justifies their maintenance cost. +- Update the relevant canonical prompt, module, validator, and runtime documents + as behavior lands. Update maintained examples where affected; check links and + whitespace. -Update canonical [configuration](../config.md), [CLI](../cli.md), -[prompt](../internal/prompts.md), [module](../internal/modules.md), -[validator](../internal/validators.md), and -[output](../integrations/output-files.md) documentation where affected, together -with maintained examples. Provide migration guidance to remove obsolete confidence -settings and fields from configuration, custom prompts, and affected output -consumers. Check links, examples, and whitespace, and remove stale descriptions -of confidence-based behavior. +Run the appropriate package checks, build, and full offline suite for the +implementation under the [development guidance](../development.md#baseline-validation). +Report the test-policy review's meaningful removals, revisions, and retained +coverage. Offline tests establish data and execution contracts; they do not +establish live-model editorial quality. -### 6. Review Tests for Policy Compliance +## Deferred Evidence and Follow-Up -Review existing and newly changed tests supporting this work against the -[testing policy](../policy/testing.md), with particular attention to prompt tests. -Each retained test should protect a meaningful contract or behavior against a -realistic defect at a maintenance cost justified by that protection. +Formal evaluation corpus preparation and live-model comparisons remain deferred +and are not prerequisites for the agreed changes. Broader validation context and +additional handling of combined semantic effects should follow concrete evidence +rather than hypothetical restrictions. Any later evaluation should examine +needed and unnecessary edits, missed corrections, validator disagreement, and +repair outcomes using independent editorial judgment. Keep live calls opt-in +and separate from the default offline suite. -Prompt wording checks carry an unusually high cost: assertions that a particular -word or phrase appears in a prompt body obstruct development and experimentation -with alternative language without establishing that a model follows the intended -instruction. Remove these change-detector tests rather than updating their -expected phrases to match the revised prompts. Apply the same scrutiny to prompt -body snapshots, exact prose comparisons, and equivalent checks hidden behind -helpers. Do not add new tests that freeze incidental prompt wording, including -the carefully tuned meaning-reversal instructions. - -Retain or replace tests where they protect meaningful behavior, such as template -rendering with supplied data, prompt selection and operator replacement, response -schema compatibility, repair context assembly, and processing of validation and -repair responses. Exact comparisons of test-controlled data or explicit machine -contracts can be appropriate; distinguish those from assertions about bundled -instruction prose. Replace a removed test only when a genuine coverage gap -justifies it, and consolidate redundant coverage. - -Review prompt language directly for the agreed editorial intent. Keyword checks -are not a substitute for that review or for the deferred live-model evaluation. -Summarize the tests removed, revised, and retained with their behavioral rationale, -and run the checks appropriate to any test changes made in this final stage. - -## Completion and Test Review - -Confidence has been removed from schemas, models, validators, configuration, -CLI flags, feedback, and fixtures. All eleven prompt definitions and their shared -contracts have been revised. Current behavior and cutover instructions are -recorded in the canonical documentation linked above and the -[v3 migration guide](../migrations/v3.md#confidence-removal). - -The final test-policy review made the following changes: - -- Removed assertions about bundled instruction phrases, module-guidance wording, - and exact shared-instruction repetition. These detected prose changes without - demonstrating model behavior and obstructed prompt experimentation. -- Removed confidence-specific tests with the retired contract. Repair and report - scenarios now use substantive rejection cases, preserving their coverage of - revision, withdrawal, exhaustion, partial success, and failure handling. -- Revised subprocess model fixtures to identify structured validation input - without depending on a prose heading; removed an incidental schema-formatting - assertion. -- Retained supplied-data interpolation, operator prompt replacement, schema and - response handling, repair history, protected-term behavior, and end-to-end - outcomes. Test-controlled strings and machine-contract comparisons remain where - they detect meaningful defects. - -The full offline Go test suite and application build passed. Prompt rendering, -response examples, documentation links, and whitespace were checked. Prompt -language was reviewed directly for the selected responsibilities and narrow -meaning-reversal criterion; these checks do not establish live-model editorial -quality. - -## Deferred Editorial Evaluation - -Evaluation corpus preparation, live-model comparisons, and an evaluation workflow -are outside this implementation scope. No baseline run or numerical quality gate -is required before the agreed changes proceed. - -When this work resumes, consider a modest collection of needed corrections and -text that should remain unchanged: ambiguous glossary matches, intentional -repetition, meaningful apparent filler, punctuation affecting meaning, and repairs -that should be withdrawn. Compare unnecessary edits, missed corrections, -validator disagreement, and repair outcomes using independent editorial judgment; -validator approval alone is insufficient evidence. - -Discuss representative models, expected outcomes, and an evaluation workflow at -that time. Prior repository revisions can support before/after comparisons, -including the effects of removing confidence thresholds. Keep model-dependent -evaluation separate from the default offline test suite. +No blocking product decisions remain for the implementation scope above.