From 6a843481d2372afdaffb22bcdf74236fa6a3c2f5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 15:18:20 +0000 Subject: [PATCH] Plan removal of confidence throughout correction workflow --- docs/roadmap/prompt-quality.md | 63 ++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 15 deletions(-) diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index a6b7d2b..f564bb8 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -2,8 +2,8 @@ **Updated:** 2026-11-11 -**Status:** Topic 1 resolved in ADR 0003; topic 2 decided, with prompt updates -pending; topics 3–6 remain for discussion. +**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation +pending; topics 4–6 remain for discussion. Work through the remaining topics in order. Their recommendations are starting points for discussion, not newly adopted restrictions or guarantees. @@ -18,7 +18,8 @@ Retain the strengths: conservative transcript editing, explicit literal-span and deletion semantics, distinct glossary context and protection, full before/after validation, independent repair conversations with withdrawal, and native PromptKit/ConfigKit prompt replacement. Improve consistency across the three -roles, useful rejection feedback, and maintainability. +roles, useful rejection feedback, and maintainability. Remove model-reported +confidence scores and confidence thresholds throughout the correction workflow. ## Discussion and Work Order @@ -46,8 +47,9 @@ found without requiring an exhaustive analysis. Approvals require no explanation: instruct the model to return `reason: ""`. The existing schema requires the field but permits an empty string, which the -response handler already accepts. This convention needs no schema or -response-handling changes. The rejection guidance is a prompt instruction; +response handler already accepts. This reason convention itself needs no schema +or response-handling changes; topic 3 separately removes confidence from those +contracts. The rejection guidance is a prompt instruction; this decision adds no runtime rule rejecting blank explanations. **Remaining work:** update the shared validation guidance and all three validator @@ -56,18 +58,47 @@ convention. Check rendered prompts and representative repair payloads to confirm that rejection feedback identifies defects without prescribing replacement corrections. Consider schema expansion only if a concrete need emerges. -### 3. Make Repair a Reconsideration of the Edit's Merits +### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided -Withdrawal is already explicit. Clarify when the model should use it and how it -should respond when the original evidence remains insufficient. +**Decision:** remove confidence entirely from proposals, validation, and repair, +including the threshold validator and its configuration. Audita has no established +calibration that gives its model-reported scores a reliable numerical meaning +across transcripts and models. Proposal scores currently affect acceptance; +validator scores are range-checked but do not determine approval. Retaining these +numbers adds complexity and encourages repairs aimed at meeting a threshold. +The target workflow relies on substantive validation criteria and supplied +evidence, without a replacement numerical or categorical confidence scale. -**Recommendation:** ask the model to reassess the correction from the supplied -evidence, retain the module's focus and the accepted editorial allowance, and -withdraw unsupported proposals. Address the risk that numerical confidence -feedback merely encourages raising the score. -Do not automatically prohibit confidence-only revisions: legitimate reassessment -is possible. Discuss and evaluate that behavior before adding a business rule. -Retain the agreed full-history conversation model and existing repair lifecycle. +Repair instructions should ask the model to reassess a correction using the +original transcript, supplied context, and rejection feedback. Revise when the +evidence supports a correction that addresses the identified problems; withdraw +when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal +is a successful resolution that preserves the original text. Explain concrete +uncertainty in rejection feedback where relevant, such as context that cannot +distinguish two possible names. Preserve the module's requested focus, the +accepted minor editorial allowance, the full-history conversation model, and +the existing repair lifecycle and revalidation. + +**Remaining work:** + +- Remove confidence fields, instructions, and examples from all proposal, + validation, and repair prompts and response schemas, and from their Go models, + decoding, and validation paths. +- Remove the confidence-threshold validator, its registration and chain entries, + root and stage threshold settings, and any corresponding CLI exposure. +- Remove confidence-specific rejection feedback, reason codes, report/debug + fields, and other plumbing wherever present. Update affected tests, fixtures, + configuration examples, and canonical documentation when implementation lands. +- Treat this as a deliberate contract change for configuration, custom prompts, + and affected serialized outputs; provide migration guidance to remove obsolete + fields and settings. Do not retain inert confidence fields or threshold knobs. +- Verify that confidence-free proposals, validation decisions, and repair + responses work end to end, while substantive validators, revision, withdrawal, + and revalidation continue to operate as intended. + +Removing the proposal threshold can admit corrections it previously rejected. +Use the evaluation work in topic 5 to assess that effect; removal is the selected +direction, not conditional on a further confidence-policy decision. ### 4. Simplify Instructions and Improve Examples @@ -93,6 +124,8 @@ intentional repetition, meaningful uses of apparent filler, punctuation that changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, missed corrections, validator disagreement, and repair outcomes using independent editorial assessment; validator approval alone is insufficient evidence. +Include a comparison against the prior confidence-threshold behavior to identify +changes in both useful corrections and erroneous edits after confidence removal. Capture candidate examples during topics 1–4 and establish a baseline before changing prompts. Discuss representative models, expected outcomes, and how to