Plan removal of confidence throughout correction workflow

This commit is contained in:
2026-11-11 15:18:20 +00:00
parent d2eb5a5662
commit 6a843481d2

View File

@@ -2,8 +2,8 @@
**Updated:** 2026-11-11 **Updated:** 2026-11-11
**Status:** Topic 1 resolved in ADR 0003; topic 2 decided, with prompt updates **Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
pending; topics 3–6 remain for discussion. pending; topics 4–6 remain for discussion.
Work through the remaining topics in order. Their recommendations are starting Work through the remaining topics in order. Their recommendations are starting
points for discussion, not newly adopted restrictions or guarantees. points for discussion, not newly adopted restrictions or guarantees.
@@ -18,7 +18,8 @@ Retain the strengths: conservative transcript editing, explicit literal-span and
deletion semantics, distinct glossary context and protection, full before/after deletion semantics, distinct glossary context and protection, full before/after
validation, independent repair conversations with withdrawal, and native validation, independent repair conversations with withdrawal, and native
PromptKit/ConfigKit prompt replacement. Improve consistency across the three PromptKit/ConfigKit prompt replacement. Improve consistency across the three
roles, useful rejection feedback, and maintainability. roles, useful rejection feedback, and maintainability. Remove model-reported
confidence scores and confidence thresholds throughout the correction workflow.
## Discussion and Work Order ## Discussion and Work Order
@@ -46,8 +47,9 @@ found without requiring an exhaustive analysis.
Approvals require no explanation: instruct the model to return `reason: ""`. Approvals require no explanation: instruct the model to return `reason: ""`.
The existing schema requires the field but permits an empty string, which the The existing schema requires the field but permits an empty string, which the
response handler already accepts. This convention needs no schema or response handler already accepts. This reason convention itself needs no schema
response-handling changes. The rejection guidance is a prompt instruction; or response-handling changes; topic 3 separately removes confidence from those
contracts. The rejection guidance is a prompt instruction;
this decision adds no runtime rule rejecting blank explanations. this decision adds no runtime rule rejecting blank explanations.
**Remaining work:** update the shared validation guidance and all three validator **Remaining work:** update the shared validation guidance and all three validator
@@ -56,18 +58,47 @@ convention. Check rendered prompts and representative repair payloads to confirm
that rejection feedback identifies defects without prescribing replacement that rejection feedback identifies defects without prescribing replacement
corrections. Consider schema expansion only if a concrete need emerges. corrections. Consider schema expansion only if a concrete need emerges.
### 3. Make Repair a Reconsideration of the Edit's Merits ### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided
Withdrawal is already explicit. Clarify when the model should use it and how it **Decision:** remove confidence entirely from proposals, validation, and repair,
should respond when the original evidence remains insufficient. including the threshold validator and its configuration. Audita has no established
calibration that gives its model-reported scores a reliable numerical meaning
across transcripts and models. Proposal scores currently affect acceptance;
validator scores are range-checked but do not determine approval. Retaining these
numbers adds complexity and encourages repairs aimed at meeting a threshold.
The target workflow relies on substantive validation criteria and supplied
evidence, without a replacement numerical or categorical confidence scale.
**Recommendation:** ask the model to reassess the correction from the supplied Repair instructions should ask the model to reassess a correction using the
evidence, retain the module's focus and the accepted editorial allowance, and original transcript, supplied context, and rejection feedback. Revise when the
withdraw unsupported proposals. Address the risk that numerical confidence evidence supports a correction that addresses the identified problems; withdraw
feedback merely encourages raising the score. when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal
Do not automatically prohibit confidence-only revisions: legitimate reassessment is a successful resolution that preserves the original text. Explain concrete
is possible. Discuss and evaluate that behavior before adding a business rule. uncertainty in rejection feedback where relevant, such as context that cannot
Retain the agreed full-history conversation model and existing repair lifecycle. distinguish two possible names. Preserve the module's requested focus, the
accepted minor editorial allowance, the full-history conversation model, and
the existing repair lifecycle and revalidation.
**Remaining work:**
- Remove confidence fields, instructions, and examples from all proposal,
validation, and repair prompts and response schemas, and from their Go models,
decoding, and validation paths.
- Remove the confidence-threshold validator, its registration and chain entries,
root and stage threshold settings, and any corresponding CLI exposure.
- Remove confidence-specific rejection feedback, reason codes, report/debug
fields, and other plumbing wherever present. Update affected tests, fixtures,
configuration examples, and canonical documentation when implementation lands.
- Treat this as a deliberate contract change for configuration, custom prompts,
and affected serialized outputs; provide migration guidance to remove obsolete
fields and settings. Do not retain inert confidence fields or threshold knobs.
- Verify that confidence-free proposals, validation decisions, and repair
responses work end to end, while substantive validators, revision, withdrawal,
and revalidation continue to operate as intended.
Removing the proposal threshold can admit corrections it previously rejected.
Use the evaluation work in topic 5 to assess that effect; removal is the selected
direction, not conditional on a further confidence-policy decision.
### 4. Simplify Instructions and Improve Examples ### 4. Simplify Instructions and Improve Examples
@@ -93,6 +124,8 @@ intentional repetition, meaningful uses of apparent filler, punctuation that
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits, changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
missed corrections, validator disagreement, and repair outcomes using independent missed corrections, validator disagreement, and repair outcomes using independent
editorial assessment; validator approval alone is insufficient evidence. editorial assessment; validator approval alone is insufficient evidence.
Include a comparison against the prior confidence-threshold behavior to identify
changes in both useful corrections and erroneous edits after confidence removal.
Capture candidate examples during topics 1–4 and establish a baseline before Capture candidate examples during topics 1–4 and establish a baseline before
changing prompts. Discuss representative models, expected outcomes, and how to changing prompts. Discuss representative models, expected outcomes, and how to