Plan removal of confidence throughout correction workflow
This commit is contained in:
@@ -2,8 +2,8 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Topic 1 resolved in ADR 0003; topic 2 decided, with prompt updates
|
||||
pending; topics 3–6 remain for discussion.
|
||||
**Status:** Topic 1 resolved in ADR 0003; topics 2–3 decided, with implementation
|
||||
pending; topics 4–6 remain for discussion.
|
||||
Work through the remaining topics in order. Their recommendations are starting
|
||||
points for discussion, not newly adopted restrictions or guarantees.
|
||||
|
||||
@@ -18,7 +18,8 @@ Retain the strengths: conservative transcript editing, explicit literal-span and
|
||||
deletion semantics, distinct glossary context and protection, full before/after
|
||||
validation, independent repair conversations with withdrawal, and native
|
||||
PromptKit/ConfigKit prompt replacement. Improve consistency across the three
|
||||
roles, useful rejection feedback, and maintainability.
|
||||
roles, useful rejection feedback, and maintainability. Remove model-reported
|
||||
confidence scores and confidence thresholds throughout the correction workflow.
|
||||
|
||||
## Discussion and Work Order
|
||||
|
||||
@@ -46,8 +47,9 @@ found without requiring an exhaustive analysis.
|
||||
|
||||
Approvals require no explanation: instruct the model to return `reason: ""`.
|
||||
The existing schema requires the field but permits an empty string, which the
|
||||
response handler already accepts. This convention needs no schema or
|
||||
response-handling changes. The rejection guidance is a prompt instruction;
|
||||
response handler already accepts. This reason convention itself needs no schema
|
||||
or response-handling changes; topic 3 separately removes confidence from those
|
||||
contracts. The rejection guidance is a prompt instruction;
|
||||
this decision adds no runtime rule rejecting blank explanations.
|
||||
|
||||
**Remaining work:** update the shared validation guidance and all three validator
|
||||
@@ -56,18 +58,47 @@ convention. Check rendered prompts and representative repair payloads to confirm
|
||||
that rejection feedback identifies defects without prescribing replacement
|
||||
corrections. Consider schema expansion only if a concrete need emerges.
|
||||
|
||||
### 3. Make Repair a Reconsideration of the Edit's Merits
|
||||
### 3. Reconsider Corrections on Their Merits and Remove Confidence — Decided
|
||||
|
||||
Withdrawal is already explicit. Clarify when the model should use it and how it
|
||||
should respond when the original evidence remains insufficient.
|
||||
**Decision:** remove confidence entirely from proposals, validation, and repair,
|
||||
including the threshold validator and its configuration. Audita has no established
|
||||
calibration that gives its model-reported scores a reliable numerical meaning
|
||||
across transcripts and models. Proposal scores currently affect acceptance;
|
||||
validator scores are range-checked but do not determine approval. Retaining these
|
||||
numbers adds complexity and encourages repairs aimed at meeting a threshold.
|
||||
The target workflow relies on substantive validation criteria and supplied
|
||||
evidence, without a replacement numerical or categorical confidence scale.
|
||||
|
||||
**Recommendation:** ask the model to reassess the correction from the supplied
|
||||
evidence, retain the module's focus and the accepted editorial allowance, and
|
||||
withdraw unsupported proposals. Address the risk that numerical confidence
|
||||
feedback merely encourages raising the score.
|
||||
Do not automatically prohibit confidence-only revisions: legitimate reassessment
|
||||
is possible. Discuss and evaluate that behavior before adding a business rule.
|
||||
Retain the agreed full-history conversation model and existing repair lifecycle.
|
||||
Repair instructions should ask the model to reassess a correction using the
|
||||
original transcript, supplied context, and rejection feedback. Revise when the
|
||||
evidence supports a correction that addresses the identified problems; withdraw
|
||||
when the proposal remains unsupported, unnecessary, or too uncertain. Withdrawal
|
||||
is a successful resolution that preserves the original text. Explain concrete
|
||||
uncertainty in rejection feedback where relevant, such as context that cannot
|
||||
distinguish two possible names. Preserve the module's requested focus, the
|
||||
accepted minor editorial allowance, the full-history conversation model, and
|
||||
the existing repair lifecycle and revalidation.
|
||||
|
||||
**Remaining work:**
|
||||
|
||||
- Remove confidence fields, instructions, and examples from all proposal,
|
||||
validation, and repair prompts and response schemas, and from their Go models,
|
||||
decoding, and validation paths.
|
||||
- Remove the confidence-threshold validator, its registration and chain entries,
|
||||
root and stage threshold settings, and any corresponding CLI exposure.
|
||||
- Remove confidence-specific rejection feedback, reason codes, report/debug
|
||||
fields, and other plumbing wherever present. Update affected tests, fixtures,
|
||||
configuration examples, and canonical documentation when implementation lands.
|
||||
- Treat this as a deliberate contract change for configuration, custom prompts,
|
||||
and affected serialized outputs; provide migration guidance to remove obsolete
|
||||
fields and settings. Do not retain inert confidence fields or threshold knobs.
|
||||
- Verify that confidence-free proposals, validation decisions, and repair
|
||||
responses work end to end, while substantive validators, revision, withdrawal,
|
||||
and revalidation continue to operate as intended.
|
||||
|
||||
Removing the proposal threshold can admit corrections it previously rejected.
|
||||
Use the evaluation work in topic 5 to assess that effect; removal is the selected
|
||||
direction, not conditional on a further confidence-policy decision.
|
||||
|
||||
### 4. Simplify Instructions and Improve Examples
|
||||
|
||||
@@ -93,6 +124,8 @@ intentional repetition, meaningful uses of apparent filler, punctuation that
|
||||
changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
|
||||
missed corrections, validator disagreement, and repair outcomes using independent
|
||||
editorial assessment; validator approval alone is insufficient evidence.
|
||||
Include a comparison against the prior confidence-threshold behavior to identify
|
||||
changes in both useful corrections and erroneous edits after confidence removal.
|
||||
|
||||
Capture candidate examples during topics 1–4 and establish a baseline before
|
||||
changing prompts. Discuss representative models, expected outcomes, and how to
|
||||
|
||||
Reference in New Issue
Block a user