Remove confidence scoring and refine correction prompts

This commit is contained in:
2026-11-11 17:05:52 +00:00
parent bbf89bb4f5
commit d155bf4cb3
92 changed files with 448 additions and 786 deletions

View File

@@ -24,7 +24,7 @@ Explicit process flags override root settings only. `--glossaries` and
`--protected-terms` each accept one JSON array of collection names or `null`;
`[]` clears the selection. Set pipeline order and per-entry overrides in YAML.
Other supplied flags cover profiles, output schema, description, reporting,
chunking, normalization, confidence thresholds, validation budget, and debug
chunking, normalization, validation budget, and debug
work directory. `audita process --help` lists the available flags. An omitted flag
does not contribute an override. [Configuration](config.md) defines their values
and inheritance.

View File

@@ -44,7 +44,7 @@ protection is omitted, it derives from the root glossary selection.
`glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`. Repeated modules
run separately; `[]` runs normalization without model processing. An optional
nonempty, unique `id` identifies a record in evidence. Each record may override
`glossaries`, `protected_terms`, `confidence_threshold`, and fields under `llm`,
`glossaries`, `protected_terms`, and fields under `llm`,
`validation`, and `context`. Omitted fields inherit their resolved root values.
Explicit empty lists and null list selections clear a selection.
@@ -55,6 +55,16 @@ inside it. A glossary correction must have at least one selected entry at
processing preflight. Empty definitions and empty protection remain valid
configuration.
The bundled prompts use English. The modules request complementary work:
- `glossary`: context-supported corrections toward selected glossary terms.
- `homophones`: plausible spoken-word mistranscriptions, including glossary terms.
- `spoken_word`: conservative filler, hesitation, and accidental repetition cleanup.
- `grammar`: punctuation, capitalization, spacing, and whole-word `a`/`an` cleanup.
Otherwise acceptable minor editorial cleanup may be accepted across modules, as
specified by the [editorial acceptance boundary](policy/architecture.md#correction-and-concurrency-invariants).
## Settings
| Setting | Default | Meaning |
@@ -67,7 +77,6 @@ configuration.
| `llm.repair_attempts` | `2` | Structural repairs, from zero through three. |
| `llm.operation_timeout` | `15m` | Positive duration per logical operation. |
| `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. |
| `thresholds.<module>` | `0.8` | Root confidence threshold for each module, in `[0, 1]`. |
| `context.description` | empty | Prompt context; at most 500 bytes after trimming. |
| `chunking.target_sections` | unset | Optional positive target. |
| `chunking.max_section_tokens`, `chunking.min_section_tokens` | `8192`, `2048` | Positive section bounds; minimum must not exceed maximum. |

View File

@@ -67,7 +67,7 @@ Each recovery identifies its work unit, optional `stage_id`, and successful oper
application outcome is recorded in the Markdown report. Filtering covers
defective proposals and revisions under their originating operation. Only
original proposal operations count toward `proposal_responses_with_filtering`.
Low confidence, protected-term rejection,
Protected-term rejection,
ordinary editorial rejection, and an apply-time conflict caused by an earlier
accepted correction are not filtering recoveries. Failed operations are errors,
not recoveries. The initial chunk identity remains stable even if earlier

View File

@@ -44,7 +44,7 @@ Each stage prepares the encoded glossary context and protected spellings once.
Proposal generation adds the selected transcript section and trimmed description
for each request. PromptKit renders the selected definition and validates the
correction-set schema before Audita maps proposals. Domain validators still
check confidence, applicability, protected glossary terms, and replacement
check applicability, protected glossary terms, and replacement
policy. Model-facing segment IDs are one-based positions within the proposal
request. Generation maps returned references back to the current chunk's actual
segment IDs, preserving response order and multiple proposals per segment.
@@ -76,6 +76,11 @@ outcomes are defined by the [report contract](../integrations/output-files.md).
## Changing Modules
Frequently reevaluate the default module combination and sequence against what
has been observed to work best for most workloads. Prior empirical results inform
the current choice without fixing a permanent number or order of passes. Maintain
the selected default in the [configuration contract](../config.md#pipeline-and-glossary-selections).
1. Add or update the narrowly scoped package under `internal/modules/<module_key>`.
2. Keep module prompt selection in the module and its definition in
`internal/prompts`; see [prompt assets](prompts.md).

View File

@@ -59,12 +59,14 @@ pass these values through the role-bound adapter. See the
[configuration](../config.md) for operator-selected profiles.
Proposal instructions describe literal matching, each module's replacement policy,
source-based proposal identity, and confidence in whether an edit is warranted.
and source-based proposal identity.
Proposal `id` and validator `correction_index` values are short, one-based
references scoped to each request, not internal transcript/proposal identities.
Mapping belongs to [proposal generation](modules.md) and
[validator reconciliation](validators.md). Validator instructions distinguish
confidence in the verdict from the verdict itself. Compact output examples
[validator reconciliation](validators.md). Validator decisions use `approved` for the verdict. Approval examples use an empty
`reason`; rejection guidance requests a specific defect and supplied evidence
within the validator's remit. The model explains defects; repair owns revision or
withdrawal. This is prompt guidance, not a runtime nonempty-reason check. Compact output examples
illustrate the supplied schema without requiring fixed JSON formatting or key
order. The grammar article exception and glossary mishearing exception apply
consistently across their instructions.
@@ -72,3 +74,19 @@ consistently across their instructions.
Only the three active validator definitions are retained. The unused internal
`grammar_review` and `spoken_word_review` definitions and type selections were
removed; built-in chains and operator configuration are unchanged.
Repair guidance reassesses the proposal against the original transcript, context,
and findings. Unsupported, unnecessary, or unresolved uncertain corrections may
be withdrawn, preserving the original text. Otherwise acceptable minor editorial
cleanup remains permissible during repair.
The meaning-reversal validator has a deliberately narrow remit: reversal of the
full segment's assertion, not every change of meaning. Its prompt distinguishes
changes of referent from reversal and asks rejection reasons to identify the
opposing assertions. Preserve its carefully tuned core instructions when editing;
other validators assess other criteria.
Prompt tests protect rendering with supplied data, resource selection, schemas,
and integration behavior. They do not freeze bundled prose with word/phrase
checks or snapshots. Review editorial intent directly; offline tests cannot
establish live-model editorial quality.

View File

@@ -21,7 +21,6 @@ Execution-class metadata:
Deterministic:
- `proposal_shape`
- `confidence_threshold`
- `original_text_presence`
- `non_empty_resulting_segment` — rejects edits that would leave the segment
blank, regardless of replacement-span length.
@@ -80,9 +79,7 @@ makes all requested candidates unavailable. Decisions and unavailable indices
map back to internal proposal identities; results are ordered by request identity
and satisfy strict internal cardinality before the runner advances
approved candidates. An unavailable candidate never advances as approved.
Invalid confidence in a decoded test double also makes only its candidate
unavailable; production schema validation still rejects out-of-range confidence
before reconciliation. A partially unavailable batch records one failed
A partially unavailable batch records one failed
operation with only those requested indices, while a successful batch with
ignored extras records a decision-filtering recovery. Fatal errors stop the
chain without returning unchecked candidates as approved. A local result builder

View File

@@ -21,6 +21,20 @@ configuration. Use the [configuration reference](../config.md) and
stderr, including when ordinary reports are disabled. `--debug` also shows
counts unless followed by `--quiet`. See [CLI reporting](../cli.md#processing).
## Confidence removal
Remove root `thresholds` and stage `confidence_threshold` settings, and the
`--glossary-confidence-threshold`, `--homophones-confidence-threshold`,
`--spoken-word-confidence-threshold`, and `--grammar-confidence-threshold` flags.
The `confidence_threshold` validator has been removed from the built-in chains.
Update custom proposal, validation, and repair prompts to omit `confidence` from
response examples and instructions. Audita's response schemas no longer accept
that field. Validator responses still require `approved` and `reason`; approvals
may use an empty reason. Corrections in debug records and other affected
serialized proposal/repair data no longer carry confidence, so consumers must
stop requiring or interpreting it. No replacement score or threshold is provided.
Inspect configuration and resources before processing, then try a controlled
small transcript. Processing preflight checks every glossary correction and
selected role before preparing outputs or invoking a model. See [operations](../operations.md)

View File

@@ -2,7 +2,7 @@
**Updated:** 2026-11-11
**Status:** Decisions settled; implementation pending. Editorial evaluation is
**Status:** All six implementation stages completed. Editorial evaluation remains
deferred. No open decisions remain in the current scope.
## Target State and Constraints
@@ -215,6 +215,36 @@ are not a substitute for that review or for the deferred live-model evaluation.
Summarize the tests removed, revised, and retained with their behavioral rationale,
and run the checks appropriate to any test changes made in this final stage.
## Completion and Test Review
Confidence has been removed from schemas, models, validators, configuration,
CLI flags, feedback, and fixtures. All eleven prompt definitions and their shared
contracts have been revised. Current behavior and cutover instructions are
recorded in the canonical documentation linked above and the
[v3 migration guide](../migrations/v3.md#confidence-removal).
The final test-policy review made the following changes:
- Removed assertions about bundled instruction phrases, module-guidance wording,
and exact shared-instruction repetition. These detected prose changes without
demonstrating model behavior and obstructed prompt experimentation.
- Removed confidence-specific tests with the retired contract. Repair and report
scenarios now use substantive rejection cases, preserving their coverage of
revision, withdrawal, exhaustion, partial success, and failure handling.
- Revised subprocess model fixtures to identify structured validation input
without depending on a prose heading; removed an incidental schema-formatting
assertion.
- Retained supplied-data interpolation, operator prompt replacement, schema and
response handling, repair history, protected-term behavior, and end-to-end
outcomes. Test-controlled strings and machine-contract comparisons remain where
they detect meaningful defects.
The full offline Go test suite and application build passed. Prompt rendering,
response examples, documentation links, and whitespace were checked. Prompt
language was reviewed directly for the selected responsibilities and narrow
meaning-reversal criterion; these checks do not establish live-model editorial
quality.
## Deferred Editorial Evaluation
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow