Remove confidence scoring and refine correction prompts
This commit is contained in:
@@ -24,7 +24,7 @@ Explicit process flags override root settings only. `--glossaries` and
|
||||
`--protected-terms` each accept one JSON array of collection names or `null`;
|
||||
`[]` clears the selection. Set pipeline order and per-entry overrides in YAML.
|
||||
Other supplied flags cover profiles, output schema, description, reporting,
|
||||
chunking, normalization, confidence thresholds, validation budget, and debug
|
||||
chunking, normalization, validation budget, and debug
|
||||
work directory. `audita process --help` lists the available flags. An omitted flag
|
||||
does not contribute an override. [Configuration](config.md) defines their values
|
||||
and inheritance.
|
||||
|
||||
@@ -44,7 +44,7 @@ protection is omitted, it derives from the root glossary selection.
|
||||
`glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`. Repeated modules
|
||||
run separately; `[]` runs normalization without model processing. An optional
|
||||
nonempty, unique `id` identifies a record in evidence. Each record may override
|
||||
`glossaries`, `protected_terms`, `confidence_threshold`, and fields under `llm`,
|
||||
`glossaries`, `protected_terms`, and fields under `llm`,
|
||||
`validation`, and `context`. Omitted fields inherit their resolved root values.
|
||||
Explicit empty lists and null list selections clear a selection.
|
||||
|
||||
@@ -55,6 +55,16 @@ inside it. A glossary correction must have at least one selected entry at
|
||||
processing preflight. Empty definitions and empty protection remain valid
|
||||
configuration.
|
||||
|
||||
The bundled prompts use English. The modules request complementary work:
|
||||
|
||||
- `glossary`: context-supported corrections toward selected glossary terms.
|
||||
- `homophones`: plausible spoken-word mistranscriptions, including glossary terms.
|
||||
- `spoken_word`: conservative filler, hesitation, and accidental repetition cleanup.
|
||||
- `grammar`: punctuation, capitalization, spacing, and whole-word `a`/`an` cleanup.
|
||||
|
||||
Otherwise acceptable minor editorial cleanup may be accepted across modules, as
|
||||
specified by the [editorial acceptance boundary](policy/architecture.md#correction-and-concurrency-invariants).
|
||||
|
||||
## Settings
|
||||
|
||||
| Setting | Default | Meaning |
|
||||
@@ -67,7 +77,6 @@ configuration.
|
||||
| `llm.repair_attempts` | `2` | Structural repairs, from zero through three. |
|
||||
| `llm.operation_timeout` | `15m` | Positive duration per logical operation. |
|
||||
| `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. |
|
||||
| `thresholds.<module>` | `0.8` | Root confidence threshold for each module, in `[0, 1]`. |
|
||||
| `context.description` | empty | Prompt context; at most 500 bytes after trimming. |
|
||||
| `chunking.target_sections` | unset | Optional positive target. |
|
||||
| `chunking.max_section_tokens`, `chunking.min_section_tokens` | `8192`, `2048` | Positive section bounds; minimum must not exceed maximum. |
|
||||
|
||||
@@ -67,7 +67,7 @@ Each recovery identifies its work unit, optional `stage_id`, and successful oper
|
||||
application outcome is recorded in the Markdown report. Filtering covers
|
||||
defective proposals and revisions under their originating operation. Only
|
||||
original proposal operations count toward `proposal_responses_with_filtering`.
|
||||
Low confidence, protected-term rejection,
|
||||
Protected-term rejection,
|
||||
ordinary editorial rejection, and an apply-time conflict caused by an earlier
|
||||
accepted correction are not filtering recoveries. Failed operations are errors,
|
||||
not recoveries. The initial chunk identity remains stable even if earlier
|
||||
|
||||
@@ -44,7 +44,7 @@ Each stage prepares the encoded glossary context and protected spellings once.
|
||||
Proposal generation adds the selected transcript section and trimmed description
|
||||
for each request. PromptKit renders the selected definition and validates the
|
||||
correction-set schema before Audita maps proposals. Domain validators still
|
||||
check confidence, applicability, protected glossary terms, and replacement
|
||||
check applicability, protected glossary terms, and replacement
|
||||
policy. Model-facing segment IDs are one-based positions within the proposal
|
||||
request. Generation maps returned references back to the current chunk's actual
|
||||
segment IDs, preserving response order and multiple proposals per segment.
|
||||
@@ -76,6 +76,11 @@ outcomes are defined by the [report contract](../integrations/output-files.md).
|
||||
|
||||
## Changing Modules
|
||||
|
||||
Frequently reevaluate the default module combination and sequence against what
|
||||
has been observed to work best for most workloads. Prior empirical results inform
|
||||
the current choice without fixing a permanent number or order of passes. Maintain
|
||||
the selected default in the [configuration contract](../config.md#pipeline-and-glossary-selections).
|
||||
|
||||
1. Add or update the narrowly scoped package under `internal/modules/<module_key>`.
|
||||
2. Keep module prompt selection in the module and its definition in
|
||||
`internal/prompts`; see [prompt assets](prompts.md).
|
||||
|
||||
@@ -59,12 +59,14 @@ pass these values through the role-bound adapter. See the
|
||||
[configuration](../config.md) for operator-selected profiles.
|
||||
|
||||
Proposal instructions describe literal matching, each module's replacement policy,
|
||||
source-based proposal identity, and confidence in whether an edit is warranted.
|
||||
and source-based proposal identity.
|
||||
Proposal `id` and validator `correction_index` values are short, one-based
|
||||
references scoped to each request, not internal transcript/proposal identities.
|
||||
Mapping belongs to [proposal generation](modules.md) and
|
||||
[validator reconciliation](validators.md). Validator instructions distinguish
|
||||
confidence in the verdict from the verdict itself. Compact output examples
|
||||
[validator reconciliation](validators.md). Validator decisions use `approved` for the verdict. Approval examples use an empty
|
||||
`reason`; rejection guidance requests a specific defect and supplied evidence
|
||||
within the validator's remit. The model explains defects; repair owns revision or
|
||||
withdrawal. This is prompt guidance, not a runtime nonempty-reason check. Compact output examples
|
||||
illustrate the supplied schema without requiring fixed JSON formatting or key
|
||||
order. The grammar article exception and glossary mishearing exception apply
|
||||
consistently across their instructions.
|
||||
@@ -72,3 +74,19 @@ consistently across their instructions.
|
||||
Only the three active validator definitions are retained. The unused internal
|
||||
`grammar_review` and `spoken_word_review` definitions and type selections were
|
||||
removed; built-in chains and operator configuration are unchanged.
|
||||
|
||||
Repair guidance reassesses the proposal against the original transcript, context,
|
||||
and findings. Unsupported, unnecessary, or unresolved uncertain corrections may
|
||||
be withdrawn, preserving the original text. Otherwise acceptable minor editorial
|
||||
cleanup remains permissible during repair.
|
||||
|
||||
The meaning-reversal validator has a deliberately narrow remit: reversal of the
|
||||
full segment's assertion, not every change of meaning. Its prompt distinguishes
|
||||
changes of referent from reversal and asks rejection reasons to identify the
|
||||
opposing assertions. Preserve its carefully tuned core instructions when editing;
|
||||
other validators assess other criteria.
|
||||
|
||||
Prompt tests protect rendering with supplied data, resource selection, schemas,
|
||||
and integration behavior. They do not freeze bundled prose with word/phrase
|
||||
checks or snapshots. Review editorial intent directly; offline tests cannot
|
||||
establish live-model editorial quality.
|
||||
|
||||
@@ -21,7 +21,6 @@ Execution-class metadata:
|
||||
|
||||
Deterministic:
|
||||
- `proposal_shape`
|
||||
- `confidence_threshold`
|
||||
- `original_text_presence`
|
||||
- `non_empty_resulting_segment` — rejects edits that would leave the segment
|
||||
blank, regardless of replacement-span length.
|
||||
@@ -80,9 +79,7 @@ makes all requested candidates unavailable. Decisions and unavailable indices
|
||||
map back to internal proposal identities; results are ordered by request identity
|
||||
and satisfy strict internal cardinality before the runner advances
|
||||
approved candidates. An unavailable candidate never advances as approved.
|
||||
Invalid confidence in a decoded test double also makes only its candidate
|
||||
unavailable; production schema validation still rejects out-of-range confidence
|
||||
before reconciliation. A partially unavailable batch records one failed
|
||||
A partially unavailable batch records one failed
|
||||
operation with only those requested indices, while a successful batch with
|
||||
ignored extras records a decision-filtering recovery. Fatal errors stop the
|
||||
chain without returning unchecked candidates as approved. A local result builder
|
||||
|
||||
@@ -21,6 +21,20 @@ configuration. Use the [configuration reference](../config.md) and
|
||||
stderr, including when ordinary reports are disabled. `--debug` also shows
|
||||
counts unless followed by `--quiet`. See [CLI reporting](../cli.md#processing).
|
||||
|
||||
## Confidence removal
|
||||
|
||||
Remove root `thresholds` and stage `confidence_threshold` settings, and the
|
||||
`--glossary-confidence-threshold`, `--homophones-confidence-threshold`,
|
||||
`--spoken-word-confidence-threshold`, and `--grammar-confidence-threshold` flags.
|
||||
The `confidence_threshold` validator has been removed from the built-in chains.
|
||||
|
||||
Update custom proposal, validation, and repair prompts to omit `confidence` from
|
||||
response examples and instructions. Audita's response schemas no longer accept
|
||||
that field. Validator responses still require `approved` and `reason`; approvals
|
||||
may use an empty reason. Corrections in debug records and other affected
|
||||
serialized proposal/repair data no longer carry confidence, so consumers must
|
||||
stop requiring or interpreting it. No replacement score or threshold is provided.
|
||||
|
||||
Inspect configuration and resources before processing, then try a controlled
|
||||
small transcript. Processing preflight checks every glossary correction and
|
||||
selected role before preparing outputs or invoking a model. See [operations](../operations.md)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
|
||||
**Status:** Decisions settled; implementation pending. Editorial evaluation is
|
||||
**Status:** All six implementation stages completed. Editorial evaluation remains
|
||||
deferred. No open decisions remain in the current scope.
|
||||
|
||||
## Target State and Constraints
|
||||
@@ -215,6 +215,36 @@ are not a substitute for that review or for the deferred live-model evaluation.
|
||||
Summarize the tests removed, revised, and retained with their behavioral rationale,
|
||||
and run the checks appropriate to any test changes made in this final stage.
|
||||
|
||||
## Completion and Test Review
|
||||
|
||||
Confidence has been removed from schemas, models, validators, configuration,
|
||||
CLI flags, feedback, and fixtures. All eleven prompt definitions and their shared
|
||||
contracts have been revised. Current behavior and cutover instructions are
|
||||
recorded in the canonical documentation linked above and the
|
||||
[v3 migration guide](../migrations/v3.md#confidence-removal).
|
||||
|
||||
The final test-policy review made the following changes:
|
||||
|
||||
- Removed assertions about bundled instruction phrases, module-guidance wording,
|
||||
and exact shared-instruction repetition. These detected prose changes without
|
||||
demonstrating model behavior and obstructed prompt experimentation.
|
||||
- Removed confidence-specific tests with the retired contract. Repair and report
|
||||
scenarios now use substantive rejection cases, preserving their coverage of
|
||||
revision, withdrawal, exhaustion, partial success, and failure handling.
|
||||
- Revised subprocess model fixtures to identify structured validation input
|
||||
without depending on a prose heading; removed an incidental schema-formatting
|
||||
assertion.
|
||||
- Retained supplied-data interpolation, operator prompt replacement, schema and
|
||||
response handling, repair history, protected-term behavior, and end-to-end
|
||||
outcomes. Test-controlled strings and machine-contract comparisons remain where
|
||||
they detect meaningful defects.
|
||||
|
||||
The full offline Go test suite and application build passed. Prompt rendering,
|
||||
response examples, documentation links, and whitespace were checked. Prompt
|
||||
language was reviewed directly for the selected responsibilities and narrow
|
||||
meaning-reversal criterion; these checks do not establish live-model editorial
|
||||
quality.
|
||||
|
||||
## Deferred Editorial Evaluation
|
||||
|
||||
Evaluation corpus preparation, live-model comparisons, and an evaluation workflow
|
||||
|
||||
Reference in New Issue
Block a user