2026-09-25 12:15:44 +00:00
|
|
|
|
# Prompt Audit: Findings and Recommended Changes
|
2026-09-24 03:07:36 +00:00
|
|
|
|
|
2026-09-25 12:15:44 +00:00
|
|
|
|
**Reviewed:** 2026-09-25, source commit `b8089f4`.
|
|
|
|
|
|
**Scope:** All four proposal prompts, three active LLM-validator prompts, two
|
|
|
|
|
|
inactive validator definitions, shared hardening text, response schemas, supplied
|
|
|
|
|
|
context, and downstream validation/application behavior.
|
|
|
|
|
|
**Status:** R16–R26 open. R1–R15 are resolved; their findings and resolution records
|
|
|
|
|
|
remain in Git history through `b8089f4`.
|
2026-09-24 03:07:36 +00:00
|
|
|
|
|
|
|
|
|
|
## Architecture Assessment
|
|
|
|
|
|
|
2026-09-25 12:15:44 +00:00
|
|
|
|
PromptKit owns structural response validation and repair. Audita owns domain
|
|
|
|
|
|
validation, decision reconciliation, and application. The recent span-deletion
|
|
|
|
|
|
and partial-decision changes are aligned across these boundaries. This audit found
|
|
|
|
|
|
prompt contradictions, incomplete explanations of application semantics, and
|
|
|
|
|
|
opportunities to improve instructions and supplied context.
|
|
|
|
|
|
|
|
|
|
|
|
R16–R19 and R22–R23/R25–R26 are focused wording improvements suitable for
|
|
|
|
|
|
consideration before v2.0.0. R20–R21 require separate design or policy decisions;
|
|
|
|
|
|
R24 is optional cleanup after checking consumers. Their inclusion here does not
|
|
|
|
|
|
make every item a release gate or authorize an unreviewed behavioral expansion.
|
|
|
|
|
|
The shared editorial validator's broad remit is intentional and needs no new
|
|
|
|
|
|
module-scope enforcement as part of this audit.
|
|
|
|
|
|
|
|
|
|
|
|
## R16 — P2: Make the Grammar Article Exception Consistent
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Conflicting instructions.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [grammar prompt](../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml)
|
|
|
|
|
|
allows whole-word `a`/`an` substitution, while its system instruction prohibits word
|
|
|
|
|
|
substitution and a later rule permits only capitalization changes to written words.
|
|
|
|
|
|
A model can follow those prohibitions and omit the expressly permitted correction.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** State the existing article exception consistently in the
|
|
|
|
|
|
system message and word-preservation rule. Consolidate overlapping prohibitions
|
|
|
|
|
|
where helpful without expanding grammar into spelling, homophone, or content edits.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** The prompt unambiguously permits `a`/`an` correction under the
|
|
|
|
|
|
existing conditions while retaining the module's conservative formatting remit.
|
|
|
|
|
|
|
|
|
|
|
|
## R17 — P2: Clarify the Glossary Mishearing Exception
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Prompt conflict with intended glossary correction behavior.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [glossary prompt](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
|
|
|
|
|
|
permits replacing an existing glossary term with another for a plausible mishearing,
|
|
|
|
|
|
then treats present glossary names as protected and prohibits altering them.
|
|
|
|
|
|
The glossary-specific [protected-term validation](../../internal/framework/validators/protected_terms.go)
|
|
|
|
|
|
deliberately allows protected-term swaps, subject to subsequent validation.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Distinguish preserving correct glossary spellings from
|
|
|
|
|
|
correcting a contextually supported misidentification. Make the plausible-mishearing
|
|
|
|
|
|
exception explicit while retaining acoustic plausibility and context requirements.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Recognizing a source word as a glossary term does not itself imply
|
|
|
|
|
|
that the prompt forbids a supported correction to another term. Unrelated name
|
|
|
|
|
|
substitution and gratuitous spelling/capitalization normalization remain outside
|
|
|
|
|
|
scope; no validator relaxation is implied by this wording change.
|
|
|
|
|
|
|
|
|
|
|
|
## R18 — P2: Explain Literal Replacement and Context Spans
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Incomplete application contract in proposal instructions.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Glossary](../../internal/prompts/assets/prompts/modules.glossary.proposal.yaml)
|
|
|
|
|
|
uses `replace_all` without explaining that every matching occurrence in the target
|
|
|
|
|
|
segment changes. Grammar, homophones, and spoken-word require unique matches, but
|
|
|
|
|
|
all proposal prompts ask for only the exact span needing replacement, which can
|
|
|
|
|
|
discourage including unchanged context. [Preview/application](../../internal/framework/proposals/preview.go)
|
|
|
|
|
|
uses literal, case-sensitive matching.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Explain glossary's replace-all behavior explicitly. Allow
|
|
|
|
|
|
unchanged surrounding text in a source span when needed to identify the intended
|
|
|
|
|
|
edit, preserving that context in the replacement. Explain literal matching without
|
|
|
|
|
|
imposing uniqueness on glossary or adding automatic whitespace cleanup.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Guidance explains how to target one intended occurrence when a
|
|
|
|
|
|
shorter span repeats, and that a glossary proposal using a repeated span affects
|
|
|
|
|
|
all matches. Existing module replacement policies and eligible whitespace spans
|
|
|
|
|
|
remain unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
## R19 — P2: Align Output Instructions with Correction and Decision Identity
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Ambiguous response shape and identity guidance.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** Proposal prompts request only changed “segments,” although their
|
|
|
|
|
|
[schema](../../internal/framework/responseschema/assets/correction_set.v1.json)
|
|
|
|
|
|
contains correction proposals and multiple proposals can target one segment.
|
|
|
|
|
|
Validator payloads contain both segment `id` and `correction_index`; the validator
|
|
|
|
|
|
prompts request a decision for each correction index without expressly warning
|
|
|
|
|
|
against substituting segment IDs or renumbering. Both schemas require a top-level
|
|
|
|
|
|
object containing a named list, which prose currently leaves largely to the schema.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Request proposed corrections rather than changed segments,
|
|
|
|
|
|
allow multiple proposals per segment, and explain that their source spans refer to
|
|
|
|
|
|
the supplied text rather than imagined results of other proposals. Tell validators
|
|
|
|
|
|
to copy each supplied `correction_index` exactly, including negative decisions,
|
|
|
|
|
|
without renumbering or substituting segment IDs. Show the top-level
|
|
|
|
|
|
`{"corrections":[...]}` and `{"validations":[...]}` shapes concisely.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Instructions match the supplied schemas and identity model without
|
|
|
|
|
|
requiring one proposal per segment. Validators are still asked for complete decision
|
|
|
|
|
|
sets; tolerant reconciliation remains a fallback, not the preferred output.
|
|
|
|
|
|
|
|
|
|
|
|
## R20 — P2: Evaluate Context Parity for LLM Validation
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Input-context limitation; separate design work.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** Proposal generation supplies section text and glossary data. The
|
|
|
|
|
|
[validator variable builder](../../internal/framework/validators/llm_prompt_variables.go)
|
|
|
|
|
|
supplies candidate before/after segments, categories, and transcript description,
|
|
|
|
|
|
but not neighboring transcript segments or glossary definitions. Spoken-form and
|
|
|
|
|
|
editorial prompts ask for contextual judgments; evidence supporting the proposal
|
|
|
|
|
|
may be absent during validation. Other batched candidates are not reliable context.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Evaluate supplying relevant glossary information and
|
|
|
|
|
|
neighboring transcript context to validators, including its effect on token
|
|
|
|
|
|
estimation and batching. Choose the context scope in a separate design pass.
|
|
|
|
|
|
Avoid asking models to rely on evidence they do not receive; do not silently weaken
|
|
|
|
|
|
validation or imply access to audio.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** A selected design states which evidence each validator receives
|
|
|
|
|
|
and how it fits existing budgets. Representative examples distinguish cases needing
|
|
|
|
|
|
additional context from cases already decidable from the segment. This is not a
|
|
|
|
|
|
requirement to copy the entire section or glossary into every request.
|
|
|
|
|
|
|
|
|
|
|
|
## R21 — P2: Decide How Protected-Term Repetition Should Be Handled
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Prompt/validator policy tension; separate decision required.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [spoken-word prompt](../../internal/prompts/assets/prompts/modules.spoken_word.proposal.yaml)
|
|
|
|
|
|
permits accidental repetition cleanup while preserving named entities. The
|
|
|
|
|
|
[protected-term check](../../internal/framework/validators/protected_terms.go)
|
|
|
|
|
|
requires each protected identity's occurrence count to be preserved. Thus
|
|
|
|
|
|
`Jesters Jesters arrived` to `Jesters arrived` can be rejected even when repetition
|
|
|
|
|
|
is accidental and the named entity remains.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Explicitly decide whether preservation of identity should
|
|
|
|
|
|
permit removing a redundant occurrence. Evaluate a narrow allowance for supported
|
|
|
|
|
|
repetition cleanup, retaining ordinary editorial and meaning checks. If current
|
|
|
|
|
|
count preservation is retained, document that as a deliberate policy choice rather
|
|
|
|
|
|
than silently making the prompt more restrictive to fit existing code.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Prompt and validation expectations agree after the policy decision.
|
|
|
|
|
|
Any allowance preserves substantive references and intentional emphasis; any
|
|
|
|
|
|
restriction is explicitly selected. This audit does not authorize a general
|
|
|
|
|
|
relaxation of protected-term validation.
|
|
|
|
|
|
|
|
|
|
|
|
## R22 — P3: Clarify Full-Segment Spoken-Form Assessment
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Validator wording improvement.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [spoken-form validator](../../internal/prompts/assets/prompts/validators.spoken_form_plausibility.yaml)
|
|
|
|
|
|
explicitly directs full before/after comparison for deletions, while its general
|
|
|
|
|
|
criteria discuss source and replacement spans and surrounding segment context.
|
|
|
|
|
|
Full segment text is available for every candidate.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Explicitly direct full before/after assessment for all
|
|
|
|
|
|
corrections, retaining the narrow acoustic/mistranscription remit and the distinction
|
|
|
|
|
|
from style or readability review.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Guidance consistently uses the available segment evidence without
|
|
|
|
|
|
turning this validator into a general editorial or semantic-equivalence check.
|
|
|
|
|
|
|
|
|
|
|
|
## R23 — P3: Add a Deletion Example to Meaning-Reversal Guidance
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Optional illustrative improvement.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [meaning-reversal prompt](../../internal/prompts/assets/prompts/validators.meaning_reversal_review.yaml)
|
|
|
|
|
|
correctly explains empty replacement and full-segment comparison, but its concrete
|
|
|
|
|
|
examples focus on word substitutions such as visible/invisible and up/down.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Add a concise example where deleting a negation reverses
|
|
|
|
|
|
meaning. Preserve context-sensitive judgment and the validator's narrow scope.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** The example illustrates why a deletion can reverse meaning without
|
|
|
|
|
|
prohibiting empty replacements or making every word deletion a reversal. It is not
|
|
|
|
|
|
a new literal-token rule or a guarantee of model accuracy.
|
|
|
|
|
|
|
|
|
|
|
|
## R24 — P3: Review the Two Inactive Validator Definitions
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Optional maintenance cleanup.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** Registered [grammar_review](../../internal/prompts/assets/prompts/validators.grammar_review.yaml)
|
|
|
|
|
|
and [spoken_word_review](../../internal/prompts/assets/prompts/validators.spoken_word_review.yaml)
|
|
|
|
|
|
definitions largely duplicate editorial review and have already diverged from its
|
|
|
|
|
|
updated deletion guidance. Neither is used by the [built-in chains](../../internal/validators/chains.go).
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Check consumers and retained API uses, then retire the
|
|
|
|
|
|
unused definitions and associated registrations if nothing needs them. If retained,
|
|
|
|
|
|
document their purpose and keep relevant guidance consistent. Do not activate them
|
|
|
|
|
|
as extra validation stages merely because they exist.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Each retained definition has a clear purpose; removal, if selected,
|
|
|
|
|
|
updates actual consumers, tests, and prompt inventory, with any compatibility
|
|
|
|
|
|
implications considered explicitly.
|
|
|
|
|
|
|
|
|
|
|
|
## R25 — P3: Define What Confidence Means
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Underspecified output semantics.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** Proposal and validator prompts specify the confidence range but not
|
|
|
|
|
|
its interpretation. Proposal confidence drives threshold filtering. Validator
|
|
|
|
|
|
confidence is range-checked but does not determine the verdict, which comes from
|
|
|
|
|
|
`approved`.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Describe proposal confidence as confidence that the proposed
|
|
|
|
|
|
correction is warranted, and validator confidence as confidence in the supplied
|
|
|
|
|
|
verdict, including a negative verdict. Do not introduce a validator confidence
|
|
|
|
|
|
threshold or request inflated scores to pass existing thresholds.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** A confident rejection can legitimately have high confidence;
|
|
|
|
|
|
`approved` remains authoritative for the validator verdict. No promise of numerical
|
|
|
|
|
|
calibration or new runtime filtering behavior is added.
|
|
|
|
|
|
|
|
|
|
|
|
## R26 — P3: Clarify Hardening and Remove Assumed Pipeline Ordering
|
|
|
|
|
|
|
|
|
|
|
|
**Category:** Shared wording conflicts with the correction task or configuration.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [hardening fragment](../../internal/prompts/assets/shared/prompt_hardening.md)
|
|
|
|
|
|
calls the transcript the “source of truth for what was said,” although correcting
|
|
|
|
|
|
transcription errors is the task. The grammar prompt claims other corrections occur
|
|
|
|
|
|
in a later review stage, although pipeline selection/order is configurable.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Describe the transcript as source material and require
|
|
|
|
|
|
only evidence-supported corrections while retaining the data-versus-instructions
|
|
|
|
|
|
boundary. Replace the grammar scheduling claim with a statement that homophone and
|
|
|
|
|
|
mistranscription corrections are outside this module's scope.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Hardening discourages invention without implying the transcript
|
|
|
|
|
|
cannot contain errors. Module guidance remains true regardless of configured stage
|
|
|
|
|
|
order or whether another module is enabled. No pipeline ordering restriction is
|
|
|
|
|
|
introduced.
|
2026-09-25 03:03:14 +00:00
|
|
|
|
|
|
|
|
|
|
## Review Verification and Limits
|
|
|
|
|
|
|
2026-09-25 12:15:44 +00:00
|
|
|
|
- This is a source/contract audit, not evidence of the frequency of any particular
|
|
|
|
|
|
model failure in production.
|
|
|
|
|
|
- `go test ./internal/prompts ./internal/framework/validators ./internal/framework/responseschema`
|
|
|
|
|
|
passed during the audit. Those checks verify existing deterministic behavior,
|
|
|
|
|
|
not the effectiveness of proposed language changes.
|
|
|
|
|
|
- No live-provider evaluation was performed and none is made a release gate here.
|
|
|
|
|
|
- Remediation should use focused tests for changed contracts or payloads and retain
|
|
|
|
|
|
prompt parsing/rendering checks. Avoid exact-prose snapshots and a new evaluation
|
|
|
|
|
|
framework solely to implement wording improvements.
|
|
|
|
|
|
- This document catalogs observations and recommendations only; no prompts,
|
|
|
|
|
|
application behavior, or release artifacts were changed by this update.
|