230 lines
13 KiB
Markdown
230 lines
13 KiB
Markdown
# Prompt Quality Roadmap
|
|
|
|
**Updated:** 2026-11-11
|
|
|
|
**Status:** All six stages implemented and verified. Broader context changes and
|
|
live-model editorial evaluation remain deferred as described below. The plan
|
|
followed the review of revision `d155bf4`, which covered all eleven prompt
|
|
definitions, shared instructions, response schemas, and request assembly.
|
|
That review identified design concerns, not measured live-model regressions.
|
|
|
|
## Target State and Boundaries
|
|
|
|
Audita's proposal, validation, and repair prompts will give each model call a
|
|
coherent assignment and the evidence needed to carry it out. Proposals will
|
|
better account for interacting edits, repair will receive an explicit view of
|
|
its current proposed result, and validation will identify the target speaker.
|
|
Focused examples will clarify difficult editorial judgments.
|
|
|
|
Preserve the established design:
|
|
|
|
- Module focus guides proposal generation while otherwise acceptable minor
|
|
editorial cleanup remains permissible across modules, as selected in
|
|
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
|
- Validators retain distinct responsibilities. Preserve the carefully tuned
|
|
meaning-reversal criterion and its focus on reversal of the full assertion.
|
|
- Keep literal-span matching, deletion semantics, separate glossary context and
|
|
protected spellings, and full before/after validation.
|
|
- Keep independent repair conversations with full history, successful withdrawal,
|
|
and full revalidation of revisions. Validators explain defects; repair owns
|
|
revision or withdrawal. Approvals continue to need no explanation.
|
|
- Retain confidence-free responses, English defaults, existing module identities,
|
|
and native operator prompt replacement. The current module sequence is outside
|
|
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
|
|
|
|
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
|
|
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
|
|
Update these canonical owners when the corresponding changes land.
|
|
|
|
## Implementation Sequence
|
|
|
|
### 1. Align Proposal and Repair Instruction Authority
|
|
|
|
Proposal system messages currently include categorical task restrictions, such
|
|
as prohibiting punctuation or grammar edits in the homophone module. Repair
|
|
inherits those messages and then permits acceptable minor editorial cleanup in a
|
|
later user message. These instructions can compete across authority levels.
|
|
|
|
- Put durable correction principles in the system messages: evidence-supported
|
|
work, established protection and meaning constraints, and the distinction
|
|
between requested focus and acceptable editorial corrections.
|
|
- Put the initial proposal's focused assignment and task-specific exclusions in
|
|
its user message. Preserve the focused tasks rather than asking every module
|
|
to perform every kind of cleanup.
|
|
- Let each repair instruction give the subsequent reassessment assignment,
|
|
including the selected editorial allowance, without conflicting with an
|
|
inherited system prohibition.
|
|
- Review each complete proposal-to-repair conversation, including later repair
|
|
attempts, for coherent instructions and response-format transitions.
|
|
|
|
This implements the existing acceptance decision; it does not tighten validators
|
|
or change the repair lifecycle. Keep the validator prompts' separate remits,
|
|
especially meaning-reversal review, intact.
|
|
|
|
### 2. Guide Independently Useful and Overlapping Corrections
|
|
|
|
The shared proposal contract already makes each edit refer to the supplied
|
|
segment rather than the imagined result of another proposal. Add guidance for
|
|
edits that overlap or depend on one another:
|
|
|
|
- Prefer independently useful corrections with literal spans that identify the
|
|
intended changes.
|
|
- Combine changes into one proposal when their spans overlap or when one edit
|
|
requires the other. Preserve unchanged context needed for unique matching or
|
|
for selecting an occurrence under the glossary replacement policy.
|
|
- Continue to allow multiple independent corrections to the same segment.
|
|
|
|
For example, separate proposals to capitalize `hello` and replace
|
|
`hello world` with `hello world` may both pass validation against the original,
|
|
but applying the capitalization first prevents the second span from matching.
|
|
A combined replacement can express both edits against the original text.
|
|
|
|
These are prompt instructions. Add no new runtime overlap rejection, proposal
|
|
limit, or application-order rule. Assess combined semantic effects separately
|
|
if practical examples establish a need for additional processing changes.
|
|
|
|
### 3. Supply Repair with the Current Before/After Segment
|
|
|
|
The repair request currently identifies the proposal, its current correction,
|
|
and validation findings. Although the original section remains available in
|
|
conversation history, repair must locate it and reconstruct the proposed result.
|
|
|
|
- Add the original target segment and its proposed resulting text to repair
|
|
context when the current proposal is applicable.
|
|
- Reuse the same domain preview semantics used by validation, including literal
|
|
deletion, whitespace, and replacement of every matching occurrence where
|
|
applicable. Avoid a second implementation of replacement behavior.
|
|
- Refresh this view for each repair attempt, using the current proposal version
|
|
and target rather than a stale view of the initial proposal.
|
|
- When a preview cannot be produced, retain the relevant findings and available
|
|
source context without fabricating a resulting segment or implying application
|
|
succeeded.
|
|
- Keep the full conversation history, proposal references, and revision or
|
|
withdrawal response contract. Explain the added view as evidence for assessing
|
|
the current correction, not an already-applied transcript change.
|
|
|
|
This should make spacing, deletion, and replacement consequences easier to
|
|
judge without requiring the model to reproduce Audita's application mechanics.
|
|
|
|
### 4. Improve Validation Evidence
|
|
|
|
Add the target segment's speaker to each validation item. Neighboring segments
|
|
already carry their speakers; the candidate's identity completes that context
|
|
and helps distinguish continued speech, interruptions, and another speaker's
|
|
response. Carry the supplied value through the existing payload construction
|
|
and token accounting without inferring a missing speaker.
|
|
|
|
Investigate the broader evidence imbalance using available practical cases:
|
|
proposal sees a section, while validation sees immediate neighbors and selected
|
|
glossary entries. A correction supported several turns earlier may appear
|
|
unsupported to a validator. Compare the evidence actually supplied to both
|
|
calls before attributing a rejection to prompt wording.
|
|
|
|
Adding the target speaker is agreed implementation work. Substantial expansion
|
|
of validation context remains evidence-dependent: record a concrete case and a
|
|
focused follow-up recommendation if needed. Retain the current context selection
|
|
and fitting behavior otherwise. Stronger instructions cannot supply missing
|
|
evidence, and this roadmap does not require a broader context redesign.
|
|
|
|
### 5. Add Focused Editorial and Repair Examples
|
|
|
|
Add a small number of examples where they clarify judgments that current
|
|
instructions leave implicit:
|
|
|
|
- Apparent filler that contributes meaning, such as `like` expressing
|
|
approximation or `I mean` introducing a substantive self-correction. Use
|
|
surrounding context to contrast meaningful usage with removable dysfluency.
|
|
- One compact repair revision and one withdrawal, using the existing response
|
|
schema and making withdrawal a successful preservation of the original text.
|
|
- A combined overlapping edit illustrating step 2.
|
|
|
|
Place examples in the relevant proposal, validation, or shared repair guidance.
|
|
Keep existing useful contextual contrasts. Examples should illustrate reasoning
|
|
without becoming unconditional token-based editing rules or an extensive
|
|
rulebook. Maintain the narrow meaning-reversal instructions.
|
|
|
|
### 6. Verify Integration, Documentation, and Test-Policy Compliance
|
|
|
|
Follow the [testing policy](../policy/testing.md) throughout implementation and
|
|
finish with a focused review of affected tests:
|
|
|
|
- Verify repair context reflects the current proposal and the shared preview
|
|
behavior, including representative deletion, replacement-policy, subsequent
|
|
revision, and unavailable-preview cases where coverage is warranted.
|
|
- Verify target-speaker data reaches validation and participates in existing
|
|
payload accounting. Preserve coverage of response handling, revision,
|
|
withdrawal, revalidation, and operator prompt replacement.
|
|
- Review rendered messages and representative complete repair conversations
|
|
directly for instruction authority, editorial intent, and format consistency.
|
|
Check response examples against their schemas.
|
|
- Remove any affected tests that freeze bundled prompt wording through phrase
|
|
checks, exact prose comparisons, or snapshots. Do not replace them with checks
|
|
for the new wording. Retain tests of supplied data and meaningful machine
|
|
contracts where their protective value justifies their maintenance cost.
|
|
- Update the relevant canonical prompt, module, validator, and runtime documents
|
|
as behavior lands. Update maintained examples where affected; check links and
|
|
whitespace.
|
|
|
|
Run the appropriate package checks, build, and full offline suite for the
|
|
implementation under the [development guidance](../development.md#baseline-validation).
|
|
Report the test-policy review's meaningful removals, revisions, and retained
|
|
coverage. Offline tests establish data and execution contracts; they do not
|
|
establish live-model editorial quality.
|
|
|
|
## Deferred Evidence and Follow-Up
|
|
|
|
The evidence review examined the maintained local, campaign, and prompt-edit
|
|
transcript examples, adjacent-context validation fixtures, and proposal
|
|
application fixtures. The glossary example is a single-segment correction;
|
|
the validation fixture supplies the troupe definition and adjacent evidence
|
|
needed to assess its proposed name change. These sources contain no recorded
|
|
live-model rejection demonstrating a need for more distant context. The target
|
|
speaker is now included; broader selection and context fitting remain unchanged.
|
|
Revisit the window when a captured case shows material evidence in the proposal
|
|
request that was absent from validation.
|
|
|
|
The existing stale-span application fixture demonstrates the mechanical overlap
|
|
addressed by the new guidance. It does not establish a need for new combined
|
|
semantic validation, so application ordering and rejection rules remain unchanged.
|
|
|
|
Formal evaluation corpus preparation and live-model comparisons remain deferred
|
|
and are not prerequisites for the agreed changes. Broader validation context and
|
|
additional handling of combined semantic effects should follow concrete evidence
|
|
rather than hypothetical restrictions. Any later evaluation should examine
|
|
needed and unnecessary edits, missed corrections, validator disagreement, and
|
|
repair outcomes using independent editorial judgment. Keep live calls opt-in
|
|
and separate from the default offline suite.
|
|
|
|
No blocking product decisions remain for the implementation scope above.
|
|
|
|
## Completion and Verification
|
|
|
|
- The four proposal definitions now separate durable system principles from
|
|
their initial user assignments. The shared proposal guidance covers overlapping
|
|
and dependent edits; the grammar prompt illustrates a combined edit.
|
|
- Every repair attempt receives the current target's original text when
|
|
available and its previewed result when applicable, using the same domain
|
|
replacement function as validation. Revision and withdrawal examples accompany
|
|
the shared repair guidance.
|
|
- Validation includes the target speaker in rendered input and token accounting.
|
|
Spoken-word proposal and editorial review examples distinguish meaningful
|
|
apparent filler from removable hesitation. The meaning-reversal definition and
|
|
all response schemas retain their existing contracts.
|
|
- Behavioral tests cover repair deletion and replace-all previews, ambiguous or
|
|
missing spans, blank results, missing targets, and refresh after retargeting.
|
|
Existing validation tests now cover target-speaker propagation, absence, and
|
|
budget fitting, including parity with serialized payload estimates.
|
|
- The affected prompt, module, request-assembly, and conversation tests were
|
|
reviewed for wording checks. No further bundled-prose change detectors were
|
|
found to remove; the earlier cleanup remains effective. Retained assertions
|
|
protect supplied-data rendering, schema selection, operator replacement,
|
|
conversation history, revalidation, withdrawal, and outcomes. New assertions
|
|
check request data and behavior rather than prompt phrases.
|
|
- All eleven definitions prepared through the CLI. Four illustrative complete
|
|
conversations, including subsequent repair requests, were reviewed, and the
|
|
sixteen embedded JSON response examples passed schema checks. The application
|
|
build and full offline Go suite passed. Canonical documentation and maintained
|
|
preview examples were updated; links and whitespace were checked.
|
|
|
|
These checks verify implementation and editorial intent, not live-model quality.
|