Files
audita/docs/roadmap/prompt-quality.md

230 lines
13 KiB
Markdown
Raw Normal View History

# Prompt Quality Roadmap
**Updated:** 2026-11-11
**Status:** All six stages implemented and verified. Broader context changes and
live-model editorial evaluation remain deferred as described below. The plan
followed the review of revision `d155bf4`, which covered all eleven prompt
definitions, shared instructions, response schemas, and request assembly.
That review identified design concerns, not measured live-model regressions.
## Target State and Boundaries
Audita's proposal, validation, and repair prompts will give each model call a
coherent assignment and the evidence needed to carry it out. Proposals will
better account for interacting edits, repair will receive an explicit view of
its current proposed result, and validation will identify the target speaker.
Focused examples will clarify difficult editorial judgments.
Preserve the established design:
- Module focus guides proposal generation while otherwise acceptable minor
editorial cleanup remains permissible across modules, as selected in
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
- Validators retain distinct responsibilities. Preserve the carefully tuned
meaning-reversal criterion and its focus on reversal of the full assertion.
- Keep literal-span matching, deletion semantics, separate glossary context and
protected spellings, and full before/after validation.
- Keep independent repair conversations with full history, successful withdrawal,
and full revalidation of revisions. Validators explain defects; repair owns
revision or withdrawal. Approvals continue to need no explanation.
- Retain confidence-free responses, English defaults, existing module identities,
and native operator prompt replacement. The current module sequence is outside
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
Update these canonical owners when the corresponding changes land.
## Implementation Sequence
### 1. Align Proposal and Repair Instruction Authority
Proposal system messages currently include categorical task restrictions, such
as prohibiting punctuation or grammar edits in the homophone module. Repair
inherits those messages and then permits acceptable minor editorial cleanup in a
later user message. These instructions can compete across authority levels.
- Put durable correction principles in the system messages: evidence-supported
work, established protection and meaning constraints, and the distinction
between requested focus and acceptable editorial corrections.
- Put the initial proposal's focused assignment and task-specific exclusions in
its user message. Preserve the focused tasks rather than asking every module
to perform every kind of cleanup.
- Let each repair instruction give the subsequent reassessment assignment,
including the selected editorial allowance, without conflicting with an
inherited system prohibition.
- Review each complete proposal-to-repair conversation, including later repair
attempts, for coherent instructions and response-format transitions.
This implements the existing acceptance decision; it does not tighten validators
or change the repair lifecycle. Keep the validator prompts' separate remits,
especially meaning-reversal review, intact.
### 2. Guide Independently Useful and Overlapping Corrections
The shared proposal contract already makes each edit refer to the supplied
segment rather than the imagined result of another proposal. Add guidance for
edits that overlap or depend on one another:
- Prefer independently useful corrections with literal spans that identify the
intended changes.
- Combine changes into one proposal when their spans overlap or when one edit
requires the other. Preserve unchanged context needed for unique matching or
for selecting an occurrence under the glossary replacement policy.
- Continue to allow multiple independent corrections to the same segment.
For example, separate proposals to capitalize `hello` and replace
`hello world` with `hello world` may both pass validation against the original,
but applying the capitalization first prevents the second span from matching.
A combined replacement can express both edits against the original text.
These are prompt instructions. Add no new runtime overlap rejection, proposal
limit, or application-order rule. Assess combined semantic effects separately
if practical examples establish a need for additional processing changes.
### 3. Supply Repair with the Current Before/After Segment
The repair request currently identifies the proposal, its current correction,
and validation findings. Although the original section remains available in
conversation history, repair must locate it and reconstruct the proposed result.
- Add the original target segment and its proposed resulting text to repair
context when the current proposal is applicable.
- Reuse the same domain preview semantics used by validation, including literal
deletion, whitespace, and replacement of every matching occurrence where
applicable. Avoid a second implementation of replacement behavior.
- Refresh this view for each repair attempt, using the current proposal version
and target rather than a stale view of the initial proposal.
- When a preview cannot be produced, retain the relevant findings and available
source context without fabricating a resulting segment or implying application
succeeded.
- Keep the full conversation history, proposal references, and revision or
withdrawal response contract. Explain the added view as evidence for assessing
the current correction, not an already-applied transcript change.
This should make spacing, deletion, and replacement consequences easier to
judge without requiring the model to reproduce Audita's application mechanics.
### 4. Improve Validation Evidence
Add the target segment's speaker to each validation item. Neighboring segments
already carry their speakers; the candidate's identity completes that context
and helps distinguish continued speech, interruptions, and another speaker's
response. Carry the supplied value through the existing payload construction
and token accounting without inferring a missing speaker.
Investigate the broader evidence imbalance using available practical cases:
proposal sees a section, while validation sees immediate neighbors and selected
glossary entries. A correction supported several turns earlier may appear
unsupported to a validator. Compare the evidence actually supplied to both
calls before attributing a rejection to prompt wording.
Adding the target speaker is agreed implementation work. Substantial expansion
of validation context remains evidence-dependent: record a concrete case and a
focused follow-up recommendation if needed. Retain the current context selection
and fitting behavior otherwise. Stronger instructions cannot supply missing
evidence, and this roadmap does not require a broader context redesign.
### 5. Add Focused Editorial and Repair Examples
Add a small number of examples where they clarify judgments that current
instructions leave implicit:
- Apparent filler that contributes meaning, such as `like` expressing
approximation or `I mean` introducing a substantive self-correction. Use
surrounding context to contrast meaningful usage with removable dysfluency.
- One compact repair revision and one withdrawal, using the existing response
schema and making withdrawal a successful preservation of the original text.
- A combined overlapping edit illustrating step 2.
Place examples in the relevant proposal, validation, or shared repair guidance.
Keep existing useful contextual contrasts. Examples should illustrate reasoning
without becoming unconditional token-based editing rules or an extensive
rulebook. Maintain the narrow meaning-reversal instructions.
### 6. Verify Integration, Documentation, and Test-Policy Compliance
Follow the [testing policy](../policy/testing.md) throughout implementation and
finish with a focused review of affected tests:
- Verify repair context reflects the current proposal and the shared preview
behavior, including representative deletion, replacement-policy, subsequent
revision, and unavailable-preview cases where coverage is warranted.
- Verify target-speaker data reaches validation and participates in existing
payload accounting. Preserve coverage of response handling, revision,
withdrawal, revalidation, and operator prompt replacement.
- Review rendered messages and representative complete repair conversations
directly for instruction authority, editorial intent, and format consistency.
Check response examples against their schemas.
- Remove any affected tests that freeze bundled prompt wording through phrase
checks, exact prose comparisons, or snapshots. Do not replace them with checks
for the new wording. Retain tests of supplied data and meaningful machine
contracts where their protective value justifies their maintenance cost.
- Update the relevant canonical prompt, module, validator, and runtime documents
as behavior lands. Update maintained examples where affected; check links and
whitespace.
Run the appropriate package checks, build, and full offline suite for the
implementation under the [development guidance](../development.md#baseline-validation).
Report the test-policy review's meaningful removals, revisions, and retained
coverage. Offline tests establish data and execution contracts; they do not
establish live-model editorial quality.
## Deferred Evidence and Follow-Up
The evidence review examined the maintained local, campaign, and prompt-edit
transcript examples, adjacent-context validation fixtures, and proposal
application fixtures. The glossary example is a single-segment correction;
the validation fixture supplies the troupe definition and adjacent evidence
needed to assess its proposed name change. These sources contain no recorded
live-model rejection demonstrating a need for more distant context. The target
speaker is now included; broader selection and context fitting remain unchanged.
Revisit the window when a captured case shows material evidence in the proposal
request that was absent from validation.
The existing stale-span application fixture demonstrates the mechanical overlap
addressed by the new guidance. It does not establish a need for new combined
semantic validation, so application ordering and rejection rules remain unchanged.
Formal evaluation corpus preparation and live-model comparisons remain deferred
and are not prerequisites for the agreed changes. Broader validation context and
additional handling of combined semantic effects should follow concrete evidence
rather than hypothetical restrictions. Any later evaluation should examine
needed and unnecessary edits, missed corrections, validator disagreement, and
repair outcomes using independent editorial judgment. Keep live calls opt-in
and separate from the default offline suite.
No blocking product decisions remain for the implementation scope above.
## Completion and Verification
- The four proposal definitions now separate durable system principles from
their initial user assignments. The shared proposal guidance covers overlapping
and dependent edits; the grammar prompt illustrates a combined edit.
- Every repair attempt receives the current target's original text when
available and its previewed result when applicable, using the same domain
replacement function as validation. Revision and withdrawal examples accompany
the shared repair guidance.
- Validation includes the target speaker in rendered input and token accounting.
Spoken-word proposal and editorial review examples distinguish meaningful
apparent filler from removable hesitation. The meaning-reversal definition and
all response schemas retain their existing contracts.
- Behavioral tests cover repair deletion and replace-all previews, ambiguous or
missing spans, blank results, missing targets, and refresh after retargeting.
Existing validation tests now cover target-speaker propagation, absence, and
budget fitting, including parity with serialized payload estimates.
- The affected prompt, module, request-assembly, and conversation tests were
reviewed for wording checks. No further bundled-prose change detectors were
found to remove; the earlier cleanup remains effective. Retained assertions
protect supplied-data rendering, schema selection, operator replacement,
conversation history, revalidation, withdrawal, and outcomes. New assertions
check request data and behavior rather than prompt phrases.
- All eleven definitions prepared through the CLI. Four illustrative complete
conversations, including subsequent repair requests, were reviewed, and the
sixteen embedded JSON response examples passed schema checks. The application
build and full offline Go suite passed. Canonical documentation and maintained
preview examples were updated; links and whitespace were checked.
These checks verify implementation and editorial intent, not live-model quality.