Remove completed prompt quality plans

This commit is contained in:
2026-11-15 02:31:59 +00:00
parent 5277a1ca19
commit e183708021
3 changed files with 14 additions and 225 deletions

View File

@@ -1,6 +1,6 @@
# Deferred Feature Work
**Last updated:** 2026-11-11
**Last updated:** 2026-11-15
**Status:** Deferred or exploratory work; not implemented or scheduled. These
entries preserve work beyond completed feature and library upgrades.
@@ -83,3 +83,16 @@ semantics. It must address meaning preservation and the effects on segment
identity, timestamps, fixed chunks, subsequent stages, recombination, and output
consumers. Do not infer authorization to remove a segment from an empty string
or introduce segment-removal machinery as part of span-deletion consistency.
## Live-Model Editorial Evaluation
Consider an opt-in transcript corpus and live-model comparisons to assess needed
and unnecessary edits, missed corrections, validator disagreement, and repair
outcomes through independent editorial review. Native PromptKit execution
summaries can supply request and usage measurements. Keep live calls outside
the offline test suite.
Broader validator context and combined semantic validation remain exploratory.
A proposal to expand context should identify a captured case and the evidence
missing from the validator request.
See the [current validator context](../internal/validators.md#supplemental-validation-context).

View File

@@ -1,150 +0,0 @@
# Prompt Quality Implementation Plan
Follow the stages in order within one sprint. Intermediate stages may leave
dependent edits in progress. Verify the integrated result at the end of the
sprint. Read the [feature roadmap](prompt-quality.md) for the selected behavior
and [developer guide](../development.md) for the applicable policies.
## Stage 1 — Finish the PromptKit Adoption Follow-ups
**Complete.** Focused decoder, runner, reporting, and subprocess checks passed.
1. In the debug case of `TestSubprocessCorrectionRepairConversation`, read the
run's `llm-execution.json` and compare its top-level
`summary.provider_attempts` with the number of requests captured by the
provider fixture. Use the existing fixture with an initial proposal, repair
branches, and nested validation. This checks that the shared proposal is
counted once and that repair and validation calls join the invocation total.
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
2. In `repairCandidates`, remove the extra JSON unmarshal and use the typed
response from one decode for the domain decision and the interaction
artifact. Reuse `DecodeRepairResponse` and its existing proposal-reference
and action checks. Where a decoded response fails a correspondence check,
retain it as diagnostic evidence alongside the raw response and failure.
Assess only delivered, structurally valid content. Keep the existing
invalid-output disposition, failure attribution, and revalidation behavior.
[Repair runner](../../internal/framework/runner/repair.go)
[Repair decoder](../../internal/framework/proposal_generation/repair.go)
3. In `BuildLLMExecution`, call `llm.Summarize` once per call and use the result
for both terminal coverage and the serialized summary. Keep the invocation
summary sourced from the native group. Preserve field names, ordering, and
the distinction between unknown usage and zero usage.
[Execution report](../../internal/framework/processreport/llm.go)
## Stage 2 — Review Proposal and Repair Conversations
**Complete.** The shipped and operator-replaced grammar requests retained the
proposal context, including a synthetic glossary entry and protected spelling.
The wire schema changed from a correction set to one repair result. The second
turn retained its branch's prior revision, and the other branch started from
the original proposal exchange. The adapter fixture confirmed separate proposal
and repair profiles. Runner fixtures confirmed updated previews after retargeting,
omission of an inapplicable after-preview, and withdrawal.
The shipped grammar repair wording could read as a blanket restriction on
changes to other words. The revised text describes the grammar focus and leaves
the shared editorial allowance explicit in the rendered request.
1. Use the existing subprocess fixture to exercise the production adapter and
coordinator with scripted responses. Inspect its captured request messages
and response schemas for the initial proposal and the two-turn repair.
Inspect the existing withdrawal case in the runner fixture as well. Use
prepared request artifacts for the rendered instructions and wire requests
for the effective schema. Inspect the shipped definition and an operator
replacement from the existing fixture. Use temporary test logging if
captured requests or artifacts are otherwise discarded after a passing run.
[Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go)
[Runner fixtures](../../internal/framework/runner/repair_test.go)
2. Check whether the original system instructions, the initial assignment,
and the subsequent repair assignment give a coherent task. Check that the
continuation retains the relevant glossary and protected spellings when
supplied, selects the repair definition, role profile, and one-result
schema, and includes each branch's own revision history. Compare the latest
target and before/after preview with the candidate under review, including
a retargeted edit and the existing omission of an inapplicable after-preview.
Use `TestRoleClientContinuesIndependentRepairConversations` for distinct
proposal and repair profiles.
[Adapter conversation fixture](../../internal/framework/llm/conversation_test.go)
3. Check whether the findings give the model a specific defect to address,
whether the current preview identifies the edit being assessed, and whether
the instruction makes revision and withdrawal clear. Fix concrete
ambiguities in the affected bundled prompt or feedback presentation. Keep
module focus, editorial acceptance, and required evidence as specified by
the feature roadmap. If the requests are coherent, record the reviewed
cases and the outcome in the sprint summary. [Prompt registry](../internal/prompts.md)
## Stage 3 — Review Evidence at the Validator Boundary
**Complete.** Both model validators received speaker `A`, the literal
`second` → `Second` edit, full `second text` → `Second text` previews,
the preceding and following segments, and the matching `Second text` glossary
entry for the original and revised candidate. The capitalization objection
reached the next repair request with each validator's identity. The fixture
scripts that same objection for both validators, so it confirms feedback routing.
The rendered meaning-reversal instructions still require opposing full
assertions. This case revealed no missing evidence or unclear validator
instruction. The captured items retained their supplemental context, and
the fitting tests exercised omission of optional context under a smaller budget.
The native invocation reported 10 provider attempts with complete terminal
coverage and unknown token usage. Per-call roles accounted for one proposal,
two initial validator requests, three repair requests, and four validator
requests across two revisions. The seven repair-related requests are already
included in the invocation total. The fixture's last revision repeats the
same edit and receives a scripted approval, so the counts measure execution.
1. Follow a rejected candidate and, where available, its revision through the
captured proposal, validator request, decision, and next repair request.
Compare the target speaker, literal edit, before/after text, supplied
neighbors, matching glossary entries, and validator's stated reason.
Account for optional context omitted by the existing payload fitting.
[Validator payload and fitting](../internal/validators.md#supplemental-validation-context)
2. If evidence is already present but the instruction obscures how to use it,
revise the relevant prompt guidance or example. Preserve each validator's
remit, including the full-assertion reversal criterion. If the case needs
unavailable evidence, describe the missing evidence and the smallest
candidate payload change in the sprint summary. Broader context expansion
and combined semantic validation remain deferred. If no concrete gap
appears, the captured comparison completes this stage.
3. Use the native invocation summary to describe any additional requests
attributable to repair and revalidation. Use per-call evidence to identify
their roles. Treat group totals as encompassing potentially overlapping
work and account for terminal coverage separately from known or complete
usage. Report editorial findings from the requests themselves.
[Diagnostics and reporting](../internal/diagnostics-reporting.md)
## Stage 4 — Verify the Integrated Result
**Complete.** Subprocess, adapter, and runner fixtures together verified
independent repair histories, revalidation from the beginning, acceptance on
the last allowed turn, withdrawal, exhaustion, recoverable provider failure,
and preservation of independently approved edits. The adapter's cancellation
fixture retained the prepared request and raw response with no assessable
content. The runner records the cancellation before attempting to decode, and
the subprocess cancellation fixture verified that no transcript was published.
Reviewed the revised shipped grammar request and the operator replacement as
sent by PromptKit. Supplied-data rendering and schema checks passed, including
the shared revision and withdrawal response examples. The focused tests and
full offline `go test ./...` suite passed with Go 1.27.3. Roadmap links and
anchors resolved, and `git diff --check` passed.
1. Review the existing runner, adapter, and subprocess evidence together for
independent branches, full revalidation, final-turn acceptance, withdrawal,
exhaustion, recoverable unavailability, and cancellation before delivery.
Confirm that accepted siblings survive recoverable failures and that
undelivered content cannot become an approved correction. Extend the nearest
existing fixture only if a changed behavior needs coverage. Use the real
adapter and coordinator with controlled model responses for execution tests.
2. For any prompt edits, inspect representative rendered requests and verify
supplied-data rendering and affected examples against the applicable
response schemas. Use durable assertions for changed behavior and inspect
the prose as part of review. Follow the [testing policy](../policy/testing.md).
Update the [prompt registry](../internal/prompts.md),
[module guide](../internal/modules.md), or [validator guide](../internal/validators.md)
where their owned behavior changes.
3. Use focused tests for iteration in the touched packages. At sprint
completion, run `go test ./...` with the repository's Go toolchain and
offline fixtures. Verify affected links and run `git diff --check`.
Summarize the cases inspected, any evidence-backed edits, and the results
of verification. Live provider comparisons remain deferred.

View File

@@ -1,74 +0,0 @@
# Prompt Quality Roadmap
**Status:** The earlier six prompt improvements are implemented. This follow-up
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
validation, and repair prompts through the same execution path used in production.
Live-model comparisons and broader context expansion remain deferred. See the
[implementation plan](implementation.md) for the sprint sequence.
## Baseline and Boundaries
The [prompt registry](../internal/prompts.md),
[validator guide](../internal/validators.md), and
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
prompts separate durable system principles from initial proposal assignments,
guide independent and overlapping edits, supply current before/after repair
previews, include the target speaker in validation, and illustrate editorial
judgments, revision, and withdrawal.
Preserve these selected decisions:
- Module focus guides generation; otherwise acceptable minor editorial cleanup
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
- Keep the carefully tuned meaning-reversal criterion and distinct validator
responsibilities. Approvals require no explanation; confidence scores remain absent.
- Preserve literal spans, deletion, replacement policies, glossary context,
protected spellings, full revalidation, and deterministic application.
- Retain English defaults and native operator prompt replacement. Module sequence
reevaluation follows the [module guidance](../internal/modules.md).
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
coordination, per-operation deadlines, prepared-request notification, and selected
run accounting. Audita owns candidate identity, findings, previews, acceptance,
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
it must not recreate message replay, turn loops, or token aggregation locally.
## Target State
Proposal and repair instructions should remain coherent across a complete
conversation. The original context and supplied glossary and protected
spellings remain available, each repair branch keeps its own history, and
continuations select the repair definition, profile, and output schema.
Both shipped and operator-replaced definitions use native prompt selection.
Repair feedback identifies the current edit and the validator's findings so
that the model can revise the proposal or withdraw it.
Audita assesses every delivered revision, including the last allowed response.
Coordination stopping and correction acceptance are distinct outcomes.
Undelivered content is diagnostic evidence. Independent approvals survive
recoverable failures. Transient retries, structural repair, and correction
revision retain their separate budgets.
Prompt and feedback changes should address concrete issues in captured
requests while preserving validator acceptance and the allowance for minor
editorial cleanup. Broader context and combined semantic validation remain
deferred. Any proposal to expand validator context should identify a concrete
case and the evidence missing from its request. This follow-up retains current
context fitting and apply-time conflict handling.
Native execution diagnostics count the shared proposal once and include each
new repair and validator run. Terminal coverage and usage availability
remain distinct. Execution measurements describe work and cost. Review
prompt clarity using the supplied evidence and rendered instructions.
The adoption follow-up also removes duplicate repair decoding and per-call
summarization while preserving the existing diagnostic evidence.
## Deferred Evaluation
Formal corpus preparation and live-model comparisons remain optional follow-up
work. Keep live calls opt-in and outside the offline suite. When undertaken,
compare needed and unnecessary edits, missed corrections, validator disagreement,
and repair outcomes using independent editorial judgment. Native groups can
supply execution measurements without introducing a second accounting system.
No blocking product decisions remain for this follow-up scope.