diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index ce5a5cd..b0e7eff 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -1,6 +1,6 @@ # Deferred Feature Work -**Last updated:** 2026-11-11 +**Last updated:** 2026-11-15 **Status:** Deferred or exploratory work; not implemented or scheduled. These entries preserve work beyond completed feature and library upgrades. @@ -83,3 +83,16 @@ semantics. It must address meaning preservation and the effects on segment identity, timestamps, fixed chunks, subsequent stages, recombination, and output consumers. Do not infer authorization to remove a segment from an empty string or introduce segment-removal machinery as part of span-deletion consistency. + +## Live-Model Editorial Evaluation + +Consider an opt-in transcript corpus and live-model comparisons to assess needed +and unnecessary edits, missed corrections, validator disagreement, and repair +outcomes through independent editorial review. Native PromptKit execution +summaries can supply request and usage measurements. Keep live calls outside +the offline test suite. + +Broader validator context and combined semantic validation remain exploratory. +A proposal to expand context should identify a captured case and the evidence +missing from the validator request. +See the [current validator context](../internal/validators.md#supplemental-validation-context). diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index e2f3b3b..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,150 +0,0 @@ -# Prompt Quality Implementation Plan - -Follow the stages in order within one sprint. Intermediate stages may leave -dependent edits in progress. Verify the integrated result at the end of the -sprint. Read the [feature roadmap](prompt-quality.md) for the selected behavior -and [developer guide](../development.md) for the applicable policies. - -## Stage 1 — Finish the PromptKit Adoption Follow-ups - -**Complete.** Focused decoder, runner, reporting, and subprocess checks passed. - -1. In the debug case of `TestSubprocessCorrectionRepairConversation`, read the - run's `llm-execution.json` and compare its top-level - `summary.provider_attempts` with the number of requests captured by the - provider fixture. Use the existing fixture with an initial proposal, repair - branches, and nested validation. This checks that the shared proposal is - counted once and that repair and validation calls join the invocation total. - [Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go) -2. In `repairCandidates`, remove the extra JSON unmarshal and use the typed - response from one decode for the domain decision and the interaction - artifact. Reuse `DecodeRepairResponse` and its existing proposal-reference - and action checks. Where a decoded response fails a correspondence check, - retain it as diagnostic evidence alongside the raw response and failure. - Assess only delivered, structurally valid content. Keep the existing - invalid-output disposition, failure attribution, and revalidation behavior. - [Repair runner](../../internal/framework/runner/repair.go) - [Repair decoder](../../internal/framework/proposal_generation/repair.go) -3. In `BuildLLMExecution`, call `llm.Summarize` once per call and use the result - for both terminal coverage and the serialized summary. Keep the invocation - summary sourced from the native group. Preserve field names, ordering, and - the distinction between unknown usage and zero usage. - [Execution report](../../internal/framework/processreport/llm.go) - -## Stage 2 — Review Proposal and Repair Conversations - -**Complete.** The shipped and operator-replaced grammar requests retained the -proposal context, including a synthetic glossary entry and protected spelling. -The wire schema changed from a correction set to one repair result. The second -turn retained its branch's prior revision, and the other branch started from -the original proposal exchange. The adapter fixture confirmed separate proposal -and repair profiles. Runner fixtures confirmed updated previews after retargeting, -omission of an inapplicable after-preview, and withdrawal. - -The shipped grammar repair wording could read as a blanket restriction on -changes to other words. The revised text describes the grammar focus and leaves -the shared editorial allowance explicit in the rendered request. - -1. Use the existing subprocess fixture to exercise the production adapter and - coordinator with scripted responses. Inspect its captured request messages - and response schemas for the initial proposal and the two-turn repair. - Inspect the existing withdrawal case in the runner fixture as well. Use - prepared request artifacts for the rendered instructions and wire requests - for the effective schema. Inspect the shipped definition and an operator - replacement from the existing fixture. Use temporary test logging if - captured requests or artifacts are otherwise discarded after a passing run. - [Subprocess fixture](../../cmd/audita/correction_repair_integration_test.go) - [Runner fixtures](../../internal/framework/runner/repair_test.go) -2. Check whether the original system instructions, the initial assignment, - and the subsequent repair assignment give a coherent task. Check that the - continuation retains the relevant glossary and protected spellings when - supplied, selects the repair definition, role profile, and one-result - schema, and includes each branch's own revision history. Compare the latest - target and before/after preview with the candidate under review, including - a retargeted edit and the existing omission of an inapplicable after-preview. - Use `TestRoleClientContinuesIndependentRepairConversations` for distinct - proposal and repair profiles. - [Adapter conversation fixture](../../internal/framework/llm/conversation_test.go) -3. Check whether the findings give the model a specific defect to address, - whether the current preview identifies the edit being assessed, and whether - the instruction makes revision and withdrawal clear. Fix concrete - ambiguities in the affected bundled prompt or feedback presentation. Keep - module focus, editorial acceptance, and required evidence as specified by - the feature roadmap. If the requests are coherent, record the reviewed - cases and the outcome in the sprint summary. [Prompt registry](../internal/prompts.md) - -## Stage 3 — Review Evidence at the Validator Boundary - -**Complete.** Both model validators received speaker `A`, the literal -`second` → `Second` edit, full `second text` → `Second text` previews, -the preceding and following segments, and the matching `Second text` glossary -entry for the original and revised candidate. The capitalization objection -reached the next repair request with each validator's identity. The fixture -scripts that same objection for both validators, so it confirms feedback routing. -The rendered meaning-reversal instructions still require opposing full -assertions. This case revealed no missing evidence or unclear validator -instruction. The captured items retained their supplemental context, and -the fitting tests exercised omission of optional context under a smaller budget. - -The native invocation reported 10 provider attempts with complete terminal -coverage and unknown token usage. Per-call roles accounted for one proposal, -two initial validator requests, three repair requests, and four validator -requests across two revisions. The seven repair-related requests are already -included in the invocation total. The fixture's last revision repeats the -same edit and receives a scripted approval, so the counts measure execution. - -1. Follow a rejected candidate and, where available, its revision through the - captured proposal, validator request, decision, and next repair request. - Compare the target speaker, literal edit, before/after text, supplied - neighbors, matching glossary entries, and validator's stated reason. - Account for optional context omitted by the existing payload fitting. - [Validator payload and fitting](../internal/validators.md#supplemental-validation-context) -2. If evidence is already present but the instruction obscures how to use it, - revise the relevant prompt guidance or example. Preserve each validator's - remit, including the full-assertion reversal criterion. If the case needs - unavailable evidence, describe the missing evidence and the smallest - candidate payload change in the sprint summary. Broader context expansion - and combined semantic validation remain deferred. If no concrete gap - appears, the captured comparison completes this stage. -3. Use the native invocation summary to describe any additional requests - attributable to repair and revalidation. Use per-call evidence to identify - their roles. Treat group totals as encompassing potentially overlapping - work and account for terminal coverage separately from known or complete - usage. Report editorial findings from the requests themselves. - [Diagnostics and reporting](../internal/diagnostics-reporting.md) - -## Stage 4 — Verify the Integrated Result - -**Complete.** Subprocess, adapter, and runner fixtures together verified -independent repair histories, revalidation from the beginning, acceptance on -the last allowed turn, withdrawal, exhaustion, recoverable provider failure, -and preservation of independently approved edits. The adapter's cancellation -fixture retained the prepared request and raw response with no assessable -content. The runner records the cancellation before attempting to decode, and -the subprocess cancellation fixture verified that no transcript was published. - -Reviewed the revised shipped grammar request and the operator replacement as -sent by PromptKit. Supplied-data rendering and schema checks passed, including -the shared revision and withdrawal response examples. The focused tests and -full offline `go test ./...` suite passed with Go 1.27.3. Roadmap links and -anchors resolved, and `git diff --check` passed. - -1. Review the existing runner, adapter, and subprocess evidence together for - independent branches, full revalidation, final-turn acceptance, withdrawal, - exhaustion, recoverable unavailability, and cancellation before delivery. - Confirm that accepted siblings survive recoverable failures and that - undelivered content cannot become an approved correction. Extend the nearest - existing fixture only if a changed behavior needs coverage. Use the real - adapter and coordinator with controlled model responses for execution tests. -2. For any prompt edits, inspect representative rendered requests and verify - supplied-data rendering and affected examples against the applicable - response schemas. Use durable assertions for changed behavior and inspect - the prose as part of review. Follow the [testing policy](../policy/testing.md). - Update the [prompt registry](../internal/prompts.md), - [module guide](../internal/modules.md), or [validator guide](../internal/validators.md) - where their owned behavior changes. -3. Use focused tests for iteration in the touched packages. At sprint - completion, run `go test ./...` with the repository's Go toolchain and - offline fixtures. Verify affected links and run `git diff --check`. - Summarize the cases inspected, any evidence-backed edits, and the results - of verification. Live provider comparisons remain deferred. diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md deleted file mode 100644 index d62afe3..0000000 --- a/docs/roadmap/prompt-quality.md +++ /dev/null @@ -1,74 +0,0 @@ -# Prompt Quality Roadmap - -**Status:** The earlier six prompt improvements are implemented. This follow-up -uses Audita's PromptKit v0.23.0 integration to review and develop proposal, -validation, and repair prompts through the same execution path used in production. -Live-model comparisons and broader context expansion remain deferred. See the -[implementation plan](implementation.md) for the sprint sequence. - -## Baseline and Boundaries - -The [prompt registry](../internal/prompts.md), -[validator guide](../internal/validators.md), and -[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing -prompts separate durable system principles from initial proposal assignments, -guide independent and overlapping edits, supply current before/after repair -previews, include the target speaker in validation, and illustrate editorial -judgments, revision, and withdrawal. - -Preserve these selected decisions: - -- Module focus guides generation; otherwise acceptable minor editorial cleanup - remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). -- Keep the carefully tuned meaning-reversal criterion and distinct validator - responsibilities. Approvals require no explanation; confidence scores remain absent. -- Preserve literal spans, deletion, replacement policies, glossary context, - protected spellings, full revalidation, and deterministic application. -- Retain English defaults and native operator prompt replacement. Module sequence - reevaluation follows the [module guidance](../internal/modules.md). - -PromptKit now owns immutable captured exchanges, continuation assembly, bounded -coordination, per-operation deadlines, prepared-request notification, and selected -run accounting. Audita owns candidate identity, findings, previews, acceptance, -withdrawal, and transcript mutation. Follow-up prompt work must use these APIs; -it must not recreate message replay, turn loops, or token aggregation locally. - -## Target State - -Proposal and repair instructions should remain coherent across a complete -conversation. The original context and supplied glossary and protected -spellings remain available, each repair branch keeps its own history, and -continuations select the repair definition, profile, and output schema. -Both shipped and operator-replaced definitions use native prompt selection. -Repair feedback identifies the current edit and the validator's findings so -that the model can revise the proposal or withdraw it. - -Audita assesses every delivered revision, including the last allowed response. -Coordination stopping and correction acceptance are distinct outcomes. -Undelivered content is diagnostic evidence. Independent approvals survive -recoverable failures. Transient retries, structural repair, and correction -revision retain their separate budgets. - -Prompt and feedback changes should address concrete issues in captured -requests while preserving validator acceptance and the allowance for minor -editorial cleanup. Broader context and combined semantic validation remain -deferred. Any proposal to expand validator context should identify a concrete -case and the evidence missing from its request. This follow-up retains current -context fitting and apply-time conflict handling. - -Native execution diagnostics count the shared proposal once and include each -new repair and validator run. Terminal coverage and usage availability -remain distinct. Execution measurements describe work and cost. Review -prompt clarity using the supplied evidence and rendered instructions. -The adoption follow-up also removes duplicate repair decoding and per-call -summarization while preserving the existing diagnostic evidence. - -## Deferred Evaluation - -Formal corpus preparation and live-model comparisons remain optional follow-up -work. Keep live calls opt-in and outside the offline suite. When undertaken, -compare needed and unnecessary edits, missed corrections, validator disagreement, -and repair outcomes using independent editorial judgment. Native groups can -supply execution measurements without introducing a second accounting system. - -No blocking product decisions remain for this follow-up scope.