diff --git a/docs/config.md b/docs/config.md index 8adae11..33d1545 100644 --- a/docs/config.md +++ b/docs/config.md @@ -27,7 +27,7 @@ PromptKit loads its default catalog and Audita's embedded prompts, followed by optional `prompts.yml` and recursive `prompts/` YAML files beside the application main. Native imports, resource collections, inheritance, and replacement apply. Profiles, backends, models, credentials, admission, and provider parameters belong -to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/formats.md). +to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/formats.md). The engine captures its sources for an invocation. Edit files and run a new invocation to pick up changes. Inspection does not require model credentials. diff --git a/docs/integrations/openai-compatible-llm.md b/docs/integrations/openai-compatible-llm.md index 27d20c4..6b4ba99 100644 --- a/docs/integrations/openai-compatible-llm.md +++ b/docs/integrations/openai-compatible-llm.md @@ -19,7 +19,7 @@ failures; explicit zero makes one attempt. `llm.repair_attempts` counts additional generations for schema-invalid model output. The operation timeout covers preparation, capacity waiting, retries, generation, and repair. Caller cancellation or an earlier deadline also applies. See [configuration](../config.md#settings) -for defaults and the [PromptKit consumer contract](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/consumers/pkg-promptkit.md) +for defaults and the [PromptKit consumer contract](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/consumers/pkg-promptkit.md) for protocol and retry details. A model operation that exhausts structural repair, a live-parent local operation diff --git a/docs/internal/diagnostics-reporting.md b/docs/internal/diagnostics-reporting.md index ec3c924..2ba98be 100644 --- a/docs/internal/diagnostics-reporting.md +++ b/docs/internal/diagnostics-reporting.md @@ -48,6 +48,14 @@ artifacts can still contain sensitive transcript and prompt content. Execution position and optional operator ID are carried separately from generated safe artifact path coordinates. +Execution diagnostics include a native-group invocation summary and per-call +summaries. Terminal coverage is separate from usage completeness; unknown usage +is distinct from known zero. Overlapping summaries must not be added together. +The collector snapshots after work joins and retains domain labels and stable +presentation order. Prepared evidence comes from `OnPrepared`, so preparation or +admission failures may have no request snapshot. Coordinator execution failures +still finalize the attempted logical call and its available evidence. + ## Key Tests Useful tests: `internal/core/reporting/*_test.go`, diff --git a/docs/internal/llm-runtime.md b/docs/internal/llm-runtime.md index 565bf12..adb48ba 100644 --- a/docs/internal/llm-runtime.md +++ b/docs/internal/llm-runtime.md @@ -9,27 +9,44 @@ it never replaces the requested engine. Each resolved pipeline entry receives role-bound clients with its selected profiles and effective budgets. Clients share engine-local backend admission. -The adapter uses `PrepareExecution` and `RunPrepared` on the same handle, passes +The adapter uses ordinary `Run` with `OperationTimeout` and `OnPrepared`, passes Audita's per-call schema and repair budget, and decodes only validated results. +The operation budget includes preparation, admission, retries, and structural +repair. `ErrOperationTimeout` remains recoverable even when wrapped by a +preparation error. Parent cancellation remains distinct. The prepared notification +provides the actual request snapshot; failures before notification have no snapshot. PromptKit owns backend and credential selection. The adapter uses captured credential source names for artifact redaction without inspecting unrelated environment values. See the [integration contract](../integrations/openai-compatible-llm.md) for failure behavior and the [configuration reference](../config.md) for sources. -When correction repair is enabled, the proposal call retains its prompt, -effective variables, prepared messages, and full final response for section-local -replay. The adapter renders the repair definition with the selected profile, -appends only that branch’s history, and checks the original prepared prefix -before execution. The complete conversation is sent to the selected provider; -provider rejection ends that branch as unavailable. The requested repair -definition is retained in call attribution alongside native base-prompt metadata. +When correction repair is enabled, the proposal call captures a native immutable +PromptKit exchange. Each actionable correction starts an independent +`Engine.Coordinate` branch from that exchange. PromptKit assembles continuation +history, owns the additional-turn budget, and delivers the last permitted response +for assessment. Follow-ups select the repair prompt, profile, and output schema +directly; Audita does not rerender the original proposal or assemble replay messages. +An opaque exchange in the structured-output contract keeps provider state outside +modules and validators. The adapter shares request construction, notifications, +observers, and result/failure finalization between ordinary and coordinated calls. + +Audita owns actionable rejection, revision decoding, withdrawal, current previews, +and full revalidation. Accepted siblings remain independent. Requesting more work +at the budget boundary produces exhaustion without allocating another logical call. +A canceled, undelivered response supplies diagnostic evidence only; it cannot enter +the approved set. Repair disabled in configuration bypasses coordination entirely. The runner supplements each repair instruction with its current proposal preview as described in the [module guide](modules.md#failure-and-outcome-behavior). The bundled prompts separate durable system principles from the initial user assignment so later repair instructions can allow acceptable minor cleanup; see [prompt instruction ownership](prompts.md). -The observation collector tracks logical calls and execution events. Ordinary +The observation collector tracks logical calls and execution events, and enrolls +selected runs in a native `ExecutionGroup`. Proposal, repair, and nested validator +clients explicitly share the invocation observer; history reuse adds no extra run. +Native summaries deduplicate terminal totals by run ID and retain terminal coverage +separately from usage presence and completeness. Logical calls and run membership +can differ, especially on failures before execution. Ordinary reports use compact safe outcomes; request and response evidence is optional debug material. The runner preserves ordered pipeline traversal and deterministic section collection while canceling and joining peers on fatal errors. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index ff44960..2bfaf7f 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -20,7 +20,9 @@ formats; [configuration](../config.md) owns settings; the Recoverable failed batches mark their candidates unavailable; successful survivors continue through all required validators. 6. Actionable rejections can enter independent section-local repair branches. - Each revision is mapped to the original section and revalidated from the + PromptKit coordinates each branch from the captured proposal exchange, owning + its history and additional-turn limit. Each revision, including the final + permitted response, is mapped to the original section and revalidated from the start. Withdrawal, exhaustion, and recoverable failures finish the branch without affecting independent approvals. 7. Only fully approved proposals reach application. Corrections apply in diff --git a/docs/oss/attribution.md b/docs/oss/attribution.md index d7dbc01..5c3f54e 100644 --- a/docs/oss/attribution.md +++ b/docs/oss/attribution.md @@ -210,7 +210,7 @@ License and notices: [Apache 2.0](licenses/Apache-2.0.txt). ## PromptKit -Selected module: `go.maximumdirect.net/promptkit` at `v0.21.0`. +Selected module: `go.maximumdirect.net/promptkit` at `v0.23.0`. Copyright (C) 2026 Eric Rakestraw @@ -218,9 +218,9 @@ SPDX-License-Identifier: LGPL-3.0-or-later License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | [GPL v3](licenses/GPL-3.0.txt) | -[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/COPYRIGHT). +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/fcbff80ab227397a03935ab3bf8114481c0736a7/COPYRIGHT). -[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09). +[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/fcbff80ab227397a03935ab3bf8114481c0736a7). ## yaml.v3 diff --git a/docs/roadmap/prompt-quality.md b/docs/roadmap/prompt-quality.md index 1e94b59..a24f749 100644 --- a/docs/roadmap/prompt-quality.md +++ b/docs/roadmap/prompt-quality.md @@ -1,229 +1,106 @@ # Prompt Quality Roadmap -**Updated:** 2026-11-11 +**Status:** The earlier six prompt improvements are implemented. This follow-up +uses Audita's PromptKit v0.23.0 integration to review and develop proposal, +validation, and repair prompts through the same execution path used in production. +Live-model comparisons and broader context expansion remain deferred. -**Status:** All six stages implemented and verified. Broader context changes and -live-model editorial evaluation remain deferred as described below. The plan -followed the review of revision `d155bf4`, which covered all eleven prompt -definitions, shared instructions, response schemas, and request assembly. -That review identified design concerns, not measured live-model regressions. +## Baseline and Boundaries -## Target State and Boundaries +The [prompt registry](../internal/prompts.md), +[validator guide](../internal/validators.md), and +[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing +prompts separate durable system principles from initial proposal assignments, +guide independent and overlapping edits, supply current before/after repair +previews, include the target speaker in validation, and illustrate editorial +judgments, revision, and withdrawal. -Audita's proposal, validation, and repair prompts will give each model call a -coherent assignment and the evidence needed to carry it out. Proposals will -better account for interacting edits, repair will receive an explicit view of -its current proposed result, and validation will identify the target speaker. -Focused examples will clarify difficult editorial judgments. +Preserve these selected decisions: -Preserve the established design: +- Module focus guides generation; otherwise acceptable minor editorial cleanup + remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). +- Keep the carefully tuned meaning-reversal criterion and distinct validator + responsibilities. Approvals require no explanation; confidence scores remain absent. +- Preserve literal spans, deletion, replacement policies, glossary context, + protected spellings, full revalidation, and deterministic application. +- Retain English defaults and native operator prompt replacement. Module sequence + reevaluation follows the [module guidance](../internal/modules.md). -- Module focus guides proposal generation while otherwise acceptable minor - editorial cleanup remains permissible across modules, as selected in - [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md). -- Validators retain distinct responsibilities. Preserve the carefully tuned - meaning-reversal criterion and its focus on reversal of the full assertion. -- Keep literal-span matching, deletion semantics, separate glossary context and - protected spellings, and full before/after validation. -- Keep independent repair conversations with full history, successful withdrawal, - and full revalidation of revisions. Validators explain defects; repair owns - revision or withdrawal. Approvals continue to need no explanation. -- Retain confidence-free responses, English defaults, existing module identities, - and native operator prompt replacement. The current module sequence is outside - this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md). +PromptKit now owns immutable captured exchanges, continuation assembly, bounded +coordination, per-operation deadlines, prepared-request notification, and selected +run accounting. Audita owns candidate identity, findings, previews, acceptance, +withdrawal, and transcript mutation. Follow-up prompt work must use these APIs; +it must not recreate message replay, turn loops, or token aggregation locally. -The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md), -and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline. -Update these canonical owners when the corresponding changes land. +## 1. Review Complete Native Conversations -## Implementation Sequence +Use captured proposal exchanges and independent coordinator branches to inspect +representative proposal-to-repair conversations, including a second revision and +withdrawal. Review actual `OnPrepared` snapshots for instruction authority, +response-format transitions, glossary/protected-term context, and fresh previews. +A follow-up selects its repair definition, role profile, and schema directly. -### 1. Align Proposal and Repair Instruction Authority +Exercise operator replacements through native prompt discovery and precedence. +Compare the request that actually executes rather than separately rendering an +approximation. Reusing captured history must preserve the proposal context while +keeping each correction's repair history private. -Proposal system messages currently include categorical task restrictions, such -as prohibiting punctuation or grammar edits in the homophone module. Repair -inherits those messages and then permits acceptable minor editorial cleanup in a -later user message. These instructions can compete across authority levels. +Record concrete editorial concerns and revise affected prompts when warranted. +Do not change validator acceptance or narrow allowable cleanup merely to make +proposal and repair assignments identical. -- Put durable correction principles in the system messages: evidence-supported - work, established protection and meaning constraints, and the distinction - between requested focus and acceptable editorial corrections. -- Put the initial proposal's focused assignment and task-specific exclusions in - its user message. Preserve the focused tasks rather than asking every module - to perform every kind of cleanup. -- Let each repair instruction give the subsequent reassessment assignment, - including the selected editorial allowance, without conflicting with an - inherited system prohibition. -- Review each complete proposal-to-repair conversation, including later repair - attempts, for coherent instructions and response-format transitions. +## 2. Assess Repair Feedback and Outcomes -This implements the existing acceptance decision; it does not tighten validators -or change the repair lifecycle. Keep the validator prompts' separate remits, -especially meaning-reversal review, intact. +Use the real coordinator with controlled model responses for deterministic +integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion, +unavailable responses, and independent accepted siblings. Revalidate each revision +from the beginning. A normal coordination stop is not evidence of acceptance; +Audita's domain outcome determines that. Canceled undelivered content is evidence +only and must never enter the approved set. -### 2. Guide Independently Useful and Overlapping Corrections +Use current target previews and validator-owned findings to assess whether a +repair request explains the defect sufficiently. Keep presentation changes in +prompt assets and domain feedback construction; keep history and budget mechanics +upstream. Retain separate transient retry, structural repair, and correction +revision budgets. -The shared proposal contract already makes each edit refer to the supplied -segment rather than the imagined result of another proposal. Add guidance for -edits that overlap or depend on one another: +## 3. Diagnose Evidence Gaps Using Actual Requests -- Prefer independently useful corrections with literal spans that identify the - intended changes. -- Combine changes into one proposal when their spans overlap or when one edit - requires the other. Preserve unchanged context needed for unique matching or - for selecting an occurrence under the glossary replacement policy. -- Continue to allow multiple independent corrections to the same segment. +For a concrete rejected correction, compare the captured proposal evidence with +what the validator actually received. The target speaker is already included. +Broader validation context remains evidence-dependent: propose a focused change +only when a captured case demonstrates relevant evidence missing from validation. -For example, separate proposals to capitalize `hello` and replace -`hello world` with `hello world` may both pass validation against the original, -but applying the capitalization first prevents the second span from matching. -A combined replacement can express both edits against the original text. +Use native execution-group totals to distinguish additional model work from +shared initial history. Select validator membership explicitly, count the initial +proposal once, and do not add overlapping group totals. Preserve unknown usage +and terminal coverage separately. These measurements describe execution and cost; +they do not establish editorial quality. -These are prompt instructions. Add no new runtime overlap rejection, proposal -limit, or application-order rule. Assess combined semantic effects separately -if practical examples establish a need for additional processing changes. +Existing local examples have not established a need for broader validation windows +or combined semantic validation. Keep context fitting and apply-time conflict +handling unchanged until concrete cases justify further work. -### 3. Supply Repair with the Current Before/After Segment +## 4. Verification and Documentation -The repair request currently identifies the proposal, its current correction, -and validation findings. Although the original section remains available in -conversation history, repair must locate it and reconstruct the proposed result. +Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter +and coordinator with deterministic model-boundary fixtures. Protect supplied-data +rendering, schemas, profile selection, current previews, branching, revalidation, +withdrawal, and operator replacement. Inspect representative rendered prose as +part of review; avoid phrase checks, prose snapshots, and tests that duplicate +upstream history or accounting algorithms. -- Add the original target segment and its proposed resulting text to repair - context when the current proposal is applicable. -- Reuse the same domain preview semantics used by validation, including literal - deletion, whitespace, and replacement of every matching occurrence where - applicable. Avoid a second implementation of replacement behavior. -- Refresh this view for each repair attempt, using the current proposal version - and target rather than a stale view of the initial proposal. -- When a preview cannot be produced, retain the relevant findings and available - source context without fabricating a resulting segment or implying application - succeeded. -- Keep the full conversation history, proposal references, and revision or - withdrawal response contract. Explain the added view as evidence for assessing - the current correction, not an already-applied transcript change. +Update canonical prompt and validator documentation when behavior changes. +Keep examples valid against their response schemas and verify affected links. +Run focused tests and the full offline suite for runtime changes. Pure wording +changes need proportionate review, not new change-detector tests. -This should make spacing, deletion, and replacement consequences easier to -judge without requiring the model to reproduce Audita's application mechanics. +## Deferred Evaluation -### 4. Improve Validation Evidence +Formal corpus preparation and live-model comparisons remain optional follow-up +work. Keep live calls opt-in and outside the offline suite. When undertaken, +compare needed and unnecessary edits, missed corrections, validator disagreement, +and repair outcomes using independent editorial judgment. Native groups can +supply execution measurements without introducing a second accounting system. -Add the target segment's speaker to each validation item. Neighboring segments -already carry their speakers; the candidate's identity completes that context -and helps distinguish continued speech, interruptions, and another speaker's -response. Carry the supplied value through the existing payload construction -and token accounting without inferring a missing speaker. - -Investigate the broader evidence imbalance using available practical cases: -proposal sees a section, while validation sees immediate neighbors and selected -glossary entries. A correction supported several turns earlier may appear -unsupported to a validator. Compare the evidence actually supplied to both -calls before attributing a rejection to prompt wording. - -Adding the target speaker is agreed implementation work. Substantial expansion -of validation context remains evidence-dependent: record a concrete case and a -focused follow-up recommendation if needed. Retain the current context selection -and fitting behavior otherwise. Stronger instructions cannot supply missing -evidence, and this roadmap does not require a broader context redesign. - -### 5. Add Focused Editorial and Repair Examples - -Add a small number of examples where they clarify judgments that current -instructions leave implicit: - -- Apparent filler that contributes meaning, such as `like` expressing - approximation or `I mean` introducing a substantive self-correction. Use - surrounding context to contrast meaningful usage with removable dysfluency. -- One compact repair revision and one withdrawal, using the existing response - schema and making withdrawal a successful preservation of the original text. -- A combined overlapping edit illustrating step 2. - -Place examples in the relevant proposal, validation, or shared repair guidance. -Keep existing useful contextual contrasts. Examples should illustrate reasoning -without becoming unconditional token-based editing rules or an extensive -rulebook. Maintain the narrow meaning-reversal instructions. - -### 6. Verify Integration, Documentation, and Test-Policy Compliance - -Follow the [testing policy](../policy/testing.md) throughout implementation and -finish with a focused review of affected tests: - -- Verify repair context reflects the current proposal and the shared preview - behavior, including representative deletion, replacement-policy, subsequent - revision, and unavailable-preview cases where coverage is warranted. -- Verify target-speaker data reaches validation and participates in existing - payload accounting. Preserve coverage of response handling, revision, - withdrawal, revalidation, and operator prompt replacement. -- Review rendered messages and representative complete repair conversations - directly for instruction authority, editorial intent, and format consistency. - Check response examples against their schemas. -- Remove any affected tests that freeze bundled prompt wording through phrase - checks, exact prose comparisons, or snapshots. Do not replace them with checks - for the new wording. Retain tests of supplied data and meaningful machine - contracts where their protective value justifies their maintenance cost. -- Update the relevant canonical prompt, module, validator, and runtime documents - as behavior lands. Update maintained examples where affected; check links and - whitespace. - -Run the appropriate package checks, build, and full offline suite for the -implementation under the [development guidance](../development.md#baseline-validation). -Report the test-policy review's meaningful removals, revisions, and retained -coverage. Offline tests establish data and execution contracts; they do not -establish live-model editorial quality. - -## Deferred Evidence and Follow-Up - -The evidence review examined the maintained local, campaign, and prompt-edit -transcript examples, adjacent-context validation fixtures, and proposal -application fixtures. The glossary example is a single-segment correction; -the validation fixture supplies the troupe definition and adjacent evidence -needed to assess its proposed name change. These sources contain no recorded -live-model rejection demonstrating a need for more distant context. The target -speaker is now included; broader selection and context fitting remain unchanged. -Revisit the window when a captured case shows material evidence in the proposal -request that was absent from validation. - -The existing stale-span application fixture demonstrates the mechanical overlap -addressed by the new guidance. It does not establish a need for new combined -semantic validation, so application ordering and rejection rules remain unchanged. - -Formal evaluation corpus preparation and live-model comparisons remain deferred -and are not prerequisites for the agreed changes. Broader validation context and -additional handling of combined semantic effects should follow concrete evidence -rather than hypothetical restrictions. Any later evaluation should examine -needed and unnecessary edits, missed corrections, validator disagreement, and -repair outcomes using independent editorial judgment. Keep live calls opt-in -and separate from the default offline suite. - -No blocking product decisions remain for the implementation scope above. - -## Completion and Verification - -- The four proposal definitions now separate durable system principles from - their initial user assignments. The shared proposal guidance covers overlapping - and dependent edits; the grammar prompt illustrates a combined edit. -- Every repair attempt receives the current target's original text when - available and its previewed result when applicable, using the same domain - replacement function as validation. Revision and withdrawal examples accompany - the shared repair guidance. -- Validation includes the target speaker in rendered input and token accounting. - Spoken-word proposal and editorial review examples distinguish meaningful - apparent filler from removable hesitation. The meaning-reversal definition and - all response schemas retain their existing contracts. -- Behavioral tests cover repair deletion and replace-all previews, ambiguous or - missing spans, blank results, missing targets, and refresh after retargeting. - Existing validation tests now cover target-speaker propagation, absence, and - budget fitting, including parity with serialized payload estimates. -- The affected prompt, module, request-assembly, and conversation tests were - reviewed for wording checks. No further bundled-prose change detectors were - found to remove; the earlier cleanup remains effective. Retained assertions - protect supplied-data rendering, schema selection, operator replacement, - conversation history, revalidation, withdrawal, and outcomes. New assertions - check request data and behavior rather than prompt phrases. -- All eleven definitions prepared through the CLI. Four illustrative complete - conversations, including subsequent repair requests, were reviewed, and the - sixteen embedded JSON response examples passed schema checks. The application - build and full offline Go suite passed. Canonical documentation and maintained - preview examples were updated; links and whitespace were checked. - -These checks verify implementation and editorial intent, not live-model quality. +No blocking product decisions remain for this follow-up scope. diff --git a/docs/roadmap/promptkit-implementation.md b/docs/roadmap/promptkit-implementation.md deleted file mode 100644 index 0f6bae5..0000000 --- a/docs/roadmap/promptkit-implementation.md +++ /dev/null @@ -1,168 +0,0 @@ -# PromptKit v0.23.0 Implementation Plan - -**Status:** Ready for implementation; all tasks below are pending. - -The [adoption roadmap](promptkit.md) owns the target behavior and boundaries. -This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed -to establish full adoption. It does not authorize application code changes during -the current documentation pass. - -## 1. Upgrade and unify execution - -- [ ] Update `go.mod` and `go.sum` directly to - `go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove - temporary replacements/workspace files. Check the selected module graph and - compatibility with the existing CommandKit/ckprompt integration. -- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around - shared request construction, response interpretation, and error mapping that - can serve both ordinary calls and coordinator-owned turns. Keep those helpers - narrow; do not wrap `Coordinate` in another turn loop. -- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary - `Run` for calls whose prepare/run split exists only to obtain diagnostics. - Avoid retaining a redundant child timer around the same ordinary operation. -- [ ] Supply `OnPrepared` to collect effective metadata and selected credential - source names, and retain prepared content for debug when enabled. Use the - delivered snapshot without a separate `Prepare` call. Clear per-call state - so a failure before notification cannot reuse a prior turn's evidence. -- [ ] Preserve per-role profile/schema/variable selection, transient and - structural-repair budgets, ordinary observers, output decoding, and error - disposition. Map `ErrOperationTimeout` explicitly, including timeouts during - preparation. Distinguish parent cancellation and provider-local deadlines. - Retain the public processing reason codes where their meanings still apply. -- [ ] Ensure call finalization runs on both results and errors; coordinator - execution errors do not arrive as ordinary response callbacks. Preserve the - last attempted operation's diagnostic and failure context for final reporting. - -**Verification:** focused adapter tests for exact prepared evidence after a -provider failure, absent evidence before notification, independent operation -budgets, timeout classification, metadata redaction, and ordinary observer -composition. Use controlled model clients and cancellation. Retain explicit -prepared execution only where an identified consumer actually requires it. - -## 2. Replace history and repair-loop mechanics - -- [ ] Replace the local `Exchange`/`Replay` model in - [structured completion contracts](../../internal/framework/structuredoutput/call.go) - with a minimal opaque bridge to native captured exchanges. The adapter owns - PromptKit execution; modules and validators must not reconstruct messages or - depend on provider-specific state. -- [ ] Capture the original proposal run through `CaptureExchange`. Select the - repair prompt directly for a continuation; remove original-prompt rerendering, - manual assistant/suffix assembly, prefix comparison, and `RepairInstruction` - history plumbing. Preserve prepared snapshots used only for diagnostic output. -- [ ] Replace the attempt loop in - [repair coordination](../../internal/framework/runner/repair.go) with one - adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal. - Keep scheduling across proposals in Audita. Pass the resolved correction repair - limit as `AdditionalTurns`, and bypass coordination when repair is disabled. -- [ ] Keep domain branch state in the callback: original proposal identity, - current revision, local segment mapping, current before/after preview, safe - rejection findings, and domain outcome. The initial callback uses existing - findings without rerunning the initial batch validation. -- [ ] Build each follow-up request with the repair definition, role profile, - repair schema, variables, operation timeout, prepared notification, and selected - observers. Let the coordinator attach `ContinueFrom` and manage history. -- [ ] Process `ConversationTurn.Result` only when present. Check structural - validation before schema decoding, interpret withdrawal, and revalidate every - revision from the beginning using the parent callback context. Validator calls - select their own profiles, timeouts, and observers through the shared adapter. -- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain - outcomes. If rejection remains actionable, request a further turn even at the - budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate - an independent exhaustion loop. The final permitted revision is still assessed. -- [ ] Keep diagnostic attempt coordinates and operation records, but allocate - execution evidence only for runs actually started. The callback may construct - a request that is never executed because the turn budget is exhausted. -- [ ] Map execution errors, callback errors, stopping, exhaustion, and - `Undelivered` separately. Unassessed or structurally invalid content never - reaches the approved set. Preserve accepted sibling proposals and apply-time - checks; fatal/canceled work still cancels and joins peers. - -**Verification:** a controlled workflow with one initial approval and two rejected -proposals demonstrates independent histories, withdrawal, revised acceptance, -full revalidation, and deterministic application. Cover final-turn acceptance, -exhaustion, zero repair, structural invalidity, recoverable failures, parent -cancellation before delivery, and capacity-one validator calls. Assertions should -exercise Audita outcomes through the real upstream coordinator with a fake model -boundary, rather than test a fake coordinator's loop behavior. - -## 3. Adopt native execution accounting - -- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe` - callback with existing diagnostic observers using `ComposeObservers`. Ensure - observation remains enabled wherever required accounting is consumed, - independently of whether debug artifacts are enabled. -- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use - `CoordinateRequest.Observer` for continuation membership and select membership - separately for nested validation requests. Keep conversation parent identity - distinct from Audita's stage/section/proposal/validator labels. -- [ ] Replace the terminal-map deduplication and usage summation in - [observation summaries](../../internal/framework/llm/observations.go) with - `ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation - boundaries. Retain logical call metadata, safe event evidence, statuses, and - deterministic ordering that the upstream group does not supply. -- [ ] Carry terminal coverage separately from usage known/completeness into the - internal summary and its diagnostic representation. Do not present missing - terminals as complete zero accounting. Keep the established ordinary report - scope; adding token payloads to the ordinary errors/recoveries report is not - part of this upgrade. -- [ ] Count shared initial work once in the invocation group; count each new - continuation and validator run separately. Do not sum overlapping subgroup - totals or attempt events on top of their terminal totals. Sort any presented - member lists deterministically rather than relying on native snapshot ordering. -- [ ] Snapshot final totals after all selected work has joined. Groups add no - separate close/wait lifecycle. Audit and delete local aggregation that became - redundant; document the purpose of any remaining application-specific counters. - -**Verification:** integration tests demonstrate composition with existing -observers, explicit validator membership, shared initial work, failure terminal -facts, unknown versus zero usage, missing terminal coverage, and concurrent calls. -Reuse upstream group semantics; test Audita's enrollment and presentation instead -of duplicating the library's unit suite. - -## 4. Cleanup, documentation, and acceptance - -- [ ] Audit references to the old exchange/replay types, prefix checks, - `RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and - terminal aggregation. Delete dead production helpers and replace tests tied to - those implementations with focused behavior coverage. A thin local adapter is - justified; a parallel conversation/coordinator/accounting implementation is not. -- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md), - [pipeline](../internal/pipeline.md), - [diagnostics/reporting internals](../internal/diagnostics-reporting.md), and - [LLM integration contract](../integrations/openai-compatible-llm.md) as behavior - lands. Update pinned upstream references and dependency attribution as needed. - Document the selected notification boundary and timeout mapping without claiming - that preparation or admission failures always have a prepared snapshot. -- [ ] Update configuration/operations references only for actual externally - visible changes. No prompt rewrite, validator tuning, new configuration format, - compatibility shim, new scheduler, or unavailable-decision regeneration is - required by this migration. -- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect - domain and integration behavior, remove obsolete implementation-coupled tests, - and avoid prompt word/phrase change detectors. Review final tests for redundancy. -- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled - check of changed concurrent adapter/runner/accounting paths. Build the CLI and - exercise existing offline processing and inspection coverage. No paid provider - calls are required for acceptance. -- [ ] Review the final diff, verify local Markdown links and `git diff --check`, - and confirm no local dependency replacements or workspace artifacts remain. - Verify the selected PromptKit version is v0.23.0. -- [ ] Confirm all three upstream capability families are used in production - processing and the agreed local machinery is retired. Record completion only - after this end-state audit; move useful current contracts into their canonical - owners and retire these temporary planning documents under repository policy. - -Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain: - -```sh -go list -m go.maximumdirect.net/promptkit -go test ./... -go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport -go build -o ./bin/audita ./cmd/audita -git diff --check -``` - -Adjust focused package checks if code ownership moves. A green dependency-only -suite is not evidence that the new workflow has been adopted; acceptance also -requires the coordinator, conversation, and group integration scenarios above. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md deleted file mode 100644 index 0cb8969..0000000 --- a/docs/roadmap/promptkit.md +++ /dev/null @@ -1,136 +0,0 @@ -# PromptKit v0.23.0 Adoption Roadmap - -**Status:** Accepted target; implementation pending. This pass changes planning -documents only. The [implementation plan](promptkit-implementation.md) owns the -ordered tasks and verification. - -## Objective and Baseline - -Upgrade Audita directly from PromptKit v0.21.0 to v0.23.0 and fully adopt its -conversation, coordinator, and execution-group APIs. Removing redundant local -machinery is an explicit completion requirement. A dependency-only upgrade or -continuation adoption with the old repair loop retained is an intermediate state. - -The current [dependency manifest](../../go.mod) pins v0.21.0. Audita captures its -own exchange and replay values, reconstructs and compares conversation prefixes, -appends repair history, manages the revision loop, and aggregates terminal -observations locally. Each role client wraps explicit preparation and execution -in a local deadline. See the [LLM runtime](../internal/llm-runtime.md), -[pipeline](../internal/pipeline.md), and -[diagnostic implementation](../internal/diagnostics-reporting.md). - -[PromptKit v0.23.0](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/releases/v0.23.0.md) -includes the v0.22.0 conversation and accounting capabilities plus the execution -controls needed for full coordinator adoption. The former upstream timeout and -prepared-request gaps are closed; no intermediate v0.22.0 rollout or further -upstream feature gate is required. The Go minimum and dependency versions are -unchanged by these upstream releases. - -The release review verified Audita's existing suite against the tagged v0.23.0 -source through a temporary module override and selected upstream packages with -race detection. That is compatibility evidence, not verification of the migration -described here. Implementation must use the released dependency, without a local -replacement or committed workspace file. - -## Target Ownership - -| Responsibility | Target owner | -| --- | --- | -| Immutable captured history, continuation assembly, and parent run identity | PromptKit `Exchange`, `CaptureExchange`, and `ContinueFrom` | -| Sequential turns, additional-turn budget, callback delivery, and coordination termination | PromptKit `Engine.Coordinate` | -| Whole-operation deadlines and timeout attribution | PromptKit `OperationTimeout` and `ErrOperationTimeout` | -| Exact prepared-request notification before generation | PromptKit `OnPrepared` | -| Selected run membership, deduplication, and terminal count/usage aggregation | PromptKit `ExecutionGroup` and `ComposeObservers` | -| Candidate identity, actionable rejection, feedback, revision decoding, withdrawal, and full revalidation | Audita | -| Branch scheduling, accepted sibling proposals, application order, and transcript mutation | Audita | -| Domain operation labels, failure disposition, reports, redaction, and debug artifact storage | Audita | - -Use the supported public API through Audita's framework adapter boundaries. -Keep module and validator contracts domain-oriented. Any bridge needed by the -runner must delegate to the real coordinator and retain opaque conversation -state, rather than implementing another conversation engine or general workflow -framework. - -## Selected Behavior - -### Execution and diagnostics - -Use ordinary `Engine.Run` with per-request `OperationTimeout` and `OnPrepared` -where explicit preparation currently serves only to inspect and execute the same -request. Map the resolved Audita operation timeout onto each proposal, validation, -and repair operation. Keep provider, transport-retry, structural-repair, and -correction-revision budgets distinct. - -Use `ErrOperationTimeout` to identify the selected local budget, while retaining -Audita's parent-cancellation and recoverable-provider-failure handling. Do not -classify every wrapped preparation error as fatal before considering its timeout -identity. Ordinary configuration failures remain fatal. If split preparation is -still needed for a concrete consumer, one caller context must cover both calls -when a total preparation-plus-execution deadline is required; the upstream -prepared-handle duration starts only at execution. - -`OnPrepared` receives the actual detached snapshot. Retain the metadata needed -for attribution and credential-source redaction, and capture content for optional -debug evidence. Keep the callback short; application validation belongs in the -next-turn callback after admission is released. Notification may be absent on -preparation/admission failure or cancellation before delivery. Reports must not -invent missing snapshots. Optional debug write failures remain nonfatal. - -### Conversations and correction repair - -Capture the proposal exchange when repair needs it. Every rejected candidate's -branch starts from that same immutable exchange; subsequent turns remain private -to that branch. Follow-ups select the module's repair prompt, role profile, -variables, and repair output contract through ordinary PromptKit requests. -Retain native operator prompt overrides and the existing domain prompt content. - -Use one coordinator per actionable correction. Its initial callback uses existing -rejection findings. Each newly delivered response must pass structural validation -before decoding; captured content alone does not imply validity. A decoded -revision is fully revalidated before acceptance. Withdrawal, unavailable results, -and accepted revisions remain Audita outcomes; a normal coordinator stop is not -itself an approval. - -The last permitted response follows the same assessment path. Further requested -work beyond the budget maps to exhaustion. Skip coordination when Audita's -correction-repair setting disables it: upstream's zero budget still invokes an -initial callback. Cancellation's `Undelivered` response is diagnostic evidence, -not an approved correction, and must not trigger model work after cancellation. -Recoverable branch failures preserve independent approvals; fatal failures and -caller cancellation retain Audita's cancel-and-join behavior. - -### Execution groups - -Use native groups as the source of aggregate provider-attempt, retry, repair, -and available token totals. Explicitly select membership for proposal calls, -coordinated continuations, and validator calls. Coordinator observers cover its -own turns; nested validators require their own observer selection. - -Maintain an invocation group for combined execution totals. Add narrower groups -only for accounting views Audita consumes. A shared initial proposal run belongs -once to the invocation total; replaying its history does not enroll it again. -Each new call's reported history tokens still count. Overlapping group totals -must not be summed as disjoint work. - -Preserve upstream usage presence/completeness and terminal coverage separately. -Keep domain logical-call records and deterministic presentation where needed; -their count need not equal observed run membership. Group summaries do not replace -Audita's errors/recoveries, validation decisions, or per-operation failure facts. - -## Delivery and Completion - -Implement in ordered, reviewable stages: dependency and execution adapter; -captured conversations and coordinator; execution groups; cleanup, integration -verification, and canonical documentation. All stages belong to this adoption. - -Completion requires removal of local exchange/replay history, prefix checks, -repair turn-budget loops, and duplicate terminal aggregation where replaced by -PromptKit. Retain local behavior only where it has a concrete Audita domain or -presentation responsibility. Preserve transcript correctness, repair prompts, -validation semantics, deterministic reports, standard inspection commands, and -operator configuration unless a deliberate change is documented and tested. - -No product-level open decisions remain. Exact internal bridge signatures are -implementation choices governed by this ownership split. Future validator -decision recovery and new scheduling behavior remain in the -[deferred work catalog](future.md). diff --git a/go.mod b/go.mod index 25aa514..2588365 100644 --- a/go.mod +++ b/go.mod @@ -3,11 +3,12 @@ module gitea.maximumdirect.net/eric/audita/v3 go 1.27.0 require ( + github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 go.maximumdirect.net/ckconfig v0.7.0 go.maximumdirect.net/ckprompt v0.7.0 go.maximumdirect.net/commandkit v0.6.0 go.maximumdirect.net/configkit v1.2.0 - go.maximumdirect.net/promptkit v0.21.0 + go.maximumdirect.net/promptkit v0.23.0 go.maximumdirect.net/releasekit v0.9.0 gopkg.in/yaml.v3 v3.0.1 ) @@ -16,7 +17,6 @@ require ( github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/rogpeppe/go-internal v1.14.1 // indirect - github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect github.com/spf13/cobra v1.10.1 // indirect github.com/spf13/pflag v1.0.9 // indirect github.com/yuin/goldmark/v2 v2.1.6 // indirect diff --git a/go.sum b/go.sum index f6ca140..dffb4c4 100644 --- a/go.sum +++ b/go.sum @@ -32,8 +32,8 @@ go.maximumdirect.net/commandkit v0.6.0 h1:4PHhMIlZuuvreIxZhUazeweg6zoYixEvhM9LJn go.maximumdirect.net/commandkit v0.6.0/go.mod h1:ZS98DewKNVBygKVDoh5MFD09CnPlP37dS9pVsID5Eoc= go.maximumdirect.net/configkit v1.2.0 h1:c4tDJ6Xeh/Jg90UZplLxsnf500lvrZBVj0E3i1ysir8= go.maximumdirect.net/configkit v1.2.0/go.mod h1:s1zLJ9GGB/1Q4a+aOHMLISonJxWFgNWpA6SpPlOafeA= -go.maximumdirect.net/promptkit v0.21.0 h1:Ws4y28lnW+lIXAq4tGhXVFnQV/6AqkDChhg8knIZFrU= -go.maximumdirect.net/promptkit v0.21.0/go.mod h1:pi9acdCi80HWyzY+XYwRvMEoCiTcDWKyeBh3mzHdIFo= +go.maximumdirect.net/promptkit v0.23.0 h1:Q17wmRHWSznc1yW1MuB7dxuT3KVYRCzHI9pKBzpXhq0= +go.maximumdirect.net/promptkit v0.23.0/go.mod h1:pi9acdCi80HWyzY+XYwRvMEoCiTcDWKyeBh3mzHdIFo= go.maximumdirect.net/releasekit v0.9.0 h1:sK+Aoj6gS6ATU/1reocjwBekp4YETYENwkcX5ti6mrE= go.maximumdirect.net/releasekit v0.9.0/go.mod h1:QE4BLrAhVJTzi+/YTcVl21K/qZp9ojRLg5kuoy3Zb44= go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4= diff --git a/internal/framework/llm/adapter.go b/internal/framework/llm/adapter.go index 21f5861..709858f 100644 --- a/internal/framework/llm/adapter.go +++ b/internal/framework/llm/adapter.go @@ -5,7 +5,6 @@ import ( "encoding/json" "errors" "fmt" - "reflect" "sync" "time" @@ -47,13 +46,20 @@ func (r *Runtime) NewRoleClient(role, profileID string, settings config.LLM, hoo return &RoleClient{runtime: r, role: role, profileID: profileID, settings: settings, hooks: hooks}, nil } -func (c *RoleClient) CompleteStructured(parent context.Context, req structuredoutput.Request, out any) (response structuredoutput.Response, err error) { - ctx, cancel := context.WithTimeout(parent, c.settings.OperationTimeout) - defer cancel() +type pendingCall struct { + request pk.RunRequest + finish func(*pk.RunResult, error) (structuredoutput.Response, error) +} + +func (c *RoleClient) prepareCall(parent context.Context, req structuredoutput.Request, out any, observer pk.ExecutionObserver) (*pendingCall, error) { + var response structuredoutput.Response + var err error + var finishHooks []func() + ctx := parent var terminalMu sync.Mutex var terminal pk.ExecutionEvent terminalAvailable := false - defer func() { + finishHooks = append(finishHooks, func() { terminalMu.Lock() if terminalAvailable { response.Execution = outcomes.ExecutionCounts{ @@ -69,7 +75,7 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou } } terminalMu.Unlock() - }() + }) var record structuredoutput.CallRecord if c.hooks.OnCall != nil || c.hooks.OnEventWithCall != nil { record.CallID = c.runtime.nextCallID.Add(1) @@ -81,7 +87,7 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou record.Started = time.Now().UTC() record.Prompt = structuredoutput.PromptMetadata{PromptID: req.PromptID, ProfileID: c.profileID} onCall(record) - defer func() { + finishHooks = append(finishHooks, func() { record.Finished = time.Now().UTC() if err != nil { var failure *structuredoutput.Failure @@ -92,123 +98,120 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou } } onCall(record) - }() + }) + } + finish := func() { + for i := len(finishHooks) - 1; i >= 0; i-- { + finishHooks[i]() + } + } + fail := func(failure error) (*pendingCall, error) { + err = failure + finish() + return nil, failure } if parent.Err() != nil { - return response, classifiedFailure(parent, ctx, parent.Err(), nil) + return fail(classifiedFailure(parent, ctx, parent.Err(), nil)) } if err = validateOutputTarget(out); err != nil { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "invalid structured output target", Cause: err} + return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "invalid structured output target", Cause: err}) } contract, contractErr := OutputContract(req.Schema, c.settings.RepairAttempts) if contractErr != nil { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "unknown structured response schema", Cause: contractErr} + return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "unknown structured response schema", Cause: contractErr}) } vars := prompts.WithSharedVariables(req.Variables) for name, value := range prompts.SharedVariables() { vars[name] = value } - baseID, baseVars := req.PromptID, vars - var suffix []pk.RenderedMessage - if req.Replay != nil { - if req.Replay.Exchange == nil { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "repair exchange is unavailable"} + var exchange *pk.Exchange + if req.ContinueFrom != nil { + var ok bool + exchange, ok = req.ContinueFrom.State.(*pk.Exchange) + if !ok || exchange == nil { + return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "repair exchange is unavailable"}) } - instruction, renderErr := c.runtime.Engine.Prepare(ctx, pk.RunRequest{PromptID: req.PromptID, ProfileID: c.profileID, Vars: vars}) - if renderErr != nil { - return response, classifiedFailure(parent, ctx, renderErr, nil) - } - response.RepairInstruction = convertMessages(instruction.Messages) - exchange := req.Replay.Exchange - baseID, baseVars = exchange.PromptID, exchange.Variables - suffix = append(suffix, pk.RenderedMessage{Role: pk.RoleAssistant, Content: exchange.FinalContent}) - for _, message := range req.Replay.Turns { - native := pk.RenderedMessage{Role: message.Role, Content: message.Content} - if cache := message.CacheControl; cache != nil { - native.CacheControl = &pk.CacheControl{Type: pk.CacheControlType(cache.Type), TTL: cache.TTL} - } - suffix = append(suffix, native) - } - suffix = append(suffix, instruction.Messages...) } - handle, prepErr := c.runtime.Engine.PrepareExecution(ctx, pk.RunRequest{ - PromptID: baseID, ProfileID: c.profileID, Vars: baseVars, AppendedMessages: suffix, + runRequest := pk.RunRequest{ + PromptID: req.PromptID, ProfileID: c.profileID, Vars: vars, + CaptureExchange: req.CaptureExchange, ContinueFrom: exchange, TransientRetries: &c.settings.TransientRetries, Validation: contract, - Observer: func(event pk.ExecutionEvent) { + Observer: pk.ComposeObservers(observer, func(event pk.ExecutionEvent) { if event.Kind == pk.ExecutionFinished { terminalMu.Lock() terminal = event terminalAvailable = true terminalMu.Unlock() } - if c.hooks.OnEvent != nil { - c.hooks.OnEvent(event) - } if c.hooks.OnEventWithCall != nil { c.hooks.OnEventWithCall(record.CallID, record.Context, event) } - }, - }) - if prepErr != nil { - return response, classifiedFailure(parent, ctx, prepErr, nil) + }), } - defer handle.Discard() - details := handle.Details() - if req.Replay != nil { - prefix := req.Replay.Exchange.Messages - if len(details.Messages) < len(prefix) || !reflect.DeepEqual(convertMessages(details.Messages[:len(prefix)]), prefix) { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "repair conversation prefix changed"} + var details pk.PreparedRun + capturePrepared := func(_ context.Context, prepared pk.PreparedRun) { + details = prepared + if name := details.EffectiveModelParams.APIKeyEnv; name != "" { + c.credentialMu.Lock() + if c.credentialNames == nil { + c.credentialNames = make(map[string]struct{}) + } + c.credentialNames[name] = struct{}{} + c.credentialMu.Unlock() } + response.Prompt = promptMetadata(details) + record.Prompt = response.Prompt + response.Model = details.EffectiveModelParams.Model + response.Prepared = preparedSnapshot(details) + record.Prepared = response.Prepared } - if name := details.EffectiveModelParams.APIKeyEnv; name != "" { - c.credentialMu.Lock() - if c.credentialNames == nil { - c.credentialNames = make(map[string]struct{}) + runRequest.OperationTimeout = c.settings.OperationTimeout + runRequest.OnPrepared = capturePrepared + return &pendingCall{request: runRequest, finish: func(result *pk.RunResult, runErr error) (completed structuredoutput.Response, finalErr error) { + defer func() { err = finalErr; finish(); completed = response }() + + if result != nil { + response.FinalContent = result.RawOutput + record.FinalContent = result.RawOutput + response.Model = result.ModelName + response.PromptTokens = result.Usage.PromptTokens + response.CompletionTokens = result.Usage.CompletionTokens + response.TotalTokens = result.Usage.TotalTokens } - c.credentialNames[name] = struct{}{} - c.credentialMu.Unlock() - } - response.Prompt = promptMetadata(details) - record.Prompt = response.Prompt - response.Model = details.EffectiveModelParams.Model - response.Prepared = preparedSnapshot(details) - record.Prepared = response.Prepared - result, runErr := c.runtime.Engine.RunPrepared(ctx, handle) - if result != nil { - response.FinalContent = result.RawOutput - record.FinalContent = result.RawOutput - response.Model = result.ModelName - response.PromptTokens = result.Usage.PromptTokens - response.CompletionTokens = result.Usage.CompletionTokens - response.TotalTokens = result.Usage.TotalTokens - } - if runErr != nil { - return response, classifiedFailure(parent, ctx, runErr, result) - } - if result == nil { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "structured model call returned no result"} - } - if !result.Validation.IsValid { - return response, classifiedFailure(parent, ctx, nil, result) - } - if decodeErr := json.Unmarshal([]byte(result.RawOutput), out); decodeErr != nil { - var typeError *json.UnmarshalTypeError - if errors.As(decodeErr, &typeError) { - return response, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonInvalidOutput, Message: "structured output contains an unusable value", Cause: decodeErr} + if runErr != nil { + return response, classifiedFailure(parent, ctx, runErr, result) } - return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "validated structured output could not be decoded", Cause: decodeErr} - } - if req.CaptureExchange && req.Replay == nil { - frozen := make(map[string]string, len(vars)) - for key, value := range vars { - frozen[key] = value + if result == nil { + return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "structured model call returned no result"} } - response.Exchange = &structuredoutput.Exchange{PromptID: req.PromptID, Variables: frozen, Messages: convertMessages(details.Messages), FinalContent: result.RawOutput} + if !result.Validation.IsValid { + return response, classifiedFailure(parent, ctx, nil, result) + } + if decodeErr := json.Unmarshal([]byte(result.RawOutput), out); decodeErr != nil { + var typeError *json.UnmarshalTypeError + if errors.As(decodeErr, &typeError) { + return response, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonInvalidOutput, Message: "structured output contains an unusable value", Cause: decodeErr} + } + return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "validated structured output could not be decoded", Cause: decodeErr} + } + if result.Exchange != nil { + response.Exchange = &structuredoutput.Exchange{State: result.Exchange} + } + + response.Content = json.RawMessage(result.RawOutput) + response.Provider = "openai-compatible" + return response, nil + }}, nil +} + +func (c *RoleClient) CompleteStructured(ctx context.Context, req structuredoutput.Request, out any) (structuredoutput.Response, error) { + call, err := c.prepareCall(ctx, req, out, c.hooks.OnEvent) + if err != nil { + return structuredoutput.Response{}, err } - response.Content = json.RawMessage(result.RawOutput) - response.Provider = "openai-compatible" - return response, nil + result, err := c.runtime.Engine.Run(ctx, call.request) + return call.finish(result, err) } // OutputContract maps an Audita domain schema to the native request contract. @@ -231,6 +234,12 @@ func classifiedFailure(parent, local context.Context, cause error, result *pk.Ru failure.Cause = parent.Err() return failure } + if errors.Is(cause, pk.ErrOperationTimeout) { + failure.Class = structuredoutput.FailureRecoverable + failure.Reason = outcomes.ReasonOperationTimeout + failure.Message = "model operation timed out" + return failure + } if isSetupFailure(cause) { failure.Reason = outcomes.ReasonSetupFailure failure.Message = "structured model setup failed" @@ -298,10 +307,7 @@ func promptMetadata(details pk.PreparedRun) structuredoutput.PromptMetadata { } func preparedSnapshot(details pk.PreparedRun) *structuredoutput.PreparedSnapshot { - messages := make([]structuredoutput.Message, len(details.Messages)) - for i, message := range details.Messages { - messages[i] = structuredoutput.Message{Role: string(message.Role), Content: message.Content} - } + messages := convertMessages(details.Messages) return &structuredoutput.PreparedSnapshot{Messages: messages, Prompt: promptMetadata(details)} } diff --git a/internal/framework/llm/adapter_test.go b/internal/framework/llm/adapter_test.go index 2c8065a..0d6e8f8 100644 --- a/internal/framework/llm/adapter_test.go +++ b/internal/framework/llm/adapter_test.go @@ -482,3 +482,14 @@ func TestPreparedAdapterUnknownValidationFailureRemainsFatal(t *testing.T) { t.Fatalf("unknown validation failure: %#v", failure) } } + +// Preparation errors can carry the native operation deadline as their cause. +// That deadline is recoverable even when the outer category is a setup phase. +func TestOperationTimeoutDuringPreparationIsRecoverable(t *testing.T) { + ctx := context.Background() + cause := errors.Join(pk.ErrPromptLoad, pk.ErrOperationTimeout, context.DeadlineExceeded) + failure := classifiedFailure(ctx, ctx, cause, nil) + if failure.Class != structuredoutput.FailureRecoverable || failure.Reason != outcomes.ReasonOperationTimeout || !errors.Is(failure, pk.ErrOperationTimeout) { + t.Fatalf("failure = %#v", failure) + } +} diff --git a/internal/framework/llm/conversation_test.go b/internal/framework/llm/conversation_test.go index cce449b..f399cff 100644 --- a/internal/framework/llm/conversation_test.go +++ b/internal/framework/llm/conversation_test.go @@ -19,141 +19,9 @@ import ( "gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput" "gitea.maximumdirect.net/eric/audita/v3/internal/prompts" ck "go.maximumdirect.net/configkit" - pk "go.maximumdirect.net/promptkit" ) -func TestNativeConversationReplayWithRepairContract(t *testing.T) { - dir := t.TempDir() - files := map[string]string{ - "config.yml": "version: 3\n", - "prompts.yml": `profiles: - - id: proposal-profile - endpoint: https://example.invalid/v1 - model: proposal-model - - id: repair-profile - endpoint: https://example.invalid/v1 - model: repair-model -prompts: - - id: proposal-fixture - messages: - - role: system - content: 'System {{.Subject}}' - - role: user - content: 'Proposal {{.Subject}}' - output: - format: json - validation_mode: json_schema - schema_path: correction_set.v1.json - - id: repair-fixture - messages: - - role: user - content: 'Repair {{.Finding}}' - output: - format: text - validation_mode: none -`, - } - for name, content := range files { - if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0644); err != nil { - t.Fatal(err) - } - } - loaded, err := config.Load(ck.Input{Source: ck.File(filepath.Join(dir, "config.yml"))}, "") - if err != nil { - t.Fatal(err) - } - var requests []struct { - Model string `json:"model"` - Messages []struct{ Role, Content string } `json:"messages"` - ResponseFormat map[string]any `json:"response_format"` - } - outputs := []string{`{"corrections":[]}`, `{"proposal_ref":1,"action":"revise","correction":{"id":1,"original_text":" ","corrected_text":""}}`, `{"proposal_ref":1,"action":"withdraw","correction":null}`} - transport := transportFunc(func(r *http.Request) (*http.Response, error) { - var request struct { - Model string `json:"model"` - Messages []struct{ Role, Content string } `json:"messages"` - ResponseFormat map[string]any `json:"response_format"` - } - if err := json.NewDecoder(r.Body).Decode(&request); err != nil { - t.Fatal(err) - } - requests = append(requests, request) - content, _ := json.Marshal(outputs[len(requests)-1]) - body := fmt.Sprintf(`{"model":%q,"choices":[{"message":{"content":%s}}]}`, request.Model, content) - return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body)), Request: r}, nil - }) - runtime, err := NewRuntime(context.Background(), loaded, &http.Client{Transport: transport}) - if err != nil { - t.Fatal(err) - } - ctx := context.Background() - base := pk.RunRequest{PromptID: "proposal-fixture", ProfileID: "proposal-profile", Vars: map[string]string{"Subject": "source"}} - original, err := runtime.Engine.PrepareExecution(ctx, base) - if err != nil { - t.Fatal(err) - } - prefix := original.Details().Messages - initial, err := runtime.Engine.RunPrepared(ctx, original) - if err != nil { - t.Fatal(err) - } - suffix := []pk.RenderedMessage{{Role: pk.RoleAssistant, Content: initial.RawOutput}} - contract, err := OutputContract(responseschema.CorrectionRepairKey, 0) - if err != nil { - t.Fatal(err) - } - for _, finding := range []string{"first", "second"} { - instruction, err := runtime.Engine.Prepare(ctx, pk.RunRequest{PromptID: "repair-fixture", ProfileID: "repair-profile", Vars: map[string]string{"Finding": finding}}) - if err != nil { - t.Fatal(err) - } - suffix = append(suffix, instruction.Messages...) - req := base - req.ProfileID = "repair-profile" - req.AppendedMessages = suffix - req.Validation = contract - handle, err := runtime.Engine.PrepareExecution(ctx, req) - if err != nil { - t.Fatal(err) - } - details := handle.Details() - if !reflect.DeepEqual(details.Messages[:len(prefix)], prefix) || !reflect.DeepEqual(details.Messages[len(prefix):], suffix) { - t.Fatalf("prepared conversation = %#v", details.Messages) - } - result, err := runtime.Engine.RunPrepared(ctx, handle) - if err != nil { - t.Fatal(err) - } - if !result.Validation.IsValid { - t.Fatalf("invalid result: %+v", result.Validation) - } - suffix = append(suffix, pk.RenderedMessage{Role: pk.RoleAssistant, Content: result.RawOutput}) - } - if len(requests) != 3 || requests[0].Model != "proposal-model" || requests[1].Model != "repair-model" || requests[2].Model != "repair-model" { - t.Fatalf("models: %+v", requests) - } - for i, wantLen := range []int{2, 4, 6} { - req := requests[i] - if len(req.Messages) != wantLen { - t.Fatalf("request %d messages: %+v", i, req.Messages) - } - for j, message := range req.Messages { - if message.Role != string(append(prefix, suffix...)[j].Role) || message.Content != append(prefix, suffix...)[j].Content { - t.Fatalf("request %d message %d: %+v", i, j, message) - } - } - encoded, _ := json.Marshal(req.ResponseFormat) - expected := "corrections" - if i > 0 { - expected = "proposal_ref" - } - if !strings.Contains(string(encoded), expected) { - t.Fatalf("request %d contract: %s", i, encoded) - } - } -} - -func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) { +func TestRoleClientContinuesIndependentRepairConversations(t *testing.T) { dir := t.TempDir() for name, content := range map[string]string{ "config.yml": "version: 3\n", @@ -207,17 +75,13 @@ func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) { t.Fatalf("capture: %+v %v", initial, err) } vars["SectionJSON"] = "changed" - replay := &structuredoutput.Replay{Exchange: initial.Exchange} + exchange := initial.Exchange for _, finding := range []string{"first", "second"} { - result, err := repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": finding}, Replay: replay}, &map[string]any{}) + result, err := repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": finding}, ContinueFrom: exchange}, &map[string]any{}) if err != nil { t.Fatal(err) } - if len(result.RepairInstruction) != 1 { - t.Fatalf("instruction: %+v", result) - } - replay.Turns = append(replay.Turns, result.RepairInstruction...) - replay.Turns = append(replay.Turns, structuredoutput.Message{Role: "assistant", Content: result.FinalContent}) + exchange = result.Exchange } if !reflect.DeepEqual(models, []string{"proposal", "repair", "repair"}) { t.Fatalf("models = %v", models) @@ -235,21 +99,18 @@ func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) { !strings.Contains(string(first.Content), `"text":"first"`) || !reflect.DeepEqual(first, messages[2][len(prefix)+1]) { t.Fatalf("cached instruction changed during replay: first=%s replay=%s", first.Content, messages[2][len(prefix)+1].Content) } - _, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "independent"}, Replay: &structuredoutput.Replay{Exchange: initial.Exchange}}, &map[string]any{}) + _, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "independent"}, ContinueFrom: initial.Exchange}, &map[string]any{}) if err != nil || len(messages) != 4 || len(messages[3]) != len(prefix)+2 || !strings.Contains(string(messages[3][len(prefix)+1].Content), "independent") { t.Fatalf("independent branch: %v %+v", err, messages) } - bad := *initial.Exchange - bad.Messages = append([]structuredoutput.Message(nil), bad.Messages...) - bad.Messages[0].Content = "changed" - _, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "failure"}, Replay: &structuredoutput.Replay{Exchange: &bad}}, &map[string]any{}) + _, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "failure"}, ContinueFrom: &structuredoutput.Exchange{}}, &map[string]any{}) var failure *structuredoutput.Failure if !errors.As(err, &failure) || failure.Class != structuredoutput.FailureFatal || len(messages) != 4 { t.Fatalf("prefix failure: %v, calls %d", err, len(messages)) } } -func TestRepairReplayUsesReplacementAndCountsStructuralRepair(t *testing.T) { +func TestRepairContinuationUsesReplacementAndCountsStructuralRepair(t *testing.T) { dir := t.TempDir() files := map[string]string{ "config.yml": "version: 3\n", @@ -291,11 +152,11 @@ func TestRepairReplayUsesReplacementAndCountsStructuralRepair(t *testing.T) { if err != nil { t.Fatal(err) } - result, err := client.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "fixture"}, Replay: &structuredoutput.Replay{Exchange: base.Exchange}}, &map[string]any{}) + result, err := client.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "fixture"}, ContinueFrom: base.Exchange}, &map[string]any{}) if err != nil { t.Fatal(err) } - if len(requests) != 3 || result.Execution.ProviderAttempts != 2 || result.Execution.StructuralRepairs != 1 || result.FinalContent != outputs[2] || len(result.RepairInstruction) != 1 || result.RepairInstruction[0].Content != "Replace fixture" { + if len(requests) != 3 || result.Execution.ProviderAttempts != 2 || result.Execution.StructuralRepairs != 1 || result.FinalContent != outputs[2] { t.Fatalf("result=%+v calls=%d", result, len(requests)) } } diff --git a/internal/framework/llm/coordinate.go b/internal/framework/llm/coordinate.go new file mode 100644 index 0000000..ade437b --- /dev/null +++ b/internal/framework/llm/coordinate.go @@ -0,0 +1,63 @@ +package llm + +import ( + "context" + "encoding/json" + + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput" + pk "go.maximumdirect.net/promptkit" +) + +var _ structuredoutput.Coordinator = (*RoleClient)(nil) + +// Coordinate shares request construction and finalization with ordinary calls. +// PromptKit alone advances history and consumes the additional-turn budget. +func (c *RoleClient) Coordinate(ctx context.Context, start *structuredoutput.Exchange, budget int, next func(context.Context, structuredoutput.Turn) (*structuredoutput.Request, error)) (bool, error) { + var exchange *pk.Exchange + if start != nil { + exchange, _ = start.State.(*pk.Exchange) + } + var pending *pendingCall + result, err := c.runtime.Engine.Coordinate(ctx, pk.CoordinateRequest{ + Start: exchange, AdditionalTurns: budget, Observer: c.hooks.OnEvent, + Next: func(ctx context.Context, turn pk.ConversationTurn) (*pk.RunRequest, error) { + var delivered structuredoutput.Turn + if turn.Result != nil { + response, failure := pending.finish(turn.Result, nil) + pending = nil + delivered = structuredoutput.Turn{Response: &response, Err: failure} + } + selected, err := next(ctx, delivered) + if err != nil || selected == nil { + return nil, err + } + // Exhaustion needs a requested turn, but must not allocate a logical call. + if turn.RemainingTurns == 0 { + return &pk.RunRequest{}, nil + } + var output json.RawMessage + pending, err = c.prepareCall(ctx, *selected, &output, nil) + if err != nil { + _, domainErr := next(ctx, structuredoutput.Turn{Response: &structuredoutput.Response{}, Err: err}) + if domainErr != nil { + return nil, domainErr + } + return nil, err + } + return &pending.request, nil + }, + }) + if pending != nil { + // Execution errors and withheld responses do not enter Next. Finalize their + // diagnostic evidence and let the domain record the terminal failure once. + response, failure := pending.finish(result.Undelivered, err) + _, domainErr := next(ctx, structuredoutput.Turn{Response: &response, Err: failure}) + if domainErr != nil { + return false, domainErr + } + if structuredoutput.IsRecoverable(failure) { + return false, nil + } + } + return result.Outcome == pk.CoordinationExhausted, err +} diff --git a/internal/framework/llm/coordinate_test.go b/internal/framework/llm/coordinate_test.go new file mode 100644 index 0000000..51a5203 --- /dev/null +++ b/internal/framework/llm/coordinate_test.go @@ -0,0 +1,141 @@ +package llm + +import ( + "context" + "errors" + "testing" + + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/outcomes" + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/responseschema" + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput" + "gitea.maximumdirect.net/eric/audita/v3/internal/prompts" + pk "go.maximumdirect.net/promptkit" +) + +func TestCoordinatorFinalTurnFailuresAndAccounting(t *testing.T) { + for _, tc := range []struct { + name string + reply reply + again bool + reason outcomes.ReasonCode + }{ + {name: "last turn assessed", reply: reply{200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}}, + {name: "exhaustion", reply: reply{200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}, again: true}, + {name: "invalid structure", reply: reply{200, `{"wrong":true}`}, reason: outcomes.ReasonInvalidOutputExhausted}, + {name: "provider error", reply: reply{400, ""}, reason: outcomes.ReasonProviderFailed}, + } { + t.Run(tc.name, func(t *testing.T) { + runtime, count, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, tc.reply}, 0, 0, nil) + collector := NewObservationCollector() + client, err := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, collector.Hooks()) + if err != nil { + t.Fatal(err) + } + req := adapterRequest(responseschema.CorrectionSetKey) + req.CaptureExchange = true + base, err := client.CompleteStructured(context.Background(), req, &map[string]any{}) + if err != nil { + t.Fatal(err) + } + deliveries := 0 + exhausted, err := client.Coordinate(context.Background(), base.Exchange, 1, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) { + if turn.Response != nil { + deliveries++ + if tc.reason != "" { + var failure *structuredoutput.Failure + if !errors.As(turn.Err, &failure) || failure.Reason != tc.reason { + t.Fatalf("failure=%v", turn.Err) + } + return nil, nil + } + if turn.Err != nil || len(turn.Response.Content) == 0 { + t.Fatalf("turn=%+v", turn) + } + if !tc.again { + return nil, nil + } + } + return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil + }) + if err != nil || exhausted != tc.again || deliveries != 1 || count.Load() != 2 { + t.Fatalf("exhausted=%v delivered=%d calls=%d err=%v", exhausted, deliveries, count.Load(), err) + } + snapshot := collector.Snapshot() + summary := Summarize(snapshot) + if len(snapshot.Calls) != 2 || summary.ProviderAttempts != 2 || !summary.TerminalCoverageComplete { + t.Fatalf("summary=%+v calls=%+v", summary, snapshot.Calls) + } + for _, call := range snapshot.Calls { + if call.Finished.IsZero() { + t.Fatal("unfinished logical call") + } + } + }) + } +} + +func TestCoordinatorCanceledDeliveryRetainsEvidence(t *testing.T) { + runtime, _, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, {200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}}, 0, 0, nil) + collector := NewObservationCollector() + proposal, _ := runtime.NewRoleClient("proposal", "base", runtime.Config.LLM, collector.Hooks()) + req := adapterRequest(responseschema.CorrectionSetKey) + req.CaptureExchange = true + base, err := proposal.CompleteStructured(context.Background(), req, &map[string]any{}) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + hooks := collector.Hooks() + hooks.OnEvent = func(e pk.ExecutionEvent) { + if e.Kind == pk.ExecutionFinished { + cancel() + } + } + client, _ := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, hooks) + deliveries := 0 + _, err = client.Coordinate(ctx, base.Exchange, 1, func(_ context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) { + if turn.Response != nil { + deliveries++ + if !errors.Is(turn.Err, context.Canceled) || turn.Response.Prepared == nil || turn.Response.FinalContent == "" || len(turn.Response.Content) != 0 { + t.Fatalf("canceled turn=%+v", turn) + } + return nil, turn.Err + } + return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil + }) + if !errors.Is(err, context.Canceled) || deliveries != 1 { + t.Fatalf("deliveries=%d err=%v", deliveries, err) + } + if got := Summarize(collector.Snapshot()); got.ProviderAttempts != 2 || !got.TerminalCoverageComplete { + t.Fatalf("summary=%+v", got) + } +} + +func TestCoordinatorReleasesAdmissionBeforeNestedValidation(t *testing.T) { + runtime, count, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, {200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}, {200, `{"validations":[]}`}}, 0, 0, nil) + collector := NewObservationCollector() + proposal, _ := runtime.NewRoleClient("proposal", "base", runtime.Config.LLM, collector.Hooks()) + repair, _ := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, collector.Hooks()) + validation, _ := runtime.NewRoleClient("validation", "base", runtime.Config.LLM, collector.Hooks()) + req := adapterRequest(responseschema.CorrectionSetKey) + req.CaptureExchange = true + base, err := proposal.CompleteStructured(context.Background(), req, &map[string]any{}) + if err != nil { + t.Fatal(err) + } + _, err = repair.Coordinate(context.Background(), base.Exchange, 1, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) { + if turn.Response != nil { + if turn.Err != nil { + return nil, turn.Err + } + _, err := validation.CompleteStructured(ctx, adapterRequest(responseschema.ValidatorDecisionSetKey), &map[string]any{}) + return nil, err + } + return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil + }) + summary := Summarize(collector.Snapshot()) + if err != nil || count.Load() != 3 || summary.ProviderAttempts != 3 || summary.LogicalCalls != 3 || !summary.TerminalCoverageComplete { + t.Fatalf("calls=%d summary=%+v err=%v", count.Load(), summary, err) + } +} diff --git a/internal/framework/llm/native_adapter_test.go b/internal/framework/llm/native_adapter_test.go index 506b834..c7205f7 100644 --- a/internal/framework/llm/native_adapter_test.go +++ b/internal/framework/llm/native_adapter_test.go @@ -115,7 +115,7 @@ func TestRoleClientExecutionBudgetsAndFailureClasses(t *testing.T) { capturing := request capturing.CaptureExchange = true response, err := client.CompleteStructured(context.Background(), capturing, &result) - if tc.success && (response.Exchange == nil || response.Exchange.FinalContent != tc.outputs[len(tc.outputs)-1]) { + if tc.success && (response.Exchange == nil || response.FinalContent != tc.outputs[len(tc.outputs)-1]) { t.Fatalf("captured final response: %+v", response.Exchange) } if (err == nil) != tc.success || calls != tc.wantCalls { diff --git a/internal/framework/llm/observations.go b/internal/framework/llm/observations.go index d0f5fad..1a88f4c 100644 --- a/internal/framework/llm/observations.go +++ b/internal/framework/llm/observations.go @@ -64,21 +64,24 @@ type LogicalCall struct { } type ObservationSnapshot struct { - Calls []LogicalCall `json:"calls"` - Events []ExecutionObservation `json:"events"` + Group *pk.ExecutionGroupSummary `json:"-"` + Calls []LogicalCall `json:"calls"` + Events []ExecutionObservation `json:"events"` } type ExecutionSummary struct { - LogicalCalls uint64 `json:"logical_calls"` - ProviderAttempts uint64 `json:"provider_attempts"` - TransientRetries uint64 `json:"transient_retries"` - StructuralRepairs uint64 `json:"structural_repairs"` - Usage Usage `json:"usage"` + TerminalCoverageComplete bool `json:"terminal_coverage_complete"` + LogicalCalls uint64 `json:"logical_calls"` + ProviderAttempts uint64 `json:"provider_attempts"` + TransientRetries uint64 `json:"transient_retries"` + StructuralRepairs uint64 `json:"structural_repairs"` + Usage Usage `json:"usage"` } // ObservationCollector accepts short synchronous callbacks from parallel calls. // Snapshot should be taken after the runner has joined active section work. type ObservationCollector struct { + group pk.ExecutionGroup mu sync.Mutex calls map[uint64]LogicalCall events []ExecutionObservation @@ -137,6 +140,7 @@ func (c *ObservationCollector) recordEvent(callID uint64, context structuredoutp if c == nil { return } + c.group.Observe(event) count := func(value pk.ObservedTokenCount) TokenCount { return TokenCount{Value: value.Value, Known: value.Known, Complete: value.Complete} } @@ -176,52 +180,28 @@ func (c *ObservationCollector) Snapshot() ObservationSnapshot { } return snapshot.Events[i].Time.Before(snapshot.Events[j].Time) }) + group := c.group.Snapshot() + snapshot.Group = &group return snapshot } -// Summarize counts terminal usage once per logical call. PromptKit's terminal -// usage already aggregates its attempts, retries, and repairs. +// Summarize delegates deduplication and terminal accounting to PromptKit. A +// filtered diagnostic view selects its own group; overlapping views are not additive. func Summarize(snapshot ObservationSnapshot) ExecutionSummary { - result := ExecutionSummary{LogicalCalls: uint64(len(snapshot.Calls))} - terminal := make(map[uint64]ExecutionObservation) - for _, event := range snapshot.Events { - if event.Kind == string(pk.ExecutionFinished) { - terminal[event.CallID] = event + summary := snapshot.Group + if summary == nil { + var group pk.ExecutionGroup + count := func(c TokenCount) pk.ObservedTokenCount { + return pk.ObservedTokenCount{Value: c.Value, Known: c.Known, Complete: c.Complete} } - } - dispatched := false - aggregate := func(dst *TokenCount, source TokenCount) { - if source.Known { - dst.Value += source.Value - dst.Known = true + for _, event := range snapshot.Events { + group.Observe(pk.ExecutionEvent{Kind: pk.ExecutionEventKind(event.Kind), RunID: event.RunID, Attempts: event.Attempts, TransientRetries: event.TransientRetries, RepairCalls: event.StructuralRepairs, Usage: pk.ObservedTokenUsage{PromptTokens: count(event.Usage.Prompt), CompletionTokens: count(event.Usage.Completion), TotalTokens: count(event.Usage.Total), CachedTokens: count(event.Usage.Cached), CacheWriteTokens: count(event.Usage.CacheWrite)}}) } - dst.Complete = dst.Complete && source.Complete + value := group.Snapshot() + summary = &value } - result.Usage = Usage{Prompt: TokenCount{Complete: true}, Completion: TokenCount{Complete: true}, Total: TokenCount{Complete: true}, Cached: TokenCount{Complete: true}, CacheWrite: TokenCount{Complete: true}} - for _, call := range snapshot.Calls { - event, ok := terminal[call.CallID] - if !ok { - continue - } - result.ProviderAttempts += event.Attempts - result.TransientRetries += event.TransientRetries - result.StructuralRepairs += event.StructuralRepairs - if event.Attempts == 0 { - continue - } - dispatched = true - aggregate(&result.Usage.Prompt, event.Usage.Prompt) - aggregate(&result.Usage.Completion, event.Usage.Completion) - aggregate(&result.Usage.Total, event.Usage.Total) - aggregate(&result.Usage.Cached, event.Usage.Cached) - aggregate(&result.Usage.CacheWrite, event.Usage.CacheWrite) + count := func(c pk.ObservedTokenCount) TokenCount { + return TokenCount{Value: c.Value, Known: c.Known, Complete: c.Complete} } - if !dispatched { - result.Usage.Prompt.Known = true - result.Usage.Completion.Known = true - result.Usage.Total.Known = true - result.Usage.Cached.Known = true - result.Usage.CacheWrite.Known = true - } - return result + return ExecutionSummary{LogicalCalls: uint64(len(snapshot.Calls)), TerminalCoverageComplete: summary.TerminalCoverageComplete, ProviderAttempts: summary.Attempts, TransientRetries: summary.TransientRetries, StructuralRepairs: summary.RepairCalls, Usage: Usage{Prompt: count(summary.Usage.PromptTokens), Completion: count(summary.Usage.CompletionTokens), Total: count(summary.Usage.TotalTokens), Cached: count(summary.Usage.CachedTokens), CacheWrite: count(summary.Usage.CacheWriteTokens)}} } diff --git a/internal/framework/llm/observations_test.go b/internal/framework/llm/observations_test.go index 8668703..285b95a 100644 --- a/internal/framework/llm/observations_test.go +++ b/internal/framework/llm/observations_test.go @@ -3,6 +3,7 @@ package llm import ( "context" "encoding/json" + "fmt" "strings" "sync" "testing" @@ -23,8 +24,8 @@ func TestObservationSummarySeparatesLogicalCallsAttemptsAndUsage(t *testing.T) { } call(1) // preparation failed before PromptKit emitted events call(2) // capacity rejected without a dispatched attempt - hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{Kind: pk.ExecutionAttempt, Dispatched: false}) - hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 0}) + hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{RunID: "run-2", Kind: pk.ExecutionAttempt, Dispatched: false}) + hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{RunID: "run-2", Kind: pk.ExecutionFinished, Attempts: 0, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CompletionTokens: pk.ObservedTokenCount{Known: true, Complete: true}, TotalTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CachedTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CacheWriteTokens: pk.ObservedTokenCount{Known: true, Complete: true}}}) summary := Summarize(collector.Snapshot()) if summary.LogicalCalls != 2 || summary.ProviderAttempts != 0 || summary.Usage.Total != (TokenCount{Known: true, Complete: true}) { t.Fatalf("zero-attempt accounting: %+v", summary) @@ -32,16 +33,16 @@ func TestObservationSummarySeparatesLogicalCallsAttemptsAndUsage(t *testing.T) { call(3) context := structuredoutput.DiagnosticContext{Role: "validation", ModuleInstance: "grammar", ValidatorName: "editorial_review"} - hooks.OnEventWithCall(3, context, pk.ExecutionEvent{Kind: pk.ExecutionAttempt, Dispatched: true, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 100, Known: true, Complete: true}}}) - hooks.OnEventWithCall(3, context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 3, TransientRetries: 1, RepairCalls: 1, Usage: pk.ObservedTokenUsage{ + hooks.OnEventWithCall(3, context, pk.ExecutionEvent{RunID: "run-3", Kind: pk.ExecutionAttempt, Dispatched: true, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 100, Known: true, Complete: true}}}) + hooks.OnEventWithCall(3, context, pk.ExecutionEvent{RunID: "run-3", Kind: pk.ExecutionFinished, Attempts: 3, TransientRetries: 1, RepairCalls: 1, Usage: pk.ObservedTokenUsage{ PromptTokens: pk.ObservedTokenCount{Value: 5, Known: true, Complete: false}, TotalTokens: pk.ObservedTokenCount{Value: 8, Known: true, Complete: false}, }}) call(4) - hooks.OnEventWithCall(4, context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: false, Complete: false}, TotalTokens: pk.ObservedTokenCount{Known: false, Complete: false}}}) + hooks.OnEventWithCall(4, context, pk.ExecutionEvent{RunID: "run-4", Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: false, Complete: false}, TotalTokens: pk.ObservedTokenCount{Known: false, Complete: false}}}) snapshot := collector.Snapshot() summary = Summarize(snapshot) - if summary.LogicalCalls != 4 || summary.ProviderAttempts != 4 || summary.TransientRetries != 1 || summary.StructuralRepairs != 1 || summary.Usage.Total.Value != 8 || !summary.Usage.Total.Known || summary.Usage.Total.Complete || summary.Usage.Prompt.Value != 5 || !summary.Usage.Prompt.Known || summary.Usage.Prompt.Complete || summary.Usage.Cached.Known || summary.Usage.Cached.Complete { + if summary.LogicalCalls != 4 || summary.ProviderAttempts != 4 || summary.TransientRetries != 1 || summary.StructuralRepairs != 1 || summary.Usage.Total.Value != 8 || !summary.Usage.Total.Known || summary.Usage.Total.Complete || summary.Usage.Prompt.Value != 5 || !summary.Usage.Prompt.Known || summary.Usage.Prompt.Complete || !summary.Usage.Cached.Known || summary.Usage.Cached.Complete { t.Fatalf("partial usage or double count: %+v", summary) } raw, err := json.Marshal(snapshot) @@ -68,7 +69,7 @@ func TestObservationCollectorConcurrentAttributionAndSnapshotCopy(t *testing.T) context := structuredoutput.DiagnosticContext{Role: "proposal", ModuleInstance: "grammar", SectionIndex: &index} started := time.Now().UTC() hooks.OnCall(structuredoutput.CallRecord{CallID: uint64(id), Context: context, Started: started}) - hooks.OnEventWithCall(uint64(id), context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, RunID: "run", Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 1, Known: true, Complete: true}}}) + hooks.OnEventWithCall(uint64(id), context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, RunID: fmt.Sprint(id), Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 1, Known: true, Complete: true}}}) hooks.OnCall(structuredoutput.CallRecord{CallID: uint64(id), Context: context, Started: started, Finished: time.Now().UTC()}) index = -1 }(i) @@ -124,3 +125,18 @@ func TestRoleClientFeedsCollectorAcrossPreparationFailure(t *testing.T) { t.Fatalf("logical failures: %+v", snapshot.Calls) } } + +func TestObservationCoverageIsIndependentOfUsage(t *testing.T) { + collector := NewObservationCollector() + hooks := collector.Hooks() + hooks.OnEventWithCall(1, structuredoutput.DiagnosticContext{}, pk.ExecutionEvent{RunID: "pending", Kind: pk.ExecutionAttempt}) + got := Summarize(collector.Snapshot()) + if got.TerminalCoverageComplete || got.Usage.Total.Known || got.Usage.Total.Complete { + t.Fatalf("incomplete group=%+v", got) + } + hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{}, pk.ExecutionEvent{RunID: "finished", Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 7, Known: true, Complete: true}}}) + got = Summarize(collector.Snapshot()) + if got.TerminalCoverageComplete || !got.Usage.Total.Known || !got.Usage.Total.Complete || got.Usage.Total.Value != 7 { + t.Fatalf("partial coverage=%+v", got) + } +} diff --git a/internal/framework/processreport/llm.go b/internal/framework/processreport/llm.go index 7f9c2f6..350db1f 100644 --- a/internal/framework/processreport/llm.go +++ b/internal/framework/processreport/llm.go @@ -20,15 +20,16 @@ func mapLLMSummary(source llm.ExecutionSummary) reporting.LLMSummary { } type executionCall struct { - RequestedPromptID string `json:"requested_prompt_id,omitempty"` - CallID uint64 `json:"call_id"` - Context structuredoutput.DiagnosticContext `json:"context"` - Prompt structuredoutput.PromptMetadata `json:"prompt"` - Started time.Time `json:"started"` - Finished *time.Time `json:"finished,omitempty"` - ElapsedMS int64 `json:"elapsed_ms"` - Failure structuredoutput.FailureKind `json:"failure,omitempty"` - Summary reporting.LLMSummary `json:"summary"` + TerminalCoverageComplete bool `json:"terminal_coverage_complete"` + RequestedPromptID string `json:"requested_prompt_id,omitempty"` + CallID uint64 `json:"call_id"` + Context structuredoutput.DiagnosticContext `json:"context"` + Prompt structuredoutput.PromptMetadata `json:"prompt"` + Started time.Time `json:"started"` + Finished *time.Time `json:"finished,omitempty"` + ElapsedMS int64 `json:"elapsed_ms"` + Failure structuredoutput.FailureKind `json:"failure,omitempty"` + Summary reporting.LLMSummary `json:"summary"` } type executionEvent struct { @@ -72,7 +73,7 @@ func BuildLLMExecution(snapshot llm.ObservationSnapshot) any { value := call.Finished finished = &value } - calls = append(calls, executionCall{RequestedPromptID: call.RequestedPromptID, CallID: call.CallID, Context: call.Context, Prompt: call.Prompt, Started: call.Started, Finished: finished, ElapsedMS: call.WallDuration.Milliseconds(), Failure: call.Failure, Summary: mapLLMSummary(llm.Summarize(group))}) + calls = append(calls, executionCall{TerminalCoverageComplete: llm.Summarize(group).TerminalCoverageComplete, RequestedPromptID: call.RequestedPromptID, CallID: call.CallID, Context: call.Context, Prompt: call.Prompt, Started: call.Started, Finished: finished, ElapsedMS: call.WallDuration.Milliseconds(), Failure: call.Failure, Summary: mapLLMSummary(llm.Summarize(group))}) } for _, event := range snapshot.Events { var generation, attempt *uint64 @@ -87,7 +88,8 @@ func BuildLLMExecution(snapshot llm.ObservationSnapshot) any { events = append(events, executionEvent{CallID: event.CallID, Context: event.Context, Kind: event.Kind, RunID: event.RunID, Time: event.Time, BackendID: event.BackendID, ProfileID: event.ProfileID, Model: event.Model, Generation: generation, Attempt: attempt, Dispatched: event.Dispatched, CapacityWaitMS: event.CapacityWait.Milliseconds(), GenerationMS: event.GenerationDuration.Milliseconds(), RetryDelayMS: event.RetryDelay.Milliseconds(), DurationMS: event.Duration.Milliseconds(), Outcome: event.Outcome, Failure: event.Failure, StatusCode: event.StatusCode, RetriesExhausted: event.RetriesExhausted, Attempts: event.Attempts, TransientRetries: event.TransientRetries, StructuralRepairs: event.StructuralRepairs, Usage: mapLLMSummary(llm.ExecutionSummary{Usage: event.Usage}).Usage}) } return struct { - Calls []executionCall `json:"calls"` - Events []executionEvent `json:"events"` - }{calls, events} + Summary llm.ExecutionSummary `json:"summary"` + Calls []executionCall `json:"calls"` + Events []executionEvent `json:"events"` + }{llm.Summarize(snapshot), calls, events} } diff --git a/internal/framework/processreport/llm_test.go b/internal/framework/processreport/llm_test.go index 0b618cb..05ee0f5 100644 --- a/internal/framework/processreport/llm_test.go +++ b/internal/framework/processreport/llm_test.go @@ -19,7 +19,7 @@ func TestExecutionReportingKeepsSafeMilliseconds(t *testing.T) { {CallID: 1, Context: structuredoutput.DiagnosticContext{Role: "proposal", ModuleInstance: "grammar-1"}, Prompt: structuredoutput.PromptMetadata{BackendID: "primary", ProfileID: "draft", Model: "m"}, Started: start, Finished: start.Add(1500 * time.Millisecond), WallDuration: 1500 * time.Millisecond}, {CallID: 2, Context: structuredoutput.DiagnosticContext{Role: "validation"}, Prompt: structuredoutput.PromptMetadata{BackendID: "secondary"}, Started: start, Finished: start.Add(5 * time.Millisecond), WallDuration: 5 * time.Millisecond, Failure: structuredoutput.FailureRuntime}, }, - Events: []llm.ExecutionObservation{{CallID: 1, Kind: string(pk.ExecutionFinished), BackendID: "primary", Attempts: 2, TransientRetries: 1, CapacityWait: 20 * time.Millisecond, GenerationDuration: 500 * time.Millisecond, RetryDelay: 100 * time.Millisecond}}, + Events: []llm.ExecutionObservation{{CallID: 1, RunID: "run-1", Kind: string(pk.ExecutionFinished), BackendID: "primary", Attempts: 2, TransientRetries: 1, CapacityWait: 20 * time.Millisecond, GenerationDuration: 500 * time.Millisecond, RetryDelay: 100 * time.Millisecond}}, } raw, err := json.Marshal(BuildLLMExecution(snapshot)) if err != nil { @@ -46,7 +46,7 @@ func BenchmarkExecutionReports(b *testing.B) { snapshot := llm.ObservationSnapshot{} for i := 0; i < n; i++ { snapshot.Calls = append(snapshot.Calls, llm.LogicalCall{CallID: uint64(i + 1), Prompt: structuredoutput.PromptMetadata{BackendID: "local"}, Context: structuredoutput.DiagnosticContext{ModuleInstance: "grammar", Role: "proposal"}}) - snapshot.Events = append(snapshot.Events, llm.ExecutionObservation{CallID: uint64(i + 1), Kind: string(pk.ExecutionFinished), Attempts: 1}) + snapshot.Events = append(snapshot.Events, llm.ExecutionObservation{RunID: fmt.Sprint(i), CallID: uint64(i + 1), Kind: string(pk.ExecutionFinished), Attempts: 1}) } b.Run(fmt.Sprintf("execution/%d", n), func(b *testing.B) { for i := 0; i < b.N; i++ { @@ -64,9 +64,9 @@ func TestIndexedReportsPreserveOrderAndPartialUsage(t *testing.T) { {CallID: 2, Context: structuredoutput.DiagnosticContext{Role: "validation", ModuleInstance: "grammar_1"}, Prompt: structuredoutput.PromptMetadata{BackendID: "a"}}, }, Events: []llm.ExecutionObservation{ - {CallID: 1, Kind: string(pk.ExecutionFinished), Attempts: 1, Usage: llm.Usage{Total: llm.TokenCount{Value: 7, Known: true, Complete: true}}}, - {CallID: 3, Kind: string(pk.ExecutionFinished), Attempts: 1}, - {CallID: 2, Kind: string(pk.ExecutionFinished), Attempts: 1}, + {CallID: 1, RunID: "run-1", Kind: string(pk.ExecutionFinished), Attempts: 1, Usage: llm.Usage{Total: llm.TokenCount{Value: 7, Known: true, Complete: true}}}, + {CallID: 3, RunID: "run-3", Kind: string(pk.ExecutionFinished), Attempts: 1}, + {CallID: 2, RunID: "run-2", Kind: string(pk.ExecutionFinished), Attempts: 1}, }, } raw, err := json.Marshal(BuildLLMExecution(snapshot)) diff --git a/internal/framework/proposal_generation/module_proposal_test.go b/internal/framework/proposal_generation/module_proposal_test.go index 3747b0b..e3e134f 100644 --- a/internal/framework/proposal_generation/module_proposal_test.go +++ b/internal/framework/proposal_generation/module_proposal_test.go @@ -18,7 +18,7 @@ import ( ) func TestExecuteModuleProposalBuildsMessagesFromSectionAndDescription(t *testing.T) { - exchange := &structuredoutput.Exchange{PromptID: "captured", FinalContent: "response"} + exchange := &structuredoutput.Exchange{} client := &fakeStructuredClient{ completion: structuredoutput.Response{Exchange: exchange}, responses: []StructuredCorrectionSet{ diff --git a/internal/framework/runner/repair.go b/internal/framework/runner/repair.go index 75191cd..3553c0c 100644 --- a/internal/framework/runner/repair.go +++ b/internal/framework/runner/repair.go @@ -2,6 +2,7 @@ package runner import ( "context" + "encoding/json" "errors" "fmt" "path/filepath" @@ -117,8 +118,11 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput, return findings } findings := safeFindings(rejected) - replay := &structuredoutput.Replay{Exchange: input.Exchange} - for attempt := 0; attempt < input.Spec.Settings.LLM.CorrectionRepairAttempts; attempt++ { + coordinator, ok := input.Spec.RepairClient.(structuredoutput.Coordinator) + if !ok { + return finish(fmt.Errorf("correction repair coordinator is unavailable")) + } + buildRequest := func() (*structuredoutput.Request, error) { request := proposal_generation.RepairRequest{ ProposalRef: position + 1, Correction: proposal_generation.StructuredCorrectionProposal{SegmentReference: localRef, OriginalText: current.OriginalText, CorrectedText: current.CorrectedText}, @@ -134,12 +138,23 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput, } payload, err := request.JSON() if err != nil { - branches = append(branches, branch) - return finish(err) + return nil, err } + + proposalIndex, attemptNumber := original.ProposalIndex, branch.AttemptCount+1 + return &structuredoutput.Request{PromptID: promptID, Variables: map[string]string{"RepairRequestJSON": payload}, Schema: responseschema.CorrectionRepairKey, Context: structuredoutput.DiagnosticContext{StageIndex: &input.Spec.Index, StageID: input.Spec.ID, Role: "correction_repair", ModuleKey: input.Spec.ModuleKey, ModuleInstance: input.Spec.InstanceName, SectionIndex: input.SectionIndex, ProposalIndex: &proposalIndex, RepairAttempt: &attemptNumber}}, nil + } + exhausted, coordinationErr := coordinator.Coordinate(ctx, input.Exchange, input.Spec.Settings.LLM.CorrectionRepairAttempts, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) { + if turn.Response == nil { + return buildRequest() + } + completion, callErr := *turn.Response, turn.Err + proposalIndex, attemptNumber := original.ProposalIndex, branch.AttemptCount+1 var response proposal_generation.RepairResponse - proposalIndex, attemptNumber := original.ProposalIndex, attempt+1 - completion, callErr := input.Spec.RepairClient.CompleteStructured(ctx, structuredoutput.Request{PromptID: promptID, Variables: map[string]string{"RepairRequestJSON": payload}, Schema: responseschema.CorrectionRepairKey, Replay: replay, Context: structuredoutput.DiagnosticContext{StageIndex: &input.Spec.Index, StageID: input.Spec.ID, Role: "correction_repair", ModuleKey: input.Spec.ModuleKey, ModuleInstance: input.Spec.InstanceName, SectionIndex: input.SectionIndex, ProposalIndex: &proposalIndex, RepairAttempt: &attemptNumber}}, &response) + if callErr == nil { + _ = json.Unmarshal(completion.Content, &response) + } + operation := outcomes.Operation{ Reference: outcomes.Reference{ StageIndex: &input.Spec.Index, StageID: input.Spec.ID, ModuleKey: input.Spec.ModuleKey, @@ -183,26 +198,23 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput, finishAttempt(callErr, nil, validateSectionCandidatesResult{}) if structuredoutput.IsRecoverable(callErr) { branch.Outcome = "unavailable" - break + return nil, nil } - branches = append(branches, branch) - return finish(callErr) + return nil, callErr } decoded, err := proposal_generation.DecodeRepairResponse(completion.Content, position+1) if err != nil { operation.FailureReason = outcomes.ReasonInvalidOutput finishAttempt(err, nil, validateSectionCandidatesResult{}) branch.Outcome = "unavailable" - break + return nil, nil } operation.Succeeded = true operation.RecoveryReasons = outcomes.ExecutionRecoveries(operation.Counts) - replay.Turns = append(replay.Turns, completion.RepairInstruction...) - replay.Turns = append(replay.Turns, structuredoutput.Message{Role: "assistant", Content: completion.FinalContent}) if decoded.Action == "withdraw" { finishAttempt(nil, nil, validateSectionCandidatesResult{}) branch.Outcome = "withdrawn" - break + return nil, nil } revision := decoded.Correction localRef = revision.SegmentReference @@ -225,7 +237,7 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput, operation.RecoveryReasons = append(operation.RecoveryReasons, outcomes.RecoveryProposalsFiltered) } for i := range validated.operations { - proposalIndex, attemptNumber := original.ProposalIndex, attempt+1 + proposalIndex, attemptNumber := original.ProposalIndex, attemptNumber validated.operations[i].Reference.ProposalIndex = &proposalIndex validated.operations[i].Reference.RepairAttempt = &attemptNumber } @@ -236,32 +248,35 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput, finishAttempt(validationErr, ¤t, validated) initial.operations = append(initial.operations, validated.operations...) if validationErr != nil { - branches = append(branches, branch) - return finish(validationErr) + return nil, validationErr } if approved { branch.Outcome = "approved" initial.approved = append(initial.approved, current) superseded[original.ProposalIndex] = true - break + return nil, nil } if actionable(original.ProposalIndex, validated.rejected) { - if attempt+1 == input.Spec.Settings.LLM.CorrectionRepairAttempts { - branch.Outcome = "exhausted" - break - } findings = safeFindings(validated.rejected) - continue + return buildRequest() } branch.Outcome = "unavailable" for _, d := range validated.rejected { if d.Disposition != validators.DispositionUnavailable { branch.Outcome = "rejected" - break + return nil, nil } } - break + return nil, nil + }) + if exhausted { + branch.Outcome = "exhausted" } + if coordinationErr != nil { + branches = append(branches, branch) + return finish(coordinationErr) + } + branches = append(branches, branch) } return finish(nil) diff --git a/internal/framework/runner/repair_test.go b/internal/framework/runner/repair_test.go index 576e627..4be7cc3 100644 --- a/internal/framework/runner/repair_test.go +++ b/internal/framework/runner/repair_test.go @@ -4,10 +4,17 @@ import ( "context" "encoding/json" "errors" + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/llm" + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/responseschema" + "gitea.maximumdirect.net/eric/audita/v3/internal/prompts" + ck "go.maximumdirect.net/configkit" + "io" + "net/http" "os" "path/filepath" "strings" "testing" + "time" "gitea.maximumdirect.net/eric/audita/v3/internal/core/config" "gitea.maximumdirect.net/eric/audita/v3/internal/core/schema" @@ -20,6 +27,7 @@ import ( ) type repairClient struct { + t *testing.T outputs []string calls []structuredoutput.Request err error @@ -27,11 +35,7 @@ type repairClient struct { } func (c *repairClient) CompleteStructured(_ context.Context, req structuredoutput.Request, out any) (structuredoutput.Response, error) { - if req.Replay != nil { - copyReplay := *req.Replay - copyReplay.Turns = append([]structuredoutput.Message(nil), req.Replay.Turns...) - req.Replay = ©Replay - } + c.calls = append(c.calls, req) if c.err != nil { return structuredoutput.Response{}, c.err @@ -40,18 +44,18 @@ func (c *repairClient) CompleteStructured(_ context.Context, req structuredoutpu if err := json.Unmarshal([]byte(value), out); err != nil { return structuredoutput.Response{}, err } - return structuredoutput.Response{Prepared: &structuredoutput.PreparedSnapshot{Messages: []structuredoutput.Message{{Role: "user", Content: "synthetic-secret"}}}, Content: json.RawMessage(value), FinalContent: value, RepairInstruction: []structuredoutput.Message{{Role: "user", Content: "instruction"}}}, nil + return structuredoutput.Response{Prepared: &structuredoutput.PreparedSnapshot{Messages: []structuredoutput.Message{{Role: "user", Content: "synthetic-secret"}}}, Content: json.RawMessage(value), FinalContent: value, Exchange: &structuredoutput.Exchange{}}, nil } func (c *repairClient) ConfiguredSecrets(context.Context) []string { return c.secrets } func TestRepairBranchesRetargetAndPreserveOtherApprovals(t *testing.T) { - client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`}} + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`}} candidates := []proposals.EnrichedCorrectionProposal{ {CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 4}}, {CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 0, OriginalText: "x", CorrectedText: "y"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 5}}, } - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{ModuleKey: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}, {ID: 2, Text: "uh hello"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal", FinalContent: "original response"}} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{ModuleKey: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}, {ID: 2, Text: "uh hello"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -85,7 +89,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) { {"blank result", "uh", "uh", "", "", proposals.ReplacementPolicyRequireUnique, false}, } { t.Run(tc.name, func(t *testing.T) { - client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 81, OriginalText: tc.original, CorrectedText: tc.replacement}} input := validateSectionCandidatesInput{ Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, @@ -93,7 +97,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) { WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 81, Text: tc.segment}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}, rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, - Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, + Exchange: &structuredoutput.Exchange{}, } initial, err := validateSectionCandidates(context.Background(), input) if err != nil { @@ -125,7 +129,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) { } func TestRepairRequestRefreshesPreviewAfterRetarget(t *testing.T) { - client := &repairClient{outputs: []string{ + client := &repairClient{t: t, outputs: []string{ `{"proposal_ref":1,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`, `{"proposal_ref":1,"action":"withdraw","correction":null}`, }} @@ -138,7 +142,7 @@ func TestRepairRequestRefreshesPreviewAfterRetarget(t *testing.T) { WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 81, Text: "cat"}, {ID: 104, Text: "uh hello"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, reject}, SectionEnriched: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 81, OriginalText: "cat", CorrectedText: "dog"}}}, - Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, + Exchange: &structuredoutput.Exchange{}, } initial, err := validateSectionCandidates(context.Background(), input) if err != nil { @@ -175,9 +179,9 @@ func TestRepairBranchTerminalOutcomes(t *testing.T) { {"provider failure", nil, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonProviderFailed, Message: "provider failed"}, 1, "unavailable", 1}, } { t.Run(tc.name, func(t *testing.T) { - client := &repairClient{outputs: tc.outputs, err: tc.err} + client := &repairClient{t: t, outputs: tc.outputs, err: tc.err} candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}} - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: tc.budget}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: tc.budget}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -189,8 +193,8 @@ func TestRepairBranchTerminalOutcomes(t *testing.T) { if len(branches) != 1 || branches[0].Outcome != tc.outcome || len(client.calls) != tc.calls || branches[0].AttemptCount != tc.calls || len(branches[0].Attempts) != 0 { t.Fatalf("branches=%+v calls=%d", branches, len(client.calls)) } - if tc.calls == 2 && len(client.calls[1].Replay.Turns) != 2 { - t.Fatalf("history: %+v", client.calls[1].Replay.Turns) + if tc.calls == 2 && client.calls[1].Context.RepairAttempt == nil { + t.Fatalf("history: %+v", client.calls[1].ContinueFrom) } }) } @@ -202,11 +206,11 @@ func (m repairModule) RepairPromptID() string { return "repair" } func TestRunnerAppliesRevisionsInOriginalOrder(t *testing.T) { candidates := []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, {TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "fox"}, {TargetSegmentID: 1, OriginalText: "bird", CorrectedText: "owl"}} - client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"cat","corrected_text":"fox"}}`, + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"cat","corrected_text":"fox"}}`, `{"proposal_ref":3,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`, }} module := repairModule{outcomeModule{fakeModule: fakeModule{key: "grammar", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair(0)}}, proposeResult: func(req contracts.ProposalRequest) (contracts.ProposalResult, error) { - return contracts.ProposalResult{Proposals: candidates, SegmentReferences: []int{1, 1, 1}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}}, nil + return contracts.ProposalResult{Proposals: candidates, SegmentReferences: []int{1, 1, 1}, Exchange: &structuredoutput.Exchange{}}, nil }}} runner := New(fakeFactory{modules: map[string]contracts.TranscriptModule{"grammar": module}}) out, err := runner.Run(context.Background(), RunInput{Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat and bird"}}}, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "grammar", InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}}}) @@ -246,11 +250,11 @@ func TestRepairRevalidatesEarlierApprovalAndRetainsUnrelatedApprovalOnFailure(t } return validators.Result{Decisions: decisions}, nil }} - client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`}} + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`}} if tc.failure { client.err = &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonProviderFailed, Message: "failed"} } - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat bird"}}}, Validators: []contracts.Validator{check, rejectUntilRepair(0)}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat bird"}}}, Validators: []contracts.Validator{check, rejectUntilRepair(0)}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -271,9 +275,9 @@ func TestRepairRevalidatesEarlierApprovalAndRetainsUnrelatedApprovalOnFailure(t } func TestRepairInterruptionRetainsAttempt(t *testing.T) { - client := &repairClient{err: &structuredoutput.Failure{Class: structuredoutput.FailureCanceled, Reason: outcomes.ReasonCanceled, Message: "canceled", Cause: context.Canceled}} + client := &repairClient{t: t, err: &structuredoutput.Failure{Class: structuredoutput.FailureCanceled, Reason: outcomes.ReasonCanceled, Message: "canceled", Cause: context.Canceled}} candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}} - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -285,12 +289,12 @@ func TestRepairInterruptionRetainsAttempt(t *testing.T) { } func TestRepairArtifactsAreUniqueAndRedacted(t *testing.T) { - client := &repairClient{secrets: []string{"synthetic-secret"}, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`, `{"proposal_ref":2,"action":"withdraw","correction":null}`}} + client := &repairClient{t: t, secrets: []string{"synthetic-secret"}, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`, `{"proposal_ref":2,"action":"withdraw","correction":null}`}} candidates := []proposals.EnrichedCorrectionProposal{ {CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}}, {CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "fox"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 1}}, } - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar_1", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: t.TempDir()} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar_1", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: t.TempDir()} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -318,14 +322,14 @@ func TestRepairArtifactsAreUniqueAndRedacted(t *testing.T) { } func TestRepairArtifactFailureDoesNotChangeOutcome(t *testing.T) { - client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} path := filepath.Join(t.TempDir(), "file") if err := os.WriteFile(path, []byte("x"), 0600); err != nil { t.Fatal(err) } warnings := 0 candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}} - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: path, ReportDiagnosticError: func(error) { warnings++ }} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: path, ReportDiagnosticError: func(error) { warnings++ }} initial, err := validateSectionCandidates(context.Background(), input) if err != nil { t.Fatal(err) @@ -337,7 +341,7 @@ func TestRepairArtifactFailureDoesNotChangeOutcome(t *testing.T) { } func TestNoEffectAndUnavailableOnlyDoNotStartRepair(t *testing.T) { - client := &repairClient{} + client := &repairClient{t: t} for _, rejected := range [][]ValidatorRejectedChange{ {{ProposalIndex: 0, Disposition: validators.DispositionRejected, ReasonCode: validators.ReasonNoEffect}}, {{ProposalIndex: 0, Disposition: validators.DispositionUnavailable, ReasonCode: "provider_failed"}}, @@ -352,10 +356,10 @@ func TestNoEffectAndUnavailableOnlyDoNotStartRepair(t *testing.T) { } func TestRepairFeedbackRedactsValidationCredentials(t *testing.T) { - client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} - validation := &repairClient{secrets: []string{"validation-secret"}} + client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}} + validation := &repairClient{t: t, secrets: []string{"validation-secret"}} candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}} - input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, ValidationLLMClient: validation, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: t.TempDir()} + input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, ValidationLLMClient: validation, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: t.TempDir()} initial := validateSectionCandidatesResult{rejected: []ValidatorRejectedChange{{ProposalIndex: 0, ValidatorName: "editorial_review", Disposition: validators.DispositionRejected, ReasonCode: "rejected", Message: "model echoed validation-secret"}}} _, branches, err := repairCandidates(context.Background(), input, []int{1}, initial, "repair") if err != nil { @@ -393,3 +397,66 @@ func rejectUntilRepair(preserved ...int) contracts.Validator { return validators.Result{Decisions: decisions}, nil }} } + +type repairTransport func(*http.Request) (*http.Response, error) + +func (f repairTransport) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +// Exercise the real adapter and upstream coordinator; only model responses are scripted. +func (c *repairClient) Coordinate(ctx context.Context, _ *structuredoutput.Exchange, budget int, next func(context.Context, structuredoutput.Turn) (*structuredoutput.Request, error)) (bool, error) { + t := c.t + dir := t.TempDir() + for name, content := range map[string]string{ + "config.yml": "version: 3\n", + "prompts.yml": "profiles:\n - id: fixture\n endpoint: https://example.invalid/v1\n model: fixture\nprompts:\n - id: repair\n messages:\n - role: user\n content: 'synthetic-secret {{.RepairRequestJSON}}'\n output:\n format: text\n validation_mode: none\n", + } { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0600); err != nil { + t.Fatal(err) + } + } + loaded, err := config.Load(ck.Input{Source: ck.File(filepath.Join(dir, "config.yml"))}, "") + if err != nil { + t.Fatal(err) + } + var selected *structuredoutput.Request + initial := true + runtime, err := llm.NewRuntime(ctx, loaded, &http.Client{Transport: repairTransport(func(*http.Request) (*http.Response, error) { + content := `{"corrections":[]}` + if initial { + initial = false + } else { + c.calls = append(c.calls, *selected) + content = c.outputs[len(c.calls)-1] + } + body, _ := json.Marshal(map[string]any{"choices": []any{map[string]any{"message": map[string]any{"role": "assistant", "content": content}, "finish_reason": "stop"}}}) + return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(string(body)))}, nil + })}) + if err != nil { + t.Fatal(err) + } + client, err := runtime.NewRoleClient("repair", "fixture", config.LLM{OperationTimeout: time.Second}, llm.CallHooks{}) + if err != nil { + t.Fatal(err) + } + var output map[string]any + base, err := client.CompleteStructured(ctx, structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarProposal, Schema: responseschema.CorrectionSetKey, CaptureExchange: true, Variables: map[string]string{"TranscriptDescriptionBlock": "", "GlossaryJSON": "[]", "SectionJSON": "[]"}}, &output) + if err != nil { + return false, err + } + return client.Coordinate(ctx, base.Exchange, budget, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) { + req, err := next(ctx, turn) + if req != nil && err == nil { + selected = req + if c.err != nil { + c.calls = append(c.calls, *req) + _, err = next(ctx, structuredoutput.Turn{Response: &structuredoutput.Response{}, Err: c.err}) + return nil, err + } + // Test prompt IDs vary; all use the same supplied-data fixture definition. + copy := *req + copy.PromptID = "repair" + return ©, nil + } + return req, err + }) +} diff --git a/internal/framework/structuredoutput/call.go b/internal/framework/structuredoutput/call.go index 371401a..2a2786e 100644 --- a/internal/framework/structuredoutput/call.go +++ b/internal/framework/structuredoutput/call.go @@ -31,7 +31,7 @@ type Request struct { Schema responseschema.Key `json:"schema"` Context DiagnosticContext `json:"context"` CaptureExchange bool `json:"-"` - Replay *Replay `json:"-"` + ContinueFrom *Exchange `json:"-"` } // PromptMetadata is the execution snapshot returned by the runtime when available. @@ -44,33 +44,23 @@ type PromptMetadata struct { Model string `json:"model,omitempty"` } -// Exchange holds a section's original prepared conversation for read-only replay. +// Exchange is an opaque immutable conversation owned by the runtime adapter. type Exchange struct { - PromptID string - Variables map[string]string - Messages []Message - FinalContent string -} - -// Replay carries a section exchange and the prior turns of one repair branch. -type Replay struct { - Exchange *Exchange - Turns []Message + State any } type Response struct { - Exchange *Exchange `json:"-"` - RepairInstruction []Message `json:"-"` - Execution outcomes.ExecutionCounts `json:"execution"` - Content json.RawMessage `json:"content,omitempty"` - Provider string `json:"provider,omitempty"` - Model string `json:"model,omitempty"` - PromptTokens int `json:"prompt_tokens,omitempty"` - CompletionTokens int `json:"completion_tokens,omitempty"` - TotalTokens int `json:"total_tokens,omitempty"` - Prompt PromptMetadata `json:"prompt"` - Prepared *PreparedSnapshot `json:"prepared,omitempty"` - FinalContent string `json:"final_content,omitempty"` + Exchange *Exchange `json:"-"` + Execution outcomes.ExecutionCounts `json:"execution"` + Content json.RawMessage `json:"content,omitempty"` + Provider string `json:"provider,omitempty"` + Model string `json:"model,omitempty"` + PromptTokens int `json:"prompt_tokens,omitempty"` + CompletionTokens int `json:"completion_tokens,omitempty"` + TotalTokens int `json:"total_tokens,omitempty"` + Prompt PromptMetadata `json:"prompt"` + Prepared *PreparedSnapshot `json:"prepared,omitempty"` + FinalContent string `json:"final_content,omitempty"` } // PreparedSnapshot is the exact rendered request snapshot. Its messages may @@ -108,3 +98,16 @@ type CallRecord struct { type Client interface { CompleteStructured(ctx context.Context, req Request, out any) (Response, error) } + +// Turn delivers a completed correction request, or its failure. The initial +// callback has no response. A failure is terminal for this coordination. +type Turn struct { + Response *Response + Err error +} + +// Coordinator delegates bounded conversation progression to the runtime. +// Returning no request stops the branch; requesting beyond the budget exhausts it. +type Coordinator interface { + Coordinate(context.Context, *Exchange, int, func(context.Context, Turn) (*Request, error)) (bool, error) +}