Adopt PromptKit v0.23.0 conversations and execution accounting
This commit is contained in:
@@ -27,7 +27,7 @@ PromptKit loads its default catalog and Audita's embedded prompts, followed by
|
||||
optional `prompts.yml` and recursive `prompts/` YAML files beside the application
|
||||
main. Native imports, resource collections, inheritance, and replacement apply.
|
||||
Profiles, backends, models, credentials, admission, and provider parameters belong
|
||||
to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/formats.md).
|
||||
to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/formats.md).
|
||||
The engine captures its sources for an invocation. Edit files and run a new
|
||||
invocation to pick up changes. Inspection does not require model credentials.
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ failures; explicit zero makes one attempt. `llm.repair_attempts` counts
|
||||
additional generations for schema-invalid model output. The operation timeout
|
||||
covers preparation, capacity waiting, retries, generation, and repair. Caller
|
||||
cancellation or an earlier deadline also applies. See [configuration](../config.md#settings)
|
||||
for defaults and the [PromptKit consumer contract](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/consumers/pkg-promptkit.md)
|
||||
for defaults and the [PromptKit consumer contract](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/consumers/pkg-promptkit.md)
|
||||
for protocol and retry details.
|
||||
|
||||
A model operation that exhausts structural repair, a live-parent local operation
|
||||
|
||||
@@ -48,6 +48,14 @@ artifacts can still contain sensitive transcript and prompt content. Execution
|
||||
position and optional operator ID are carried separately from generated safe
|
||||
artifact path coordinates.
|
||||
|
||||
Execution diagnostics include a native-group invocation summary and per-call
|
||||
summaries. Terminal coverage is separate from usage completeness; unknown usage
|
||||
is distinct from known zero. Overlapping summaries must not be added together.
|
||||
The collector snapshots after work joins and retains domain labels and stable
|
||||
presentation order. Prepared evidence comes from `OnPrepared`, so preparation or
|
||||
admission failures may have no request snapshot. Coordinator execution failures
|
||||
still finalize the attempted logical call and its available evidence.
|
||||
|
||||
## Key Tests
|
||||
|
||||
Useful tests: `internal/core/reporting/*_test.go`,
|
||||
|
||||
@@ -9,27 +9,44 @@ it never replaces the requested engine.
|
||||
|
||||
Each resolved pipeline entry receives role-bound clients with its selected
|
||||
profiles and effective budgets. Clients share engine-local backend admission.
|
||||
The adapter uses `PrepareExecution` and `RunPrepared` on the same handle, passes
|
||||
The adapter uses ordinary `Run` with `OperationTimeout` and `OnPrepared`, passes
|
||||
Audita's per-call schema and repair budget, and decodes only validated results.
|
||||
The operation budget includes preparation, admission, retries, and structural
|
||||
repair. `ErrOperationTimeout` remains recoverable even when wrapped by a
|
||||
preparation error. Parent cancellation remains distinct. The prepared notification
|
||||
provides the actual request snapshot; failures before notification have no snapshot.
|
||||
PromptKit owns backend and credential selection. The adapter uses captured
|
||||
credential source names for artifact redaction without inspecting unrelated
|
||||
environment values. See the [integration contract](../integrations/openai-compatible-llm.md)
|
||||
for failure behavior and the [configuration reference](../config.md) for sources.
|
||||
|
||||
When correction repair is enabled, the proposal call retains its prompt,
|
||||
effective variables, prepared messages, and full final response for section-local
|
||||
replay. The adapter renders the repair definition with the selected profile,
|
||||
appends only that branch’s history, and checks the original prepared prefix
|
||||
before execution. The complete conversation is sent to the selected provider;
|
||||
provider rejection ends that branch as unavailable. The requested repair
|
||||
definition is retained in call attribution alongside native base-prompt metadata.
|
||||
When correction repair is enabled, the proposal call captures a native immutable
|
||||
PromptKit exchange. Each actionable correction starts an independent
|
||||
`Engine.Coordinate` branch from that exchange. PromptKit assembles continuation
|
||||
history, owns the additional-turn budget, and delivers the last permitted response
|
||||
for assessment. Follow-ups select the repair prompt, profile, and output schema
|
||||
directly; Audita does not rerender the original proposal or assemble replay messages.
|
||||
An opaque exchange in the structured-output contract keeps provider state outside
|
||||
modules and validators. The adapter shares request construction, notifications,
|
||||
observers, and result/failure finalization between ordinary and coordinated calls.
|
||||
|
||||
Audita owns actionable rejection, revision decoding, withdrawal, current previews,
|
||||
and full revalidation. Accepted siblings remain independent. Requesting more work
|
||||
at the budget boundary produces exhaustion without allocating another logical call.
|
||||
A canceled, undelivered response supplies diagnostic evidence only; it cannot enter
|
||||
the approved set. Repair disabled in configuration bypasses coordination entirely.
|
||||
The runner supplements each repair instruction with its current proposal preview
|
||||
as described in the [module guide](modules.md#failure-and-outcome-behavior).
|
||||
The bundled prompts separate durable system principles from the initial user
|
||||
assignment so later repair instructions can allow acceptable minor cleanup;
|
||||
see [prompt instruction ownership](prompts.md).
|
||||
|
||||
The observation collector tracks logical calls and execution events. Ordinary
|
||||
The observation collector tracks logical calls and execution events, and enrolls
|
||||
selected runs in a native `ExecutionGroup`. Proposal, repair, and nested validator
|
||||
clients explicitly share the invocation observer; history reuse adds no extra run.
|
||||
Native summaries deduplicate terminal totals by run ID and retain terminal coverage
|
||||
separately from usage presence and completeness. Logical calls and run membership
|
||||
can differ, especially on failures before execution. Ordinary
|
||||
reports use compact safe outcomes; request and response evidence is optional
|
||||
debug material. The runner preserves ordered pipeline traversal and deterministic
|
||||
section collection while canceling and joining peers on fatal errors.
|
||||
|
||||
@@ -20,7 +20,9 @@ formats; [configuration](../config.md) owns settings; the
|
||||
Recoverable failed batches mark their candidates unavailable; successful
|
||||
survivors continue through all required validators.
|
||||
6. Actionable rejections can enter independent section-local repair branches.
|
||||
Each revision is mapped to the original section and revalidated from the
|
||||
PromptKit coordinates each branch from the captured proposal exchange, owning
|
||||
its history and additional-turn limit. Each revision, including the final
|
||||
permitted response, is mapped to the original section and revalidated from the
|
||||
start. Withdrawal, exhaustion, and recoverable failures finish the branch
|
||||
without affecting independent approvals.
|
||||
7. Only fully approved proposals reach application. Corrections apply in
|
||||
|
||||
@@ -210,7 +210,7 @@ License and notices: [Apache 2.0](licenses/Apache-2.0.txt).
|
||||
|
||||
## PromptKit
|
||||
|
||||
Selected module: `go.maximumdirect.net/promptkit` at `v0.21.0`.
|
||||
Selected module: `go.maximumdirect.net/promptkit` at `v0.23.0`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
@@ -218,9 +218,9 @@ SPDX-License-Identifier: LGPL-3.0-or-later
|
||||
|
||||
License and notices: [LGPL v3](licenses/LGPL-3.0.txt) |
|
||||
[GPL v3](licenses/GPL-3.0.txt) |
|
||||
[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/COPYRIGHT).
|
||||
[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/fcbff80ab227397a03935ab3bf8114481c0736a7/COPYRIGHT).
|
||||
|
||||
[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09).
|
||||
[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/fcbff80ab227397a03935ab3bf8114481c0736a7).
|
||||
|
||||
## yaml.v3
|
||||
|
||||
|
||||
@@ -1,229 +1,106 @@
|
||||
# Prompt Quality Roadmap
|
||||
|
||||
**Updated:** 2026-11-11
|
||||
**Status:** The earlier six prompt improvements are implemented. This follow-up
|
||||
uses Audita's PromptKit v0.23.0 integration to review and develop proposal,
|
||||
validation, and repair prompts through the same execution path used in production.
|
||||
Live-model comparisons and broader context expansion remain deferred.
|
||||
|
||||
**Status:** All six stages implemented and verified. Broader context changes and
|
||||
live-model editorial evaluation remain deferred as described below. The plan
|
||||
followed the review of revision `d155bf4`, which covered all eleven prompt
|
||||
definitions, shared instructions, response schemas, and request assembly.
|
||||
That review identified design concerns, not measured live-model regressions.
|
||||
## Baseline and Boundaries
|
||||
|
||||
## Target State and Boundaries
|
||||
The [prompt registry](../internal/prompts.md),
|
||||
[validator guide](../internal/validators.md), and
|
||||
[LLM runtime](../internal/llm-runtime.md) own implemented behavior. The existing
|
||||
prompts separate durable system principles from initial proposal assignments,
|
||||
guide independent and overlapping edits, supply current before/after repair
|
||||
previews, include the target speaker in validation, and illustrate editorial
|
||||
judgments, revision, and withdrawal.
|
||||
|
||||
Audita's proposal, validation, and repair prompts will give each model call a
|
||||
coherent assignment and the evidence needed to carry it out. Proposals will
|
||||
better account for interacting edits, repair will receive an explicit view of
|
||||
its current proposed result, and validation will identify the target speaker.
|
||||
Focused examples will clarify difficult editorial judgments.
|
||||
Preserve these selected decisions:
|
||||
|
||||
Preserve the established design:
|
||||
- Module focus guides generation; otherwise acceptable minor editorial cleanup
|
||||
remains permissible across modules under [ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
||||
- Keep the carefully tuned meaning-reversal criterion and distinct validator
|
||||
responsibilities. Approvals require no explanation; confidence scores remain absent.
|
||||
- Preserve literal spans, deletion, replacement policies, glossary context,
|
||||
protected spellings, full revalidation, and deterministic application.
|
||||
- Retain English defaults and native operator prompt replacement. Module sequence
|
||||
reevaluation follows the [module guidance](../internal/modules.md).
|
||||
|
||||
- Module focus guides proposal generation while otherwise acceptable minor
|
||||
editorial cleanup remains permissible across modules, as selected in
|
||||
[ADR 0003](../adr/0003-module-focus-and-editorial-acceptance.md).
|
||||
- Validators retain distinct responsibilities. Preserve the carefully tuned
|
||||
meaning-reversal criterion and its focus on reversal of the full assertion.
|
||||
- Keep literal-span matching, deletion semantics, separate glossary context and
|
||||
protected spellings, and full before/after validation.
|
||||
- Keep independent repair conversations with full history, successful withdrawal,
|
||||
and full revalidation of revisions. Validators explain defects; repair owns
|
||||
revision or withdrawal. Approvals continue to need no explanation.
|
||||
- Retain confidence-free responses, English defaults, existing module identities,
|
||||
and native operator prompt replacement. The current module sequence is outside
|
||||
this pass; ongoing reevaluation follows the [module guidance](../internal/modules.md).
|
||||
PromptKit now owns immutable captured exchanges, continuation assembly, bounded
|
||||
coordination, per-operation deadlines, prepared-request notification, and selected
|
||||
run accounting. Audita owns candidate identity, findings, previews, acceptance,
|
||||
withdrawal, and transcript mutation. Follow-up prompt work must use these APIs;
|
||||
it must not recreate message replay, turn loops, or token aggregation locally.
|
||||
|
||||
The [prompt registry](../internal/prompts.md), [validator guide](../internal/validators.md),
|
||||
and [LLM runtime](../internal/llm-runtime.md) describe the implemented baseline.
|
||||
Update these canonical owners when the corresponding changes land.
|
||||
## 1. Review Complete Native Conversations
|
||||
|
||||
## Implementation Sequence
|
||||
Use captured proposal exchanges and independent coordinator branches to inspect
|
||||
representative proposal-to-repair conversations, including a second revision and
|
||||
withdrawal. Review actual `OnPrepared` snapshots for instruction authority,
|
||||
response-format transitions, glossary/protected-term context, and fresh previews.
|
||||
A follow-up selects its repair definition, role profile, and schema directly.
|
||||
|
||||
### 1. Align Proposal and Repair Instruction Authority
|
||||
Exercise operator replacements through native prompt discovery and precedence.
|
||||
Compare the request that actually executes rather than separately rendering an
|
||||
approximation. Reusing captured history must preserve the proposal context while
|
||||
keeping each correction's repair history private.
|
||||
|
||||
Proposal system messages currently include categorical task restrictions, such
|
||||
as prohibiting punctuation or grammar edits in the homophone module. Repair
|
||||
inherits those messages and then permits acceptable minor editorial cleanup in a
|
||||
later user message. These instructions can compete across authority levels.
|
||||
Record concrete editorial concerns and revise affected prompts when warranted.
|
||||
Do not change validator acceptance or narrow allowable cleanup merely to make
|
||||
proposal and repair assignments identical.
|
||||
|
||||
- Put durable correction principles in the system messages: evidence-supported
|
||||
work, established protection and meaning constraints, and the distinction
|
||||
between requested focus and acceptable editorial corrections.
|
||||
- Put the initial proposal's focused assignment and task-specific exclusions in
|
||||
its user message. Preserve the focused tasks rather than asking every module
|
||||
to perform every kind of cleanup.
|
||||
- Let each repair instruction give the subsequent reassessment assignment,
|
||||
including the selected editorial allowance, without conflicting with an
|
||||
inherited system prohibition.
|
||||
- Review each complete proposal-to-repair conversation, including later repair
|
||||
attempts, for coherent instructions and response-format transitions.
|
||||
## 2. Assess Repair Feedback and Outcomes
|
||||
|
||||
This implements the existing acceptance decision; it does not tighten validators
|
||||
or change the repair lifecycle. Keep the validator prompts' separate remits,
|
||||
especially meaning-reversal review, intact.
|
||||
Use the real coordinator with controlled model responses for deterministic
|
||||
integration checks. Cover revision, withdrawal, final-turn acceptance, exhaustion,
|
||||
unavailable responses, and independent accepted siblings. Revalidate each revision
|
||||
from the beginning. A normal coordination stop is not evidence of acceptance;
|
||||
Audita's domain outcome determines that. Canceled undelivered content is evidence
|
||||
only and must never enter the approved set.
|
||||
|
||||
### 2. Guide Independently Useful and Overlapping Corrections
|
||||
Use current target previews and validator-owned findings to assess whether a
|
||||
repair request explains the defect sufficiently. Keep presentation changes in
|
||||
prompt assets and domain feedback construction; keep history and budget mechanics
|
||||
upstream. Retain separate transient retry, structural repair, and correction
|
||||
revision budgets.
|
||||
|
||||
The shared proposal contract already makes each edit refer to the supplied
|
||||
segment rather than the imagined result of another proposal. Add guidance for
|
||||
edits that overlap or depend on one another:
|
||||
## 3. Diagnose Evidence Gaps Using Actual Requests
|
||||
|
||||
- Prefer independently useful corrections with literal spans that identify the
|
||||
intended changes.
|
||||
- Combine changes into one proposal when their spans overlap or when one edit
|
||||
requires the other. Preserve unchanged context needed for unique matching or
|
||||
for selecting an occurrence under the glossary replacement policy.
|
||||
- Continue to allow multiple independent corrections to the same segment.
|
||||
For a concrete rejected correction, compare the captured proposal evidence with
|
||||
what the validator actually received. The target speaker is already included.
|
||||
Broader validation context remains evidence-dependent: propose a focused change
|
||||
only when a captured case demonstrates relevant evidence missing from validation.
|
||||
|
||||
For example, separate proposals to capitalize `hello` and replace
|
||||
`hello world` with `hello world` may both pass validation against the original,
|
||||
but applying the capitalization first prevents the second span from matching.
|
||||
A combined replacement can express both edits against the original text.
|
||||
Use native execution-group totals to distinguish additional model work from
|
||||
shared initial history. Select validator membership explicitly, count the initial
|
||||
proposal once, and do not add overlapping group totals. Preserve unknown usage
|
||||
and terminal coverage separately. These measurements describe execution and cost;
|
||||
they do not establish editorial quality.
|
||||
|
||||
These are prompt instructions. Add no new runtime overlap rejection, proposal
|
||||
limit, or application-order rule. Assess combined semantic effects separately
|
||||
if practical examples establish a need for additional processing changes.
|
||||
Existing local examples have not established a need for broader validation windows
|
||||
or combined semantic validation. Keep context fitting and apply-time conflict
|
||||
handling unchanged until concrete cases justify further work.
|
||||
|
||||
### 3. Supply Repair with the Current Before/After Segment
|
||||
## 4. Verification and Documentation
|
||||
|
||||
The repair request currently identifies the proposal, its current correction,
|
||||
and validation findings. Although the original section remains available in
|
||||
conversation history, repair must locate it and reconstruct the proposed result.
|
||||
Follow the [testing policy](../policy/testing.md). Use the real PromptKit adapter
|
||||
and coordinator with deterministic model-boundary fixtures. Protect supplied-data
|
||||
rendering, schemas, profile selection, current previews, branching, revalidation,
|
||||
withdrawal, and operator replacement. Inspect representative rendered prose as
|
||||
part of review; avoid phrase checks, prose snapshots, and tests that duplicate
|
||||
upstream history or accounting algorithms.
|
||||
|
||||
- Add the original target segment and its proposed resulting text to repair
|
||||
context when the current proposal is applicable.
|
||||
- Reuse the same domain preview semantics used by validation, including literal
|
||||
deletion, whitespace, and replacement of every matching occurrence where
|
||||
applicable. Avoid a second implementation of replacement behavior.
|
||||
- Refresh this view for each repair attempt, using the current proposal version
|
||||
and target rather than a stale view of the initial proposal.
|
||||
- When a preview cannot be produced, retain the relevant findings and available
|
||||
source context without fabricating a resulting segment or implying application
|
||||
succeeded.
|
||||
- Keep the full conversation history, proposal references, and revision or
|
||||
withdrawal response contract. Explain the added view as evidence for assessing
|
||||
the current correction, not an already-applied transcript change.
|
||||
Update canonical prompt and validator documentation when behavior changes.
|
||||
Keep examples valid against their response schemas and verify affected links.
|
||||
Run focused tests and the full offline suite for runtime changes. Pure wording
|
||||
changes need proportionate review, not new change-detector tests.
|
||||
|
||||
This should make spacing, deletion, and replacement consequences easier to
|
||||
judge without requiring the model to reproduce Audita's application mechanics.
|
||||
## Deferred Evaluation
|
||||
|
||||
### 4. Improve Validation Evidence
|
||||
Formal corpus preparation and live-model comparisons remain optional follow-up
|
||||
work. Keep live calls opt-in and outside the offline suite. When undertaken,
|
||||
compare needed and unnecessary edits, missed corrections, validator disagreement,
|
||||
and repair outcomes using independent editorial judgment. Native groups can
|
||||
supply execution measurements without introducing a second accounting system.
|
||||
|
||||
Add the target segment's speaker to each validation item. Neighboring segments
|
||||
already carry their speakers; the candidate's identity completes that context
|
||||
and helps distinguish continued speech, interruptions, and another speaker's
|
||||
response. Carry the supplied value through the existing payload construction
|
||||
and token accounting without inferring a missing speaker.
|
||||
|
||||
Investigate the broader evidence imbalance using available practical cases:
|
||||
proposal sees a section, while validation sees immediate neighbors and selected
|
||||
glossary entries. A correction supported several turns earlier may appear
|
||||
unsupported to a validator. Compare the evidence actually supplied to both
|
||||
calls before attributing a rejection to prompt wording.
|
||||
|
||||
Adding the target speaker is agreed implementation work. Substantial expansion
|
||||
of validation context remains evidence-dependent: record a concrete case and a
|
||||
focused follow-up recommendation if needed. Retain the current context selection
|
||||
and fitting behavior otherwise. Stronger instructions cannot supply missing
|
||||
evidence, and this roadmap does not require a broader context redesign.
|
||||
|
||||
### 5. Add Focused Editorial and Repair Examples
|
||||
|
||||
Add a small number of examples where they clarify judgments that current
|
||||
instructions leave implicit:
|
||||
|
||||
- Apparent filler that contributes meaning, such as `like` expressing
|
||||
approximation or `I mean` introducing a substantive self-correction. Use
|
||||
surrounding context to contrast meaningful usage with removable dysfluency.
|
||||
- One compact repair revision and one withdrawal, using the existing response
|
||||
schema and making withdrawal a successful preservation of the original text.
|
||||
- A combined overlapping edit illustrating step 2.
|
||||
|
||||
Place examples in the relevant proposal, validation, or shared repair guidance.
|
||||
Keep existing useful contextual contrasts. Examples should illustrate reasoning
|
||||
without becoming unconditional token-based editing rules or an extensive
|
||||
rulebook. Maintain the narrow meaning-reversal instructions.
|
||||
|
||||
### 6. Verify Integration, Documentation, and Test-Policy Compliance
|
||||
|
||||
Follow the [testing policy](../policy/testing.md) throughout implementation and
|
||||
finish with a focused review of affected tests:
|
||||
|
||||
- Verify repair context reflects the current proposal and the shared preview
|
||||
behavior, including representative deletion, replacement-policy, subsequent
|
||||
revision, and unavailable-preview cases where coverage is warranted.
|
||||
- Verify target-speaker data reaches validation and participates in existing
|
||||
payload accounting. Preserve coverage of response handling, revision,
|
||||
withdrawal, revalidation, and operator prompt replacement.
|
||||
- Review rendered messages and representative complete repair conversations
|
||||
directly for instruction authority, editorial intent, and format consistency.
|
||||
Check response examples against their schemas.
|
||||
- Remove any affected tests that freeze bundled prompt wording through phrase
|
||||
checks, exact prose comparisons, or snapshots. Do not replace them with checks
|
||||
for the new wording. Retain tests of supplied data and meaningful machine
|
||||
contracts where their protective value justifies their maintenance cost.
|
||||
- Update the relevant canonical prompt, module, validator, and runtime documents
|
||||
as behavior lands. Update maintained examples where affected; check links and
|
||||
whitespace.
|
||||
|
||||
Run the appropriate package checks, build, and full offline suite for the
|
||||
implementation under the [development guidance](../development.md#baseline-validation).
|
||||
Report the test-policy review's meaningful removals, revisions, and retained
|
||||
coverage. Offline tests establish data and execution contracts; they do not
|
||||
establish live-model editorial quality.
|
||||
|
||||
## Deferred Evidence and Follow-Up
|
||||
|
||||
The evidence review examined the maintained local, campaign, and prompt-edit
|
||||
transcript examples, adjacent-context validation fixtures, and proposal
|
||||
application fixtures. The glossary example is a single-segment correction;
|
||||
the validation fixture supplies the troupe definition and adjacent evidence
|
||||
needed to assess its proposed name change. These sources contain no recorded
|
||||
live-model rejection demonstrating a need for more distant context. The target
|
||||
speaker is now included; broader selection and context fitting remain unchanged.
|
||||
Revisit the window when a captured case shows material evidence in the proposal
|
||||
request that was absent from validation.
|
||||
|
||||
The existing stale-span application fixture demonstrates the mechanical overlap
|
||||
addressed by the new guidance. It does not establish a need for new combined
|
||||
semantic validation, so application ordering and rejection rules remain unchanged.
|
||||
|
||||
Formal evaluation corpus preparation and live-model comparisons remain deferred
|
||||
and are not prerequisites for the agreed changes. Broader validation context and
|
||||
additional handling of combined semantic effects should follow concrete evidence
|
||||
rather than hypothetical restrictions. Any later evaluation should examine
|
||||
needed and unnecessary edits, missed corrections, validator disagreement, and
|
||||
repair outcomes using independent editorial judgment. Keep live calls opt-in
|
||||
and separate from the default offline suite.
|
||||
|
||||
No blocking product decisions remain for the implementation scope above.
|
||||
|
||||
## Completion and Verification
|
||||
|
||||
- The four proposal definitions now separate durable system principles from
|
||||
their initial user assignments. The shared proposal guidance covers overlapping
|
||||
and dependent edits; the grammar prompt illustrates a combined edit.
|
||||
- Every repair attempt receives the current target's original text when
|
||||
available and its previewed result when applicable, using the same domain
|
||||
replacement function as validation. Revision and withdrawal examples accompany
|
||||
the shared repair guidance.
|
||||
- Validation includes the target speaker in rendered input and token accounting.
|
||||
Spoken-word proposal and editorial review examples distinguish meaningful
|
||||
apparent filler from removable hesitation. The meaning-reversal definition and
|
||||
all response schemas retain their existing contracts.
|
||||
- Behavioral tests cover repair deletion and replace-all previews, ambiguous or
|
||||
missing spans, blank results, missing targets, and refresh after retargeting.
|
||||
Existing validation tests now cover target-speaker propagation, absence, and
|
||||
budget fitting, including parity with serialized payload estimates.
|
||||
- The affected prompt, module, request-assembly, and conversation tests were
|
||||
reviewed for wording checks. No further bundled-prose change detectors were
|
||||
found to remove; the earlier cleanup remains effective. Retained assertions
|
||||
protect supplied-data rendering, schema selection, operator replacement,
|
||||
conversation history, revalidation, withdrawal, and outcomes. New assertions
|
||||
check request data and behavior rather than prompt phrases.
|
||||
- All eleven definitions prepared through the CLI. Four illustrative complete
|
||||
conversations, including subsequent repair requests, were reviewed, and the
|
||||
sixteen embedded JSON response examples passed schema checks. The application
|
||||
build and full offline Go suite passed. Canonical documentation and maintained
|
||||
preview examples were updated; links and whitespace were checked.
|
||||
|
||||
These checks verify implementation and editorial intent, not live-model quality.
|
||||
No blocking product decisions remain for this follow-up scope.
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
# PromptKit v0.23.0 Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation; all tasks below are pending.
|
||||
|
||||
The [adoption roadmap](promptkit.md) owns the target behavior and boundaries.
|
||||
This plan sequences the direct v0.21.0-to-v0.23.0 migration and the evidence needed
|
||||
to establish full adoption. It does not authorize application code changes during
|
||||
the current documentation pass.
|
||||
|
||||
## 1. Upgrade and unify execution
|
||||
|
||||
- [ ] Update `go.mod` and `go.sum` directly to
|
||||
`go.maximumdirect.net/promptkit v0.23.0`. Use released module resolution; remove
|
||||
temporary replacements/workspace files. Check the selected module graph and
|
||||
compatibility with the existing CommandKit/ckprompt integration.
|
||||
- [ ] Refactor [the LLM adapter](../../internal/framework/llm/adapter.go) around
|
||||
shared request construction, response interpretation, and error mapping that
|
||||
can serve both ordinary calls and coordinator-owned turns. Keep those helpers
|
||||
narrow; do not wrap `Coordinate` in another turn loop.
|
||||
- [ ] Move whole-operation timing to `RunRequest.OperationTimeout`. Use ordinary
|
||||
`Run` for calls whose prepare/run split exists only to obtain diagnostics.
|
||||
Avoid retaining a redundant child timer around the same ordinary operation.
|
||||
- [ ] Supply `OnPrepared` to collect effective metadata and selected credential
|
||||
source names, and retain prepared content for debug when enabled. Use the
|
||||
delivered snapshot without a separate `Prepare` call. Clear per-call state
|
||||
so a failure before notification cannot reuse a prior turn's evidence.
|
||||
- [ ] Preserve per-role profile/schema/variable selection, transient and
|
||||
structural-repair budgets, ordinary observers, output decoding, and error
|
||||
disposition. Map `ErrOperationTimeout` explicitly, including timeouts during
|
||||
preparation. Distinguish parent cancellation and provider-local deadlines.
|
||||
Retain the public processing reason codes where their meanings still apply.
|
||||
- [ ] Ensure call finalization runs on both results and errors; coordinator
|
||||
execution errors do not arrive as ordinary response callbacks. Preserve the
|
||||
last attempted operation's diagnostic and failure context for final reporting.
|
||||
|
||||
**Verification:** focused adapter tests for exact prepared evidence after a
|
||||
provider failure, absent evidence before notification, independent operation
|
||||
budgets, timeout classification, metadata redaction, and ordinary observer
|
||||
composition. Use controlled model clients and cancellation. Retain explicit
|
||||
prepared execution only where an identified consumer actually requires it.
|
||||
|
||||
## 2. Replace history and repair-loop mechanics
|
||||
|
||||
- [ ] Replace the local `Exchange`/`Replay` model in
|
||||
[structured completion contracts](../../internal/framework/structuredoutput/call.go)
|
||||
with a minimal opaque bridge to native captured exchanges. The adapter owns
|
||||
PromptKit execution; modules and validators must not reconstruct messages or
|
||||
depend on provider-specific state.
|
||||
- [ ] Capture the original proposal run through `CaptureExchange`. Select the
|
||||
repair prompt directly for a continuation; remove original-prompt rerendering,
|
||||
manual assistant/suffix assembly, prefix comparison, and `RepairInstruction`
|
||||
history plumbing. Preserve prepared snapshots used only for diagnostic output.
|
||||
- [ ] Replace the attempt loop in
|
||||
[repair coordination](../../internal/framework/runner/repair.go) with one
|
||||
adapter-backed `Engine.Coordinate` invocation per actionable rejected proposal.
|
||||
Keep scheduling across proposals in Audita. Pass the resolved correction repair
|
||||
limit as `AdditionalTurns`, and bypass coordination when repair is disabled.
|
||||
- [ ] Keep domain branch state in the callback: original proposal identity,
|
||||
current revision, local segment mapping, current before/after preview, safe
|
||||
rejection findings, and domain outcome. The initial callback uses existing
|
||||
findings without rerunning the initial batch validation.
|
||||
- [ ] Build each follow-up request with the repair definition, role profile,
|
||||
repair schema, variables, operation timeout, prepared notification, and selected
|
||||
observers. Let the coordinator attach `ContinueFrom` and manage history.
|
||||
- [ ] Process `ConversationTurn.Result` only when present. Check structural
|
||||
validation before schema decoding, interpret withdrawal, and revalidate every
|
||||
revision from the beginning using the parent callback context. Validator calls
|
||||
select their own profiles, timeouts, and observers through the shared adapter.
|
||||
- [ ] Return a nil next request for accepted, withdrawn, or unavailable domain
|
||||
outcomes. If rejection remains actionable, request a further turn even at the
|
||||
budget boundary so PromptKit reports `CoordinationExhausted`; do not recreate
|
||||
an independent exhaustion loop. The final permitted revision is still assessed.
|
||||
- [ ] Keep diagnostic attempt coordinates and operation records, but allocate
|
||||
execution evidence only for runs actually started. The callback may construct
|
||||
a request that is never executed because the turn budget is exhausted.
|
||||
- [ ] Map execution errors, callback errors, stopping, exhaustion, and
|
||||
`Undelivered` separately. Unassessed or structurally invalid content never
|
||||
reaches the approved set. Preserve accepted sibling proposals and apply-time
|
||||
checks; fatal/canceled work still cancels and joins peers.
|
||||
|
||||
**Verification:** a controlled workflow with one initial approval and two rejected
|
||||
proposals demonstrates independent histories, withdrawal, revised acceptance,
|
||||
full revalidation, and deterministic application. Cover final-turn acceptance,
|
||||
exhaustion, zero repair, structural invalidity, recoverable failures, parent
|
||||
cancellation before delivery, and capacity-one validator calls. Assertions should
|
||||
exercise Audita outcomes through the real upstream coordinator with a fake model
|
||||
boundary, rather than test a fake coordinator's loop behavior.
|
||||
|
||||
## 3. Adopt native execution accounting
|
||||
|
||||
- [ ] Introduce an invocation-scoped `ExecutionGroup` and compose its `Observe`
|
||||
callback with existing diagnostic observers using `ComposeObservers`. Ensure
|
||||
observation remains enabled wherever required accounting is consumed,
|
||||
independently of whether debug artifacts are enabled.
|
||||
- [ ] Explicitly enroll proposal, repair-generation, and validator runs. Use
|
||||
`CoordinateRequest.Observer` for continuation membership and select membership
|
||||
separately for nested validation requests. Keep conversation parent identity
|
||||
distinct from Audita's stage/section/proposal/validator labels.
|
||||
- [ ] Replace the terminal-map deduplication and usage summation in
|
||||
[observation summaries](../../internal/framework/llm/observations.go) with
|
||||
`ExecutionGroup.Snapshot()`. Convert upstream summary fields only at presentation
|
||||
boundaries. Retain logical call metadata, safe event evidence, statuses, and
|
||||
deterministic ordering that the upstream group does not supply.
|
||||
- [ ] Carry terminal coverage separately from usage known/completeness into the
|
||||
internal summary and its diagnostic representation. Do not present missing
|
||||
terminals as complete zero accounting. Keep the established ordinary report
|
||||
scope; adding token payloads to the ordinary errors/recoveries report is not
|
||||
part of this upgrade.
|
||||
- [ ] Count shared initial work once in the invocation group; count each new
|
||||
continuation and validator run separately. Do not sum overlapping subgroup
|
||||
totals or attempt events on top of their terminal totals. Sort any presented
|
||||
member lists deterministically rather than relying on native snapshot ordering.
|
||||
- [ ] Snapshot final totals after all selected work has joined. Groups add no
|
||||
separate close/wait lifecycle. Audit and delete local aggregation that became
|
||||
redundant; document the purpose of any remaining application-specific counters.
|
||||
|
||||
**Verification:** integration tests demonstrate composition with existing
|
||||
observers, explicit validator membership, shared initial work, failure terminal
|
||||
facts, unknown versus zero usage, missing terminal coverage, and concurrent calls.
|
||||
Reuse upstream group semantics; test Audita's enrollment and presentation instead
|
||||
of duplicating the library's unit suite.
|
||||
|
||||
## 4. Cleanup, documentation, and acceptance
|
||||
|
||||
- [ ] Audit references to the old exchange/replay types, prefix checks,
|
||||
`RepairInstruction`, local repair attempt loop, redundant timeout wrapper, and
|
||||
terminal aggregation. Delete dead production helpers and replace tests tied to
|
||||
those implementations with focused behavior coverage. A thin local adapter is
|
||||
justified; a parallel conversation/coordinator/accounting implementation is not.
|
||||
- [ ] Update the canonical [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md),
|
||||
[diagnostics/reporting internals](../internal/diagnostics-reporting.md), and
|
||||
[LLM integration contract](../integrations/openai-compatible-llm.md) as behavior
|
||||
lands. Update pinned upstream references and dependency attribution as needed.
|
||||
Document the selected notification boundary and timeout mapping without claiming
|
||||
that preparation or admission failures always have a prepared snapshot.
|
||||
- [ ] Update configuration/operations references only for actual externally
|
||||
visible changes. No prompt rewrite, validator tuning, new configuration format,
|
||||
compatibility shim, new scheduler, or unavailable-decision regeneration is
|
||||
required by this migration.
|
||||
- [ ] Follow the [testing policy](../policy/testing.md): keep tests that protect
|
||||
domain and integration behavior, remove obsolete implementation-coupled tests,
|
||||
and avoid prompt word/phrase change detectors. Review final tests for redundancy.
|
||||
- [ ] Run the focused migration tests, the complete Audita suite, and a race-enabled
|
||||
check of changed concurrent adapter/runner/accounting paths. Build the CLI and
|
||||
exercise existing offline processing and inspection coverage. No paid provider
|
||||
calls are required for acceptance.
|
||||
- [ ] Review the final diff, verify local Markdown links and `git diff --check`,
|
||||
and confirm no local dependency replacements or workspace artifacts remain.
|
||||
Verify the selected PromptKit version is v0.23.0.
|
||||
- [ ] Confirm all three upstream capability families are used in production
|
||||
processing and the agreed local machinery is retired. Record completion only
|
||||
after this end-state audit; move useful current contracts into their canonical
|
||||
owners and retire these temporary planning documents under repository policy.
|
||||
|
||||
Suggested final checks, using the locally selected Go 1.27.0-or-newer toolchain:
|
||||
|
||||
```sh
|
||||
go list -m go.maximumdirect.net/promptkit
|
||||
go test ./...
|
||||
go test -race ./internal/framework/llm ./internal/framework/runner ./internal/framework/processreport
|
||||
go build -o ./bin/audita ./cmd/audita
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Adjust focused package checks if code ownership moves. A green dependency-only
|
||||
suite is not evidence that the new workflow has been adopted; acceptance also
|
||||
requires the coordinator, conversation, and group integration scenarios above.
|
||||
@@ -1,136 +0,0 @@
|
||||
# PromptKit v0.23.0 Adoption Roadmap
|
||||
|
||||
**Status:** Accepted target; implementation pending. This pass changes planning
|
||||
documents only. The [implementation plan](promptkit-implementation.md) owns the
|
||||
ordered tasks and verification.
|
||||
|
||||
## Objective and Baseline
|
||||
|
||||
Upgrade Audita directly from PromptKit v0.21.0 to v0.23.0 and fully adopt its
|
||||
conversation, coordinator, and execution-group APIs. Removing redundant local
|
||||
machinery is an explicit completion requirement. A dependency-only upgrade or
|
||||
continuation adoption with the old repair loop retained is an intermediate state.
|
||||
|
||||
The current [dependency manifest](../../go.mod) pins v0.21.0. Audita captures its
|
||||
own exchange and replay values, reconstructs and compares conversation prefixes,
|
||||
appends repair history, manages the revision loop, and aggregates terminal
|
||||
observations locally. Each role client wraps explicit preparation and execution
|
||||
in a local deadline. See the [LLM runtime](../internal/llm-runtime.md),
|
||||
[pipeline](../internal/pipeline.md), and
|
||||
[diagnostic implementation](../internal/diagnostics-reporting.md).
|
||||
|
||||
[PromptKit v0.23.0](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.0/docs/releases/v0.23.0.md)
|
||||
includes the v0.22.0 conversation and accounting capabilities plus the execution
|
||||
controls needed for full coordinator adoption. The former upstream timeout and
|
||||
prepared-request gaps are closed; no intermediate v0.22.0 rollout or further
|
||||
upstream feature gate is required. The Go minimum and dependency versions are
|
||||
unchanged by these upstream releases.
|
||||
|
||||
The release review verified Audita's existing suite against the tagged v0.23.0
|
||||
source through a temporary module override and selected upstream packages with
|
||||
race detection. That is compatibility evidence, not verification of the migration
|
||||
described here. Implementation must use the released dependency, without a local
|
||||
replacement or committed workspace file.
|
||||
|
||||
## Target Ownership
|
||||
|
||||
| Responsibility | Target owner |
|
||||
| --- | --- |
|
||||
| Immutable captured history, continuation assembly, and parent run identity | PromptKit `Exchange`, `CaptureExchange`, and `ContinueFrom` |
|
||||
| Sequential turns, additional-turn budget, callback delivery, and coordination termination | PromptKit `Engine.Coordinate` |
|
||||
| Whole-operation deadlines and timeout attribution | PromptKit `OperationTimeout` and `ErrOperationTimeout` |
|
||||
| Exact prepared-request notification before generation | PromptKit `OnPrepared` |
|
||||
| Selected run membership, deduplication, and terminal count/usage aggregation | PromptKit `ExecutionGroup` and `ComposeObservers` |
|
||||
| Candidate identity, actionable rejection, feedback, revision decoding, withdrawal, and full revalidation | Audita |
|
||||
| Branch scheduling, accepted sibling proposals, application order, and transcript mutation | Audita |
|
||||
| Domain operation labels, failure disposition, reports, redaction, and debug artifact storage | Audita |
|
||||
|
||||
Use the supported public API through Audita's framework adapter boundaries.
|
||||
Keep module and validator contracts domain-oriented. Any bridge needed by the
|
||||
runner must delegate to the real coordinator and retain opaque conversation
|
||||
state, rather than implementing another conversation engine or general workflow
|
||||
framework.
|
||||
|
||||
## Selected Behavior
|
||||
|
||||
### Execution and diagnostics
|
||||
|
||||
Use ordinary `Engine.Run` with per-request `OperationTimeout` and `OnPrepared`
|
||||
where explicit preparation currently serves only to inspect and execute the same
|
||||
request. Map the resolved Audita operation timeout onto each proposal, validation,
|
||||
and repair operation. Keep provider, transport-retry, structural-repair, and
|
||||
correction-revision budgets distinct.
|
||||
|
||||
Use `ErrOperationTimeout` to identify the selected local budget, while retaining
|
||||
Audita's parent-cancellation and recoverable-provider-failure handling. Do not
|
||||
classify every wrapped preparation error as fatal before considering its timeout
|
||||
identity. Ordinary configuration failures remain fatal. If split preparation is
|
||||
still needed for a concrete consumer, one caller context must cover both calls
|
||||
when a total preparation-plus-execution deadline is required; the upstream
|
||||
prepared-handle duration starts only at execution.
|
||||
|
||||
`OnPrepared` receives the actual detached snapshot. Retain the metadata needed
|
||||
for attribution and credential-source redaction, and capture content for optional
|
||||
debug evidence. Keep the callback short; application validation belongs in the
|
||||
next-turn callback after admission is released. Notification may be absent on
|
||||
preparation/admission failure or cancellation before delivery. Reports must not
|
||||
invent missing snapshots. Optional debug write failures remain nonfatal.
|
||||
|
||||
### Conversations and correction repair
|
||||
|
||||
Capture the proposal exchange when repair needs it. Every rejected candidate's
|
||||
branch starts from that same immutable exchange; subsequent turns remain private
|
||||
to that branch. Follow-ups select the module's repair prompt, role profile,
|
||||
variables, and repair output contract through ordinary PromptKit requests.
|
||||
Retain native operator prompt overrides and the existing domain prompt content.
|
||||
|
||||
Use one coordinator per actionable correction. Its initial callback uses existing
|
||||
rejection findings. Each newly delivered response must pass structural validation
|
||||
before decoding; captured content alone does not imply validity. A decoded
|
||||
revision is fully revalidated before acceptance. Withdrawal, unavailable results,
|
||||
and accepted revisions remain Audita outcomes; a normal coordinator stop is not
|
||||
itself an approval.
|
||||
|
||||
The last permitted response follows the same assessment path. Further requested
|
||||
work beyond the budget maps to exhaustion. Skip coordination when Audita's
|
||||
correction-repair setting disables it: upstream's zero budget still invokes an
|
||||
initial callback. Cancellation's `Undelivered` response is diagnostic evidence,
|
||||
not an approved correction, and must not trigger model work after cancellation.
|
||||
Recoverable branch failures preserve independent approvals; fatal failures and
|
||||
caller cancellation retain Audita's cancel-and-join behavior.
|
||||
|
||||
### Execution groups
|
||||
|
||||
Use native groups as the source of aggregate provider-attempt, retry, repair,
|
||||
and available token totals. Explicitly select membership for proposal calls,
|
||||
coordinated continuations, and validator calls. Coordinator observers cover its
|
||||
own turns; nested validators require their own observer selection.
|
||||
|
||||
Maintain an invocation group for combined execution totals. Add narrower groups
|
||||
only for accounting views Audita consumes. A shared initial proposal run belongs
|
||||
once to the invocation total; replaying its history does not enroll it again.
|
||||
Each new call's reported history tokens still count. Overlapping group totals
|
||||
must not be summed as disjoint work.
|
||||
|
||||
Preserve upstream usage presence/completeness and terminal coverage separately.
|
||||
Keep domain logical-call records and deterministic presentation where needed;
|
||||
their count need not equal observed run membership. Group summaries do not replace
|
||||
Audita's errors/recoveries, validation decisions, or per-operation failure facts.
|
||||
|
||||
## Delivery and Completion
|
||||
|
||||
Implement in ordered, reviewable stages: dependency and execution adapter;
|
||||
captured conversations and coordinator; execution groups; cleanup, integration
|
||||
verification, and canonical documentation. All stages belong to this adoption.
|
||||
|
||||
Completion requires removal of local exchange/replay history, prefix checks,
|
||||
repair turn-budget loops, and duplicate terminal aggregation where replaced by
|
||||
PromptKit. Retain local behavior only where it has a concrete Audita domain or
|
||||
presentation responsibility. Preserve transcript correctness, repair prompts,
|
||||
validation semantics, deterministic reports, standard inspection commands, and
|
||||
operator configuration unless a deliberate change is documented and tested.
|
||||
|
||||
No product-level open decisions remain. Exact internal bridge signatures are
|
||||
implementation choices governed by this ownership split. Future validator
|
||||
decision recovery and new scheduling behavior remain in the
|
||||
[deferred work catalog](future.md).
|
||||
4
go.mod
4
go.mod
@@ -3,11 +3,12 @@ module gitea.maximumdirect.net/eric/audita/v3
|
||||
go 1.27.0
|
||||
|
||||
require (
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
go.maximumdirect.net/ckconfig v0.7.0
|
||||
go.maximumdirect.net/ckprompt v0.7.0
|
||||
go.maximumdirect.net/commandkit v0.6.0
|
||||
go.maximumdirect.net/configkit v1.2.0
|
||||
go.maximumdirect.net/promptkit v0.21.0
|
||||
go.maximumdirect.net/promptkit v0.23.0
|
||||
go.maximumdirect.net/releasekit v0.9.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
@@ -16,7 +17,6 @@ require (
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/kr/pretty v0.3.1 // indirect
|
||||
github.com/rogpeppe/go-internal v1.14.1 // indirect
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
|
||||
github.com/spf13/cobra v1.10.1 // indirect
|
||||
github.com/spf13/pflag v1.0.9 // indirect
|
||||
github.com/yuin/goldmark/v2 v2.1.6 // indirect
|
||||
|
||||
4
go.sum
4
go.sum
@@ -32,8 +32,8 @@ go.maximumdirect.net/commandkit v0.6.0 h1:4PHhMIlZuuvreIxZhUazeweg6zoYixEvhM9LJn
|
||||
go.maximumdirect.net/commandkit v0.6.0/go.mod h1:ZS98DewKNVBygKVDoh5MFD09CnPlP37dS9pVsID5Eoc=
|
||||
go.maximumdirect.net/configkit v1.2.0 h1:c4tDJ6Xeh/Jg90UZplLxsnf500lvrZBVj0E3i1ysir8=
|
||||
go.maximumdirect.net/configkit v1.2.0/go.mod h1:s1zLJ9GGB/1Q4a+aOHMLISonJxWFgNWpA6SpPlOafeA=
|
||||
go.maximumdirect.net/promptkit v0.21.0 h1:Ws4y28lnW+lIXAq4tGhXVFnQV/6AqkDChhg8knIZFrU=
|
||||
go.maximumdirect.net/promptkit v0.21.0/go.mod h1:pi9acdCi80HWyzY+XYwRvMEoCiTcDWKyeBh3mzHdIFo=
|
||||
go.maximumdirect.net/promptkit v0.23.0 h1:Q17wmRHWSznc1yW1MuB7dxuT3KVYRCzHI9pKBzpXhq0=
|
||||
go.maximumdirect.net/promptkit v0.23.0/go.mod h1:pi9acdCi80HWyzY+XYwRvMEoCiTcDWKyeBh3mzHdIFo=
|
||||
go.maximumdirect.net/releasekit v0.9.0 h1:sK+Aoj6gS6ATU/1reocjwBekp4YETYENwkcX5ti6mrE=
|
||||
go.maximumdirect.net/releasekit v0.9.0/go.mod h1:QE4BLrAhVJTzi+/YTcVl21K/qZp9ojRLg5kuoy3Zb44=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -47,13 +46,20 @@ func (r *Runtime) NewRoleClient(role, profileID string, settings config.LLM, hoo
|
||||
return &RoleClient{runtime: r, role: role, profileID: profileID, settings: settings, hooks: hooks}, nil
|
||||
}
|
||||
|
||||
func (c *RoleClient) CompleteStructured(parent context.Context, req structuredoutput.Request, out any) (response structuredoutput.Response, err error) {
|
||||
ctx, cancel := context.WithTimeout(parent, c.settings.OperationTimeout)
|
||||
defer cancel()
|
||||
type pendingCall struct {
|
||||
request pk.RunRequest
|
||||
finish func(*pk.RunResult, error) (structuredoutput.Response, error)
|
||||
}
|
||||
|
||||
func (c *RoleClient) prepareCall(parent context.Context, req structuredoutput.Request, out any, observer pk.ExecutionObserver) (*pendingCall, error) {
|
||||
var response structuredoutput.Response
|
||||
var err error
|
||||
var finishHooks []func()
|
||||
ctx := parent
|
||||
var terminalMu sync.Mutex
|
||||
var terminal pk.ExecutionEvent
|
||||
terminalAvailable := false
|
||||
defer func() {
|
||||
finishHooks = append(finishHooks, func() {
|
||||
terminalMu.Lock()
|
||||
if terminalAvailable {
|
||||
response.Execution = outcomes.ExecutionCounts{
|
||||
@@ -69,7 +75,7 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou
|
||||
}
|
||||
}
|
||||
terminalMu.Unlock()
|
||||
}()
|
||||
})
|
||||
var record structuredoutput.CallRecord
|
||||
if c.hooks.OnCall != nil || c.hooks.OnEventWithCall != nil {
|
||||
record.CallID = c.runtime.nextCallID.Add(1)
|
||||
@@ -81,7 +87,7 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou
|
||||
record.Started = time.Now().UTC()
|
||||
record.Prompt = structuredoutput.PromptMetadata{PromptID: req.PromptID, ProfileID: c.profileID}
|
||||
onCall(record)
|
||||
defer func() {
|
||||
finishHooks = append(finishHooks, func() {
|
||||
record.Finished = time.Now().UTC()
|
||||
if err != nil {
|
||||
var failure *structuredoutput.Failure
|
||||
@@ -92,123 +98,120 @@ func (c *RoleClient) CompleteStructured(parent context.Context, req structuredou
|
||||
}
|
||||
}
|
||||
onCall(record)
|
||||
}()
|
||||
})
|
||||
}
|
||||
finish := func() {
|
||||
for i := len(finishHooks) - 1; i >= 0; i-- {
|
||||
finishHooks[i]()
|
||||
}
|
||||
}
|
||||
fail := func(failure error) (*pendingCall, error) {
|
||||
err = failure
|
||||
finish()
|
||||
return nil, failure
|
||||
}
|
||||
if parent.Err() != nil {
|
||||
return response, classifiedFailure(parent, ctx, parent.Err(), nil)
|
||||
return fail(classifiedFailure(parent, ctx, parent.Err(), nil))
|
||||
}
|
||||
if err = validateOutputTarget(out); err != nil {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "invalid structured output target", Cause: err}
|
||||
return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "invalid structured output target", Cause: err})
|
||||
}
|
||||
contract, contractErr := OutputContract(req.Schema, c.settings.RepairAttempts)
|
||||
if contractErr != nil {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "unknown structured response schema", Cause: contractErr}
|
||||
return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "unknown structured response schema", Cause: contractErr})
|
||||
}
|
||||
vars := prompts.WithSharedVariables(req.Variables)
|
||||
for name, value := range prompts.SharedVariables() {
|
||||
vars[name] = value
|
||||
}
|
||||
baseID, baseVars := req.PromptID, vars
|
||||
var suffix []pk.RenderedMessage
|
||||
if req.Replay != nil {
|
||||
if req.Replay.Exchange == nil {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "repair exchange is unavailable"}
|
||||
var exchange *pk.Exchange
|
||||
if req.ContinueFrom != nil {
|
||||
var ok bool
|
||||
exchange, ok = req.ContinueFrom.State.(*pk.Exchange)
|
||||
if !ok || exchange == nil {
|
||||
return fail(&structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonSetupFailure, Message: "repair exchange is unavailable"})
|
||||
}
|
||||
instruction, renderErr := c.runtime.Engine.Prepare(ctx, pk.RunRequest{PromptID: req.PromptID, ProfileID: c.profileID, Vars: vars})
|
||||
if renderErr != nil {
|
||||
return response, classifiedFailure(parent, ctx, renderErr, nil)
|
||||
}
|
||||
response.RepairInstruction = convertMessages(instruction.Messages)
|
||||
exchange := req.Replay.Exchange
|
||||
baseID, baseVars = exchange.PromptID, exchange.Variables
|
||||
suffix = append(suffix, pk.RenderedMessage{Role: pk.RoleAssistant, Content: exchange.FinalContent})
|
||||
for _, message := range req.Replay.Turns {
|
||||
native := pk.RenderedMessage{Role: message.Role, Content: message.Content}
|
||||
if cache := message.CacheControl; cache != nil {
|
||||
native.CacheControl = &pk.CacheControl{Type: pk.CacheControlType(cache.Type), TTL: cache.TTL}
|
||||
}
|
||||
suffix = append(suffix, native)
|
||||
}
|
||||
suffix = append(suffix, instruction.Messages...)
|
||||
}
|
||||
handle, prepErr := c.runtime.Engine.PrepareExecution(ctx, pk.RunRequest{
|
||||
PromptID: baseID, ProfileID: c.profileID, Vars: baseVars, AppendedMessages: suffix,
|
||||
runRequest := pk.RunRequest{
|
||||
PromptID: req.PromptID, ProfileID: c.profileID, Vars: vars,
|
||||
CaptureExchange: req.CaptureExchange, ContinueFrom: exchange,
|
||||
TransientRetries: &c.settings.TransientRetries,
|
||||
Validation: contract,
|
||||
Observer: func(event pk.ExecutionEvent) {
|
||||
Observer: pk.ComposeObservers(observer, func(event pk.ExecutionEvent) {
|
||||
if event.Kind == pk.ExecutionFinished {
|
||||
terminalMu.Lock()
|
||||
terminal = event
|
||||
terminalAvailable = true
|
||||
terminalMu.Unlock()
|
||||
}
|
||||
if c.hooks.OnEvent != nil {
|
||||
c.hooks.OnEvent(event)
|
||||
}
|
||||
if c.hooks.OnEventWithCall != nil {
|
||||
c.hooks.OnEventWithCall(record.CallID, record.Context, event)
|
||||
}
|
||||
},
|
||||
})
|
||||
if prepErr != nil {
|
||||
return response, classifiedFailure(parent, ctx, prepErr, nil)
|
||||
}),
|
||||
}
|
||||
defer handle.Discard()
|
||||
details := handle.Details()
|
||||
if req.Replay != nil {
|
||||
prefix := req.Replay.Exchange.Messages
|
||||
if len(details.Messages) < len(prefix) || !reflect.DeepEqual(convertMessages(details.Messages[:len(prefix)]), prefix) {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "repair conversation prefix changed"}
|
||||
var details pk.PreparedRun
|
||||
capturePrepared := func(_ context.Context, prepared pk.PreparedRun) {
|
||||
details = prepared
|
||||
if name := details.EffectiveModelParams.APIKeyEnv; name != "" {
|
||||
c.credentialMu.Lock()
|
||||
if c.credentialNames == nil {
|
||||
c.credentialNames = make(map[string]struct{})
|
||||
}
|
||||
c.credentialNames[name] = struct{}{}
|
||||
c.credentialMu.Unlock()
|
||||
}
|
||||
response.Prompt = promptMetadata(details)
|
||||
record.Prompt = response.Prompt
|
||||
response.Model = details.EffectiveModelParams.Model
|
||||
response.Prepared = preparedSnapshot(details)
|
||||
record.Prepared = response.Prepared
|
||||
}
|
||||
if name := details.EffectiveModelParams.APIKeyEnv; name != "" {
|
||||
c.credentialMu.Lock()
|
||||
if c.credentialNames == nil {
|
||||
c.credentialNames = make(map[string]struct{})
|
||||
runRequest.OperationTimeout = c.settings.OperationTimeout
|
||||
runRequest.OnPrepared = capturePrepared
|
||||
return &pendingCall{request: runRequest, finish: func(result *pk.RunResult, runErr error) (completed structuredoutput.Response, finalErr error) {
|
||||
defer func() { err = finalErr; finish(); completed = response }()
|
||||
|
||||
if result != nil {
|
||||
response.FinalContent = result.RawOutput
|
||||
record.FinalContent = result.RawOutput
|
||||
response.Model = result.ModelName
|
||||
response.PromptTokens = result.Usage.PromptTokens
|
||||
response.CompletionTokens = result.Usage.CompletionTokens
|
||||
response.TotalTokens = result.Usage.TotalTokens
|
||||
}
|
||||
c.credentialNames[name] = struct{}{}
|
||||
c.credentialMu.Unlock()
|
||||
}
|
||||
response.Prompt = promptMetadata(details)
|
||||
record.Prompt = response.Prompt
|
||||
response.Model = details.EffectiveModelParams.Model
|
||||
response.Prepared = preparedSnapshot(details)
|
||||
record.Prepared = response.Prepared
|
||||
result, runErr := c.runtime.Engine.RunPrepared(ctx, handle)
|
||||
if result != nil {
|
||||
response.FinalContent = result.RawOutput
|
||||
record.FinalContent = result.RawOutput
|
||||
response.Model = result.ModelName
|
||||
response.PromptTokens = result.Usage.PromptTokens
|
||||
response.CompletionTokens = result.Usage.CompletionTokens
|
||||
response.TotalTokens = result.Usage.TotalTokens
|
||||
}
|
||||
if runErr != nil {
|
||||
return response, classifiedFailure(parent, ctx, runErr, result)
|
||||
}
|
||||
if result == nil {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "structured model call returned no result"}
|
||||
}
|
||||
if !result.Validation.IsValid {
|
||||
return response, classifiedFailure(parent, ctx, nil, result)
|
||||
}
|
||||
if decodeErr := json.Unmarshal([]byte(result.RawOutput), out); decodeErr != nil {
|
||||
var typeError *json.UnmarshalTypeError
|
||||
if errors.As(decodeErr, &typeError) {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonInvalidOutput, Message: "structured output contains an unusable value", Cause: decodeErr}
|
||||
if runErr != nil {
|
||||
return response, classifiedFailure(parent, ctx, runErr, result)
|
||||
}
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "validated structured output could not be decoded", Cause: decodeErr}
|
||||
}
|
||||
if req.CaptureExchange && req.Replay == nil {
|
||||
frozen := make(map[string]string, len(vars))
|
||||
for key, value := range vars {
|
||||
frozen[key] = value
|
||||
if result == nil {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "structured model call returned no result"}
|
||||
}
|
||||
response.Exchange = &structuredoutput.Exchange{PromptID: req.PromptID, Variables: frozen, Messages: convertMessages(details.Messages), FinalContent: result.RawOutput}
|
||||
if !result.Validation.IsValid {
|
||||
return response, classifiedFailure(parent, ctx, nil, result)
|
||||
}
|
||||
if decodeErr := json.Unmarshal([]byte(result.RawOutput), out); decodeErr != nil {
|
||||
var typeError *json.UnmarshalTypeError
|
||||
if errors.As(decodeErr, &typeError) {
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonInvalidOutput, Message: "structured output contains an unusable value", Cause: decodeErr}
|
||||
}
|
||||
return response, &structuredoutput.Failure{Class: structuredoutput.FailureFatal, Reason: outcomes.ReasonInternalFailure, Message: "validated structured output could not be decoded", Cause: decodeErr}
|
||||
}
|
||||
if result.Exchange != nil {
|
||||
response.Exchange = &structuredoutput.Exchange{State: result.Exchange}
|
||||
}
|
||||
|
||||
response.Content = json.RawMessage(result.RawOutput)
|
||||
response.Provider = "openai-compatible"
|
||||
return response, nil
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func (c *RoleClient) CompleteStructured(ctx context.Context, req structuredoutput.Request, out any) (structuredoutput.Response, error) {
|
||||
call, err := c.prepareCall(ctx, req, out, c.hooks.OnEvent)
|
||||
if err != nil {
|
||||
return structuredoutput.Response{}, err
|
||||
}
|
||||
response.Content = json.RawMessage(result.RawOutput)
|
||||
response.Provider = "openai-compatible"
|
||||
return response, nil
|
||||
result, err := c.runtime.Engine.Run(ctx, call.request)
|
||||
return call.finish(result, err)
|
||||
}
|
||||
|
||||
// OutputContract maps an Audita domain schema to the native request contract.
|
||||
@@ -231,6 +234,12 @@ func classifiedFailure(parent, local context.Context, cause error, result *pk.Ru
|
||||
failure.Cause = parent.Err()
|
||||
return failure
|
||||
}
|
||||
if errors.Is(cause, pk.ErrOperationTimeout) {
|
||||
failure.Class = structuredoutput.FailureRecoverable
|
||||
failure.Reason = outcomes.ReasonOperationTimeout
|
||||
failure.Message = "model operation timed out"
|
||||
return failure
|
||||
}
|
||||
if isSetupFailure(cause) {
|
||||
failure.Reason = outcomes.ReasonSetupFailure
|
||||
failure.Message = "structured model setup failed"
|
||||
@@ -298,10 +307,7 @@ func promptMetadata(details pk.PreparedRun) structuredoutput.PromptMetadata {
|
||||
}
|
||||
|
||||
func preparedSnapshot(details pk.PreparedRun) *structuredoutput.PreparedSnapshot {
|
||||
messages := make([]structuredoutput.Message, len(details.Messages))
|
||||
for i, message := range details.Messages {
|
||||
messages[i] = structuredoutput.Message{Role: string(message.Role), Content: message.Content}
|
||||
}
|
||||
messages := convertMessages(details.Messages)
|
||||
return &structuredoutput.PreparedSnapshot{Messages: messages, Prompt: promptMetadata(details)}
|
||||
}
|
||||
|
||||
|
||||
@@ -482,3 +482,14 @@ func TestPreparedAdapterUnknownValidationFailureRemainsFatal(t *testing.T) {
|
||||
t.Fatalf("unknown validation failure: %#v", failure)
|
||||
}
|
||||
}
|
||||
|
||||
// Preparation errors can carry the native operation deadline as their cause.
|
||||
// That deadline is recoverable even when the outer category is a setup phase.
|
||||
func TestOperationTimeoutDuringPreparationIsRecoverable(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
cause := errors.Join(pk.ErrPromptLoad, pk.ErrOperationTimeout, context.DeadlineExceeded)
|
||||
failure := classifiedFailure(ctx, ctx, cause, nil)
|
||||
if failure.Class != structuredoutput.FailureRecoverable || failure.Reason != outcomes.ReasonOperationTimeout || !errors.Is(failure, pk.ErrOperationTimeout) {
|
||||
t.Fatalf("failure = %#v", failure)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,141 +19,9 @@ import (
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/prompts"
|
||||
ck "go.maximumdirect.net/configkit"
|
||||
pk "go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestNativeConversationReplayWithRepairContract(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
files := map[string]string{
|
||||
"config.yml": "version: 3\n",
|
||||
"prompts.yml": `profiles:
|
||||
- id: proposal-profile
|
||||
endpoint: https://example.invalid/v1
|
||||
model: proposal-model
|
||||
- id: repair-profile
|
||||
endpoint: https://example.invalid/v1
|
||||
model: repair-model
|
||||
prompts:
|
||||
- id: proposal-fixture
|
||||
messages:
|
||||
- role: system
|
||||
content: 'System {{.Subject}}'
|
||||
- role: user
|
||||
content: 'Proposal {{.Subject}}'
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: correction_set.v1.json
|
||||
- id: repair-fixture
|
||||
messages:
|
||||
- role: user
|
||||
content: 'Repair {{.Finding}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`,
|
||||
}
|
||||
for name, content := range files {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
loaded, err := config.Load(ck.Input{Source: ck.File(filepath.Join(dir, "config.yml"))}, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var requests []struct {
|
||||
Model string `json:"model"`
|
||||
Messages []struct{ Role, Content string } `json:"messages"`
|
||||
ResponseFormat map[string]any `json:"response_format"`
|
||||
}
|
||||
outputs := []string{`{"corrections":[]}`, `{"proposal_ref":1,"action":"revise","correction":{"id":1,"original_text":" ","corrected_text":""}}`, `{"proposal_ref":1,"action":"withdraw","correction":null}`}
|
||||
transport := transportFunc(func(r *http.Request) (*http.Response, error) {
|
||||
var request struct {
|
||||
Model string `json:"model"`
|
||||
Messages []struct{ Role, Content string } `json:"messages"`
|
||||
ResponseFormat map[string]any `json:"response_format"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
requests = append(requests, request)
|
||||
content, _ := json.Marshal(outputs[len(requests)-1])
|
||||
body := fmt.Sprintf(`{"model":%q,"choices":[{"message":{"content":%s}}]}`, request.Model, content)
|
||||
return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body)), Request: r}, nil
|
||||
})
|
||||
runtime, err := NewRuntime(context.Background(), loaded, &http.Client{Transport: transport})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
base := pk.RunRequest{PromptID: "proposal-fixture", ProfileID: "proposal-profile", Vars: map[string]string{"Subject": "source"}}
|
||||
original, err := runtime.Engine.PrepareExecution(ctx, base)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prefix := original.Details().Messages
|
||||
initial, err := runtime.Engine.RunPrepared(ctx, original)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
suffix := []pk.RenderedMessage{{Role: pk.RoleAssistant, Content: initial.RawOutput}}
|
||||
contract, err := OutputContract(responseschema.CorrectionRepairKey, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, finding := range []string{"first", "second"} {
|
||||
instruction, err := runtime.Engine.Prepare(ctx, pk.RunRequest{PromptID: "repair-fixture", ProfileID: "repair-profile", Vars: map[string]string{"Finding": finding}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
suffix = append(suffix, instruction.Messages...)
|
||||
req := base
|
||||
req.ProfileID = "repair-profile"
|
||||
req.AppendedMessages = suffix
|
||||
req.Validation = contract
|
||||
handle, err := runtime.Engine.PrepareExecution(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
details := handle.Details()
|
||||
if !reflect.DeepEqual(details.Messages[:len(prefix)], prefix) || !reflect.DeepEqual(details.Messages[len(prefix):], suffix) {
|
||||
t.Fatalf("prepared conversation = %#v", details.Messages)
|
||||
}
|
||||
result, err := runtime.Engine.RunPrepared(ctx, handle)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !result.Validation.IsValid {
|
||||
t.Fatalf("invalid result: %+v", result.Validation)
|
||||
}
|
||||
suffix = append(suffix, pk.RenderedMessage{Role: pk.RoleAssistant, Content: result.RawOutput})
|
||||
}
|
||||
if len(requests) != 3 || requests[0].Model != "proposal-model" || requests[1].Model != "repair-model" || requests[2].Model != "repair-model" {
|
||||
t.Fatalf("models: %+v", requests)
|
||||
}
|
||||
for i, wantLen := range []int{2, 4, 6} {
|
||||
req := requests[i]
|
||||
if len(req.Messages) != wantLen {
|
||||
t.Fatalf("request %d messages: %+v", i, req.Messages)
|
||||
}
|
||||
for j, message := range req.Messages {
|
||||
if message.Role != string(append(prefix, suffix...)[j].Role) || message.Content != append(prefix, suffix...)[j].Content {
|
||||
t.Fatalf("request %d message %d: %+v", i, j, message)
|
||||
}
|
||||
}
|
||||
encoded, _ := json.Marshal(req.ResponseFormat)
|
||||
expected := "corrections"
|
||||
if i > 0 {
|
||||
expected = "proposal_ref"
|
||||
}
|
||||
if !strings.Contains(string(encoded), expected) {
|
||||
t.Fatalf("request %d contract: %s", i, encoded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) {
|
||||
func TestRoleClientContinuesIndependentRepairConversations(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for name, content := range map[string]string{
|
||||
"config.yml": "version: 3\n",
|
||||
@@ -207,17 +75,13 @@ func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) {
|
||||
t.Fatalf("capture: %+v %v", initial, err)
|
||||
}
|
||||
vars["SectionJSON"] = "changed"
|
||||
replay := &structuredoutput.Replay{Exchange: initial.Exchange}
|
||||
exchange := initial.Exchange
|
||||
for _, finding := range []string{"first", "second"} {
|
||||
result, err := repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": finding}, Replay: replay}, &map[string]any{})
|
||||
result, err := repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": finding}, ContinueFrom: exchange}, &map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(result.RepairInstruction) != 1 {
|
||||
t.Fatalf("instruction: %+v", result)
|
||||
}
|
||||
replay.Turns = append(replay.Turns, result.RepairInstruction...)
|
||||
replay.Turns = append(replay.Turns, structuredoutput.Message{Role: "assistant", Content: result.FinalContent})
|
||||
exchange = result.Exchange
|
||||
}
|
||||
if !reflect.DeepEqual(models, []string{"proposal", "repair", "repair"}) {
|
||||
t.Fatalf("models = %v", models)
|
||||
@@ -235,21 +99,18 @@ func TestRoleClientCapturesAndReplaysRepairConversation(t *testing.T) {
|
||||
!strings.Contains(string(first.Content), `"text":"first"`) || !reflect.DeepEqual(first, messages[2][len(prefix)+1]) {
|
||||
t.Fatalf("cached instruction changed during replay: first=%s replay=%s", first.Content, messages[2][len(prefix)+1].Content)
|
||||
}
|
||||
_, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "independent"}, Replay: &structuredoutput.Replay{Exchange: initial.Exchange}}, &map[string]any{})
|
||||
_, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "independent"}, ContinueFrom: initial.Exchange}, &map[string]any{})
|
||||
if err != nil || len(messages) != 4 || len(messages[3]) != len(prefix)+2 || !strings.Contains(string(messages[3][len(prefix)+1].Content), "independent") {
|
||||
t.Fatalf("independent branch: %v %+v", err, messages)
|
||||
}
|
||||
bad := *initial.Exchange
|
||||
bad.Messages = append([]structuredoutput.Message(nil), bad.Messages...)
|
||||
bad.Messages[0].Content = "changed"
|
||||
_, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "failure"}, Replay: &structuredoutput.Replay{Exchange: &bad}}, &map[string]any{})
|
||||
_, err = repair.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "failure"}, ContinueFrom: &structuredoutput.Exchange{}}, &map[string]any{})
|
||||
var failure *structuredoutput.Failure
|
||||
if !errors.As(err, &failure) || failure.Class != structuredoutput.FailureFatal || len(messages) != 4 {
|
||||
t.Fatalf("prefix failure: %v, calls %d", err, len(messages))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepairReplayUsesReplacementAndCountsStructuralRepair(t *testing.T) {
|
||||
func TestRepairContinuationUsesReplacementAndCountsStructuralRepair(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
files := map[string]string{
|
||||
"config.yml": "version: 3\n",
|
||||
@@ -291,11 +152,11 @@ func TestRepairReplayUsesReplacementAndCountsStructuralRepair(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := client.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "fixture"}, Replay: &structuredoutput.Replay{Exchange: base.Exchange}}, &map[string]any{})
|
||||
result, err := client.CompleteStructured(context.Background(), structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "fixture"}, ContinueFrom: base.Exchange}, &map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(requests) != 3 || result.Execution.ProviderAttempts != 2 || result.Execution.StructuralRepairs != 1 || result.FinalContent != outputs[2] || len(result.RepairInstruction) != 1 || result.RepairInstruction[0].Content != "Replace fixture" {
|
||||
if len(requests) != 3 || result.Execution.ProviderAttempts != 2 || result.Execution.StructuralRepairs != 1 || result.FinalContent != outputs[2] {
|
||||
t.Fatalf("result=%+v calls=%d", result, len(requests))
|
||||
}
|
||||
}
|
||||
|
||||
63
internal/framework/llm/coordinate.go
Normal file
63
internal/framework/llm/coordinate.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput"
|
||||
pk "go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
var _ structuredoutput.Coordinator = (*RoleClient)(nil)
|
||||
|
||||
// Coordinate shares request construction and finalization with ordinary calls.
|
||||
// PromptKit alone advances history and consumes the additional-turn budget.
|
||||
func (c *RoleClient) Coordinate(ctx context.Context, start *structuredoutput.Exchange, budget int, next func(context.Context, structuredoutput.Turn) (*structuredoutput.Request, error)) (bool, error) {
|
||||
var exchange *pk.Exchange
|
||||
if start != nil {
|
||||
exchange, _ = start.State.(*pk.Exchange)
|
||||
}
|
||||
var pending *pendingCall
|
||||
result, err := c.runtime.Engine.Coordinate(ctx, pk.CoordinateRequest{
|
||||
Start: exchange, AdditionalTurns: budget, Observer: c.hooks.OnEvent,
|
||||
Next: func(ctx context.Context, turn pk.ConversationTurn) (*pk.RunRequest, error) {
|
||||
var delivered structuredoutput.Turn
|
||||
if turn.Result != nil {
|
||||
response, failure := pending.finish(turn.Result, nil)
|
||||
pending = nil
|
||||
delivered = structuredoutput.Turn{Response: &response, Err: failure}
|
||||
}
|
||||
selected, err := next(ctx, delivered)
|
||||
if err != nil || selected == nil {
|
||||
return nil, err
|
||||
}
|
||||
// Exhaustion needs a requested turn, but must not allocate a logical call.
|
||||
if turn.RemainingTurns == 0 {
|
||||
return &pk.RunRequest{}, nil
|
||||
}
|
||||
var output json.RawMessage
|
||||
pending, err = c.prepareCall(ctx, *selected, &output, nil)
|
||||
if err != nil {
|
||||
_, domainErr := next(ctx, structuredoutput.Turn{Response: &structuredoutput.Response{}, Err: err})
|
||||
if domainErr != nil {
|
||||
return nil, domainErr
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return &pending.request, nil
|
||||
},
|
||||
})
|
||||
if pending != nil {
|
||||
// Execution errors and withheld responses do not enter Next. Finalize their
|
||||
// diagnostic evidence and let the domain record the terminal failure once.
|
||||
response, failure := pending.finish(result.Undelivered, err)
|
||||
_, domainErr := next(ctx, structuredoutput.Turn{Response: &response, Err: failure})
|
||||
if domainErr != nil {
|
||||
return false, domainErr
|
||||
}
|
||||
if structuredoutput.IsRecoverable(failure) {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return result.Outcome == pk.CoordinationExhausted, err
|
||||
}
|
||||
141
internal/framework/llm/coordinate_test.go
Normal file
141
internal/framework/llm/coordinate_test.go
Normal file
@@ -0,0 +1,141 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/outcomes"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/responseschema"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/structuredoutput"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/prompts"
|
||||
pk "go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestCoordinatorFinalTurnFailuresAndAccounting(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
reply reply
|
||||
again bool
|
||||
reason outcomes.ReasonCode
|
||||
}{
|
||||
{name: "last turn assessed", reply: reply{200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}},
|
||||
{name: "exhaustion", reply: reply{200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}, again: true},
|
||||
{name: "invalid structure", reply: reply{200, `{"wrong":true}`}, reason: outcomes.ReasonInvalidOutputExhausted},
|
||||
{name: "provider error", reply: reply{400, ""}, reason: outcomes.ReasonProviderFailed},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
runtime, count, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, tc.reply}, 0, 0, nil)
|
||||
collector := NewObservationCollector()
|
||||
client, err := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, collector.Hooks())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := adapterRequest(responseschema.CorrectionSetKey)
|
||||
req.CaptureExchange = true
|
||||
base, err := client.CompleteStructured(context.Background(), req, &map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deliveries := 0
|
||||
exhausted, err := client.Coordinate(context.Background(), base.Exchange, 1, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) {
|
||||
if turn.Response != nil {
|
||||
deliveries++
|
||||
if tc.reason != "" {
|
||||
var failure *structuredoutput.Failure
|
||||
if !errors.As(turn.Err, &failure) || failure.Reason != tc.reason {
|
||||
t.Fatalf("failure=%v", turn.Err)
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
if turn.Err != nil || len(turn.Response.Content) == 0 {
|
||||
t.Fatalf("turn=%+v", turn)
|
||||
}
|
||||
if !tc.again {
|
||||
return nil, nil
|
||||
}
|
||||
}
|
||||
return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil
|
||||
})
|
||||
if err != nil || exhausted != tc.again || deliveries != 1 || count.Load() != 2 {
|
||||
t.Fatalf("exhausted=%v delivered=%d calls=%d err=%v", exhausted, deliveries, count.Load(), err)
|
||||
}
|
||||
snapshot := collector.Snapshot()
|
||||
summary := Summarize(snapshot)
|
||||
if len(snapshot.Calls) != 2 || summary.ProviderAttempts != 2 || !summary.TerminalCoverageComplete {
|
||||
t.Fatalf("summary=%+v calls=%+v", summary, snapshot.Calls)
|
||||
}
|
||||
for _, call := range snapshot.Calls {
|
||||
if call.Finished.IsZero() {
|
||||
t.Fatal("unfinished logical call")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoordinatorCanceledDeliveryRetainsEvidence(t *testing.T) {
|
||||
runtime, _, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, {200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}}, 0, 0, nil)
|
||||
collector := NewObservationCollector()
|
||||
proposal, _ := runtime.NewRoleClient("proposal", "base", runtime.Config.LLM, collector.Hooks())
|
||||
req := adapterRequest(responseschema.CorrectionSetKey)
|
||||
req.CaptureExchange = true
|
||||
base, err := proposal.CompleteStructured(context.Background(), req, &map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
hooks := collector.Hooks()
|
||||
hooks.OnEvent = func(e pk.ExecutionEvent) {
|
||||
if e.Kind == pk.ExecutionFinished {
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
client, _ := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, hooks)
|
||||
deliveries := 0
|
||||
_, err = client.Coordinate(ctx, base.Exchange, 1, func(_ context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) {
|
||||
if turn.Response != nil {
|
||||
deliveries++
|
||||
if !errors.Is(turn.Err, context.Canceled) || turn.Response.Prepared == nil || turn.Response.FinalContent == "" || len(turn.Response.Content) != 0 {
|
||||
t.Fatalf("canceled turn=%+v", turn)
|
||||
}
|
||||
return nil, turn.Err
|
||||
}
|
||||
return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil
|
||||
})
|
||||
if !errors.Is(err, context.Canceled) || deliveries != 1 {
|
||||
t.Fatalf("deliveries=%d err=%v", deliveries, err)
|
||||
}
|
||||
if got := Summarize(collector.Snapshot()); got.ProviderAttempts != 2 || !got.TerminalCoverageComplete {
|
||||
t.Fatalf("summary=%+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoordinatorReleasesAdmissionBeforeNestedValidation(t *testing.T) {
|
||||
runtime, count, _ := adapterFixture(t, []reply{{200, `{"corrections":[]}`}, {200, `{"proposal_ref":1,"action":"withdraw","correction":null}`}, {200, `{"validations":[]}`}}, 0, 0, nil)
|
||||
collector := NewObservationCollector()
|
||||
proposal, _ := runtime.NewRoleClient("proposal", "base", runtime.Config.LLM, collector.Hooks())
|
||||
repair, _ := runtime.NewRoleClient("repair", "base", runtime.Config.LLM, collector.Hooks())
|
||||
validation, _ := runtime.NewRoleClient("validation", "base", runtime.Config.LLM, collector.Hooks())
|
||||
req := adapterRequest(responseschema.CorrectionSetKey)
|
||||
req.CaptureExchange = true
|
||||
base, err := proposal.CompleteStructured(context.Background(), req, &map[string]any{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = repair.Coordinate(context.Background(), base.Exchange, 1, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) {
|
||||
if turn.Response != nil {
|
||||
if turn.Err != nil {
|
||||
return nil, turn.Err
|
||||
}
|
||||
_, err := validation.CompleteStructured(ctx, adapterRequest(responseschema.ValidatorDecisionSetKey), &map[string]any{})
|
||||
return nil, err
|
||||
}
|
||||
return &structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarRepair, Schema: responseschema.CorrectionRepairKey, Variables: map[string]string{"RepairRequestJSON": "{}"}}, nil
|
||||
})
|
||||
summary := Summarize(collector.Snapshot())
|
||||
if err != nil || count.Load() != 3 || summary.ProviderAttempts != 3 || summary.LogicalCalls != 3 || !summary.TerminalCoverageComplete {
|
||||
t.Fatalf("calls=%d summary=%+v err=%v", count.Load(), summary, err)
|
||||
}
|
||||
}
|
||||
@@ -115,7 +115,7 @@ func TestRoleClientExecutionBudgetsAndFailureClasses(t *testing.T) {
|
||||
capturing := request
|
||||
capturing.CaptureExchange = true
|
||||
response, err := client.CompleteStructured(context.Background(), capturing, &result)
|
||||
if tc.success && (response.Exchange == nil || response.Exchange.FinalContent != tc.outputs[len(tc.outputs)-1]) {
|
||||
if tc.success && (response.Exchange == nil || response.FinalContent != tc.outputs[len(tc.outputs)-1]) {
|
||||
t.Fatalf("captured final response: %+v", response.Exchange)
|
||||
}
|
||||
if (err == nil) != tc.success || calls != tc.wantCalls {
|
||||
|
||||
@@ -64,21 +64,24 @@ type LogicalCall struct {
|
||||
}
|
||||
|
||||
type ObservationSnapshot struct {
|
||||
Calls []LogicalCall `json:"calls"`
|
||||
Events []ExecutionObservation `json:"events"`
|
||||
Group *pk.ExecutionGroupSummary `json:"-"`
|
||||
Calls []LogicalCall `json:"calls"`
|
||||
Events []ExecutionObservation `json:"events"`
|
||||
}
|
||||
|
||||
type ExecutionSummary struct {
|
||||
LogicalCalls uint64 `json:"logical_calls"`
|
||||
ProviderAttempts uint64 `json:"provider_attempts"`
|
||||
TransientRetries uint64 `json:"transient_retries"`
|
||||
StructuralRepairs uint64 `json:"structural_repairs"`
|
||||
Usage Usage `json:"usage"`
|
||||
TerminalCoverageComplete bool `json:"terminal_coverage_complete"`
|
||||
LogicalCalls uint64 `json:"logical_calls"`
|
||||
ProviderAttempts uint64 `json:"provider_attempts"`
|
||||
TransientRetries uint64 `json:"transient_retries"`
|
||||
StructuralRepairs uint64 `json:"structural_repairs"`
|
||||
Usage Usage `json:"usage"`
|
||||
}
|
||||
|
||||
// ObservationCollector accepts short synchronous callbacks from parallel calls.
|
||||
// Snapshot should be taken after the runner has joined active section work.
|
||||
type ObservationCollector struct {
|
||||
group pk.ExecutionGroup
|
||||
mu sync.Mutex
|
||||
calls map[uint64]LogicalCall
|
||||
events []ExecutionObservation
|
||||
@@ -137,6 +140,7 @@ func (c *ObservationCollector) recordEvent(callID uint64, context structuredoutp
|
||||
if c == nil {
|
||||
return
|
||||
}
|
||||
c.group.Observe(event)
|
||||
count := func(value pk.ObservedTokenCount) TokenCount {
|
||||
return TokenCount{Value: value.Value, Known: value.Known, Complete: value.Complete}
|
||||
}
|
||||
@@ -176,52 +180,28 @@ func (c *ObservationCollector) Snapshot() ObservationSnapshot {
|
||||
}
|
||||
return snapshot.Events[i].Time.Before(snapshot.Events[j].Time)
|
||||
})
|
||||
group := c.group.Snapshot()
|
||||
snapshot.Group = &group
|
||||
return snapshot
|
||||
}
|
||||
|
||||
// Summarize counts terminal usage once per logical call. PromptKit's terminal
|
||||
// usage already aggregates its attempts, retries, and repairs.
|
||||
// Summarize delegates deduplication and terminal accounting to PromptKit. A
|
||||
// filtered diagnostic view selects its own group; overlapping views are not additive.
|
||||
func Summarize(snapshot ObservationSnapshot) ExecutionSummary {
|
||||
result := ExecutionSummary{LogicalCalls: uint64(len(snapshot.Calls))}
|
||||
terminal := make(map[uint64]ExecutionObservation)
|
||||
for _, event := range snapshot.Events {
|
||||
if event.Kind == string(pk.ExecutionFinished) {
|
||||
terminal[event.CallID] = event
|
||||
summary := snapshot.Group
|
||||
if summary == nil {
|
||||
var group pk.ExecutionGroup
|
||||
count := func(c TokenCount) pk.ObservedTokenCount {
|
||||
return pk.ObservedTokenCount{Value: c.Value, Known: c.Known, Complete: c.Complete}
|
||||
}
|
||||
}
|
||||
dispatched := false
|
||||
aggregate := func(dst *TokenCount, source TokenCount) {
|
||||
if source.Known {
|
||||
dst.Value += source.Value
|
||||
dst.Known = true
|
||||
for _, event := range snapshot.Events {
|
||||
group.Observe(pk.ExecutionEvent{Kind: pk.ExecutionEventKind(event.Kind), RunID: event.RunID, Attempts: event.Attempts, TransientRetries: event.TransientRetries, RepairCalls: event.StructuralRepairs, Usage: pk.ObservedTokenUsage{PromptTokens: count(event.Usage.Prompt), CompletionTokens: count(event.Usage.Completion), TotalTokens: count(event.Usage.Total), CachedTokens: count(event.Usage.Cached), CacheWriteTokens: count(event.Usage.CacheWrite)}})
|
||||
}
|
||||
dst.Complete = dst.Complete && source.Complete
|
||||
value := group.Snapshot()
|
||||
summary = &value
|
||||
}
|
||||
result.Usage = Usage{Prompt: TokenCount{Complete: true}, Completion: TokenCount{Complete: true}, Total: TokenCount{Complete: true}, Cached: TokenCount{Complete: true}, CacheWrite: TokenCount{Complete: true}}
|
||||
for _, call := range snapshot.Calls {
|
||||
event, ok := terminal[call.CallID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
result.ProviderAttempts += event.Attempts
|
||||
result.TransientRetries += event.TransientRetries
|
||||
result.StructuralRepairs += event.StructuralRepairs
|
||||
if event.Attempts == 0 {
|
||||
continue
|
||||
}
|
||||
dispatched = true
|
||||
aggregate(&result.Usage.Prompt, event.Usage.Prompt)
|
||||
aggregate(&result.Usage.Completion, event.Usage.Completion)
|
||||
aggregate(&result.Usage.Total, event.Usage.Total)
|
||||
aggregate(&result.Usage.Cached, event.Usage.Cached)
|
||||
aggregate(&result.Usage.CacheWrite, event.Usage.CacheWrite)
|
||||
count := func(c pk.ObservedTokenCount) TokenCount {
|
||||
return TokenCount{Value: c.Value, Known: c.Known, Complete: c.Complete}
|
||||
}
|
||||
if !dispatched {
|
||||
result.Usage.Prompt.Known = true
|
||||
result.Usage.Completion.Known = true
|
||||
result.Usage.Total.Known = true
|
||||
result.Usage.Cached.Known = true
|
||||
result.Usage.CacheWrite.Known = true
|
||||
}
|
||||
return result
|
||||
return ExecutionSummary{LogicalCalls: uint64(len(snapshot.Calls)), TerminalCoverageComplete: summary.TerminalCoverageComplete, ProviderAttempts: summary.Attempts, TransientRetries: summary.TransientRetries, StructuralRepairs: summary.RepairCalls, Usage: Usage{Prompt: count(summary.Usage.PromptTokens), Completion: count(summary.Usage.CompletionTokens), Total: count(summary.Usage.TotalTokens), Cached: count(summary.Usage.CachedTokens), CacheWrite: count(summary.Usage.CacheWriteTokens)}}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package llm
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
@@ -23,8 +24,8 @@ func TestObservationSummarySeparatesLogicalCallsAttemptsAndUsage(t *testing.T) {
|
||||
}
|
||||
call(1) // preparation failed before PromptKit emitted events
|
||||
call(2) // capacity rejected without a dispatched attempt
|
||||
hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{Kind: pk.ExecutionAttempt, Dispatched: false})
|
||||
hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 0})
|
||||
hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{RunID: "run-2", Kind: pk.ExecutionAttempt, Dispatched: false})
|
||||
hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{Role: "proposal"}, pk.ExecutionEvent{RunID: "run-2", Kind: pk.ExecutionFinished, Attempts: 0, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CompletionTokens: pk.ObservedTokenCount{Known: true, Complete: true}, TotalTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CachedTokens: pk.ObservedTokenCount{Known: true, Complete: true}, CacheWriteTokens: pk.ObservedTokenCount{Known: true, Complete: true}}})
|
||||
summary := Summarize(collector.Snapshot())
|
||||
if summary.LogicalCalls != 2 || summary.ProviderAttempts != 0 || summary.Usage.Total != (TokenCount{Known: true, Complete: true}) {
|
||||
t.Fatalf("zero-attempt accounting: %+v", summary)
|
||||
@@ -32,16 +33,16 @@ func TestObservationSummarySeparatesLogicalCallsAttemptsAndUsage(t *testing.T) {
|
||||
|
||||
call(3)
|
||||
context := structuredoutput.DiagnosticContext{Role: "validation", ModuleInstance: "grammar", ValidatorName: "editorial_review"}
|
||||
hooks.OnEventWithCall(3, context, pk.ExecutionEvent{Kind: pk.ExecutionAttempt, Dispatched: true, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 100, Known: true, Complete: true}}})
|
||||
hooks.OnEventWithCall(3, context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 3, TransientRetries: 1, RepairCalls: 1, Usage: pk.ObservedTokenUsage{
|
||||
hooks.OnEventWithCall(3, context, pk.ExecutionEvent{RunID: "run-3", Kind: pk.ExecutionAttempt, Dispatched: true, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 100, Known: true, Complete: true}}})
|
||||
hooks.OnEventWithCall(3, context, pk.ExecutionEvent{RunID: "run-3", Kind: pk.ExecutionFinished, Attempts: 3, TransientRetries: 1, RepairCalls: 1, Usage: pk.ObservedTokenUsage{
|
||||
PromptTokens: pk.ObservedTokenCount{Value: 5, Known: true, Complete: false},
|
||||
TotalTokens: pk.ObservedTokenCount{Value: 8, Known: true, Complete: false},
|
||||
}})
|
||||
call(4)
|
||||
hooks.OnEventWithCall(4, context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: false, Complete: false}, TotalTokens: pk.ObservedTokenCount{Known: false, Complete: false}}})
|
||||
hooks.OnEventWithCall(4, context, pk.ExecutionEvent{RunID: "run-4", Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{PromptTokens: pk.ObservedTokenCount{Known: false, Complete: false}, TotalTokens: pk.ObservedTokenCount{Known: false, Complete: false}}})
|
||||
snapshot := collector.Snapshot()
|
||||
summary = Summarize(snapshot)
|
||||
if summary.LogicalCalls != 4 || summary.ProviderAttempts != 4 || summary.TransientRetries != 1 || summary.StructuralRepairs != 1 || summary.Usage.Total.Value != 8 || !summary.Usage.Total.Known || summary.Usage.Total.Complete || summary.Usage.Prompt.Value != 5 || !summary.Usage.Prompt.Known || summary.Usage.Prompt.Complete || summary.Usage.Cached.Known || summary.Usage.Cached.Complete {
|
||||
if summary.LogicalCalls != 4 || summary.ProviderAttempts != 4 || summary.TransientRetries != 1 || summary.StructuralRepairs != 1 || summary.Usage.Total.Value != 8 || !summary.Usage.Total.Known || summary.Usage.Total.Complete || summary.Usage.Prompt.Value != 5 || !summary.Usage.Prompt.Known || summary.Usage.Prompt.Complete || !summary.Usage.Cached.Known || summary.Usage.Cached.Complete {
|
||||
t.Fatalf("partial usage or double count: %+v", summary)
|
||||
}
|
||||
raw, err := json.Marshal(snapshot)
|
||||
@@ -68,7 +69,7 @@ func TestObservationCollectorConcurrentAttributionAndSnapshotCopy(t *testing.T)
|
||||
context := structuredoutput.DiagnosticContext{Role: "proposal", ModuleInstance: "grammar", SectionIndex: &index}
|
||||
started := time.Now().UTC()
|
||||
hooks.OnCall(structuredoutput.CallRecord{CallID: uint64(id), Context: context, Started: started})
|
||||
hooks.OnEventWithCall(uint64(id), context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, RunID: "run", Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 1, Known: true, Complete: true}}})
|
||||
hooks.OnEventWithCall(uint64(id), context, pk.ExecutionEvent{Kind: pk.ExecutionFinished, RunID: fmt.Sprint(id), Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 1, Known: true, Complete: true}}})
|
||||
hooks.OnCall(structuredoutput.CallRecord{CallID: uint64(id), Context: context, Started: started, Finished: time.Now().UTC()})
|
||||
index = -1
|
||||
}(i)
|
||||
@@ -124,3 +125,18 @@ func TestRoleClientFeedsCollectorAcrossPreparationFailure(t *testing.T) {
|
||||
t.Fatalf("logical failures: %+v", snapshot.Calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationCoverageIsIndependentOfUsage(t *testing.T) {
|
||||
collector := NewObservationCollector()
|
||||
hooks := collector.Hooks()
|
||||
hooks.OnEventWithCall(1, structuredoutput.DiagnosticContext{}, pk.ExecutionEvent{RunID: "pending", Kind: pk.ExecutionAttempt})
|
||||
got := Summarize(collector.Snapshot())
|
||||
if got.TerminalCoverageComplete || got.Usage.Total.Known || got.Usage.Total.Complete {
|
||||
t.Fatalf("incomplete group=%+v", got)
|
||||
}
|
||||
hooks.OnEventWithCall(2, structuredoutput.DiagnosticContext{}, pk.ExecutionEvent{RunID: "finished", Kind: pk.ExecutionFinished, Attempts: 1, Usage: pk.ObservedTokenUsage{TotalTokens: pk.ObservedTokenCount{Value: 7, Known: true, Complete: true}}})
|
||||
got = Summarize(collector.Snapshot())
|
||||
if got.TerminalCoverageComplete || !got.Usage.Total.Known || !got.Usage.Total.Complete || got.Usage.Total.Value != 7 {
|
||||
t.Fatalf("partial coverage=%+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,15 +20,16 @@ func mapLLMSummary(source llm.ExecutionSummary) reporting.LLMSummary {
|
||||
}
|
||||
|
||||
type executionCall struct {
|
||||
RequestedPromptID string `json:"requested_prompt_id,omitempty"`
|
||||
CallID uint64 `json:"call_id"`
|
||||
Context structuredoutput.DiagnosticContext `json:"context"`
|
||||
Prompt structuredoutput.PromptMetadata `json:"prompt"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished *time.Time `json:"finished,omitempty"`
|
||||
ElapsedMS int64 `json:"elapsed_ms"`
|
||||
Failure structuredoutput.FailureKind `json:"failure,omitempty"`
|
||||
Summary reporting.LLMSummary `json:"summary"`
|
||||
TerminalCoverageComplete bool `json:"terminal_coverage_complete"`
|
||||
RequestedPromptID string `json:"requested_prompt_id,omitempty"`
|
||||
CallID uint64 `json:"call_id"`
|
||||
Context structuredoutput.DiagnosticContext `json:"context"`
|
||||
Prompt structuredoutput.PromptMetadata `json:"prompt"`
|
||||
Started time.Time `json:"started"`
|
||||
Finished *time.Time `json:"finished,omitempty"`
|
||||
ElapsedMS int64 `json:"elapsed_ms"`
|
||||
Failure structuredoutput.FailureKind `json:"failure,omitempty"`
|
||||
Summary reporting.LLMSummary `json:"summary"`
|
||||
}
|
||||
|
||||
type executionEvent struct {
|
||||
@@ -72,7 +73,7 @@ func BuildLLMExecution(snapshot llm.ObservationSnapshot) any {
|
||||
value := call.Finished
|
||||
finished = &value
|
||||
}
|
||||
calls = append(calls, executionCall{RequestedPromptID: call.RequestedPromptID, CallID: call.CallID, Context: call.Context, Prompt: call.Prompt, Started: call.Started, Finished: finished, ElapsedMS: call.WallDuration.Milliseconds(), Failure: call.Failure, Summary: mapLLMSummary(llm.Summarize(group))})
|
||||
calls = append(calls, executionCall{TerminalCoverageComplete: llm.Summarize(group).TerminalCoverageComplete, RequestedPromptID: call.RequestedPromptID, CallID: call.CallID, Context: call.Context, Prompt: call.Prompt, Started: call.Started, Finished: finished, ElapsedMS: call.WallDuration.Milliseconds(), Failure: call.Failure, Summary: mapLLMSummary(llm.Summarize(group))})
|
||||
}
|
||||
for _, event := range snapshot.Events {
|
||||
var generation, attempt *uint64
|
||||
@@ -87,7 +88,8 @@ func BuildLLMExecution(snapshot llm.ObservationSnapshot) any {
|
||||
events = append(events, executionEvent{CallID: event.CallID, Context: event.Context, Kind: event.Kind, RunID: event.RunID, Time: event.Time, BackendID: event.BackendID, ProfileID: event.ProfileID, Model: event.Model, Generation: generation, Attempt: attempt, Dispatched: event.Dispatched, CapacityWaitMS: event.CapacityWait.Milliseconds(), GenerationMS: event.GenerationDuration.Milliseconds(), RetryDelayMS: event.RetryDelay.Milliseconds(), DurationMS: event.Duration.Milliseconds(), Outcome: event.Outcome, Failure: event.Failure, StatusCode: event.StatusCode, RetriesExhausted: event.RetriesExhausted, Attempts: event.Attempts, TransientRetries: event.TransientRetries, StructuralRepairs: event.StructuralRepairs, Usage: mapLLMSummary(llm.ExecutionSummary{Usage: event.Usage}).Usage})
|
||||
}
|
||||
return struct {
|
||||
Calls []executionCall `json:"calls"`
|
||||
Events []executionEvent `json:"events"`
|
||||
}{calls, events}
|
||||
Summary llm.ExecutionSummary `json:"summary"`
|
||||
Calls []executionCall `json:"calls"`
|
||||
Events []executionEvent `json:"events"`
|
||||
}{llm.Summarize(snapshot), calls, events}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestExecutionReportingKeepsSafeMilliseconds(t *testing.T) {
|
||||
{CallID: 1, Context: structuredoutput.DiagnosticContext{Role: "proposal", ModuleInstance: "grammar-1"}, Prompt: structuredoutput.PromptMetadata{BackendID: "primary", ProfileID: "draft", Model: "m"}, Started: start, Finished: start.Add(1500 * time.Millisecond), WallDuration: 1500 * time.Millisecond},
|
||||
{CallID: 2, Context: structuredoutput.DiagnosticContext{Role: "validation"}, Prompt: structuredoutput.PromptMetadata{BackendID: "secondary"}, Started: start, Finished: start.Add(5 * time.Millisecond), WallDuration: 5 * time.Millisecond, Failure: structuredoutput.FailureRuntime},
|
||||
},
|
||||
Events: []llm.ExecutionObservation{{CallID: 1, Kind: string(pk.ExecutionFinished), BackendID: "primary", Attempts: 2, TransientRetries: 1, CapacityWait: 20 * time.Millisecond, GenerationDuration: 500 * time.Millisecond, RetryDelay: 100 * time.Millisecond}},
|
||||
Events: []llm.ExecutionObservation{{CallID: 1, RunID: "run-1", Kind: string(pk.ExecutionFinished), BackendID: "primary", Attempts: 2, TransientRetries: 1, CapacityWait: 20 * time.Millisecond, GenerationDuration: 500 * time.Millisecond, RetryDelay: 100 * time.Millisecond}},
|
||||
}
|
||||
raw, err := json.Marshal(BuildLLMExecution(snapshot))
|
||||
if err != nil {
|
||||
@@ -46,7 +46,7 @@ func BenchmarkExecutionReports(b *testing.B) {
|
||||
snapshot := llm.ObservationSnapshot{}
|
||||
for i := 0; i < n; i++ {
|
||||
snapshot.Calls = append(snapshot.Calls, llm.LogicalCall{CallID: uint64(i + 1), Prompt: structuredoutput.PromptMetadata{BackendID: "local"}, Context: structuredoutput.DiagnosticContext{ModuleInstance: "grammar", Role: "proposal"}})
|
||||
snapshot.Events = append(snapshot.Events, llm.ExecutionObservation{CallID: uint64(i + 1), Kind: string(pk.ExecutionFinished), Attempts: 1})
|
||||
snapshot.Events = append(snapshot.Events, llm.ExecutionObservation{RunID: fmt.Sprint(i), CallID: uint64(i + 1), Kind: string(pk.ExecutionFinished), Attempts: 1})
|
||||
}
|
||||
b.Run(fmt.Sprintf("execution/%d", n), func(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
@@ -64,9 +64,9 @@ func TestIndexedReportsPreserveOrderAndPartialUsage(t *testing.T) {
|
||||
{CallID: 2, Context: structuredoutput.DiagnosticContext{Role: "validation", ModuleInstance: "grammar_1"}, Prompt: structuredoutput.PromptMetadata{BackendID: "a"}},
|
||||
},
|
||||
Events: []llm.ExecutionObservation{
|
||||
{CallID: 1, Kind: string(pk.ExecutionFinished), Attempts: 1, Usage: llm.Usage{Total: llm.TokenCount{Value: 7, Known: true, Complete: true}}},
|
||||
{CallID: 3, Kind: string(pk.ExecutionFinished), Attempts: 1},
|
||||
{CallID: 2, Kind: string(pk.ExecutionFinished), Attempts: 1},
|
||||
{CallID: 1, RunID: "run-1", Kind: string(pk.ExecutionFinished), Attempts: 1, Usage: llm.Usage{Total: llm.TokenCount{Value: 7, Known: true, Complete: true}}},
|
||||
{CallID: 3, RunID: "run-3", Kind: string(pk.ExecutionFinished), Attempts: 1},
|
||||
{CallID: 2, RunID: "run-2", Kind: string(pk.ExecutionFinished), Attempts: 1},
|
||||
},
|
||||
}
|
||||
raw, err := json.Marshal(BuildLLMExecution(snapshot))
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
)
|
||||
|
||||
func TestExecuteModuleProposalBuildsMessagesFromSectionAndDescription(t *testing.T) {
|
||||
exchange := &structuredoutput.Exchange{PromptID: "captured", FinalContent: "response"}
|
||||
exchange := &structuredoutput.Exchange{}
|
||||
client := &fakeStructuredClient{
|
||||
completion: structuredoutput.Response{Exchange: exchange},
|
||||
responses: []StructuredCorrectionSet{
|
||||
|
||||
@@ -2,6 +2,7 @@ package runner
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
@@ -117,8 +118,11 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput,
|
||||
return findings
|
||||
}
|
||||
findings := safeFindings(rejected)
|
||||
replay := &structuredoutput.Replay{Exchange: input.Exchange}
|
||||
for attempt := 0; attempt < input.Spec.Settings.LLM.CorrectionRepairAttempts; attempt++ {
|
||||
coordinator, ok := input.Spec.RepairClient.(structuredoutput.Coordinator)
|
||||
if !ok {
|
||||
return finish(fmt.Errorf("correction repair coordinator is unavailable"))
|
||||
}
|
||||
buildRequest := func() (*structuredoutput.Request, error) {
|
||||
request := proposal_generation.RepairRequest{
|
||||
ProposalRef: position + 1,
|
||||
Correction: proposal_generation.StructuredCorrectionProposal{SegmentReference: localRef, OriginalText: current.OriginalText, CorrectedText: current.CorrectedText},
|
||||
@@ -134,12 +138,23 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput,
|
||||
}
|
||||
payload, err := request.JSON()
|
||||
if err != nil {
|
||||
branches = append(branches, branch)
|
||||
return finish(err)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
proposalIndex, attemptNumber := original.ProposalIndex, branch.AttemptCount+1
|
||||
return &structuredoutput.Request{PromptID: promptID, Variables: map[string]string{"RepairRequestJSON": payload}, Schema: responseschema.CorrectionRepairKey, Context: structuredoutput.DiagnosticContext{StageIndex: &input.Spec.Index, StageID: input.Spec.ID, Role: "correction_repair", ModuleKey: input.Spec.ModuleKey, ModuleInstance: input.Spec.InstanceName, SectionIndex: input.SectionIndex, ProposalIndex: &proposalIndex, RepairAttempt: &attemptNumber}}, nil
|
||||
}
|
||||
exhausted, coordinationErr := coordinator.Coordinate(ctx, input.Exchange, input.Spec.Settings.LLM.CorrectionRepairAttempts, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) {
|
||||
if turn.Response == nil {
|
||||
return buildRequest()
|
||||
}
|
||||
completion, callErr := *turn.Response, turn.Err
|
||||
proposalIndex, attemptNumber := original.ProposalIndex, branch.AttemptCount+1
|
||||
var response proposal_generation.RepairResponse
|
||||
proposalIndex, attemptNumber := original.ProposalIndex, attempt+1
|
||||
completion, callErr := input.Spec.RepairClient.CompleteStructured(ctx, structuredoutput.Request{PromptID: promptID, Variables: map[string]string{"RepairRequestJSON": payload}, Schema: responseschema.CorrectionRepairKey, Replay: replay, Context: structuredoutput.DiagnosticContext{StageIndex: &input.Spec.Index, StageID: input.Spec.ID, Role: "correction_repair", ModuleKey: input.Spec.ModuleKey, ModuleInstance: input.Spec.InstanceName, SectionIndex: input.SectionIndex, ProposalIndex: &proposalIndex, RepairAttempt: &attemptNumber}}, &response)
|
||||
if callErr == nil {
|
||||
_ = json.Unmarshal(completion.Content, &response)
|
||||
}
|
||||
|
||||
operation := outcomes.Operation{
|
||||
Reference: outcomes.Reference{
|
||||
StageIndex: &input.Spec.Index, StageID: input.Spec.ID, ModuleKey: input.Spec.ModuleKey,
|
||||
@@ -183,26 +198,23 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput,
|
||||
finishAttempt(callErr, nil, validateSectionCandidatesResult{})
|
||||
if structuredoutput.IsRecoverable(callErr) {
|
||||
branch.Outcome = "unavailable"
|
||||
break
|
||||
return nil, nil
|
||||
}
|
||||
branches = append(branches, branch)
|
||||
return finish(callErr)
|
||||
return nil, callErr
|
||||
}
|
||||
decoded, err := proposal_generation.DecodeRepairResponse(completion.Content, position+1)
|
||||
if err != nil {
|
||||
operation.FailureReason = outcomes.ReasonInvalidOutput
|
||||
finishAttempt(err, nil, validateSectionCandidatesResult{})
|
||||
branch.Outcome = "unavailable"
|
||||
break
|
||||
return nil, nil
|
||||
}
|
||||
operation.Succeeded = true
|
||||
operation.RecoveryReasons = outcomes.ExecutionRecoveries(operation.Counts)
|
||||
replay.Turns = append(replay.Turns, completion.RepairInstruction...)
|
||||
replay.Turns = append(replay.Turns, structuredoutput.Message{Role: "assistant", Content: completion.FinalContent})
|
||||
if decoded.Action == "withdraw" {
|
||||
finishAttempt(nil, nil, validateSectionCandidatesResult{})
|
||||
branch.Outcome = "withdrawn"
|
||||
break
|
||||
return nil, nil
|
||||
}
|
||||
revision := decoded.Correction
|
||||
localRef = revision.SegmentReference
|
||||
@@ -225,7 +237,7 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput,
|
||||
operation.RecoveryReasons = append(operation.RecoveryReasons, outcomes.RecoveryProposalsFiltered)
|
||||
}
|
||||
for i := range validated.operations {
|
||||
proposalIndex, attemptNumber := original.ProposalIndex, attempt+1
|
||||
proposalIndex, attemptNumber := original.ProposalIndex, attemptNumber
|
||||
validated.operations[i].Reference.ProposalIndex = &proposalIndex
|
||||
validated.operations[i].Reference.RepairAttempt = &attemptNumber
|
||||
}
|
||||
@@ -236,32 +248,35 @@ func repairCandidates(ctx context.Context, input validateSectionCandidatesInput,
|
||||
finishAttempt(validationErr, ¤t, validated)
|
||||
initial.operations = append(initial.operations, validated.operations...)
|
||||
if validationErr != nil {
|
||||
branches = append(branches, branch)
|
||||
return finish(validationErr)
|
||||
return nil, validationErr
|
||||
}
|
||||
if approved {
|
||||
branch.Outcome = "approved"
|
||||
initial.approved = append(initial.approved, current)
|
||||
superseded[original.ProposalIndex] = true
|
||||
break
|
||||
return nil, nil
|
||||
}
|
||||
if actionable(original.ProposalIndex, validated.rejected) {
|
||||
if attempt+1 == input.Spec.Settings.LLM.CorrectionRepairAttempts {
|
||||
branch.Outcome = "exhausted"
|
||||
break
|
||||
}
|
||||
findings = safeFindings(validated.rejected)
|
||||
continue
|
||||
return buildRequest()
|
||||
}
|
||||
branch.Outcome = "unavailable"
|
||||
for _, d := range validated.rejected {
|
||||
if d.Disposition != validators.DispositionUnavailable {
|
||||
branch.Outcome = "rejected"
|
||||
break
|
||||
return nil, nil
|
||||
}
|
||||
}
|
||||
break
|
||||
return nil, nil
|
||||
})
|
||||
if exhausted {
|
||||
branch.Outcome = "exhausted"
|
||||
}
|
||||
if coordinationErr != nil {
|
||||
branches = append(branches, branch)
|
||||
return finish(coordinationErr)
|
||||
}
|
||||
|
||||
branches = append(branches, branch)
|
||||
}
|
||||
return finish(nil)
|
||||
|
||||
@@ -4,10 +4,17 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/responseschema"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/prompts"
|
||||
ck "go.maximumdirect.net/configkit"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/audita/v3/internal/core/schema"
|
||||
@@ -20,6 +27,7 @@ import (
|
||||
)
|
||||
|
||||
type repairClient struct {
|
||||
t *testing.T
|
||||
outputs []string
|
||||
calls []structuredoutput.Request
|
||||
err error
|
||||
@@ -27,11 +35,7 @@ type repairClient struct {
|
||||
}
|
||||
|
||||
func (c *repairClient) CompleteStructured(_ context.Context, req structuredoutput.Request, out any) (structuredoutput.Response, error) {
|
||||
if req.Replay != nil {
|
||||
copyReplay := *req.Replay
|
||||
copyReplay.Turns = append([]structuredoutput.Message(nil), req.Replay.Turns...)
|
||||
req.Replay = ©Replay
|
||||
}
|
||||
|
||||
c.calls = append(c.calls, req)
|
||||
if c.err != nil {
|
||||
return structuredoutput.Response{}, c.err
|
||||
@@ -40,18 +44,18 @@ func (c *repairClient) CompleteStructured(_ context.Context, req structuredoutpu
|
||||
if err := json.Unmarshal([]byte(value), out); err != nil {
|
||||
return structuredoutput.Response{}, err
|
||||
}
|
||||
return structuredoutput.Response{Prepared: &structuredoutput.PreparedSnapshot{Messages: []structuredoutput.Message{{Role: "user", Content: "synthetic-secret"}}}, Content: json.RawMessage(value), FinalContent: value, RepairInstruction: []structuredoutput.Message{{Role: "user", Content: "instruction"}}}, nil
|
||||
return structuredoutput.Response{Prepared: &structuredoutput.PreparedSnapshot{Messages: []structuredoutput.Message{{Role: "user", Content: "synthetic-secret"}}}, Content: json.RawMessage(value), FinalContent: value, Exchange: &structuredoutput.Exchange{}}, nil
|
||||
}
|
||||
|
||||
func (c *repairClient) ConfiguredSecrets(context.Context) []string { return c.secrets }
|
||||
|
||||
func TestRepairBranchesRetargetAndPreserveOtherApprovals(t *testing.T) {
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`}}
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`}}
|
||||
candidates := []proposals.EnrichedCorrectionProposal{
|
||||
{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 4}},
|
||||
{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 0, OriginalText: "x", CorrectedText: "y"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 5}},
|
||||
}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{ModuleKey: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}, {ID: 2, Text: "uh hello"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal", FinalContent: "original response"}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{ModuleKey: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}, {ID: 2, Text: "uh hello"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -85,7 +89,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) {
|
||||
{"blank result", "uh", "uh", "", "", proposals.ReplacementPolicyRequireUnique, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 81, OriginalText: tc.original, CorrectedText: tc.replacement}}
|
||||
input := validateSectionCandidatesInput{
|
||||
Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}},
|
||||
@@ -93,7 +97,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) {
|
||||
WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 81, Text: tc.segment}}},
|
||||
Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, validators.NonEmptyResultingSegmentValidator{}, rejectUntilRepair()},
|
||||
SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate},
|
||||
Exchange: &structuredoutput.Exchange{PromptID: "proposal"},
|
||||
Exchange: &structuredoutput.Exchange{},
|
||||
}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
@@ -125,7 +129,7 @@ func TestRepairRequestCarriesLiteralPreview(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRepairRequestRefreshesPreviewAfterRetarget(t *testing.T) {
|
||||
client := &repairClient{outputs: []string{
|
||||
client := &repairClient{t: t, outputs: []string{
|
||||
`{"proposal_ref":1,"action":"revise","correction":{"id":2,"original_text":"uh ","corrected_text":""}}`,
|
||||
`{"proposal_ref":1,"action":"withdraw","correction":null}`,
|
||||
}}
|
||||
@@ -138,7 +142,7 @@ func TestRepairRequestRefreshesPreviewAfterRetarget(t *testing.T) {
|
||||
WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 81, Text: "cat"}, {ID: 104, Text: "uh hello"}}},
|
||||
Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, reject},
|
||||
SectionEnriched: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 81, OriginalText: "cat", CorrectedText: "dog"}}},
|
||||
Exchange: &structuredoutput.Exchange{PromptID: "proposal"},
|
||||
Exchange: &structuredoutput.Exchange{},
|
||||
}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
@@ -175,9 +179,9 @@ func TestRepairBranchTerminalOutcomes(t *testing.T) {
|
||||
{"provider failure", nil, &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonProviderFailed, Message: "provider failed"}, 1, "unavailable", 1},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
client := &repairClient{outputs: tc.outputs, err: tc.err}
|
||||
client := &repairClient{t: t, outputs: tc.outputs, err: tc.err}
|
||||
candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: tc.budget}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: tc.budget}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -189,8 +193,8 @@ func TestRepairBranchTerminalOutcomes(t *testing.T) {
|
||||
if len(branches) != 1 || branches[0].Outcome != tc.outcome || len(client.calls) != tc.calls || branches[0].AttemptCount != tc.calls || len(branches[0].Attempts) != 0 {
|
||||
t.Fatalf("branches=%+v calls=%d", branches, len(client.calls))
|
||||
}
|
||||
if tc.calls == 2 && len(client.calls[1].Replay.Turns) != 2 {
|
||||
t.Fatalf("history: %+v", client.calls[1].Replay.Turns)
|
||||
if tc.calls == 2 && client.calls[1].Context.RepairAttempt == nil {
|
||||
t.Fatalf("history: %+v", client.calls[1].ContinueFrom)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -202,11 +206,11 @@ func (m repairModule) RepairPromptID() string { return "repair" }
|
||||
|
||||
func TestRunnerAppliesRevisionsInOriginalOrder(t *testing.T) {
|
||||
candidates := []proposals.CorrectionProposal{{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, {TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "fox"}, {TargetSegmentID: 1, OriginalText: "bird", CorrectedText: "owl"}}
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"cat","corrected_text":"fox"}}`,
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"cat","corrected_text":"fox"}}`,
|
||||
`{"proposal_ref":3,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`,
|
||||
}}
|
||||
module := repairModule{outcomeModule{fakeModule: fakeModule{key: "grammar", policy: proposals.ReplacementPolicyRequireUnique, validators: []contracts.Validator{validators.ProposalShapeValidator{}, validators.NoEffectValidator{}, validators.OriginalTextPresenceValidator{}, rejectUntilRepair(0)}}, proposeResult: func(req contracts.ProposalRequest) (contracts.ProposalResult, error) {
|
||||
return contracts.ProposalResult{Proposals: candidates, SegmentReferences: []int{1, 1, 1}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}}, nil
|
||||
return contracts.ProposalResult{Proposals: candidates, SegmentReferences: []int{1, 1, 1}, Exchange: &structuredoutput.Exchange{}}, nil
|
||||
}}}
|
||||
runner := New(fakeFactory{modules: map[string]contracts.TranscriptModule{"grammar": module}})
|
||||
out, err := runner.Run(context.Background(), RunInput{Transcript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat and bird"}}}, ModuleSpecs: []contracts.ModuleRunSpec{{ModuleKey: "grammar", InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}}})
|
||||
@@ -246,11 +250,11 @@ func TestRepairRevalidatesEarlierApprovalAndRetainsUnrelatedApprovalOnFailure(t
|
||||
}
|
||||
return validators.Result{Decisions: decisions}, nil
|
||||
}}
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`}}
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":2,"action":"revise","correction":{"id":1,"original_text":"bird","corrected_text":"owl"}}`}}
|
||||
if tc.failure {
|
||||
client.err = &structuredoutput.Failure{Class: structuredoutput.FailureRecoverable, Reason: outcomes.ReasonProviderFailed, Message: "failed"}
|
||||
}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat bird"}}}, Validators: []contracts.Validator{check, rejectUntilRepair(0)}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat bird"}}}, Validators: []contracts.Validator{check, rejectUntilRepair(0)}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -271,9 +275,9 @@ func TestRepairRevalidatesEarlierApprovalAndRetainsUnrelatedApprovalOnFailure(t
|
||||
}
|
||||
|
||||
func TestRepairInterruptionRetainsAttempt(t *testing.T) {
|
||||
client := &repairClient{err: &structuredoutput.Failure{Class: structuredoutput.FailureCanceled, Reason: outcomes.ReasonCanceled, Message: "canceled", Cause: context.Canceled}}
|
||||
client := &repairClient{t: t, err: &structuredoutput.Failure{Class: structuredoutput.FailureCanceled, Reason: outcomes.ReasonCanceled, Message: "canceled", Cause: context.Canceled}}
|
||||
candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -285,12 +289,12 @@ func TestRepairInterruptionRetainsAttempt(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRepairArtifactsAreUniqueAndRedacted(t *testing.T) {
|
||||
client := &repairClient{secrets: []string{"synthetic-secret"}, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`, `{"proposal_ref":2,"action":"withdraw","correction":null}`}}
|
||||
client := &repairClient{t: t, secrets: []string{"synthetic-secret"}, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`, `{"proposal_ref":2,"action":"withdraw","correction":null}`}}
|
||||
candidates := []proposals.EnrichedCorrectionProposal{
|
||||
{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}},
|
||||
{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "fox"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 1}},
|
||||
}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar_1", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: t.TempDir()}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar_1", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: candidates, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: t.TempDir()}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -318,14 +322,14 @@ func TestRepairArtifactsAreUniqueAndRedacted(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRepairArtifactFailureDoesNotChangeOutcome(t *testing.T) {
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
path := filepath.Join(t.TempDir(), "file")
|
||||
if err := os.WriteFile(path, []byte("x"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
warnings := 0
|
||||
candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: path, ReportDiagnosticError: func(error) { warnings++ }}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, Policy: proposals.ReplacementPolicyRequireUnique, WorkingTranscript: &schema.Transcript{Segments: []schema.Segment{{ID: 1, Text: "cat"}}}, Validators: []contracts.Validator{rejectUntilRepair()}, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: path, ReportDiagnosticError: func(error) { warnings++ }}
|
||||
initial, err := validateSectionCandidates(context.Background(), input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -337,7 +341,7 @@ func TestRepairArtifactFailureDoesNotChangeOutcome(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestNoEffectAndUnavailableOnlyDoNotStartRepair(t *testing.T) {
|
||||
client := &repairClient{}
|
||||
client := &repairClient{t: t}
|
||||
for _, rejected := range [][]ValidatorRejectedChange{
|
||||
{{ProposalIndex: 0, Disposition: validators.DispositionRejected, ReasonCode: validators.ReasonNoEffect}},
|
||||
{{ProposalIndex: 0, Disposition: validators.DispositionUnavailable, ReasonCode: "provider_failed"}},
|
||||
@@ -352,10 +356,10 @@ func TestNoEffectAndUnavailableOnlyDoNotStartRepair(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRepairFeedbackRedactsValidationCredentials(t *testing.T) {
|
||||
client := &repairClient{outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
validation := &repairClient{secrets: []string{"validation-secret"}}
|
||||
client := &repairClient{t: t, outputs: []string{`{"proposal_ref":1,"action":"withdraw","correction":null}`}}
|
||||
validation := &repairClient{t: t, secrets: []string{"validation-secret"}}
|
||||
candidate := proposals.EnrichedCorrectionProposal{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 1, OriginalText: "cat", CorrectedText: "dog"}, ProposalMetadata: proposals.ProposalMetadata{ProposalIndex: 0}}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, ValidationLLMClient: validation, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{PromptID: "proposal"}, DiagnosticsDir: t.TempDir()}
|
||||
input := validateSectionCandidatesInput{Spec: contracts.ModuleRunSpec{InstanceName: "grammar", RepairClient: client, Settings: config.Stage{LLM: config.LLM{CorrectionRepairAttempts: 1}}}, ValidationLLMClient: validation, SectionEnriched: []proposals.EnrichedCorrectionProposal{candidate}, Exchange: &structuredoutput.Exchange{}, DiagnosticsDir: t.TempDir()}
|
||||
initial := validateSectionCandidatesResult{rejected: []ValidatorRejectedChange{{ProposalIndex: 0, ValidatorName: "editorial_review", Disposition: validators.DispositionRejected, ReasonCode: "rejected", Message: "model echoed validation-secret"}}}
|
||||
_, branches, err := repairCandidates(context.Background(), input, []int{1}, initial, "repair")
|
||||
if err != nil {
|
||||
@@ -393,3 +397,66 @@ func rejectUntilRepair(preserved ...int) contracts.Validator {
|
||||
return validators.Result{Decisions: decisions}, nil
|
||||
}}
|
||||
}
|
||||
|
||||
type repairTransport func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f repairTransport) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
|
||||
|
||||
// Exercise the real adapter and upstream coordinator; only model responses are scripted.
|
||||
func (c *repairClient) Coordinate(ctx context.Context, _ *structuredoutput.Exchange, budget int, next func(context.Context, structuredoutput.Turn) (*structuredoutput.Request, error)) (bool, error) {
|
||||
t := c.t
|
||||
dir := t.TempDir()
|
||||
for name, content := range map[string]string{
|
||||
"config.yml": "version: 3\n",
|
||||
"prompts.yml": "profiles:\n - id: fixture\n endpoint: https://example.invalid/v1\n model: fixture\nprompts:\n - id: repair\n messages:\n - role: user\n content: 'synthetic-secret {{.RepairRequestJSON}}'\n output:\n format: text\n validation_mode: none\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
loaded, err := config.Load(ck.Input{Source: ck.File(filepath.Join(dir, "config.yml"))}, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var selected *structuredoutput.Request
|
||||
initial := true
|
||||
runtime, err := llm.NewRuntime(ctx, loaded, &http.Client{Transport: repairTransport(func(*http.Request) (*http.Response, error) {
|
||||
content := `{"corrections":[]}`
|
||||
if initial {
|
||||
initial = false
|
||||
} else {
|
||||
c.calls = append(c.calls, *selected)
|
||||
content = c.outputs[len(c.calls)-1]
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"choices": []any{map[string]any{"message": map[string]any{"role": "assistant", "content": content}, "finish_reason": "stop"}}})
|
||||
return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(string(body)))}, nil
|
||||
})})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client, err := runtime.NewRoleClient("repair", "fixture", config.LLM{OperationTimeout: time.Second}, llm.CallHooks{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var output map[string]any
|
||||
base, err := client.CompleteStructured(ctx, structuredoutput.Request{PromptID: prompts.PromptIDModuleGrammarProposal, Schema: responseschema.CorrectionSetKey, CaptureExchange: true, Variables: map[string]string{"TranscriptDescriptionBlock": "", "GlossaryJSON": "[]", "SectionJSON": "[]"}}, &output)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return client.Coordinate(ctx, base.Exchange, budget, func(ctx context.Context, turn structuredoutput.Turn) (*structuredoutput.Request, error) {
|
||||
req, err := next(ctx, turn)
|
||||
if req != nil && err == nil {
|
||||
selected = req
|
||||
if c.err != nil {
|
||||
c.calls = append(c.calls, *req)
|
||||
_, err = next(ctx, structuredoutput.Turn{Response: &structuredoutput.Response{}, Err: c.err})
|
||||
return nil, err
|
||||
}
|
||||
// Test prompt IDs vary; all use the same supplied-data fixture definition.
|
||||
copy := *req
|
||||
copy.PromptID = "repair"
|
||||
return ©, nil
|
||||
}
|
||||
return req, err
|
||||
})
|
||||
}
|
||||
|
||||
@@ -31,7 +31,7 @@ type Request struct {
|
||||
Schema responseschema.Key `json:"schema"`
|
||||
Context DiagnosticContext `json:"context"`
|
||||
CaptureExchange bool `json:"-"`
|
||||
Replay *Replay `json:"-"`
|
||||
ContinueFrom *Exchange `json:"-"`
|
||||
}
|
||||
|
||||
// PromptMetadata is the execution snapshot returned by the runtime when available.
|
||||
@@ -44,33 +44,23 @@ type PromptMetadata struct {
|
||||
Model string `json:"model,omitempty"`
|
||||
}
|
||||
|
||||
// Exchange holds a section's original prepared conversation for read-only replay.
|
||||
// Exchange is an opaque immutable conversation owned by the runtime adapter.
|
||||
type Exchange struct {
|
||||
PromptID string
|
||||
Variables map[string]string
|
||||
Messages []Message
|
||||
FinalContent string
|
||||
}
|
||||
|
||||
// Replay carries a section exchange and the prior turns of one repair branch.
|
||||
type Replay struct {
|
||||
Exchange *Exchange
|
||||
Turns []Message
|
||||
State any
|
||||
}
|
||||
|
||||
type Response struct {
|
||||
Exchange *Exchange `json:"-"`
|
||||
RepairInstruction []Message `json:"-"`
|
||||
Execution outcomes.ExecutionCounts `json:"execution"`
|
||||
Content json.RawMessage `json:"content,omitempty"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
PromptTokens int `json:"prompt_tokens,omitempty"`
|
||||
CompletionTokens int `json:"completion_tokens,omitempty"`
|
||||
TotalTokens int `json:"total_tokens,omitempty"`
|
||||
Prompt PromptMetadata `json:"prompt"`
|
||||
Prepared *PreparedSnapshot `json:"prepared,omitempty"`
|
||||
FinalContent string `json:"final_content,omitempty"`
|
||||
Exchange *Exchange `json:"-"`
|
||||
Execution outcomes.ExecutionCounts `json:"execution"`
|
||||
Content json.RawMessage `json:"content,omitempty"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
PromptTokens int `json:"prompt_tokens,omitempty"`
|
||||
CompletionTokens int `json:"completion_tokens,omitempty"`
|
||||
TotalTokens int `json:"total_tokens,omitempty"`
|
||||
Prompt PromptMetadata `json:"prompt"`
|
||||
Prepared *PreparedSnapshot `json:"prepared,omitempty"`
|
||||
FinalContent string `json:"final_content,omitempty"`
|
||||
}
|
||||
|
||||
// PreparedSnapshot is the exact rendered request snapshot. Its messages may
|
||||
@@ -108,3 +98,16 @@ type CallRecord struct {
|
||||
type Client interface {
|
||||
CompleteStructured(ctx context.Context, req Request, out any) (Response, error)
|
||||
}
|
||||
|
||||
// Turn delivers a completed correction request, or its failure. The initial
|
||||
// callback has no response. A failure is terminal for this coordination.
|
||||
type Turn struct {
|
||||
Response *Response
|
||||
Err error
|
||||
}
|
||||
|
||||
// Coordinator delegates bounded conversation progression to the runtime.
|
||||
// Returning no request stops the branch; requesting beyond the budget exhausts it.
|
||||
type Coordinator interface {
|
||||
Coordinate(context.Context, *Exchange, int, func(context.Context, Turn) (*Request, error)) (bool, error)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user