Record per-run accounting and continuation relationships

This commit is contained in:
2026-11-13 16:33:21 +00:00
parent ee810b8fbe
commit 45bbf7bb2c

View File

@@ -54,9 +54,10 @@ The proposed capabilities are:
Continuation has direct value by removing consumer capture, reconstruction,
prefix comparison, and history assembly. The coordinator centralizes turn
progression, execution relationships, cancellation, budgets, and diagnostics.
Its value should come from removing that repeated lifecycle work, beyond merely
replacing a consumer's loop counter.
progression, cancellation, and budgets while reusing ordinary execution
observations. Captured exchanges supply the minimal parent relationship selected
in Decision 5. Its value comes from removing repeated lifecycle work, beyond
merely replacing a consumer's loop counter.
Audita would continue owning candidate selection, domain validation, feedback
meaning, branch scheduling, accepted sibling proposals, and transcript changes.
Its domain-specific repair code would not move wholesale into PromptKit.
@@ -178,9 +179,9 @@ interpret the purpose of that work.
**Implementation planning:** select exact request/stop and outcome types under
Decision 4's accepted response delivery, turn-budget consumption, zero-turn
behavior, cancellation, and minimal error/result semantics.
Execution association and usage accounting belong to Decision 5. The sketches
below illustrate the selected responsibility split; they do not settle the
remaining public contracts.
Decision 5 records the selected execution association and accounting approach.
The sketches below illustrate the selected responsibility split; they do not
settle the remaining public contracts.
#### Illustrative Consumer Sketches
@@ -270,7 +271,8 @@ already receive each completed response through the callback. Completed work
must not be mistaken for application acceptance of its content.
The concrete interface should let both examples use the same coordination
mechanism without manual message-history assembly or per-turn lifecycle
bookkeeping. Callback model-call association remains part of Decision 5.
bookkeeping. Calls made inside callbacks retain independent observations;
applications explicitly group them as selected in Decision 5.
### 4. How Are Application Next Actions And Execution Outcomes Represented?
@@ -386,8 +388,9 @@ Failure-stage fields, additional run-ID references, a separate failure
classification hierarchy, and aggregate reporting are not part of this minimal
termination contract. They require a demonstrated consumer need before addition.
This does not remove fields from existing response values or observation APIs.
Decision 5 separately evaluates accounting, and Decision 6 inspection, without
presuming that either needs to expand the coordinator's termination result.
Decision 5 reuses ordinary run accounting without expanding this result.
Decision 6 separately evaluates inspection without presuming that it needs to
expand the coordinator's termination result.
Audita retains its own meanings for acceptance, rejection, withdrawal, and
assessment unavailability. A normal stop does not assert any of those outcomes.
@@ -408,33 +411,59 @@ result.
### 5. How Are Related Executions And Usage Accounted For?
**Status: open.** Define accounting for the initial exchange, each continuation,
its structural repairs and transport retries, and model calls made by
application callbacks, both for standalone continuations and coordinated turns.
**Status: accepted; exact placement of relationship metadata remains for
implementation planning.** Reuse ordinary per-run accounting, expose a minimal
parent relationship for continuations, and leave grouping of other calls and
overall totals to applications. This applies to both standalone continuations
and coordinated turns.
The current [observation boundary](../internal/observation.md) keeps nested
invocations independent. A callback's arbitrary model calls cannot simply be
assumed to belong to the enclosing interaction's totals.
**Recommendation:** preserve independent run accounting and add explicit
relationships between exchanges, branches, sequences, and associated runs.
Choose a small association mechanism, such as a scoped observation facility or
explicit execution references, so consumers need not rebuild lifecycle tracking.
Shared initial work should be referenced by branches and counted once when
reporting the combined operation. Report available usage with its existing
presence/completeness meaning; do not promise complete cost accounting.
**Accepted per-run accounting:** each continuation reports its own usage and
ordinary execution observations, including the existing distinctions between
transport attempts, structural repairs, and available failure-path diagnostics.
Keep existing usage presence/completeness semantics. The coordinator adds no
second accounting system or cumulative totals to its termination result.
**Alternative:** have PromptKit automatically capture every nested run during an
application callback. This minimizes explicit association but can include
unrelated work and changes the existing invocation-isolation boundary.
Another simpler option is to report continuation usage only and let applications
attach callback accounting; that leaves more work downstream and needs checking
against Audita's stated reporting requirement.
An initial exchange A that produces branches B and C contributes its usage only
to A. B and C do not inherit that usage. Applications reporting the combined work
can count A, B, and C once each. Usage reported for sending captured history again
in B or C belongs to those new executions and still counts. Reusing content does
not eliminate newly reported usage. Existing observation totals must not be
added to the attempt events they already summarize.
**Details to settle:** association of concurrent callback calls, observer
composition, failure-path diagnostics, and which totals include which child
executions. A turn budget would bound continuation attempts, not all provider
calls performed by structural repair, transport retries, or application work.
**Accepted relationship metadata:** retain the originating run identity with
each captured exchange and expose the immediate parent relationship in
continuation metadata. Use existing run identifiers without introducing mandatory
conversation, branch, or sequence identifiers. This supplies information that
PromptKit knows directly without requiring consumers to reconstruct which
exchange each continuation extends. Its exact public placement remains for
implementation planning and need not enlarge the coordinator's termination
result.
**Accepted application boundary:** model calls made inside a callback remain
independently observed and accounted for. PromptKit does not infer that they
belong to a continuation or aggregate merely because they execute inside its
callback. Applications explicitly group those existing observations when useful,
including concurrent calls. Audita retains assembly of its overall usage report;
PromptKit supplies execution facts without owning an application-wide reporting
model or promising complete cost accounting.
**Alternatives considered, not selected:**
- Automatically group nested calls. This could simplify obtaining an overall
total, but requires rules about which calls belong together and how observation
context propagates, and could include unrelated work.
- Add no relationship metadata. This would keep the API smaller, but observation
consumers would have to reconstruct continuation relationships themselves.
**Implementation planning:** place the minimal parent reference and preserve
ordinary per-request observation behavior for every continuation. Broader scoped
observation facilities or aggregate reporting require a separate demonstrated
need. The additional-turn budget continues to bound continuation executions,
not all provider calls performed inside those executions or application callbacks.
### 6. Where Should Continuation Inspection Live?
@@ -483,8 +512,11 @@ coverage should establish:
- Independent branches retaining their own history and outcomes.
- Assessment calls against the same backend at capacity one, without capacity
retained by the coordinator blocking them.
- Associated validator usage and shared initial usage reported without
double-counting, including incomplete usage and operational failures.
- Per-run usage and failure diagnostics available through ordinary observations,
including incomplete usage. Consumer examples explicitly group callback calls
and count shared initial work once, without coordinator-provided totals.
- Immediate parent relationships distinguish successive turns and independent
branches using existing run identities.
These are planning targets, not new implemented guarantees. Keep additional
mechanisms and constraints tied to the selected decisions and demonstrated