Record per-run accounting and continuation relationships

This commit is contained in:
2026-11-13 16:33:21 +00:00
parent ee810b8fbe
commit 45bbf7bb2c

View File

@@ -54,9 +54,10 @@ The proposed capabilities are:
Continuation has direct value by removing consumer capture, reconstruction, Continuation has direct value by removing consumer capture, reconstruction,
prefix comparison, and history assembly. The coordinator centralizes turn prefix comparison, and history assembly. The coordinator centralizes turn
progression, execution relationships, cancellation, budgets, and diagnostics. progression, cancellation, and budgets while reusing ordinary execution
Its value should come from removing that repeated lifecycle work, beyond merely observations. Captured exchanges supply the minimal parent relationship selected
replacing a consumer's loop counter. in Decision 5. Its value comes from removing repeated lifecycle work, beyond
merely replacing a consumer's loop counter.
Audita would continue owning candidate selection, domain validation, feedback Audita would continue owning candidate selection, domain validation, feedback
meaning, branch scheduling, accepted sibling proposals, and transcript changes. meaning, branch scheduling, accepted sibling proposals, and transcript changes.
Its domain-specific repair code would not move wholesale into PromptKit. Its domain-specific repair code would not move wholesale into PromptKit.
@@ -178,9 +179,9 @@ interpret the purpose of that work.
**Implementation planning:** select exact request/stop and outcome types under **Implementation planning:** select exact request/stop and outcome types under
Decision 4's accepted response delivery, turn-budget consumption, zero-turn Decision 4's accepted response delivery, turn-budget consumption, zero-turn
behavior, cancellation, and minimal error/result semantics. behavior, cancellation, and minimal error/result semantics.
Execution association and usage accounting belong to Decision 5. The sketches Decision 5 records the selected execution association and accounting approach.
below illustrate the selected responsibility split; they do not settle the The sketches below illustrate the selected responsibility split; they do not
remaining public contracts. settle the remaining public contracts.
#### Illustrative Consumer Sketches #### Illustrative Consumer Sketches
@@ -270,7 +271,8 @@ already receive each completed response through the callback. Completed work
must not be mistaken for application acceptance of its content. must not be mistaken for application acceptance of its content.
The concrete interface should let both examples use the same coordination The concrete interface should let both examples use the same coordination
mechanism without manual message-history assembly or per-turn lifecycle mechanism without manual message-history assembly or per-turn lifecycle
bookkeeping. Callback model-call association remains part of Decision 5. bookkeeping. Calls made inside callbacks retain independent observations;
applications explicitly group them as selected in Decision 5.
### 4. How Are Application Next Actions And Execution Outcomes Represented? ### 4. How Are Application Next Actions And Execution Outcomes Represented?
@@ -386,8 +388,9 @@ Failure-stage fields, additional run-ID references, a separate failure
classification hierarchy, and aggregate reporting are not part of this minimal classification hierarchy, and aggregate reporting are not part of this minimal
termination contract. They require a demonstrated consumer need before addition. termination contract. They require a demonstrated consumer need before addition.
This does not remove fields from existing response values or observation APIs. This does not remove fields from existing response values or observation APIs.
Decision 5 separately evaluates accounting, and Decision 6 inspection, without Decision 5 reuses ordinary run accounting without expanding this result.
presuming that either needs to expand the coordinator's termination result. Decision 6 separately evaluates inspection without presuming that it needs to
expand the coordinator's termination result.
Audita retains its own meanings for acceptance, rejection, withdrawal, and Audita retains its own meanings for acceptance, rejection, withdrawal, and
assessment unavailability. A normal stop does not assert any of those outcomes. assessment unavailability. A normal stop does not assert any of those outcomes.
@@ -408,33 +411,59 @@ result.
### 5. How Are Related Executions And Usage Accounted For? ### 5. How Are Related Executions And Usage Accounted For?
**Status: open.** Define accounting for the initial exchange, each continuation, **Status: accepted; exact placement of relationship metadata remains for
its structural repairs and transport retries, and model calls made by implementation planning.** Reuse ordinary per-run accounting, expose a minimal
application callbacks, both for standalone continuations and coordinated turns. parent relationship for continuations, and leave grouping of other calls and
overall totals to applications. This applies to both standalone continuations
and coordinated turns.
The current [observation boundary](../internal/observation.md) keeps nested The current [observation boundary](../internal/observation.md) keeps nested
invocations independent. A callback's arbitrary model calls cannot simply be invocations independent. A callback's arbitrary model calls cannot simply be
assumed to belong to the enclosing interaction's totals. assumed to belong to the enclosing interaction's totals.
**Recommendation:** preserve independent run accounting and add explicit **Accepted per-run accounting:** each continuation reports its own usage and
relationships between exchanges, branches, sequences, and associated runs. ordinary execution observations, including the existing distinctions between
Choose a small association mechanism, such as a scoped observation facility or transport attempts, structural repairs, and available failure-path diagnostics.
explicit execution references, so consumers need not rebuild lifecycle tracking. Keep existing usage presence/completeness semantics. The coordinator adds no
Shared initial work should be referenced by branches and counted once when second accounting system or cumulative totals to its termination result.
reporting the combined operation. Report available usage with its existing
presence/completeness meaning; do not promise complete cost accounting.
**Alternative:** have PromptKit automatically capture every nested run during an An initial exchange A that produces branches B and C contributes its usage only
application callback. This minimizes explicit association but can include to A. B and C do not inherit that usage. Applications reporting the combined work
unrelated work and changes the existing invocation-isolation boundary. can count A, B, and C once each. Usage reported for sending captured history again
Another simpler option is to report continuation usage only and let applications in B or C belongs to those new executions and still counts. Reusing content does
attach callback accounting; that leaves more work downstream and needs checking not eliminate newly reported usage. Existing observation totals must not be
against Audita's stated reporting requirement. added to the attempt events they already summarize.
**Details to settle:** association of concurrent callback calls, observer **Accepted relationship metadata:** retain the originating run identity with
composition, failure-path diagnostics, and which totals include which child each captured exchange and expose the immediate parent relationship in
executions. A turn budget would bound continuation attempts, not all provider continuation metadata. Use existing run identifiers without introducing mandatory
calls performed by structural repair, transport retries, or application work. conversation, branch, or sequence identifiers. This supplies information that
PromptKit knows directly without requiring consumers to reconstruct which
exchange each continuation extends. Its exact public placement remains for
implementation planning and need not enlarge the coordinator's termination
result.
**Accepted application boundary:** model calls made inside a callback remain
independently observed and accounted for. PromptKit does not infer that they
belong to a continuation or aggregate merely because they execute inside its
callback. Applications explicitly group those existing observations when useful,
including concurrent calls. Audita retains assembly of its overall usage report;
PromptKit supplies execution facts without owning an application-wide reporting
model or promising complete cost accounting.
**Alternatives considered, not selected:**
- Automatically group nested calls. This could simplify obtaining an overall
total, but requires rules about which calls belong together and how observation
context propagates, and could include unrelated work.
- Add no relationship metadata. This would keep the API smaller, but observation
consumers would have to reconstruct continuation relationships themselves.
**Implementation planning:** place the minimal parent reference and preserve
ordinary per-request observation behavior for every continuation. Broader scoped
observation facilities or aggregate reporting require a separate demonstrated
need. The additional-turn budget continues to bound continuation executions,
not all provider calls performed inside those executions or application callbacks.
### 6. Where Should Continuation Inspection Live? ### 6. Where Should Continuation Inspection Live?
@@ -483,8 +512,11 @@ coverage should establish:
- Independent branches retaining their own history and outcomes. - Independent branches retaining their own history and outcomes.
- Assessment calls against the same backend at capacity one, without capacity - Assessment calls against the same backend at capacity one, without capacity
retained by the coordinator blocking them. retained by the coordinator blocking them.
- Associated validator usage and shared initial usage reported without - Per-run usage and failure diagnostics available through ordinary observations,
double-counting, including incomplete usage and operational failures. including incomplete usage. Consumer examples explicitly group callback calls
and count shared initial work once, without coordinator-provided totals.
- Immediate parent relationships distinguish successive turns and independent
branches using existing run identities.
These are planning targets, not new implemented guarantees. Keep additional These are planning targets, not new implemented guarantees. Keep additional
mechanisms and constraints tied to the selected decisions and demonstrated mechanisms and constraints tied to the selected decisions and demonstrated