Record per-run accounting and continuation relationships
This commit is contained in:
@@ -54,9 +54,10 @@ The proposed capabilities are:
|
|||||||
|
|
||||||
Continuation has direct value by removing consumer capture, reconstruction,
|
Continuation has direct value by removing consumer capture, reconstruction,
|
||||||
prefix comparison, and history assembly. The coordinator centralizes turn
|
prefix comparison, and history assembly. The coordinator centralizes turn
|
||||||
progression, execution relationships, cancellation, budgets, and diagnostics.
|
progression, cancellation, and budgets while reusing ordinary execution
|
||||||
Its value should come from removing that repeated lifecycle work, beyond merely
|
observations. Captured exchanges supply the minimal parent relationship selected
|
||||||
replacing a consumer's loop counter.
|
in Decision 5. Its value comes from removing repeated lifecycle work, beyond
|
||||||
|
merely replacing a consumer's loop counter.
|
||||||
Audita would continue owning candidate selection, domain validation, feedback
|
Audita would continue owning candidate selection, domain validation, feedback
|
||||||
meaning, branch scheduling, accepted sibling proposals, and transcript changes.
|
meaning, branch scheduling, accepted sibling proposals, and transcript changes.
|
||||||
Its domain-specific repair code would not move wholesale into PromptKit.
|
Its domain-specific repair code would not move wholesale into PromptKit.
|
||||||
@@ -178,9 +179,9 @@ interpret the purpose of that work.
|
|||||||
**Implementation planning:** select exact request/stop and outcome types under
|
**Implementation planning:** select exact request/stop and outcome types under
|
||||||
Decision 4's accepted response delivery, turn-budget consumption, zero-turn
|
Decision 4's accepted response delivery, turn-budget consumption, zero-turn
|
||||||
behavior, cancellation, and minimal error/result semantics.
|
behavior, cancellation, and minimal error/result semantics.
|
||||||
Execution association and usage accounting belong to Decision 5. The sketches
|
Decision 5 records the selected execution association and accounting approach.
|
||||||
below illustrate the selected responsibility split; they do not settle the
|
The sketches below illustrate the selected responsibility split; they do not
|
||||||
remaining public contracts.
|
settle the remaining public contracts.
|
||||||
|
|
||||||
#### Illustrative Consumer Sketches
|
#### Illustrative Consumer Sketches
|
||||||
|
|
||||||
@@ -270,7 +271,8 @@ already receive each completed response through the callback. Completed work
|
|||||||
must not be mistaken for application acceptance of its content.
|
must not be mistaken for application acceptance of its content.
|
||||||
The concrete interface should let both examples use the same coordination
|
The concrete interface should let both examples use the same coordination
|
||||||
mechanism without manual message-history assembly or per-turn lifecycle
|
mechanism without manual message-history assembly or per-turn lifecycle
|
||||||
bookkeeping. Callback model-call association remains part of Decision 5.
|
bookkeeping. Calls made inside callbacks retain independent observations;
|
||||||
|
applications explicitly group them as selected in Decision 5.
|
||||||
|
|
||||||
### 4. How Are Application Next Actions And Execution Outcomes Represented?
|
### 4. How Are Application Next Actions And Execution Outcomes Represented?
|
||||||
|
|
||||||
@@ -386,8 +388,9 @@ Failure-stage fields, additional run-ID references, a separate failure
|
|||||||
classification hierarchy, and aggregate reporting are not part of this minimal
|
classification hierarchy, and aggregate reporting are not part of this minimal
|
||||||
termination contract. They require a demonstrated consumer need before addition.
|
termination contract. They require a demonstrated consumer need before addition.
|
||||||
This does not remove fields from existing response values or observation APIs.
|
This does not remove fields from existing response values or observation APIs.
|
||||||
Decision 5 separately evaluates accounting, and Decision 6 inspection, without
|
Decision 5 reuses ordinary run accounting without expanding this result.
|
||||||
presuming that either needs to expand the coordinator's termination result.
|
Decision 6 separately evaluates inspection without presuming that it needs to
|
||||||
|
expand the coordinator's termination result.
|
||||||
|
|
||||||
Audita retains its own meanings for acceptance, rejection, withdrawal, and
|
Audita retains its own meanings for acceptance, rejection, withdrawal, and
|
||||||
assessment unavailability. A normal stop does not assert any of those outcomes.
|
assessment unavailability. A normal stop does not assert any of those outcomes.
|
||||||
@@ -408,33 +411,59 @@ result.
|
|||||||
|
|
||||||
### 5. How Are Related Executions And Usage Accounted For?
|
### 5. How Are Related Executions And Usage Accounted For?
|
||||||
|
|
||||||
**Status: open.** Define accounting for the initial exchange, each continuation,
|
**Status: accepted; exact placement of relationship metadata remains for
|
||||||
its structural repairs and transport retries, and model calls made by
|
implementation planning.** Reuse ordinary per-run accounting, expose a minimal
|
||||||
application callbacks, both for standalone continuations and coordinated turns.
|
parent relationship for continuations, and leave grouping of other calls and
|
||||||
|
overall totals to applications. This applies to both standalone continuations
|
||||||
|
and coordinated turns.
|
||||||
|
|
||||||
The current [observation boundary](../internal/observation.md) keeps nested
|
The current [observation boundary](../internal/observation.md) keeps nested
|
||||||
invocations independent. A callback's arbitrary model calls cannot simply be
|
invocations independent. A callback's arbitrary model calls cannot simply be
|
||||||
assumed to belong to the enclosing interaction's totals.
|
assumed to belong to the enclosing interaction's totals.
|
||||||
|
|
||||||
**Recommendation:** preserve independent run accounting and add explicit
|
**Accepted per-run accounting:** each continuation reports its own usage and
|
||||||
relationships between exchanges, branches, sequences, and associated runs.
|
ordinary execution observations, including the existing distinctions between
|
||||||
Choose a small association mechanism, such as a scoped observation facility or
|
transport attempts, structural repairs, and available failure-path diagnostics.
|
||||||
explicit execution references, so consumers need not rebuild lifecycle tracking.
|
Keep existing usage presence/completeness semantics. The coordinator adds no
|
||||||
Shared initial work should be referenced by branches and counted once when
|
second accounting system or cumulative totals to its termination result.
|
||||||
reporting the combined operation. Report available usage with its existing
|
|
||||||
presence/completeness meaning; do not promise complete cost accounting.
|
|
||||||
|
|
||||||
**Alternative:** have PromptKit automatically capture every nested run during an
|
An initial exchange A that produces branches B and C contributes its usage only
|
||||||
application callback. This minimizes explicit association but can include
|
to A. B and C do not inherit that usage. Applications reporting the combined work
|
||||||
unrelated work and changes the existing invocation-isolation boundary.
|
can count A, B, and C once each. Usage reported for sending captured history again
|
||||||
Another simpler option is to report continuation usage only and let applications
|
in B or C belongs to those new executions and still counts. Reusing content does
|
||||||
attach callback accounting; that leaves more work downstream and needs checking
|
not eliminate newly reported usage. Existing observation totals must not be
|
||||||
against Audita's stated reporting requirement.
|
added to the attempt events they already summarize.
|
||||||
|
|
||||||
**Details to settle:** association of concurrent callback calls, observer
|
**Accepted relationship metadata:** retain the originating run identity with
|
||||||
composition, failure-path diagnostics, and which totals include which child
|
each captured exchange and expose the immediate parent relationship in
|
||||||
executions. A turn budget would bound continuation attempts, not all provider
|
continuation metadata. Use existing run identifiers without introducing mandatory
|
||||||
calls performed by structural repair, transport retries, or application work.
|
conversation, branch, or sequence identifiers. This supplies information that
|
||||||
|
PromptKit knows directly without requiring consumers to reconstruct which
|
||||||
|
exchange each continuation extends. Its exact public placement remains for
|
||||||
|
implementation planning and need not enlarge the coordinator's termination
|
||||||
|
result.
|
||||||
|
|
||||||
|
**Accepted application boundary:** model calls made inside a callback remain
|
||||||
|
independently observed and accounted for. PromptKit does not infer that they
|
||||||
|
belong to a continuation or aggregate merely because they execute inside its
|
||||||
|
callback. Applications explicitly group those existing observations when useful,
|
||||||
|
including concurrent calls. Audita retains assembly of its overall usage report;
|
||||||
|
PromptKit supplies execution facts without owning an application-wide reporting
|
||||||
|
model or promising complete cost accounting.
|
||||||
|
|
||||||
|
**Alternatives considered, not selected:**
|
||||||
|
|
||||||
|
- Automatically group nested calls. This could simplify obtaining an overall
|
||||||
|
total, but requires rules about which calls belong together and how observation
|
||||||
|
context propagates, and could include unrelated work.
|
||||||
|
- Add no relationship metadata. This would keep the API smaller, but observation
|
||||||
|
consumers would have to reconstruct continuation relationships themselves.
|
||||||
|
|
||||||
|
**Implementation planning:** place the minimal parent reference and preserve
|
||||||
|
ordinary per-request observation behavior for every continuation. Broader scoped
|
||||||
|
observation facilities or aggregate reporting require a separate demonstrated
|
||||||
|
need. The additional-turn budget continues to bound continuation executions,
|
||||||
|
not all provider calls performed inside those executions or application callbacks.
|
||||||
|
|
||||||
### 6. Where Should Continuation Inspection Live?
|
### 6. Where Should Continuation Inspection Live?
|
||||||
|
|
||||||
@@ -483,8 +512,11 @@ coverage should establish:
|
|||||||
- Independent branches retaining their own history and outcomes.
|
- Independent branches retaining their own history and outcomes.
|
||||||
- Assessment calls against the same backend at capacity one, without capacity
|
- Assessment calls against the same backend at capacity one, without capacity
|
||||||
retained by the coordinator blocking them.
|
retained by the coordinator blocking them.
|
||||||
- Associated validator usage and shared initial usage reported without
|
- Per-run usage and failure diagnostics available through ordinary observations,
|
||||||
double-counting, including incomplete usage and operational failures.
|
including incomplete usage. Consumer examples explicitly group callback calls
|
||||||
|
and count shared initial work once, without coordinator-provided totals.
|
||||||
|
- Immediate parent relationships distinguish successive turns and independent
|
||||||
|
branches using existing run identities.
|
||||||
|
|
||||||
These are planning targets, not new implemented guarantees. Keep additional
|
These are planning targets, not new implemented guarantees. Keep additional
|
||||||
mechanisms and constraints tied to the selected decisions and demonstrated
|
mechanisms and constraints tied to the selected decisions and demonstrated
|
||||||
|
|||||||
Reference in New Issue
Block a user