Record agreement to retire the global LLM scheduler
This commit is contained in:
@@ -5,7 +5,7 @@
|
|||||||
Working notepad for reviewing the ownership boundaries in the
|
Working notepad for reviewing the ownership boundaries in the
|
||||||
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
||||||
below in order, recording questions, evidence, decisions, and follow-up work.
|
below in order, recording questions, evidence, decisions, and follow-up work.
|
||||||
Items 1–3 record selected architectural directions. Detailed design,
|
Items 1–4 record selected architectural directions. Detailed design,
|
||||||
read-only filesystem and schema decisions, and the remaining topics are still
|
read-only filesystem and schema decisions, and the remaining topics are still
|
||||||
under discussion.
|
under discussion.
|
||||||
Item 7 records the completed Audita comparison and its implications for the
|
Item 7 records the completed Audita comparison and its implications for the
|
||||||
@@ -19,8 +19,9 @@ features where PromptKit does not yet meet an application need.
|
|||||||
|
|
||||||
No application changes have been implemented through this notepad. The adoption
|
No application changes have been implemented through this notepad. The adoption
|
||||||
roadmap still needs reconciliation with the selected direction, including its
|
roadmap still needs reconciliation with the selected direction, including its
|
||||||
adapter-preservation and latest-response-only language. Significant changes to
|
adapter-preservation, latest-response-only, and cross-backend scheduler
|
||||||
accepted architecture need successor ADRs under the
|
preservation language. Significant changes to accepted architecture need
|
||||||
|
successor ADRs under the
|
||||||
[documentation policy](../policy/documentation.md#architecture-decision-records).
|
[documentation policy](../policy/documentation.md#architecture-decision-records).
|
||||||
Current boundaries are described in [Architecture](../policy/architecture.md)
|
Current boundaries are described in [Architecture](../policy/architecture.md)
|
||||||
and [LLM Runtime Internals](../internal/llm.md).
|
and [LLM Runtime Internals](../internal/llm.md).
|
||||||
@@ -273,21 +274,75 @@ the retired message-reconstruction protocol rather than preserving its mechanics
|
|||||||
|
|
||||||
## 4. Global Scheduling And Admission
|
## 4. Global Scheduling And Admission
|
||||||
|
|
||||||
Consider whether backend capacity limits and pipeline worker bounds meet the
|
**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and
|
||||||
operational need, or whether a separate cross-backend ceiling remains useful.
|
retire the separate cross-backend ceiling. Rely on PromptKit's native backend
|
||||||
If it remains useful, consider implementing that reusable mechanism upstream.
|
admission and capacity limits while keeping bounded Notarius pipeline workers.
|
||||||
|
This deliberately changes the existing global-limit contract; reproducing that
|
||||||
|
ceiling in PromptKit is not a migration prerequisite. No application changes
|
||||||
|
have been implemented.
|
||||||
|
|
||||||
Questions:
|
### Agreed Ownership
|
||||||
|
|
||||||
- What resource or operational budget does each limit protect?
|
| Concern | Owner |
|
||||||
- Should a global permit remain occupied during backend queueing and repair?
|
| --- | --- |
|
||||||
- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide
|
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
|
||||||
admission support?
|
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
|
||||||
|
| Application disposition after admission or execution failure | Notarius application policy |
|
||||||
|
|
||||||
**Discussion / decision:** Pending.
|
For limited backends, PromptKit v0.23.2 bounds active model attempts by
|
||||||
|
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||||||
|
An operation holds admission
|
||||||
|
across preparation, retries, and structural repair, while active permits cover
|
||||||
|
individual model attempts and are released during retry waits. A completed
|
||||||
|
continuation turn releases admission before application assessment, so its
|
||||||
|
callback can invoke an LLM validator using the same backend.
|
||||||
|
|
||||||
**Follow-up:** Account for the existing global-limit contract. PromptKit
|
The current Notarius wrapper holds a global permit around the complete
|
||||||
v0.23.2's per-backend limits do not directly replace it.
|
structured call, including backend waiting and repairs. Removing it eliminates
|
||||||
|
that additional queue and avoids making independent backends share those
|
||||||
|
permits. Independent native backend limits may allow aggregate concurrency above
|
||||||
|
the former global ceiling; that is an accepted behavior change. Pipeline worker
|
||||||
|
bounds remain application work controls, not a replacement global LLM ceiling.
|
||||||
|
|
||||||
|
### Removal And Configuration Consequences
|
||||||
|
|
||||||
|
- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom
|
||||||
|
queue/cancellation/permit bookkeeping.
|
||||||
|
- Retire `concurrency.total_llm`. Give pipeline worker settings an independent
|
||||||
|
default and remove the extraction-worker maximum derived from `total_llm`.
|
||||||
|
- Use one shared execution engine for producers, validators, and repair turns.
|
||||||
|
Native capacity belongs to an engine and selected backend ID; separate engine
|
||||||
|
instances have independent pools.
|
||||||
|
- Configure native limits at the intended backend capacity boundary. Profiles
|
||||||
|
selecting the same backend ID share its pool; separate IDs pointing to the
|
||||||
|
same endpoint do not automatically share capacity. Endpoint-only profiles and
|
||||||
|
backends without a positive limit remain unrestricted by backend admission.
|
||||||
|
- Handle native `ErrCapacityExceeded` as an admission failure through application
|
||||||
|
policy, consistently with item 3's removal of blanket execution retries.
|
||||||
|
- Retire scheduler implementation tests with the removed code. Retain focused
|
||||||
|
integration checks for shared native admission across roles, independent
|
||||||
|
backends, bounded pipeline work, queue exhaustion, and cancellation.
|
||||||
|
|
||||||
|
### Remaining Details
|
||||||
|
|
||||||
|
Select independent worker defaults, native backend/queue defaults where the
|
||||||
|
application supplies them, and the configuration migration treatment of the
|
||||||
|
retired field. Specify how admission failures are reported and handled for each
|
||||||
|
application role. These details do not reopen the selected ownership boundary.
|
||||||
|
|
||||||
|
A future demonstrated need for capacity shared across several backend IDs could
|
||||||
|
justify an upstream PromptKit feature request. No such feature is required for
|
||||||
|
this agreed design, and it does not justify retaining the Notarius scheduler.
|
||||||
|
|
||||||
|
Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
|
||||||
|
[scheduler](../../internal/framework/llm/scheduler.go),
|
||||||
|
[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and
|
||||||
|
[native capacity behavior](../../../promptkit/docs/internal/capacity.md).
|
||||||
|
|
||||||
|
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
|
||||||
|
and record the changed ownership in an ADR. Update architecture, configuration,
|
||||||
|
operations, and implementation documentation when the migration lands. Continue
|
||||||
|
with item 5's execution identity and checkpoint reuse discussion.
|
||||||
|
|
||||||
## 5. Execution Identity And Checkpoint Reuse
|
## 5. Execution Identity And Checkpoint Reuse
|
||||||
|
|
||||||
@@ -452,8 +507,8 @@ and [prompt guidance](../../../audita/docs/internal/prompts.md).
|
|||||||
adapter-removal direction remains in force.
|
adapter-removal direction remains in force.
|
||||||
- Proposal, validator, and repair roles share one engine and its per-backend
|
- Proposal, validator, and repair roles share one engine and its per-backend
|
||||||
admission. The runner separately bounds section work. This demonstrates useful
|
admission. The runner separately bounds section work. This demonstrates useful
|
||||||
separation of application work scheduling and provider capacity, but does not
|
separation of application work scheduling and provider capacity. Item 4 now
|
||||||
supply Notarius's independent cross-backend ceiling.
|
selects that ownership and retires Notarius's independent cross-backend ceiling.
|
||||||
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
|
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
|
||||||
feed a native `ExecutionGroup`; application records add logical-call and
|
feed a native `ExecutionGroup`; application records add logical-call and
|
||||||
stage/section/proposal attribution. Nested validator executions enroll
|
stage/section/proposal attribution. Nested validator executions enroll
|
||||||
@@ -485,8 +540,9 @@ conversation assembly or the turn loop local. Item 3 owns the agreed Notarius
|
|||||||
flow and the remaining repair-prompt, response-contract, and policy details;
|
flow and the remaining repair-prompt, response-contract, and policy details;
|
||||||
the comparison alone did not settle those details.
|
the comparison alone did not settle those details.
|
||||||
|
|
||||||
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit
|
Item 4 now records agreement to use native admission with bounded application
|
||||||
integration. Item 5 remains independent upstream/application identity work.
|
workers. Item 6 can draw on Audita's observations and CommandKit integration.
|
||||||
|
Item 5 remains independent upstream/application identity work.
|
||||||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||||||
feature is needed merely to provide replaceable defaults or domain-directed
|
feature is needed merely to provide replaceable defaults or domain-directed
|
||||||
continuation. Any remaining upstream request should identify the specific unmet
|
continuation. Any remaining upstream request should identify the specific unmet
|
||||||
|
|||||||
Reference in New Issue
Block a user