From 40cd4fe97dbc1ff3c34729e0190454d7cd9385c7 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 18 Nov 2026 20:04:48 +0000 Subject: [PATCH] Record agreement to retire the global LLM scheduler --- .../promptkit-architecture-discussion.md | 92 +++++++++++++++---- 1 file changed, 74 insertions(+), 18 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 3d7d9bfd..ede70fa8 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -5,7 +5,7 @@ Working notepad for reviewing the ownership boundaries in the [PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. -Items 1–3 record selected architectural directions. Detailed design, +Items 1–4 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. Item 7 records the completed Audita comparison and its implications for the @@ -19,8 +19,9 @@ features where PromptKit does not yet meet an application need. No application changes have been implemented through this notepad. The adoption roadmap still needs reconciliation with the selected direction, including its -adapter-preservation and latest-response-only language. Significant changes to -accepted architecture need successor ADRs under the +adapter-preservation, latest-response-only, and cross-backend scheduler +preservation language. Significant changes to accepted architecture need +successor ADRs under the [documentation policy](../policy/documentation.md#architecture-decision-records). Current boundaries are described in [Architecture](../policy/architecture.md) and [LLM Runtime Internals](../internal/llm.md). @@ -273,21 +274,75 @@ the retired message-reconstruction protocol rather than preserving its mechanics ## 4. Global Scheduling And Admission -Consider whether backend capacity limits and pipeline worker bounds meet the -operational need, or whether a separate cross-backend ceiling remains useful. -If it remains useful, consider implementing that reusable mechanism upstream. +**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and +retire the separate cross-backend ceiling. Rely on PromptKit's native backend +admission and capacity limits while keeping bounded Notarius pipeline workers. +This deliberately changes the existing global-limit contract; reproducing that +ceiling in PromptKit is not a migration prerequisite. No application changes +have been implemented. -Questions: +### Agreed Ownership -- What resource or operational budget does each limit protect? -- Should a global permit remain occupied during backend queueing and repair? -- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide - admission support? +| Concern | Owner | +| --- | --- | +| Bounded extraction jobs and lane continuations | Notarius pipeline workers | +| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit | +| Application disposition after admission or execution failure | Notarius application policy | -**Discussion / decision:** Pending. +For limited backends, PromptKit v0.23.2 bounds active model attempts by +`concurrency_limit` and admitted operations by that limit plus `queue_capacity`. +An operation holds admission +across preparation, retries, and structural repair, while active permits cover +individual model attempts and are released during retry waits. A completed +continuation turn releases admission before application assessment, so its +callback can invoke an LLM validator using the same backend. -**Follow-up:** Account for the existing global-limit contract. PromptKit -v0.23.2's per-backend limits do not directly replace it. +The current Notarius wrapper holds a global permit around the complete +structured call, including backend waiting and repairs. Removing it eliminates +that additional queue and avoids making independent backends share those +permits. Independent native backend limits may allow aggregate concurrency above +the former global ceiling; that is an accepted behavior change. Pipeline worker +bounds remain application work controls, not a replacement global LLM ceiling. + +### Removal And Configuration Consequences + +- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom + queue/cancellation/permit bookkeeping. +- Retire `concurrency.total_llm`. Give pipeline worker settings an independent + default and remove the extraction-worker maximum derived from `total_llm`. +- Use one shared execution engine for producers, validators, and repair turns. + Native capacity belongs to an engine and selected backend ID; separate engine + instances have independent pools. +- Configure native limits at the intended backend capacity boundary. Profiles + selecting the same backend ID share its pool; separate IDs pointing to the + same endpoint do not automatically share capacity. Endpoint-only profiles and + backends without a positive limit remain unrestricted by backend admission. +- Handle native `ErrCapacityExceeded` as an admission failure through application + policy, consistently with item 3's removal of blanket execution retries. +- Retire scheduler implementation tests with the removed code. Retain focused + integration checks for shared native admission across roles, independent + backends, bounded pipeline work, queue exhaustion, and cancellation. + +### Remaining Details + +Select independent worker defaults, native backend/queue defaults where the +application supplies them, and the configuration migration treatment of the +retired field. Specify how admission failures are reported and handled for each +application role. These details do not reopen the selected ownership boundary. + +A future demonstrated need for capacity shared across several backend IDs could +justify an upstream PromptKit feature request. No such feature is required for +this agreed design, and it does not justify retaining the Notarius scheduler. + +Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go), +[scheduler](../../internal/framework/llm/scheduler.go), +[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and +[native capacity behavior](../../../promptkit/docs/internal/capacity.md). + +**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language +and record the changed ownership in an ADR. Update architecture, configuration, +operations, and implementation documentation when the migration lands. Continue +with item 5's execution identity and checkpoint reuse discussion. ## 5. Execution Identity And Checkpoint Reuse @@ -452,8 +507,8 @@ and [prompt guidance](../../../audita/docs/internal/prompts.md). adapter-removal direction remains in force. - Proposal, validator, and repair roles share one engine and its per-backend admission. The runner separately bounds section work. This demonstrates useful - separation of application work scheduling and provider capacity, but does not - supply Notarius's independent cross-backend ceiling. + separation of application work scheduling and provider capacity. Item 4 now + selects that ownership and retires Notarius's independent cross-backend ceiling. - Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations feed a native `ExecutionGroup`; application records add logical-call and stage/section/proposal attribution. Nested validator executions enroll @@ -485,8 +540,9 @@ conversation assembly or the turn loop local. Item 3 owns the agreed Notarius flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details. -Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit -integration. Item 5 remains independent upstream/application identity work. +Item 4 now records agreement to use native admission with bounded application +workers. Item 6 can draw on Audita's observations and CommandKit integration. +Item 5 remains independent upstream/application identity work. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet