From f883316d7af7a3ffec7e803e56a393c468596fdb Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 20 Nov 2026 12:57:33 +0000 Subject: [PATCH] Resolve native module execution and pipeline assessment design --- .../promptkit-architecture-discussion.md | 33 ++++--- docs/roadmap/promptkit.md | 86 +++++++++++++++---- 2 files changed, 89 insertions(+), 30 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index d976d322..74a48608 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -68,9 +68,11 @@ resulting change to the adoption target remains to be selected. Engine construction becomes ordinary application wiring using item 2's embedded defaults and operator sources. Profile selection is expressed through native -requests. The current `internal/framework/llm` package can shrink to engine -assembly and focused helpers; whether those warrant a separate package remains -an organizational choice, not a reason to retain an adapter. +requests. U3 now places shared loaders and engine construction in `internal/cli` +and the producer helper beside pipeline assessment. Retire the emptied +`internal/framework/llm` package rather than retaining a constructor-only package. +The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns +the resolved design; implementation is pending. Use native requests, results, exchanges, errors, and observers directly in model-executing application code. Pure domain models, codecs, source-reference @@ -100,11 +102,19 @@ pipeline assessment, and native execution: and assessment for revised responses. The module supplies the repair request; PromptKit maintains history and enforces the turn budget. -A small shared application function can connect these responsibilities: the -module supplies native request construction and response interpretation, while -the pipeline supplies configured validation and terminal policy. It uses native -PromptKit types and returns a typed application outcome. It does not introduce -another completion API, conversation protocol, or retry engine. +The resolved U3 design uses a small shared helper in `internal/framework/pipeline`. +Modules supply native initial/correction requests and response interpretation; +the pipeline supplies an operation-scoped assessment callback and relevant +cache/debug collaborators. Local variables or closures hold operation state. +Use the same assessor for initial and revised candidates without an additional +final pass. Native preparation supports cache lookup before execution; hits load +the accepted artifact directly. The helper returns a typed application outcome +and saves only final accepted artifacts under the initial hash. + +Keep deterministic stages on their ordinary typed production/validation path. +LLM validators call the shared engine directly and return findings without an +automatic producer correction loop. No replacement completion API, execution-plan +language, or copied conversation model is selected. Item 3 owns the detailed continuation and assessment rules, including final-turn assessment, failure handling, complete replacement responses, and safe fallback. @@ -140,10 +150,9 @@ not require preserving a private completion interface. ### Remaining Implementation Work -- Select concrete module/pipeline function signatures that connect request - construction, interpretation, configured validation, and terminal disposition. -- Choose the location of engine assembly and focused application helpers without - introducing a replacement LLM service layer. +- Implement the resolved U3 ownership and placement; refine function signatures + and argument grouping against representative extraction and normalization. + These implementation details do not reopen the selected boundaries. - Implement item 2's resolved asset/schema design and settle the remaining migration details recorded under items 3–6, including durable compatibility. - Evaluate any remaining upstream gap against an actual application requirement, diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 2c74b885..6f266512 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -84,13 +84,54 @@ API. Pure domain models, codecs, source-reference resolution, and deterministic transformations need no PromptKit dependency. Application interfaces continue to express artifact production, typed validation, output encoding, and state access. -A small shared application function connects module request construction and -response interpretation to pipeline validation and terminal policy. It calls -native `Run` and `Coordinate` and returns a typed application outcome. It owns no -replacement completion API, conversation protocol, or general retry engine. -Exact signatures and helper placement remain open. The current -`internal/framework/llm` package may shrink to engine assembly and focused helpers -or cease to be a separate package. +#### Module Execution And Assessment (U3 Resolved) + +Keep the existing typed stage boundaries and use native PromptKit values in +model-executing code. Place engine construction and shared loaders in focused +files in the `internal/cli` composition root. It constructs one native engine +for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset +registry, and reporting responsibilities disappear; do not retain a package +solely to hold a small constructor or introduce another LLM service interface. + +Place one small shared producer helper alongside assessment in +`internal/framework/pipeline`. Connect modules to it through ordinary functions: + +1. The pipeline supplies an operation-scoped assessment callback capturing the + configured validator chain and acceptance/terminal policy. +2. The module prepares source material, references, and any normalization handle + mappings once. It builds the native initial request and supplies response + interpretation and correction-request functions. +3. The helper receives those functions, the shared engine, and relevant + cache/debug collaborators. For cacheable work it prepares the initial request + and looks up the accepted-artifact cache; on a miss it executes the retained + request. Otherwise it uses ordinary native execution. +4. For each executed response, the helper invokes module interpretation and + pipeline assessment. When correction is appropriate, it supplies the module's + next request to native coordination. PromptKit owns conversation state and + turn iteration; the helper does not implement another retry engine. +5. Return a typed application outcome. Cache only final accepted artifacts under + the initial request's key. A cache hit follows the existing direct-reuse policy + without model execution or renewed semantic assessment. + +Request builders return `promptkit.RunRequest`; interpreters consume native +results. Candidates, findings, and acceptance outcomes remain application-owned. +Keep operation-specific state in local variables or closures, without an +execution-plan language, replacement completion API, or copied conversation model. + +Use one pipeline assessment implementation for the initial candidate and every +correction, including the final permitted response. Do not repeat final validation +merely because the helper returns to the outer runner. Deterministic stages keep +their ordinary typed production/validation path without synthetic requests or +conversation machinery. LLM validators call the shared engine directly and return +domain findings; they do not automatically acquire a producer correction loop. + +Modules receive operation data and collaborators, not physical storage roots or +CommandKit reporters. Pure domain packages retain artifact models, codecs, +reference resolution, and deterministic rules. `internal/preflight` retains its +separate known-dependency checks. Exact argument grouping and helper signatures +can be refined against representative extraction and normalization during +implementation; ownership and placement are settled. Role-specific prompts, +budgets, and fallback policy remain U4–U6. Implementation is pending. Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` shape, ordered steps and generated references, typed artifact-family boundaries, @@ -711,7 +752,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. | | U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. | | U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | @@ -725,6 +765,12 @@ machinery are selected in [the schema design](#prompt-assets-and-response-schema Implementation remains in Phase 2; role-specific prompts and retry budgets remain U4/U5. +**U3 resolved:** Typed stage boundaries, composition-root engine construction, +and the shared pipeline producer helper with module functions and a pipeline +assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). +Implement in Phase 3 and refine signatures during Phase 4 without reopening +ownership. U4–U6 still own role-specific behavior. + **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. The former later CommandKit phase is retired. CLI compatibility is a non-goal; @@ -745,8 +791,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas, - CommandKit, debug, cache, or output choices as requirements to redesign. +- Preserve U4–U9 with their dependent phases; do not reopen resolved module + execution, assets/schemas, CommandKit, debug, cache, or output ownership. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, and outstanding choices are explicit. This phase does not claim that the hash @@ -770,8 +816,9 @@ without credentials or synthetic requests. `config diff` handles two selections. ### Phase 3: Prove A Representative Native Pipeline -- Resolve U3 and relevant U4–U6 details. Connect native generation, interpretation, - configured validation, and continuation through the application function. +- Implement the resolved U3 collaboration and settle relevant U4–U6 details. + Connect module request/interpretation functions to the pipeline assessment + callback through the shared producer helper; refine concrete signatures here. - Exercise an LLM validator on the same engine and normalization with fixed handles and fresh plan application. Select actual observer/group enrollment. - Run through the new command surface and alias, using the small completion @@ -779,9 +826,10 @@ without credentials or synthetic requests. `config diff` handles two selections. Begin U8/U9 integration without a provisional local message hash. **Completion evidence:** fake-provider workflows exercise accumulated history, -full candidate revalidation, final-turn acceptance, selected exhaustion/fallback, -and useful failures. The executable publishes artifacts and reports them through -native presentation without the private completion/correction API. +full candidate revalidation through the same assessor, final-turn acceptance +without duplicate final validation, selected exhaustion/fallback, and useful +failures. The executable publishes artifacts and reports them through native +presentation without the private completion/correction API. ### Phase 4: Migrate All Roles And Remove Superseded Machinery @@ -790,7 +838,9 @@ native presentation without the private completion/correction API. - Remove the adapter, copied completion contracts, manual replay/capability checks, outer LLM retries, global scheduler/ceiling, and coupled worker settings. - Finish native command/inspection migration and remove custom dispatch, flags, - encoders, verbosity handling, and obsolete asset registration. + encoders, verbosity handling, and obsolete asset registration. Retire the + emptied `internal/framework/llm` package after moving engine assembly to the + composition root and producer collaboration to the pipeline. **Completion evidence:** every role uses the shared native engine and selected admission policy. No production path depends on the private completion API, @@ -832,7 +882,7 @@ Release preparation/publication is governed separately by release policy. | Retire or replace | Retained application responsibility | | --- | --- | -| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. | +| `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. | | `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. | | Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | @@ -858,7 +908,7 @@ retry tests or freeze complete help text and native debug snapshots. | Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. | | Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. | | Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. | -| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. | +| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |