Resolve native module execution and pipeline assessment design

This commit is contained in:
2026-11-20 12:57:33 +00:00
parent 0833450c71
commit f883316d7a
2 changed files with 89 additions and 30 deletions

View File

@@ -68,9 +68,11 @@ resulting change to the adoption target remains to be selected.
Engine construction becomes ordinary application wiring using item 2's embedded Engine construction becomes ordinary application wiring using item 2's embedded
defaults and operator sources. Profile selection is expressed through native defaults and operator sources. Profile selection is expressed through native
requests. The current `internal/framework/llm` package can shrink to engine requests. U3 now places shared loaders and engine construction in `internal/cli`
assembly and focused helpers; whether those warrant a separate package remains and the producer helper beside pipeline assessment. Retire the emptied
an organizational choice, not a reason to retain an adapter. `internal/framework/llm` package rather than retaining a constructor-only package.
The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns
the resolved design; implementation is pending.
Use native requests, results, exchanges, errors, and observers directly in Use native requests, results, exchanges, errors, and observers directly in
model-executing application code. Pure domain models, codecs, source-reference model-executing application code. Pure domain models, codecs, source-reference
@@ -100,11 +102,19 @@ pipeline assessment, and native execution:
and assessment for revised responses. The module supplies the repair request; and assessment for revised responses. The module supplies the repair request;
PromptKit maintains history and enforces the turn budget. PromptKit maintains history and enforces the turn budget.
A small shared application function can connect these responsibilities: the The resolved U3 design uses a small shared helper in `internal/framework/pipeline`.
module supplies native request construction and response interpretation, while Modules supply native initial/correction requests and response interpretation;
the pipeline supplies configured validation and terminal policy. It uses native the pipeline supplies an operation-scoped assessment callback and relevant
PromptKit types and returns a typed application outcome. It does not introduce cache/debug collaborators. Local variables or closures hold operation state.
another completion API, conversation protocol, or retry engine. Use the same assessor for initial and revised candidates without an additional
final pass. Native preparation supports cache lookup before execution; hits load
the accepted artifact directly. The helper returns a typed application outcome
and saves only final accepted artifacts under the initial hash.
Keep deterministic stages on their ordinary typed production/validation path.
LLM validators call the shared engine directly and return findings without an
automatic producer correction loop. No replacement completion API, execution-plan
language, or copied conversation model is selected.
Item 3 owns the detailed continuation and assessment rules, including final-turn Item 3 owns the detailed continuation and assessment rules, including final-turn
assessment, failure handling, complete replacement responses, and safe fallback. assessment, failure handling, complete replacement responses, and safe fallback.
@@ -140,10 +150,9 @@ not require preserving a private completion interface.
### Remaining Implementation Work ### Remaining Implementation Work
- Select concrete module/pipeline function signatures that connect request - Implement the resolved U3 ownership and placement; refine function signatures
construction, interpretation, configured validation, and terminal disposition. and argument grouping against representative extraction and normalization.
- Choose the location of engine assembly and focused application helpers without These implementation details do not reopen the selected boundaries.
introducing a replacement LLM service layer.
- Implement item 2's resolved asset/schema design and settle the remaining - Implement item 2's resolved asset/schema design and settle the remaining
migration details recorded under items 3–6, including durable compatibility. migration details recorded under items 3–6, including durable compatibility.
- Evaluate any remaining upstream gap against an actual application requirement, - Evaluate any remaining upstream gap against an actual application requirement,

View File

@@ -84,13 +84,54 @@ API. Pure domain models, codecs, source-reference resolution, and deterministic
transformations need no PromptKit dependency. Application interfaces continue to transformations need no PromptKit dependency. Application interfaces continue to
express artifact production, typed validation, output encoding, and state access. express artifact production, typed validation, output encoding, and state access.
A small shared application function connects module request construction and #### Module Execution And Assessment (U3 Resolved)
response interpretation to pipeline validation and terminal policy. It calls
native `Run` and `Coordinate` and returns a typed application outcome. It owns no Keep the existing typed stage boundaries and use native PromptKit values in
replacement completion API, conversation protocol, or general retry engine. model-executing code. Place engine construction and shared loaders in focused
Exact signatures and helper placement remain open. The current files in the `internal/cli` composition root. It constructs one native engine
`internal/framework/llm` package may shrink to engine assembly and focused helpers for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset
or cease to be a separate package. registry, and reporting responsibilities disappear; do not retain a package
solely to hold a small constructor or introduce another LLM service interface.
Place one small shared producer helper alongside assessment in
`internal/framework/pipeline`. Connect modules to it through ordinary functions:
1. The pipeline supplies an operation-scoped assessment callback capturing the
configured validator chain and acceptance/terminal policy.
2. The module prepares source material, references, and any normalization handle
mappings once. It builds the native initial request and supplies response
interpretation and correction-request functions.
3. The helper receives those functions, the shared engine, and relevant
cache/debug collaborators. For cacheable work it prepares the initial request
and looks up the accepted-artifact cache; on a miss it executes the retained
request. Otherwise it uses ordinary native execution.
4. For each executed response, the helper invokes module interpretation and
pipeline assessment. When correction is appropriate, it supplies the module's
next request to native coordination. PromptKit owns conversation state and
turn iteration; the helper does not implement another retry engine.
5. Return a typed application outcome. Cache only final accepted artifacts under
the initial request's key. A cache hit follows the existing direct-reuse policy
without model execution or renewed semantic assessment.
Request builders return `promptkit.RunRequest`; interpreters consume native
results. Candidates, findings, and acceptance outcomes remain application-owned.
Keep operation-specific state in local variables or closures, without an
execution-plan language, replacement completion API, or copied conversation model.
Use one pipeline assessment implementation for the initial candidate and every
correction, including the final permitted response. Do not repeat final validation
merely because the helper returns to the outer runner. Deterministic stages keep
their ordinary typed production/validation path without synthetic requests or
conversation machinery. LLM validators call the shared engine directly and return
domain findings; they do not automatically acquire a producer correction loop.
Modules receive operation data and collaborators, not physical storage roots or
CommandKit reporters. Pure domain packages retain artifact models, codecs,
reference resolution, and deterministic rules. `internal/preflight` retains its
separate known-dependency checks. Exact argument grouping and helper signatures
can be refined against representative extraction and normalization during
implementation; ownership and placement are settled. Role-specific prompts,
budgets, and fallback policy remain U4–U6. Implementation is pending.
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
shape, ordered steps and generated references, typed artifact-family boundaries, shape, ordered steps and generated references, typed artifact-family boundaries,
@@ -711,7 +752,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
| ID | Remaining choice or dependency | Resolve for | | ID | Remaining choice or dependency | Resolve for |
| --- | --- | --- | | --- | --- | --- |
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. | | U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. |
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. | | U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | | U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
@@ -725,6 +765,12 @@ machinery are selected in [the schema design](#prompt-assets-and-response-schema
Implementation remains in Phase 2; role-specific prompts and retry budgets remain Implementation remains in Phase 2; role-specific prompts and retry budgets remain
U4/U5. U4/U5.
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
and the shared pipeline producer helper with module functions and a pipeline
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
Implement in Phase 3 and refine signatures during Phase 4 without reopening
ownership. U4–U6 still own role-specific behavior.
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
output, debug policy, and coordinated cutover sequencing are selected above. output, debug policy, and coordinated cutover sequencing are selected above.
The former later CommandKit phase is retired. CLI compatibility is a non-goal; The former later CommandKit phase is retired. CLI compatibility is a non-goal;
@@ -745,8 +791,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
Keep the cache dependency explicit without blocking independent work. Keep the cache dependency explicit without blocking independent work.
- Map affected ADRs, current callers, contracts, and removal inventory. Carry - Map affected ADRs, current callers, contracts, and removal inventory. Carry
forward Audita evidence without adopting its application wrappers. forward Audita evidence without adopting its application wrappers.
- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas, - Preserve U4–U9 with their dependent phases; do not reopen resolved module
CommandKit, debug, cache, or output choices as requirements to redesign. execution, assets/schemas, CommandKit, debug, cache, or output ownership.
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope, **Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
and outstanding choices are explicit. This phase does not claim that the hash and outstanding choices are explicit. This phase does not claim that the hash
@@ -770,8 +816,9 @@ without credentials or synthetic requests. `config diff` handles two selections.
### Phase 3: Prove A Representative Native Pipeline ### Phase 3: Prove A Representative Native Pipeline
- Resolve U3 and relevant U4–U6 details. Connect native generation, interpretation, - Implement the resolved U3 collaboration and settle relevant U4–U6 details.
configured validation, and continuation through the application function. Connect module request/interpretation functions to the pipeline assessment
callback through the shared producer helper; refine concrete signatures here.
- Exercise an LLM validator on the same engine and normalization with fixed - Exercise an LLM validator on the same engine and normalization with fixed
handles and fresh plan application. Select actual observer/group enrollment. handles and fresh plan application. Select actual observer/group enrollment.
- Run through the new command surface and alias, using the small completion - Run through the new command surface and alias, using the small completion
@@ -779,9 +826,10 @@ without credentials or synthetic requests. `config diff` handles two selections.
Begin U8/U9 integration without a provisional local message hash. Begin U8/U9 integration without a provisional local message hash.
**Completion evidence:** fake-provider workflows exercise accumulated history, **Completion evidence:** fake-provider workflows exercise accumulated history,
full candidate revalidation, final-turn acceptance, selected exhaustion/fallback, full candidate revalidation through the same assessor, final-turn acceptance
and useful failures. The executable publishes artifacts and reports them through without duplicate final validation, selected exhaustion/fallback, and useful
native presentation without the private completion/correction API. failures. The executable publishes artifacts and reports them through native
presentation without the private completion/correction API.
### Phase 4: Migrate All Roles And Remove Superseded Machinery ### Phase 4: Migrate All Roles And Remove Superseded Machinery
@@ -790,7 +838,9 @@ native presentation without the private completion/correction API.
- Remove the adapter, copied completion contracts, manual replay/capability checks, - Remove the adapter, copied completion contracts, manual replay/capability checks,
outer LLM retries, global scheduler/ceiling, and coupled worker settings. outer LLM retries, global scheduler/ceiling, and coupled worker settings.
- Finish native command/inspection migration and remove custom dispatch, flags, - Finish native command/inspection migration and remove custom dispatch, flags,
encoders, verbosity handling, and obsolete asset registration. encoders, verbosity handling, and obsolete asset registration. Retire the
emptied `internal/framework/llm` package after moving engine assembly to the
composition root and producer collaboration to the pipeline.
**Completion evidence:** every role uses the shared native engine and selected **Completion evidence:** every role uses the shared native engine and selected
admission policy. No production path depends on the private completion API, admission policy. No production path depends on the private completion API,
@@ -832,7 +882,7 @@ Release preparation/publication is governed separately by release policy.
| Retire or replace | Retained application responsibility | | Retire or replace | Retained application responsibility |
| --- | --- | | --- | --- |
| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. | | `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. |
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. | | `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
@@ -858,7 +908,7 @@ retry tests or freeze complete help text and native debug snapshots.
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. | | Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. | | Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. | | Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. | | Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |