Resolve native module execution and pipeline assessment design

This commit is contained in:
2026-11-20 12:57:33 +00:00
parent 0833450c71
commit f883316d7a
2 changed files with 89 additions and 30 deletions

View File

@@ -68,9 +68,11 @@ resulting change to the adoption target remains to be selected.
Engine construction becomes ordinary application wiring using item 2's embedded
defaults and operator sources. Profile selection is expressed through native
requests. The current `internal/framework/llm` package can shrink to engine
assembly and focused helpers; whether those warrant a separate package remains
an organizational choice, not a reason to retain an adapter.
requests. U3 now places shared loaders and engine construction in `internal/cli`
and the producer helper beside pipeline assessment. Retire the emptied
`internal/framework/llm` package rather than retaining a constructor-only package.
The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns
the resolved design; implementation is pending.
Use native requests, results, exchanges, errors, and observers directly in
model-executing application code. Pure domain models, codecs, source-reference
@@ -100,11 +102,19 @@ pipeline assessment, and native execution:
and assessment for revised responses. The module supplies the repair request;
PromptKit maintains history and enforces the turn budget.
A small shared application function can connect these responsibilities: the
module supplies native request construction and response interpretation, while
the pipeline supplies configured validation and terminal policy. It uses native
PromptKit types and returns a typed application outcome. It does not introduce
another completion API, conversation protocol, or retry engine.
The resolved U3 design uses a small shared helper in `internal/framework/pipeline`.
Modules supply native initial/correction requests and response interpretation;
the pipeline supplies an operation-scoped assessment callback and relevant
cache/debug collaborators. Local variables or closures hold operation state.
Use the same assessor for initial and revised candidates without an additional
final pass. Native preparation supports cache lookup before execution; hits load
the accepted artifact directly. The helper returns a typed application outcome
and saves only final accepted artifacts under the initial hash.
Keep deterministic stages on their ordinary typed production/validation path.
LLM validators call the shared engine directly and return findings without an
automatic producer correction loop. No replacement completion API, execution-plan
language, or copied conversation model is selected.
Item 3 owns the detailed continuation and assessment rules, including final-turn
assessment, failure handling, complete replacement responses, and safe fallback.
@@ -140,10 +150,9 @@ not require preserving a private completion interface.
### Remaining Implementation Work
- Select concrete module/pipeline function signatures that connect request
construction, interpretation, configured validation, and terminal disposition.
- Choose the location of engine assembly and focused application helpers without
introducing a replacement LLM service layer.
- Implement the resolved U3 ownership and placement; refine function signatures
and argument grouping against representative extraction and normalization.
These implementation details do not reopen the selected boundaries.
- Implement item 2's resolved asset/schema design and settle the remaining
migration details recorded under items 3–6, including durable compatibility.
- Evaluate any remaining upstream gap against an actual application requirement,

View File

@@ -84,13 +84,54 @@ API. Pure domain models, codecs, source-reference resolution, and deterministic
transformations need no PromptKit dependency. Application interfaces continue to
express artifact production, typed validation, output encoding, and state access.
A small shared application function connects module request construction and
response interpretation to pipeline validation and terminal policy. It calls
native `Run` and `Coordinate` and returns a typed application outcome. It owns no
replacement completion API, conversation protocol, or general retry engine.
Exact signatures and helper placement remain open. The current
`internal/framework/llm` package may shrink to engine assembly and focused helpers
or cease to be a separate package.
#### Module Execution And Assessment (U3 Resolved)
Keep the existing typed stage boundaries and use native PromptKit values in
model-executing code. Place engine construction and shared loaders in focused
files in the `internal/cli` composition root. It constructs one native engine
for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset
registry, and reporting responsibilities disappear; do not retain a package
solely to hold a small constructor or introduce another LLM service interface.
Place one small shared producer helper alongside assessment in
`internal/framework/pipeline`. Connect modules to it through ordinary functions:
1. The pipeline supplies an operation-scoped assessment callback capturing the
configured validator chain and acceptance/terminal policy.
2. The module prepares source material, references, and any normalization handle
mappings once. It builds the native initial request and supplies response
interpretation and correction-request functions.
3. The helper receives those functions, the shared engine, and relevant
cache/debug collaborators. For cacheable work it prepares the initial request
and looks up the accepted-artifact cache; on a miss it executes the retained
request. Otherwise it uses ordinary native execution.
4. For each executed response, the helper invokes module interpretation and
pipeline assessment. When correction is appropriate, it supplies the module's
next request to native coordination. PromptKit owns conversation state and
turn iteration; the helper does not implement another retry engine.
5. Return a typed application outcome. Cache only final accepted artifacts under
the initial request's key. A cache hit follows the existing direct-reuse policy
without model execution or renewed semantic assessment.
Request builders return `promptkit.RunRequest`; interpreters consume native
results. Candidates, findings, and acceptance outcomes remain application-owned.
Keep operation-specific state in local variables or closures, without an
execution-plan language, replacement completion API, or copied conversation model.
Use one pipeline assessment implementation for the initial candidate and every
correction, including the final permitted response. Do not repeat final validation
merely because the helper returns to the outer runner. Deterministic stages keep
their ordinary typed production/validation path without synthetic requests or
conversation machinery. LLM validators call the shared engine directly and return
domain findings; they do not automatically acquire a producer correction loop.
Modules receive operation data and collaborators, not physical storage roots or
CommandKit reporters. Pure domain packages retain artifact models, codecs,
reference resolution, and deterministic rules. `internal/preflight` retains its
separate known-dependency checks. Exact argument grouping and helper signatures
can be refined against representative extraction and normalization during
implementation; ownership and placement are settled. Role-specific prompts,
budgets, and fallback policy remain U4–U6. Implementation is pending.
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
shape, ordered steps and generated references, typed artifact-family boundaries,
@@ -711,7 +752,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
| ID | Remaining choice or dependency | Resolve for |
| --- | --- | --- |
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. |
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
@@ -725,6 +765,12 @@ machinery are selected in [the schema design](#prompt-assets-and-response-schema
Implementation remains in Phase 2; role-specific prompts and retry budgets remain
U4/U5.
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
and the shared pipeline producer helper with module functions and a pipeline
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
Implement in Phase 3 and refine signatures during Phase 4 without reopening
ownership. U4–U6 still own role-specific behavior.
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
output, debug policy, and coordinated cutover sequencing are selected above.
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
@@ -745,8 +791,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
Keep the cache dependency explicit without blocking independent work.
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
forward Audita evidence without adopting its application wrappers.
- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas,
CommandKit, debug, cache, or output choices as requirements to redesign.
- Preserve U4–U9 with their dependent phases; do not reopen resolved module
execution, assets/schemas, CommandKit, debug, cache, or output ownership.
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
and outstanding choices are explicit. This phase does not claim that the hash
@@ -770,8 +816,9 @@ without credentials or synthetic requests. `config diff` handles two selections.
### Phase 3: Prove A Representative Native Pipeline
- Resolve U3 and relevant U4–U6 details. Connect native generation, interpretation,
configured validation, and continuation through the application function.
- Implement the resolved U3 collaboration and settle relevant U4–U6 details.
Connect module request/interpretation functions to the pipeline assessment
callback through the shared producer helper; refine concrete signatures here.
- Exercise an LLM validator on the same engine and normalization with fixed
handles and fresh plan application. Select actual observer/group enrollment.
- Run through the new command surface and alias, using the small completion
@@ -779,9 +826,10 @@ without credentials or synthetic requests. `config diff` handles two selections.
Begin U8/U9 integration without a provisional local message hash.
**Completion evidence:** fake-provider workflows exercise accumulated history,
full candidate revalidation, final-turn acceptance, selected exhaustion/fallback,
and useful failures. The executable publishes artifacts and reports them through
native presentation without the private completion/correction API.
full candidate revalidation through the same assessor, final-turn acceptance
without duplicate final validation, selected exhaustion/fallback, and useful
failures. The executable publishes artifacts and reports them through native
presentation without the private completion/correction API.
### Phase 4: Migrate All Roles And Remove Superseded Machinery
@@ -790,7 +838,9 @@ native presentation without the private completion/correction API.
- Remove the adapter, copied completion contracts, manual replay/capability checks,
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
- Finish native command/inspection migration and remove custom dispatch, flags,
encoders, verbosity handling, and obsolete asset registration.
encoders, verbosity handling, and obsolete asset registration. Retire the
emptied `internal/framework/llm` package after moving engine assembly to the
composition root and producer collaboration to the pipeline.
**Completion evidence:** every role uses the shared native engine and selected
admission policy. No production path depends on the private completion API,
@@ -832,7 +882,7 @@ Release preparation/publication is governed separately by release policy.
| Retire or replace | Retained application responsibility |
| --- | --- |
| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. |
| `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. |
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
@@ -858,7 +908,7 @@ retry tests or freeze complete help text and native debug snapshots.
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. |
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |