Resolve native module execution and pipeline assessment design
This commit is contained in:
@@ -68,9 +68,11 @@ resulting change to the adoption target remains to be selected.
|
||||
|
||||
Engine construction becomes ordinary application wiring using item 2's embedded
|
||||
defaults and operator sources. Profile selection is expressed through native
|
||||
requests. The current `internal/framework/llm` package can shrink to engine
|
||||
assembly and focused helpers; whether those warrant a separate package remains
|
||||
an organizational choice, not a reason to retain an adapter.
|
||||
requests. U3 now places shared loaders and engine construction in `internal/cli`
|
||||
and the producer helper beside pipeline assessment. Retire the emptied
|
||||
`internal/framework/llm` package rather than retaining a constructor-only package.
|
||||
The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns
|
||||
the resolved design; implementation is pending.
|
||||
|
||||
Use native requests, results, exchanges, errors, and observers directly in
|
||||
model-executing application code. Pure domain models, codecs, source-reference
|
||||
@@ -100,11 +102,19 @@ pipeline assessment, and native execution:
|
||||
and assessment for revised responses. The module supplies the repair request;
|
||||
PromptKit maintains history and enforces the turn budget.
|
||||
|
||||
A small shared application function can connect these responsibilities: the
|
||||
module supplies native request construction and response interpretation, while
|
||||
the pipeline supplies configured validation and terminal policy. It uses native
|
||||
PromptKit types and returns a typed application outcome. It does not introduce
|
||||
another completion API, conversation protocol, or retry engine.
|
||||
The resolved U3 design uses a small shared helper in `internal/framework/pipeline`.
|
||||
Modules supply native initial/correction requests and response interpretation;
|
||||
the pipeline supplies an operation-scoped assessment callback and relevant
|
||||
cache/debug collaborators. Local variables or closures hold operation state.
|
||||
Use the same assessor for initial and revised candidates without an additional
|
||||
final pass. Native preparation supports cache lookup before execution; hits load
|
||||
the accepted artifact directly. The helper returns a typed application outcome
|
||||
and saves only final accepted artifacts under the initial hash.
|
||||
|
||||
Keep deterministic stages on their ordinary typed production/validation path.
|
||||
LLM validators call the shared engine directly and return findings without an
|
||||
automatic producer correction loop. No replacement completion API, execution-plan
|
||||
language, or copied conversation model is selected.
|
||||
|
||||
Item 3 owns the detailed continuation and assessment rules, including final-turn
|
||||
assessment, failure handling, complete replacement responses, and safe fallback.
|
||||
@@ -140,10 +150,9 @@ not require preserving a private completion interface.
|
||||
|
||||
### Remaining Implementation Work
|
||||
|
||||
- Select concrete module/pipeline function signatures that connect request
|
||||
construction, interpretation, configured validation, and terminal disposition.
|
||||
- Choose the location of engine assembly and focused application helpers without
|
||||
introducing a replacement LLM service layer.
|
||||
- Implement the resolved U3 ownership and placement; refine function signatures
|
||||
and argument grouping against representative extraction and normalization.
|
||||
These implementation details do not reopen the selected boundaries.
|
||||
- Implement item 2's resolved asset/schema design and settle the remaining
|
||||
migration details recorded under items 3–6, including durable compatibility.
|
||||
- Evaluate any remaining upstream gap against an actual application requirement,
|
||||
|
||||
@@ -84,13 +84,54 @@ API. Pure domain models, codecs, source-reference resolution, and deterministic
|
||||
transformations need no PromptKit dependency. Application interfaces continue to
|
||||
express artifact production, typed validation, output encoding, and state access.
|
||||
|
||||
A small shared application function connects module request construction and
|
||||
response interpretation to pipeline validation and terminal policy. It calls
|
||||
native `Run` and `Coordinate` and returns a typed application outcome. It owns no
|
||||
replacement completion API, conversation protocol, or general retry engine.
|
||||
Exact signatures and helper placement remain open. The current
|
||||
`internal/framework/llm` package may shrink to engine assembly and focused helpers
|
||||
or cease to be a separate package.
|
||||
#### Module Execution And Assessment (U3 Resolved)
|
||||
|
||||
Keep the existing typed stage boundaries and use native PromptKit values in
|
||||
model-executing code. Place engine construction and shared loaders in focused
|
||||
files in the `internal/cli` composition root. It constructs one native engine
|
||||
for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset
|
||||
registry, and reporting responsibilities disappear; do not retain a package
|
||||
solely to hold a small constructor or introduce another LLM service interface.
|
||||
|
||||
Place one small shared producer helper alongside assessment in
|
||||
`internal/framework/pipeline`. Connect modules to it through ordinary functions:
|
||||
|
||||
1. The pipeline supplies an operation-scoped assessment callback capturing the
|
||||
configured validator chain and acceptance/terminal policy.
|
||||
2. The module prepares source material, references, and any normalization handle
|
||||
mappings once. It builds the native initial request and supplies response
|
||||
interpretation and correction-request functions.
|
||||
3. The helper receives those functions, the shared engine, and relevant
|
||||
cache/debug collaborators. For cacheable work it prepares the initial request
|
||||
and looks up the accepted-artifact cache; on a miss it executes the retained
|
||||
request. Otherwise it uses ordinary native execution.
|
||||
4. For each executed response, the helper invokes module interpretation and
|
||||
pipeline assessment. When correction is appropriate, it supplies the module's
|
||||
next request to native coordination. PromptKit owns conversation state and
|
||||
turn iteration; the helper does not implement another retry engine.
|
||||
5. Return a typed application outcome. Cache only final accepted artifacts under
|
||||
the initial request's key. A cache hit follows the existing direct-reuse policy
|
||||
without model execution or renewed semantic assessment.
|
||||
|
||||
Request builders return `promptkit.RunRequest`; interpreters consume native
|
||||
results. Candidates, findings, and acceptance outcomes remain application-owned.
|
||||
Keep operation-specific state in local variables or closures, without an
|
||||
execution-plan language, replacement completion API, or copied conversation model.
|
||||
|
||||
Use one pipeline assessment implementation for the initial candidate and every
|
||||
correction, including the final permitted response. Do not repeat final validation
|
||||
merely because the helper returns to the outer runner. Deterministic stages keep
|
||||
their ordinary typed production/validation path without synthetic requests or
|
||||
conversation machinery. LLM validators call the shared engine directly and return
|
||||
domain findings; they do not automatically acquire a producer correction loop.
|
||||
|
||||
Modules receive operation data and collaborators, not physical storage roots or
|
||||
CommandKit reporters. Pure domain packages retain artifact models, codecs,
|
||||
reference resolution, and deterministic rules. `internal/preflight` retains its
|
||||
separate known-dependency checks. Exact argument grouping and helper signatures
|
||||
can be refined against representative extraction and normalization during
|
||||
implementation; ownership and placement are settled. Role-specific prompts,
|
||||
budgets, and fallback policy remain U4–U6. Implementation is pending.
|
||||
|
||||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||||
@@ -711,7 +752,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
| ID | Remaining choice or dependency | Resolve for |
|
||||
| --- | --- | --- |
|
||||
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. |
|
||||
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
|
||||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||||
@@ -725,6 +765,12 @@ machinery are selected in [the schema design](#prompt-assets-and-response-schema
|
||||
Implementation remains in Phase 2; role-specific prompts and retry budgets remain
|
||||
U4/U5.
|
||||
|
||||
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
|
||||
and the shared pipeline producer helper with module functions and a pipeline
|
||||
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
|
||||
Implement in Phase 3 and refine signatures during Phase 4 without reopening
|
||||
ownership. U4–U6 still own role-specific behavior.
|
||||
|
||||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||||
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
|
||||
@@ -745,8 +791,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
Keep the cache dependency explicit without blocking independent work.
|
||||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||||
forward Audita evidence without adopting its application wrappers.
|
||||
- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas,
|
||||
CommandKit, debug, cache, or output choices as requirements to redesign.
|
||||
- Preserve U4–U9 with their dependent phases; do not reopen resolved module
|
||||
execution, assets/schemas, CommandKit, debug, cache, or output ownership.
|
||||
|
||||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||||
and outstanding choices are explicit. This phase does not claim that the hash
|
||||
@@ -770,8 +816,9 @@ without credentials or synthetic requests. `config diff` handles two selections.
|
||||
|
||||
### Phase 3: Prove A Representative Native Pipeline
|
||||
|
||||
- Resolve U3 and relevant U4–U6 details. Connect native generation, interpretation,
|
||||
configured validation, and continuation through the application function.
|
||||
- Implement the resolved U3 collaboration and settle relevant U4–U6 details.
|
||||
Connect module request/interpretation functions to the pipeline assessment
|
||||
callback through the shared producer helper; refine concrete signatures here.
|
||||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||||
handles and fresh plan application. Select actual observer/group enrollment.
|
||||
- Run through the new command surface and alias, using the small completion
|
||||
@@ -779,9 +826,10 @@ without credentials or synthetic requests. `config diff` handles two selections.
|
||||
Begin U8/U9 integration without a provisional local message hash.
|
||||
|
||||
**Completion evidence:** fake-provider workflows exercise accumulated history,
|
||||
full candidate revalidation, final-turn acceptance, selected exhaustion/fallback,
|
||||
and useful failures. The executable publishes artifacts and reports them through
|
||||
native presentation without the private completion/correction API.
|
||||
full candidate revalidation through the same assessor, final-turn acceptance
|
||||
without duplicate final validation, selected exhaustion/fallback, and useful
|
||||
failures. The executable publishes artifacts and reports them through native
|
||||
presentation without the private completion/correction API.
|
||||
|
||||
### Phase 4: Migrate All Roles And Remove Superseded Machinery
|
||||
|
||||
@@ -790,7 +838,9 @@ native presentation without the private completion/correction API.
|
||||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||||
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
|
||||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||||
encoders, verbosity handling, and obsolete asset registration.
|
||||
encoders, verbosity handling, and obsolete asset registration. Retire the
|
||||
emptied `internal/framework/llm` package after moving engine assembly to the
|
||||
composition root and producer collaboration to the pipeline.
|
||||
|
||||
**Completion evidence:** every role uses the shared native engine and selected
|
||||
admission policy. No production path depends on the private completion API,
|
||||
@@ -832,7 +882,7 @@ Release preparation/publication is governed separately by release policy.
|
||||
|
||||
| Retire or replace | Retained application responsibility |
|
||||
| --- | --- |
|
||||
| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. |
|
||||
| `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. |
|
||||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||||
@@ -858,7 +908,7 @@ retry tests or freeze complete help text and native debug snapshots.
|
||||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
|
||||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. |
|
||||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
|
||||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
|
||||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |
|
||||
|
||||
Reference in New Issue
Block a user