diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 5187c8cf..21b485a6 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -149,8 +149,8 @@ can be refined against representative extraction and normalization during implementation; ownership and placement are settled. U4 selects prompt roles and guidance ownership below; U5 selects retry controls and the correction budget. U6 selects strict-by-default handling and module-owned fallbacks below; -configuration scope and concrete module fallback choices remain open. -Implementation is pending. +configuration precedence and fallback behavior are resolved. Implementation is +pending. Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` shape, ordered steps and generated references, typed artifact-family boundaries, @@ -471,8 +471,7 @@ apply U6's strict/permissive handling when no acceptable result is obtained. Ownership, response contracts, and default profile behavior are resolved. Refine exact template prose, correction IDs, and necessary operation-specific inputs while implementing representative workflows. Implementation is pending; U5 selects -semantic budgets and U6 selects strict-by-default terminal handling, with -configuration scope and concrete module fallback choices still open. +semantic budgets and U6 selects strict-by-default terminal handling and fallbacks. #### Execution And Assessment Flow @@ -591,10 +590,9 @@ for semantic correction becomes `max_additional_turns`; native transport control and prompt-owned structural customization cover their own concerns. Remove non-LLM retry settings without a replacement generic allowance. Do not promise equivalent total provider-attempt counts. U5 is resolved; U6 selects strict-by-default -handling, with configuration scope and concrete module fallback choices still -open. Implementation is pending. +handling and module-owned fallbacks. Implementation is pending. -#### Terminal Outcomes And Module Fallbacks (U6 Direction Resolved) +#### Terminal Outcomes And Module Fallbacks (U6 Resolved) Use two modes when an operation does not obtain a successful, validated result: @@ -610,8 +608,16 @@ fallback where the domain permits it; no separate omit-output action is needed. The application default is strict for every role, including normalization. Modules do not declare their own defaults. Permissive handling requires an explicit configuration selection; normalization has no automatic exception. -Configuration spelling/scope/precedence and concrete module fallback definitions -remain open. Do not infer them from existing policy fields. + +Use `mode: strict` or `mode: permissive` at pipeline level, with an optional +`mode` override on each configured producer module binding. Resolve the module +binding first, then the pipeline setting, then the application default `strict`. +Omission inherits; either explicit value overrides the broader setting. A strict +pipeline can therefore permit fallback for one module, and a permissive pipeline +can require strict handling for an essential module. This is binding configuration, +not a default declared by the module implementation. Validators report findings; +the producer's resolved mode governs disposition. Use ordinary configuration and +the existing `--set` mechanism, without a dedicated mode flag. Attempt application-directed correction when appropriate within the selected budget before terminal handling. Neither mode extends PromptKit's native retry @@ -628,12 +634,12 @@ validator chain as a condition of using the fallback or claim normal validation passed. Keep the necessary checks with module fallback construction; no separate configurable fallback-validation framework is needed. -The selected direction supports these domain-specific fallbacks: +Use these domain-specific fallbacks: | Module | Fallback direction | | --- | --- | | D&D spell extraction | A valid empty spell artifact, so that the failed chunk contributes no extracted spells. | -| D&D transcript chunking | A deterministic partition preserving the entire transcript. Whole-transcript-as-one-chunk versus deterministic bounded slices remains open. | +| D&D transcript chunking | One chunk containing the entire transcript, preserving source order and references. This is the default fallback; no deterministic slicing machinery or sizing policy is needed for this migration. | | Normalization | Apply only safe consolidation groups from the final usable response afresh to the fixed originals, retaining everything else. If that response is unusable, use the deterministic merge of the input chunk artifacts. Reuse an already available merged input. | Keep source grounding, reference resolution, candidate isolation, and @@ -642,7 +648,10 @@ partial normalization applies only compatible safe groups and leaves other original candidates separate. Use the final response and fixed original inputs, without recovering an earlier candidate or combining proposals across turns. Concrete module algorithms must make their supported fallback and required -invariants explicit; these examples do not introduce a generic fallback language. +invariants explicit. Refine module-specific construction without introducing a +generic fallback language. Whole-transcript chunking deliberately favors simple +machinery for a rarely expected fallback; it does not promise bounded downstream +request size. Fallbacks may be published and consumed downstream, but they and dependent results affected by their incompleteness are ineligible for accepted-artifact cache lookup @@ -655,9 +664,8 @@ Record why fallback was used and expose that qualification in the ordinary completion report, even when diagnostic verbosity hides individual warnings. Use the same application outcome for reporting and optional debug detail. Keep conversation state in process, outside ordinary artifact/cache payloads; -explicitly requested detailed debug is separate. Implementation is pending, and -U6 remains open only for the configuration contract and concrete module fallback -choices. +explicitly requested detailed debug is separate. U6 is resolved; implementation +and module-specific construction details remain in Phases 3–4. ### Admission And Pipeline Workers @@ -983,7 +991,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U6 | Configuration spelling, scope, and override precedence; concrete module fallbacks, particularly whole-transcript versus sliced chunking. Strict is the application default for every role; the two-mode model, module ownership, and final-response-only handling are settled. | Phases 3–4. | | U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. | | U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. | | U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. | @@ -999,7 +1006,7 @@ and the shared pipeline producer helper with module functions and a pipeline assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). Implement in Phase 3 and refine signatures during Phase 4 without reopening ownership. U4/U5 select prompt/guidance and retry behavior; U6 selects terminal -handling, with its remaining configuration and module choices listed above. +handling, configuration precedence, and module-owned fallbacks. **U4 resolved:** Domain-specific initial/correction prompt pairs, shared response schemas, default profile behavior, validator-owned guidance, and complete @@ -1016,14 +1023,16 @@ also removes generic non-LLM retries, all producer/validator `retries` fields, and the generic outer attempt loop. Any demonstrated transient I/O recovery stays with its specific operation. Implement in Phases 3–4; U6 selects terminal handling. -**U6 direction resolved:** [Strict/permissive handling and module-owned fallbacks](#terminal-outcomes-and-module-fallbacks-u6-direction-resolved) +**U6 resolved:** [Strict/permissive handling and module-owned fallbacks](#terminal-outcomes-and-module-fallbacks-u6-resolved) replace failure-category policies. Fallbacks satisfy artifact/downstream invariants without asserting ordinary semantic acceptance; use only the final response and fixed originals. Fallback and affected downstream work remain outside accepted- artifact caching, and completion reports disclose fallback use. Strict is the application default, including normalization; there are no module-declared -defaults. Configuration contract and concrete module fallback choices remain -open above. +defaults. `mode` resolves module binding, then pipeline, then that default. +Chunking falls back to one whole-transcript chunk; spell extraction uses its empty +artifact, and normalization uses safe final-response groups or merged input. +Implement module-specific construction details in Phases 3–4. **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. @@ -1045,7 +1054,7 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U6–U9 with their dependent phases; do not reopen resolved module +- Preserve U7–U9 with their dependent phases; do not reopen resolved module execution, prompt/guidance ownership, retry controls/defaults, assets/schemas, CommandKit, debug, cache, or output ownership. @@ -1074,8 +1083,7 @@ without credentials or synthetic requests. `config diff` handles two selections. ### Phase 3: Prove A Representative Native Pipeline - Implement the resolved U3–U5 collaboration and retry policy and U6's - strict-by-default model. Settle its configuration contract and representative - module fallbacks. + strict-by-default model, `mode` precedence, and representative module fallbacks. Connect module request/interpretation functions to the pipeline assessment callback through the shared producer helper; refine concrete signatures here. Implement domain-specific initial/correction prompts using the same schema and @@ -1100,7 +1108,7 @@ without the private completion/correction API. - Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve deterministic paths and ordered validation. Apply U4's prompt/guidance design - and U5's retry policy across roles; settle remaining U6/U7 choices. + and U5's retry policy across roles; implement U6's module fallbacks and settle U7. - Remove the adapter, copied completion contracts, manual replay/capability checks, generic outer attempt loop, all producer/validator `retries` fields, global scheduler/ceiling, and coupled worker settings. @@ -1188,7 +1196,7 @@ retry tests or freeze complete help text and native debug snapshots. | Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. | | Non-LLM failure handling | Deterministic errors propagate without generic retries; rejected deterministic candidates reach terminal policy without regeneration; cancellation stops work. Producer/validator `retries` fields and the generic outer attempt loop are retired. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; no earlier-candidate recovery or cross-turn proposal union; cancellation stops without fallback work. | -| Strict/permissive outcomes | Strict is the default for every role, including normalization, without module-specific defaults. Strict failure when no acceptable result is obtained; explicitly selected permissive handling produces module fallback with required artifact/downstream invariants, without rerunning the ordinary full validation chain or claiming normal acceptance. Fallback construction failure fails execution; successful fallback use is disclosed in human/machine completion reports. | +| Strict/permissive outcomes | `mode` resolves module binding, pipeline, then strict; omission inherits and either explicit value overrides. Strict is the default for every role, including normalization, without module-specific defaults. Strict failure when no acceptable result is obtained; permissive handling produces module fallback with required artifact/downstream invariants, without rerunning the ordinary full validation chain or claiming normal acceptance. Chunking fallback is one whole-transcript chunk preserving source order/references. Fallback construction failure fails execution; successful fallback use is disclosed in human/machine completion reports. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss. Failed/rejected/incomplete work is not stored; fallbacks and affected downstream work skip lookup/storage while unaffected accepted predecessors remain reusable; no required history. | | Cache migration | Selected reuse/bypass/refresh and old-record treatment work; invocation configuration no longer blocks lookup; canonical chunk caching retains its distinct policy. | @@ -1227,8 +1235,9 @@ Provide practical release guidance for operators and subprocess callers: Explain retirement of all producer/validator `retries` fields and generic non-LLM retries, upstream-owned structural defaults, removal of application repair controls and bundled counts, native operator customization, accumulated-history - costs, the universal strict default, explicit permissive selection and precedence, - and each module's fallback behavior and downstream guarantees. + costs, the universal strict default, pipeline `mode` with module-binding overrides, + and each module's fallback behavior and downstream guarantees, including the + whole-transcript chunking fallback. - Accepted-artifact caching, message-based misses, settings-only reuse, refresh/ bypass for changed interpretation or validation, fallback/affected-work exclusion, and old-checkpoint handling.