Set semantic turn defaults and delegate structural repair upstream
This commit is contained in:
@@ -146,8 +146,8 @@ reference resolution, and deterministic rules. `internal/preflight` retains its
|
||||
separate known-dependency checks. Exact argument grouping and helper signatures
|
||||
can be refined against representative extraction and normalization during
|
||||
implementation; ownership and placement are settled. U4 selects prompt roles and
|
||||
guidance ownership below; budgets and fallback policy remain U5/U6. Implementation
|
||||
is pending.
|
||||
guidance ownership below; U5 selects retry controls and the correction budget.
|
||||
Remaining failure and fallback policies stay in U5/U6. Implementation is pending.
|
||||
|
||||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||||
@@ -194,10 +194,11 @@ a singleton flag and use the last successfully parsed value.
|
||||
| Capture destination | Application `--debug-dir PATH`, with the semantics below. |
|
||||
|
||||
Keep worker counts and semantic budgets in configuration, adjustable through
|
||||
application `--set`; their fields/defaults remain open. Use ckprompt's reusable
|
||||
profile, session, and execution flags, including model, temperature, timeout,
|
||||
reasoning effort, and transient retries, for run-wide overrides across producers,
|
||||
validators, and continuations. Omission preserves normal selection; explicitly
|
||||
application `--set`. U5 selects `max_additional_turns`; worker defaults remain U7.
|
||||
Use ckprompt's reusable profile, session, and execution flags, including model,
|
||||
temperature, timeout, reasoning effort, and transient retries, for run-wide
|
||||
overrides across producers, validators, and continuations. Omission preserves
|
||||
normal selection; explicitly
|
||||
empty reasoning effort clears inheritance. Use native `--llm-profile` and
|
||||
`--llm-session-id`, retiring the separate reasoning-clearing control. Native
|
||||
transient retries remain distinct from semantic additional-turn budgets.
|
||||
@@ -275,7 +276,8 @@ ConfigKit continues to own loading and composition under
|
||||
Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of
|
||||
native companion sources; do not introduce a separate `promptkit.config_file`.
|
||||
Retire `concurrency.total_llm` and revise retry/worker settings as described below.
|
||||
Their replacement defaults and detailed migration treatment remain open.
|
||||
U5 selects `max_additional_turns` and removes application structural-repair
|
||||
controls; worker defaults and retired global-limit migration remain U7.
|
||||
|
||||
Use one source assembly for runtime, preflight, and inspection:
|
||||
|
||||
@@ -334,21 +336,23 @@ Remove synthetic mounts, file copying, flattened schema namespaces, and module
|
||||
asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening
|
||||
consumers disappear; no replacement filesystem abstraction is needed.
|
||||
|
||||
Declare response format, validation mode, schema path, and structural-repair
|
||||
budget in native prompt definitions by default. Modules select prompt IDs, supply
|
||||
inputs, and interpret responses. U4 selects the same schema file for each
|
||||
initial/correction pair. Execution, inspection, and prepared-request preview then
|
||||
use the same declared contract without repeating schema selection in Go.
|
||||
U4 selects prompt roles and contracts; exact prose is implementation work, while
|
||||
retry defaults remain U5.
|
||||
Declare response format, validation mode, and schema path in native prompt
|
||||
definitions by default. Omit structural-repair attempt settings in bundled
|
||||
prompts to inherit PromptKit's upstream default under U5. Modules select prompt
|
||||
IDs, supply inputs, and interpret responses. U4 selects the same schema file for
|
||||
each initial/correction pair. Execution, inspection, and prepared-request preview
|
||||
then use the same declared contract without repeating schema selection in Go.
|
||||
U4 selects prompt roles and contracts; exact prose is implementation work.
|
||||
U5 selects budget ownership and defaults.
|
||||
|
||||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||||
dependent contract. It replaces the whole configured contract, so supply the
|
||||
intended format, validation mode, schema, and repair policy together. Do not
|
||||
introduce a wrapper or per-call registry solely to reproduce Audita's approach.
|
||||
Operator prompt replacements must still produce responses the selected module
|
||||
can interpret; that module boundary does not require duplicating each prompt's
|
||||
schema selection in Go.
|
||||
intended format, validation mode, and schema together while leaving structural
|
||||
repair at its native default. Do not inject a repair count merely to construct
|
||||
the override, or introduce a wrapper or per-call registry solely to reproduce
|
||||
Audita's approach. Operator prompt replacements must still produce responses the
|
||||
selected module can interpret; that module boundary does not require duplicating
|
||||
each prompt's schema selection in Go.
|
||||
|
||||
Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to
|
||||
that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`.
|
||||
@@ -459,12 +463,12 @@ candidate, its module supplies guidance and artifact validators cannot yet run.
|
||||
Cancellation or fatal execution failures can interrupt assessment. An unavailable
|
||||
validator assessment is a separate outcome, not a detected defect or an instruction
|
||||
to change the artifact. Do not convert provider errors into correction guidance;
|
||||
detailed failure disposition remains U5/U6.
|
||||
terminal/fallback disposition remains U6.
|
||||
|
||||
Ownership, response contracts, and default profile behavior are resolved. Refine
|
||||
exact template prose, correction IDs, and necessary operation-specific inputs
|
||||
while implementing representative workflows. Implementation is pending; semantic
|
||||
budgets and terminal/fallback policies remain U5/U6.
|
||||
while implementing representative workflows. Implementation is pending; U5 selects
|
||||
semantic budgets and U6 retains terminal/fallback policy choices.
|
||||
|
||||
#### Execution And Assessment Flow
|
||||
|
||||
@@ -481,8 +485,9 @@ The application flow is:
|
||||
all applicable configured validators and collect their actionable guidance
|
||||
before requesting correction, as selected in U4.
|
||||
3. For actionable correction, call `Engine.Coordinate` with the captured exchange
|
||||
and `AdditionalTurns`. Its callback selects a repair prompt and supplies
|
||||
findings; PromptKit retains the initial exchange and subsequent turns.
|
||||
and the resolved `max_additional_turns` as native `AdditionalTurns`. Its callback
|
||||
selects a repair prompt and supplies findings; PromptKit retains the initial
|
||||
exchange and subsequent turns.
|
||||
4. Interpret every delivered revised response and rebuild from the fixed original
|
||||
inputs. Run the complete configured validation chain again for changed
|
||||
candidates, including validators that previously approved. Assess the final
|
||||
@@ -491,8 +496,8 @@ The application flow is:
|
||||
failure, or supported safe-fallback policy after exhaustion. Native `stopped`
|
||||
and `exhausted` outcomes do not themselves mean application acceptance.
|
||||
|
||||
Accumulated history increases context use; select the semantic turn budget
|
||||
accordingly. Exact budgets remain U5.
|
||||
Accumulated history increases context use; U5's allowance bounds additional
|
||||
semantic correction while stopping as soon as the candidate is accepted.
|
||||
|
||||
Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection`
|
||||
transport, manual assistant/user replay, and correction-protocol capability
|
||||
@@ -502,16 +507,71 @@ findings and explicit fallback information. Retain useful ordered collection and
|
||||
deduplication of actionable findings, rendered through a native repair prompt.
|
||||
Reason codes and operator messages are not automatically model instructions.
|
||||
|
||||
Transport retries, structural repair, and semantic additional turns remain
|
||||
separate controls. Preserve the ability to override native transport retries,
|
||||
using the upstream default when omitted. Remove blanket outer LLM retries after
|
||||
native transport/structural budgets are exhausted, including repeated LLM
|
||||
validator invocations in `executeValidationChain`. Keep the ordered validator
|
||||
chain and aggregation. A validator execution failure is distinct from rejection
|
||||
of a producer candidate. Existing producer/validator `retries` settings cannot be
|
||||
mechanically renamed; decide their migration and non-LLM failure handling
|
||||
explicitly. Use native `OperationTimeout` when that operation scope is needed,
|
||||
without adding an arbitrary default or replacing broader application deadlines.
|
||||
#### Retry Controls And Correction Budget (U5 Controls Resolved)
|
||||
|
||||
Delegate transport retries and structural repair to PromptKit. Notarius selects
|
||||
only the allowance for application-directed semantic correction:
|
||||
|
||||
| Concern | Control and selected default |
|
||||
| --- | --- |
|
||||
| Temporary provider failures | Native PromptKit transient retries; inherit the upstream default when omitted and preserve native execution overrides. |
|
||||
| Structurally invalid model output | Native output-contract policy; bundled prompts and application requests inherit PromptKit's upstream structural-repair default. |
|
||||
| Actionable domain defects | Application `max_additional_turns`, passed directly to native `AdditionalTurns`; application default `2`. |
|
||||
|
||||
`max_additional_turns` is a nonnegative maximum, excluding the initial producer
|
||||
response. Resolve producer-binding override, then pipeline override, then the
|
||||
application default of `2`. Omission inherits; explicit `0` disables semantic
|
||||
correction. It applies to LLM-backed producers. Validators do not receive a
|
||||
separate semantic continuation budget, and deterministic stages do not acquire
|
||||
conversation machinery. Keep the field in configuration, adjustable through the
|
||||
existing application `--set` mechanism, without another dedicated CLI flag.
|
||||
|
||||
For example, a producer binding can override its pipeline allowance:
|
||||
|
||||
```yaml
|
||||
extract:
|
||||
module: dnd/spells
|
||||
max_additional_turns: 2
|
||||
```
|
||||
|
||||
Two additional turns are an allowance, not mandatory executions. An accepted
|
||||
initial candidate needs none; an accepted first correction needs only one. A
|
||||
second turn is available when actionable defects remain and continuation is
|
||||
appropriate. Assess every delivered response, including the final permitted turn,
|
||||
and stop on acceptance or a terminal condition. PromptKit enforces the turn
|
||||
limit; do not add an application counter or retry loop. Internal transport
|
||||
attempts and structural repairs do not consume this semantic-turn allowance.
|
||||
The default is an application policy choice, not a measured claim about how
|
||||
frequently corrections will be needed.
|
||||
|
||||
For structural repair, omit `output.repair_attempts` from all bundled initial,
|
||||
correction, and validator definitions. Remove Notarius's pipeline/binding/
|
||||
validator `structured_output_repair_attempts` fields and resolution plumbing.
|
||||
Do not replace them with a hard-coded default, including zero, in request
|
||||
construction or a native output-contract override. Explicit operator
|
||||
customization remains available through native PromptKit prompt configuration.
|
||||
|
||||
Notarius deliberately inherits future upstream default changes. An anticipated
|
||||
PromptKit default change is not a migration prerequisite, and its timing or
|
||||
chosen count is not a Notarius contract. Notarius still declares response format,
|
||||
validation mode, and schema; PromptKit owns the mechanics of obtaining a valid
|
||||
response. At the public boundary, handle a usable result or terminal failure,
|
||||
including a returned result reporting unsuccessful structural validation.
|
||||
Do not restart the operation to extend PromptKit's exhausted repair budget.
|
||||
|
||||
Remove blanket outer LLM retries, including repeated LLM validator invocations in
|
||||
`executeValidationChain`. Keep ordered candidate assessment and distinguish a
|
||||
validator execution failure from a domain rejection. Use native `OperationTimeout`
|
||||
when that operation scope is needed, without adding an arbitrary default or
|
||||
replacing broader application deadlines. Cancellation stops work.
|
||||
|
||||
Retire the broad producer/validator `retries` settings for LLM execution rather
|
||||
than automatically renaming them. Migration guidance maps intent: a producer
|
||||
allowance intended for semantic correction becomes `max_additional_turns`; native
|
||||
transport controls and prompt-owned structural customization cover their own
|
||||
concerns. Do not promise equivalent total provider-attempt counts. Generic retry
|
||||
handling for non-LLM operations remains the outstanding U5 choice; detailed
|
||||
terminal and fallback dispositions remain U6. Implementation is pending.
|
||||
|
||||
Keep source grounding, reference resolution, candidate isolation, domain
|
||||
invariants, and reconciliation safety checks. Unknown or overlapping handles
|
||||
@@ -846,7 +906,7 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
| ID | Remaining choice or dependency | Resolve for |
|
||||
| --- | --- | --- |
|
||||
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||||
| U5 | Decide generic retry handling for non-LLM operations. LLM controls, `max_additional_turns` naming/default/precedence, structural-default delegation, and intent-based LLM setting migration are resolved below. | Phases 3–4 and configuration migration. |
|
||||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||||
| U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. |
|
||||
| U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. |
|
||||
@@ -855,22 +915,29 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||||
native schema-source selection, and removal of duplicate schema/filesystem
|
||||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||||
Implementation remains in Phase 2; U4 selects prompt roles and contracts, while
|
||||
retry budgets remain U5.
|
||||
Implementation remains in Phase 2; U4 selects prompt roles and contracts, and U5
|
||||
selects native repair delegation and the application correction allowance.
|
||||
|
||||
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
|
||||
and the shared pipeline producer helper with module functions and a pipeline
|
||||
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
|
||||
Implement in Phase 3 and refine signatures during Phase 4 without reopening
|
||||
ownership. U4 selects prompt/guidance behavior; U5/U6 retain budget and terminal
|
||||
policy choices.
|
||||
ownership. U4/U5 select prompt/guidance and budget behavior; remaining failure
|
||||
and terminal-policy choices stay in U5/U6.
|
||||
|
||||
**U4 resolved:** Domain-specific initial/correction prompt pairs, shared response
|
||||
schemas, default profile behavior, validator-owned guidance, and complete
|
||||
applicable-chain assessment are selected in
|
||||
[the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved).
|
||||
Implement templates and operation-specific inputs in Phases 3–4; exact prose does
|
||||
not reopen ownership. Budgets and terminal/fallback policy remain U5/U6.
|
||||
not reopen ownership. U5 selects budgets; terminal/fallback policy remains U6.
|
||||
|
||||
**U5 controls resolved:** `max_additional_turns` defaults to `2`, with binding over
|
||||
pipeline precedence and explicit zero disabling correction. Native transient and
|
||||
structural-repair defaults remain upstream-owned; bundled prompts and application
|
||||
requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-controls-resolved)
|
||||
also selects intent-based LLM setting migration and removal of outer LLM retries.
|
||||
Non-LLM retry handling remains open; U6 owns terminal/fallback decisions.
|
||||
|
||||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||||
@@ -893,8 +960,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||||
forward Audita evidence without adopting its application wrappers.
|
||||
- Preserve U5–U9 with their dependent phases; do not reopen resolved module
|
||||
execution, prompt/guidance ownership, assets/schemas, CommandKit, debug, cache,
|
||||
or output ownership.
|
||||
execution, prompt/guidance ownership, retry controls/defaults, assets/schemas,
|
||||
CommandKit, debug, cache, or output ownership.
|
||||
|
||||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||||
and outstanding choices are explicit. Upstream availability alone does not
|
||||
@@ -905,7 +972,8 @@ of the migration.
|
||||
|
||||
- Implement the resolved U2 design, update available dependencies, introduce
|
||||
configuration version 6, convert native embedded/companion resources, and use
|
||||
native schema loading with embedded/default or operator-selected roots.
|
||||
native schema loading with embedded/default or operator-selected roots. Omit
|
||||
bundled structural-repair settings and remove their application override fields.
|
||||
- Build the small CommandKit application, shared ConfigKit/engine loaders,
|
||||
standard suites, pipeline declarations/native alias, and `internal/preflight`.
|
||||
- Add maintained embedded-only and operator-override examples. Remove duplicated
|
||||
@@ -919,11 +987,14 @@ without credentials or synthetic requests. `config diff` handles two selections.
|
||||
|
||||
### Phase 3: Prove A Representative Native Pipeline
|
||||
|
||||
- Implement the resolved U3/U4 collaboration and settle relevant U5/U6 details.
|
||||
- Implement the resolved U3/U4 collaboration and U5 controls; settle remaining
|
||||
U5/U6 failure-policy details.
|
||||
Connect module request/interpretation functions to the pipeline assessment
|
||||
callback through the shared producer helper; refine concrete signatures here.
|
||||
Implement domain-specific initial/correction prompts using the same schema and
|
||||
default profile, supplied overrides, and validator-owned correction guidance.
|
||||
Resolve `max_additional_turns` with omission/zero distinctions and pass it to
|
||||
native coordination; inherit native transport and structural-repair defaults.
|
||||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||||
handles and fresh plan application. Select actual observer/group enrollment.
|
||||
- Run through the new command surface and alias, using the small completion
|
||||
@@ -942,7 +1013,7 @@ without the private completion/correction API.
|
||||
|
||||
- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve
|
||||
deterministic paths and ordered validation. Apply U4's prompt/guidance design
|
||||
across roles and complete U5–U7.
|
||||
and U5's selected controls across roles; settle remaining U5–U7 choices.
|
||||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||||
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
|
||||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||||
@@ -996,6 +1067,7 @@ Release preparation/publication is governed separately by release policy.
|
||||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. |
|
||||
| Application `structured_output_repair_attempts` fields/resolution and explicit bundled `output.repair_attempts` | Native upstream structural-repair default, with operator customization through native prompt configuration. |
|
||||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||||
| Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. |
|
||||
| Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. |
|
||||
@@ -1019,6 +1091,7 @@ retry tests or freeze complete help text and native debug snapshots.
|
||||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
|
||||
| Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. |
|
||||
| Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. |
|
||||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
|
||||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |
|
||||
@@ -1053,8 +1126,11 @@ Provide practical release guidance for operators and subprocess callers:
|
||||
- Version 6 and native resources, embedded-only versus companion examples in
|
||||
appropriate directories, imports, qualified IDs, replacement/inheritance,
|
||||
native local backends, removed catalogs/versions, and separate source versions.
|
||||
- Native admission and worker defaults, removed global limit, retry-budget
|
||||
migration, accumulated-history costs, and selected role failure policy.
|
||||
- Native admission and worker defaults, removed global limit, `max_additional_turns`
|
||||
default/precedence/zero behavior, and intent-based migration from LLM `retries`.
|
||||
Explain upstream-owned structural defaults, removal of application repair
|
||||
controls and bundled counts, native operator customization, accumulated-history
|
||||
costs, and selected role failure policy.
|
||||
- Accepted-artifact caching, message-based misses, settings-only reuse, refresh/
|
||||
bypass for changed interpretation or validation, and old-checkpoint handling.
|
||||
- Artifact delivery without mandatory manifests/receipts; preserve intrinsic
|
||||
|
||||
Reference in New Issue
Block a user