1302 lines
87 KiB
Markdown
1302 lines
87 KiB
Markdown
# Native PromptKit And CommandKit Migration
|
||
|
||
## Purpose And Status
|
||
|
||
Make Notarius fully native to PromptKit, ConfigKit, and CommandKit. Use upstream
|
||
configuration, execution, continuation, admission, inspection, commands, and
|
||
presentation directly. Retain the application work of producing, assessing,
|
||
publishing, and reusing typed artifacts. Historical duplication of library
|
||
behavior is not a requirement to preserve it.
|
||
|
||
This is the single migration plan for PromptKit, ConfigKit, CommandKit,
|
||
accepted-artifact caching, and artifact-first output. It owns the selected target,
|
||
remaining choices, implementation sequence, removal inventory, acceptance
|
||
criteria, and supporting evidence. The earlier discussion documents are retired;
|
||
Git history preserves the discussion. All implementation phases are pending.
|
||
Recording these decisions has not changed application behavior. Unrelated future
|
||
work remains in the [separate backlog](future.md).
|
||
|
||
Notarius predates many modern PromptKit capabilities: the library grew out of
|
||
mechanics shared by Notarius and related applications. Historical duplication
|
||
and slight behavioral differences do not express an intentional application
|
||
requirement. Evaluate remaining machinery against actual domain needs, and
|
||
consider upstream features when a reusable capability is missing.
|
||
|
||
Complete one coordinated cutover for configuration, PromptKit resources, and the
|
||
CLI. Adopt native public conventions fully; preserving old CLI behavior is a
|
||
non-goal. Supply migration guidance and maintained examples instead of aliases,
|
||
parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is
|
||
part of the main migration, not a later follow-up. Release preparation and
|
||
publication remain separate work.
|
||
|
||
Adopting the kits together lets standard inspection and execution share the same
|
||
loaders and native facts. It avoids building temporary command and reporting
|
||
adapters while those boundaries are already being replaced.
|
||
|
||
## Baseline And Dependency Target
|
||
|
||
At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application
|
||
uses configuration version 5 and Go 1.27. Its dependencies include old-path
|
||
PromptKit v0.12.0 and ConfigKit v0.5.0, new-path ConfigKit v1.2.0, go-yaml
|
||
v4.0.0-rc.6, and two external PromptKit provider catalogs.
|
||
|
||
The selected reviewed baseline is:
|
||
|
||
| Module | Version |
|
||
| --- | --- |
|
||
| `go.maximumdirect.net/commandkit` | `v0.7.0` |
|
||
| `go.maximumdirect.net/ckconfig` | `v0.9.0` |
|
||
| `go.maximumdirect.net/ckprompt` | `v0.10.0` |
|
||
| `go.maximumdirect.net/configkit` | `v1.2.1` |
|
||
| `go.maximumdirect.net/promptkit` | `v0.23.3`, subject to U1's hash-capable release review |
|
||
|
||
Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible
|
||
validation dependencies. The selected CommandKit integration releases document
|
||
compatibility with PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout
|
||
attribution and cancellation/error identity fixes; that patch requires no
|
||
configuration or request API migration from v0.23.2 and retains its dependencies.
|
||
The original review used local release tags and tagged dependency manifests;
|
||
remote release lookup was unavailable, so these are selected versions, not a
|
||
claim that they are the latest releases. See the
|
||
[integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
|
||
and tagged [release notes][release], [migration guide][migration],
|
||
[format reference][formats], and [Go consumer guide][consumer].
|
||
|
||
The cache requires native `MessageContentHash`; v0.23.3's `RenderedPromptHash`
|
||
is not a substitute. The sibling PromptKit repository has retired the original
|
||
feature request, and its [v0.24.0 release notes](../../../promptkit/docs/releases/v0.24.0.md)
|
||
document the addition and a ConfigKit v1.3.0 dependency. U1 remains open to review
|
||
the supplying API/release and verify a compatible assembled dependency set before
|
||
changing the selected baseline. Independent work can proceed; completed cache
|
||
integration must use the upstream capability. Do not add a local hash-policy fork.
|
||
Neither a cross-backend global limit nor a comprehensive execution fingerprint
|
||
is required. Remove obsolete imports and catalog dependencies when unused;
|
||
avoid permanent `replace` directives and update [attribution](../oss/attribution.md)
|
||
under the [OSS policy](../policy/oss.md).
|
||
|
||
Audita's reviewed v0.23.2 integration demonstrates native source assembly,
|
||
continuation, observations, and CommandKit wiring. Adopt those mechanisms
|
||
without copying its `RoleClient`, structured-output interface, or opaque exchange
|
||
wrapper. Its per-proposal repair branches and lack of checkpoints do not define
|
||
Notarius's artifact and recovery policies.
|
||
|
||
## Target Architecture
|
||
|
||
### Application Structure And Ownership
|
||
|
||
| Part | Responsibility |
|
||
| --- | --- |
|
||
| Application composition | Load configuration and native sources; construct one shared PromptKit engine for producers, validators, and continuation runs. |
|
||
| Artifact-family modules | Prepare domain inputs, select prompt IDs and response requirements, interpret responses, apply deterministic domain rules, and provide supported fallbacks. |
|
||
| Pipeline execution | Connect generation to configured validation and strict/permissive handling; manage ordered dependencies, typed handoffs, bounded workers, and stable result ordering. |
|
||
| PromptKit | Prepare and execute requests; maintain captured exchanges and conversations; enforce native retry, repair, turn, and admission budgets; report execution facts. |
|
||
| Application state and records | Publish artifacts, cache final accepted artifacts, provide a small completion summary, and capture optional debug evidence with pipeline context. |
|
||
| ConfigKit and CommandKit | ConfigKit owns configuration mechanics; CommandKit and ckconfig/ckprompt own standardized commands and presentation. Notarius supplies loaders, request builders, report content, and outcome mapping. |
|
||
|
||
Use native PromptKit requests, results, exchanges, errors, and observers in
|
||
model-executing application code. Remove the private provider-neutral completion
|
||
API. Pure domain models, codecs, source-reference resolution, and deterministic
|
||
transformations need no PromptKit dependency. Application interfaces continue to
|
||
express artifact production, typed validation, output encoding, and state access.
|
||
|
||
#### Module Execution And Assessment (U3 Resolved)
|
||
|
||
Keep the existing typed stage boundaries and use native PromptKit values in
|
||
model-executing code. Place engine construction and shared loaders in focused
|
||
files in the `internal/cli` composition root. It constructs one native engine
|
||
for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset
|
||
registry, and reporting responsibilities disappear; do not retain a package
|
||
solely to hold a small constructor or introduce another LLM service interface.
|
||
|
||
Place one small shared producer helper alongside assessment in
|
||
`internal/framework/pipeline`. Connect modules to it through ordinary functions:
|
||
|
||
1. The pipeline supplies an operation-scoped assessment callback capturing the
|
||
configured validator chain and acceptance criteria. It applies the selected
|
||
strict/permissive mode when no acceptable result is obtained.
|
||
2. The module prepares source material, references, and any normalization handle
|
||
mappings once. It builds the native initial request and supplies response
|
||
interpretation, correction-request, and supported fallback functions.
|
||
3. The helper receives those functions, the shared engine, and relevant
|
||
cache/debug collaborators. For cacheable work it prepares the initial request
|
||
and looks up the accepted-artifact cache; on a miss it executes the retained
|
||
request. Otherwise it uses ordinary native execution.
|
||
4. For each executed response, the helper invokes module interpretation and
|
||
pipeline assessment. When correction is appropriate, it supplies the module's
|
||
next request to native coordination. PromptKit owns conversation state and
|
||
turn iteration; the helper does not implement another retry engine.
|
||
5. Return a typed application outcome. Cache only final accepted artifacts under
|
||
the initial request's key. A cache hit follows the existing direct-reuse policy
|
||
without model execution or renewed semantic assessment.
|
||
|
||
Request builders return `promptkit.RunRequest`; interpreters consume native
|
||
results. Candidates, findings, and acceptance outcomes remain application-owned.
|
||
Keep operation-specific state in local variables or closures, without an
|
||
execution-plan language, replacement completion API, or copied conversation model.
|
||
|
||
Use one pipeline assessment implementation for the initial candidate and every
|
||
correction, including the final permitted response. Do not repeat final validation
|
||
merely because the helper returns to the outer runner. Deterministic stages keep
|
||
their ordinary typed production/validation path without synthetic requests or
|
||
conversation machinery. LLM validators call the shared engine directly and return
|
||
domain findings; they do not automatically acquire a producer correction loop.
|
||
|
||
Modules receive operation data and collaborators, not physical storage roots or
|
||
CommandKit reporters. Pure domain packages retain artifact models, codecs,
|
||
reference resolution, and deterministic rules. `internal/preflight` retains its
|
||
separate known-dependency checks. Exact argument grouping and helper signatures
|
||
can be refined against representative extraction and normalization during
|
||
implementation; ownership and placement are settled. U4 selects prompt roles and
|
||
guidance ownership below; U5 selects retry controls and the correction budget.
|
||
U6 selects strict/permissive handling and module-owned fallbacks below; its default,
|
||
configuration scope, and concrete module fallback choices remain open.
|
||
Implementation is pending.
|
||
|
||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||
and deterministic public ordering. Keep modules independent of physical output,
|
||
cache, and debug roots. Native adoption does not turn the pipeline into a general
|
||
workflow engine or redesign durable D&D artifacts.
|
||
|
||
### Command Surface And Native Inputs
|
||
|
||
Install the application `pipeline` family alongside ckconfig and ckprompt:
|
||
|
||
| Command | Responsibility |
|
||
| --- | --- |
|
||
| `pipeline list` | Discover configured pipelines. |
|
||
| `pipeline show <pipeline-id>` | Report resolved steps, lanes, modules, validators, and reference relationships. |
|
||
| `pipeline validate <pipeline-id>` | Check composition and known prompt/profile dependencies offline. |
|
||
| `pipeline run <pipeline-id> --input-file PATH` | Execute the pipeline and publish artifacts. |
|
||
| `run <pipeline-id> --input-file PATH` | Native alias for `pipeline run`, with the same handler and inputs. |
|
||
|
||
Use `commandkit.Alias("run", "pipeline run")`, not a forwarding command. Use
|
||
`notarius run` in quick starts and document both routes. Install the full selected
|
||
standard suite: native help/version; Bash, Zsh, and Fish completion; configuration
|
||
scopes, validation, show, sources, explain, and diff; PromptKit resource listings
|
||
and inspection for prompts, profiles, backends, namespaces, and collections;
|
||
requirements discovery/validation and prepared-request preview; and explicit
|
||
reachability, authentication, text, structured-output, and aggregate smoke tests.
|
||
Include upstream convenience aliases and useful completion hints.
|
||
|
||
Use native typed declarations and supplied-value semantics. Defaults apply to
|
||
omission, not explicit empty/false/zero values. Preserve native command-path,
|
||
operand, repeated-flag, and `--` parsing conventions. Check every occurrence of
|
||
a singleton flag and use the last successfully parsed value.
|
||
|
||
| Concern | Selected input |
|
||
| --- | --- |
|
||
| Configuration | Native `--config`, `--overlay`, and supported application `--set POINTER=JSON`; scope named `app`. |
|
||
| Pipeline input | Required pipeline-ID operand and `--input-file PATH`. |
|
||
| Lanes | Repeatable `--lane ID`; omission selects the normal lane set. Commas are literal values, not separators. |
|
||
| Output | Application `--output-dir PATH`. |
|
||
| Chunk-plan cache | `--chunk-cache auto\|bypass\|refresh`. |
|
||
| Accepted-artifact cache | Reuse/bypass/refresh controls; spelling, scope, and defaults remain U8. |
|
||
| References | Repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`. |
|
||
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
|
||
| Capture destination | Application `--debug-dir PATH`, with the semantics below. |
|
||
|
||
Keep worker counts and semantic budgets in configuration, adjustable through
|
||
application `--set`. U5 selects `max_additional_turns`; worker defaults remain U7.
|
||
Use ckprompt's reusable profile, session, and execution flags, including model,
|
||
temperature, timeout, reasoning effort, and transient retries, for run-wide
|
||
overrides across producers, validators, and continuations. Omission preserves
|
||
normal selection; explicitly
|
||
empty reasoning effort clears inheritance. Use native `--llm-profile` and
|
||
`--llm-session-id`, retiring the separate reasoning-clearing control. Native
|
||
transient retries remain distinct from semantic additional-turn budgets.
|
||
Individual prompt selection and named prompt inputs belong on standard PromptKit
|
||
commands; pipeline modules build their own requests.
|
||
|
||
Notarius retains reference-selector expansion, specificity, required slots, and
|
||
external/generated-reference conflict checks. Pipeline reference selectors are
|
||
not individual-request `--llm-input-file` bindings. Remove custom dispatch,
|
||
assembled help, standard flag parsing, comma-separated `--only`, and bespoke
|
||
`--json` switches. Replace `config effective` with `config show` and `config
|
||
profile` with `promptkit show profile`. Old flags are migration evidence only.
|
||
|
||
### Application Composition, Loaders, And Preflight
|
||
|
||
One constructor registers application commands, the native alias, ckconfig's
|
||
`app` and `promptkit` scopes, ckprompt, and native help/version/completion/reporting.
|
||
It does not load configuration, build an engine, or initialize run storage.
|
||
Help, version, and completion work without application configuration.
|
||
|
||
Share two loading responsibilities across application and standard commands:
|
||
|
||
1. Select application configuration through explicit `--config`, then existing
|
||
environment/discovery policy. Let ConfigKit apply overlays, imports,
|
||
inheritance, overrides, defaults, normalization, and structural validation.
|
||
Retain its native result and selected main-source context. Validate an
|
||
explicitly empty configuration path rather than falling through to discovery.
|
||
2. Build the native engine from that application's source context using the source
|
||
assembly below. Across commands, `--config` selects the application document;
|
||
the PromptKit scope inspects the engine assembled for it. Request overrides
|
||
do not mutate captured engine configuration. Do not emulate unsupported
|
||
arbitrary `--set` overrides for the PromptKit scope.
|
||
|
||
Use small invocation-owned state through CommandKit's facilities to reuse loaded
|
||
values. A pipeline shares one engine across all roles. `config diff` may load two
|
||
distinct configurations within one invocation: keep their selections separate,
|
||
including overlays/overrides. No persistent loader cache or configuration hash
|
||
framework is needed.
|
||
|
||
Reuse the application resolver for module/options checks, typed compatibility,
|
||
ordered dependencies, validator chains, and reference bindings. `pipeline list`
|
||
only enumerates configuration; `show` renders resolved composition rather than
|
||
building an engine to expand every profile. ConfigKit supplies loaded values and
|
||
native provenance/secret-aware inspection, not domain pipeline semantics.
|
||
|
||
Centralize known prompt/profile dependency checks in `internal/preflight`. It
|
||
accepts the resolved pipeline, module declarations, and native engine. Check
|
||
every statically known dependency, including LLM validators and correction
|
||
prompts, plus applicable/default profiles. Partial profiles alone are not errors.
|
||
Share identifiers and selection logic with execution; no independent inventory,
|
||
asset registry, or synthetic request execution is needed. Dynamic selections
|
||
receive their final checks at runtime. Findings identify pipeline, step/lane,
|
||
stage/validator, and resource where applicable.
|
||
|
||
`pipeline validate` and `run` call the same resolution/preflight functions;
|
||
handlers do not invoke other commands or parse their reports. Keep `config
|
||
validate` focused on configuration and remove its pipeline-specific mode.
|
||
Offline preflight reads configuration and prompt resources but needs no transcript,
|
||
reference contents, credential values, or provider calls. Concrete PromptKit
|
||
preparation/validation owns completed-request checks, including schemas and
|
||
credentials; provider smoke tests require explicit invocation.
|
||
|
||
Preflight does not discover configuration, register modules, build engines,
|
||
initialize storage, or render CLI reports. Modules do not import it. Use native
|
||
request builders for standard commands and add application callbacks only where
|
||
needed, preserving supplied operator values. Output/cache/debug allocation
|
||
belongs to the run path, with debug timing selected below.
|
||
|
||
### Configuration, Sources, And Prompt Assets
|
||
|
||
Introduce application configuration version 6 as the only supported version.
|
||
Preserve unrelated discovery, pipeline structure, and resolution behavior.
|
||
ConfigKit continues to own loading and composition under
|
||
[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
|
||
Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of
|
||
native companion sources; do not introduce a separate `promptkit.config_file`.
|
||
Retire `concurrency.total_llm` and revise retry/worker settings as described below.
|
||
U5 selects `max_additional_turns` and removes application structural-repair
|
||
controls; worker defaults and retired global-limit migration remain U7.
|
||
|
||
Use one source assembly for runtime, preflight, and inspection:
|
||
|
||
1. PromptKit's bundled OpenRouter underlay.
|
||
2. Notarius's embedded native prompts and profiles under `collections.notarius`.
|
||
3. An optional, higher-priority operator underlay containing `prompts.yml` and
|
||
recursive `.yml`/`.yaml` files under `prompts/`, beside the main application
|
||
configuration actually selected by ConfigKit.
|
||
|
||
Anchor companion discovery to that selected main source, whether explicit or
|
||
discovered. Configurations in the same directory share companions; there is no
|
||
working-directory fallback. The two companion locations are strict peers, so
|
||
filenames do not resolve conflicting operator records. Missing companions are
|
||
fine; loading errors do not silently fall back to embedded defaults. Preserve
|
||
native imports and source-relative resource paths.
|
||
|
||
Native declarations become the sole prompt assembly mechanism. Keep the embedded
|
||
content tree and module ownership of prompt meaning, response requirements, and
|
||
domain interpretation. Remove prompt/fallback-profile registration in
|
||
`AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`,
|
||
shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration
|
||
for checkpoint fingerprints. The resolved schema design below also removes
|
||
schema registration and filesystem-flattening machinery.
|
||
|
||
Convert maintained resources, examples, and fixtures to native identified lists,
|
||
`config:` source controls, and sibling inheritance (`extends: {id: parent}`).
|
||
Same-ID replacement supplies a complete record; distinct inherited variants can
|
||
customize an existing resource. Explain native profile-null/backend fallback
|
||
semantics. The optional PromptKit source header is version 2, independently of
|
||
Notarius's application version. Do not translate legacy resource formats or
|
||
synthesize compatibility aliases.
|
||
|
||
Select prompts by ID alone; remove obsolete PromptKit version selectors and
|
||
related request/debug metadata. Retain current local resource IDs and use exact
|
||
qualified references, such as `notarius/dnd.spells` and
|
||
`notarius/dnd-extraction`. Keep the extraction profile's existing model/settings
|
||
and its reference to the bundled unqualified `openrouter` backend. Inherited
|
||
application profiles use sibling local IDs in the same collection. The bundled
|
||
inventory differs from the old external catalogs and selects no model for an
|
||
application request; document intentional replacements for removed catalog IDs.
|
||
The bundled backend supplies no concurrency limit. Application/operator limits
|
||
need explicit native configuration; application-supplied defaults remain U7.
|
||
|
||
Preserve prompt prose, message ordering, roles, and cache boundaries where their
|
||
meaning carries forward. Review and deliberately revise initial/repair prompts
|
||
for accumulated conversation history; preserving every old byte or retry message
|
||
is not a migration requirement. Shared content should use native filesystem
|
||
contexts and imports rather than reconstructed virtual trees.
|
||
|
||
#### Prompt Assets And Response Schemas (U2 Resolved)
|
||
|
||
Use the existing root `assets` package's ordinary `embed.FS` directly for native
|
||
prompt sources and response schemas, with `fs.Sub` only where useful. Shared
|
||
message files use their actual paths within the native filesystem context.
|
||
Remove synthetic mounts, file copying, flattened schema namespaces, and module
|
||
asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening
|
||
consumers disappear; no replacement filesystem abstraction is needed.
|
||
|
||
Declare response format, validation mode, and schema path in native prompt
|
||
definitions by default. Omit structural-repair attempt settings in bundled
|
||
prompts to inherit PromptKit's upstream default under U5. Modules select prompt
|
||
IDs, supply inputs, and interpret responses. U4 selects the same schema file for
|
||
each initial/correction pair. Execution, inspection, and prepared-request preview
|
||
then use the same declared contract without repeating schema selection in Go.
|
||
U4 selects prompt roles and contracts; exact prose is implementation work.
|
||
U5 selects budget ownership and defaults.
|
||
|
||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||
dependent contract. It replaces the whole configured contract, so supply the
|
||
intended format, validation mode, and schema together while leaving structural
|
||
repair at its native default. Do not inject a repair count merely to construct
|
||
the override, or introduce a wrapper or per-call registry solely to reproduce
|
||
Audita's approach. Operator prompt replacements must still produce responses the
|
||
selected module can interpret; that module boundary does not require duplicating
|
||
each prompt's schema selection in Go.
|
||
|
||
Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to
|
||
that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`.
|
||
They are separate from message-file source contexts: prompt underlays do not
|
||
implicitly overlay schema files. When an operator configures an external schema
|
||
directory, select PromptKit's native directory source instead of the embedded
|
||
root. It replaces the whole schema source and must supply the schemas used by
|
||
the selected prompts. Do not build an application filesystem overlay to mix
|
||
embedded and external schemas. Transparent layering would be a separate upstream
|
||
capability, not a requirement for this migration. Concrete configuration binding
|
||
is an implementation detail within this selected source policy.
|
||
|
||
Keep private response schemas as self-contained JSON documents. PromptKit owns
|
||
loading, compilation, structural validation, provider structured-output settings,
|
||
and structural repair. Remove Notarius's response-schema byte registry, copied
|
||
schema wrappers, defensive-copy layer, digest calculation, and duplicate
|
||
structural-validation machinery. Native prepared snapshots provide diagnostic
|
||
schema details; no schema fingerprint is added to the message-content cache key.
|
||
|
||
Keep typed decoding, source/reference checks, reconciliation safety, semantic
|
||
validators, artifact codecs, and durable artifact schemas in Notarius. Response
|
||
schemas describe model output; artifact schemas describe published and consumed
|
||
domain artifacts. Preserve identifiers and versions belonging to actual artifact
|
||
contracts. Retire private schema keys, registration records, hashes, and copied
|
||
identity/version/name metadata whose only consumers were adapters, checkpoints,
|
||
or reporting. No independent schema metadata system replaces them.
|
||
|
||
Implementation is pending. Verify native preparation of embedded and operator
|
||
prompt definitions, shared-file paths, schema selection from embedded and external
|
||
roots, and complete request-contract replacement where actually used. Missing or
|
||
invalid schemas should follow native preparation behavior. No new schema check
|
||
requiring synthetic requests is added to offline pipeline preflight.
|
||
|
||
### Native Production, Continuation, And Assessment
|
||
|
||
Start with one conversation per producer operation: a chunk-plan generation,
|
||
one lane's extraction of one chunk, or one lane's normalization pass. Use
|
||
complete replacement model responses. Normalization may continue to return a
|
||
reconciliation plan that deterministic code applies to produce the artifact.
|
||
Per-entity branching is not part of this migration.
|
||
|
||
#### Prompt Roles And Correction Guidance (U4 Resolved)
|
||
|
||
Give each correctable producer an initial native prompt and a domain-specific
|
||
correction prompt. Retain existing initial IDs and identify correction IDs
|
||
explicitly in the module, for example `notarius/dnd.spells.repair`. Preflight uses
|
||
those same declarations to check known prompt dependencies. Shared correction
|
||
wording can live in ordinary asset files; distinct domain/operation templates
|
||
supply the task and response expectations without a universal application-aware
|
||
correction protocol.
|
||
|
||
The initial request supplies source material, references, and the task. Enduring
|
||
domain principles belong in its system instructions. The correction template
|
||
supplies a new user assignment with the current candidate's actionable findings;
|
||
PromptKit carries the preceding exchange. Do not rerender the original source or
|
||
manually replay prior responses. Review initial wording so later correction
|
||
assignments remain valid. Include additional application-derived context
|
||
only when needed to interpret the findings. Keep source/reference meanings and
|
||
normalization candidate handles fixed throughout the conversation.
|
||
|
||
Use the same declared response schema for each initial/correction pair:
|
||
|
||
| Role | Response contract |
|
||
| --- | --- |
|
||
| LLM chunk planning | Complete replacement chunk plan. |
|
||
| Extraction | Complete replacement extracted artifact candidate. |
|
||
| Normalization | Complete replacement reconciliation plan, applied afresh to the fixed original candidates. |
|
||
| LLM validation | The validator's assessment response; no separate semantic correction conversation by default. |
|
||
|
||
No correction envelope, patch language, or special repair-result type is needed.
|
||
PromptKit retains structural validation and repair inside each native execution.
|
||
Notarius assesses the response returned by that public boundary, independently
|
||
of how many internal attempts produced it.
|
||
|
||
Use the same native default profile for an initial/correction pair. Preserve
|
||
explicit application profile selections and run-wide execution overrides across
|
||
both requests. An operator may select a different default profile in a correction
|
||
prompt, subject to explicit request overrides. Validators retain their own
|
||
profile selection. Pass native selections and overrides without copying resolved
|
||
settings or adding a Notarius-wide repair-model configuration layer.
|
||
|
||
The component that detects a defect owns its actionable correction guidance:
|
||
|
||
- Each rejecting validator supplies guidance describing the defect and the
|
||
constraints a corrected answer must satisfy. A deterministic validator authors
|
||
it alongside the check; an LLM validator requests an actionable explanation in
|
||
its prompt and returns it through the application validation contract.
|
||
- Preserve the explicit `CorrectionGuidance` boundary separately from operator
|
||
diagnostics. Guidance need not prescribe an exact replacement, and validators
|
||
do not construct the complete correction prompt.
|
||
- Module interpretation and normalization safety checks supply their own guidance
|
||
when defects arise before an artifact candidate can be materialized.
|
||
- The pipeline collects guidance in configured validator order. The module's
|
||
domain-specific template renders it into the correction assignment. Neither
|
||
the pipeline helper nor the template duplicates validators' defect-detection
|
||
logic. Reason codes and operator messages are not automatically instructions.
|
||
|
||
Assess the same candidate through every applicable configured validator before
|
||
requesting correction. A rejection does not short-circuit later validators;
|
||
collect all actionable findings they return. Reassess every revised candidate
|
||
through the complete applicable chain, including earlier approvals and the final
|
||
permitted response. The existing validator-owned guidance and ordered-chain
|
||
semantics remain useful; replace the hard-coded `CorrectionRequest` message
|
||
builder with native template rendering and retain useful ordered aggregation.
|
||
|
||
Applicability requires usable input: if interpretation cannot construct a typed
|
||
candidate, its module supplies guidance and artifact validators cannot yet run.
|
||
Cancellation or fatal execution failures can interrupt assessment. An unavailable
|
||
validator assessment is a separate outcome, not a detected defect or an instruction
|
||
to change the artifact. Do not convert provider errors into correction guidance;
|
||
apply U6's strict/permissive handling when no acceptable result is obtained.
|
||
|
||
Ownership, response contracts, and default profile behavior are resolved. Refine
|
||
exact template prose, correction IDs, and necessary operation-specific inputs
|
||
while implementing representative workflows. Implementation is pending; U5 selects
|
||
semantic budgets and U6 selects terminal handling, with defaults, configuration
|
||
scope, and concrete module fallback choices still open.
|
||
|
||
#### Execution And Assessment Flow
|
||
|
||
The application flow is:
|
||
|
||
1. Prepare source context, references, and candidate-handle mappings once. Build
|
||
the native initial request. For cacheable work, prepare and check the cache
|
||
before execution as described below; execute the retained request on a miss.
|
||
Otherwise use ordinary `Engine.Run`. Capture an exchange when continuation
|
||
or debug inspection needs one.
|
||
2. Consume the native usable response or terminal failure, decode, and apply
|
||
module interpretation and safety checks. A module can supply actionable
|
||
findings before it has a materialized candidate. Assess each candidate through
|
||
all applicable configured validators and collect their actionable guidance
|
||
before requesting correction, as selected in U4.
|
||
3. For actionable correction, call `Engine.Coordinate` with the captured exchange
|
||
and the resolved `max_additional_turns` as native `AdditionalTurns`. Its callback
|
||
selects a repair prompt and supplies findings; PromptKit retains the initial
|
||
exchange and subsequent turns.
|
||
4. Interpret every delivered revised response and rebuild from the fixed original
|
||
inputs. Run the complete configured validation chain again for changed
|
||
candidates, including validators that previously approved. Assess the final
|
||
permitted response before declaring exhaustion.
|
||
5. Stop on acceptance or a terminal condition. If no acceptable result is obtained,
|
||
fail in strict mode or produce the module's supported fallback in permissive
|
||
mode. Native `stopped` and `exhausted` outcomes do not themselves mean application
|
||
acceptance. Cancellation stops work without constructing a fallback.
|
||
|
||
Accumulated history increases context use; U5's allowance bounds additional
|
||
semantic correction while stopping as soon as the candidate is accepted.
|
||
|
||
Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection`
|
||
transport, manual assistant/user replay, and correction-protocol capability
|
||
checks. Replace `runProducerAttempts`' LLM retry loop with native coordination and
|
||
application assessment. Adapt `NormalizeRetry` and `producerRetryDirective` to
|
||
findings and explicit fallback information. Retain useful ordered collection and
|
||
deduplication of actionable findings, rendered through a native repair prompt.
|
||
Reason codes and operator messages are not automatically model instructions.
|
||
|
||
#### Retry Controls And Correction Budget (U5 Resolved)
|
||
|
||
Delegate transport retries and structural repair to PromptKit. Notarius selects
|
||
only the allowance for application-directed semantic correction:
|
||
|
||
| Concern | Control and selected default |
|
||
| --- | --- |
|
||
| Temporary provider failures | Native PromptKit transient retries; inherit the upstream default when omitted and preserve native execution overrides. |
|
||
| Structurally invalid model output | Native output-contract policy; bundled prompts and application requests inherit PromptKit's upstream structural-repair default. |
|
||
| Actionable domain defects | Application `max_additional_turns`, passed directly to native `AdditionalTurns`; application default `2`. |
|
||
|
||
`max_additional_turns` is a nonnegative maximum, excluding the initial producer
|
||
response. Resolve producer-binding override, then pipeline override, then the
|
||
application default of `2`. Omission inherits; explicit `0` disables semantic
|
||
correction. It applies to LLM-backed producers. Validators do not receive a
|
||
separate semantic continuation budget, and deterministic stages do not acquire
|
||
conversation machinery. Keep the field in configuration, adjustable through the
|
||
existing application `--set` mechanism, without another dedicated CLI flag.
|
||
|
||
For example, a producer binding can override its pipeline allowance:
|
||
|
||
```yaml
|
||
extract:
|
||
module: dnd/spells
|
||
max_additional_turns: 2
|
||
```
|
||
|
||
Two additional turns are an allowance, not mandatory executions. An accepted
|
||
initial candidate needs none; an accepted first correction needs only one. A
|
||
second turn is available when actionable defects remain and continuation is
|
||
appropriate. Assess every delivered response, including the final permitted turn,
|
||
and stop on acceptance or a terminal condition. PromptKit enforces the turn
|
||
limit; do not add an application counter or retry loop. Internal transport
|
||
attempts and structural repairs do not consume this semantic-turn allowance.
|
||
The default is an application policy choice, not a measured claim about how
|
||
frequently corrections will be needed.
|
||
|
||
For structural repair, omit `output.repair_attempts` from all bundled initial,
|
||
correction, and validator definitions. Remove Notarius's pipeline/binding/
|
||
validator `structured_output_repair_attempts` fields and resolution plumbing.
|
||
Do not replace them with a hard-coded default, including zero, in request
|
||
construction or a native output-contract override. Explicit operator
|
||
customization remains available through native PromptKit prompt configuration.
|
||
|
||
Notarius deliberately inherits future upstream default changes. An anticipated
|
||
PromptKit default change is not a migration prerequisite, and its timing or
|
||
chosen count is not a Notarius contract. Notarius still declares response format,
|
||
validation mode, and schema; PromptKit owns the mechanics of obtaining a valid
|
||
response. At the public boundary, handle a usable result or terminal failure,
|
||
including a returned result reporting unsuccessful structural validation.
|
||
Do not restart the operation to extend PromptKit's exhausted repair budget.
|
||
|
||
Remove blanket outer LLM retries, including repeated LLM validator invocations in
|
||
`executeValidationChain`. Keep ordered candidate assessment and distinguish a
|
||
validator execution failure from a domain rejection. Use native `OperationTimeout`
|
||
when that operation scope is needed, without adding an arbitrary default or
|
||
replacing broader application deadlines. Cancellation stops work.
|
||
|
||
Remove generic retries for non-LLM operations as well. Parsing, chunk
|
||
materialization, deterministic merging, normalization-plan application, and
|
||
encoding run without an outer attempt loop. Repeating those operations against
|
||
unchanged inputs ordinarily cannot correct a defect. Return operation errors
|
||
through normal pipeline failure handling; apply the selected terminal policy to
|
||
a rejected deterministic candidate instead of regenerating the identical value.
|
||
Cancellation stops work.
|
||
|
||
A demonstrated transient I/O recovery need belongs with that specific operation,
|
||
where repeatability and failure conditions are understood. Do not retain a generic
|
||
stage retry setting or framework in anticipation of filesystem or future remote-
|
||
storage failures. After a fix, an operator can rerun the pipeline using the
|
||
accepted-artifact cache for successful predecessors while cheap deterministic
|
||
preparation runs again.
|
||
|
||
Retire producer and validator `retries` fields entirely, together with the generic
|
||
outer attempt loop. Migration guidance maps intent: a producer allowance intended
|
||
for semantic correction becomes `max_additional_turns`; native transport controls
|
||
and prompt-owned structural customization cover their own concerns. Remove
|
||
non-LLM retry settings without a replacement generic allowance. Do not promise
|
||
equivalent total provider-attempt counts. U5 is resolved; U6 selects strict/permissive
|
||
handling, with defaults, configuration scope, and concrete module fallback choices
|
||
still open. Implementation is pending.
|
||
|
||
#### Terminal Outcomes And Module Fallbacks (U6 Direction Resolved)
|
||
|
||
Use two modes when an operation does not obtain a successful, validated result:
|
||
|
||
| Mode | Terminal behavior |
|
||
| --- | --- |
|
||
| Strict | Require a successful, validated result; otherwise fail the pipeline. |
|
||
| Permissive | Produce the module's supported fallback, record a warning, and continue with that artifact. |
|
||
|
||
The pipeline selects the mode; the module owns the fallback algorithm. Replace
|
||
the failure-category policy object and separate `fail_run`, `reject_output`, and
|
||
`warn_continue` choices with this selection. An empty artifact is a supported
|
||
fallback where the domain permits it; no separate omit-output action is needed.
|
||
The default mode, configuration spelling/scope/precedence, and concrete module
|
||
fallback definitions remain open. Do not infer them from existing policy fields.
|
||
|
||
Attempt application-directed correction when appropriate within the selected
|
||
budget before terminal handling. Neither mode extends PromptKit's native retry
|
||
or repair budgets or adds retries to deterministic operations. Preserve the
|
||
distinction between a detected defect and an unavailable assessment for correction
|
||
guidance and diagnostics; neither establishes ordinary acceptance. Cancellation
|
||
remains cancellation and does not initiate fallback work. If a supported fallback
|
||
cannot be constructed, fail execution.
|
||
|
||
Fallbacks must satisfy the structural artifact contract and invariants required
|
||
by downstream stages. They need not satisfy every ordinary semantic acceptance
|
||
criterion whose failure led to fallback. Do not rerun the complete ordinary
|
||
validator chain as a condition of using the fallback or claim normal validation
|
||
passed. Keep the necessary checks with module fallback construction; no separate
|
||
configurable fallback-validation framework is needed.
|
||
|
||
The selected direction supports these domain-specific fallbacks:
|
||
|
||
| Module | Fallback direction |
|
||
| --- | --- |
|
||
| D&D spell extraction | A valid empty spell artifact, so that the failed chunk contributes no extracted spells. |
|
||
| D&D transcript chunking | A deterministic partition preserving the entire transcript. Whole-transcript-as-one-chunk versus deterministic bounded slices remains open. |
|
||
| Normalization | Apply only safe consolidation groups from the final usable response afresh to the fixed originals, retaining everything else. If that response is unusable, use the deterministic merge of the input chunk artifacts. Reuse an already available merged input. |
|
||
|
||
Keep source grounding, reference resolution, candidate isolation, and
|
||
reconciliation safety checks. Unknown or overlapping handles remain invalid;
|
||
partial normalization applies only compatible safe groups and leaves other
|
||
original candidates separate. Use the final response and fixed original inputs,
|
||
without recovering an earlier candidate or combining proposals across turns.
|
||
Concrete module algorithms must make their supported fallback and required
|
||
invariants explicit; these examples do not introduce a generic fallback language.
|
||
|
||
Fallbacks may be published and consumed downstream, but they and dependent results
|
||
affected by their incompleteness are ineligible for accepted-artifact cache lookup
|
||
or storage. Carry the small application outcome facts needed to skip that reuse;
|
||
do not add dependency fingerprints or a provenance ledger. Fully validated,
|
||
unaffected predecessors remain reusable, allowing later runs to retry work that
|
||
previously needed fallback.
|
||
|
||
Record why fallback was used and expose that qualification in the ordinary
|
||
completion report, even when diagnostic verbosity hides individual warnings.
|
||
Use the same application outcome for reporting and optional debug detail. Keep
|
||
conversation state in process, outside ordinary artifact/cache payloads;
|
||
explicitly requested detailed debug is separate. Implementation is pending, and
|
||
U6 remains open only for the default mode, configuration contract, and concrete
|
||
module fallback choices.
|
||
|
||
### Admission And Pipeline Workers
|
||
|
||
Remove `Scheduler`, `scheduledClient`, and custom global queue/permit plumbing.
|
||
Retire the separate cross-backend ceiling and the default/maximum extraction
|
||
worker coupling to `concurrency.total_llm`. Keep bounded extraction workers and
|
||
bounded lane continuations with independent defaults to be selected.
|
||
|
||
Use one shared native engine across all roles. Native capacity belongs to the
|
||
engine and backend ID: profiles selecting the same ID share its pool; different
|
||
backend IDs or separate engines do not automatically share limits, even for the
|
||
same endpoint. Endpoint-only profiles and backends without a positive limit
|
||
remain unrestricted by native backend admission.
|
||
|
||
For limited backends, the reviewed baseline bounds active model attempts by
|
||
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||
Ordinary execution admission spans preparation, retries, and repairs; split
|
||
`PrepareExecution` itself reserves no capacity, with admission occurring during
|
||
`RunPrepared`. Active permits are released during retry waits.
|
||
A completed continuation turn releases admission before assessment, allowing a
|
||
nested validator to use the same backend. Configure native limits at the intended
|
||
capacity boundary and handle `ErrCapacityExceeded` through application policy.
|
||
|
||
Aggregate concurrency across independent backends may exceed the former global
|
||
ceiling. That change is intentional. Bounded pipeline workers are application
|
||
work controls, not a substitute cross-backend LLM ceiling. Defaults, retired-field
|
||
migration, and role-specific admission-failure dispositions remain open.
|
||
|
||
### Accepted-Artifact Cache
|
||
|
||
Replace the checkpoint subsystem with a cache around the complete Notarius
|
||
producer operation. Its value is an accepted answer to the original task. This
|
||
avoids repeating expensive model work after a downstream failure without caching
|
||
every cheap deterministic pipeline step.
|
||
|
||
```text
|
||
Prepare initial request -> obtain MessageContentHash -> look up cache/<hash>.json
|
||
Hit: decode the accepted artifact and continue the pipeline
|
||
Miss: execute the retained request and complete the Notarius operation
|
||
-> interpret, validate, and perform application-directed corrections
|
||
-> on final acceptance, store the eligible artifact under the INITIAL hash
|
||
```
|
||
|
||
Use the upstream message-content hash of ordered message roles and exact text.
|
||
Include message boundaries and order, excluding model/settings, session/execution IDs,
|
||
timing, provider cache hints, and other non-content metadata. Add no stage,
|
||
prompt-ID, library-version, whole-configuration, schema-policy, or implementation
|
||
fingerprint. Actual input or schema text rendered into messages participates as
|
||
message content; metadata outside those messages does not.
|
||
|
||
| Change | Intended reuse behavior |
|
||
| --- | --- |
|
||
| Fix a downstream stage | Earlier artifacts remain reusable when their initial messages match. |
|
||
| Change initial message text, roles, order, or boundaries | Cache miss. |
|
||
| Change a prompt ID, model, or settings without changing messages | Cache hit remains possible. |
|
||
| Change deterministic interpretation or validation | Existing accepted artifacts remain reusable; explicit bypass/refresh applies the new policy. |
|
||
|
||
Obtain the key before model execution through native preparation. Preserve the
|
||
prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared`
|
||
where supported, and discard the handle on a hit. `OnPrepared` execution hooks
|
||
alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes
|
||
session identity and cache hints; it is not the requested hash. The upstream
|
||
API/release review in U1 must finish before completing this cache integration.
|
||
|
||
Keep the initial key throughout corrections. Cache the final Notarius artifact
|
||
through its existing codec, not an intermediate response, `RunResult`, exchange,
|
||
or prepared handle. For normalization, save the accepted normalized artifact
|
||
after applying the reconciliation plan, not the proposed plan itself. Save only
|
||
after the entire Notarius operation succeeds, including domain validation and
|
||
any application-directed corrections. Failed, rejected, and validation-incomplete
|
||
work is not cacheable. U6's fallbacks and dependent results affected by their
|
||
incompleteness are ineligible for both lookup and storage, even if the downstream
|
||
operation itself validates successfully. An unrelated accepted predecessor remains
|
||
reusable. This requires a small eligibility fact, not a dependency fingerprint graph.
|
||
|
||
A hit loads the accepted artifact directly, with ordinary decoding/usability
|
||
checks; absent or unusable entries are misses. It makes no model calls, rebuilds
|
||
no result/exchange, replays no corrections, and performs no new semantic
|
||
validation merely because the artifact was cached. It makes no claim of acceptance
|
||
under changed validation policy and contributes no historical token usage to
|
||
current execution totals.
|
||
|
||
PromptKit owns everything inside each native request, including structural
|
||
validation, transport retries, and repairs. Notarius consumes its public response
|
||
or terminal failure and owns domain acceptance. Internal PromptKit attempts do
|
||
not determine cache eligibility. Native diagnostic evidence does not transfer
|
||
that ownership. No cached-response injection, replay API, exchange restoration,
|
||
or PromptKit-owned write-through cache is requested.
|
||
|
||
Use a small filesystem lookup/write facility and ordinary temporary-file/rename
|
||
publication. `cache/<hash>.json` contains the artifact, with no required manifest,
|
||
acceptance ledger, historical provenance, or versioned checkpoint envelope.
|
||
Lookup does not require original output or debug directories. Remove checkpoint
|
||
identities, dependency fingerprints, invocation-wide barriers, and recovery
|
||
bookkeeping. Preserve the separate canonical chunk-plan cache and its materialized
|
||
chunk validation. Cache directory configuration, reuse/bypass/refresh controls,
|
||
and handling of old checkpoint files remain U8; old `--resume`/`--recompute-step`
|
||
conventions are not compatibility requirements.
|
||
|
||
### Inspection And Execution Bookkeeping
|
||
|
||
Delegate configuration/resource inspection, requirements, concrete preview and
|
||
validation, and explicit smoke diagnostics to ckconfig/ckprompt through the shared
|
||
loaders. Remove effective-YAML parsing, the profile-inspector wrapper, and copied
|
||
effective-parameter types. A configured profile may be partial; use native
|
||
prepared/result details for the actual execution target, including overrides.
|
||
Standard commands retain their native reports, support/applicability distinctions,
|
||
and failure behavior. Smoke status zero alone does not prove a check passed.
|
||
|
||
Use ordinary `Run` with `OnPrepared` when actual request details are needed for
|
||
diagnostics; split preparation has a separate purpose for cache lookup. Native
|
||
observations provide terminal facts even without a returned `RunResult`.
|
||
`OnPrepared` is absent on preparation/admission failures and is not a completion
|
||
record. Keep synchronous callbacks lightweight, compose capture/accounting
|
||
observers, and synchronize shared state where operations overlap. `OnPrepared`
|
||
runs while admission is held; persist retained evidence at normal operation
|
||
boundaries.
|
||
|
||
Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope.
|
||
Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning,
|
||
independent accumulators, debug completion wrappers, synthetic call IDs, and
|
||
adapter profile-recording/forwarding. Native snapshots serve actual-target
|
||
inspection; do not reconstruct a durable profile inventory.
|
||
|
||
Use `ExecutionGroup` for selected diagnostic totals, explicitly enrolling initial,
|
||
continuation, and nested validator executions without overlapping double counts.
|
||
Preserve terminal coverage and unknown-versus-zero usage. Reuse adds no historical
|
||
usage to current totals. These facts support diagnostics and capture, not a
|
||
mandatory consumer execution ledger. Notarius's application outcome supplies
|
||
publication and fallback qualifications for the completion summary; assessment,
|
||
failure, and reuse details support optional diagnostics and debug capture without
|
||
independently maintained reporting state.
|
||
|
||
### Artifact Delivery And Completion Output
|
||
|
||
Artifacts are the primary deliverable: publish accepted domain artifacts and
|
||
permitted module fallbacks as files in the output directory. Preserve intrinsic
|
||
domain content and artifact schemas.
|
||
CLI `--format` controls the completion report, not payload encoding. Consumers
|
||
need no run manifest, metadata bundle, persisted receipt, or execution history to
|
||
use the artifacts. Retire reporting-only metadata and mandatory manifests instead
|
||
of rebuilding them with native values.
|
||
|
||
Submit one small structured value through `Reporter.Report`:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `pipeline` | Selected pipeline identifier. |
|
||
| `artifact_count` | Deliverable artifacts, including reused artifacts; excludes intermediates, cache entries, and debug files. |
|
||
| `output_directory` | Absolute published artifact directory. |
|
||
| `debug_directory` | Optional absolute completed run-specific capture directory; omitted when capture is disabled. |
|
||
|
||
Supply a native text template for the ordinary successful case:
|
||
|
||
```text
|
||
Success: dnd-session produced 6 artifacts in /work/results/run-123
|
||
```
|
||
|
||
When capture is available, append `Debug capture: <run-directory>`. Provide the
|
||
path rather than mirroring the bundle. JSON/YAML render the same structured
|
||
value through native serializers. A plain string would render natively as text
|
||
but lose named fields in machine output; the template gives both presentations
|
||
without parallel encoders. No report-version, run-ID, token, profile, warning-count,
|
||
or validation-history collection is required.
|
||
|
||
When a completed run used a fallback, qualify the human summary and structured
|
||
value with that fact, even if native diagnostic verbosity suppresses individual
|
||
warnings. For example, a small fallback count can accompany the completion fields;
|
||
exact field names remain implementation details. Do not present the run as if all
|
||
operations obtained normally validated results. Detailed reasons belong in optional
|
||
diagnostics/debug capture, without per-producer histories in the completion report.
|
||
Quiet mode continues to suppress stdout.
|
||
|
||
Wire results to stdout and diagnostics to stderr:
|
||
|
||
| Mode | Stdout | Stderr |
|
||
| --- | --- | --- |
|
||
| Ordinary | Brief completion report. | Failure explanation when needed. |
|
||
| `--verbose` | Same completion report. | Native information, warnings, and failures. |
|
||
| `--quiet` | Nothing. | Failure explanation when needed. |
|
||
| `--debug` | Completion report with capture path when enabled. | Native diagnostics and default progress. |
|
||
|
||
Use native mode ordering (last enabled mode wins), explicit progress selection,
|
||
and quiet suppression. Quiet still permits failure explanations and does not
|
||
prevent rendering errors from failing the invocation. `--format json --verbose`
|
||
keeps stdout valid report JSON. Standard configuration/PromptKit commands retain
|
||
their own native values, with no universal Notarius envelope. Artifact-quality
|
||
advisories retain their domain classification.
|
||
|
||
Optional saved reports use shell redirection, without a report-file flag or
|
||
automatic duplicate persistence:
|
||
|
||
```sh
|
||
notarius run dnd-session --input-file transcript.json --format json > run-report.json
|
||
```
|
||
|
||
Publish artifacts and finish requested capture before submitting a successful
|
||
completion report. Use native outcomes: status 0 for command completion, 1 for
|
||
operational/cancellation/publication/presentation failure, 2 for invalid invocation
|
||
or identifiable invalid configuration, and 3 for native `config diff --check`
|
||
differences. Preserve ckconfig's structured classification and underlying causes;
|
||
not every preflight failure is invalid input. A permissive run that completes with
|
||
supported fallbacks succeeds with an explicit fallback qualification. Strict-mode
|
||
failure, inability to construct a supported fallback, and other terminal pipeline
|
||
failures fail the command.
|
||
|
||
On failure, supply a useful native explanation and nonzero status, without a new
|
||
structured error-envelope protocol. `--format` does not serialize diagnostics.
|
||
Retain CommandKit's separation of operation causes from output/diagnostic delivery
|
||
failures. If publication succeeded before capture/report delivery failed, identify
|
||
the published path and available debug location. Published artifacts remain valid;
|
||
do not rerun model work or rewrite execution history. Subprocess callers check
|
||
status before decoding a successful report; failed writes may leave partial stdout.
|
||
|
||
### Debug Capture And Optional Diagnostics
|
||
|
||
Capture the **most convenient** debug representation, rather than optimizing
|
||
for conciseness, exhaustive completeness, or strict deduplication. Native snapshots
|
||
and readable views may repeat evidence. Reuse application decisions and native
|
||
execution facts instead of maintaining a second ledger or transcript model.
|
||
Debug formats are diagnostic data, not stable consumer contracts.
|
||
|
||
#### Controls And Location
|
||
|
||
Bind an application boolean to native `--debug`. Capture requires that boolean
|
||
and a nonempty effective directory; a directory alone does not enable capture.
|
||
Select `--debug-dir PATH` over `debug.directory` over `/tmp/notarius-debug`.
|
||
Resolve nonempty configuration-relative paths against their source and CLI-relative
|
||
paths against the invocation directory. Recognize explicit empty values first:
|
||
`--debug-dir ""` or `debug.directory: ""` disables capture without fallback.
|
||
A nonempty CLI override can re-enable capture over an empty configured root.
|
||
An unusable nonempty destination fails requested capture rather than selecting
|
||
another directory. Remove the configuration-relative `./notarius-debug` default.
|
||
|
||
Capture selection is independent of the final reporting mode. `--debug --quiet`
|
||
captures while suppressing stdout; `--quiet --debug` selects debug verbosity and
|
||
capture. `--debug=false` disables capture through native binding semantics.
|
||
Empty-directory debug retains native verbosity. No separate capture flag or
|
||
capture-depth switch is needed.
|
||
|
||
#### Lifecycle And Failure Handling
|
||
|
||
Allocate the run bundle after application configuration and run identity are
|
||
available, before engine construction and pipeline preflight. Argument/config-load
|
||
failures may have no bundle. Keep allocation out of shared loaders.
|
||
|
||
Create `<debug-root>/<run-id>/` and update `<debug-root>/latest` to a relative
|
||
symlink to that run at session start. It points to the newly started session even
|
||
if that session fails. Do not retarget at completion; concurrent starts use the
|
||
last successful link update. Runs without capture leave it untouched.
|
||
|
||
One application recorder accepts initial context, native evidence, and application
|
||
decisions, then finalizes once with pipeline/publication/capture outcomes. Persist
|
||
as operations advance, without holding the whole pipeline's content in memory.
|
||
Keep debug separate from output/cache and do not automatically delete bundles.
|
||
|
||
Requested capture is required work: allocation/link failures stop before model
|
||
work; recording failures surface at the next normal application boundary;
|
||
finalization failure fails the command. Observer callbacks retain errors for
|
||
boundary checks because they cannot return errors. An existing pipeline failure
|
||
remains primary, with capture failures reported additionally and causes retained.
|
||
Already-written data and published artifacts remain available. Capture failures
|
||
never cause model retries. Finalize before submitting the success report.
|
||
|
||
#### Inventory And Depth
|
||
|
||
Replace separate summary/trace categories with a compact run overview,
|
||
configuration/composition snapshots, parsed source, and per-operation records.
|
||
Use native execution IDs with application scope for navigation. `latest` is the
|
||
selected link name; other filenames and serialized application fields remain
|
||
implementation choices.
|
||
|
||
| Capture | Selected contents and depth |
|
||
| --- | --- |
|
||
| Run overview | Application version, run/pipeline identity, timing, outcome, publication location, effective invocation selections/overrides, and navigation. Terminal facts come from the same application outcome used by the CLI. |
|
||
| Configuration and composition | Native application and PromptKit effective snapshots, normally once each, plus resolved steps, lanes, modules, validators, and references. |
|
||
| Parsed source | Generic input document with unit identities and content, independent of the original input file. If parsing fails, retain the input location and available diagnostics. |
|
||
| Operation inputs | Actual source/chunk and reference content, incoming artifacts, prompt bindings/variables, and fixed original normalization candidates/handle mappings. Include directly even when duplicated elsewhere. |
|
||
| Prepared requests | Native `PreparedRun`, including settings, messages, and exposed output-contract/schema details when available. |
|
||
| Execution events | Native per-execution events in occurrence order, including attempts/retries/repairs, timing, usage availability, and terminal metadata. |
|
||
| Results and responses | Native `RunResult` plus exact `RawOutput` in a separate editor-readable text file. |
|
||
| Conversations | Full `Exchange.Details()` for every application-visible turn, including rejected turns, plus simple readable role-labeled messages. Repeated prefixes are acceptable. Use prepared messages when no completed exchange exists and label the representation. |
|
||
| Interpretation and assessment | Candidate via its codec before acceptance; decode/safety failures; each validator's findings and correction guidance; application disposition, any fallback artifact and reason, and links to LLM validator executions. |
|
||
| Normalization and corrections | Proposed reconciliation plan and applied candidate; constructed correction inputs even if the next preparation fails. |
|
||
| Deterministic stages and reuse | Chunk plans/materialized chunks, merged/normalized artifacts, and the loaded cached artifact with the current hit decision. No reconstructed history or fabricated execution on reuse. |
|
||
|
||
Use indented JSON for native structured snapshots and existing domain codecs.
|
||
Retain selected content in full, with no arbitrary truncation. Enable exchange
|
||
capture where debug inspection requires it, including single-turn calls;
|
||
continuation's in-process exchange needs remain independent of durable capture.
|
||
Keep operation inputs self-contained enough for direct investigation without
|
||
requiring original inputs, a cache entry, or reconstruction of deltas. Do not
|
||
archive every original file or serialize the pipeline's entire internal state.
|
||
|
||
On failure, retain available evidence, where execution stopped, errors, and
|
||
application disposition. Preparation/admission failure may have no `OnPrepared`;
|
||
operational failure may have no `RunResult`. Capture a coordination `Undelivered`
|
||
response as unassessed. Native internal-attempt metadata is useful but intermediate
|
||
provider response bodies are not required. No provider wrapper or complete
|
||
provider-transcript guarantee is part of this design.
|
||
|
||
#### Data Handling And Diagnostic Commands
|
||
|
||
Operators secure debug data as they secure ordinary application inputs and
|
||
outputs, for example with filesystem permissions. Debug contains internal data;
|
||
no separate security/retention model or sanitized-bundle promise is required.
|
||
Do not directly print/serialize ConfigKit-marked secrets, and preserve native
|
||
credential omissions through secret-aware inspection. Do not heuristically scan,
|
||
scrub, or rewrite prompts, source material, model output, or arbitrary strings.
|
||
|
||
Inspection/diagnostic commands use native reports and do not automatically create
|
||
pipeline debug bundles. Operators may redirect those reports. Neither debug mode
|
||
nor capture initiates provider probes. Explicit smoke checks retain native
|
||
support, applicability, and outcome distinctions.
|
||
|
||
## Unresolved Choices
|
||
|
||
The architecture and public direction above are selected. Resolve each remaining
|
||
choice before its dependent work; independent work need not wait. Routine API
|
||
placement and serialization choices are implementation details, not reasons to
|
||
reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||
|
||
| ID | Remaining choice or dependency | Resolve for |
|
||
| --- | --- | --- |
|
||
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||
| U6 | Default strict/permissive mode; configuration spelling, scope, and override precedence; concrete module fallbacks, particularly whole-transcript versus sliced chunking. The two-mode model, module ownership, and final-response-only handling are settled. | Phases 3–4. |
|
||
| U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. |
|
||
| U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. |
|
||
| U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. |
|
||
|
||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||
native schema-source selection, and removal of duplicate schema/filesystem
|
||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||
Implementation remains in Phase 2; U4 selects prompt roles and contracts, and U5
|
||
selects native repair delegation and the application correction allowance.
|
||
|
||
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
|
||
and the shared pipeline producer helper with module functions and a pipeline
|
||
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
|
||
Implement in Phase 3 and refine signatures during Phase 4 without reopening
|
||
ownership. U4/U5 select prompt/guidance and retry behavior; U6 selects terminal
|
||
handling, with its remaining configuration and module choices listed above.
|
||
|
||
**U4 resolved:** Domain-specific initial/correction prompt pairs, shared response
|
||
schemas, default profile behavior, validator-owned guidance, and complete
|
||
applicable-chain assessment are selected in
|
||
[the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved).
|
||
Implement templates and operation-specific inputs in Phases 3–4; exact prose does
|
||
not reopen ownership. U5 selects budgets; U6 selects strict/permissive handling.
|
||
|
||
**U5 resolved:** `max_additional_turns` defaults to `2`, with binding over
|
||
pipeline precedence and explicit zero disabling correction. Native transient and
|
||
structural-repair defaults remain upstream-owned; bundled prompts and application
|
||
requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-resolved)
|
||
also removes generic non-LLM retries, all producer/validator `retries` fields,
|
||
and the generic outer attempt loop. Any demonstrated transient I/O recovery stays
|
||
with its specific operation. Implement in Phases 3–4; U6 selects terminal handling.
|
||
|
||
**U6 direction resolved:** [Strict/permissive handling and module-owned fallbacks](#terminal-outcomes-and-module-fallbacks-u6-direction-resolved)
|
||
replace failure-category policies. Fallbacks satisfy artifact/downstream invariants
|
||
without asserting ordinary semantic acceptance; use only the final response and
|
||
fixed originals. Fallback and affected downstream work remain outside accepted-
|
||
artifact caching, and completion reports disclose fallback use. Default mode,
|
||
configuration contract, and concrete module fallback choices remain open above.
|
||
|
||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
|
||
existing artifact schemas remain domain contracts. Any actual artifact-format
|
||
change discovered during implementation needs an explicit compatibility decision,
|
||
not an assumption that native CLI adoption authorizes it.
|
||
|
||
## Implementation Sequence
|
||
|
||
All six phases are pending. Each delivers a reviewable outcome, not a separate
|
||
release or compatibility layer. Tests and canonical documentation accompany
|
||
behavior changes; the last phase verifies integration and completeness. Record
|
||
major boundary changes in successor ADRs without rewriting accepted decisions.
|
||
|
||
### Phase 1: Establish The Migration Baseline
|
||
|
||
- Record the selected dependency set and review U1 against the upstream release.
|
||
Keep the cache dependency explicit without blocking independent work.
|
||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||
forward Audita evidence without adopting its application wrappers.
|
||
- Preserve U6–U9 with their dependent phases; do not reopen resolved module
|
||
execution, prompt/guidance ownership, retry controls/defaults, assets/schemas,
|
||
CommandKit, debug, cache, or output ownership.
|
||
|
||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||
and outstanding choices are explicit. Upstream availability alone does not
|
||
establish compatibility of the assembled Notarius dependencies or completion
|
||
of the migration.
|
||
|
||
### Phase 2: Build Native Composition, Sources, And Preflight
|
||
|
||
- Implement the resolved U2 design, update available dependencies, introduce
|
||
configuration version 6, convert native embedded/companion resources, and use
|
||
native schema loading with embedded/default or operator-selected roots. Omit
|
||
bundled structural-repair settings and remove their application override fields.
|
||
- Build the small CommandKit application, shared ConfigKit/engine loaders,
|
||
standard suites, pipeline declarations/native alias, and `internal/preflight`.
|
||
- Add maintained embedded-only and operator-override examples. Remove duplicated
|
||
loading and asset assembly as callers migrate.
|
||
|
||
**Completion evidence:** runtime/inspection/preflight share intended definitions;
|
||
source contexts, shared message paths, prompt-declared output contracts, selected
|
||
schema roots, strict peer conflicts, and replacement work offline. Help,
|
||
version, and completion need no configuration. Preflight checks known dependencies
|
||
without credentials or synthetic requests. `config diff` handles two selections.
|
||
|
||
### Phase 3: Prove A Representative Native Pipeline
|
||
|
||
- Implement the resolved U3–U5 collaboration and retry policy and U6's
|
||
strict/permissive model. Settle its default mode, configuration contract, and
|
||
representative module fallbacks.
|
||
Connect module request/interpretation functions to the pipeline assessment
|
||
callback through the shared producer helper; refine concrete signatures here.
|
||
Implement domain-specific initial/correction prompts using the same schema and
|
||
default profile, supplied overrides, and validator-owned correction guidance.
|
||
Resolve `max_additional_turns` with omission/zero distinctions and pass it to
|
||
native coordination; inherit native transport and structural-repair defaults.
|
||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||
handles and fresh plan application. Select actual observer/group enrollment.
|
||
- Run through the new command surface and alias, using the small completion
|
||
report, native outcomes/streams, and useful debug capture from the outset.
|
||
Begin U8/U9 integration without a provisional local message hash.
|
||
|
||
**Completion evidence:** fake-provider workflows exercise accumulated history,
|
||
collection of independent validator findings without rejection short-circuiting,
|
||
full candidate revalidation through the same assessor, final-turn acceptance
|
||
without duplicate final validation, strict failure, permissive fallback, and useful
|
||
failures. Unavailable assessments remain distinct from actionable defects. The
|
||
executable publishes artifacts and reports them through native presentation
|
||
without the private completion/correction API.
|
||
|
||
### Phase 4: Migrate All Roles And Remove Superseded Machinery
|
||
|
||
- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve
|
||
deterministic paths and ordered validation. Apply U4's prompt/guidance design
|
||
and U5's retry policy across roles; settle remaining U6/U7 choices.
|
||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||
generic outer attempt loop, all producer/validator `retries` fields, global
|
||
scheduler/ceiling, and coupled worker settings.
|
||
- Replace failure-category policy fields with the selected mode configuration
|
||
and module-owned fallback behavior.
|
||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||
encoders, verbosity handling, and obsolete asset registration. Retire the
|
||
emptied `internal/framework/llm` package after moving engine assembly to the
|
||
composition root and producer collaboration to the pipeline.
|
||
|
||
**Completion evidence:** every role uses the shared native engine and selected
|
||
admission policy. No production path depends on the private completion API,
|
||
latest-response-only correction, generic stage retries, global ceiling, or legacy
|
||
CLI infrastructure. Obsolete tests retire with the machinery; surviving domain
|
||
requirements remain.
|
||
|
||
### Phase 5: Complete Cache, Output, And Capture Integration
|
||
|
||
- Resolve U1's supplying release and complete U8. Implement lookup at initial
|
||
preparation, accepted-artifact storage under that hash, explicit cache controls,
|
||
and deliberate old-checkpoint handling. Keep canonical chunk caching separate.
|
||
- Complete the artifact-first publication/report contract and U9 details. Remove
|
||
checkpoint fingerprint graphs, mandatory manifests/receipts, durable execution
|
||
ledgers, and duplicate accounting; do not retain them to populate summaries.
|
||
- Complete capture inventory, per-operation evidence, directory/empty-value
|
||
behavior, `latest`, and recording/finalization failure handling.
|
||
|
||
**Completion evidence:** downstream failure/fix/retry reuses accepted predecessors;
|
||
message changes miss and settings-only changes can hit; corrected answers use the
|
||
initial key; hits make no model calls or fresh validation claims. Fallbacks and
|
||
affected downstream work skip cache lookup/storage; unaffected accepted work
|
||
remains reusable. Artifacts need no metadata bundle. Machine output and debug
|
||
capture obey the selected contracts, including failures after publication and
|
||
quiet-mode capture.
|
||
|
||
### Phase 6: Verify And Document The Coordinated Cutover
|
||
|
||
- Complete the acceptance checks and removal inventory, including dependency
|
||
cleanup, maintained examples, and the assembled command surface.
|
||
- Finish release migration guidance, attribution, successor ADRs, and affected
|
||
canonical documentation. Update this roadmap's status and reconcile the
|
||
separate future-work backlog.
|
||
- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash
|
||
policy, or obsolete metadata requirement remains in the finished migration.
|
||
|
||
**Completion evidence:** PromptKit, ConfigKit, and CommandKit adoption, accepted-
|
||
artifact reuse, artifact delivery, diagnostics, and maintained offline workflows
|
||
are verified together. Remaining work is explicitly separate future scope.
|
||
Release preparation/publication is governed separately by release policy.
|
||
|
||
## Removal Inventory
|
||
|
||
| Retire or replace | Retained application responsibility |
|
||
| --- | --- |
|
||
| `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. |
|
||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||
| Generic `runProducerAttempts` outer loop, repeated validator executions in `executeValidationChain`, all producer/validator `retries` fields, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordinary deterministic execution, native LLM execution/continuation, ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. |
|
||
| Application `structured_output_repair_attempts` fields/resolution and explicit bundled `output.repair_attempts` | Native upstream structural-repair default, with operator customization through native prompt configuration. |
|
||
| Failure-category `validation_policy` fields and separate `fail_run`/`reject_output`/`warn_continue` actions | Strict/permissive selection, module-owned fallback construction, and explicit fallback outcomes. |
|
||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||
| Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. |
|
||
| Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. |
|
||
| Custom dispatch/help/version/parsing, command-specific loading, copied inspection, legacy aliases/flags, parallel encoders and verbosity handling | Native declarations/loaders, pipeline handlers, domain input checks. |
|
||
| Mandatory run manifests, reporting-only metadata bundles, extensive receipts and durable execution ledgers | Domain artifact files, one small native completion report, optional debug evidence. |
|
||
| Summary/trace hierarchy, heuristic debug-content scrubbing, custom transcript/deduplication machinery | Convenient run/operation capture and native secret/credential omissions. |
|
||
|
||
## Acceptance And Verification
|
||
|
||
Use real CommandKit, ConfigKit, and PromptKit in representative assembled flows
|
||
with a fake at the provider boundary. Keep the suite deterministic, offline, and
|
||
independent of real credentials. Test domain interpretation/assessment at their
|
||
package boundaries; use subprocess tests for streams, exit status, alias wiring,
|
||
and representative machine output. Do not duplicate upstream parser/renderer/
|
||
retry tests or freeze complete help text and native debug snapshots.
|
||
|
||
| Area | Required evidence |
|
||
| --- | --- |
|
||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
|
||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
|
||
| Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. |
|
||
| Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. |
|
||
| Non-LLM failure handling | Deterministic errors propagate without generic retries; rejected deterministic candidates reach terminal policy without regeneration; cancellation stops work. Producer/validator `retries` fields and the generic outer attempt loop are retired. |
|
||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; no earlier-candidate recovery or cross-turn proposal union; cancellation stops without fallback work. |
|
||
| Strict/permissive outcomes | Strict failure when no acceptable result is obtained; permissive module fallback with required artifact/downstream invariants, without rerunning the ordinary full validation chain or claiming normal acceptance. Fallback construction failure fails execution; successful fallback use is disclosed in human/machine completion reports. |
|
||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss. Failed/rejected/incomplete work is not stored; fallbacks and affected downstream work skip lookup/storage while unaffected accepted predecessors remain reusable; no required history. |
|
||
| Cache migration | Selected reuse/bypass/refresh and old-record treatment work; invocation configuration no longer blocks lookup; canonical chunk caching retains its distinct policy. |
|
||
| Output/outcomes | Artifact files independent of report format; native text/JSON/YAML summary and qualified outcomes; no mandatory metadata bundle; quiet suppression, verbose stderr, debug path; native nonzero failures and useful explanations. Publication survives later capture/report failure without model reruns. |
|
||
| Execution observations | Initial/continuation/validator totals preserve unknown-versus-zero usage without double counts; result-less failures observable; reuse does not count historical work; native execution success remains distinct from acceptance. |
|
||
| Debug controls/lifecycle | Default/CLI/config/explicit-empty paths and native mode ordering; quiet capture; allocation before engine/preflight; start-time relative `latest`, no completion retarget; disabled runs leave it untouched; capture failures preserve primary causes and publication facts. |
|
||
| Debug evidence | Available input/source/native snapshots, readable text, all visible turns, assessments, normalization and deterministic outputs, cache hits, and failed-run evidence; no invented provider transcript; ConfigKit secrets/native credentials omitted, no heuristic content scrubbing. |
|
||
| End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit phase. |
|
||
|
||
Run focused checks during implementation and the repository-wide checks for the
|
||
completed cutover under the [testing policy](../policy/testing.md):
|
||
|
||
```sh
|
||
go test ./...
|
||
go vet ./...
|
||
go build ./cmd/notarius
|
||
```
|
||
|
||
Check maintained examples, dependencies, attribution, documentation links, and
|
||
removal coverage. Live provider smoke and human model-quality evaluation remain
|
||
optional, outside the default suite. These migration checks add no permanent
|
||
release gates. Documentation completion does not claim that implementation
|
||
acceptance checks have passed.
|
||
|
||
## Migration Guidance And Documentation
|
||
|
||
Provide practical release guidance for operators and subprocess callers:
|
||
|
||
- New commands/flags, native `run` alias, `--format`, stdout/stderr, outcomes, and
|
||
saving optional reports by redirection. No legacy CLI compatibility promise.
|
||
- Version 6 and native resources, embedded-only versus companion examples in
|
||
appropriate directories, imports, qualified IDs, replacement/inheritance,
|
||
native local backends, removed catalogs/versions, and separate source versions.
|
||
- Native admission and worker defaults, removed global limit, `max_additional_turns`
|
||
default/precedence/zero behavior, and intent-based migration from LLM `retries`.
|
||
Explain retirement of all producer/validator `retries` fields and generic non-LLM
|
||
retries, upstream-owned structural defaults, removal of application repair
|
||
controls and bundled counts, native operator customization, accumulated-history
|
||
costs, strict/permissive selection and precedence, and each module's fallback
|
||
behavior and downstream guarantees.
|
||
- Accepted-artifact caching, message-based misses, settings-only reuse, refresh/
|
||
bypass for changed interpretation or validation, fallback/affected-work exclusion,
|
||
and old-checkpoint handling.
|
||
- Artifact delivery without mandatory manifests/receipts; preserve intrinsic
|
||
artifact schemas and document any explicitly selected format change.
|
||
- Debug root/default/empty behavior, `latest`, native mode ordering, failure
|
||
behavior, and ordinary operator responsibility for application data.
|
||
|
||
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
||
[operations](../operations.md), [internal overview](../internal/overview.md),
|
||
[LLM](../internal/llm.md), [pipeline](../internal/pipeline.md),
|
||
[modules](../internal/modules.md), [state](../internal/state.md), and affected
|
||
integration/consumer guides as behavior lands. Remove superseded manifest/receipt
|
||
contracts then; keep current-behavior documentation accurate until implementation.
|
||
Update [architecture policy](../policy/architecture.md) for native dependencies
|
||
and changed scheduler, retry, asset, cache, and reporting boundaries.
|
||
|
||
Record significant changes in successor ADRs, including
|
||
[asset assembly](../adr/0011-centralize-llm-assets.md),
|
||
[validation retries](../adr/0014-feedback-aware-validation-retries.md),
|
||
[module retries](../adr/0016-feedback-aware-module-requested-retries.md),
|
||
[PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md),
|
||
admission, caching, and CLI/reporting ownership. Preserve accepted decision text
|
||
and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||
This roadmap owns implementation status; Git history retains earlier discussions.
|
||
|
||
## Evidence And Reading
|
||
|
||
These sources support the selected design, not a claim of completed Notarius
|
||
implementation. Sibling links assume the local repositories are available; the
|
||
tagged PromptKit links below preserve the reviewed baseline.
|
||
|
||
### Native Library Guidance
|
||
|
||
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md) and
|
||
[CLI conventions](../../../commandkit/docs/cli.md): module selection, native
|
||
aliases/inputs, shared loaders, reporting modes, outcomes, and delivery failures.
|
||
- [ckconfig](../../../commandkit/integrations/ckconfig/README.md) and
|
||
[ckprompt](../../../commandkit/integrations/ckprompt/README.md): standard suites
|
||
supplied with application loaders and request builders.
|
||
- [ConfigKit](../../../ConfigKit/README.md): native loading and inspection.
|
||
- [PromptKit consumer guide][consumer] and
|
||
[capacity behavior](../../../promptkit/docs/internal/capacity.md): native
|
||
preparation/execution, continuation, observations, and admission.
|
||
|
||
### Audita Comparison
|
||
|
||
The comparison was completed on 2026-11-18 against Audita revision
|
||
`da5b8913d1206a8e8f0a784b2322760385548a60`, using PromptKit v0.23.2,
|
||
ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
|
||
The observations below concern that revision; linked sibling files may evolve.
|
||
|
||
| Evidence | Lesson carried into this roadmap |
|
||
| --- | --- |
|
||
| [Engine assembly](../../../audita/internal/framework/llm/runtime.go), [source tests](../../../audita/internal/framework/llm/engine_sources_test.go), and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go) | One captured engine, native underlays and main-source-relative companions, whole-record replacement, preserved file contexts, strict peer conflicts, and no silent fallback on operator-source errors. |
|
||
| [Adapter tests](../../../audita/internal/framework/llm/adapter_test.go) | Ordinary embedded schemas and complete per-request output contracts work with operator prompt replacement. U2 chooses prompt-declared defaults for Notarius; Audita's schema registry and per-call overrides are not requirements to copy. |
|
||
| [Native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), [repair flow](../../../audita/internal/framework/runner/repair.go), [conversation tests](../../../audita/internal/framework/llm/conversation_test.go), and [repair tests](../../../audita/internal/framework/runner/repair_test.go) | Captured exchanges, accumulated history, complete candidate revalidation, final-turn assessment, and canceled/undelivered response handling. |
|
||
| [Prompt guidance](../../../audita/docs/internal/prompts.md) | Durable system instructions, an initial user task, and subsequent repair assignments suit native continuation. |
|
||
| [Observations](../../../audita/internal/framework/llm/observations.go) and [CLI composition](../../../audita/internal/cli/app_wiring.go) | Native execution totals with application scope and explicit nested-validator enrollment; standard CommandKit suites using shared loaders. |
|
||
|
||
Audita repairs individual transcript corrections in independent branches;
|
||
Notarius's selected unit is a whole producer operation and typed artifact.
|
||
Audita's wrappers, response-schema registry, per-proposal policies, and absence
|
||
of a cache do not define Notarius requirements.
|
||
|
||
At the reviewed revision, focused LLM, prompt/schema, runner, and proposal-generation
|
||
tests passed, together with subprocess checks for correction conversations,
|
||
cancellation during repair, and repair-disabled/output selections using a local
|
||
fake provider. These historical checks do not establish live-model quality or
|
||
completion of Notarius's migration acceptance checks.
|
||
|
||
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md
|
||
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md
|
||
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/formats.md
|
||
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/pkg-promptkit.md
|