1208 lines
80 KiB
Markdown
1208 lines
80 KiB
Markdown
# Native PromptKit And CommandKit Migration
|
||
|
||
## Purpose And Status
|
||
|
||
Make Notarius fully native to PromptKit, ConfigKit, and CommandKit. Use upstream
|
||
configuration, execution, continuation, admission, inspection, commands, and
|
||
presentation directly. Retain the application work of producing, assessing,
|
||
publishing, and reusing typed artifacts. Historical duplication of library
|
||
behavior is not a requirement to preserve it.
|
||
|
||
This is the single migration plan for PromptKit, ConfigKit, CommandKit,
|
||
accepted-artifact caching, and artifact-first output. It owns the selected target,
|
||
remaining choices, implementation sequence, removal inventory, acceptance
|
||
criteria, and supporting evidence. The earlier discussion documents are retired;
|
||
Git history preserves the discussion. All implementation phases are pending.
|
||
Recording these decisions has not changed application behavior. Unrelated future
|
||
work remains in the [separate backlog](future.md).
|
||
|
||
Notarius predates many modern PromptKit capabilities: the library grew out of
|
||
mechanics shared by Notarius and related applications. Historical duplication
|
||
and slight behavioral differences do not express an intentional application
|
||
requirement. Evaluate remaining machinery against actual domain needs, and
|
||
consider upstream features when a reusable capability is missing.
|
||
|
||
Complete one coordinated cutover for configuration, PromptKit resources, and the
|
||
CLI. Adopt native public conventions fully; preserving old CLI behavior is a
|
||
non-goal. Supply migration guidance and maintained examples instead of aliases,
|
||
parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is
|
||
part of the main migration, not a later follow-up. Release preparation and
|
||
publication remain separate work.
|
||
|
||
Adopting the kits together lets standard inspection and execution share the same
|
||
loaders and native facts. It avoids building temporary command and reporting
|
||
adapters while those boundaries are already being replaced.
|
||
|
||
## Baseline And Dependency Target
|
||
|
||
At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application
|
||
uses configuration version 5 and Go 1.27. Its dependencies include old-path
|
||
PromptKit v0.12.0 and ConfigKit v0.5.0, new-path ConfigKit v1.2.0, go-yaml
|
||
v4.0.0-rc.6, and two external PromptKit provider catalogs.
|
||
|
||
The selected reviewed baseline is:
|
||
|
||
| Module | Version |
|
||
| --- | --- |
|
||
| `go.maximumdirect.net/commandkit` | `v0.7.0` |
|
||
| `go.maximumdirect.net/ckconfig` | `v0.9.0` |
|
||
| `go.maximumdirect.net/ckprompt` | `v0.10.0` |
|
||
| `go.maximumdirect.net/configkit` | `v1.2.1` |
|
||
| `go.maximumdirect.net/promptkit` | `v0.23.3`, subject to U1's hash-capable release review |
|
||
|
||
Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible
|
||
validation dependencies. The selected CommandKit integration releases document
|
||
compatibility with PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout
|
||
attribution and cancellation/error identity fixes; that patch requires no
|
||
configuration or request API migration from v0.23.2 and retains its dependencies.
|
||
The original review used local release tags and tagged dependency manifests;
|
||
remote release lookup was unavailable, so these are selected versions, not a
|
||
claim that they are the latest releases. See the
|
||
[integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
|
||
and tagged [release notes][release], [migration guide][migration],
|
||
[format reference][formats], and [Go consumer guide][consumer].
|
||
|
||
The cache requires native `MessageContentHash`; v0.23.3's `RenderedPromptHash`
|
||
is not a substitute. The sibling PromptKit repository has retired the original
|
||
feature request, and its [v0.24.0 release notes](../../../promptkit/docs/releases/v0.24.0.md)
|
||
document the addition and a ConfigKit v1.3.0 dependency. U1 remains open to review
|
||
the supplying API/release and verify a compatible assembled dependency set before
|
||
changing the selected baseline. Independent work can proceed; completed cache
|
||
integration must use the upstream capability. Do not add a local hash-policy fork.
|
||
Neither a cross-backend global limit nor a comprehensive execution fingerprint
|
||
is required. Remove obsolete imports and catalog dependencies when unused;
|
||
avoid permanent `replace` directives and update [attribution](../oss/attribution.md)
|
||
under the [OSS policy](../policy/oss.md).
|
||
|
||
Audita's reviewed v0.23.2 integration demonstrates native source assembly,
|
||
continuation, observations, and CommandKit wiring. Adopt those mechanisms
|
||
without copying its `RoleClient`, structured-output interface, or opaque exchange
|
||
wrapper. Its per-proposal repair branches and lack of checkpoints do not define
|
||
Notarius's artifact and recovery policies.
|
||
|
||
## Target Architecture
|
||
|
||
### Application Structure And Ownership
|
||
|
||
| Part | Responsibility |
|
||
| --- | --- |
|
||
| Application composition | Load configuration and native sources; construct one shared PromptKit engine for producers, validators, and continuation runs. |
|
||
| Artifact-family modules | Prepare domain inputs, select prompt IDs and response requirements, interpret responses, and apply deterministic domain rules. |
|
||
| Pipeline execution | Connect generation to configured validation and terminal policy; manage ordered dependencies, typed handoffs, bounded workers, and stable result ordering. |
|
||
| PromptKit | Prepare and execute requests; maintain captured exchanges and conversations; enforce native retry, repair, turn, and admission budgets; report execution facts. |
|
||
| Application state and records | Publish artifacts, cache final accepted artifacts, provide a small completion summary, and capture optional debug evidence with pipeline context. |
|
||
| ConfigKit and CommandKit | ConfigKit owns configuration mechanics; CommandKit and ckconfig/ckprompt own standardized commands and presentation. Notarius supplies loaders, request builders, report content, and outcome mapping. |
|
||
|
||
Use native PromptKit requests, results, exchanges, errors, and observers in
|
||
model-executing application code. Remove the private provider-neutral completion
|
||
API. Pure domain models, codecs, source-reference resolution, and deterministic
|
||
transformations need no PromptKit dependency. Application interfaces continue to
|
||
express artifact production, typed validation, output encoding, and state access.
|
||
|
||
#### Module Execution And Assessment (U3 Resolved)
|
||
|
||
Keep the existing typed stage boundaries and use native PromptKit values in
|
||
model-executing code. Place engine construction and shared loaders in focused
|
||
files in the `internal/cli` composition root. It constructs one native engine
|
||
for the run. Retire `internal/framework/llm` once its adapter, scheduler, asset
|
||
registry, and reporting responsibilities disappear; do not retain a package
|
||
solely to hold a small constructor or introduce another LLM service interface.
|
||
|
||
Place one small shared producer helper alongside assessment in
|
||
`internal/framework/pipeline`. Connect modules to it through ordinary functions:
|
||
|
||
1. The pipeline supplies an operation-scoped assessment callback capturing the
|
||
configured validator chain and acceptance/terminal policy.
|
||
2. The module prepares source material, references, and any normalization handle
|
||
mappings once. It builds the native initial request and supplies response
|
||
interpretation and correction-request functions.
|
||
3. The helper receives those functions, the shared engine, and relevant
|
||
cache/debug collaborators. For cacheable work it prepares the initial request
|
||
and looks up the accepted-artifact cache; on a miss it executes the retained
|
||
request. Otherwise it uses ordinary native execution.
|
||
4. For each executed response, the helper invokes module interpretation and
|
||
pipeline assessment. When correction is appropriate, it supplies the module's
|
||
next request to native coordination. PromptKit owns conversation state and
|
||
turn iteration; the helper does not implement another retry engine.
|
||
5. Return a typed application outcome. Cache only final accepted artifacts under
|
||
the initial request's key. A cache hit follows the existing direct-reuse policy
|
||
without model execution or renewed semantic assessment.
|
||
|
||
Request builders return `promptkit.RunRequest`; interpreters consume native
|
||
results. Candidates, findings, and acceptance outcomes remain application-owned.
|
||
Keep operation-specific state in local variables or closures, without an
|
||
execution-plan language, replacement completion API, or copied conversation model.
|
||
|
||
Use one pipeline assessment implementation for the initial candidate and every
|
||
correction, including the final permitted response. Do not repeat final validation
|
||
merely because the helper returns to the outer runner. Deterministic stages keep
|
||
their ordinary typed production/validation path without synthetic requests or
|
||
conversation machinery. LLM validators call the shared engine directly and return
|
||
domain findings; they do not automatically acquire a producer correction loop.
|
||
|
||
Modules receive operation data and collaborators, not physical storage roots or
|
||
CommandKit reporters. Pure domain packages retain artifact models, codecs,
|
||
reference resolution, and deterministic rules. `internal/preflight` retains its
|
||
separate known-dependency checks. Exact argument grouping and helper signatures
|
||
can be refined against representative extraction and normalization during
|
||
implementation; ownership and placement are settled. U4 selects prompt roles and
|
||
guidance ownership below; U5 selects retry controls and the correction budget.
|
||
Remaining failure and fallback policies stay in U5/U6. Implementation is pending.
|
||
|
||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||
and deterministic public ordering. Keep modules independent of physical output,
|
||
cache, and debug roots. Native adoption does not turn the pipeline into a general
|
||
workflow engine or redesign durable D&D artifacts.
|
||
|
||
### Command Surface And Native Inputs
|
||
|
||
Install the application `pipeline` family alongside ckconfig and ckprompt:
|
||
|
||
| Command | Responsibility |
|
||
| --- | --- |
|
||
| `pipeline list` | Discover configured pipelines. |
|
||
| `pipeline show <pipeline-id>` | Report resolved steps, lanes, modules, validators, and reference relationships. |
|
||
| `pipeline validate <pipeline-id>` | Check composition and known prompt/profile dependencies offline. |
|
||
| `pipeline run <pipeline-id> --input-file PATH` | Execute the pipeline and publish artifacts. |
|
||
| `run <pipeline-id> --input-file PATH` | Native alias for `pipeline run`, with the same handler and inputs. |
|
||
|
||
Use `commandkit.Alias("run", "pipeline run")`, not a forwarding command. Use
|
||
`notarius run` in quick starts and document both routes. Install the full selected
|
||
standard suite: native help/version; Bash, Zsh, and Fish completion; configuration
|
||
scopes, validation, show, sources, explain, and diff; PromptKit resource listings
|
||
and inspection for prompts, profiles, backends, namespaces, and collections;
|
||
requirements discovery/validation and prepared-request preview; and explicit
|
||
reachability, authentication, text, structured-output, and aggregate smoke tests.
|
||
Include upstream convenience aliases and useful completion hints.
|
||
|
||
Use native typed declarations and supplied-value semantics. Defaults apply to
|
||
omission, not explicit empty/false/zero values. Preserve native command-path,
|
||
operand, repeated-flag, and `--` parsing conventions. Check every occurrence of
|
||
a singleton flag and use the last successfully parsed value.
|
||
|
||
| Concern | Selected input |
|
||
| --- | --- |
|
||
| Configuration | Native `--config`, `--overlay`, and supported application `--set POINTER=JSON`; scope named `app`. |
|
||
| Pipeline input | Required pipeline-ID operand and `--input-file PATH`. |
|
||
| Lanes | Repeatable `--lane ID`; omission selects the normal lane set. Commas are literal values, not separators. |
|
||
| Output | Application `--output-dir PATH`. |
|
||
| Chunk-plan cache | `--chunk-cache auto\|bypass\|refresh`. |
|
||
| Accepted-artifact cache | Reuse/bypass/refresh controls; spelling, scope, and defaults remain U8. |
|
||
| References | Repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`. |
|
||
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
|
||
| Capture destination | Application `--debug-dir PATH`, with the semantics below. |
|
||
|
||
Keep worker counts and semantic budgets in configuration, adjustable through
|
||
application `--set`. U5 selects `max_additional_turns`; worker defaults remain U7.
|
||
Use ckprompt's reusable profile, session, and execution flags, including model,
|
||
temperature, timeout, reasoning effort, and transient retries, for run-wide
|
||
overrides across producers, validators, and continuations. Omission preserves
|
||
normal selection; explicitly
|
||
empty reasoning effort clears inheritance. Use native `--llm-profile` and
|
||
`--llm-session-id`, retiring the separate reasoning-clearing control. Native
|
||
transient retries remain distinct from semantic additional-turn budgets.
|
||
Individual prompt selection and named prompt inputs belong on standard PromptKit
|
||
commands; pipeline modules build their own requests.
|
||
|
||
Notarius retains reference-selector expansion, specificity, required slots, and
|
||
external/generated-reference conflict checks. Pipeline reference selectors are
|
||
not individual-request `--llm-input-file` bindings. Remove custom dispatch,
|
||
assembled help, standard flag parsing, comma-separated `--only`, and bespoke
|
||
`--json` switches. Replace `config effective` with `config show` and `config
|
||
profile` with `promptkit show profile`. Old flags are migration evidence only.
|
||
|
||
### Application Composition, Loaders, And Preflight
|
||
|
||
One constructor registers application commands, the native alias, ckconfig's
|
||
`app` and `promptkit` scopes, ckprompt, and native help/version/completion/reporting.
|
||
It does not load configuration, build an engine, or initialize run storage.
|
||
Help, version, and completion work without application configuration.
|
||
|
||
Share two loading responsibilities across application and standard commands:
|
||
|
||
1. Select application configuration through explicit `--config`, then existing
|
||
environment/discovery policy. Let ConfigKit apply overlays, imports,
|
||
inheritance, overrides, defaults, normalization, and structural validation.
|
||
Retain its native result and selected main-source context. Validate an
|
||
explicitly empty configuration path rather than falling through to discovery.
|
||
2. Build the native engine from that application's source context using the source
|
||
assembly below. Across commands, `--config` selects the application document;
|
||
the PromptKit scope inspects the engine assembled for it. Request overrides
|
||
do not mutate captured engine configuration. Do not emulate unsupported
|
||
arbitrary `--set` overrides for the PromptKit scope.
|
||
|
||
Use small invocation-owned state through CommandKit's facilities to reuse loaded
|
||
values. A pipeline shares one engine across all roles. `config diff` may load two
|
||
distinct configurations within one invocation: keep their selections separate,
|
||
including overlays/overrides. No persistent loader cache or configuration hash
|
||
framework is needed.
|
||
|
||
Reuse the application resolver for module/options checks, typed compatibility,
|
||
ordered dependencies, validator chains, and reference bindings. `pipeline list`
|
||
only enumerates configuration; `show` renders resolved composition rather than
|
||
building an engine to expand every profile. ConfigKit supplies loaded values and
|
||
native provenance/secret-aware inspection, not domain pipeline semantics.
|
||
|
||
Centralize known prompt/profile dependency checks in `internal/preflight`. It
|
||
accepts the resolved pipeline, module declarations, and native engine. Check
|
||
every statically known dependency, including LLM validators and correction
|
||
prompts, plus applicable/default profiles. Partial profiles alone are not errors.
|
||
Share identifiers and selection logic with execution; no independent inventory,
|
||
asset registry, or synthetic request execution is needed. Dynamic selections
|
||
receive their final checks at runtime. Findings identify pipeline, step/lane,
|
||
stage/validator, and resource where applicable.
|
||
|
||
`pipeline validate` and `run` call the same resolution/preflight functions;
|
||
handlers do not invoke other commands or parse their reports. Keep `config
|
||
validate` focused on configuration and remove its pipeline-specific mode.
|
||
Offline preflight reads configuration and prompt resources but needs no transcript,
|
||
reference contents, credential values, or provider calls. Concrete PromptKit
|
||
preparation/validation owns completed-request checks, including schemas and
|
||
credentials; provider smoke tests require explicit invocation.
|
||
|
||
Preflight does not discover configuration, register modules, build engines,
|
||
initialize storage, or render CLI reports. Modules do not import it. Use native
|
||
request builders for standard commands and add application callbacks only where
|
||
needed, preserving supplied operator values. Output/cache/debug allocation
|
||
belongs to the run path, with debug timing selected below.
|
||
|
||
### Configuration, Sources, And Prompt Assets
|
||
|
||
Introduce application configuration version 6 as the only supported version.
|
||
Preserve unrelated discovery, pipeline structure, and resolution behavior.
|
||
ConfigKit continues to own loading and composition under
|
||
[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
|
||
Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of
|
||
native companion sources; do not introduce a separate `promptkit.config_file`.
|
||
Retire `concurrency.total_llm` and revise retry/worker settings as described below.
|
||
U5 selects `max_additional_turns` and removes application structural-repair
|
||
controls; worker defaults and retired global-limit migration remain U7.
|
||
|
||
Use one source assembly for runtime, preflight, and inspection:
|
||
|
||
1. PromptKit's bundled OpenRouter underlay.
|
||
2. Notarius's embedded native prompts and profiles under `collections.notarius`.
|
||
3. An optional, higher-priority operator underlay containing `prompts.yml` and
|
||
recursive `.yml`/`.yaml` files under `prompts/`, beside the main application
|
||
configuration actually selected by ConfigKit.
|
||
|
||
Anchor companion discovery to that selected main source, whether explicit or
|
||
discovered. Configurations in the same directory share companions; there is no
|
||
working-directory fallback. The two companion locations are strict peers, so
|
||
filenames do not resolve conflicting operator records. Missing companions are
|
||
fine; loading errors do not silently fall back to embedded defaults. Preserve
|
||
native imports and source-relative resource paths.
|
||
|
||
Native declarations become the sole prompt assembly mechanism. Keep the embedded
|
||
content tree and module ownership of prompt meaning, response requirements, and
|
||
domain interpretation. Remove prompt/fallback-profile registration in
|
||
`AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`,
|
||
shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration
|
||
for checkpoint fingerprints. The resolved schema design below also removes
|
||
schema registration and filesystem-flattening machinery.
|
||
|
||
Convert maintained resources, examples, and fixtures to native identified lists,
|
||
`config:` source controls, and sibling inheritance (`extends: {id: parent}`).
|
||
Same-ID replacement supplies a complete record; distinct inherited variants can
|
||
customize an existing resource. Explain native profile-null/backend fallback
|
||
semantics. The optional PromptKit source header is version 2, independently of
|
||
Notarius's application version. Do not translate legacy resource formats or
|
||
synthesize compatibility aliases.
|
||
|
||
Select prompts by ID alone; remove obsolete PromptKit version selectors and
|
||
related request/debug metadata. Retain current local resource IDs and use exact
|
||
qualified references, such as `notarius/dnd.spells` and
|
||
`notarius/dnd-extraction`. Keep the extraction profile's existing model/settings
|
||
and its reference to the bundled unqualified `openrouter` backend. Inherited
|
||
application profiles use sibling local IDs in the same collection. The bundled
|
||
inventory differs from the old external catalogs and selects no model for an
|
||
application request; document intentional replacements for removed catalog IDs.
|
||
The bundled backend supplies no concurrency limit. Application/operator limits
|
||
need explicit native configuration; application-supplied defaults remain U7.
|
||
|
||
Preserve prompt prose, message ordering, roles, and cache boundaries where their
|
||
meaning carries forward. Review and deliberately revise initial/repair prompts
|
||
for accumulated conversation history; preserving every old byte or retry message
|
||
is not a migration requirement. Shared content should use native filesystem
|
||
contexts and imports rather than reconstructed virtual trees.
|
||
|
||
#### Prompt Assets And Response Schemas (U2 Resolved)
|
||
|
||
Use the existing root `assets` package's ordinary `embed.FS` directly for native
|
||
prompt sources and response schemas, with `fs.Sub` only where useful. Shared
|
||
message files use their actual paths within the native filesystem context.
|
||
Remove synthetic mounts, file copying, flattened schema namespaces, and module
|
||
asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening
|
||
consumers disappear; no replacement filesystem abstraction is needed.
|
||
|
||
Declare response format, validation mode, and schema path in native prompt
|
||
definitions by default. Omit structural-repair attempt settings in bundled
|
||
prompts to inherit PromptKit's upstream default under U5. Modules select prompt
|
||
IDs, supply inputs, and interpret responses. U4 selects the same schema file for
|
||
each initial/correction pair. Execution, inspection, and prepared-request preview
|
||
then use the same declared contract without repeating schema selection in Go.
|
||
U4 selects prompt roles and contracts; exact prose is implementation work.
|
||
U5 selects budget ownership and defaults.
|
||
|
||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||
dependent contract. It replaces the whole configured contract, so supply the
|
||
intended format, validation mode, and schema together while leaving structural
|
||
repair at its native default. Do not inject a repair count merely to construct
|
||
the override, or introduce a wrapper or per-call registry solely to reproduce
|
||
Audita's approach. Operator prompt replacements must still produce responses the
|
||
selected module can interpret; that module boundary does not require duplicating
|
||
each prompt's schema selection in Go.
|
||
|
||
Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to
|
||
that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`.
|
||
They are separate from message-file source contexts: prompt underlays do not
|
||
implicitly overlay schema files. When an operator configures an external schema
|
||
directory, select PromptKit's native directory source instead of the embedded
|
||
root. It replaces the whole schema source and must supply the schemas used by
|
||
the selected prompts. Do not build an application filesystem overlay to mix
|
||
embedded and external schemas. Transparent layering would be a separate upstream
|
||
capability, not a requirement for this migration. Concrete configuration binding
|
||
is an implementation detail within this selected source policy.
|
||
|
||
Keep private response schemas as self-contained JSON documents. PromptKit owns
|
||
loading, compilation, structural validation, provider structured-output settings,
|
||
and structural repair. Remove Notarius's response-schema byte registry, copied
|
||
schema wrappers, defensive-copy layer, digest calculation, and duplicate
|
||
structural-validation machinery. Native prepared snapshots provide diagnostic
|
||
schema details; no schema fingerprint is added to the message-content cache key.
|
||
|
||
Keep typed decoding, source/reference checks, reconciliation safety, semantic
|
||
validators, artifact codecs, and durable artifact schemas in Notarius. Response
|
||
schemas describe model output; artifact schemas describe published and consumed
|
||
domain artifacts. Preserve identifiers and versions belonging to actual artifact
|
||
contracts. Retire private schema keys, registration records, hashes, and copied
|
||
identity/version/name metadata whose only consumers were adapters, checkpoints,
|
||
or reporting. No independent schema metadata system replaces them.
|
||
|
||
Implementation is pending. Verify native preparation of embedded and operator
|
||
prompt definitions, shared-file paths, schema selection from embedded and external
|
||
roots, and complete request-contract replacement where actually used. Missing or
|
||
invalid schemas should follow native preparation behavior. No new schema check
|
||
requiring synthetic requests is added to offline pipeline preflight.
|
||
|
||
### Native Production, Continuation, And Assessment
|
||
|
||
Start with one conversation per producer operation: a chunk-plan generation,
|
||
one lane's extraction of one chunk, or one lane's normalization pass. Use
|
||
complete replacement model responses. Normalization may continue to return a
|
||
reconciliation plan that deterministic code applies to produce the artifact.
|
||
Per-entity branching is not part of this migration.
|
||
|
||
#### Prompt Roles And Correction Guidance (U4 Resolved)
|
||
|
||
Give each correctable producer an initial native prompt and a domain-specific
|
||
correction prompt. Retain existing initial IDs and identify correction IDs
|
||
explicitly in the module, for example `notarius/dnd.spells.repair`. Preflight uses
|
||
those same declarations to check known prompt dependencies. Shared correction
|
||
wording can live in ordinary asset files; distinct domain/operation templates
|
||
supply the task and response expectations without a universal application-aware
|
||
correction protocol.
|
||
|
||
The initial request supplies source material, references, and the task. Enduring
|
||
domain principles belong in its system instructions. The correction template
|
||
supplies a new user assignment with the current candidate's actionable findings;
|
||
PromptKit carries the preceding exchange. Do not rerender the original source or
|
||
manually replay prior responses. Review initial wording so later correction
|
||
assignments remain valid. Include additional application-derived context
|
||
only when needed to interpret the findings. Keep source/reference meanings and
|
||
normalization candidate handles fixed throughout the conversation.
|
||
|
||
Use the same declared response schema for each initial/correction pair:
|
||
|
||
| Role | Response contract |
|
||
| --- | --- |
|
||
| LLM chunk planning | Complete replacement chunk plan. |
|
||
| Extraction | Complete replacement extracted artifact candidate. |
|
||
| Normalization | Complete replacement reconciliation plan, applied afresh to the fixed original candidates. |
|
||
| LLM validation | The validator's assessment response; no separate semantic correction conversation by default. |
|
||
|
||
No correction envelope, patch language, or special repair-result type is needed.
|
||
PromptKit retains structural validation and repair inside each native execution.
|
||
Notarius assesses the response returned by that public boundary, independently
|
||
of how many internal attempts produced it.
|
||
|
||
Use the same native default profile for an initial/correction pair. Preserve
|
||
explicit application profile selections and run-wide execution overrides across
|
||
both requests. An operator may select a different default profile in a correction
|
||
prompt, subject to explicit request overrides. Validators retain their own
|
||
profile selection. Pass native selections and overrides without copying resolved
|
||
settings or adding a Notarius-wide repair-model configuration layer.
|
||
|
||
The component that detects a defect owns its actionable correction guidance:
|
||
|
||
- Each rejecting validator supplies guidance describing the defect and the
|
||
constraints a corrected answer must satisfy. A deterministic validator authors
|
||
it alongside the check; an LLM validator requests an actionable explanation in
|
||
its prompt and returns it through the application validation contract.
|
||
- Preserve the explicit `CorrectionGuidance` boundary separately from operator
|
||
diagnostics. Guidance need not prescribe an exact replacement, and validators
|
||
do not construct the complete correction prompt.
|
||
- Module interpretation and normalization safety checks supply their own guidance
|
||
when defects arise before an artifact candidate can be materialized.
|
||
- The pipeline collects guidance in configured validator order. The module's
|
||
domain-specific template renders it into the correction assignment. Neither
|
||
the pipeline helper nor the template duplicates validators' defect-detection
|
||
logic. Reason codes and operator messages are not automatically instructions.
|
||
|
||
Assess the same candidate through every applicable configured validator before
|
||
requesting correction. A rejection does not short-circuit later validators;
|
||
collect all actionable findings they return. Reassess every revised candidate
|
||
through the complete applicable chain, including earlier approvals and the final
|
||
permitted response. The existing validator-owned guidance and ordered-chain
|
||
semantics remain useful; replace the hard-coded `CorrectionRequest` message
|
||
builder with native template rendering and retain useful ordered aggregation.
|
||
|
||
Applicability requires usable input: if interpretation cannot construct a typed
|
||
candidate, its module supplies guidance and artifact validators cannot yet run.
|
||
Cancellation or fatal execution failures can interrupt assessment. An unavailable
|
||
validator assessment is a separate outcome, not a detected defect or an instruction
|
||
to change the artifact. Do not convert provider errors into correction guidance;
|
||
terminal/fallback disposition remains U6.
|
||
|
||
Ownership, response contracts, and default profile behavior are resolved. Refine
|
||
exact template prose, correction IDs, and necessary operation-specific inputs
|
||
while implementing representative workflows. Implementation is pending; U5 selects
|
||
semantic budgets and U6 retains terminal/fallback policy choices.
|
||
|
||
#### Execution And Assessment Flow
|
||
|
||
The application flow is:
|
||
|
||
1. Prepare source context, references, and candidate-handle mappings once. Build
|
||
the native initial request. For cacheable work, prepare and check the cache
|
||
before execution as described below; execute the retained request on a miss.
|
||
Otherwise use ordinary `Engine.Run`. Capture an exchange when continuation
|
||
or debug inspection needs one.
|
||
2. Consume the native usable response or terminal failure, decode, and apply
|
||
module interpretation and safety checks. A module can supply actionable
|
||
findings before it has a materialized candidate. Assess each candidate through
|
||
all applicable configured validators and collect their actionable guidance
|
||
before requesting correction, as selected in U4.
|
||
3. For actionable correction, call `Engine.Coordinate` with the captured exchange
|
||
and the resolved `max_additional_turns` as native `AdditionalTurns`. Its callback
|
||
selects a repair prompt and supplies findings; PromptKit retains the initial
|
||
exchange and subsequent turns.
|
||
4. Interpret every delivered revised response and rebuild from the fixed original
|
||
inputs. Run the complete configured validation chain again for changed
|
||
candidates, including validators that previously approved. Assess the final
|
||
permitted response before declaring exhaustion.
|
||
5. Stop on acceptance or a terminal condition. Apply application rejection,
|
||
failure, or supported safe-fallback policy after exhaustion. Native `stopped`
|
||
and `exhausted` outcomes do not themselves mean application acceptance.
|
||
|
||
Accumulated history increases context use; U5's allowance bounds additional
|
||
semantic correction while stopping as soon as the candidate is accepted.
|
||
|
||
Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection`
|
||
transport, manual assistant/user replay, and correction-protocol capability
|
||
checks. Replace `runProducerAttempts`' LLM retry loop with native coordination and
|
||
application assessment. Adapt `NormalizeRetry` and `producerRetryDirective` to
|
||
findings and explicit fallback information. Retain useful ordered collection and
|
||
deduplication of actionable findings, rendered through a native repair prompt.
|
||
Reason codes and operator messages are not automatically model instructions.
|
||
|
||
#### Retry Controls And Correction Budget (U5 Controls Resolved)
|
||
|
||
Delegate transport retries and structural repair to PromptKit. Notarius selects
|
||
only the allowance for application-directed semantic correction:
|
||
|
||
| Concern | Control and selected default |
|
||
| --- | --- |
|
||
| Temporary provider failures | Native PromptKit transient retries; inherit the upstream default when omitted and preserve native execution overrides. |
|
||
| Structurally invalid model output | Native output-contract policy; bundled prompts and application requests inherit PromptKit's upstream structural-repair default. |
|
||
| Actionable domain defects | Application `max_additional_turns`, passed directly to native `AdditionalTurns`; application default `2`. |
|
||
|
||
`max_additional_turns` is a nonnegative maximum, excluding the initial producer
|
||
response. Resolve producer-binding override, then pipeline override, then the
|
||
application default of `2`. Omission inherits; explicit `0` disables semantic
|
||
correction. It applies to LLM-backed producers. Validators do not receive a
|
||
separate semantic continuation budget, and deterministic stages do not acquire
|
||
conversation machinery. Keep the field in configuration, adjustable through the
|
||
existing application `--set` mechanism, without another dedicated CLI flag.
|
||
|
||
For example, a producer binding can override its pipeline allowance:
|
||
|
||
```yaml
|
||
extract:
|
||
module: dnd/spells
|
||
max_additional_turns: 2
|
||
```
|
||
|
||
Two additional turns are an allowance, not mandatory executions. An accepted
|
||
initial candidate needs none; an accepted first correction needs only one. A
|
||
second turn is available when actionable defects remain and continuation is
|
||
appropriate. Assess every delivered response, including the final permitted turn,
|
||
and stop on acceptance or a terminal condition. PromptKit enforces the turn
|
||
limit; do not add an application counter or retry loop. Internal transport
|
||
attempts and structural repairs do not consume this semantic-turn allowance.
|
||
The default is an application policy choice, not a measured claim about how
|
||
frequently corrections will be needed.
|
||
|
||
For structural repair, omit `output.repair_attempts` from all bundled initial,
|
||
correction, and validator definitions. Remove Notarius's pipeline/binding/
|
||
validator `structured_output_repair_attempts` fields and resolution plumbing.
|
||
Do not replace them with a hard-coded default, including zero, in request
|
||
construction or a native output-contract override. Explicit operator
|
||
customization remains available through native PromptKit prompt configuration.
|
||
|
||
Notarius deliberately inherits future upstream default changes. An anticipated
|
||
PromptKit default change is not a migration prerequisite, and its timing or
|
||
chosen count is not a Notarius contract. Notarius still declares response format,
|
||
validation mode, and schema; PromptKit owns the mechanics of obtaining a valid
|
||
response. At the public boundary, handle a usable result or terminal failure,
|
||
including a returned result reporting unsuccessful structural validation.
|
||
Do not restart the operation to extend PromptKit's exhausted repair budget.
|
||
|
||
Remove blanket outer LLM retries, including repeated LLM validator invocations in
|
||
`executeValidationChain`. Keep ordered candidate assessment and distinguish a
|
||
validator execution failure from a domain rejection. Use native `OperationTimeout`
|
||
when that operation scope is needed, without adding an arbitrary default or
|
||
replacing broader application deadlines. Cancellation stops work.
|
||
|
||
Retire the broad producer/validator `retries` settings for LLM execution rather
|
||
than automatically renaming them. Migration guidance maps intent: a producer
|
||
allowance intended for semantic correction becomes `max_additional_turns`; native
|
||
transport controls and prompt-owned structural customization cover their own
|
||
concerns. Do not promise equivalent total provider-attempt counts. Generic retry
|
||
handling for non-LLM operations remains the outstanding U5 choice; detailed
|
||
terminal and fallback dispositions remain U6. Implementation is pending.
|
||
|
||
Keep source grounding, reference resolution, candidate isolation, domain
|
||
invariants, and reconciliation safety checks. Unknown or overlapping handles
|
||
remain invalid. A supported safe fallback may retain safe merges while leaving
|
||
unsafe groups separate; it must pass configured artifact validation and terminal
|
||
policy. After structural-repair exhaustion it can be assessed directly without
|
||
restarting the same model operation. Whether an earlier safe candidate can
|
||
survive a later failed turn remains open; do not silently introduce best-so-far
|
||
acceptance.
|
||
|
||
Preserve rejection versus unavailable/incomplete validation. A structurally
|
||
valid candidate may advance under the existing `warn_continue` policy with
|
||
explicit incomplete-validation outcome information, but it and its affected
|
||
derivatives are ineligible for accepted-artifact cache reuse or publication. A
|
||
completed rejection never advances. Cancellation cannot make an undelivered
|
||
response accepted. Keep conversation state in process, outside ordinary
|
||
artifact/cache payloads; explicitly requested detailed debug is separate.
|
||
|
||
### Admission And Pipeline Workers
|
||
|
||
Remove `Scheduler`, `scheduledClient`, and custom global queue/permit plumbing.
|
||
Retire the separate cross-backend ceiling and the default/maximum extraction
|
||
worker coupling to `concurrency.total_llm`. Keep bounded extraction workers and
|
||
bounded lane continuations with independent defaults to be selected.
|
||
|
||
Use one shared native engine across all roles. Native capacity belongs to the
|
||
engine and backend ID: profiles selecting the same ID share its pool; different
|
||
backend IDs or separate engines do not automatically share limits, even for the
|
||
same endpoint. Endpoint-only profiles and backends without a positive limit
|
||
remain unrestricted by native backend admission.
|
||
|
||
For limited backends, the reviewed baseline bounds active model attempts by
|
||
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||
Ordinary execution admission spans preparation, retries, and repairs; split
|
||
`PrepareExecution` itself reserves no capacity, with admission occurring during
|
||
`RunPrepared`. Active permits are released during retry waits.
|
||
A completed continuation turn releases admission before assessment, allowing a
|
||
nested validator to use the same backend. Configure native limits at the intended
|
||
capacity boundary and handle `ErrCapacityExceeded` through application policy.
|
||
|
||
Aggregate concurrency across independent backends may exceed the former global
|
||
ceiling. That change is intentional. Bounded pipeline workers are application
|
||
work controls, not a substitute cross-backend LLM ceiling. Defaults, retired-field
|
||
migration, and role-specific admission-failure dispositions remain open.
|
||
|
||
### Accepted-Artifact Cache
|
||
|
||
Replace the checkpoint subsystem with a cache around the complete Notarius
|
||
producer operation. Its value is an accepted answer to the original task. This
|
||
avoids repeating expensive model work after a downstream failure without caching
|
||
every cheap deterministic pipeline step.
|
||
|
||
```text
|
||
Prepare initial request -> obtain MessageContentHash -> look up cache/<hash>.json
|
||
Hit: decode the accepted artifact and continue the pipeline
|
||
Miss: execute the retained request and complete the Notarius operation
|
||
-> interpret, validate, and perform application-directed corrections
|
||
-> on final acceptance, store the artifact under the INITIAL hash
|
||
```
|
||
|
||
Use the upstream message-content hash of ordered message roles and exact text.
|
||
Include message boundaries and order, excluding model/settings, session/execution IDs,
|
||
timing, provider cache hints, and other non-content metadata. Add no stage,
|
||
prompt-ID, library-version, whole-configuration, schema-policy, or implementation
|
||
fingerprint. Actual input or schema text rendered into messages participates as
|
||
message content; metadata outside those messages does not.
|
||
|
||
| Change | Intended reuse behavior |
|
||
| --- | --- |
|
||
| Fix a downstream stage | Earlier artifacts remain reusable when their initial messages match. |
|
||
| Change initial message text, roles, order, or boundaries | Cache miss. |
|
||
| Change a prompt ID, model, or settings without changing messages | Cache hit remains possible. |
|
||
| Change deterministic interpretation or validation | Existing accepted artifacts remain reusable; explicit bypass/refresh applies the new policy. |
|
||
|
||
Obtain the key before model execution through native preparation. Preserve the
|
||
prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared`
|
||
where supported, and discard the handle on a hit. `OnPrepared` execution hooks
|
||
alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes
|
||
session identity and cache hints; it is not the requested hash. The upstream
|
||
API/release review in U1 must finish before completing this cache integration.
|
||
|
||
Keep the initial key throughout corrections. Cache the final Notarius artifact
|
||
through its existing codec, not an intermediate response, `RunResult`, exchange,
|
||
or prepared handle. For normalization, save the accepted normalized artifact
|
||
after applying the reconciliation plan, not the proposed plan itself. Save only
|
||
after the entire Notarius operation succeeds, including domain validation and
|
||
any application-directed corrections. Failed, rejected, and validation-incomplete
|
||
work and ineligible derivatives are not cacheable.
|
||
|
||
A hit loads the accepted artifact directly, with ordinary decoding/usability
|
||
checks; absent or unusable entries are misses. It makes no model calls, rebuilds
|
||
no result/exchange, replays no corrections, and performs no new semantic
|
||
validation merely because the artifact was cached. It makes no claim of acceptance
|
||
under changed validation policy and contributes no historical token usage to
|
||
current execution totals.
|
||
|
||
PromptKit owns everything inside each native request, including structural
|
||
validation, transport retries, and repairs. Notarius consumes its public response
|
||
or terminal failure and owns domain acceptance. Internal PromptKit attempts do
|
||
not determine cache eligibility. Native diagnostic evidence does not transfer
|
||
that ownership. No cached-response injection, replay API, exchange restoration,
|
||
or PromptKit-owned write-through cache is requested.
|
||
|
||
Use a small filesystem lookup/write facility and ordinary temporary-file/rename
|
||
publication. `cache/<hash>.json` contains the artifact, with no required manifest,
|
||
acceptance ledger, historical provenance, or versioned checkpoint envelope.
|
||
Lookup does not require original output or debug directories. Remove checkpoint
|
||
identities, dependency fingerprints, invocation-wide barriers, and recovery
|
||
bookkeeping. Preserve the separate canonical chunk-plan cache and its materialized
|
||
chunk validation. Cache directory configuration, reuse/bypass/refresh controls,
|
||
and handling of old checkpoint files remain U8; old `--resume`/`--recompute-step`
|
||
conventions are not compatibility requirements.
|
||
|
||
### Inspection And Execution Bookkeeping
|
||
|
||
Delegate configuration/resource inspection, requirements, concrete preview and
|
||
validation, and explicit smoke diagnostics to ckconfig/ckprompt through the shared
|
||
loaders. Remove effective-YAML parsing, the profile-inspector wrapper, and copied
|
||
effective-parameter types. A configured profile may be partial; use native
|
||
prepared/result details for the actual execution target, including overrides.
|
||
Standard commands retain their native reports, support/applicability distinctions,
|
||
and failure behavior. Smoke status zero alone does not prove a check passed.
|
||
|
||
Use ordinary `Run` with `OnPrepared` when actual request details are needed for
|
||
diagnostics; split preparation has a separate purpose for cache lookup. Native
|
||
observations provide terminal facts even without a returned `RunResult`.
|
||
`OnPrepared` is absent on preparation/admission failures and is not a completion
|
||
record. Keep synchronous callbacks lightweight, compose capture/accounting
|
||
observers, and synchronize shared state where operations overlap. `OnPrepared`
|
||
runs while admission is held; persist retained evidence at normal operation
|
||
boundaries.
|
||
|
||
Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope.
|
||
Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning,
|
||
independent accumulators, debug completion wrappers, synthetic call IDs, and
|
||
adapter profile-recording/forwarding. Native snapshots serve actual-target
|
||
inspection; do not reconstruct a durable profile inventory.
|
||
|
||
Use `ExecutionGroup` for selected diagnostic totals, explicitly enrolling initial,
|
||
continuation, and nested validator executions without overlapping double counts.
|
||
Preserve terminal coverage and unknown-versus-zero usage. Reuse adds no historical
|
||
usage to current totals. These facts support diagnostics and capture, not a
|
||
mandatory consumer execution ledger. Notarius's acceptance, fallback, rejection,
|
||
incomplete-validation, publication, and reuse facts feed the completion summary
|
||
and debug overview without independently maintained reporting state.
|
||
|
||
### Artifact Delivery And Completion Output
|
||
|
||
Artifacts are the primary deliverable: publish accepted domain artifacts as files
|
||
in the output directory. Preserve intrinsic domain content and artifact schemas.
|
||
CLI `--format` controls the completion report, not payload encoding. Consumers
|
||
need no run manifest, metadata bundle, persisted receipt, or execution history to
|
||
use the artifacts. Retire reporting-only metadata and mandatory manifests instead
|
||
of rebuilding them with native values.
|
||
|
||
Submit one small structured value through `Reporter.Report`:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `pipeline` | Selected pipeline identifier. |
|
||
| `artifact_count` | Deliverable artifacts, including reused artifacts; excludes intermediates, cache entries, and debug files. |
|
||
| `output_directory` | Absolute published artifact directory. |
|
||
| `debug_directory` | Optional absolute completed run-specific capture directory; omitted when capture is disabled. |
|
||
|
||
Supply a native text template for the ordinary successful case:
|
||
|
||
```text
|
||
Success: dnd-session produced 6 artifacts in /work/results/run-123
|
||
```
|
||
|
||
When capture is available, append `Debug capture: <run-directory>`. Provide the
|
||
path rather than mirroring the bundle. JSON/YAML render the same structured
|
||
value through native serializers. A plain string would render natively as text
|
||
but lose named fields in machine output; the template gives both presentations
|
||
without parallel encoders. No report-version, run-ID, token, profile, warning-count,
|
||
or validation-history collection is required.
|
||
|
||
When a completed run includes rejected work or permitted incomplete validation,
|
||
qualify the human summary and structured value with the rejection count and/or
|
||
incomplete-validation fact. Keep these facts visible without verbose/debug mode;
|
||
small outcome qualifications do not require per-producer histories. Exact names
|
||
of those conditional fields are implementation details.
|
||
|
||
Wire results to stdout and diagnostics to stderr:
|
||
|
||
| Mode | Stdout | Stderr |
|
||
| --- | --- | --- |
|
||
| Ordinary | Brief completion report. | Failure explanation when needed. |
|
||
| `--verbose` | Same completion report. | Native information, warnings, and failures. |
|
||
| `--quiet` | Nothing. | Failure explanation when needed. |
|
||
| `--debug` | Completion report with capture path when enabled. | Native diagnostics and default progress. |
|
||
|
||
Use native mode ordering (last enabled mode wins), explicit progress selection,
|
||
and quiet suppression. Quiet still permits failure explanations and does not
|
||
prevent rendering errors from failing the invocation. `--format json --verbose`
|
||
keeps stdout valid report JSON. Standard configuration/PromptKit commands retain
|
||
their own native values, with no universal Notarius envelope. Artifact-quality
|
||
advisories retain their domain classification.
|
||
|
||
Optional saved reports use shell redirection, without a report-file flag or
|
||
automatic duplicate persistence:
|
||
|
||
```sh
|
||
notarius run dnd-session --input-file transcript.json --format json > run-report.json
|
||
```
|
||
|
||
Publish artifacts and finish requested capture before submitting a successful
|
||
completion report. Use native outcomes: status 0 for command completion, 1 for
|
||
operational/cancellation/publication/presentation failure, 2 for invalid invocation
|
||
or identifiable invalid configuration, and 3 for native `config diff --check`
|
||
differences. Preserve ckconfig's structured classification and underlying causes;
|
||
not every preflight failure is invalid input. Command completion is distinct from
|
||
domain acceptance: permitted rejections/incomplete validation qualify the report,
|
||
while terminal pipeline failure fails the command.
|
||
|
||
On failure, supply a useful native explanation and nonzero status, without a new
|
||
structured error-envelope protocol. `--format` does not serialize diagnostics.
|
||
Retain CommandKit's separation of operation causes from output/diagnostic delivery
|
||
failures. If publication succeeded before capture/report delivery failed, identify
|
||
the published path and available debug location. Published artifacts remain valid;
|
||
do not rerun model work or rewrite execution history. Subprocess callers check
|
||
status before decoding a successful report; failed writes may leave partial stdout.
|
||
|
||
### Debug Capture And Optional Diagnostics
|
||
|
||
Capture the **most convenient** debug representation, rather than optimizing
|
||
for conciseness, exhaustive completeness, or strict deduplication. Native snapshots
|
||
and readable views may repeat evidence. Reuse application decisions and native
|
||
execution facts instead of maintaining a second ledger or transcript model.
|
||
Debug formats are diagnostic data, not stable consumer contracts.
|
||
|
||
#### Controls And Location
|
||
|
||
Bind an application boolean to native `--debug`. Capture requires that boolean
|
||
and a nonempty effective directory; a directory alone does not enable capture.
|
||
Select `--debug-dir PATH` over `debug.directory` over `/tmp/notarius-debug`.
|
||
Resolve nonempty configuration-relative paths against their source and CLI-relative
|
||
paths against the invocation directory. Recognize explicit empty values first:
|
||
`--debug-dir ""` or `debug.directory: ""` disables capture without fallback.
|
||
A nonempty CLI override can re-enable capture over an empty configured root.
|
||
An unusable nonempty destination fails requested capture rather than selecting
|
||
another directory. Remove the configuration-relative `./notarius-debug` default.
|
||
|
||
Capture selection is independent of the final reporting mode. `--debug --quiet`
|
||
captures while suppressing stdout; `--quiet --debug` selects debug verbosity and
|
||
capture. `--debug=false` disables capture through native binding semantics.
|
||
Empty-directory debug retains native verbosity. No separate capture flag or
|
||
capture-depth switch is needed.
|
||
|
||
#### Lifecycle And Failure Handling
|
||
|
||
Allocate the run bundle after application configuration and run identity are
|
||
available, before engine construction and pipeline preflight. Argument/config-load
|
||
failures may have no bundle. Keep allocation out of shared loaders.
|
||
|
||
Create `<debug-root>/<run-id>/` and update `<debug-root>/latest` to a relative
|
||
symlink to that run at session start. It points to the newly started session even
|
||
if that session fails. Do not retarget at completion; concurrent starts use the
|
||
last successful link update. Runs without capture leave it untouched.
|
||
|
||
One application recorder accepts initial context, native evidence, and application
|
||
decisions, then finalizes once with pipeline/publication/capture outcomes. Persist
|
||
as operations advance, without holding the whole pipeline's content in memory.
|
||
Keep debug separate from output/cache and do not automatically delete bundles.
|
||
|
||
Requested capture is required work: allocation/link failures stop before model
|
||
work; recording failures surface at the next normal application boundary;
|
||
finalization failure fails the command. Observer callbacks retain errors for
|
||
boundary checks because they cannot return errors. An existing pipeline failure
|
||
remains primary, with capture failures reported additionally and causes retained.
|
||
Already-written data and published artifacts remain available. Capture failures
|
||
never cause model retries. Finalize before submitting the success report.
|
||
|
||
#### Inventory And Depth
|
||
|
||
Replace separate summary/trace categories with a compact run overview,
|
||
configuration/composition snapshots, parsed source, and per-operation records.
|
||
Use native execution IDs with application scope for navigation. `latest` is the
|
||
selected link name; other filenames and serialized application fields remain
|
||
implementation choices.
|
||
|
||
| Capture | Selected contents and depth |
|
||
| --- | --- |
|
||
| Run overview | Application version, run/pipeline identity, timing, outcome, publication location, effective invocation selections/overrides, and navigation. Terminal facts come from the same application outcome used by the CLI. |
|
||
| Configuration and composition | Native application and PromptKit effective snapshots, normally once each, plus resolved steps, lanes, modules, validators, and references. |
|
||
| Parsed source | Generic input document with unit identities and content, independent of the original input file. If parsing fails, retain the input location and available diagnostics. |
|
||
| Operation inputs | Actual source/chunk and reference content, incoming artifacts, prompt bindings/variables, and fixed original normalization candidates/handle mappings. Include directly even when duplicated elsewhere. |
|
||
| Prepared requests | Native `PreparedRun`, including settings, messages, and exposed output-contract/schema details when available. |
|
||
| Execution events | Native per-execution events in occurrence order, including attempts/retries/repairs, timing, usage availability, and terminal metadata. |
|
||
| Results and responses | Native `RunResult` plus exact `RawOutput` in a separate editor-readable text file. |
|
||
| Conversations | Full `Exchange.Details()` for every application-visible turn, including rejected turns, plus simple readable role-labeled messages. Repeated prefixes are acceptable. Use prepared messages when no completed exchange exists and label the representation. |
|
||
| Interpretation and assessment | Candidate via its codec before acceptance; decode/safety failures; each validator's findings and correction guidance; application disposition and links to LLM validator executions. |
|
||
| Normalization and corrections | Proposed reconciliation plan and applied candidate; constructed correction inputs even if the next preparation fails. |
|
||
| Deterministic stages and reuse | Chunk plans/materialized chunks, merged/normalized artifacts, and the loaded cached artifact with the current hit decision. No reconstructed history or fabricated execution on reuse. |
|
||
|
||
Use indented JSON for native structured snapshots and existing domain codecs.
|
||
Retain selected content in full, with no arbitrary truncation. Enable exchange
|
||
capture where debug inspection requires it, including single-turn calls;
|
||
continuation's in-process exchange needs remain independent of durable capture.
|
||
Keep operation inputs self-contained enough for direct investigation without
|
||
requiring original inputs, a cache entry, or reconstruction of deltas. Do not
|
||
archive every original file or serialize the pipeline's entire internal state.
|
||
|
||
On failure, retain available evidence, where execution stopped, errors, and
|
||
application disposition. Preparation/admission failure may have no `OnPrepared`;
|
||
operational failure may have no `RunResult`. Capture a coordination `Undelivered`
|
||
response as unassessed. Native internal-attempt metadata is useful but intermediate
|
||
provider response bodies are not required. No provider wrapper or complete
|
||
provider-transcript guarantee is part of this design.
|
||
|
||
#### Data Handling And Diagnostic Commands
|
||
|
||
Operators secure debug data as they secure ordinary application inputs and
|
||
outputs, for example with filesystem permissions. Debug contains internal data;
|
||
no separate security/retention model or sanitized-bundle promise is required.
|
||
Do not directly print/serialize ConfigKit-marked secrets, and preserve native
|
||
credential omissions through secret-aware inspection. Do not heuristically scan,
|
||
scrub, or rewrite prompts, source material, model output, or arbitrary strings.
|
||
|
||
Inspection/diagnostic commands use native reports and do not automatically create
|
||
pipeline debug bundles. Operators may redirect those reports. Neither debug mode
|
||
nor capture initiates provider probes. Explicit smoke checks retain native
|
||
support, applicability, and outcome distinctions.
|
||
|
||
## Unresolved Choices
|
||
|
||
The architecture and public direction above are selected. Resolve each remaining
|
||
choice before its dependent work; independent work need not wait. Routine API
|
||
placement and serialization choices are implementation details, not reasons to
|
||
reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||
|
||
| ID | Remaining choice or dependency | Resolve for |
|
||
| --- | --- | --- |
|
||
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||
| U5 | Decide generic retry handling for non-LLM operations. LLM controls, `max_additional_turns` naming/default/precedence, structural-default delegation, and intent-based LLM setting migration are resolved below. | Phases 3–4 and configuration migration. |
|
||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||
| U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. |
|
||
| U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. |
|
||
| U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. |
|
||
|
||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||
native schema-source selection, and removal of duplicate schema/filesystem
|
||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||
Implementation remains in Phase 2; U4 selects prompt roles and contracts, and U5
|
||
selects native repair delegation and the application correction allowance.
|
||
|
||
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
|
||
and the shared pipeline producer helper with module functions and a pipeline
|
||
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
|
||
Implement in Phase 3 and refine signatures during Phase 4 without reopening
|
||
ownership. U4/U5 select prompt/guidance and budget behavior; remaining failure
|
||
and terminal-policy choices stay in U5/U6.
|
||
|
||
**U4 resolved:** Domain-specific initial/correction prompt pairs, shared response
|
||
schemas, default profile behavior, validator-owned guidance, and complete
|
||
applicable-chain assessment are selected in
|
||
[the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved).
|
||
Implement templates and operation-specific inputs in Phases 3–4; exact prose does
|
||
not reopen ownership. U5 selects budgets; terminal/fallback policy remains U6.
|
||
|
||
**U5 controls resolved:** `max_additional_turns` defaults to `2`, with binding over
|
||
pipeline precedence and explicit zero disabling correction. Native transient and
|
||
structural-repair defaults remain upstream-owned; bundled prompts and application
|
||
requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-controls-resolved)
|
||
also selects intent-based LLM setting migration and removal of outer LLM retries.
|
||
Non-LLM retry handling remains open; U6 owns terminal/fallback decisions.
|
||
|
||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
|
||
existing artifact schemas remain domain contracts. Any actual artifact-format
|
||
change discovered during implementation needs an explicit compatibility decision,
|
||
not an assumption that native CLI adoption authorizes it.
|
||
|
||
## Implementation Sequence
|
||
|
||
All six phases are pending. Each delivers a reviewable outcome, not a separate
|
||
release or compatibility layer. Tests and canonical documentation accompany
|
||
behavior changes; the last phase verifies integration and completeness. Record
|
||
major boundary changes in successor ADRs without rewriting accepted decisions.
|
||
|
||
### Phase 1: Establish The Migration Baseline
|
||
|
||
- Record the selected dependency set and review U1 against the upstream release.
|
||
Keep the cache dependency explicit without blocking independent work.
|
||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||
forward Audita evidence without adopting its application wrappers.
|
||
- Preserve U5–U9 with their dependent phases; do not reopen resolved module
|
||
execution, prompt/guidance ownership, retry controls/defaults, assets/schemas,
|
||
CommandKit, debug, cache, or output ownership.
|
||
|
||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||
and outstanding choices are explicit. Upstream availability alone does not
|
||
establish compatibility of the assembled Notarius dependencies or completion
|
||
of the migration.
|
||
|
||
### Phase 2: Build Native Composition, Sources, And Preflight
|
||
|
||
- Implement the resolved U2 design, update available dependencies, introduce
|
||
configuration version 6, convert native embedded/companion resources, and use
|
||
native schema loading with embedded/default or operator-selected roots. Omit
|
||
bundled structural-repair settings and remove their application override fields.
|
||
- Build the small CommandKit application, shared ConfigKit/engine loaders,
|
||
standard suites, pipeline declarations/native alias, and `internal/preflight`.
|
||
- Add maintained embedded-only and operator-override examples. Remove duplicated
|
||
loading and asset assembly as callers migrate.
|
||
|
||
**Completion evidence:** runtime/inspection/preflight share intended definitions;
|
||
source contexts, shared message paths, prompt-declared output contracts, selected
|
||
schema roots, strict peer conflicts, and replacement work offline. Help,
|
||
version, and completion need no configuration. Preflight checks known dependencies
|
||
without credentials or synthetic requests. `config diff` handles two selections.
|
||
|
||
### Phase 3: Prove A Representative Native Pipeline
|
||
|
||
- Implement the resolved U3/U4 collaboration and U5 controls; settle remaining
|
||
U5/U6 failure-policy details.
|
||
Connect module request/interpretation functions to the pipeline assessment
|
||
callback through the shared producer helper; refine concrete signatures here.
|
||
Implement domain-specific initial/correction prompts using the same schema and
|
||
default profile, supplied overrides, and validator-owned correction guidance.
|
||
Resolve `max_additional_turns` with omission/zero distinctions and pass it to
|
||
native coordination; inherit native transport and structural-repair defaults.
|
||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||
handles and fresh plan application. Select actual observer/group enrollment.
|
||
- Run through the new command surface and alias, using the small completion
|
||
report, native outcomes/streams, and useful debug capture from the outset.
|
||
Begin U8/U9 integration without a provisional local message hash.
|
||
|
||
**Completion evidence:** fake-provider workflows exercise accumulated history,
|
||
collection of independent validator findings without rejection short-circuiting,
|
||
full candidate revalidation through the same assessor, final-turn acceptance
|
||
without duplicate final validation, selected exhaustion/fallback, and useful
|
||
failures. Unavailable assessments remain distinct from actionable defects. The
|
||
executable publishes artifacts and reports them through native presentation
|
||
without the private completion/correction API.
|
||
|
||
### Phase 4: Migrate All Roles And Remove Superseded Machinery
|
||
|
||
- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve
|
||
deterministic paths and ordered validation. Apply U4's prompt/guidance design
|
||
and U5's selected controls across roles; settle remaining U5–U7 choices.
|
||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
|
||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||
encoders, verbosity handling, and obsolete asset registration. Retire the
|
||
emptied `internal/framework/llm` package after moving engine assembly to the
|
||
composition root and producer collaboration to the pipeline.
|
||
|
||
**Completion evidence:** every role uses the shared native engine and selected
|
||
admission policy. No production path depends on the private completion API,
|
||
latest-response-only correction, global ceiling, or legacy CLI infrastructure.
|
||
Obsolete tests retire with the machinery; surviving domain requirements remain.
|
||
|
||
### Phase 5: Complete Cache, Output, And Capture Integration
|
||
|
||
- Resolve U1's supplying release and complete U8. Implement lookup at initial
|
||
preparation, accepted-artifact storage under that hash, explicit cache controls,
|
||
and deliberate old-checkpoint handling. Keep canonical chunk caching separate.
|
||
- Complete the artifact-first publication/report contract and U9 details. Remove
|
||
checkpoint fingerprint graphs, mandatory manifests/receipts, durable execution
|
||
ledgers, and duplicate accounting; do not retain them to populate summaries.
|
||
- Complete capture inventory, per-operation evidence, directory/empty-value
|
||
behavior, `latest`, and recording/finalization failure handling.
|
||
|
||
**Completion evidence:** downstream failure/fix/retry reuses accepted predecessors;
|
||
message changes miss and settings-only changes can hit; corrected answers use the
|
||
initial key; hits make no model calls or fresh validation claims. Artifacts need
|
||
no metadata bundle. Machine output and debug capture obey the selected contracts,
|
||
including failures after publication and quiet-mode capture.
|
||
|
||
### Phase 6: Verify And Document The Coordinated Cutover
|
||
|
||
- Complete the acceptance checks and removal inventory, including dependency
|
||
cleanup, maintained examples, and the assembled command surface.
|
||
- Finish release migration guidance, attribution, successor ADRs, and affected
|
||
canonical documentation. Update this roadmap's status and reconcile the
|
||
separate future-work backlog.
|
||
- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash
|
||
policy, or obsolete metadata requirement remains in the finished migration.
|
||
|
||
**Completion evidence:** PromptKit, ConfigKit, and CommandKit adoption, accepted-
|
||
artifact reuse, artifact delivery, diagnostics, and maintained offline workflows
|
||
are verified together. Remaining work is explicitly separate future scope.
|
||
Release preparation/publication is governed separately by release policy.
|
||
|
||
## Removal Inventory
|
||
|
||
| Retire or replace | Retained application responsibility |
|
||
| --- | --- |
|
||
| `StructuredLLMClient`, private request/response types, adapter conversions, emptied `internal/framework/llm` package | Composition-root engine construction, native module request/interpretation functions, and pipeline assessment/helper. |
|
||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. |
|
||
| Application `structured_output_repair_attempts` fields/resolution and explicit bundled `output.repair_attempts` | Native upstream structural-repair default, with operator customization through native prompt configuration. |
|
||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||
| Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. |
|
||
| Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. |
|
||
| Custom dispatch/help/version/parsing, command-specific loading, copied inspection, legacy aliases/flags, parallel encoders and verbosity handling | Native declarations/loaders, pipeline handlers, domain input checks. |
|
||
| Mandatory run manifests, reporting-only metadata bundles, extensive receipts and durable execution ledgers | Domain artifact files, one small native completion report, optional debug evidence. |
|
||
| Summary/trace hierarchy, heuristic debug-content scrubbing, custom transcript/deduplication machinery | Convenient run/operation capture and native secret/credential omissions. |
|
||
|
||
## Acceptance And Verification
|
||
|
||
Use real CommandKit, ConfigKit, and PromptKit in representative assembled flows
|
||
with a fake at the provider boundary. Keep the suite deterministic, offline, and
|
||
independent of real credentials. Test domain interpretation/assessment at their
|
||
package boundaries; use subprocess tests for streams, exit status, alias wiring,
|
||
and representative machine output. Do not duplicate upstream parser/renderer/
|
||
retry tests or freeze complete help text and native debug snapshots.
|
||
|
||
| Area | Required evidence |
|
||
| --- | --- |
|
||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
|
||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
|
||
| Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. |
|
||
| Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. |
|
||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
|
||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |
|
||
| Cache migration | Selected reuse/bypass/refresh and old-record treatment work; invocation configuration no longer blocks lookup; canonical chunk caching retains its distinct policy. |
|
||
| Output/outcomes | Artifact files independent of report format; native text/JSON/YAML summary and qualified outcomes; no mandatory metadata bundle; quiet suppression, verbose stderr, debug path; native nonzero failures and useful explanations. Publication survives later capture/report failure without model reruns. |
|
||
| Execution observations | Initial/continuation/validator totals preserve unknown-versus-zero usage without double counts; result-less failures observable; reuse does not count historical work; native execution success remains distinct from acceptance. |
|
||
| Debug controls/lifecycle | Default/CLI/config/explicit-empty paths and native mode ordering; quiet capture; allocation before engine/preflight; start-time relative `latest`, no completion retarget; disabled runs leave it untouched; capture failures preserve primary causes and publication facts. |
|
||
| Debug evidence | Available input/source/native snapshots, readable text, all visible turns, assessments, normalization and deterministic outputs, cache hits, and failed-run evidence; no invented provider transcript; ConfigKit secrets/native credentials omitted, no heuristic content scrubbing. |
|
||
| End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit phase. |
|
||
|
||
Run focused checks during implementation and the repository-wide checks for the
|
||
completed cutover under the [testing policy](../policy/testing.md):
|
||
|
||
```sh
|
||
go test ./...
|
||
go vet ./...
|
||
go build ./cmd/notarius
|
||
```
|
||
|
||
Check maintained examples, dependencies, attribution, documentation links, and
|
||
removal coverage. Live provider smoke and human model-quality evaluation remain
|
||
optional, outside the default suite. These migration checks add no permanent
|
||
release gates. Documentation completion does not claim that implementation
|
||
acceptance checks have passed.
|
||
|
||
## Migration Guidance And Documentation
|
||
|
||
Provide practical release guidance for operators and subprocess callers:
|
||
|
||
- New commands/flags, native `run` alias, `--format`, stdout/stderr, outcomes, and
|
||
saving optional reports by redirection. No legacy CLI compatibility promise.
|
||
- Version 6 and native resources, embedded-only versus companion examples in
|
||
appropriate directories, imports, qualified IDs, replacement/inheritance,
|
||
native local backends, removed catalogs/versions, and separate source versions.
|
||
- Native admission and worker defaults, removed global limit, `max_additional_turns`
|
||
default/precedence/zero behavior, and intent-based migration from LLM `retries`.
|
||
Explain upstream-owned structural defaults, removal of application repair
|
||
controls and bundled counts, native operator customization, accumulated-history
|
||
costs, and selected role failure policy.
|
||
- Accepted-artifact caching, message-based misses, settings-only reuse, refresh/
|
||
bypass for changed interpretation or validation, and old-checkpoint handling.
|
||
- Artifact delivery without mandatory manifests/receipts; preserve intrinsic
|
||
artifact schemas and document any explicitly selected format change.
|
||
- Debug root/default/empty behavior, `latest`, native mode ordering, failure
|
||
behavior, and ordinary operator responsibility for application data.
|
||
|
||
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
||
[operations](../operations.md), [internal overview](../internal/overview.md),
|
||
[LLM](../internal/llm.md), [pipeline](../internal/pipeline.md),
|
||
[modules](../internal/modules.md), [state](../internal/state.md), and affected
|
||
integration/consumer guides as behavior lands. Remove superseded manifest/receipt
|
||
contracts then; keep current-behavior documentation accurate until implementation.
|
||
Update [architecture policy](../policy/architecture.md) for native dependencies
|
||
and changed scheduler, retry, asset, cache, and reporting boundaries.
|
||
|
||
Record significant changes in successor ADRs, including
|
||
[asset assembly](../adr/0011-centralize-llm-assets.md),
|
||
[validation retries](../adr/0014-feedback-aware-validation-retries.md),
|
||
[module retries](../adr/0016-feedback-aware-module-requested-retries.md),
|
||
[PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md),
|
||
admission, caching, and CLI/reporting ownership. Preserve accepted decision text
|
||
and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||
This roadmap owns implementation status; Git history retains earlier discussions.
|
||
|
||
## Evidence And Reading
|
||
|
||
These sources support the selected design, not a claim of completed Notarius
|
||
implementation. Sibling links assume the local repositories are available; the
|
||
tagged PromptKit links below preserve the reviewed baseline.
|
||
|
||
### Native Library Guidance
|
||
|
||
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md) and
|
||
[CLI conventions](../../../commandkit/docs/cli.md): module selection, native
|
||
aliases/inputs, shared loaders, reporting modes, outcomes, and delivery failures.
|
||
- [ckconfig](../../../commandkit/integrations/ckconfig/README.md) and
|
||
[ckprompt](../../../commandkit/integrations/ckprompt/README.md): standard suites
|
||
supplied with application loaders and request builders.
|
||
- [ConfigKit](../../../ConfigKit/README.md): native loading and inspection.
|
||
- [PromptKit consumer guide][consumer] and
|
||
[capacity behavior](../../../promptkit/docs/internal/capacity.md): native
|
||
preparation/execution, continuation, observations, and admission.
|
||
|
||
### Audita Comparison
|
||
|
||
The comparison was completed on 2026-11-18 against Audita revision
|
||
`da5b8913d1206a8e8f0a784b2322760385548a60`, using PromptKit v0.23.2,
|
||
ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
|
||
The observations below concern that revision; linked sibling files may evolve.
|
||
|
||
| Evidence | Lesson carried into this roadmap |
|
||
| --- | --- |
|
||
| [Engine assembly](../../../audita/internal/framework/llm/runtime.go), [source tests](../../../audita/internal/framework/llm/engine_sources_test.go), and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go) | One captured engine, native underlays and main-source-relative companions, whole-record replacement, preserved file contexts, strict peer conflicts, and no silent fallback on operator-source errors. |
|
||
| [Adapter tests](../../../audita/internal/framework/llm/adapter_test.go) | Ordinary embedded schemas and complete per-request output contracts work with operator prompt replacement. U2 chooses prompt-declared defaults for Notarius; Audita's schema registry and per-call overrides are not requirements to copy. |
|
||
| [Native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), [repair flow](../../../audita/internal/framework/runner/repair.go), [conversation tests](../../../audita/internal/framework/llm/conversation_test.go), and [repair tests](../../../audita/internal/framework/runner/repair_test.go) | Captured exchanges, accumulated history, complete candidate revalidation, final-turn assessment, and canceled/undelivered response handling. |
|
||
| [Prompt guidance](../../../audita/docs/internal/prompts.md) | Durable system instructions, an initial user task, and subsequent repair assignments suit native continuation. |
|
||
| [Observations](../../../audita/internal/framework/llm/observations.go) and [CLI composition](../../../audita/internal/cli/app_wiring.go) | Native execution totals with application scope and explicit nested-validator enrollment; standard CommandKit suites using shared loaders. |
|
||
|
||
Audita repairs individual transcript corrections in independent branches;
|
||
Notarius's selected unit is a whole producer operation and typed artifact.
|
||
Audita's wrappers, response-schema registry, per-proposal policies, and absence
|
||
of a cache do not define Notarius requirements.
|
||
|
||
At the reviewed revision, focused LLM, prompt/schema, runner, and proposal-generation
|
||
tests passed, together with subprocess checks for correction conversations,
|
||
cancellation during repair, and repair-disabled/output selections using a local
|
||
fake provider. These historical checks do not establish live-model quality or
|
||
completion of Notarius's migration acceptance checks.
|
||
|
||
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md
|
||
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md
|
||
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/formats.md
|
||
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/pkg-promptkit.md
|