927 lines
61 KiB
Markdown
927 lines
61 KiB
Markdown
# Native PromptKit And CommandKit Migration
|
||
|
||
## Purpose And Status
|
||
|
||
Make Notarius fully native to PromptKit, ConfigKit, and CommandKit. Use upstream
|
||
configuration, execution, continuation, admission, inspection, commands, and
|
||
presentation directly. Retain the application work of producing, assessing,
|
||
publishing, and reusing typed artifacts. Historical duplication of library
|
||
behavior is not a requirement to preserve it.
|
||
|
||
This roadmap incorporates the [PromptKit architecture discussion](promptkit-architecture-discussion.md)
|
||
and all six items of the [CommandKit discussion](commandkit-migration-discussion.md),
|
||
including the subsequent accepted-artifact cache and artifact-first output
|
||
decisions. The notepads retain rationale and Audita evidence; this document owns
|
||
the selected target, unresolved choices, implementation sequence, removal
|
||
inventory, and acceptance criteria. All implementation phases are pending.
|
||
Recording these decisions has not changed application behavior.
|
||
|
||
Complete one coordinated cutover for configuration, PromptKit resources, and the
|
||
CLI. Adopt native public conventions fully; preserving old CLI behavior is a
|
||
non-goal. Supply migration guidance and maintained examples instead of aliases,
|
||
parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is
|
||
part of the main migration, not a later follow-up. Release preparation and
|
||
publication remain separate work.
|
||
|
||
## Baseline And Dependency Target
|
||
|
||
At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application
|
||
uses configuration version 5 and Go 1.27. Its dependencies include old-path
|
||
PromptKit v0.12.0 and ConfigKit v0.5.0, new-path ConfigKit v1.2.0, go-yaml
|
||
v4.0.0-rc.6, and two external PromptKit provider catalogs.
|
||
|
||
The selected reviewed baseline is:
|
||
|
||
| Module | Version |
|
||
| --- | --- |
|
||
| `go.maximumdirect.net/commandkit` | `v0.7.0` |
|
||
| `go.maximumdirect.net/ckconfig` | `v0.9.0` |
|
||
| `go.maximumdirect.net/ckprompt` | `v0.10.0` |
|
||
| `go.maximumdirect.net/configkit` | `v1.2.1` |
|
||
| `go.maximumdirect.net/promptkit` | `v0.23.3`, pending the message-content hash addition below |
|
||
|
||
Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible
|
||
validation dependencies. PromptKit v0.23.3 supersedes the initial v0.23.2 target
|
||
for its timeout/cancellation fixes. These are selected versions, not a claim
|
||
that they are the latest releases. The local tag review and integration guidance
|
||
are recorded in [CommandKit item 1](commandkit-migration-discussion.md#1-adoption-scope-and-dependency-baseline).
|
||
The tagged [release notes][release], [migration guide][migration],
|
||
[format reference][formats], and [Go consumer guide][consumer] supply the baseline.
|
||
|
||
The cache requires the proposed native `MessageContentHash`. The
|
||
[upstream request](../../../promptkit/docs/roadmap/notarius-request.md) is pending;
|
||
v0.23.3's `RenderedPromptHash` is not a substitute. Confirm the supplying API and
|
||
tagged release, then verify the assembled dependency set. Independent composition
|
||
and pipeline work can proceed while that addition is pending; the completed
|
||
cache must use the upstream capability. Do not add a local hash-policy fork.
|
||
Neither a cross-backend global limit nor a comprehensive execution fingerprint
|
||
is required. Remove obsolete imports and catalog dependencies when unused;
|
||
avoid permanent `replace` directives and update [attribution](../oss/attribution.md)
|
||
under the [OSS policy](../policy/oss.md).
|
||
|
||
Audita's reviewed v0.23.2 integration demonstrates native source assembly,
|
||
continuation, observations, and CommandKit wiring. Adopt those mechanisms
|
||
without copying its `RoleClient`, structured-output interface, or opaque exchange
|
||
wrapper. Its per-proposal repair branches and lack of checkpoints do not define
|
||
Notarius's artifact and recovery policies.
|
||
|
||
## Target Architecture
|
||
|
||
### Application Structure And Ownership
|
||
|
||
| Part | Responsibility |
|
||
| --- | --- |
|
||
| Application composition | Load configuration and native sources; construct one shared PromptKit engine for producers, validators, and continuation runs. |
|
||
| Artifact-family modules | Prepare domain inputs, select prompt IDs and response requirements, interpret responses, and apply deterministic domain rules. |
|
||
| Pipeline execution | Connect generation to configured validation and terminal policy; manage ordered dependencies, typed handoffs, bounded workers, and stable result ordering. |
|
||
| PromptKit | Prepare and execute requests; maintain captured exchanges and conversations; enforce native retry, repair, turn, and admission budgets; report execution facts. |
|
||
| Application state and records | Publish artifacts, cache final accepted artifacts, provide a small completion summary, and capture optional debug evidence with pipeline context. |
|
||
| ConfigKit and CommandKit | ConfigKit owns configuration mechanics; CommandKit and ckconfig/ckprompt own standardized commands and presentation. Notarius supplies loaders, request builders, report content, and outcome mapping. |
|
||
|
||
Use native PromptKit requests, results, exchanges, errors, and observers in
|
||
model-executing application code. Remove the private provider-neutral completion
|
||
API. Pure domain models, codecs, source-reference resolution, and deterministic
|
||
transformations need no PromptKit dependency. Application interfaces continue to
|
||
express artifact production, typed validation, output encoding, and state access.
|
||
|
||
A small shared application function connects module request construction and
|
||
response interpretation to pipeline validation and terminal policy. It calls
|
||
native `Run` and `Coordinate` and returns a typed application outcome. It owns no
|
||
replacement completion API, conversation protocol, or general retry engine.
|
||
Exact signatures and helper placement remain open. The current
|
||
`internal/framework/llm` package may shrink to engine assembly and focused helpers
|
||
or cease to be a separate package.
|
||
|
||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||
and deterministic public ordering. Keep modules independent of physical output,
|
||
cache, and debug roots. Native adoption does not turn the pipeline into a general
|
||
workflow engine or redesign durable D&D artifacts.
|
||
|
||
### Command Surface And Native Inputs
|
||
|
||
Install the application `pipeline` family alongside ckconfig and ckprompt:
|
||
|
||
| Command | Responsibility |
|
||
| --- | --- |
|
||
| `pipeline list` | Discover configured pipelines. |
|
||
| `pipeline show <pipeline-id>` | Report resolved steps, lanes, modules, validators, and reference relationships. |
|
||
| `pipeline validate <pipeline-id>` | Check composition and known prompt/profile dependencies offline. |
|
||
| `pipeline run <pipeline-id> --input-file PATH` | Execute the pipeline and publish artifacts. |
|
||
| `run <pipeline-id> --input-file PATH` | Native alias for `pipeline run`, with the same handler and inputs. |
|
||
|
||
Use `commandkit.Alias("run", "pipeline run")`, not a forwarding command. Use
|
||
`notarius run` in quick starts and document both routes. Install the full selected
|
||
standard suite: native help/version; Bash, Zsh, and Fish completion; configuration
|
||
scopes, validation, show, sources, explain, and diff; PromptKit resource listings
|
||
and inspection for prompts, profiles, backends, namespaces, and collections;
|
||
requirements discovery/validation and prepared-request preview; and explicit
|
||
reachability, authentication, text, structured-output, and aggregate smoke tests.
|
||
Include upstream convenience aliases and useful completion hints.
|
||
|
||
Use native typed declarations and supplied-value semantics. Defaults apply to
|
||
omission, not explicit empty/false/zero values. Preserve native command-path,
|
||
operand, repeated-flag, and `--` parsing conventions. Check every occurrence of
|
||
a singleton flag and use the last successfully parsed value.
|
||
|
||
| Concern | Selected input |
|
||
| --- | --- |
|
||
| Configuration | Native `--config`, `--overlay`, and supported application `--set POINTER=JSON`; scope named `app`. |
|
||
| Pipeline input | Required pipeline-ID operand and `--input-file PATH`. |
|
||
| Lanes | Repeatable `--lane ID`; omission selects the normal lane set. Commas are literal values, not separators. |
|
||
| Output | Application `--output-dir PATH`. |
|
||
| Chunk-plan cache | `--chunk-cache auto\|bypass\|refresh`. |
|
||
| Accepted-artifact cache | Reuse/bypass/refresh controls; spelling, scope, and defaults remain U8. |
|
||
| References | Repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`. |
|
||
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
|
||
| Capture destination | Application `--debug-dir PATH`, with the semantics below. |
|
||
|
||
Keep worker counts and semantic budgets in configuration, adjustable through
|
||
application `--set`; their fields/defaults remain open. Use ckprompt's reusable
|
||
profile, session, and execution flags, including model, temperature, timeout,
|
||
reasoning effort, and transient retries, for run-wide overrides across producers,
|
||
validators, and continuations. Omission preserves normal selection; explicitly
|
||
empty reasoning effort clears inheritance. Use native `--llm-profile` and
|
||
`--llm-session-id`, retiring the separate reasoning-clearing control. Native
|
||
transient retries remain distinct from semantic additional-turn budgets.
|
||
Individual prompt selection and named prompt inputs belong on standard PromptKit
|
||
commands; pipeline modules build their own requests.
|
||
|
||
Notarius retains reference-selector expansion, specificity, required slots, and
|
||
external/generated-reference conflict checks. Pipeline reference selectors are
|
||
not individual-request `--llm-input-file` bindings. Remove custom dispatch,
|
||
assembled help, standard flag parsing, comma-separated `--only`, and bespoke
|
||
`--json` switches. Replace `config effective` with `config show` and `config
|
||
profile` with `promptkit show profile`. Old flags are migration evidence only.
|
||
|
||
### Application Composition, Loaders, And Preflight
|
||
|
||
One constructor registers application commands, the native alias, ckconfig's
|
||
`app` and `promptkit` scopes, ckprompt, and native help/version/completion/reporting.
|
||
It does not load configuration, build an engine, or initialize run storage.
|
||
Help, version, and completion work without application configuration.
|
||
|
||
Share two loading responsibilities across application and standard commands:
|
||
|
||
1. Select application configuration through explicit `--config`, then existing
|
||
environment/discovery policy. Let ConfigKit apply overlays, imports,
|
||
inheritance, overrides, defaults, normalization, and structural validation.
|
||
Retain its native result and selected main-source context. Validate an
|
||
explicitly empty configuration path rather than falling through to discovery.
|
||
2. Build the native engine from that application's source context using the source
|
||
assembly below. Across commands, `--config` selects the application document;
|
||
the PromptKit scope inspects the engine assembled for it. Request overrides
|
||
do not mutate captured engine configuration. Do not emulate unsupported
|
||
arbitrary `--set` overrides for the PromptKit scope.
|
||
|
||
Use small invocation-owned state through CommandKit's facilities to reuse loaded
|
||
values. A pipeline shares one engine across all roles. `config diff` may load two
|
||
distinct configurations within one invocation: keep their selections separate,
|
||
including overlays/overrides. No persistent loader cache or configuration hash
|
||
framework is needed.
|
||
|
||
Reuse the application resolver for module/options checks, typed compatibility,
|
||
ordered dependencies, validator chains, and reference bindings. `pipeline list`
|
||
only enumerates configuration; `show` renders resolved composition rather than
|
||
building an engine to expand every profile. ConfigKit supplies loaded values and
|
||
native provenance/secret-aware inspection, not domain pipeline semantics.
|
||
|
||
Centralize known prompt/profile dependency checks in `internal/preflight`. It
|
||
accepts the resolved pipeline, module declarations, and native engine. Check
|
||
every statically known dependency, including LLM validators and correction
|
||
prompts, plus applicable/default profiles. Partial profiles alone are not errors.
|
||
Share identifiers and selection logic with execution; no independent inventory,
|
||
asset registry, or synthetic request execution is needed. Dynamic selections
|
||
receive their final checks at runtime. Findings identify pipeline, step/lane,
|
||
stage/validator, and resource where applicable.
|
||
|
||
`pipeline validate` and `run` call the same resolution/preflight functions;
|
||
handlers do not invoke other commands or parse their reports. Keep `config
|
||
validate` focused on configuration and remove its pipeline-specific mode.
|
||
Offline preflight reads configuration and prompt resources but needs no transcript,
|
||
reference contents, credential values, or provider calls. Concrete PromptKit
|
||
preparation/validation owns completed-request checks, including schemas and
|
||
credentials; provider smoke tests require explicit invocation.
|
||
|
||
Preflight does not discover configuration, register modules, build engines,
|
||
initialize storage, or render CLI reports. Modules do not import it. Use native
|
||
request builders for standard commands and add application callbacks only where
|
||
needed, preserving supplied operator values. Output/cache/debug allocation
|
||
belongs to the run path, with debug timing selected below.
|
||
|
||
### Configuration, Sources, And Prompt Assets
|
||
|
||
Introduce application configuration version 6 as the only supported version.
|
||
Preserve unrelated discovery, pipeline structure, and resolution behavior.
|
||
ConfigKit continues to own loading and composition under
|
||
[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
|
||
Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of
|
||
native companion sources; do not introduce a separate `promptkit.config_file`.
|
||
Retire `concurrency.total_llm` and revise retry/worker settings as described below.
|
||
Their replacement defaults and detailed migration treatment remain open.
|
||
|
||
Use one source assembly for runtime, preflight, and inspection:
|
||
|
||
1. PromptKit's bundled OpenRouter underlay.
|
||
2. Notarius's embedded native prompts and profiles under `collections.notarius`.
|
||
3. An optional, higher-priority operator underlay containing `prompts.yml` and
|
||
recursive `.yml`/`.yaml` files under `prompts/`, beside the main application
|
||
configuration actually selected by ConfigKit.
|
||
|
||
Anchor companion discovery to that selected main source, whether explicit or
|
||
discovered. Configurations in the same directory share companions; there is no
|
||
working-directory fallback. The two companion locations are strict peers, so
|
||
filenames do not resolve conflicting operator records. Missing companions are
|
||
fine; loading errors do not silently fall back to embedded defaults. Preserve
|
||
native imports and source-relative resource paths.
|
||
|
||
Native declarations become the sole prompt assembly mechanism. Keep the embedded
|
||
content tree and module ownership of prompt meaning, response requirements, and
|
||
domain interpretation. Remove prompt/fallback-profile registration in
|
||
`AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`,
|
||
shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration
|
||
for checkpoint fingerprints. The resolved schema design below also removes
|
||
schema registration and filesystem-flattening machinery.
|
||
|
||
Convert maintained resources, examples, and fixtures to native identified lists,
|
||
`config:` source controls, and sibling inheritance (`extends: {id: parent}`).
|
||
Same-ID replacement supplies a complete record; distinct inherited variants can
|
||
customize an existing resource. Explain native profile-null/backend fallback
|
||
semantics. The optional PromptKit source header is version 2, independently of
|
||
Notarius's application version. Do not translate legacy resource formats or
|
||
synthesize compatibility aliases.
|
||
|
||
Select prompts by ID alone; remove obsolete PromptKit version selectors and
|
||
related request/debug metadata. Retain current local resource IDs and use exact
|
||
qualified references, such as `notarius/dnd.spells` and
|
||
`notarius/dnd-extraction`. Keep the extraction profile's existing model/settings
|
||
and its reference to the bundled unqualified `openrouter` backend. Inherited
|
||
application profiles use sibling local IDs in the same collection. The bundled
|
||
inventory differs from the old external catalogs and selects no model for an
|
||
application request; document intentional replacements for removed catalog IDs.
|
||
The bundled backend supplies no concurrency limit. Application/operator limits
|
||
need explicit native configuration; application-supplied defaults remain U7.
|
||
|
||
Preserve prompt prose, message ordering, roles, and cache boundaries where their
|
||
meaning carries forward. Review and deliberately revise initial/repair prompts
|
||
for accumulated conversation history; preserving every old byte or retry message
|
||
is not a migration requirement. Shared content should use native filesystem
|
||
contexts and imports rather than reconstructed virtual trees.
|
||
|
||
#### Prompt Assets And Response Schemas (U2 Resolved)
|
||
|
||
Use the existing root `assets` package's ordinary `embed.FS` directly for native
|
||
prompt sources and response schemas, with `fs.Sub` only where useful. Shared
|
||
message files use their actual paths within the native filesystem context.
|
||
Remove synthetic mounts, file copying, flattened schema namespaces, and module
|
||
asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening
|
||
consumers disappear; no replacement filesystem abstraction is needed.
|
||
|
||
Declare response format, validation mode, schema path, and structural-repair
|
||
budget in native prompt definitions by default. Modules select prompt IDs, supply
|
||
inputs, and interpret responses. Initial and correction prompts may reference
|
||
the same schema file. Execution, inspection, and prepared-request preview then
|
||
use the same declared contract without repeating schema selection in Go.
|
||
Role-specific prompt contents remain U4, and retry defaults remain U5.
|
||
|
||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||
dependent contract. It replaces the whole configured contract, so supply the
|
||
intended format, validation mode, schema, and repair policy together. Do not
|
||
introduce a wrapper or per-call registry solely to reproduce Audita's approach.
|
||
Operator prompt replacements must still produce responses the selected module
|
||
can interpret; that module boundary does not require duplicating each prompt's
|
||
schema selection in Go.
|
||
|
||
Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to
|
||
that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`.
|
||
They are separate from message-file source contexts: prompt underlays do not
|
||
implicitly overlay schema files. When an operator configures an external schema
|
||
directory, select PromptKit's native directory source instead of the embedded
|
||
root. It replaces the whole schema source and must supply the schemas used by
|
||
the selected prompts. Do not build an application filesystem overlay to mix
|
||
embedded and external schemas. Transparent layering would be a separate upstream
|
||
capability, not a requirement for this migration. Concrete configuration binding
|
||
is an implementation detail within this selected source policy.
|
||
|
||
Keep private response schemas as self-contained JSON documents. PromptKit owns
|
||
loading, compilation, structural validation, provider structured-output settings,
|
||
and structural repair. Remove Notarius's response-schema byte registry, copied
|
||
schema wrappers, defensive-copy layer, digest calculation, and duplicate
|
||
structural-validation machinery. Native prepared snapshots provide diagnostic
|
||
schema details; no schema fingerprint is added to the message-content cache key.
|
||
|
||
Keep typed decoding, source/reference checks, reconciliation safety, semantic
|
||
validators, artifact codecs, and durable artifact schemas in Notarius. Response
|
||
schemas describe model output; artifact schemas describe published and consumed
|
||
domain artifacts. Preserve identifiers and versions belonging to actual artifact
|
||
contracts. Retire private schema keys, registration records, hashes, and copied
|
||
identity/version/name metadata whose only consumers were adapters, checkpoints,
|
||
or reporting. No independent schema metadata system replaces them.
|
||
|
||
Implementation is pending. Verify native preparation of embedded and operator
|
||
prompt definitions, shared-file paths, schema selection from embedded and external
|
||
roots, and complete request-contract replacement where actually used. Missing or
|
||
invalid schemas should follow native preparation behavior. No new schema check
|
||
requiring synthetic requests is added to offline pipeline preflight.
|
||
|
||
### Native Production, Continuation, And Assessment
|
||
|
||
Start with one conversation per producer operation: a chunk-plan generation,
|
||
one lane's extraction of one chunk, or one lane's normalization pass. Use
|
||
complete replacement model responses. Normalization may continue to return a
|
||
reconciliation plan that deterministic code applies to produce the artifact.
|
||
Per-entity branching is not part of this migration.
|
||
|
||
The application flow is:
|
||
|
||
1. Prepare source context, references, and candidate-handle mappings once. Build
|
||
the native initial request. For cacheable work, prepare and check the cache
|
||
before execution as described below; execute the retained request on a miss.
|
||
Otherwise use ordinary `Engine.Run`. Capture an exchange when continuation
|
||
or debug inspection needs one.
|
||
2. Consume the native usable response or terminal failure, decode, and apply
|
||
module interpretation and safety checks. A module can supply actionable
|
||
findings before it has a materialized candidate. Run configured artifact
|
||
validation on each candidate.
|
||
3. For actionable correction, call `Engine.Coordinate` with the captured exchange
|
||
and `AdditionalTurns`. Its callback selects a repair prompt and supplies
|
||
findings; PromptKit retains the initial exchange and subsequent turns.
|
||
4. Interpret every delivered revised response and rebuild from the fixed original
|
||
inputs. Run the complete configured validation chain again for changed
|
||
candidates, including validators that previously approved. Assess the final
|
||
permitted response before declaring exhaustion.
|
||
5. Stop on acceptance or a terminal condition. Apply application rejection,
|
||
failure, or supported safe-fallback policy after exhaustion. Native `stopped`
|
||
and `exhausted` outcomes do not themselves mean application acceptance.
|
||
|
||
Repair prompts supply a new user task; enduring principles belong in the original
|
||
system instructions. Keep source/reference and candidate-handle meanings stable
|
||
throughout the conversation. Avoid replaying the original task and constraints
|
||
that permit only an initial response. Accumulated history increases context use;
|
||
select the semantic turn budget accordingly. Separate repair profiles or schemas
|
||
are supported but not yet selected.
|
||
|
||
Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection`
|
||
transport, manual assistant/user replay, and correction-protocol capability
|
||
checks. Replace `runProducerAttempts`' LLM retry loop with native coordination and
|
||
application assessment. Adapt `NormalizeRetry` and `producerRetryDirective` to
|
||
findings and explicit fallback information. Retain useful ordered collection and
|
||
deduplication of actionable findings, rendered through a native repair prompt.
|
||
Reason codes and operator messages are not automatically model instructions.
|
||
|
||
Transport retries, structural repair, and semantic additional turns remain
|
||
separate controls. Preserve the ability to override native transport retries,
|
||
using the upstream default when omitted. Remove blanket outer LLM retries after
|
||
native transport/structural budgets are exhausted, including repeated LLM
|
||
validator invocations in `executeValidationChain`. Keep the ordered validator
|
||
chain and aggregation. A validator execution failure is distinct from rejection
|
||
of a producer candidate. Existing producer/validator `retries` settings cannot be
|
||
mechanically renamed; decide their migration and non-LLM failure handling
|
||
explicitly. Use native `OperationTimeout` when that operation scope is needed,
|
||
without adding an arbitrary default or replacing broader application deadlines.
|
||
|
||
Keep source grounding, reference resolution, candidate isolation, domain
|
||
invariants, and reconciliation safety checks. Unknown or overlapping handles
|
||
remain invalid. A supported safe fallback may retain safe merges while leaving
|
||
unsafe groups separate; it must pass configured artifact validation and terminal
|
||
policy. After structural-repair exhaustion it can be assessed directly without
|
||
restarting the same model operation. Whether an earlier safe candidate can
|
||
survive a later failed turn remains open; do not silently introduce best-so-far
|
||
acceptance.
|
||
|
||
Preserve rejection versus unavailable/incomplete validation. A structurally
|
||
valid candidate may advance under the existing `warn_continue` policy with
|
||
explicit incomplete-validation outcome information, but it and its affected
|
||
derivatives are ineligible for accepted-artifact cache reuse or publication. A
|
||
completed rejection never advances. Cancellation cannot make an undelivered
|
||
response accepted. Keep conversation state in process, outside ordinary
|
||
artifact/cache payloads; explicitly requested detailed debug is separate.
|
||
|
||
### Admission And Pipeline Workers
|
||
|
||
Remove `Scheduler`, `scheduledClient`, and custom global queue/permit plumbing.
|
||
Retire the separate cross-backend ceiling and the default/maximum extraction
|
||
worker coupling to `concurrency.total_llm`. Keep bounded extraction workers and
|
||
bounded lane continuations with independent defaults to be selected.
|
||
|
||
Use one shared native engine across all roles. Native capacity belongs to the
|
||
engine and backend ID: profiles selecting the same ID share its pool; different
|
||
backend IDs or separate engines do not automatically share limits, even for the
|
||
same endpoint. Endpoint-only profiles and backends without a positive limit
|
||
remain unrestricted by native backend admission.
|
||
|
||
For limited backends, the reviewed baseline bounds active model attempts by
|
||
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
|
||
Ordinary execution admission spans preparation, retries, and repairs; split
|
||
`PrepareExecution` itself reserves no capacity, with admission occurring during
|
||
`RunPrepared`. Active permits are released during retry waits.
|
||
A completed continuation turn releases admission before assessment, allowing a
|
||
nested validator to use the same backend. Configure native limits at the intended
|
||
capacity boundary and handle `ErrCapacityExceeded` through application policy.
|
||
|
||
Aggregate concurrency across independent backends may exceed the former global
|
||
ceiling. That change is intentional. Bounded pipeline workers are application
|
||
work controls, not a substitute cross-backend LLM ceiling. Defaults, retired-field
|
||
migration, and role-specific admission-failure dispositions remain open.
|
||
|
||
### Accepted-Artifact Cache
|
||
|
||
Replace the checkpoint subsystem with a cache around the complete Notarius
|
||
producer operation. Its value is an accepted answer to the original task. This
|
||
avoids repeating expensive model work after a downstream failure without caching
|
||
every cheap deterministic pipeline step.
|
||
|
||
```text
|
||
Prepare initial request -> obtain MessageContentHash -> look up cache/<hash>.json
|
||
Hit: decode the accepted artifact and continue the pipeline
|
||
Miss: execute the retained request and complete the Notarius operation
|
||
-> interpret, validate, and perform application-directed corrections
|
||
-> on final acceptance, store the artifact under the INITIAL hash
|
||
```
|
||
|
||
Use the proposed upstream hash of ordered message roles and exact text. Include
|
||
message boundaries and order, excluding model/settings, session/execution IDs,
|
||
timing, provider cache hints, and other non-content metadata. Add no stage,
|
||
prompt-ID, library-version, whole-configuration, schema-policy, or implementation
|
||
fingerprint. Actual input or schema text rendered into messages participates as
|
||
message content; metadata outside those messages does not.
|
||
|
||
| Change | Intended reuse behavior |
|
||
| --- | --- |
|
||
| Fix a downstream stage | Earlier artifacts remain reusable when their initial messages match. |
|
||
| Change initial message text, roles, order, or boundaries | Cache miss. |
|
||
| Change a prompt ID, model, or settings without changing messages | Cache hit remains possible. |
|
||
| Change deterministic interpretation or validation | Existing accepted artifacts remain reusable; explicit bypass/refresh applies the new policy. |
|
||
|
||
Obtain the key before model execution through native preparation. Preserve the
|
||
prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared`
|
||
where supported, and discard the handle on a hit. `OnPrepared` execution hooks
|
||
alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes
|
||
session identity and cache hints; it is not the requested hash. The upstream
|
||
request in U1 must land before completing this cache integration.
|
||
|
||
Keep the initial key throughout corrections. Cache the final Notarius artifact
|
||
through its existing codec, not an intermediate response, `RunResult`, exchange,
|
||
or prepared handle. For normalization, save the accepted normalized artifact
|
||
after applying the reconciliation plan, not the proposed plan itself. Save only
|
||
after the entire Notarius operation succeeds, including domain validation and
|
||
any application-directed corrections. Failed, rejected, and validation-incomplete
|
||
work and ineligible derivatives are not cacheable.
|
||
|
||
A hit loads the accepted artifact directly, with ordinary decoding/usability
|
||
checks; absent or unusable entries are misses. It makes no model calls, rebuilds
|
||
no result/exchange, replays no corrections, and performs no new semantic
|
||
validation merely because the artifact was cached. It makes no claim of acceptance
|
||
under changed validation policy and contributes no historical token usage to
|
||
current execution totals.
|
||
|
||
PromptKit owns everything inside each native request, including structural
|
||
validation, transport retries, and repairs. Notarius consumes its public response
|
||
or terminal failure and owns domain acceptance. Internal PromptKit attempts do
|
||
not determine cache eligibility. Native diagnostic evidence does not transfer
|
||
that ownership. No cached-response injection, replay API, exchange restoration,
|
||
or PromptKit-owned write-through cache is requested.
|
||
|
||
Use a small filesystem lookup/write facility and ordinary temporary-file/rename
|
||
publication. `cache/<hash>.json` contains the artifact, with no required manifest,
|
||
acceptance ledger, historical provenance, or versioned checkpoint envelope.
|
||
Lookup does not require original output or debug directories. Remove checkpoint
|
||
identities, dependency fingerprints, invocation-wide barriers, and recovery
|
||
bookkeeping. Preserve the separate canonical chunk-plan cache and its materialized
|
||
chunk validation. Cache directory configuration, reuse/bypass/refresh controls,
|
||
and handling of old checkpoint files remain U8; old `--resume`/`--recompute-step`
|
||
conventions are not compatibility requirements.
|
||
|
||
### Inspection And Execution Bookkeeping
|
||
|
||
Delegate configuration/resource inspection, requirements, concrete preview and
|
||
validation, and explicit smoke diagnostics to ckconfig/ckprompt through the shared
|
||
loaders. Remove effective-YAML parsing, the profile-inspector wrapper, and copied
|
||
effective-parameter types. A configured profile may be partial; use native
|
||
prepared/result details for the actual execution target, including overrides.
|
||
Standard commands retain their native reports, support/applicability distinctions,
|
||
and failure behavior. Smoke status zero alone does not prove a check passed.
|
||
|
||
Use ordinary `Run` with `OnPrepared` when actual request details are needed for
|
||
diagnostics; split preparation has a separate purpose for cache lookup. Native
|
||
observations provide terminal facts even without a returned `RunResult`.
|
||
`OnPrepared` is absent on preparation/admission failures and is not a completion
|
||
record. Keep synchronous callbacks lightweight, compose capture/accounting
|
||
observers, and synchronize shared state where operations overlap.
|
||
|
||
Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope.
|
||
Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning,
|
||
independent accumulators, debug completion wrappers, synthetic call IDs, and
|
||
adapter profile-recording/forwarding. Native snapshots serve actual-target
|
||
inspection; do not reconstruct a durable profile inventory.
|
||
|
||
Use `ExecutionGroup` for selected diagnostic totals, explicitly enrolling initial,
|
||
continuation, and nested validator executions without overlapping double counts.
|
||
Preserve terminal coverage and unknown-versus-zero usage. Reuse adds no historical
|
||
usage to current totals. These facts support diagnostics and capture, not a
|
||
mandatory consumer execution ledger. Notarius's acceptance, fallback, rejection,
|
||
incomplete-validation, publication, and reuse facts feed the completion summary
|
||
and debug overview without independently maintained reporting state.
|
||
|
||
### Artifact Delivery And Completion Output
|
||
|
||
Artifacts are the primary deliverable: publish accepted domain artifacts as files
|
||
in the output directory. Preserve intrinsic domain content and artifact schemas.
|
||
CLI `--format` controls the completion report, not payload encoding. Consumers
|
||
need no run manifest, metadata bundle, persisted receipt, or execution history to
|
||
use the artifacts. Retire reporting-only metadata and mandatory manifests instead
|
||
of rebuilding them with native values.
|
||
|
||
Submit one small structured value through `Reporter.Report`:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `pipeline` | Selected pipeline identifier. |
|
||
| `artifact_count` | Deliverable artifacts, including reused artifacts; excludes intermediates, cache entries, and debug files. |
|
||
| `output_directory` | Absolute published artifact directory. |
|
||
| `debug_directory` | Optional absolute completed run-specific capture directory; omitted when capture is disabled. |
|
||
|
||
Supply a native text template for the ordinary successful case:
|
||
|
||
```text
|
||
Success: dnd-session produced 6 artifacts in /work/results/run-123
|
||
```
|
||
|
||
When capture is available, append `Debug capture: <run-directory>`. Provide the
|
||
path rather than mirroring the bundle. JSON/YAML render the same structured
|
||
value through native serializers. A plain string would render natively as text
|
||
but lose named fields in machine output; the template gives both presentations
|
||
without parallel encoders. No report-version, run-ID, token, profile, warning-count,
|
||
or validation-history collection is required.
|
||
|
||
When a completed run includes rejected work or permitted incomplete validation,
|
||
qualify the human summary and structured value with the rejection count and/or
|
||
incomplete-validation fact. Keep these facts visible without verbose/debug mode;
|
||
small outcome qualifications do not require per-producer histories. Exact names
|
||
of those conditional fields are implementation details.
|
||
|
||
Wire results to stdout and diagnostics to stderr:
|
||
|
||
| Mode | Stdout | Stderr |
|
||
| --- | --- | --- |
|
||
| Ordinary | Brief completion report. | Failure explanation when needed. |
|
||
| `--verbose` | Same completion report. | Native information, warnings, and failures. |
|
||
| `--quiet` | Nothing. | Failure explanation when needed. |
|
||
| `--debug` | Completion report with capture path when enabled. | Native diagnostics and default progress. |
|
||
|
||
Use native mode ordering (last enabled mode wins), explicit progress selection,
|
||
and quiet suppression. Quiet still permits failure explanations and does not
|
||
prevent rendering errors from failing the invocation. `--format json --verbose`
|
||
keeps stdout valid report JSON. Standard configuration/PromptKit commands retain
|
||
their own native values, with no universal Notarius envelope. Artifact-quality
|
||
advisories retain their domain classification.
|
||
|
||
Optional saved reports use shell redirection, without a report-file flag or
|
||
automatic duplicate persistence:
|
||
|
||
```sh
|
||
notarius run dnd-session --input-file transcript.json --format json > run-report.json
|
||
```
|
||
|
||
Publish artifacts and finish requested capture before submitting a successful
|
||
completion report. Use native outcomes: status 0 for command completion, 1 for
|
||
operational/cancellation/publication/presentation failure, 2 for invalid invocation
|
||
or identifiable invalid configuration, and 3 for native `config diff --check`
|
||
differences. Preserve ckconfig's structured classification and underlying causes;
|
||
not every preflight failure is invalid input. Command completion is distinct from
|
||
domain acceptance: permitted rejections/incomplete validation qualify the report,
|
||
while terminal pipeline failure fails the command.
|
||
|
||
On failure, supply a useful native explanation and nonzero status, without a new
|
||
structured error-envelope protocol. `--format` does not serialize diagnostics.
|
||
Retain CommandKit's separation of operation causes from output/diagnostic delivery
|
||
failures. If publication succeeded before capture/report delivery failed, identify
|
||
the published path and available debug location. Published artifacts remain valid;
|
||
do not rerun model work or rewrite execution history. Subprocess callers check
|
||
status before decoding a successful report; failed writes may leave partial stdout.
|
||
|
||
### Debug Capture And Optional Diagnostics
|
||
|
||
Capture the **most convenient** debug representation, rather than optimizing
|
||
for conciseness, exhaustive completeness, or strict deduplication. Native snapshots
|
||
and readable views may repeat evidence. Reuse application decisions and native
|
||
execution facts instead of maintaining a second ledger or transcript model.
|
||
Debug formats are diagnostic data, not stable consumer contracts.
|
||
|
||
#### Controls And Location
|
||
|
||
Bind an application boolean to native `--debug`. Capture requires that boolean
|
||
and a nonempty effective directory; a directory alone does not enable capture.
|
||
Select `--debug-dir PATH` over `debug.directory` over `/tmp/notarius-debug`.
|
||
Resolve nonempty configuration-relative paths against their source and CLI-relative
|
||
paths against the invocation directory. Recognize explicit empty values first:
|
||
`--debug-dir ""` or `debug.directory: ""` disables capture without fallback.
|
||
A nonempty CLI override can re-enable capture over an empty configured root.
|
||
An unusable nonempty destination fails requested capture rather than selecting
|
||
another directory. Remove the configuration-relative `./notarius-debug` default.
|
||
|
||
Capture selection is independent of the final reporting mode. `--debug --quiet`
|
||
captures while suppressing stdout; `--quiet --debug` selects debug verbosity and
|
||
capture. `--debug=false` disables capture through native binding semantics.
|
||
Empty-directory debug retains native verbosity. No separate capture flag or
|
||
capture-depth switch is needed.
|
||
|
||
#### Lifecycle And Failure Handling
|
||
|
||
Allocate the run bundle after application configuration and run identity are
|
||
available, before engine construction and pipeline preflight. Argument/config-load
|
||
failures may have no bundle. Keep allocation out of shared loaders.
|
||
|
||
Create `<debug-root>/<run-id>/` and update `<debug-root>/latest` to a relative
|
||
symlink to that run at session start. It points to the newly started session even
|
||
if that session fails. Do not retarget at completion; concurrent starts use the
|
||
last successful link update. Runs without capture leave it untouched.
|
||
|
||
One application recorder accepts initial context, native evidence, and application
|
||
decisions, then finalizes once with pipeline/publication/capture outcomes. Persist
|
||
as operations advance, without holding the whole pipeline's content in memory.
|
||
Keep debug separate from output/cache and do not automatically delete bundles.
|
||
|
||
Requested capture is required work: allocation/link failures stop before model
|
||
work; recording failures surface at the next normal application boundary;
|
||
finalization failure fails the command. Observer callbacks retain errors for
|
||
boundary checks because they cannot return errors. An existing pipeline failure
|
||
remains primary, with capture failures reported additionally and causes retained.
|
||
Already-written data and published artifacts remain available. Capture failures
|
||
never cause model retries. Finalize before submitting the success report.
|
||
|
||
#### Inventory And Depth
|
||
|
||
Replace separate summary/trace categories with a compact run overview,
|
||
configuration/composition snapshots, parsed source, and per-operation records.
|
||
Use native execution IDs with application scope for navigation. `latest` is the
|
||
selected link name; other filenames and serialized application fields remain
|
||
implementation choices.
|
||
|
||
| Capture | Selected contents and depth |
|
||
| --- | --- |
|
||
| Run overview | Application version, run/pipeline identity, timing, outcome, publication location, effective invocation selections/overrides, and navigation. Terminal facts come from the same application outcome used by the CLI. |
|
||
| Configuration and composition | Native application and PromptKit effective snapshots, normally once each, plus resolved steps, lanes, modules, validators, and references. |
|
||
| Parsed source | Generic input document with unit identities and content, independent of the original input file. If parsing fails, retain the input location and available diagnostics. |
|
||
| Operation inputs | Actual source/chunk and reference content, incoming artifacts, prompt bindings/variables, and fixed original normalization candidates/handle mappings. Include directly even when duplicated elsewhere. |
|
||
| Prepared requests | Native `PreparedRun`, including settings, messages, and exposed output-contract/schema details when available. |
|
||
| Execution events | Native per-execution events in occurrence order, including attempts/retries/repairs, timing, usage availability, and terminal metadata. |
|
||
| Results and responses | Native `RunResult` plus exact `RawOutput` in a separate editor-readable text file. |
|
||
| Conversations | Full `Exchange.Details()` for every application-visible turn, including rejected turns, plus simple readable role-labeled messages. Repeated prefixes are acceptable. Use prepared messages when no completed exchange exists and label the representation. |
|
||
| Interpretation and assessment | Candidate via its codec before acceptance; decode/safety failures; each validator's findings and correction guidance; application disposition and links to LLM validator executions. |
|
||
| Normalization and corrections | Proposed reconciliation plan and applied candidate; constructed correction inputs even if the next preparation fails. |
|
||
| Deterministic stages and reuse | Chunk plans/materialized chunks, merged/normalized artifacts, and the loaded cached artifact with the current hit decision. No reconstructed history or fabricated execution on reuse. |
|
||
|
||
Use indented JSON for native structured snapshots and existing domain codecs.
|
||
Retain selected content in full, with no arbitrary truncation. Enable exchange
|
||
capture where debug inspection requires it, including single-turn calls;
|
||
continuation's in-process exchange needs remain independent of durable capture.
|
||
Keep operation inputs self-contained enough for direct investigation without
|
||
requiring original inputs, a cache entry, or reconstruction of deltas. Do not
|
||
archive every original file or serialize the pipeline's entire internal state.
|
||
|
||
On failure, retain available evidence, where execution stopped, errors, and
|
||
application disposition. Preparation/admission failure may have no `OnPrepared`;
|
||
operational failure may have no `RunResult`. Capture a coordination `Undelivered`
|
||
response as unassessed. Native internal-attempt metadata is useful but intermediate
|
||
provider response bodies are not required. No provider wrapper or complete
|
||
provider-transcript guarantee is part of this design.
|
||
|
||
#### Data Handling And Diagnostic Commands
|
||
|
||
Operators secure debug data as they secure ordinary application inputs and
|
||
outputs, for example with filesystem permissions. Debug contains internal data;
|
||
no separate security/retention model or sanitized-bundle promise is required.
|
||
Do not directly print/serialize ConfigKit-marked secrets, and preserve native
|
||
credential omissions through secret-aware inspection. Do not heuristically scan,
|
||
scrub, or rewrite prompts, source material, model output, or arbitrary strings.
|
||
|
||
Inspection/diagnostic commands use native reports and do not automatically create
|
||
pipeline debug bundles. Operators may redirect those reports. Neither debug mode
|
||
nor capture initiates provider probes. Explicit smoke checks retain native
|
||
support, applicability, and outcome distinctions.
|
||
|
||
## Unresolved Choices
|
||
|
||
The architecture and public direction above are selected. Resolve each remaining
|
||
choice before its dependent work; independent work need not wait. Routine API
|
||
placement and serialization choices are implementation details, not reasons to
|
||
reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||
|
||
| ID | Remaining choice or dependency | Resolve for |
|
||
| --- | --- | --- |
|
||
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||
| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. |
|
||
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
|
||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||
| U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. |
|
||
| U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. |
|
||
| U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. |
|
||
|
||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||
native schema-source selection, and removal of duplicate schema/filesystem
|
||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||
Implementation remains in Phase 2; role-specific prompts and retry budgets remain
|
||
U4/U5.
|
||
|
||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
|
||
existing artifact schemas remain domain contracts. Any actual artifact-format
|
||
change discovered during implementation needs an explicit compatibility decision,
|
||
not an assumption that native CLI adoption authorizes it.
|
||
|
||
## Implementation Sequence
|
||
|
||
All six phases are pending. Each delivers a reviewable outcome, not a separate
|
||
release or compatibility layer. Tests and canonical documentation accompany
|
||
behavior changes; the last phase verifies integration and completeness. Record
|
||
major boundary changes in successor ADRs without rewriting accepted decisions.
|
||
|
||
### Phase 1: Establish The Migration Baseline
|
||
|
||
- Record the selected dependency set and track U1 against the upstream request.
|
||
Keep the cache dependency explicit without blocking independent work.
|
||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||
forward Audita evidence without adopting its application wrappers.
|
||
- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas,
|
||
CommandKit, debug, cache, or output choices as requirements to redesign.
|
||
|
||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||
and outstanding choices are explicit. This phase does not claim that the hash
|
||
already exists or that the migration is implemented.
|
||
|
||
### Phase 2: Build Native Composition, Sources, And Preflight
|
||
|
||
- Implement the resolved U2 design, update available dependencies, introduce
|
||
configuration version 6, convert native embedded/companion resources, and use
|
||
native schema loading with embedded/default or operator-selected roots.
|
||
- Build the small CommandKit application, shared ConfigKit/engine loaders,
|
||
standard suites, pipeline declarations/native alias, and `internal/preflight`.
|
||
- Add maintained embedded-only and operator-override examples. Remove duplicated
|
||
loading and asset assembly as callers migrate.
|
||
|
||
**Completion evidence:** runtime/inspection/preflight share intended definitions;
|
||
source contexts, shared message paths, prompt-declared output contracts, selected
|
||
schema roots, strict peer conflicts, and replacement work offline. Help,
|
||
version, and completion need no configuration. Preflight checks known dependencies
|
||
without credentials or synthetic requests. `config diff` handles two selections.
|
||
|
||
### Phase 3: Prove A Representative Native Pipeline
|
||
|
||
- Resolve U3 and relevant U4–U6 details. Connect native generation, interpretation,
|
||
configured validation, and continuation through the application function.
|
||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||
handles and fresh plan application. Select actual observer/group enrollment.
|
||
- Run through the new command surface and alias, using the small completion
|
||
report, native outcomes/streams, and useful debug capture from the outset.
|
||
Begin U8/U9 integration without a provisional local message hash.
|
||
|
||
**Completion evidence:** fake-provider workflows exercise accumulated history,
|
||
full candidate revalidation, final-turn acceptance, selected exhaustion/fallback,
|
||
and useful failures. The executable publishes artifacts and reports them through
|
||
native presentation without the private completion/correction API.
|
||
|
||
### Phase 4: Migrate All Roles And Remove Superseded Machinery
|
||
|
||
- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve
|
||
deterministic paths and ordered validation. Complete U4–U7 across roles.
|
||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
|
||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||
encoders, verbosity handling, and obsolete asset registration.
|
||
|
||
**Completion evidence:** every role uses the shared native engine and selected
|
||
admission policy. No production path depends on the private completion API,
|
||
latest-response-only correction, global ceiling, or legacy CLI infrastructure.
|
||
Obsolete tests retire with the machinery; surviving domain requirements remain.
|
||
|
||
### Phase 5: Complete Cache, Output, And Capture Integration
|
||
|
||
- Resolve U1's supplying release and complete U8. Implement lookup at initial
|
||
preparation, accepted-artifact storage under that hash, explicit cache controls,
|
||
and deliberate old-checkpoint handling. Keep canonical chunk caching separate.
|
||
- Complete the artifact-first publication/report contract and U9 details. Remove
|
||
checkpoint fingerprint graphs, mandatory manifests/receipts, durable execution
|
||
ledgers, and duplicate accounting; do not retain them to populate summaries.
|
||
- Complete capture inventory, per-operation evidence, directory/empty-value
|
||
behavior, `latest`, and recording/finalization failure handling.
|
||
|
||
**Completion evidence:** downstream failure/fix/retry reuses accepted predecessors;
|
||
message changes miss and settings-only changes can hit; corrected answers use the
|
||
initial key; hits make no model calls or fresh validation claims. Artifacts need
|
||
no metadata bundle. Machine output and debug capture obey the selected contracts,
|
||
including failures after publication and quiet-mode capture.
|
||
|
||
### Phase 6: Verify And Document The Coordinated Cutover
|
||
|
||
- Complete the acceptance checks and removal inventory, including dependency
|
||
cleanup, maintained examples, and the assembled command surface.
|
||
- Finish release migration guidance, attribution, successor ADRs, and affected
|
||
canonical documentation. Reconcile future-work notes and notepad status.
|
||
- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash
|
||
policy, or obsolete metadata requirement remains in the finished migration.
|
||
|
||
**Completion evidence:** PromptKit, ConfigKit, and CommandKit adoption, accepted-
|
||
artifact reuse, artifact delivery, diagnostics, and maintained offline workflows
|
||
are verified together. Remaining work is explicitly separate future scope.
|
||
Release preparation/publication is governed separately by release policy.
|
||
|
||
## Removal Inventory
|
||
|
||
| Retire or replace | Retained application responsibility |
|
||
| --- | --- |
|
||
| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. |
|
||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives | Ordered validators, domain acceptance/rejection/fallback. |
|
||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||
| Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. |
|
||
| Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. |
|
||
| Custom dispatch/help/version/parsing, command-specific loading, copied inspection, legacy aliases/flags, parallel encoders and verbosity handling | Native declarations/loaders, pipeline handlers, domain input checks. |
|
||
| Mandatory run manifests, reporting-only metadata bundles, extensive receipts and durable execution ledgers | Domain artifact files, one small native completion report, optional debug evidence. |
|
||
| Summary/trace hierarchy, heuristic debug-content scrubbing, custom transcript/deduplication machinery | Convenient run/operation capture and native secret/credential omissions. |
|
||
|
||
## Acceptance And Verification
|
||
|
||
Use real CommandKit, ConfigKit, and PromptKit in representative assembled flows
|
||
with a fake at the provider boundary. Keep the suite deterministic, offline, and
|
||
independent of real credentials. Test domain interpretation/assessment at their
|
||
package boundaries; use subprocess tests for streams, exit status, alias wiring,
|
||
and representative machine output. Do not duplicate upstream parser/renderer/
|
||
retry tests or freeze complete help text and native debug snapshots.
|
||
|
||
| Area | Required evidence |
|
||
| --- | --- |
|
||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
|
||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. |
|
||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
|
||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |
|
||
| Cache migration | Selected reuse/bypass/refresh and old-record treatment work; invocation configuration no longer blocks lookup; canonical chunk caching retains its distinct policy. |
|
||
| Output/outcomes | Artifact files independent of report format; native text/JSON/YAML summary and qualified outcomes; no mandatory metadata bundle; quiet suppression, verbose stderr, debug path; native nonzero failures and useful explanations. Publication survives later capture/report failure without model reruns. |
|
||
| Execution observations | Initial/continuation/validator totals preserve unknown-versus-zero usage without double counts; result-less failures observable; reuse does not count historical work; native execution success remains distinct from acceptance. |
|
||
| Debug controls/lifecycle | Default/CLI/config/explicit-empty paths and native mode ordering; quiet capture; allocation before engine/preflight; start-time relative `latest`, no completion retarget; disabled runs leave it untouched; capture failures preserve primary causes and publication facts. |
|
||
| Debug evidence | Available input/source/native snapshots, readable text, all visible turns, assessments, normalization and deterministic outputs, cache hits, and failed-run evidence; no invented provider transcript; ConfigKit secrets/native credentials omitted, no heuristic content scrubbing. |
|
||
| End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit phase. |
|
||
|
||
Run focused checks during implementation and the repository-wide checks for the
|
||
completed cutover under the [testing policy](../policy/testing.md):
|
||
|
||
```sh
|
||
go test ./...
|
||
go vet ./...
|
||
go build ./cmd/notarius
|
||
```
|
||
|
||
Check maintained examples, dependencies, attribution, documentation links, and
|
||
removal coverage. Live provider smoke and human model-quality evaluation remain
|
||
optional, outside the default suite. These migration checks add no permanent
|
||
release gates. Documentation completion does not claim that implementation
|
||
acceptance checks have passed.
|
||
|
||
## Migration Guidance And Documentation
|
||
|
||
Provide practical release guidance for operators and subprocess callers:
|
||
|
||
- New commands/flags, native `run` alias, `--format`, stdout/stderr, outcomes, and
|
||
saving optional reports by redirection. No legacy CLI compatibility promise.
|
||
- Version 6 and native resources, embedded-only versus companion examples in
|
||
appropriate directories, imports, qualified IDs, replacement/inheritance,
|
||
native local backends, removed catalogs/versions, and separate source versions.
|
||
- Native admission and worker defaults, removed global limit, retry-budget
|
||
migration, accumulated-history costs, and selected role failure policy.
|
||
- Accepted-artifact caching, message-based misses, settings-only reuse, refresh/
|
||
bypass for changed interpretation or validation, and old-checkpoint handling.
|
||
- Artifact delivery without mandatory manifests/receipts; preserve intrinsic
|
||
artifact schemas and document any explicitly selected format change.
|
||
- Debug root/default/empty behavior, `latest`, native mode ordering, failure
|
||
behavior, and ordinary operator responsibility for application data.
|
||
|
||
Update canonical [configuration](../config.md), [CLI](../cli.md),
|
||
[operations](../operations.md), [internal overview](../internal/overview.md),
|
||
[LLM](../internal/llm.md), [pipeline](../internal/pipeline.md),
|
||
[modules](../internal/modules.md), [state](../internal/state.md), and affected
|
||
integration/consumer guides as behavior lands. Remove superseded manifest/receipt
|
||
contracts then; keep current-behavior documentation accurate until implementation.
|
||
Update [architecture policy](../policy/architecture.md) for native dependencies
|
||
and changed scheduler, retry, asset, cache, and reporting boundaries.
|
||
|
||
Record significant changes in successor ADRs, including
|
||
[asset assembly](../adr/0011-centralize-llm-assets.md),
|
||
[validation retries](../adr/0014-feedback-aware-validation-retries.md),
|
||
[module retries](../adr/0016-feedback-aware-module-requested-retries.md),
|
||
[PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md),
|
||
admission, caching, and CLI/reporting ownership. Preserve accepted decision text
|
||
and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records).
|
||
Notepads retain discussion history; this roadmap owns implementation status.
|
||
|
||
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md
|
||
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md
|
||
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/formats.md
|
||
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/pkg-promptkit.md
|