From 2d69c26813b271112fae72e165c9f72c5f17c7cc Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 20 Nov 2026 13:06:52 +0000 Subject: [PATCH] Consolidate migration planning into the master roadmap --- README.md | 4 +- docs/development.md | 2 +- .../commandkit-migration-discussion.md | 1252 ----------------- docs/roadmap/future.md | 66 +- .../promptkit-architecture-discussion.md | 842 ----------- docs/roadmap/promptkit.md | 124 +- 6 files changed, 124 insertions(+), 2166 deletions(-) delete mode 100644 docs/roadmap/commandkit-migration-discussion.md delete mode 100644 docs/roadmap/promptkit-architecture-discussion.md diff --git a/README.md b/README.md index 43a2bbff..fb2fb3c3 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,9 @@ Releases](docs/release.md). for maintainers. - [Developer guide](docs/development.md) — contributor orientation and validation guidance. -- [Future work](docs/roadmap/future.md) — unimplemented ideas and priorities. +- [Native migration roadmap](docs/roadmap/promptkit.md) — the planned PromptKit, + ConfigKit, and CommandKit cutover. +- [Future work](docs/roadmap/future.md) — separate backlog of ideas and priorities. Notarius is licensed under the [GPL v3.0 or later](COPYRIGHT). diff --git a/docs/development.md b/docs/development.md index d4910b11..ce4e7f0e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -48,7 +48,7 @@ implementing changes. It owns implementation discipline and local commit rules. | Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. | | External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. | | Repository-specific release commands or diagnostics | [Source Releases](release.md) | The runbook supplies publication commands and recovery procedures. | -| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. | +| Proposed or unimplemented behavior | [Native migration roadmap](roadmap/promptkit.md) and [separate backlog](roadmap/future.md) | The master roadmap owns the native cutover; the backlog holds other future work. Implemented behavior belongs in its canonical documentation. | For an existing subsystem, also inspect its focused tests and the package-local types and contracts before changing behavior. diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md deleted file mode 100644 index c57eae6f..00000000 --- a/docs/roadmap/commandkit-migration-discussion.md +++ /dev/null @@ -1,1252 +0,0 @@ -# CommandKit Migration Discussion - -## Purpose And Status - -Discussion record for the decisions incorporated into the -[master native migration roadmap](promptkit.md). That roadmap now owns the -selected design, remaining choices, implementation phases, removal inventory, -and acceptance criteria. This notepad retains discussion detail and evidence; -it is not a second implementation checklist. - -**Direction agreed and incorporated:** Make a full cutover to native CommandKit, -ConfigKit, and PromptKit workflows in one coordinated migration. Adopt upstream -public conventions fully. CLI backward compatibility is a non-goal; provide -release migration guidance instead. - -All six items below are incorporated, including the accepted-artifact cache, -small completion report and native text template, debug inventory/depth, -`/tmp/notarius-debug` default, `latest`, and explicit-empty directory behavior. -No application or dependency changes have been implemented through this -planning work. Implementation remains pending. - -The roadmap resolves U10's CommandKit scope, tracks the pending upstream -message-content hash in U1, replaces U8's checkpoint design, and narrows U9 to -implementation details within the agreed output/debug contracts. U2 now resolves -asset and schema assembly. Role-specific correction/fallback policy, worker -defaults, and concrete cache controls/transition remain visible there. Item-level -directions below record the accepted design; the roadmap is the authority for -sequencing and current status. - -## Agreed Proposal - -### Guiding Rule: Full Native Cutover - -Backward compatibility with the existing CLI conventions is a non-goal. Use -CommandKit's native command, input, help/version, reporting, error, and outcome -conventions as the design baseline. This mirrors the PromptKit-native direction. - -Existing command names, flags, aliases, parsing behavior, report shapes, and exit -conventions are migration evidence, not requirements to preserve. Do not add -legacy aliases, compatibility shims, parallel parsers, or reporting wrappers -solely to reproduce the old CLI. Any application-specific behavior should serve -an actual Notarius need and fit the native conventions. - -Provide migration guidance in release documentation, explaining the new commands -and conventions and the actions operators or subprocess callers must take. -Documenting the transition replaces a requirement to keep the old CLI working. -Update canonical CLI and consumer documentation when the behavior lands. - -Application responsibilities such as producing artifacts, reporting publication -outcomes, and managing cache/debug storage remain meaningful. Their existing -CLI expression is not protected by an implicit compatibility exception. This -guiding rule does not itself choose stored artifact formats. The revised cache -decision below selects simple artifact storage rather than checkpoint envelopes. - -### Why Adopt Together - -- Source assembly and preflight are already changing. CommandKit integrations - can consume the same application and native engine loaders. -- ckconfig and ckprompt can replace standard inspection commands directly, - avoiding an intermediate adaptation of commands that would soon be replaced. -- Native execution facts and application outcomes can feed the intended reporting - facilities while duplicate LLM bookkeeping is removed. -- Configuration and command migration can be documented and verified as one - coherent transition. - -The added work is principally CLI composition and contract migration: dispatch, -flag parsing, configuration selection, reporting, and process-status mapping. -Choose their native forms deliberately and document the transition, rather than -spending migration effort recreating existing CLI conventions. - -### Ownership And Scope - -| Concern | Selected owner | -| --- | --- | -| Command hosting, standard input/help mechanics, common report formatting and visibility | CommandKit. | -| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. | -| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. | -| Pipeline execution and listing, pipeline-specific preflight, artifact publication, accepted-artifact caching, and domain outcomes | Notarius application handlers and framework. | -| Completion-summary content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. | - -Adopt the standard capability suite selected in item 1, including upstream -commands and recommendations beyond the existing Notarius surface. Items 2–4 -select application inputs, composition, and reporting; items 5–6 select debug -capture, verification, and migration. The earlier extensive receipt/manifest -proposal is superseded by item 4's artifact-first contract. - -Pipeline and domain code should remain independent of CommandKit. Native -PromptKit observations supply execution facts; Notarius attaches application -meaning and selects report content. CommandKit supplies presentation mechanics. - -Two application needs already identified for deliberate treatment are: - -- The current `run --json` publishes a receipt after output publication. Item 4 - replaces that contract with artifact files and a small completion summary, - rendered natively as text, YAML, or JSON. No automatic metadata bundle or - separately persisted report is required. -- Item 5 uses native `--debug` for diagnostic verbosity and for capture when an - effective debug directory is available, including an application default. - Capture selection remains independent of the final reporting mode. - -### Intended Sequencing - -1. Incorporate items 1–2's selected dependency baseline, standard capabilities, - and native application command design, bringing U10 forward. -2. Introduce item 3's CommandKit composition alongside native source assembly. - Connect shared loaders, standard inspection, and the focused application - preflight package. -3. Develop native execution and accounting against the intended CLI boundary, - with pipeline and domain tests independent of CommandKit. -4. Complete command and reporting migration before final end-to-end verification - and migration documentation. - -Item 6 records the incorporation decision. The master roadmap now owns the -resulting six-phase sequence; implementation is pending. - -## 1. Adoption Scope And Dependency Baseline - -**Discussion / decision:** Versions and standard capability scope agreed. Adopt -the set below, including the move to PromptKit v0.23.3. Items 2–3 select -application commands and composition; items 4–5 select reporting and debug -capture principles. These are selected migration targets; the application -dependencies have not changed. - -### Selected Versions - -| Module | Version | Role | -| --- | --- | --- | -| `go.maximumdirect.net/commandkit` | `v0.7.0` | Native command declarations, inputs, help/version, reporting, and completion. | -| `go.maximumdirect.net/ckconfig` | `v0.9.0` | Standard configuration commands and ConfigKit diagnostic classification. | -| `go.maximumdirect.net/ckprompt` | `v0.10.0` | Standard PromptKit inspection, validation, preparation, and smoke diagnostics. | -| `go.maximumdirect.net/configkit` | `v1.2.1` | Shared configuration dependency for the application and integrations. | -| `go.maximumdirect.net/promptkit` | `v0.23.3` | Native model execution, including corrected timeout/cancellation reporting. | - -The CommandKit integration releases form a documented compatible set using -PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout attribution and -cancellation/error identity fixes. It retains the same dependencies and requires -no configuration or request API migration from v0.23.2. The Go consumer minimum -remains 1.27. Verify the assembled dependency set through the migration's normal -integration checks and update attribution when the dependency changes land. - -The version review used locally available release tags and tagged dependency -manifests. Remote release lookup was unavailable, so it did not establish whether -newer tags existed remotely. These are deliberate selected versions, not a claim -that every module is the latest published release. - -Evidence: [integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection) -and [PromptKit v0.23.3 release notes](../../../promptkit/docs/releases/v0.23.3.md). - -### Selected Standard Capabilities - -| Area | Adopt in the cutover | -| --- | --- | -| Core CLI | Native help, `version`, typed flags and operands, supplied-value semantics, standard outcomes, and command declaration/dispatch. | -| Reporting | Native `--format` with text/YAML/JSON, `--quiet`, `--verbose`, `--debug`, and `--progress`, including upstream visibility and failure behavior. | -| Configuration | `config scopes`, `validate`, `show`, `sources`, `explain`, and `diff`, for application and PromptKit configuration. | -| Resource discovery | PromptKit listings for prompts, profiles, backends, namespaces, and collections, with native filters and selectors. | -| Resource inspection | Native prompt, profile, and backend reports, plus requirements discovery. | -| Concrete request checks | Requirements validation and prepared-request preview using native inputs, variables, profiles, and execution overrides. | -| Provider diagnostics | Reachability, authentication, text-inference, structured-output, and aggregate smoke tests. | -| Convenience features | Bash, Zsh, and Fish completion, useful declaration hints, and upstream PromptKit convenience aliases. | - -Use the [upstream CLI conventions](../../../commandkit/docs/cli.md) directly. -Providing the standard capability suite means supplying application loaders and -callbacks, not recreating those commands locally. Upstream convenience aliases -are selected native features; this does not reintroduce legacy Notarius aliases. - -### Wiring Boundaries And Follow-Up - -- Keep configuration validation, requirements discovery, and concrete request - validation distinct. Pipeline preflight remains application-owned. Concrete - preparation may read inputs and schemas and check credentials even though it - makes no model call. -- Install smoke commands for explicit invocation. Their availability does not - introduce provider calls during startup, ordinary validation, or offline tests. -- Enable application configuration overrides through ConfigKit. The selected - PromptKit configuration scope does not support arbitrary `--set` overrides; - use supported request-level controls rather than implementing a local equivalent. -- Retain application handlers for pipeline execution and discovery, with native - commands, inputs, and result presentation as selected in items 2–5. - -The standard capability suite needs no identified upstream addition. The later -cache decision does require a native message-content hash, which is not the -existing v0.23.3 rendered-prompt hash. Confirm the supplying API/release before -implementing cache lookup; the version table remains the reviewed baseline, not -a claim that the new hash is already available. The master roadmap now records -this baseline, the pending upstream requirement, the resolved CommandKit scope, -and the application wiring/presentation decisions. Superseded durable-record -requirements have been removed from that plan. - -## 2. Native Command Surface And Inputs - -**Discussion / decision:** Command structure and input conventions agreed. Use a -small application-owned `pipeline` family alongside the standard `config` and -`promptkit` families. These are migration targets, not implemented commands. - -### Command Structure - -```text -notarius pipeline list -notarius pipeline show -notarius pipeline validate -notarius pipeline run --input-file -notarius run --input-file - -notarius config scopes -notarius config validate|show|sources|explain|diff ... - -notarius promptkit list ... -notarius promptkit show ... -notarius promptkit requirements show|validate ... -notarius promptkit test - -notarius version -notarius completion bash|zsh|fish -notarius help [command...] -``` - -The alternatives separated by `|` above summarize the installed operations; -they are not literal shell commands. - -| Application command | Responsibility | -| --- | --- | -| `pipeline list` | Discover configured pipelines. | -| `pipeline show` | Describe the resolved ordered steps, lanes, modules, reference slots, and configured selections. | -| `pipeline validate` | Perform offline application preflight using the intended selections and overrides. | -| `pipeline run` | Execute the selected pipeline and publish artifacts. | -| `run` | Native convenience alias for `pipeline run`. | - -Move pipeline-specific validation out of `config validate --pipeline ...`. -Configuration validation, pipeline composition checks, and concrete PromptKit -request validation remain distinct. Pipeline preflight does not synthesize stage -requests or require provider credentials. PromptKit's concrete validation may -read inputs and schemas and check credentials without making a model call. - -**Naming decision:** Keep `pipeline run` as the canonical command and install -`run` as a native convenience alias using -`commandkit.Alias("run", "pipeline run")`. Both routes use the same handler, -operands, flags, validation, and reporting. CommandKit supplies alias-aware help -and diagnostics; no forwarding handler or additional parser is needed. - -The grouped family and root shortcut are application usability choices, not an -upstream naming requirement. The alias uses the new input conventions and does -not preserve legacy flags or behavior. Use `notarius run` in quick-start examples -and document both routes in the command reference when the migration lands. -See [native alias behavior](../../../commandkit/docs/cli.md#application-owned-aliases). - -### Input Conventions - -Use a required pipeline-ID operand and a required `--input-file PATH` for runs. -Use native typed declarations and supplied-value semantics, including explicit -empty, false, and zero values. Apply defaults only when the relevant value is -omitted; application validation still decides whether a supplied value is valid. - -| Concern | Selected convention | -| --- | --- | -| Configuration | Native `--config` and `--overlay`; name the application scope `app`. | -| Lane selection | Repeatable `--lane ID`; omission selects the normal pipeline lane set. Replace comma-separated `--only`. | -| Output placement | Application-owned `--output-dir PATH`. | -| Chunk-plan caching | `--chunk-cache auto\|bypass\|refresh`. | -| Accepted-artifact cache | Replace checkpoint-centric controls with cache reuse/bypass/refresh behavior; concrete CLI spelling remains to be selected. `--resume` and `--recompute-step` are migration context, not compatibility requirements. | -| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. | -| Configuration-valued tuning | Supported application `--set POINTER=JSON` overrides, including worker and semantic-budget settings once their configuration design is settled. | - -Adopt native parsing rules: flags follow the complete command path and may be -interspersed with operands; `--` ends option parsing. Singleton flags use the -last successfully parsed occurrence, with every occurrence checked. Repeatable -flags preserve occurrence order and treat commas as ordinary value content. -Use filesystem and choice completion hints where applicable. - -Keep worker counts and semantic continuation budgets in configuration rather -than introducing additional dedicated CLI controls now. Their field names, -defaults, and domain meaning remain master-roadmap decisions. Item 3 selects -shared loading and override wiring across application and standard commands. - -### Native LLM Controls - -Use ckprompt's reusable flag declarations for applicable profile, session, and -execution controls on `pipeline run`, including model, temperature, timeout, -reasoning effort, and transient retries. Run-wide overrides reach all LLM-backed -producers and validators, including continuation calls. Without an override, -normal binding/pipeline/prompt selection applies. - -| Reasoning input | Meaning | -| --- | --- | -| Omitted | Inherit the selected settings. | -| `--llm-reasoning-effort high` | Replace the inherited setting. | -| `--llm-reasoning-effort ""` | Clear the inherited setting. | - -Remove the separate `--clear-reasoning-effort` control. Native transient retries -remain distinct from application semantic continuation budgets. Individual -prompt selection and named prompt input injection belong on standard PromptKit -commands; pipeline modules construct their own requests and response contracts. -Use native `--llm-profile` and `--llm-session-id` spellings, without inventing -another LLM input parser. - -### Application Reference Inputs - -Retain repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`. -Keep useful pipeline-wide, lane, chunk, and stage-specific selectors, such as -`party=party.txt` and `npc-registry.extract.party=npc-party.txt`. CommandKit owns -repeatable input parsing and typed conversion; Notarius owns selector expansion, -specificity, required slots, and external/generated-reference conflict checks. -These selectors describe pipeline relationships and are not interchangeable -with PromptKit's individual-request `--llm-input-file` bindings. - -### Preparing Pipeline Inspection Content - -**Selected design:** Use ConfigKit for the composed, decoded -configuration, defaults, imports, overlay, inheritance, overrides, and native -provenance/redacted inspection. Feed that loaded configuration through the same -application resolution used by validation and execution to obtain selected -lanes, registered modules, validator chains, and reference relationships. -CommandKit renders a small application report of that resolved composition. - -ConfigKit does not know Notarius's module catalog or pipeline semantics. Reuse -the existing resolver rather than writing a second resolution path for `show`. -Generic effective configuration and source explanations remain available through -`config show` and `config explain`. ConfigKit's safe inspection facilities do not -automatically redact an arbitrary application report; select appropriate report -content rather than dumping the raw typed configuration. Item 3 selects loader -wiring; item 4 selects the artifact-first output and completion-report contract. - -### Removals And Remaining Work - -Retire custom dispatch, argument-count handling, assembled help, version-command -parsing, standard configuration/LLM flag parsing, comma-separated lane parsing, -and the separate reasoning-clearing machinery. Replace `config effective` with -native `config show`, `config profile` with `promptkit show profile`, and bespoke -`--json` switches with `--format json`. - -Application handlers still validate meaningful input combinations and translate -typed inputs into pipeline selections. Native `--debug` supplies reporting -behavior and conditional artifact capture as selected in item 5. -Items 3–4 select loader behavior, the small completion report, and native outcome -presentation. The master roadmap limits U9 to implementation details within -these decisions. - -Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax), -[reusable PromptKit flags](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading), -[ConfigKit loading and inspection](../../../ConfigKit/README.md), and the current -[Notarius configuration boundaries](../internal/configuration.md). -Use the current [CLI reference](../cli.md) for migration guidance, not as a -compatibility constraint. - -## 3. Application Composition, Loaders, And Preflight - -**Discussion / decision:** Agreed. Fully adopt native ConfigKit, PromptKit, and -CommandKit workflows through a small composition root, shared loaders, and -invocation-local state. Follow substantially the same composition/loading pattern -as Audita while retaining Notarius's pipeline, artifact, validation, recovery, -and publication responsibilities. Centralize application prompt/profile -dependency checks in a focused `internal/preflight` package. Implementation is -pending. - -### Native Application Composition - -One application constructor installs the pipeline declarations and native `run` -alias, a ckconfig suite with `app` and `promptkit` scopes, the selected ckprompt -capabilities, and native help/version/completion/reporting. Supply ordinary -callbacks and typed inputs; do not introduce another CLI framework or LLM wrapper. - -Constructing the application registers declarations without loading configuration, -building an engine, or initializing run storage. Help, version, and completion -remain independent of runtime configuration and provider work. Application -handlers acquire only what their operation requires. - -### Shared Configuration And Engine Loaders - -Use two shared loading responsibilities across application and standard commands: - -1. **Application configuration:** Select explicit `--config`, then the existing - environment/discovery policy; apply native `--overlay` and supported - application `--set` overrides. ConfigKit owns composition, imports, - inheritance, presence, provenance, and inspection. Supply application - defaults, normalization, and structural validation through its extension - points. Retain the native ConfigKit result and selected main-source context; - derive runtime values where needed without copying inspection machinery into - another result abstraction. -2. **PromptKit engine:** Build a native `*promptkit.Engine` from that loaded - application's source context and the agreed native source assembly. Runtime, - preflight, and inspection use the same builder and definitions. - -Preserve supplied-value information throughout loading. An explicitly empty -configuration path receives deliberate validation rather than silently falling -through to discovery. Across application and PromptKit commands, `--config` -selects the Notarius application configuration; the PromptKit scope inspects the -engine assembled for that application selection. - -Retain the master roadmap's [source assembly](promptkit.md#configuration-sources-and-prompt-assets): -PromptKit's bundled underlay, embedded Notarius resources, then optional operator -companions beside the main application source actually selected by ConfigKit. -Use native imports and filesystem contexts. There is no second source search or -working-directory fallback; strict composition, missing-companion handling, and -load-error behavior remain as selected in the master roadmap. Do not reinterpret -the application document as a standalone PromptKit configuration or add another -inspection-only assembly path. - -Application `--set` overrides flow through ConfigKit's load. Native request -execution overrides remain request controls; they do not mutate captured engine -configuration. Preserve the selected PromptKit configuration scope's lack of -arbitrary `--set` support. - -### Invocation State And Engine Ownership - -Use CommandKit's supported invocation state to retain configuration and an engine -when callbacks need them repeatedly. Application handlers use the same loader -functions with small invocation-owned state. One pipeline execution shares one -native engine across producers, validators, and continuations, preserving native -backend admission limits. - -Do not assume one invocation always means one configuration selection. Native -`config diff` acquires left and right selections with potentially different -configuration, overlays, and overrides while sharing invocation state. Load the -sides independently or distinguish selections in any local reuse. A single -unqualified cached result would be incorrect. No persistent loader cache or -cryptographic configuration identity is needed. - -### Shared Pipeline Resolution And Command Work - -Reuse the existing application resolver with loaded configuration, selected -pipeline/lanes/reference changes, applicable overrides, and the module catalog. -It owns module/options validation, typed artifact compatibility, validator-chain -selection, ordered dependencies, and reference-binding compatibility. - -| Command | Required application work | -| --- | --- | -| `pipeline list` | Load application configuration and enumerate pipelines. | -| `pipeline show` | Resolve the selected pipeline and report its composition. | -| `pipeline validate` | Resolve the pipeline and check its known native resource dependencies offline. | -| `pipeline run` / `run` | Use the same resolution and preflight checks, then prepare and execute the pipeline. | -| `config … --scope app` | Supply the native ConfigKit result directly to the integration. | -| PromptKit inspection commands | Acquire the shared engine assembly and delegate to native inspection. | - -Run handlers call shared application functions directly; they do not invoke the -validation command or parse its output. `pipeline show` can describe configured -prompt/profile selections without constructing an engine just to expand them -into execution settings or performing every execution-readiness check. - -### Prompt Dependencies And Offline Preflight - -Notarius knows which operations need prompts; PromptKit knows whether those -definitions resolve in the assembled engine. Check every statically known prompt -dependency of the selected pipeline, including LLM-backed validators and any -separately selected correction prompts. A deterministic stage may require none, -an LLM-backed stage may require several, and multiple stages may share one. -Configuration can select the required set. For data-dependent selections, check -what is knowable before execution and resolve the final choice at runtime. - -Share prompt identifiers and selection logic with execution through module or -operation declarations. Do not maintain a separate hardcoded preflight inventory. -These small dependency declarations identify required resources; they do not -restore asset manifests, prompt filesystem registration, message assembly, or -schema registries. Native sources remain the owners of prompt definitions. - -Use the same engine and intended namespaces as execution to inspect known prompts -and applicable profile selections, including a known prompt's default profile -when effective. Partial profiles alone are not failures. Report missing -dependencies with pipeline, step/lane, stage or validator, and resource context -where applicable. - -Offline preflight does not read the transcript, materialize reference contents, -synthesize stage requests, require credential values, or contact providers. -Engine assembly may read configuration and prompt resources. Availability does -not certify prompt domain correctness or guarantee that future requests pass -schema, credential, or provider checks. Concrete `Prepare`/`ValidateRequest` -remains the boundary for completed request validation, and smoke commands remain -explicit provider operations. - -### Dedicated Preflight Package - -Add a small application-level `internal/preflight` package accepting the resolved -pipeline, relevant module declarations, and the native engine. It checks the -pipeline's known prompt/profile dependencies and returns contextual findings or -errors for the command boundary to present. - -Keep configuration validation in ConfigKit/application definitions, structural -resolution in the pipeline resolver, and concrete preparation in PromptKit. -Preflight does not discover configuration, register production modules, construct -engines, prepare synthetic requests, initialize storage, or own CLI reporting. -The composition root supplies its collaborators. Module dependency declarations -remain in existing module contracts; modules do not import the preflight package. -Exact function signatures and finding representation remain implementation choices. - -### Request Construction, Storage, And Removals - -Use native request builders directly for standard PromptKit commands where -sufficient. Add an application callback only for a demonstrated application need, -preserving supplied operator values. Pipeline modules construct their own domain -requests; share concrete request-building functions when useful without -simulating pipeline execution to populate an inspection command. - -Output publication, artifact caching, debug capture, and execution records belong -to the run path. Loaders do not initialize those resources; item 5 selects debug -allocation timing. Remove command-specific loading paths, copied configuration -and resource inspection, eager runtime setup, and duplicated preflight -orchestration. No broader application/package reorganization is required by this -decision. - -Evidence: [native suite and loader ownership](../../../commandkit/docs/consumers/guide.md#configkit-scopes), -[PromptKit integration callbacks](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading), -[Audita composition](../../../audita/internal/cli/app_wiring.go), and the -[existing configuration ownership decision](../adr/0020-delegate-configuration-mechanics-to-configkit.md). - -## 4. Reports, Machine Output, And Outcomes - -**Discussion / decision:** Agreed. Deliver artifacts as files and use CommandKit -to render a small structured completion report on stdout, with a native text -template for the ordinary successful case. Additional operational diagnostics -use stderr; detailed investigation uses optional debug capture. This supersedes -the earlier extensive receipt, manifest, and durable-execution-record proposals. -Keep native outcome and delivery-failure handling. Implementation is pending. - -### Accepted-Artifact Cache And PromptKit Ownership - -**Agreed follow-up:** Rename and replace the checkpoint subsystem with the -[accepted-artifact cache](promptkit-architecture-discussion.md#5-accepted-artifact-cache). -That decision owns the matching and storage policy; this section records its -CommandKit and execution-boundary implications. - -The cached value is an accepted answer to the original task. Compute the key -from the initial prepared request using a proposed upstream `MessageContentHash` -over ordered message roles and exact text, excluding settings, session/execution -IDs, timestamps, and provider cache hints. Do not hash the entire prepared -request or introduce another Notarius-owned fingerprint policy. - -Look up `cache/.json` before model execution. A hit loads the final accepted -Notarius artifact directly, subject only to ordinary decoding/usability checks. -It does not execute a model request, reconstruct a result/exchange, replay -corrections, or repeat semantic validation merely because the artifact was cached. - -On a miss, execute the complete Notarius operation: native PromptKit calls, -domain interpretation and validation, and any application-directed correction -passes. Only after final acceptance save the artifact under the initial hash, -even if correction required several turns. For normalization, cache the accepted -normalized artifact after applying the plan, not the model's proposed plan. -Rejected, failed, and validation-incomplete work is not published to the cache. - -PromptKit owns structural response validation, internal retries, repairs, and -other mechanics inside each request execution. Notarius consumes a usable -response or terminal failure at the public boundary and owns domain acceptance. -Internal PromptKit attempts do not determine cache eligibility. Native debug -evidence about those attempts does not transfer their ownership to Notarius. - -The upstream requirement is message-content hashing. It does not require native -cached-response injection, replay, exchange restoration, or write-through -response caching. A small Notarius filesystem lookup/write facility stores the -accepted artifact without checkpoint manifests, dependency fingerprints, -acceptance ledgers, or required execution provenance. A cache hit is not a new -model execution or a claim of validation under a changed policy. - -Concrete cache flags and legacy-checkpoint handling remain implementation-planning -choices. The output/report discussion must not reintroduce cache metadata solely -to populate a report. The master roadmap now incorporates this replacement policy. - -### Artifacts Are The Deliverable - -Publish the accepted domain artifacts as files in the output directory. Their -contents and encoding are independent of CLI report formatting: `--format` -controls the completion summary, not the artifact payloads. Keep artifact -discovery understandable without requiring consumers to read execution metadata. - -Do not require a run manifest, metadata bundle, or persisted completion receipt -alongside the artifacts. Preserve information intrinsic to the domain artifacts -and their existing schemas; do not add model, retry, validation-history, or -execution inventories to those payloads merely because it is available. -Detailed operational evidence belongs in item 5's optional debug bundle. - -### Small Structured Completion Report And Native Text Template - -Submit one small application value through `Reporter.Report`. Use these fields -for an ordinary successful run: - -| Field | Meaning | -| --- | --- | -| `pipeline` | Selected pipeline identifier. | -| `artifact_count` | Number of deliverable artifacts produced, including reused artifacts; excludes internal intermediate results, cache entries, and debug files. | -| `output_directory` | Absolute path to the directory containing the published artifacts. | -| `debug_directory` | Optional absolute path to the completed run-specific debug bundle; omitted when capture is disabled. | - -For example, without debug capture: - -```json -{ - "pipeline": "dnd-session", - "artifact_count": 6, - "output_directory": "/work/results/run-123" -} -``` - -Supply a short native `text/template` to render that same value as: - -```text -Success: dnd-session produced 6 artifacts in /work/results/run-123 -``` - -CommandKit automatically renders a string value as plain text. Here the value -is structured so JSON/YAML callers receive named fields instead of a success -sentence they must parse. The native text template provides the concise human -presentation; without it, a structured value would default to YAML. Use native -`--format text|yaml|json` selection and serializers, without separate encoders or -independent human/machine report construction. - -When a completed run includes rejected work or permitted incomplete validation, -qualify the completion message clearly. Include the rejection count and/or -incomplete-validation fact in the same structured value when applicable. Do not -present unqualified success that conceals those outcomes, or make their visibility -depend on verbose/debug mode. These are small outcome qualifications, not -per-producer histories or a general metadata envelope. - -No report-version field, run-ID field, token totals, profile inventory, warning -statistics, validation-summary collection, or durable execution ledger is required -by this completion contract. The old receipt is migration context, not a shape -to preserve. Standard configuration and PromptKit commands retain their native -report values and behavior without a universal Notarius wrapper. - -### Optional Machine Output And Saved Reports - -Operators and subprocess callers select YAML or JSON through the native format -flag. The completion report is enough to find the artifacts and recognize any -qualified outcome. Consumers interested only in artifacts need no metadata file. - -Save a report with ordinary shell redirection: - -```sh -notarius run dnd-session --input-file transcript.json --format json > run-report.json -``` - -Do not add a separate report-file flag or automatically persist a duplicate report -for this design. Keep stdout as one completion report, with additional diagnostics -on stderr, so `--format json --verbose` still produces valid report JSON. - -### Publication, Streams, And Machine Consumption - -The run command executes the pipeline, publishes the output bundle, completes -required run finalization, submits its successful report, and returns a native -outcome. The report describes an actual published result. Item 5 requires -requested capture to finalize successfully before a successful run report is -submitted and defines how capture failures preserve publication facts. - -Use one primary run-result report, with progress and diagnostics separate. Wire -CommandKit's result destination to stdout and its diagnostic destination to -stderr. Subprocess consumers request `--format json`, capture streams separately, -check the exit status, and decode the successful run report. A report-write -failure may leave partial stdout even though publication already succeeded. -These run-consumption rules do not replace the native report behavior of other -commands, including configuration commands that can report failed validation. - -### Native Visibility, Diagnostics, And Debug Location - -Wire CommandKit's result destination to stdout and diagnostic destination to -stderr. Use `Info`, `Warn`, `Debug`, `Progress`, and native outcome explanations -for their respective categories; do not mix these messages into a JSON/YAML -completion report. - -| Mode | Stdout | Stderr | -| --- | --- | --- | -| Ordinary | Brief completion summary. | Failure explanation, if needed. | -| `--verbose` | Same completion report. | Additional operational information, warnings, and any failure explanation. | -| `--quiet` | Nothing. | Failure explanation, if needed. | -| `--debug` | Completion report, including capture location when enabled. | Native debug diagnostics, information, warnings, progress, and any failure explanation. | - -Adopt native reporting-mode ordering, explicit progress selection, and failure -visibility. The table describes mode defaults; `--progress` can explicitly enable -progress outside quiet mode. Machine callers expecting a report do not request -`--quiet`. Quiet suppression is not a delivery failure; native rendering errors -can still fail a quiet invocation. Artifact-quality advisories keep their domain -classification rather than becoming process warnings solely for visibility. - -When capture completes, the text template adds a line such as: - -```text -Debug capture: /tmp/notarius-debug/run-123 -``` - -Use `debug_directory` in the structured value for the same location. Provide the -path rather than mirroring the debug bundle to stdout. An explicitly empty debug -directory preserves native debug diagnostics but creates no capture or capture -path in the completion report. Item 5 owns capture controls and lifecycle. - -### Native Outcomes And Domain Acceptance - -Adopt the standard process-status conventions: - -| Status | Meaning | -| --- | --- | -| `0` | The command completed successfully. | -| `1` | Operational failure, cancellation, publication failure, or presentation failure. | -| `2` | Invalid invocation or identifiable invalid configuration input. | -| `3` | Native `config diff --check` found differences. | - -Use ckconfig's structured configuration-error classification. Do not classify -every configuration failure as operational or every preflight failure as invalid -input. Invalid operator selections and missing bundled application resources can -have different causes; preserve that distinction and native error identity. - -Command completion remains separate from domain acceptance. A successfully -published run can contain recorded rejections or validation-incomplete results -allowed by application policy; those facts qualify the completion report and do -not automatically fail the command. A terminal pipeline failure does fail the -command. Native status zero also does not replace reading standard PromptKit -reports for partial discovery or unsupported smoke checks. - -Application handlers supply useful native outcome explanations and retain the -underlying error through native cause handling. CommandKit presents the terminal -result. On failure, provide a useful explanation and nonzero status rather than -requiring a separate structured error-report protocol. `--format` controls the -completion report, not native diagnostics. If artifacts were published before a -later failure, include their location in the failure explanation. Remove duplicate -terminal-error printing and custom outcome wrappers. - -### Delivery Failures, Progress, And Accounting - -Use CommandKit's native separation of operation causes from output and diagnostic -delivery failures. If publication succeeds but stdout fails, the final command -status reflects presentation failure while the published artifacts remain a -fact. Do not rewrite pipeline history as failed execution or automatically rerun -model work because report delivery failed. - -Notarius owns stage/lane/chunk progress, domain acceptance, and cache reuse. -PromptKit supplies native execution observations and `ExecutionGroup` totals. -These facts support native operational diagnostics and optional debug capture; -they do not require a permanent consumer execution report. Explicitly enroll the -relevant initial, continuation, and validator executions without double counting. -Cache reuse does not add historical token usage to current execution totals, and -unavailable usage remains distinguishable from zero. - -Remove bespoke format selection, parallel receipt encoders, custom verbosity -handling, duplicate execution accounting, and superseded terminal-reporting -machinery. Retire automatic reporting-only metadata bundles, mandatory run -manifests, per-producer provenance inventories, and durable execution-ledger -requirements. Do not preserve or rebuild those structures to populate the small -completion value. Use existing application outcome facts for its contents and -the debug overview. Artifact schemas remain domain contracts; detailed debug -representations remain diagnostic data. - -Evidence: [native reporting](../../../commandkit/docs/consumers/guide.md#invocation-reporting), -[visibility and delivery](../../../commandkit/docs/cli.md#reporting), and -[outcome classification](../../../commandkit/docs/cli.md#outcome-status). -The current [run-result contract](../integrations/run-result.md) and -[CLI failure mapping](../internal/cli.md#failure-mapping-and-terminal-reporting) -document migration impact. Item 6 of the -[PromptKit discussion record](promptkit-architecture-discussion.md#6-inspection-and-execution-bookkeeping) -retains the native bookkeeping boundary. The master roadmap now incorporates -this contract and retires superseded durable-record requirements. - -## 5. Debug Capture And Optional Diagnostics - -**Discussion / decision:** Controls, native capture hooks, lifecycle, failure -policy, and optional diagnostics agreed. Use `--debug` for native verbosity and -durable capture with `/tmp/notarius-debug` as the default root. Operator-owned -data handling and convenient diagnostic representation are agreed. Replace the -summary/trace hierarchy with a compact run index, configuration and composition -snapshots, and operation records; maintain a `latest` symlink at the debug root. -The capture inventory and depth below are agreed: one default depth retaining -every application-visible turn, useful inputs and outputs, and application -assessments, with native snapshots and convenient readable views. An explicitly -empty debug directory disables capture while preserving native reporting-mode -selection, resolving verbosity without capture. Exact filenames and serialized -application fields remain implementation choices. Implementation is pending. - -### Shared Debug Control With Conditional Capture - -Attach a native application boolean binding to CommandKit's shared `--debug` -option on the pipeline run command. Capture is enabled when that boolean is true -and an effective debug directory is available. `--debug-dir PATH` overrides -`debug.directory` in application configuration, whose default is -`/tmp/notarius-debug`. A directory alone does not enable capture. Do not add a -separate `--capture-debug` flag. - -**Default-directory decision:** Use the absolute root `/tmp/notarius-debug`, -with each run's bundle beneath it. Temporary diagnostic output belongs under -`/tmp` or an operator-selected location under `/var`, rather than beside a -configuration file likely to live under `/etc`. Replace the old configuration- -relative `./notarius-debug` default. Precedence is CLI override, configured value, -then application default. Nonempty relative configuration paths retain their -source-relative interpretation; nonempty relative CLI paths use the invocation -directory. - -Omitting both directory overrides selects the default, so `--debug` enables -verbosity and capture without requiring a path. - -**Explicit-empty decision:** An explicitly empty `--debug-dir ""` or -`debug.directory: ""` disables capture while preserving native debug verbosity -and reporting-mode ordering. Omission inherits the configured value or default; -an explicit empty value is meaningful and must not fall back to either. CLI -selection overrides configuration, including when the CLI value is empty. A -nonempty CLI directory can therefore re-enable capture over an empty configured -directory when `--debug` is enabled. No additional flag is needed for verbosity -without capture. - -Recognize an empty value before resolving relative paths so it cannot become the -invocation or configuration directory. Disabled capture creates no bundle and -leaves `latest` untouched. A nonempty but unusable destination follows the -requested-capture failure policy when capture is enabled; it does not silently -disable capture or select a fallback directory. - -Determine capture from the application boolean and destination, not from -`Invocation.ReportingMode()`. A later quiet or verbose selection can change -presentation without clearing the debug input. Explicit `--debug=false` follows -native boolean binding semantics and disables capture. - -| Configured directory | Invocation flags | Reporting selection | Capture | -| --- | --- | --- | --- | -| Any | No `--debug`, with any directory selection | Selected native mode | None. | -| Omitted | `--debug` | Debug | `/tmp/notarius-debug`. | -| `/var/tmp/notarius` | `--debug` | Debug | `/var/tmp/notarius`. | -| Any | `--debug --debug-dir ""` | Debug | None. | -| `""` | `--debug` | Debug | None. | -| `""` | `--debug --debug-dir /tmp/investigation` | Debug | `/tmp/investigation`. | -| Omitted | `--debug --quiet` | Quiet | `/tmp/notarius-debug`. | -| Omitted | `--quiet --debug` | Debug | `/tmp/notarius-debug`. | -| Any | `--debug --quiet --debug-dir ""` | Quiet | None. | - -**Upstream ordering clarification:** CommandKit uses the last enabled reporting -mode. Thus `--debug --quiet` suppresses command output while retaining capture -when the effective directory is nonempty; `--quiet --debug` selects debug -verbosity. Preserve that native ordering rather than making quiet unconditional. -Quiet still shows failure explanations, and capture does not bypass result -suppression. On success the normal run report includes the bundle location when -capture was enabled; failure explanations include an allocated bundle path when -available. - -### Native Evidence And A Small Application Recorder - -Use native facilities directly, attaching application step/lane/chunk/validator -context where useful: - -| Facility | Available evidence | -| --- | --- | -| `OnPrepared` | Prepared request details when execution reaches that point. | -| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. | -| Returned `RunResult` | Available model output and native validation result. | -| `Exchange.Details()` | Explicit inspection of captured conversation history. | -| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and cache reuse. | - -Compose capture and accounting observers. Remove the debug completion wrapper, -copied execution summaries, synthetic call identities, and duplicate terminal -reporting. Keep callbacks lightweight and synchronize shared capture state: -callbacks are synchronous, can overlap across operations, and `OnPrepared` runs -while admission is held. Retain needed details and persist them at normal -operation boundaries without requiring an entire pipeline's content in memory. -Native snapshots may repeat conversation prefixes when that is the convenient -representation; do not introduce a custom delta format merely to remove them. - -Capture only available evidence. Preparation/admission failures may have no -`OnPrepared` notification; observer events have metadata, not response bodies; -operational failures may return no `RunResult`. Do not present the bundle as a -complete provider transcript. Use ordinary `Run` with native hooks, without -forcing split preparation/execution solely for recording. The cache's lookup -before execution is a separate reason to use native preparation; `OnPrepared` -notifications alone do not provide cache short-circuiting. - -Conversation capture required for continuation remains in-process execution -state. Persisting its history requires enabled debug capture. Cached values are -accepted domain artifacts, not conversation state or provenance records. The -separate artifact-first output and completion-report contract is selected in item 4. - -### Capture Lifecycle And Failure Policy - -For enabled capture, allocate the bundle after application configuration loads -and the destination and run identity are known, before engine construction and -pipeline preflight. Argument and configuration-load failures use native -diagnostics and may have no bundle. Loaders remain free of storage allocation. - -After creating the run directory, update `/latest` to a relative -symlink targeting that run ID before engine construction or preflight starts. -This points to the newly started debug session, including one that later fails; -it is not a last-successful-run selector. Runs without capture leave it alone. -Do not update it again at completion, so an older overlapping run cannot take -the link back when it finishes. Concurrent starts use the last successful link -update; individual run directories remain the actual storage locations. -Treat link-update failures under the requested-capture failure policy below. - -One application-owned recorder records initial context, accepts native execution -evidence and application decisions, and finalizes once with the known pipeline, -publication, and capture outcomes. Debug remains separate from output/cache, -never supplies cache reuse, and is not automatically deleted. - -Requested capture is required work: - -- Allocation failure stops before expensive model work. -- Recording failure is surfaced at the next normal application boundary. -- Finalization failure is an operational command failure. -- An existing pipeline failure remains primary; capture failures are additional - information and preserve the underlying causes. -- Already-published artifacts and already-written diagnostic data remain - available. Capture failure does not cause model retries. - -Observer callbacks cannot return errors. Retain a recording error for application -code to check at an ordinary boundary rather than adding an exception or retry -framework. Submit the successful run report only after publication and requested -capture finalization succeed. If finalization fails after publication, identify -the published output and available debug location in the native failure -explanation; preserve the successful publication as a fact. - -### Debug Data And Operator Responsibility - -Debug artifacts are expected to contain internal application data, including -source material, rendered requests, model output, and correction history where -useful for diagnosis. If the application data is sensitive, the debug artifacts -are sensitive in the same way. Operators secure them through the ordinary means -used for application inputs and outputs, such as filesystem permissions. Do not -require a special debug-only security or retention model, or use sensitivity -concerns to limit useful diagnostic content. - -Do not directly print or serialize values explicitly marked as secrets by -ConfigKit, such as resolved API keys. Use native secret-aware configuration -inspection and preserve upstream credential omissions. This boundary does not -authorize scanning or rewriting LLM output, prompts, source content, or arbitrary -strings for suspected sensitive values. Do not carry forward heuristic content -scrubbing or generic secret-detection machinery as a migration requirement. - -Separate debug/output/cache lifecycles and no automatic debug deletion remain -application organization choices. Choose capture contents for diagnostic value, -size, and duplication, rather than promising a shareable or sanitized bundle. - -### Native Optional Diagnostics - -Use the selected standard commands through the shared loaders and native inputs: - -| Command family | Purpose | -| --- | --- | -| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. | -| `pipeline validate` | Check application composition and known dependencies offline. | -| `promptkit requirements show` | Discover request requirements. | -| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. | -| `promptkit test …` | Explicit provider diagnostics. | - -Application request builders are added only for actual application needs. -Inspection/diagnostic commands use their native reports and do not automatically -allocate pipeline debug bundles; operators can redirect reports for retention. -Neither debug verbosity nor enabled capture initiates additional provider probes. -Smoke reports retain native support, applicability, and result distinctions; -status zero alone does not establish that a check passed. - -### Selected Bundle Organization - -**Guiding principle:** Capture the most convenient debug representation. Optimize -for straightforward capture and useful inspection, rather than maximum -conciseness, completeness, or deduplication. Native values, complete snapshots, -and self-contained operation records are appropriate even when fields or content -repeat. - -"Record each fact once and reference it where needed" remains a practical -guideline for avoiding independently maintained decisions and accounting, not a -strict storage constraint. Use shared application outcome facts and native -execution evidence. Repeating that evidence in useful representations does not -require another source of truth or justify custom compression/reconstruction. - -Organize the bundle around the run and its operations, replacing `summary/` and -`trace/` as separate categories. Use a compact overview for navigation and keep -detailed evidence with its operation. Summary/trace are not security classes. - -Illustrative layout: `latest` is the selected link name; other file names and -exact serialized fields remain implementation choices: - -```text -/ - latest -> / - / - run.json - config/ - application.yml - promptkit.yml - pipeline.json - source.json - operations/ - / - ... -``` - -| Component | Purpose | -| --- | --- | -| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. | -| Configuration snapshots | Preserve effective settings through native inspection. | -| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. | -| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. | -| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. | - -The overview provides navigation, with terminal facts drawn from the same -application outcome used for the CLI report. Reference detailed records where -useful; include convenient summary fields without maintaining a second execution -ledger. Configuration and resolved composition normally need one snapshot each. -Use native execution IDs with application scope to connect records. -Reused work has a loaded-artifact snapshot and current cache-hit decision, not -a fabricated model execution. Historical provenance is not required in cache -files or reconstructed solely for debug capture. - -Retain enough available context to investigate a run that fails before publishing -output. Do not depend entirely on successfully published artifacts or automatically -copy every original input and published artifact. Retain complete native -conversation snapshots for successive turns as selected below. Do not build a -competing transcript model or deduplication system to eliminate repeated history. - -### Selected Capture Inventory And Depth - -**Resolved:** Use one default capture depth preserving every application-visible -turn, its useful inputs and outputs, and the decisions made about it. An operator -should be able to inspect a failed operation without reconstructing deltas, -finding a cache entry, or extracting model text from escaped JSON strings. -Convenience governs both capture and inspection; direct native serialization -and a simple readable rendering of the same values can both be useful. - -#### Run Context - -Retain the run overview, native application and PromptKit configuration -snapshots, resolved pipeline, and parsed source identified above. Include -application version, run/pipeline identity, timing, outcome, publication location, -and navigation to operation records in the overview. Record effective invocation -selections so request overrides remain visible even when they do not change -engine configuration. - -The parsed source snapshot retains the generic document's unit identities and -content independently of the original input file. Do not automatically archive -every original configuration, template, or input file; effective snapshots and -executed requests supply the default evidence. If input parsing fails, retain -the input location and available parsing diagnostics; a parsed-source snapshot -may not exist. - -#### Operation Inputs - -An operation is meaningful application work: chunk planning, extraction for one -lane/chunk, merging, normalization, or validation. Identify its stage, lane, -chunk, role, and relationships to other operations as applicable. Retain an input -snapshot containing the actual values consumed: - -- Relevant source/chunk content. -- Loaded reference content used by the operation. -- Incoming artifacts. -- Fixed original candidates and candidate-handle mappings for normalization. -- Prompt input bindings and application variables where applicable. - -Include this content even when it also appears elsewhere in the bundle. An -extraction operation should expose its source and reference context directly. -Use existing application values and artifact codecs at meaningful boundaries; -do not invent a parallel representation of the pipeline's entire internal state. - -#### Model Executions And Conversations - -For each initial execution, continuation, and LLM validator execution, retain -the following evidence when available: - -| Evidence | Representation | -| --- | --- | -| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. | -| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. | -| Returned result | Native `RunResult` snapshot. | -| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. | -| Conversation | Full `Exchange.Details()` snapshot. | -| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. | - -Use indented JSON for structured native snapshots and existing artifact formats -for domain payloads. Readable messages are a straightforward rendering of native -data, not an independently maintained transcript model. Use the completed -exchange when available and prepared messages when execution fails before -returning a result; label which representation is shown. - -Keep full conversation snapshots for successive turns, including rejected turns. -Repeated history is acceptable and makes a later turn independently inspectable. -Enable native exchange capture for debug-enabled executions where needed, -including single-turn calls. Continuation's in-process exchange requirements -remain independent of whether durable debug capture is enabled. - -#### Interpretation, Assessment, And Stage Artifacts - -Retain beside each response: - -- The interpreted candidate, serialized through the existing candidate codec - before semantic acceptance. -- Decode, interpretation, or safety-check failures when no candidate could be - constructed. -- Each configured validator's result, findings, and actionable correction guidance. -- The application's disposition: accepted, rejected, continued, failed, or - accepted under the selected fallback/incomplete-validation policy. -- Links to LLM validator executions that informed the assessment. - -For normalization, retain the model's reconciliation plan and the candidate -produced by applying it to the fixed original inputs. Preserve constructed -correction inputs even if the next execution fails during preparation; successful -preparation captures the actual rendered correction in its native messages. - -Retain deterministic stage outputs too: chunk plans and materialized chunks, -merged artifacts, and normalized artifacts. For cache reuse, retain the loaded -artifact and current hit decision without requiring historical provenance. These -records do not fabricate a current execution or new validation of reused work. - -#### Failed Runs And Native Availability - -Persist evidence as operations advance under the selected recorder lifecycle; -do not wait for final publication. On failure, retain available evidence and -record where execution stopped, the returned error, and the application outcome. -Distinguish no returned result from a returned result that failed validation. -A completed response returned through coordination's `Undelivered` field is -captured as unassessed, without implying acceptance. - -Internal transport retries and structural repairs receive metadata capture at -the depth PromptKit exposes. Intermediate response bodies are not required by -this inventory. Operational errors may return no `RunResult`; record that -absence clearly. This scope does not require a provider wrapper or an upstream -feature request. Evaluate a later demonstrated gap on its actual diagnostic need. - -Keep selected content in full, without arbitrary truncation or a second -capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and -native credential omissions; do not scan or scrub application content. Exact -filenames and serialized application fields remain implementation choices; -the inventory and capture depth are resolved. - -Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting), -[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility), -[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run), -and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation). -The current [debug lifecycle](../internal/state.md#debug-bundles) and -[operator contract](../operations.md#debug-bundles) provide migration context. - -## 6. Verification, Migration, And Roadmap Incorporation - -**Discussion / decision:** Agreed. Complete one coordinated native cutover, with -CommandKit composition introduced early, verification focused on Notarius's -integration and domain responsibilities, and practical release migration guidance. -These decisions are incorporated into the existing master roadmap; no replacement -document was needed. Implementation is pending. - -### Focused Verification Of Application Behavior - -Use real CommandKit, ConfigKit, and PromptKit components in representative -assembled workflows, with a fake at the provider boundary. Keep the default -suite deterministic, offline, and independent of real credentials. Following the -[testing policy](../policy/testing.md), assign each behavior an appropriate test -owner and avoid duplicating upstream parser, renderer, or retry coverage. - -Select a small set of workflows that collectively establish: - -- Execution, inspection, and preflight resolve the same intended configuration - and prompt definitions, including application overrides and companion sources. -- Help, version, and completion work without application configuration; offline - preflight makes no provider calls. -- `run` and `pipeline run` reach the same application behavior, with the selected - flags, supplied-value distinctions, and overrides mapped correctly. -- Reports, diagnostics, and exit outcomes follow item 4, distinguishing domain - rejection, command failure, and presentation failure after publication. -- A downstream failure followed by another invocation reuses accepted artifacts - under the revised cache policy. Final corrected artifacts are stored under the - initial message hash; changed messages miss, settings-only changes can hit, - and hits neither make model calls nor repeat domain validation. Rejected or - validation-incomplete work is never stored as an accepted cache value. -- Debug capture follows item 5's directory selection, reporting-mode independence, - `latest` update timing, useful evidence, and allocation/write/finalization - failure behavior. - -Use subprocess tests where the actual executable boundary adds protection: -stdout/stderr separation, process exit status, and representative machine output. -Test domain interpretation, validation, continuation, and recovery at their -appropriate package boundaries. Preserve meaningful native-execution acceptance -checks from the master roadmap without repeating them throughout CLI tests. - -### Retire Tests With Superseded Machinery - -Review tests of the private adapter, global scheduler, manual continuation -protocol, broad checkpoint fingerprints, and superseded CLI contracts for removal -as those implementations retire. Where an application requirement survives, test -that requirement through its new boundary. Do not mechanically reproduce old -parser, formatter, or wrapper tests against the new implementation. - -Avoid freezing complete upstream help output, incidental diagnostic wording, or -every field of native debug snapshots. Debug verification should establish useful -evidence and correct failure handling while allowing representations to evolve -for convenience. Do not require exhaustive provider-attempt reconstruction or -strict serialization deduplication. - -Reconcile older debug-sensitivity language, special-handling requirements, and -heuristic redaction behavior/tests with item 5. Avoid directly disclosing values -marked as secrets by ConfigKit and preserve native omissions. Do not carry forward -heuristic scrubbing of prompts, source content, or model output, or special -debug-data handling requirements. Operators secure debug artifacts as they do -ordinary application inputs and outputs. - -### Migration Guidance And Documentation - -Provide release guidance for operators and subprocess callers, with a practical -old-to-new migration map covering: - -- Commands and flags, including the native `run` alias, configuration inspection, - and `--format`. -- Application and PromptKit configuration changes, source discovery, profiles, - and companion files. -- Native backend admission, independent worker settings, and revised retry controls. -- Accepted-artifact cache reuse, explicit refresh/bypass, and the selected - treatment of old checkpoint files. -- Machine reports, exit outcomes, and the distinction between successful command - execution and accepted domain results. -- Debug capture under `/tmp/notarius-debug`, `latest`, and the interaction between - capture and reporting modes. - -CLI backward compatibility remains a non-goal; migration guidance does not imply -compatibility shims or preservation of old command behavior. Legacy checkpoint -handling and published-artifact compatibility remain explicit application -decisions, without assuming that old checkpoint envelopes must survive. - -Maintain runnable examples for embedded defaults and operator overrides. Update -canonical CLI, configuration, operations, integration/consumer, and internal -documentation as the corresponding behavior lands. Record major architectural -changes in successor ADRs, preserving accepted decision text under the -[documentation policy](../policy/documentation.md#architecture-decision-records). -Keep future behavior in roadmap documents until implemented. - -### Incorporation Into The Master Roadmap - -**Completed:** The [master roadmap](promptkit.md#implementation-sequence) now -integrates CommandKit throughout six phases, beginning with native composition -and shared loaders, proving a representative pipeline through the new CLI, and -finishing cache/output/capture integration before final verification. The old -separate Phase 7 is retired. - -Its dependency baseline includes the selected versions and pending upstream hash -request. U10 is resolved; U8 now concerns cache operation/transition, and U9 -concerns implementation details within the selected output/debug contracts. -Earlier receipt/manifest inventories are retired, and U2 resolves asset/schema -assembly. Remaining correction, fallback, worker, and cache-transition choices -are explicit in its -[unresolved-choice table](promptkit.md#unresolved-choices). - -The master roadmap owns sequencing and implementation status. This notepad and -the PromptKit discussion remain discussion records, not competing implementation -checklists. - -### Completion Checks - -Use focused checks while implementing and the existing repository-wide -`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` checks for the -completed cutover. Check the affected maintained examples, documentation links, -dependency attribution, and removal inventory as part of this migration. - -Live provider smoke tests remain explicitly optional and outside the deterministic -default suite. These change-specific migration checks do not create additional -permanent release gates; publication continues to follow the -[release policy](../policy/release.md). Recording these decisions does not claim -that implementation acceptance checks have already passed. - -## Evidence And Reading - -These sources support the discussion and selected decisions; they do not -constitute a completed implementation audit. Sibling links assume the local -repositories remain available. - -- [Notarius CLI implementation overview](../internal/cli.md) and - [configuration internals](../internal/configuration.md). -- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md), including - module selection, typed inputs, raw/formatted output, reporting, and loaders. -- [ckconfig integration](../../../commandkit/integrations/ckconfig/README.md) and - [ckprompt integration](../../../commandkit/integrations/ckprompt/README.md). -- [Audita command composition](../../../audita/internal/cli/app_wiring.go), showing - application handlers and upstream integrations in one CommandKit application. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 91f666eb..e3798d18 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -1,30 +1,18 @@ # Future Work Current Notarius behavior is documented in the canonical README, CLI, -configuration, operations, internal, and integration docs. This roadmap records -future work only. Items are ordered roughly by current value and specificity, -not as committed release dates. - -## Near-Term Validation And LLM Reliability - -PromptKit now owns structural output repair within one completion. Notarius -owns stage candidates, validator chains, semantic rejection policy, bounded -feedback-aware stage retries, validation provenance, and reusable-state -eligibility, and the separation of actionable process warnings from quality -diagnostics. The remaining near-term work applies those completed foundations -to domain review and empirical evaluation. - -Near-term reliability work should now be selected from the concrete evaluation -and extension opportunities below. The retry, validation, and subprocess -foundations described above are implemented current behavior. +configuration, operations, internal, and integration docs. This backlog records +future work outside the [native migration roadmap](promptkit.md), which owns +PromptKit/ConfigKit/CommandKit adoption, continuation, admission, cache, output, +and debug decisions. Items below are ordered roughly by value and specificity, +not as committed release dates or prerequisites for that migration. ## Near-Term D&D Pipeline ### Evaluate Spell Extraction And Normalization -- Evaluate ordinary extraction retries and the completed normalization path - against a human-reviewed transcript set before and after adopting the shared - PromptKit repair and Notarius validation-retry policies above. +- Evaluate extraction, normalization, and correction quality against a + human-reviewed transcript set before and after the native migration. - Maintain a small set of human-reviewed transcripts and outputs for prompt, validator, and normalizer development. Treat model-quality review as an iterative human evaluation aid, not a deterministic correctness gate. @@ -46,8 +34,8 @@ The sections below keep broader extensions deferred. ### Large-Collection Semantic Reconciliation - Evaluate deterministic candidate blocking only after representative registry - inputs exceed the active roadmap's bounded single-request limits. Blocking - should use cheap, explainable signals to form plausible comparison sets while + inputs exceed practical single-request capacity. Blocking should use cheap, + explainable signals to form plausible comparison sets while preserving the possibility that a duplicate appears outside a lexical name match. - Define correctness for candidates that appear in more than one block, @@ -61,16 +49,15 @@ The sections below keep broader extensions deferred. ### Operator-Selected Semantic Policies -- Consider allowing an operator to select an approved semantic-policy prompt - for a typed reconciliation module without replacing the shared protocol, - response schema, or deterministic safety rules. -- Define the trusted asset source, configuration syntax, compatibility checks, - startup validation, provenance, prompt fingerprinting, checkpoint effects, - and support boundary before exposing the option. -- Prefer selection among registered, typed-policy-compatible prompt assets over - arbitrary filesystem prompt paths. Do not add this flexibility until an - operator workflow requires it; artifact-family-owned policy remains simpler - and safer for the initial implementation. +- Consider a dedicated semantic-policy selection option for a typed + reconciliation module when a concrete operator workflow needs more than the + migration's [native prompt replacement](promptkit.md#configuration-sources-and-prompt-assets). +- Use native prompt IDs and sources while retaining typed interpretation and + deterministic safety rules. Define any additional configuration and domain + compatibility checks from that workflow; do not recreate asset registration, + prompt fingerprints, or checkpoint dependency tracking. +- Reuse the selected [accepted-artifact cache policy](promptkit.md#accepted-artifact-cache) + when evaluating policy changes and the need for explicit refresh. ### Broader Reconciliation Inputs And Module Selection @@ -89,9 +76,8 @@ The sections below keep broader extensions deferred. artifact. - Add production LLM-backed validators only when a concrete review policy benefits from model judgment and deterministic checks are insufficient. -- Add validator diagnostics and timing summaries if operators need more detail - than the current [durable output bundle](../integrations/json-output.md) - provides. +- Consider additional validator diagnostic views only if a concrete need remains + after the selected [native observations and debug capture](promptkit.md#inspection-and-execution-bookkeeping). - Add validator compatibility metadata if deployments need config-time proof that a validator is suitable for a particular stage, module, or artifact kind. @@ -133,10 +119,11 @@ occurred. Define compatibility and migration policy before generated-reference chains must span multiple schema versions or long-lived historical artifacts. The -policy should address stable identifier semantics, which schema changes permit -checkpoint reuse, when an older artifact may be decoded or adapted, and when a -producer or all dependents must be recomputed. Do not add a general migration -framework until an actual contract change requires one. +policy should address stable identifier semantics, when an older artifact may +be decoded or adapted, and when consumers need regenerated artifacts. For cached +artifacts, use the [selected reuse policy](promptkit.md#accepted-artifact-cache) +and explicit refresh where needed. Do not add a general migration framework +until an actual contract change requires one. ### Artifact-family-oriented physical packaging @@ -170,7 +157,6 @@ section only after a concrete workflow, contract, and priority emerge. ### Workspace And Storage -- Default-idempotent run behavior with an explicit force override. - Remote workspace storage. - Workspace garbage collection and archival policies. -- Cross-machine checkpoint reuse. +- Cross-machine sharing of accepted-artifact cache entries. diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md deleted file mode 100644 index 74a48608..00000000 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ /dev/null @@ -1,842 +0,0 @@ -# PromptKit Architecture Discussion Record - -## Purpose And Status - -The decisions in items 1–6, the subsequent CommandKit discussion, and the -simplified cache/output designs are incorporated into the -[master native migration roadmap](promptkit.md). It owns the selected target, -unresolved choices, sequencing, acceptance criteria, and implementation status. - -The [CommandKit discussion record](commandkit-migration-discussion.md) retains -the related CLI, preflight, output, debug, and migration rationale. The master -roadmap replaces earlier checkpoint matching/provenance proposals with the -accepted-artifact cache and retires mandatory manifests, extensive receipts, -and durable execution ledgers. The upstream message-content hash remains pending; -incorporation does not claim it is already available or implemented. - -This document retains the discussion, rationale, and evidence, including item 7's -completed Audita comparison. Item-level questions and follow-ups below record -the discussion as it developed; they are not a second implementation plan. - -The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit -capabilities as fully as practical. Application ownership of a policy does not -necessarily require application ownership of its implementation. Evaluate -remaining machinery against this goal, including opportunities for upstream -features where PromptKit does not yet meet an application need. - -No application changes have been implemented through this discussion or the -roadmap rewrite. Significant changes to accepted architecture need successor ADRs -under the [documentation policy](../policy/documentation.md#architecture-decision-records). -Current boundaries are described in [Architecture](../policy/architecture.md) -and [LLM Runtime Internals](../internal/llm.md). - -## 1. LLM API And Diagnostic Boundary - -**Discussion / decision:** Direction agreed: remove the Notarius PromptKit -adapter and compose the application directly from native PromptKit execution, -typed artifact operations, and pipeline orchestration. A small application -function connects generation to configured validation and acceptance. The design -below incorporates items 2–6 and the Audita review; exact signatures and migration -details remain open. No application changes have been implemented. - -### Background And Intent - -Notarius and related applications, including Audita, originally developed their -own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by -extracting their duplicated mechanics into a shared library. Notarius adopted -an early version and predates many of the library's modern capabilities. - -The maintainer clarified that duplication of PromptKit capabilities, including -slight behavioral differences, does not reflect an intentional requirement for -an independent Notarius implementation. Existing wrappers, tests, and historical -decisions are migration evidence, not sufficient reasons to preserve that -duplication. Identify actual application needs before choosing what to retain. - -Missing reusable capabilities may justify upstream feature requests. The -discussion is not constrained to what v0.23.2 can replace immediately; any -resulting change to the adoption target remains to be selected. - -### Application Structure - -| Part | Responsibility | -| --- | --- | -| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. | -| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. | -| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. | -| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. | -| Application records and CLI | Store artifacts and accepted-artifact cache entries, attach pipeline meaning to execution facts, and present results through CommandKit. | - -Engine construction becomes ordinary application wiring using item 2's embedded -defaults and operator sources. Profile selection is expressed through native -requests. U3 now places shared loaders and engine construction in `internal/cli` -and the producer helper beside pipeline assessment. Retire the emptied -`internal/framework/llm` package rather than retaining a constructor-only package. -The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns -the resolved design; implementation is pending. - -Use native requests, results, exchanges, errors, and observers directly in -model-executing application code. Pure domain models, codecs, source-reference -resolution, and deterministic transformations need no PromptKit dependency. -Retain application interfaces for extracting artifacts, validating typed -candidates, encoding output, and loading accepted cached artifacts. Deterministic -stages continue to operate without conversation machinery. - -### Connect Modules To Pipeline Assessment - -The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go) -prepares inputs, invokes the private completion interface, interprets a response, -and returns a copied raw model candidate to the -[producer retry loop](../../internal/framework/pipeline/producer_attempts.go). -Replace that arrangement with direct collaboration between module interpretation, -pipeline assessment, and native execution: - -1. The module prepares source material, references, and any candidate-handle - mappings once, and constructs the native initial request. -2. Application execution calls `Engine.Run`, capturing an exchange when - continuation is enabled. -3. The module interprets the structurally valid response into a typed candidate - or actionable domain findings. -4. The pipeline runs configured candidate validation and decides whether to - accept, request correction, or apply terminal policy. -5. If correction is needed, native `Coordinate` invokes the same interpretation - and assessment for revised responses. The module supplies the repair request; - PromptKit maintains history and enforces the turn budget. - -The resolved U3 design uses a small shared helper in `internal/framework/pipeline`. -Modules supply native initial/correction requests and response interpretation; -the pipeline supplies an operation-scoped assessment callback and relevant -cache/debug collaborators. Local variables or closures hold operation state. -Use the same assessor for initial and revised candidates without an additional -final pass. Native preparation supports cache lookup before execution; hits load -the accepted artifact directly. The helper returns a typed application outcome -and saves only final accepted artifacts under the initial hash. - -Keep deterministic stages on their ordinary typed production/validation path. -LLM validators call the shared engine directly and return findings without an -automatic producer correction loop. No replacement completion API, execution-plan -language, or copied conversation model is selected. - -Item 3 owns the detailed continuation and assessment rules, including final-turn -assessment, failure handling, complete replacement responses, and safe fallback. -For normalization, a revised reconciliation plan is applied to the original -inputs to produce a fresh candidate. LLM-backed validators can call the same -native engine and return application validation findings; they do not need the -producer's conversation machinery. - -### Removal, Records, And Testing - -Retire `StructuredLLMClient`, the private structured-completion request/response -family, adapter conversions, and opaque correction transport. Items 3, 4, and 6 -specify removal of the outer LLM retry loop, scheduler, debug completion wrapper, -profile-recording plumbing, and duplicate execution accounting. Do not recreate -that wrapper stack around native calls. - -Attach pipeline scope through native callbacks and observers. Use native -`ExecutionGroup` totals for operational diagnostics and optional debug artifacts, -following item 6. The small completion report follows the -[CommandKit output contract](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes). -Item 5 now selects a simple accepted-artifact cache, without required durable -acceptance or execution-provenance records. - -Audita demonstrates shared engine assembly, native continuation, and CommandKit -integration. Its `RoleClient`, structured-output interface, and opaque exchange -wrapper are not part of this design; item 7 records the comparison's evidence -and limits. - -Test domain interpretation and assessment directly. Exercise model workflows -with a real PromptKit engine and its injectable client or a fake HTTP provider. -Pipeline orchestration can still use fake stage implementations. Testing does -not require preserving a private completion interface. - -### Remaining Implementation Work - -- Implement the resolved U3 ownership and placement; refine function signatures - and argument grouping against representative extraction and normalization. - These implementation details do not reopen the selected boundaries. -- Implement item 2's resolved asset/schema design and settle the remaining - migration details recorded under items 3–6, including durable compatibility. -- Evaluate any remaining upstream gap against an actual application requirement, - rather than preserving historical machinery by default. - -**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6. -Record the significant boundary change in successor ADRs. Update architecture -and implementation documentation when the change lands, explicitly replacing -the current transport-neutral completion boundary with PromptKit as the execution -dependency of application workflows. Keep domain operations and application -outcomes as the meaningful Notarius-owned boundaries. - -## 2. Prompt Assets And Schema Assembly - -**Discussion / decision:** Agreed and incorporated as resolved U2 in the -[master roadmap](promptkit.md#prompt-assets-and-response-schemas-u2-resolved). -The filesystem and schema questions initially deferred here are now settled. -Implementation is pending; the master roadmap owns the detailed design and checks. - -Use ordinary embedded assets, native prompt declarations, and native schema -loading. Keep the content tree and module ownership of prompt meaning and domain -interpretation. Shared files use real native filesystem paths. Remove duplicate -asset registration, per-module manifests, synthetic mounts, flattened schema -namespaces, and the custom read-only filesystem that served that assembly. - -Prompt definitions select output contracts by default. Initial and correction -prompts can share schema files. Use native per-request replacement only for a -demonstrated runtime-dependent contract, without an application wrapper or schema -registry. Audita demonstrates ordinary embedded schema loading and per-call -contracts; its per-call selection is not required for every Notarius operation. - -The default schema source is the embedded asset tree. Operator-selected external -schema directories use native whole-root replacement, not an application overlay. -Prompt underlays do not implicitly overlay schema files. PromptKit owns private -response-schema loading, structural checks, and repair; Notarius keeps typed -interpretation, domain validation, and distinct durable artifact schemas/codecs. - -Retire response-schema hashes and copied metadata used only by the adapter, -checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic -snapshots. Neither asset nor schema metadata adds another cache fingerprint. - -The spell extractor illustrated the duplication: prompt YAML declared ordered -messages and a schema while a Go manifest separately listed, mounted, registered, -and hashed assets. Native declarations now supply the selected assembly model. - -**Follow-up:** Implement the master roadmap's Phase 2 and retire tests of removed -mounting/registry machinery. Verify shared paths, operator replacements, native -schema sources, and any necessary complete request-contract replacement. - -## 3. Retry And Semantic Correction Ownership - -**Discussion / decision:** Direction agreed: adopt Audita's native continuation -approach. PromptKit owns captured exchanges, accumulated conversation history, -continuation execution, and turn-budget enforcement. Notarius supplies semantic -assessment, corrective findings, and acceptance or fallback decisions. Use native -PromptKit types directly, consistent with item 1; Audita's adapter and opaque -exchange wrapper are not part of the selected direction. - -The flow and removal/adaptation direction below are agreed. Begin with one -conversation per producer operation and complete replacement model responses. -Retain application assessment, safe-fallback decisions, artifact validation, -and cache eligibility. Detailed response contracts, budget migration, and -the open policy questions below remain to be resolved. No application changes -have been implemented. - -### Agreed Flow - -For cacheable operations, item 5's initial preparation and lookup precede this -flow. The steps below describe execution on a miss. - -1. Prepare the source context, references, and any request-local candidate-handle - mapping once for the producer operation. Run its initial native request with - exchange capture when continuation is enabled. Capture is independent of - detailed debug recording. -2. Handle the native call's terminal success or failure, decode a usable response, - and perform the module's deterministic interpretation and safety checks. Run - the configured validation chain on a materialized candidate. A module can - identify actionable defects before a candidate is ready for that chain. -3. If correction is needed, call `Engine.Coordinate` with the initial exchange - and a configured `AdditionalTurns` budget. The callback selects a native - repair prompt and supplies actionable findings as variables. PromptKit retains - the original exchange and each subsequent repair turn. -4. Assess every delivered response, including the last permitted turn. Rebuild - the typed candidate from the fixed operation inputs and the revised response; - do not repeatedly apply revisions to an already modified artifact. A changed - candidate receives the complete configured validation chain again, including - validators that approved an earlier version. -5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides - whether to reject, fail, or evaluate an explicitly supported safe fallback. - Native `stopped` and `exhausted` outcomes describe coordination, not acceptance. - Cancellation never turns an undelivered response into an accepted candidate. - -Start with one conversation per producer operation: a chunk-plan generation, -one lane's extraction of one chunk, or one lane's normalization pass. This aligns -with Notarius's whole-output validation boundary. Audita's per-proposal branches -are useful evidence, but per-entity repair would require its own combination and -validation design. It is not necessary to adopt native continuation. - -Repair prompts should supply a new user task, with enduring principles in the -original system instructions. Avoid replaying the initial task or embedding -initial-response-only constraints in system messages. Begin with complete -replacement model responses; for normalization, the response can remain a -reconciliation plan that Notarius applies to construct the durable artifact. -A different repair schema or profile is supported but not yet selected. Keep -source context and candidate-handle meanings stable throughout the conversation. -Accumulated history increases context and token use across turns; the semantic -turn budget should reflect that tradeoff. - -### Machinery To Remove Or Adapt - -| Current machinery | Agreed direction | -| --- | --- | -| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. | -| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. | -| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. | -| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. | -| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. | -| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. | -| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. | -| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. | - -Native transient retries, structural repair, and semantic additional turns remain -distinct controls, but Notarius need not implement all three loops. Existing -producer and validator `retries` settings cannot be mechanically renamed: their -current budgets cover different behavior. Non-LLM failure handling should be -reviewed separately rather than routed through conversation coordination. - -### Validation And Safe Fallback Still Have Application Meaning - -Keep source grounding, reference resolution, domain invariants, reconciliation -plan checks, typed application rules, and candidate isolation. In particular, -normalization must still reject unknown or overlapping candidate handles and -apply only safe groups. A final fallback can retain safe merges while leaving -unsafe groups separate; it must go through configured artifact validation and -terminal policy before it can advance. After a native terminal failure covered -by the selected fallback policy, a supported deterministic fallback can be assessed -directly instead of first -restarting the same model operation through a feedback-free module retry. Which -earlier safe candidate, if any, may survive a later failed turn remains an explicit -policy question; do not introduce implicit best-so-far acceptance. - -Retain the distinction between a rejection and unavailable validation. Under the -current `warn_continue` policy, incomplete validation can advance a structurally -valid candidate but prevents accepted-artifact cache reuse/publication, including -for derived results. Continuation adoption does not itself change that policy -or safe-fallback eligibility. Keep raw conversation state in process and -out of cached artifacts; explicit detailed debug remains separate. Item 5 -supersedes checkpoint-specific publication and provenance machinery. - -PromptKit owns everything within an individual request execution, including -structural validation, internal retries, and repairs. Notarius consumes the -public result or failure and owns domain interpretation, validation, and any -application-directed correction passes. Do not inspect or reproduce internal -PromptKit attempts to decide cache eligibility; cache only the final accepted -Notarius artifact after its complete application process succeeds. - -PromptKit should own validation and repair of the private model response. -Notarius's serialized schema validator currently receives the codec-produced -artifact and its artifact schema, which can differ from the model response. -Do not remove that boundary merely because both use JSON Schema. Audit actual -duplicate response checks individually; artifact validation and Go/domain -interpretation are not automatically redundant. Item 2's resolved native asset -and schema design preserves this distinction. - -Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go), -[validation executor](../../internal/framework/pipeline/validation_executor.go), -[typed artifact validation](../../internal/framework/pipeline/runner_typed.go), -[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go), -[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go), -and the native continuation evidence in item 7. - -### Open Implementation And Policy Questions - -- Exact repair prompt definitions, response schemas, and profile selection within - the agreed complete-replacement approach. -- Migration of producer and validator `retries` settings to native controls, - including configuration names, defaults, and compatibility. -- Detailed terminal/fallback handling, including whether an earlier safe - candidate may survive a later failed turn. -- Failure handling for non-LLM operations and any justified recovery beyond - native execution budgets. -- Item 5's remaining cache implementation details and migration to the selected - CommandKit output contract. - -**Follow-up:** Resolve the remaining details above, then reconcile the adoption -roadmap. Record the changed architecture in a successor to -[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and -[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). -Migration checks should cover accumulated history, fixed handle mappings, -revalidation, final-turn acceptance, native budget exhaustion, safe fallback, -validator failure, cancellation, and cache eligibility. Replace tests of -the retired message-reconstruction protocol rather than preserving its mechanics. - -## 4. Global Scheduling And Admission - -**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and -retire the separate cross-backend ceiling. Rely on PromptKit's native backend -admission and capacity limits while keeping bounded Notarius pipeline workers. -This deliberately changes the existing global-limit contract; reproducing that -ceiling in PromptKit is not a migration prerequisite. No application changes -have been implemented. - -### Agreed Ownership - -| Concern | Owner | -| --- | --- | -| Bounded extraction jobs and lane continuations | Notarius pipeline workers | -| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit | -| Application disposition after admission or execution failure | Notarius application policy | - -For limited backends, PromptKit v0.23.2 bounds active model attempts by -`concurrency_limit` and admitted operations by that limit plus `queue_capacity`. -An operation holds admission -across preparation, retries, and structural repair, while active permits cover -individual model attempts and are released during retry waits. A completed -continuation turn releases admission before application assessment, so its -callback can invoke an LLM validator using the same backend. - -The current Notarius wrapper holds a global permit around the complete -structured call, including backend waiting and repairs. Removing it eliminates -that additional queue and avoids making independent backends share those -permits. Independent native backend limits may allow aggregate concurrency above -the former global ceiling; that is an accepted behavior change. Pipeline worker -bounds remain application work controls, not a replacement global LLM ceiling. - -### Removal And Configuration Consequences - -- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom - queue/cancellation/permit bookkeeping. -- Retire `concurrency.total_llm`. Give pipeline worker settings an independent - default and remove the extraction-worker maximum derived from `total_llm`. -- Use one shared execution engine for producers, validators, and repair turns. - Native capacity belongs to an engine and selected backend ID; separate engine - instances have independent pools. -- Configure native limits at the intended backend capacity boundary. Profiles - selecting the same backend ID share its pool; separate IDs pointing to the - same endpoint do not automatically share capacity. Endpoint-only profiles and - backends without a positive limit remain unrestricted by backend admission. -- Handle native `ErrCapacityExceeded` as an admission failure through application - policy, consistently with item 3's removal of blanket execution retries. -- Retire scheduler implementation tests with the removed code. Retain focused - integration checks for shared native admission across roles, independent - backends, bounded pipeline work, queue exhaustion, and cancellation. - -### Remaining Details - -Select independent worker defaults, native backend/queue defaults where the -application supplies them, and the configuration migration treatment of the -retired field. Specify how admission failures are reported and handled for each -application role. These details do not reopen the selected ownership boundary. - -A future demonstrated need for capacity shared across several backend IDs could -justify an upstream PromptKit feature request. No such feature is required for -this agreed design, and it does not justify retaining the Notarius scheduler. - -Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go), -[scheduler](../../internal/framework/llm/scheduler.go), -[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and -[native capacity behavior](../../../promptkit/docs/internal/capacity.md). - -**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language -and record the changed ownership in an ADR. Update architecture, configuration, -operations, and implementation documentation when the migration lands. Continue -with item 5's accepted-artifact cache discussion. - -## 5. Accepted-Artifact Cache - -**Discussion / decision:** Agreed. Rename and replace the checkpoint subsystem -with a simple cache surrounding the complete Notarius producer operation. -Compute its key early from the initial prepared request's message contents; -store its value late, after Notarius has accepted the final artifact. This -supersedes both comprehensive execution fingerprints and the earlier -stage/input/prompt-ID matching proposal. Implementation is pending. - -### Purpose And Key - -The cached value is an accepted answer to the original task. After a downstream -failure, rerunning the pipeline can reuse that artifact without repeating the -expensive model work that produced it. Cheap deterministic work elsewhere in -the pipeline does not need a general stage-checkpoint framework. - -Use a PromptKit-provided `MessageContentHash` (proposed name) over the initial -request's ordered messages, including each message's role and exact text. Exclude -model settings, session IDs, execution IDs, timestamps, and provider cache hints. -Do not add whole-configuration, stage, prompt-ID, library-version, or validation- -policy fingerprints to this matching rule. - -Changing the actual initial messages changes the key. Changing a prompt ID, -model, or setting without changing those messages does not. Fixing deterministic -interpretation or validation may therefore require explicit refresh/bypass to -replace a previously accepted artifact; a hit does not claim fresh validation. - -This is a concrete upstream hash requirement. The reviewed PromptKit v0.23.3 -`RenderedPromptHash` includes session identity and provider cache hints, while -hashing the entire `PreparedRun` would include volatile execution metadata. -Neither is the selected key. Expose the content hash through native preparation -rather than implementing another message-hashing policy in Notarius. Confirm -the supplying upstream API/release before cache implementation; do not claim -the proposed field already exists or silently substitute a broader fingerprint. - -### Compute Early, Store After Acceptance - -```text -Prepare initial request -> obtain message-content hash -> look up cache/.json - Hit: decode the accepted Notarius artifact and continue the pipeline - Miss: execute the complete Notarius operation - -> native PromptKit calls as needed - -> domain interpretation, validation, and correction passes - -> on final acceptance, save the artifact under the initial hash -``` - -Lookup occurs before model execution. Use native preparation to inspect the -initial request; preserve that prepared request for execution on a miss where -the native API supports it. This is a reason to prepare before deciding whether -to execute, distinct from preparing solely to obtain logging details. - -Retain the initial hash throughout application-directed correction. Save the -final accepted artifact under that hash, not under a later correction request's -hash. A subsequent run can then reuse the accepted answer without replaying the -failed turns that led to it. - -The value is the final Notarius artifact, serialized through its existing codec. -For normalization, it is the accepted normalized artifact after plan application, -not the model's proposed reconciliation plan. Do not store an intermediate raw -response, `RunResult`, exchange, or prepared-execution handle as the cache value. - -Only publish after the whole Notarius process succeeds, including configured -domain validation and any application-directed retries/correction passes. -Rejected, failed, and validation-incomplete results are not cacheable. Retain -the existing exclusion of derivatives of validation-incomplete work without -restoring checkpoint manifests or fingerprint graphs to implement it. - -A hit loads the accepted artifact directly. Ordinary decoding and usability -checks apply; absent or unusable entries are misses. Do not execute a model -request, reconstruct native results/exchanges, replay corrections, or repeat -semantic validation merely because the value came from cache. Do not count -historical model work as current execution. - -### PromptKit Ownership Boundary - -PromptKit owns the mechanics within each request execution, including structural -response validation, internal retries, and repairs. Notarius handles a usable -response or terminal failure at the public call boundary, then applies its own -domain interpretation, validation, correction, and acceptance process. - -Internal PromptKit attempts are not inputs to the cache design. Native debug -observations may expose evidence about them, but Notarius does not inspect or -reproduce those mechanics to determine cache eligibility. The upstream addition -needed here is the message-content hash; no cached-response injection, replay, -exchange restoration, or PromptKit-owned write-through cache is required. - -### Storage And Removal - -Use a small filesystem lookup/write facility with `cache/.json` containing -the accepted artifact. Ordinary temporary-file/rename publication suffices to -avoid exposing partial entries. Cache lookup must not require original output -or debug directories, acceptance ledgers, execution provenance, or a separate -versioned checkpoint envelope. Debug capture can record the current hit and -loaded artifact without requiring historical metadata in the cache file. - -Retire checkpoint-specific identities, invocation-wide lookup barriers, -stage/dependency fingerprints, manifests, and recovery bookkeeping. Do not -preserve that machinery solely to cache cheap deterministic stages. The existing -standalone canonical chunk-plan cache remains a separate concern. - -The old `--resume`/`--recompute-step` conventions are migration context, not a -compatibility requirement. Select straightforward cache reuse/bypass/refresh -controls and legacy-file handling during implementation planning; do not infer -new flag spellings or a migration framework from this decision. - -**Follow-up:** The master roadmap now includes this design and the upstream hash -requirement. During implementation, verify that a downstream failure leaves -earlier accepted artifacts reusable, corrected answers are stored under the -initial hash, changed messages miss, settings-only changes can hit, and a hit makes no model calls or -new semantic-validation claims. Keep output/report contract decisions separate. - -## 6. Inspection And Execution Bookkeeping - -**Discussion / decision:** Direction agreed: use native PromptKit inspection -and execution facts, with CommandKit owning standardized operator presentation. -Remove duplicate LLM reporting and accounting machinery. Notarius retains -application context, domain decisions, artifact delivery, completion-summary -content, and optional debug capture. The CommandKit output decision below removes -mandatory durable execution provenance. No application changes have been -implemented. - -### Ownership - -| Responsibility | Owner | -| --- | --- | -| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. | -| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. | -| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. | -| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. | -| Acceptance, rejection, fallback, accepted-artifact reuse, and pipeline outcome | Notarius. | -| Artifact storage, completion-summary content, and optional debug storage | Notarius. | - -### Native Inspection And Operator Presentation - -Remove the profile-inspector wrapper, copied effective-model-parameter types, -and local reconstruction of upstream configuration. Use native inspection for -configured resources and native prepared/result data for the target actually -executed, including request overrides. Profile inspection can describe a partial -configured profile; it is not evidence of the completed execution target. - -Adopt ckconfig and ckprompt for standard inspection, requirements, validation, -preview, and opt-in smoke commands. Notarius supplies configuration/engine -loaders and application request builders. Avoid building temporary custom -commands or report machinery that the planned CommandKit adoption would replace. - -Retain application pipeline preflight: resolve modules, artifact compatibility, -references, and known prompt/profile selections. Use native definition checks -without simulating stage execution or requiring provider credentials. Concrete -request preparation remains the place to check the completed target, inputs, -schema, and credential requirements. - -CommandKit supplies formatting, visibility, progress presentation, and delivery -error handling. Notarius supplies report content, domain outcomes, and exit-status -mapping. Follow the small structured completion report and native text template -selected in [CommandKit item 4](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes); -do not preserve the old receipt shape or independent encoders. - -### Native Execution Facts - -- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed - for provenance or optional debug capture. Remove unconditional - `PrepareExecution`/`RunPrepared` use solely to obtain those details. -- Use native observations for terminal execution facts, including failures that - return no `RunResult`. `OnPrepared` is conditional and is not delivered after - preparation or admission failure; it cannot serve as a completion record. -- Use native `RunID`/`ParentRunID` with application scope to correlate executions. - Remove the private debug completion wrapper and duplicate synthetic call-ID - machinery where native identity and application attribution suffice. -- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions, - cloning, and independent attempt/usage accumulators. Use native types internally - and project selected fields only at an application reporting or durable boundary. -- Remove the adapter's `LLMProfileRecorder` and provider-forwarding plumbing. - Native snapshots provide the profiles and targets actually used for optional - diagnostics; do not reconstruct a durable profile inventory. -- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the - initial run, continuation runs, and nested validator runs as applicable. - Preserve terminal-coverage and token-usage completeness; unknown usage is not - zero. Do not count both attempt observations and terminal aggregates, or add - overlapping groups as though they were disjoint work. - -Application execution code should attach scope and observers, run and assess the -native result, and record its disposition and execution references. This does -not require a replacement general-purpose LLM interface. A successful native -execution can still produce a domain-rejected candidate; provider success and -application acceptance remain separate facts. - -### Application Records And Durable Boundaries - -Attach the application context needed to explain a pipeline in optional debug -capture: which stage/lane/chunk produced an artifact, which validators assessed -it, whether continuation or a safe fallback was used, and whether the artifact -was generated now or reused. PromptKit cannot infer that meaning from provider -execution alone. These diagnostic facts do not require consumer-facing histories. - -Under the revised item 5, cache files contain accepted artifacts without required -historical execution or acceptance records. A current debug record can identify -reuse and show the loaded artifact without counting earlier token usage as -current work or claiming new validation. - -The selected CommandKit output contract delivers artifacts as files and a small -completion report through native rendering. It does not require a run manifest, -metadata bundle, separately persisted receipt, or durable execution ledger. -Existing [manifest](../integrations/json-output.md) and -[receipt](../integrations/run-result.md) contracts document migration impact; -their field inventories are not requirements to recreate. Preserve intrinsic -domain artifact content and reuse existing outcome facts for the completion -summary and debug overview. - -Keep application debug artifact placement, opt-in capture, and lifecycle under -[CommandKit item 5](commandkit-migration-discussion.md#5-debug-capture-and-optional-diagnostics). -Native callbacks supply execution data; they do not own Notarius's storage policy. -Detailed prepared messages and execution history belong in that optional capture, -not a mandatory consumer metadata bundle. - -Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go), -[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go), -[debug completion wrapper](../../internal/framework/pipeline/debug.go), -[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md), -[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md), -[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and -[CommandKit reporting](../../../commandkit/docs/consumers/guide.md). - -### Remaining Implementation Work - -Select application scope and observer/group enrollment for the agreed diagnostics. -Implement the selected completion report and retire superseded manifest, receipt, -and execution-ledger machinery. Coordinate command migration and debug capture -with the incorporated design in the master roadmap. The core -bookkeeping ownership change needs no new PromptKit capability; any remaining -upstream request should identify a concrete unmet requirement. - -**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the -significant boundary changes in successor ADRs. Update current-behavior -documentation when implemented. Verify failure reporting without a result, -continuation/validator accounting, unknown usage, actual-target provenance, -cache reuse accounting, and optional debug capture during implementation. - -## 7. Compare Audita's PromptKit Workflow - -**Review status:** Completed on 2026-11-18 against the clean sibling checkout -`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`. -Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0, -ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision, -not new Notarius contracts. The source links below assume the sibling checkout. - -### Embedded Defaults And Operator Replacement - -- One captured engine is assembled from PromptKit's default catalog, the - embedded Audita underlay, and a higher-priority operator underlay. - `FilesystemInputs` supplies eleven native prompt definitions in - `collections.audita`: four proposal prompts, four repair prompts, and three - validator prompts. -- Operator sources are optional `prompts.yml` and recursively selected YAML - under `prompts/`, anchored to the application main source's recorded context. - Both locations feed one operator underlay and are peers; conflicting peer - definitions fail rather than receiving filename-based precedence. -- Replacement is by resource identity, not filename. For example, a complete - prompt record with local ID `modules.grammar.repair` under - `collections.audita.prompts` replaces `audita/modules.grammar.repair` while - other embedded definitions remain available. Native same-ID replacement is - whole-record replacement, not recursive merging of individual prompt fields. -- Native imports and message-file context are retained. Tests show that an - existing engine retains captured message content after a file edit and a new - engine sees the edit. They also verify inherited message-file resolution, - main-relative discovery independent of the working directory, and peer - conflict rejection. -- A diagnostic embedded-only construction can classify a loading error; it - does not silently fall back to defaults after an operator-source failure. - -Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go), -[source tests](../../../audita/internal/framework/llm/engine_sources_test.go), -and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go). - -### Prompt Content And Schema Ownership Are Separate - -Audita uses ordinary embedded filesystems for prompt and schema assets. Its -small prompt registry records IDs and embedded paths; it does not reconstruct -Notarius-style per-module virtual prompt trees. Shared instruction fragments are -supplied as template variables. Processing supplies the application-owned values -for those shared names, but an operator replacement decides whether its template -references them at all. They are not forcibly appended to replacement messages. - -The processing request independently supplies a complete native output contract: -JSON Schema validation, the schema path selected for that operation, and its -structural-repair budget. The three schemas describe a correction set, a repair -decision, and validator decisions. PromptKit receives their canonical embedded -filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not -change the processing code's expected response shape. A test replaces a prompt -with a text/no-validation declaration and still exercises the processing -request's JSON-schema repair contract. - -Audita still keeps a response-schema key/metadata/hash registry and application -decoding checks. Its use of native loading does not prove that every schema -wrapper is necessary, nor decide Notarius's deferred schema design. It supplies -a concrete option: module-owned response requirements expressed directly through -native per-request contracts. Standalone native preview/preparation can use the -prompt declaration's contract; processing supplies its own override. - -Evidence: [prompt registry](../../../audita/internal/prompts/registry.go), -[schema registry](../../../audita/internal/framework/responseschema/registry.go), -`prepareCall` and `OutputContract` in the -[processing adapter](../../../audita/internal/framework/llm/adapter.go), and -`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in -[adapter tests](../../../audita/internal/framework/llm/adapter_test.go). - -### Native Continuation With Application Assessment - -The implemented flow is: - -1. Generate a section's proposal set, capturing a native immutable exchange when - correction repair is enabled. This retains the actual proposal messages and - response; it does not require debug recording. -2. Apply application validation. An actionable rejected proposal starts its own - branch from the original proposal exchange. Unavailable-only findings and - no-effect corrections do not start repair. Existing approvals remain separate. -3. Call `Engine.Coordinate` with that exchange and an additional-turn budget. - Each follow-up selects a distinct repair prompt, repair profile, and repair - response schema. The request carries the original proposal reference, current - correction, ordered findings, and available before/after text previews. -4. PromptKit assembles history and progresses the conversation. A second turn - retains the first repair instruction and response. A sibling branch starts - from the original exchange and does not inherit another branch's repairs. - Audita does not rerender the original proposal or manually replay messages. -5. Audita decodes each delivered decision. Withdrawal ends the branch without - applying that correction. A revision undergoes the full application validation - chain again, including validators that approved its earlier version. Approval - stops the branch; actionable rejection can request another turn. -6. The final permitted response is assessed before exhaustion is determined. - Requesting another turn at that boundary does not allocate another logical - call. Recoverable provider/invalid-output failures finish the branch as - unavailable; fatal errors or parent cancellation interrupt processing. - Canceled, undelivered responses are diagnostic evidence only, never approvals. - -Transport retries and structural repair operate inside each PromptKit execution; -they are separate from the semantic branch's additional-turn budget. Tests show -one semantic repair turn containing multiple provider attempts for structural -repair. The application retains acceptance, withdrawal, application ordering, -and reporting; PromptKit retains conversation and execution mechanics. - -Audita's prompt roles support this model deliberately: system instructions carry -durable principles, the initial user message carries the initial module task, -and the repair prompt supplies a new user assignment. Adopting continuation in -Notarius would require reviewing the complete conversation and response-format -transitions, rather than mechanically appending to prompts written for fresh -replacement requests. - -Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), -[domain repair flow](../../../audita/internal/framework/runner/repair.go), -[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go), -[runner repair tests](../../../audita/internal/framework/runner/repair_test.go), -and [prompt guidance](../../../audita/docs/internal/prompts.md). - -### Other Boundaries And Limits Of The Comparison - -- Audita still has `RoleClient`, a structured-call interface, an opaque exchange - wrapper, and application diagnostic types. Adopt the demonstrated native - mechanisms without assuming Notarius should copy those wrappers; item 1's - adapter-removal direction remains in force. -- Proposal, validator, and repair roles share one engine and its per-backend - admission. The runner separately bounds section work. This demonstrates useful - separation of application work scheduling and provider capacity. Item 4 now - selects that ownership and retires Notarius's independent cross-backend ceiling. -- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations - feed a native `ExecutionGroup`; application records add logical-call and - stage/section/proposal attribution. Nested validator executions enroll - explicitly. Counts preserve terminal coverage and token-usage completeness. -- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt - commands using Audita's engine construction, including request preparation, - validation, and smoke diagnostics. -- Audita edits independently assessable transcript corrections; Notarius accepts - typed stage artifacts and has cross-stage dependencies. Audita branches at an - individual correction within a proposal response. Notarius must deliberately - choose its correction unit rather than assume per-entity branching. -- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's - cache design; item 5 records that application decision independently. - -Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md), -[observations](../../../audita/internal/framework/llm/observations.go), -[CLI integrations](../../../audita/internal/cli/app_wiring.go), and -[component scope](../../../audita/docs/internal/overview.md). - -### Implications For The Remaining Discussion - -Item 2's native underlay direction has a working application precedent. Audita -also offers evidence for ordinary embedded filesystems and per-request schema -selection. U2 now selects ordinary embedded assets and prompt-declared contracts -by default, reserving per-request replacement for demonstrated runtime needs. - -The subsequent item 3 discussion selects native continuation and accumulated -conversation history. Keeping domain validation local does not require keeping -conversation assembly or the turn loop local. Item 3 owns the agreed Notarius -flow and the remaining repair-prompt, response-contract, and policy details; -the comparison alone did not settle those details. - -Item 4 now records agreement to use native admission with bounded application -workers. The revised item 5 selects an accepted-artifact cache and a focused -upstream message-content hash instead of comprehensive execution fingerprints. -Item 6 adopts native execution observations and CommandKit presentation; -the [CommandKit output decision](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes) -selects artifact files and a small native completion report. -The two requested capabilities already work with v0.23.2; no new PromptKit -feature is needed merely to provide replaceable defaults or domain-directed -continuation. Any remaining upstream request should identify the specific unmet -requirement after reconsidering Notarius's historical behavior. - -### Verification - -Read the production composition, adapter, coordinator, domain repair callbacks, -prompt/schema definitions, and focused behavior tests. The following checks -passed from the Audita checkout: - -```sh -go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation -go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1 -``` - -The subprocess checks exercise assembled configuration, real native continuation -messages, operator replacement, final-turn assessment, sibling isolation, -repair-disabled behavior, and cancellation using a local fake provider. These -checks establish implementation behavior, not live-model quality. No live model -calls or application source changes were made. Broader migration decisions and -changes to the adoption roadmap remain for discussion. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 6f266512..218193e0 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -8,13 +8,19 @@ presentation directly. Retain the application work of producing, assessing, publishing, and reusing typed artifacts. Historical duplication of library behavior is not a requirement to preserve it. -This roadmap incorporates the [PromptKit architecture discussion](promptkit-architecture-discussion.md) -and all six items of the [CommandKit discussion](commandkit-migration-discussion.md), -including the subsequent accepted-artifact cache and artifact-first output -decisions. The notepads retain rationale and Audita evidence; this document owns -the selected target, unresolved choices, implementation sequence, removal -inventory, and acceptance criteria. All implementation phases are pending. -Recording these decisions has not changed application behavior. +This is the single migration plan for PromptKit, ConfigKit, CommandKit, +accepted-artifact caching, and artifact-first output. It owns the selected target, +remaining choices, implementation sequence, removal inventory, acceptance +criteria, and supporting evidence. The earlier discussion documents are retired; +Git history preserves the discussion. All implementation phases are pending. +Recording these decisions has not changed application behavior. Unrelated future +work remains in the [separate backlog](future.md). + +Notarius predates many modern PromptKit capabilities: the library grew out of +mechanics shared by Notarius and related applications. Historical duplication +and slight behavioral differences do not express an intentional application +requirement. Evaluate remaining machinery against actual domain needs, and +consider upstream features when a reusable capability is missing. Complete one coordinated cutover for configuration, PromptKit resources, and the CLI. Adopt native public conventions fully; preserving old CLI behavior is a @@ -23,6 +29,10 @@ parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is part of the main migration, not a later follow-up. Release preparation and publication remain separate work. +Adopting the kits together lets standard inspection and execution share the same +loaders and native facts. It avoids building temporary command and reporting +adapters while those boundaries are already being replaced. + ## Baseline And Dependency Target At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application @@ -38,22 +48,27 @@ The selected reviewed baseline is: | `go.maximumdirect.net/ckconfig` | `v0.9.0` | | `go.maximumdirect.net/ckprompt` | `v0.10.0` | | `go.maximumdirect.net/configkit` | `v1.2.1` | -| `go.maximumdirect.net/promptkit` | `v0.23.3`, pending the message-content hash addition below | +| `go.maximumdirect.net/promptkit` | `v0.23.3`, subject to U1's hash-capable release review | Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible -validation dependencies. PromptKit v0.23.3 supersedes the initial v0.23.2 target -for its timeout/cancellation fixes. These are selected versions, not a claim -that they are the latest releases. The local tag review and integration guidance -are recorded in [CommandKit item 1](commandkit-migration-discussion.md#1-adoption-scope-and-dependency-baseline). -The tagged [release notes][release], [migration guide][migration], -[format reference][formats], and [Go consumer guide][consumer] supply the baseline. +validation dependencies. The selected CommandKit integration releases document +compatibility with PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout +attribution and cancellation/error identity fixes; that patch requires no +configuration or request API migration from v0.23.2 and retains its dependencies. +The original review used local release tags and tagged dependency manifests; +remote release lookup was unavailable, so these are selected versions, not a +claim that they are the latest releases. See the +[integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection) +and tagged [release notes][release], [migration guide][migration], +[format reference][formats], and [Go consumer guide][consumer]. -The cache requires the proposed native `MessageContentHash`. The -[upstream request](../../../promptkit/docs/roadmap/notarius-request.md) is pending; -v0.23.3's `RenderedPromptHash` is not a substitute. Confirm the supplying API and -tagged release, then verify the assembled dependency set. Independent composition -and pipeline work can proceed while that addition is pending; the completed -cache must use the upstream capability. Do not add a local hash-policy fork. +The cache requires native `MessageContentHash`; v0.23.3's `RenderedPromptHash` +is not a substitute. The sibling PromptKit repository has retired the original +feature request, and its [v0.24.0 release notes](../../../promptkit/docs/releases/v0.24.0.md) +document the addition and a ConfigKit v1.3.0 dependency. U1 remains open to review +the supplying API/release and verify a compatible assembled dependency set before +changing the selected baseline. Independent work can proceed; completed cache +integration must use the upstream capability. Do not add a local hash-policy fork. Neither a cross-backend global limit nor a comprehensive execution fingerprint is required. Remove obsolete imports and catalog dependencies when unused; avoid permanent `replace` directives and update [attribution](../oss/attribution.md) @@ -480,8 +495,8 @@ Prepare initial request -> obtain MessageContentHash -> look up cache/.jso -> on final acceptance, store the artifact under the INITIAL hash ``` -Use the proposed upstream hash of ordered message roles and exact text. Include -message boundaries and order, excluding model/settings, session/execution IDs, +Use the upstream message-content hash of ordered message roles and exact text. +Include message boundaries and order, excluding model/settings, session/execution IDs, timing, provider cache hints, and other non-content metadata. Add no stage, prompt-ID, library-version, whole-configuration, schema-policy, or implementation fingerprint. Actual input or schema text rendered into messages participates as @@ -499,7 +514,7 @@ prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared` where supported, and discard the handle on a hit. `OnPrepared` execution hooks alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes session identity and cache hints; it is not the requested hash. The upstream -request in U1 must land before completing this cache integration. +API/release review in U1 must finish before completing this cache integration. Keep the initial key throughout corrections. Cache the final Notarius artifact through its existing codec, not an intermediate response, `RunResult`, exchange, @@ -548,7 +563,9 @@ diagnostics; split preparation has a separate purpose for cache lookup. Native observations provide terminal facts even without a returned `RunResult`. `OnPrepared` is absent on preparation/admission failures and is not a completion record. Keep synchronous callbacks lightweight, compose capture/accounting -observers, and synchronize shared state where operations overlap. +observers, and synchronize shared state where operations overlap. `OnPrepared` +runs while admission is held; persist retained evidence at normal operation +boundaries. Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope. Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning, @@ -751,7 +768,7 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | -| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. | +| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | | U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. | | U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | @@ -787,7 +804,7 @@ major boundary changes in successor ADRs without rewriting accepted decisions. ### Phase 1: Establish The Migration Baseline -- Record the selected dependency set and track U1 against the upstream request. +- Record the selected dependency set and review U1 against the upstream release. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. @@ -795,8 +812,9 @@ major boundary changes in successor ADRs without rewriting accepted decisions. execution, assets/schemas, CommandKit, debug, cache, or output ownership. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, -and outstanding choices are explicit. This phase does not claim that the hash -already exists or that the migration is implemented. +and outstanding choices are explicit. Upstream availability alone does not +establish compatibility of the assembled Notarius dependencies or completion +of the migration. ### Phase 2: Build Native Composition, Sources, And Preflight @@ -869,7 +887,8 @@ including failures after publication and quiet-mode capture. - Complete the acceptance checks and removal inventory, including dependency cleanup, maintained examples, and the assembled command surface. - Finish release migration guidance, attribution, successor ADRs, and affected - canonical documentation. Reconcile future-work notes and notepad status. + canonical documentation. Update this roadmap's status and reconcile the + separate future-work backlog. - Confirm no compatibility scaffold, deferred CommandKit adoption, local hash policy, or obsolete metadata requirement remains in the finished migration. @@ -968,7 +987,52 @@ Record significant changes in successor ADRs, including [PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md), admission, caching, and CLI/reporting ownership. Preserve accepted decision text and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records). -Notepads retain discussion history; this roadmap owns implementation status. +This roadmap owns implementation status; Git history retains earlier discussions. + +## Evidence And Reading + +These sources support the selected design, not a claim of completed Notarius +implementation. Sibling links assume the local repositories are available; the +tagged PromptKit links below preserve the reviewed baseline. + +### Native Library Guidance + +- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md) and + [CLI conventions](../../../commandkit/docs/cli.md): module selection, native + aliases/inputs, shared loaders, reporting modes, outcomes, and delivery failures. +- [ckconfig](../../../commandkit/integrations/ckconfig/README.md) and + [ckprompt](../../../commandkit/integrations/ckprompt/README.md): standard suites + supplied with application loaders and request builders. +- [ConfigKit](../../../ConfigKit/README.md): native loading and inspection. +- [PromptKit consumer guide][consumer] and + [capacity behavior](../../../promptkit/docs/internal/capacity.md): native + preparation/execution, continuation, observations, and admission. + +### Audita Comparison + +The comparison was completed on 2026-11-18 against Audita revision +`da5b8913d1206a8e8f0a784b2322760385548a60`, using PromptKit v0.23.2, +ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0. +The observations below concern that revision; linked sibling files may evolve. + +| Evidence | Lesson carried into this roadmap | +| --- | --- | +| [Engine assembly](../../../audita/internal/framework/llm/runtime.go), [source tests](../../../audita/internal/framework/llm/engine_sources_test.go), and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go) | One captured engine, native underlays and main-source-relative companions, whole-record replacement, preserved file contexts, strict peer conflicts, and no silent fallback on operator-source errors. | +| [Adapter tests](../../../audita/internal/framework/llm/adapter_test.go) | Ordinary embedded schemas and complete per-request output contracts work with operator prompt replacement. U2 chooses prompt-declared defaults for Notarius; Audita's schema registry and per-call overrides are not requirements to copy. | +| [Native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), [repair flow](../../../audita/internal/framework/runner/repair.go), [conversation tests](../../../audita/internal/framework/llm/conversation_test.go), and [repair tests](../../../audita/internal/framework/runner/repair_test.go) | Captured exchanges, accumulated history, complete candidate revalidation, final-turn assessment, and canceled/undelivered response handling. | +| [Prompt guidance](../../../audita/docs/internal/prompts.md) | Durable system instructions, an initial user task, and subsequent repair assignments suit native continuation. | +| [Observations](../../../audita/internal/framework/llm/observations.go) and [CLI composition](../../../audita/internal/cli/app_wiring.go) | Native execution totals with application scope and explicit nested-validator enrollment; standard CommandKit suites using shared loaders. | + +Audita repairs individual transcript corrections in independent branches; +Notarius's selected unit is a whole producer operation and typed artifact. +Audita's wrappers, response-schema registry, per-proposal policies, and absence +of a cache do not define Notarius requirements. + +At the reviewed revision, focused LLM, prompt/schema, runner, and proposal-generation +tests passed, together with subprocess checks for correction conversations, +cancellation during repair, and repair-disabled/output selections using a local +fake provider. These historical checks do not establish live-model quality or +completion of Notarius's migration acceptance checks. [release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md [migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md