1289 lines
72 KiB
Markdown
1289 lines
72 KiB
Markdown
# CommandKit Migration Discussion
|
||
|
||
## Purpose And Status
|
||
|
||
Working notepad for the CommandKit decisions that will be incorporated into the
|
||
[master PromptKit-native migration roadmap](promptkit.md). Record the proposal,
|
||
open questions, evidence, and decisions here as we discuss them.
|
||
|
||
**Direction agreed:** Make a full cutover to a CommandKit-native application as
|
||
part of the same migration as modern PromptKit, in separate, reviewable
|
||
implementation steps. Adopt CommandKit's public-facing conventions and
|
||
recommendations to the fullest extent. Bring its design decisions forward and
|
||
complete adoption before the overall cutover.
|
||
|
||
The master roadmap currently places CommandKit in a subsequent Phase 7 and leaves
|
||
its details in U10. That sequencing has not yet been revised. Item 6 selects how
|
||
to incorporate the agreed decisions into its dependency baseline, implementation
|
||
phases, removal inventory, and acceptance criteria while preserving remaining
|
||
choices explicitly. No application or dependency changes have been implemented
|
||
through this discussion.
|
||
|
||
Item 1 selects the dependency versions and standard capabilities, including
|
||
PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target.
|
||
Item 2 selects the application command structure and input conventions. Item 3
|
||
selects native composition, shared loaders, and application preflight. Item 4
|
||
selects artifact-first output, a small native completion report, and outcomes.
|
||
Item 5 selects conditional debug capture through the shared `--debug` flag,
|
||
native diagnostics, and a run/operation bundle with a `latest` symlink. Its
|
||
capture inventory, depth, and
|
||
explicit-empty directory behavior are resolved below. Item 6 selects verification,
|
||
migration guidance, and incorporation into the master roadmap; that incorporation
|
||
is still pending.
|
||
|
||
The cache follow-up in item 4 replaces the earlier checkpoint design with an
|
||
accepted-artifact cache keyed by initial message content. The output/report
|
||
direction is settled: deliver artifacts as files and render a small structured
|
||
completion value through CommandKit, with a native text template for ordinary
|
||
human output. Detailed diagnostics belong in optional debug capture. Function
|
||
signatures, debug filenames other than `latest`, and similar implementation
|
||
details do not reopen the selected architecture. Broader PromptKit choices
|
||
remain with the master roadmap.
|
||
|
||
The PromptKit ownership direction remains in force. The revised
|
||
[cache decision](promptkit-architecture-discussion.md#5-accepted-artifact-cache)
|
||
supersedes earlier checkpoint matching and provenance requirements. Master-roadmap
|
||
incorporation is pending, including U1 (the upstream content-hash capability),
|
||
U8 (cache migration), U9 (records/reporting), and U10 (CommandKit adoption).
|
||
|
||
## Agreed Proposal
|
||
|
||
### Guiding Rule: Full Native Cutover
|
||
|
||
Backward compatibility with the existing CLI conventions is a non-goal. Use
|
||
CommandKit's native command, input, help/version, reporting, error, and outcome
|
||
conventions as the design baseline. This mirrors the PromptKit-native direction.
|
||
|
||
Existing command names, flags, aliases, parsing behavior, report shapes, and exit
|
||
conventions are migration evidence, not requirements to preserve. Do not add
|
||
legacy aliases, compatibility shims, parallel parsers, or reporting wrappers
|
||
solely to reproduce the old CLI. Any application-specific behavior should serve
|
||
an actual Notarius need and fit the native conventions.
|
||
|
||
Provide migration guidance in release documentation, explaining the new commands
|
||
and conventions and the actions operators or subprocess callers must take.
|
||
Documenting the transition replaces a requirement to keep the old CLI working.
|
||
Update canonical CLI and consumer documentation when the behavior lands.
|
||
|
||
Application responsibilities such as producing artifacts, reporting publication
|
||
outcomes, and managing cache/debug storage remain meaningful. Their existing
|
||
CLI expression is not protected by an implicit compatibility exception. This
|
||
guiding rule does not itself choose stored artifact formats. The revised cache
|
||
decision below selects simple artifact storage rather than checkpoint envelopes.
|
||
|
||
### Why Adopt Together
|
||
|
||
- Source assembly and preflight are already changing. CommandKit integrations
|
||
can consume the same application and native engine loaders.
|
||
- ckconfig and ckprompt can replace standard inspection commands directly,
|
||
avoiding an intermediate adaptation of commands that would soon be replaced.
|
||
- Native execution facts and application outcomes can feed the intended reporting
|
||
facilities while duplicate LLM bookkeeping is removed.
|
||
- Configuration and command migration can be documented and verified as one
|
||
coherent transition.
|
||
|
||
The added work is principally CLI composition and contract migration: dispatch,
|
||
flag parsing, configuration selection, reporting, and process-status mapping.
|
||
Choose their native forms deliberately and document the transition, rather than
|
||
spending migration effort recreating existing CLI conventions.
|
||
|
||
### Ownership And Scope
|
||
|
||
| Concern | Selected owner |
|
||
| --- | --- |
|
||
| Command hosting, standard input/help mechanics, common report formatting and visibility | CommandKit. |
|
||
| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. |
|
||
| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. |
|
||
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, accepted-artifact caching, and domain outcomes | Notarius application handlers and framework. |
|
||
| Completion-summary content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
|
||
|
||
Adopt the standard capability suite selected in item 1, including upstream
|
||
commands and recommendations beyond the existing Notarius surface. Items 2–4
|
||
select application inputs, composition, and reporting; items 5–6 select debug
|
||
capture, verification, and migration. The earlier extensive receipt/manifest
|
||
proposal is superseded by item 4's artifact-first contract.
|
||
|
||
Pipeline and domain code should remain independent of CommandKit. Native
|
||
PromptKit observations supply execution facts; Notarius attaches application
|
||
meaning and selects report content. CommandKit supplies presentation mechanics.
|
||
|
||
Two application needs already identified for deliberate treatment are:
|
||
|
||
- The current `run --json` publishes a receipt after output publication. Item 4
|
||
replaces that contract with artifact files and a small completion summary,
|
||
rendered natively as text, YAML, or JSON. No automatic metadata bundle or
|
||
separately persisted report is required.
|
||
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when an
|
||
effective debug directory is available, including an application default.
|
||
Capture selection remains independent of the final reporting mode.
|
||
|
||
### Intended Sequencing
|
||
|
||
1. Incorporate items 1–2's selected dependency baseline, standard capabilities,
|
||
and native application command design, bringing U10 forward.
|
||
2. Introduce item 3's CommandKit composition alongside native source assembly.
|
||
Connect shared loaders, standard inspection, and the focused application
|
||
preflight package.
|
||
3. Develop native execution and accounting against the intended CLI boundary,
|
||
with pipeline and domain tests independent of CommandKit.
|
||
4. Complete command and reporting migration before final end-to-end verification
|
||
and migration documentation.
|
||
|
||
Item 6 selects the replacement phase sequence for incorporation into the master
|
||
roadmap. The master roadmap has not yet been updated.
|
||
|
||
## 1. Adoption Scope And Dependency Baseline
|
||
|
||
**Discussion / decision:** Versions and standard capability scope agreed. Adopt
|
||
the set below, including the move to PromptKit v0.23.3. Items 2–3 select
|
||
application commands and composition; items 4–5 select reporting and debug
|
||
capture principles. These are selected migration targets; the application
|
||
dependencies have not changed.
|
||
|
||
### Selected Versions
|
||
|
||
| Module | Version | Role |
|
||
| --- | --- | --- |
|
||
| `go.maximumdirect.net/commandkit` | `v0.7.0` | Native command declarations, inputs, help/version, reporting, and completion. |
|
||
| `go.maximumdirect.net/ckconfig` | `v0.9.0` | Standard configuration commands and ConfigKit diagnostic classification. |
|
||
| `go.maximumdirect.net/ckprompt` | `v0.10.0` | Standard PromptKit inspection, validation, preparation, and smoke diagnostics. |
|
||
| `go.maximumdirect.net/configkit` | `v1.2.1` | Shared configuration dependency for the application and integrations. |
|
||
| `go.maximumdirect.net/promptkit` | `v0.23.3` | Native model execution, including corrected timeout/cancellation reporting. |
|
||
|
||
The CommandKit integration releases form a documented compatible set using
|
||
PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout attribution and
|
||
cancellation/error identity fixes. It retains the same dependencies and requires
|
||
no configuration or request API migration from v0.23.2. The Go consumer minimum
|
||
remains 1.27. Verify the assembled dependency set through the migration's normal
|
||
integration checks and update attribution when the dependency changes land.
|
||
|
||
The version review used locally available release tags and tagged dependency
|
||
manifests. Remote release lookup was unavailable, so it did not establish whether
|
||
newer tags existed remotely. These are deliberate selected versions, not a claim
|
||
that every module is the latest published release.
|
||
|
||
Evidence: [integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
|
||
and [PromptKit v0.23.3 release notes](../../../promptkit/docs/releases/v0.23.3.md).
|
||
|
||
### Selected Standard Capabilities
|
||
|
||
| Area | Adopt in the cutover |
|
||
| --- | --- |
|
||
| Core CLI | Native help, `version`, typed flags and operands, supplied-value semantics, standard outcomes, and command declaration/dispatch. |
|
||
| Reporting | Native `--format` with text/YAML/JSON, `--quiet`, `--verbose`, `--debug`, and `--progress`, including upstream visibility and failure behavior. |
|
||
| Configuration | `config scopes`, `validate`, `show`, `sources`, `explain`, and `diff`, for application and PromptKit configuration. |
|
||
| Resource discovery | PromptKit listings for prompts, profiles, backends, namespaces, and collections, with native filters and selectors. |
|
||
| Resource inspection | Native prompt, profile, and backend reports, plus requirements discovery. |
|
||
| Concrete request checks | Requirements validation and prepared-request preview using native inputs, variables, profiles, and execution overrides. |
|
||
| Provider diagnostics | Reachability, authentication, text-inference, structured-output, and aggregate smoke tests. |
|
||
| Convenience features | Bash, Zsh, and Fish completion, useful declaration hints, and upstream PromptKit convenience aliases. |
|
||
|
||
Use the [upstream CLI conventions](../../../commandkit/docs/cli.md) directly.
|
||
Providing the standard capability suite means supplying application loaders and
|
||
callbacks, not recreating those commands locally. Upstream convenience aliases
|
||
are selected native features; this does not reintroduce legacy Notarius aliases.
|
||
|
||
### Wiring Boundaries And Follow-Up
|
||
|
||
- Keep configuration validation, requirements discovery, and concrete request
|
||
validation distinct. Pipeline preflight remains application-owned. Concrete
|
||
preparation may read inputs and schemas and check credentials even though it
|
||
makes no model call.
|
||
- Install smoke commands for explicit invocation. Their availability does not
|
||
introduce provider calls during startup, ordinary validation, or offline tests.
|
||
- Enable application configuration overrides through ConfigKit. The selected
|
||
PromptKit configuration scope does not support arbitrary `--set` overrides;
|
||
use supported request-level controls rather than implementing a local equivalent.
|
||
- Retain application handlers for pipeline execution and discovery, with native
|
||
commands, inputs, and result presentation as selected in items 2–5.
|
||
|
||
The standard capability suite needs no identified upstream addition. The later
|
||
cache decision does require a native message-content hash, which is not the
|
||
existing v0.23.3 rendered-prompt hash. Confirm the supplying API/release before
|
||
implementing cache lookup; the version table remains the reviewed baseline, not
|
||
a claim that the new hash is already available. On roadmap
|
||
reconciliation, incorporate these versions into U1, replace U10's open standard
|
||
capability selection with this decision, and incorporate the application wiring
|
||
and presentation decisions from items 2–5. Retire superseded durable-record
|
||
requirements as directed by item 6.
|
||
|
||
## 2. Native Command Surface And Inputs
|
||
|
||
**Discussion / decision:** Command structure and input conventions agreed. Use a
|
||
small application-owned `pipeline` family alongside the standard `config` and
|
||
`promptkit` families. These are migration targets, not implemented commands.
|
||
|
||
### Command Structure
|
||
|
||
```text
|
||
notarius pipeline list
|
||
notarius pipeline show <pipeline-id>
|
||
notarius pipeline validate <pipeline-id>
|
||
notarius pipeline run <pipeline-id> --input-file <path>
|
||
notarius run <pipeline-id> --input-file <path>
|
||
|
||
notarius config scopes
|
||
notarius config validate|show|sources|explain|diff ...
|
||
|
||
notarius promptkit list ...
|
||
notarius promptkit show ...
|
||
notarius promptkit requirements show|validate ...
|
||
notarius promptkit test <kind>
|
||
|
||
notarius version
|
||
notarius completion bash|zsh|fish
|
||
notarius help [command...]
|
||
```
|
||
|
||
The alternatives separated by `|` above summarize the installed operations;
|
||
they are not literal shell commands.
|
||
|
||
| Application command | Responsibility |
|
||
| --- | --- |
|
||
| `pipeline list` | Discover configured pipelines. |
|
||
| `pipeline show` | Describe the resolved ordered steps, lanes, modules, reference slots, and configured selections. |
|
||
| `pipeline validate` | Perform offline application preflight using the intended selections and overrides. |
|
||
| `pipeline run` | Execute the selected pipeline and publish artifacts. |
|
||
| `run` | Native convenience alias for `pipeline run`. |
|
||
|
||
Move pipeline-specific validation out of `config validate --pipeline ...`.
|
||
Configuration validation, pipeline composition checks, and concrete PromptKit
|
||
request validation remain distinct. Pipeline preflight does not synthesize stage
|
||
requests or require provider credentials. PromptKit's concrete validation may
|
||
read inputs and schemas and check credentials without making a model call.
|
||
|
||
**Naming decision:** Keep `pipeline run` as the canonical command and install
|
||
`run` as a native convenience alias using
|
||
`commandkit.Alias("run", "pipeline run")`. Both routes use the same handler,
|
||
operands, flags, validation, and reporting. CommandKit supplies alias-aware help
|
||
and diagnostics; no forwarding handler or additional parser is needed.
|
||
|
||
The grouped family and root shortcut are application usability choices, not an
|
||
upstream naming requirement. The alias uses the new input conventions and does
|
||
not preserve legacy flags or behavior. Use `notarius run` in quick-start examples
|
||
and document both routes in the command reference when the migration lands.
|
||
See [native alias behavior](../../../commandkit/docs/cli.md#application-owned-aliases).
|
||
|
||
### Input Conventions
|
||
|
||
Use a required pipeline-ID operand and a required `--input-file PATH` for runs.
|
||
Use native typed declarations and supplied-value semantics, including explicit
|
||
empty, false, and zero values. Apply defaults only when the relevant value is
|
||
omitted; application validation still decides whether a supplied value is valid.
|
||
|
||
| Concern | Selected convention |
|
||
| --- | --- |
|
||
| Configuration | Native `--config` and `--overlay`; name the application scope `app`. |
|
||
| Lane selection | Repeatable `--lane ID`; omission selects the normal pipeline lane set. Replace comma-separated `--only`. |
|
||
| Output placement | Application-owned `--output-dir PATH`. |
|
||
| Chunk-plan caching | `--chunk-cache auto\|bypass\|refresh`. |
|
||
| Accepted-artifact cache | Replace checkpoint-centric controls with cache reuse/bypass/refresh behavior; concrete CLI spelling remains to be selected. `--resume` and `--recompute-step` are migration context, not compatibility requirements. |
|
||
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
|
||
| Configuration-valued tuning | Supported application `--set POINTER=JSON` overrides, including worker and semantic-budget settings once their configuration design is settled. |
|
||
|
||
Adopt native parsing rules: flags follow the complete command path and may be
|
||
interspersed with operands; `--` ends option parsing. Singleton flags use the
|
||
last successfully parsed occurrence, with every occurrence checked. Repeatable
|
||
flags preserve occurrence order and treat commas as ordinary value content.
|
||
Use filesystem and choice completion hints where applicable.
|
||
|
||
Keep worker counts and semantic continuation budgets in configuration rather
|
||
than introducing additional dedicated CLI controls now. Their field names,
|
||
defaults, and domain meaning remain master-roadmap decisions. Item 3 selects
|
||
shared loading and override wiring across application and standard commands.
|
||
|
||
### Native LLM Controls
|
||
|
||
Use ckprompt's reusable flag declarations for applicable profile, session, and
|
||
execution controls on `pipeline run`, including model, temperature, timeout,
|
||
reasoning effort, and transient retries. Run-wide overrides reach all LLM-backed
|
||
producers and validators, including continuation calls. Without an override,
|
||
normal binding/pipeline/prompt selection applies.
|
||
|
||
| Reasoning input | Meaning |
|
||
| --- | --- |
|
||
| Omitted | Inherit the selected settings. |
|
||
| `--llm-reasoning-effort high` | Replace the inherited setting. |
|
||
| `--llm-reasoning-effort ""` | Clear the inherited setting. |
|
||
|
||
Remove the separate `--clear-reasoning-effort` control. Native transient retries
|
||
remain distinct from application semantic continuation budgets. Individual
|
||
prompt selection and named prompt input injection belong on standard PromptKit
|
||
commands; pipeline modules construct their own requests and response contracts.
|
||
Use native `--llm-profile` and `--llm-session-id` spellings, without inventing
|
||
another LLM input parser.
|
||
|
||
### Application Reference Inputs
|
||
|
||
Retain repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`.
|
||
Keep useful pipeline-wide, lane, chunk, and stage-specific selectors, such as
|
||
`party=party.txt` and `npc-registry.extract.party=npc-party.txt`. CommandKit owns
|
||
repeatable input parsing and typed conversion; Notarius owns selector expansion,
|
||
specificity, required slots, and external/generated-reference conflict checks.
|
||
These selectors describe pipeline relationships and are not interchangeable
|
||
with PromptKit's individual-request `--llm-input-file` bindings.
|
||
|
||
### Preparing Pipeline Inspection Content
|
||
|
||
**Selected design:** Use ConfigKit for the composed, decoded
|
||
configuration, defaults, imports, overlay, inheritance, overrides, and native
|
||
provenance/redacted inspection. Feed that loaded configuration through the same
|
||
application resolution used by validation and execution to obtain selected
|
||
lanes, registered modules, validator chains, and reference relationships.
|
||
CommandKit renders a small application report of that resolved composition.
|
||
|
||
ConfigKit does not know Notarius's module catalog or pipeline semantics. Reuse
|
||
the existing resolver rather than writing a second resolution path for `show`.
|
||
Generic effective configuration and source explanations remain available through
|
||
`config show` and `config explain`. ConfigKit's safe inspection facilities do not
|
||
automatically redact an arbitrary application report; select appropriate report
|
||
content rather than dumping the raw typed configuration. Item 3 selects loader
|
||
wiring; item 4 selects the artifact-first output and completion-report contract.
|
||
|
||
### Removals And Remaining Work
|
||
|
||
Retire custom dispatch, argument-count handling, assembled help, version-command
|
||
parsing, standard configuration/LLM flag parsing, comma-separated lane parsing,
|
||
and the separate reasoning-clearing machinery. Replace `config effective` with
|
||
native `config show`, `config profile` with `promptkit show profile`, and bespoke
|
||
`--json` switches with `--format json`.
|
||
|
||
Application handlers still validate meaningful input combinations and translate
|
||
typed inputs into pipeline selections. Native `--debug` supplies reporting
|
||
behavior and conditional artifact capture as selected in item 5.
|
||
Items 3–4 select loader behavior, the small completion report, and native outcome
|
||
presentation. Reconcile U9 with these decisions rather than expanding durable
|
||
metadata.
|
||
|
||
Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax),
|
||
[reusable PromptKit flags](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading),
|
||
[ConfigKit loading and inspection](../../../ConfigKit/README.md), and the current
|
||
[Notarius configuration boundaries](../internal/configuration.md).
|
||
Use the current [CLI reference](../cli.md) for migration guidance, not as a
|
||
compatibility constraint.
|
||
|
||
## 3. Application Composition, Loaders, And Preflight
|
||
|
||
**Discussion / decision:** Agreed. Fully adopt native ConfigKit, PromptKit, and
|
||
CommandKit workflows through a small composition root, shared loaders, and
|
||
invocation-local state. Follow substantially the same composition/loading pattern
|
||
as Audita while retaining Notarius's pipeline, artifact, validation, recovery,
|
||
and publication responsibilities. Centralize application prompt/profile
|
||
dependency checks in a focused `internal/preflight` package. Implementation is
|
||
pending.
|
||
|
||
### Native Application Composition
|
||
|
||
One application constructor installs the pipeline declarations and native `run`
|
||
alias, a ckconfig suite with `app` and `promptkit` scopes, the selected ckprompt
|
||
capabilities, and native help/version/completion/reporting. Supply ordinary
|
||
callbacks and typed inputs; do not introduce another CLI framework or LLM wrapper.
|
||
|
||
Constructing the application registers declarations without loading configuration,
|
||
building an engine, or initializing run storage. Help, version, and completion
|
||
remain independent of runtime configuration and provider work. Application
|
||
handlers acquire only what their operation requires.
|
||
|
||
### Shared Configuration And Engine Loaders
|
||
|
||
Use two shared loading responsibilities across application and standard commands:
|
||
|
||
1. **Application configuration:** Select explicit `--config`, then the existing
|
||
environment/discovery policy; apply native `--overlay` and supported
|
||
application `--set` overrides. ConfigKit owns composition, imports,
|
||
inheritance, presence, provenance, and inspection. Supply application
|
||
defaults, normalization, and structural validation through its extension
|
||
points. Retain the native ConfigKit result and selected main-source context;
|
||
derive runtime values where needed without copying inspection machinery into
|
||
another result abstraction.
|
||
2. **PromptKit engine:** Build a native `*promptkit.Engine` from that loaded
|
||
application's source context and the agreed native source assembly. Runtime,
|
||
preflight, and inspection use the same builder and definitions.
|
||
|
||
Preserve supplied-value information throughout loading. An explicitly empty
|
||
configuration path receives deliberate validation rather than silently falling
|
||
through to discovery. Across application and PromptKit commands, `--config`
|
||
selects the Notarius application configuration; the PromptKit scope inspects the
|
||
engine assembled for that application selection.
|
||
|
||
Retain the master roadmap's [source assembly](promptkit.md#configuration-sources-and-prompt-assets):
|
||
PromptKit's bundled underlay, embedded Notarius resources, then optional operator
|
||
companions beside the main application source actually selected by ConfigKit.
|
||
Use native imports and filesystem contexts. There is no second source search or
|
||
working-directory fallback; strict composition, missing-companion handling, and
|
||
load-error behavior remain as selected in the master roadmap. Do not reinterpret
|
||
the application document as a standalone PromptKit configuration or add another
|
||
inspection-only assembly path.
|
||
|
||
Application `--set` overrides flow through ConfigKit's load. Native request
|
||
execution overrides remain request controls; they do not mutate captured engine
|
||
configuration. Preserve the selected PromptKit configuration scope's lack of
|
||
arbitrary `--set` support.
|
||
|
||
### Invocation State And Engine Ownership
|
||
|
||
Use CommandKit's supported invocation state to retain configuration and an engine
|
||
when callbacks need them repeatedly. Application handlers use the same loader
|
||
functions with small invocation-owned state. One pipeline execution shares one
|
||
native engine across producers, validators, and continuations, preserving native
|
||
backend admission limits.
|
||
|
||
Do not assume one invocation always means one configuration selection. Native
|
||
`config diff` acquires left and right selections with potentially different
|
||
configuration, overlays, and overrides while sharing invocation state. Load the
|
||
sides independently or distinguish selections in any local reuse. A single
|
||
unqualified cached result would be incorrect. No persistent loader cache or
|
||
cryptographic configuration identity is needed.
|
||
|
||
### Shared Pipeline Resolution And Command Work
|
||
|
||
Reuse the existing application resolver with loaded configuration, selected
|
||
pipeline/lanes/reference changes, applicable overrides, and the module catalog.
|
||
It owns module/options validation, typed artifact compatibility, validator-chain
|
||
selection, ordered dependencies, and reference-binding compatibility.
|
||
|
||
| Command | Required application work |
|
||
| --- | --- |
|
||
| `pipeline list` | Load application configuration and enumerate pipelines. |
|
||
| `pipeline show` | Resolve the selected pipeline and report its composition. |
|
||
| `pipeline validate` | Resolve the pipeline and check its known native resource dependencies offline. |
|
||
| `pipeline run` / `run` | Use the same resolution and preflight checks, then prepare and execute the pipeline. |
|
||
| `config … --scope app` | Supply the native ConfigKit result directly to the integration. |
|
||
| PromptKit inspection commands | Acquire the shared engine assembly and delegate to native inspection. |
|
||
|
||
Run handlers call shared application functions directly; they do not invoke the
|
||
validation command or parse its output. `pipeline show` can describe configured
|
||
prompt/profile selections without constructing an engine just to expand them
|
||
into execution settings or performing every execution-readiness check.
|
||
|
||
### Prompt Dependencies And Offline Preflight
|
||
|
||
Notarius knows which operations need prompts; PromptKit knows whether those
|
||
definitions resolve in the assembled engine. Check every statically known prompt
|
||
dependency of the selected pipeline, including LLM-backed validators and any
|
||
separately selected correction prompts. A deterministic stage may require none,
|
||
an LLM-backed stage may require several, and multiple stages may share one.
|
||
Configuration can select the required set. For data-dependent selections, check
|
||
what is knowable before execution and resolve the final choice at runtime.
|
||
|
||
Share prompt identifiers and selection logic with execution through module or
|
||
operation declarations. Do not maintain a separate hardcoded preflight inventory.
|
||
These small dependency declarations identify required resources; they do not
|
||
restore asset manifests, prompt filesystem registration, message assembly, or
|
||
schema registries. Native sources remain the owners of prompt definitions.
|
||
|
||
Use the same engine and intended namespaces as execution to inspect known prompts
|
||
and applicable profile selections, including a known prompt's default profile
|
||
when effective. Partial profiles alone are not failures. Report missing
|
||
dependencies with pipeline, step/lane, stage or validator, and resource context
|
||
where applicable.
|
||
|
||
Offline preflight does not read the transcript, materialize reference contents,
|
||
synthesize stage requests, require credential values, or contact providers.
|
||
Engine assembly may read configuration and prompt resources. Availability does
|
||
not certify prompt domain correctness or guarantee that future requests pass
|
||
schema, credential, or provider checks. Concrete `Prepare`/`ValidateRequest`
|
||
remains the boundary for completed request validation, and smoke commands remain
|
||
explicit provider operations.
|
||
|
||
### Dedicated Preflight Package
|
||
|
||
Add a small application-level `internal/preflight` package accepting the resolved
|
||
pipeline, relevant module declarations, and the native engine. It checks the
|
||
pipeline's known prompt/profile dependencies and returns contextual findings or
|
||
errors for the command boundary to present.
|
||
|
||
Keep configuration validation in ConfigKit/application definitions, structural
|
||
resolution in the pipeline resolver, and concrete preparation in PromptKit.
|
||
Preflight does not discover configuration, register production modules, construct
|
||
engines, prepare synthetic requests, initialize storage, or own CLI reporting.
|
||
The composition root supplies its collaborators. Module dependency declarations
|
||
remain in existing module contracts; modules do not import the preflight package.
|
||
Exact function signatures and finding representation remain implementation choices.
|
||
|
||
### Request Construction, Storage, And Removals
|
||
|
||
Use native request builders directly for standard PromptKit commands where
|
||
sufficient. Add an application callback only for a demonstrated application need,
|
||
preserving supplied operator values. Pipeline modules construct their own domain
|
||
requests; share concrete request-building functions when useful without
|
||
simulating pipeline execution to populate an inspection command.
|
||
|
||
Output publication, artifact caching, debug capture, and execution records belong
|
||
to the run path. Loaders do not initialize those resources; item 5 selects debug
|
||
allocation timing. Remove command-specific loading paths, copied configuration
|
||
and resource inspection, eager runtime setup, and duplicated preflight
|
||
orchestration. No broader application/package reorganization is required by this
|
||
decision.
|
||
|
||
Evidence: [native suite and loader ownership](../../../commandkit/docs/consumers/guide.md#configkit-scopes),
|
||
[PromptKit integration callbacks](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading),
|
||
[Audita composition](../../../audita/internal/cli/app_wiring.go), and the
|
||
[existing configuration ownership decision](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
|
||
|
||
## 4. Reports, Machine Output, And Outcomes
|
||
|
||
**Discussion / decision:** Agreed. Deliver artifacts as files and use CommandKit
|
||
to render a small structured completion report on stdout, with a native text
|
||
template for the ordinary successful case. Additional operational diagnostics
|
||
use stderr; detailed investigation uses optional debug capture. This supersedes
|
||
the earlier extensive receipt, manifest, and durable-execution-record proposals.
|
||
Keep native outcome and delivery-failure handling. Implementation is pending.
|
||
|
||
### Accepted-Artifact Cache And PromptKit Ownership
|
||
|
||
**Agreed follow-up:** Rename and replace the checkpoint subsystem with the
|
||
[accepted-artifact cache](promptkit-architecture-discussion.md#5-accepted-artifact-cache).
|
||
That decision owns the matching and storage policy; this section records its
|
||
CommandKit and execution-boundary implications.
|
||
|
||
The cached value is an accepted answer to the original task. Compute the key
|
||
from the initial prepared request using a proposed upstream `MessageContentHash`
|
||
over ordered message roles and exact text, excluding settings, session/execution
|
||
IDs, timestamps, and provider cache hints. Do not hash the entire prepared
|
||
request or introduce another Notarius-owned fingerprint policy.
|
||
|
||
Look up `cache/<hash>.json` before model execution. A hit loads the final accepted
|
||
Notarius artifact directly, subject only to ordinary decoding/usability checks.
|
||
It does not execute a model request, reconstruct a result/exchange, replay
|
||
corrections, or repeat semantic validation merely because the artifact was cached.
|
||
|
||
On a miss, execute the complete Notarius operation: native PromptKit calls,
|
||
domain interpretation and validation, and any application-directed correction
|
||
passes. Only after final acceptance save the artifact under the initial hash,
|
||
even if correction required several turns. For normalization, cache the accepted
|
||
normalized artifact after applying the plan, not the model's proposed plan.
|
||
Rejected, failed, and validation-incomplete work is not published to the cache.
|
||
|
||
PromptKit owns structural response validation, internal retries, repairs, and
|
||
other mechanics inside each request execution. Notarius consumes a usable
|
||
response or terminal failure at the public boundary and owns domain acceptance.
|
||
Internal PromptKit attempts do not determine cache eligibility. Native debug
|
||
evidence about those attempts does not transfer their ownership to Notarius.
|
||
|
||
The upstream requirement is message-content hashing. It does not require native
|
||
cached-response injection, replay, exchange restoration, or write-through
|
||
response caching. A small Notarius filesystem lookup/write facility stores the
|
||
accepted artifact without checkpoint manifests, dependency fingerprints,
|
||
acceptance ledgers, or required execution provenance. A cache hit is not a new
|
||
model execution or a claim of validation under a changed policy.
|
||
|
||
Concrete cache flags and legacy-checkpoint handling remain implementation-planning
|
||
choices. The output/report discussion must not reintroduce cache metadata solely
|
||
to populate a report. The master roadmap still needs this replacement policy.
|
||
|
||
### Artifacts Are The Deliverable
|
||
|
||
Publish the accepted domain artifacts as files in the output directory. Their
|
||
contents and encoding are independent of CLI report formatting: `--format`
|
||
controls the completion summary, not the artifact payloads. Keep artifact
|
||
discovery understandable without requiring consumers to read execution metadata.
|
||
|
||
Do not require a run manifest, metadata bundle, or persisted completion receipt
|
||
alongside the artifacts. Preserve information intrinsic to the domain artifacts
|
||
and their existing schemas; do not add model, retry, validation-history, or
|
||
execution inventories to those payloads merely because it is available.
|
||
Detailed operational evidence belongs in item 5's optional debug bundle.
|
||
|
||
### Small Structured Completion Report And Native Text Template
|
||
|
||
Submit one small application value through `Reporter.Report`. Use these fields
|
||
for an ordinary successful run:
|
||
|
||
| Field | Meaning |
|
||
| --- | --- |
|
||
| `pipeline` | Selected pipeline identifier. |
|
||
| `artifact_count` | Number of deliverable artifacts produced, including reused artifacts; excludes internal intermediate results, cache entries, and debug files. |
|
||
| `output_directory` | Absolute path to the directory containing the published artifacts. |
|
||
| `debug_directory` | Optional absolute path to the completed run-specific debug bundle; omitted when capture is disabled. |
|
||
|
||
For example, without debug capture:
|
||
|
||
```json
|
||
{
|
||
"pipeline": "dnd-session",
|
||
"artifact_count": 6,
|
||
"output_directory": "/work/results/run-123"
|
||
}
|
||
```
|
||
|
||
Supply a short native `text/template` to render that same value as:
|
||
|
||
```text
|
||
Success: dnd-session produced 6 artifacts in /work/results/run-123
|
||
```
|
||
|
||
CommandKit automatically renders a string value as plain text. Here the value
|
||
is structured so JSON/YAML callers receive named fields instead of a success
|
||
sentence they must parse. The native text template provides the concise human
|
||
presentation; without it, a structured value would default to YAML. Use native
|
||
`--format text|yaml|json` selection and serializers, without separate encoders or
|
||
independent human/machine report construction.
|
||
|
||
When a completed run includes rejected work or permitted incomplete validation,
|
||
qualify the completion message clearly. Include the rejection count and/or
|
||
incomplete-validation fact in the same structured value when applicable. Do not
|
||
present unqualified success that conceals those outcomes, or make their visibility
|
||
depend on verbose/debug mode. These are small outcome qualifications, not
|
||
per-producer histories or a general metadata envelope.
|
||
|
||
No report-version field, run-ID field, token totals, profile inventory, warning
|
||
statistics, validation-summary collection, or durable execution ledger is required
|
||
by this completion contract. The old receipt is migration context, not a shape
|
||
to preserve. Standard configuration and PromptKit commands retain their native
|
||
report values and behavior without a universal Notarius wrapper.
|
||
|
||
### Optional Machine Output And Saved Reports
|
||
|
||
Operators and subprocess callers select YAML or JSON through the native format
|
||
flag. The completion report is enough to find the artifacts and recognize any
|
||
qualified outcome. Consumers interested only in artifacts need no metadata file.
|
||
|
||
Save a report with ordinary shell redirection:
|
||
|
||
```sh
|
||
notarius run dnd-session --input-file transcript.json --format json > run-report.json
|
||
```
|
||
|
||
Do not add a separate report-file flag or automatically persist a duplicate report
|
||
for this design. Keep stdout as one completion report, with additional diagnostics
|
||
on stderr, so `--format json --verbose` still produces valid report JSON.
|
||
|
||
### Publication, Streams, And Machine Consumption
|
||
|
||
The run command executes the pipeline, publishes the output bundle, completes
|
||
required run finalization, submits its successful report, and returns a native
|
||
outcome. The report describes an actual published result. Item 5 requires
|
||
requested capture to finalize successfully before a successful run report is
|
||
submitted and defines how capture failures preserve publication facts.
|
||
|
||
Use one primary run-result report, with progress and diagnostics separate. Wire
|
||
CommandKit's result destination to stdout and its diagnostic destination to
|
||
stderr. Subprocess consumers request `--format json`, capture streams separately,
|
||
check the exit status, and decode the successful run report. A report-write
|
||
failure may leave partial stdout even though publication already succeeded.
|
||
These run-consumption rules do not replace the native report behavior of other
|
||
commands, including configuration commands that can report failed validation.
|
||
|
||
### Native Visibility, Diagnostics, And Debug Location
|
||
|
||
Wire CommandKit's result destination to stdout and diagnostic destination to
|
||
stderr. Use `Info`, `Warn`, `Debug`, `Progress`, and native outcome explanations
|
||
for their respective categories; do not mix these messages into a JSON/YAML
|
||
completion report.
|
||
|
||
| Mode | Stdout | Stderr |
|
||
| --- | --- | --- |
|
||
| Ordinary | Brief completion summary. | Failure explanation, if needed. |
|
||
| `--verbose` | Same completion report. | Additional operational information, warnings, and any failure explanation. |
|
||
| `--quiet` | Nothing. | Failure explanation, if needed. |
|
||
| `--debug` | Completion report, including capture location when enabled. | Native debug diagnostics, information, warnings, progress, and any failure explanation. |
|
||
|
||
Adopt native reporting-mode ordering, explicit progress selection, and failure
|
||
visibility. The table describes mode defaults; `--progress` can explicitly enable
|
||
progress outside quiet mode. Machine callers expecting a report do not request
|
||
`--quiet`. Quiet suppression is not a delivery failure; native rendering errors
|
||
can still fail a quiet invocation. Artifact-quality advisories keep their domain
|
||
classification rather than becoming process warnings solely for visibility.
|
||
|
||
When capture completes, the text template adds a line such as:
|
||
|
||
```text
|
||
Debug capture: /tmp/notarius-debug/run-123
|
||
```
|
||
|
||
Use `debug_directory` in the structured value for the same location. Provide the
|
||
path rather than mirroring the debug bundle to stdout. An explicitly empty debug
|
||
directory preserves native debug diagnostics but creates no capture or capture
|
||
path in the completion report. Item 5 owns capture controls and lifecycle.
|
||
|
||
### Native Outcomes And Domain Acceptance
|
||
|
||
Adopt the standard process-status conventions:
|
||
|
||
| Status | Meaning |
|
||
| --- | --- |
|
||
| `0` | The command completed successfully. |
|
||
| `1` | Operational failure, cancellation, publication failure, or presentation failure. |
|
||
| `2` | Invalid invocation or identifiable invalid configuration input. |
|
||
| `3` | Native `config diff --check` found differences. |
|
||
|
||
Use ckconfig's structured configuration-error classification. Do not classify
|
||
every configuration failure as operational or every preflight failure as invalid
|
||
input. Invalid operator selections and missing bundled application resources can
|
||
have different causes; preserve that distinction and native error identity.
|
||
|
||
Command completion remains separate from domain acceptance. A successfully
|
||
published run can contain recorded rejections or validation-incomplete results
|
||
allowed by application policy; those facts qualify the completion report and do
|
||
not automatically fail the command. A terminal pipeline failure does fail the
|
||
command. Native status zero also does not replace reading standard PromptKit
|
||
reports for partial discovery or unsupported smoke checks.
|
||
|
||
Application handlers supply useful native outcome explanations and retain the
|
||
underlying error through native cause handling. CommandKit presents the terminal
|
||
result. On failure, provide a useful explanation and nonzero status rather than
|
||
requiring a separate structured error-report protocol. `--format` controls the
|
||
completion report, not native diagnostics. If artifacts were published before a
|
||
later failure, include their location in the failure explanation. Remove duplicate
|
||
terminal-error printing and custom outcome wrappers.
|
||
|
||
### Delivery Failures, Progress, And Accounting
|
||
|
||
Use CommandKit's native separation of operation causes from output and diagnostic
|
||
delivery failures. If publication succeeds but stdout fails, the final command
|
||
status reflects presentation failure while the published artifacts remain a
|
||
fact. Do not rewrite pipeline history as failed execution or automatically rerun
|
||
model work because report delivery failed.
|
||
|
||
Notarius owns stage/lane/chunk progress, domain acceptance, and cache reuse.
|
||
PromptKit supplies native execution observations and `ExecutionGroup` totals.
|
||
These facts support native operational diagnostics and optional debug capture;
|
||
they do not require a permanent consumer execution report. Explicitly enroll the
|
||
relevant initial, continuation, and validator executions without double counting.
|
||
Cache reuse does not add historical token usage to current execution totals, and
|
||
unavailable usage remains distinguishable from zero.
|
||
|
||
Remove bespoke format selection, parallel receipt encoders, custom verbosity
|
||
handling, duplicate execution accounting, and superseded terminal-reporting
|
||
machinery. Retire automatic reporting-only metadata bundles, mandatory run
|
||
manifests, per-producer provenance inventories, and durable execution-ledger
|
||
requirements. Do not preserve or rebuild those structures to populate the small
|
||
completion value. Use existing application outcome facts for its contents and
|
||
the debug overview. Artifact schemas remain domain contracts; detailed debug
|
||
representations remain diagnostic data.
|
||
|
||
Evidence: [native reporting](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
|
||
[visibility and delivery](../../../commandkit/docs/cli.md#reporting), and
|
||
[outcome classification](../../../commandkit/docs/cli.md#outcome-status).
|
||
The current [run-result contract](../integrations/run-result.md) and
|
||
[CLI failure mapping](../internal/cli.md#failure-mapping-and-terminal-reporting)
|
||
document migration impact. Item 6 of the
|
||
[PromptKit discussion record](promptkit-architecture-discussion.md#6-inspection-and-execution-bookkeeping)
|
||
retains the native bookkeeping boundary. Reconcile U9 with this simpler contract,
|
||
including retirement of superseded durable-record requirements.
|
||
|
||
## 5. Debug Capture And Optional Diagnostics
|
||
|
||
**Discussion / decision:** Controls, native capture hooks, lifecycle, failure
|
||
policy, and optional diagnostics agreed. Use `--debug` for native verbosity and
|
||
durable capture with `/tmp/notarius-debug` as the default root. Operator-owned
|
||
data handling and convenient diagnostic representation are agreed. Replace the
|
||
summary/trace hierarchy with a compact run index, configuration and composition
|
||
snapshots, and operation records; maintain a `latest` symlink at the debug root.
|
||
The capture inventory and depth below are agreed: one default depth retaining
|
||
every application-visible turn, useful inputs and outputs, and application
|
||
assessments, with native snapshots and convenient readable views. An explicitly
|
||
empty debug directory disables capture while preserving native reporting-mode
|
||
selection, resolving verbosity without capture. Exact filenames and serialized
|
||
application fields remain implementation choices. Implementation is pending.
|
||
|
||
### Shared Debug Control With Conditional Capture
|
||
|
||
Attach a native application boolean binding to CommandKit's shared `--debug`
|
||
option on the pipeline run command. Capture is enabled when that boolean is true
|
||
and an effective debug directory is available. `--debug-dir PATH` overrides
|
||
`debug.directory` in application configuration, whose default is
|
||
`/tmp/notarius-debug`. A directory alone does not enable capture. Do not add a
|
||
separate `--capture-debug` flag.
|
||
|
||
**Default-directory decision:** Use the absolute root `/tmp/notarius-debug`,
|
||
with each run's bundle beneath it. Temporary diagnostic output belongs under
|
||
`/tmp` or an operator-selected location under `/var`, rather than beside a
|
||
configuration file likely to live under `/etc`. Replace the old configuration-
|
||
relative `./notarius-debug` default. Precedence is CLI override, configured value,
|
||
then application default. Nonempty relative configuration paths retain their
|
||
source-relative interpretation; nonempty relative CLI paths use the invocation
|
||
directory.
|
||
|
||
Omitting both directory overrides selects the default, so `--debug` enables
|
||
verbosity and capture without requiring a path.
|
||
|
||
**Explicit-empty decision:** An explicitly empty `--debug-dir ""` or
|
||
`debug.directory: ""` disables capture while preserving native debug verbosity
|
||
and reporting-mode ordering. Omission inherits the configured value or default;
|
||
an explicit empty value is meaningful and must not fall back to either. CLI
|
||
selection overrides configuration, including when the CLI value is empty. A
|
||
nonempty CLI directory can therefore re-enable capture over an empty configured
|
||
directory when `--debug` is enabled. No additional flag is needed for verbosity
|
||
without capture.
|
||
|
||
Recognize an empty value before resolving relative paths so it cannot become the
|
||
invocation or configuration directory. Disabled capture creates no bundle and
|
||
leaves `latest` untouched. A nonempty but unusable destination follows the
|
||
requested-capture failure policy when capture is enabled; it does not silently
|
||
disable capture or select a fallback directory.
|
||
|
||
Determine capture from the application boolean and destination, not from
|
||
`Invocation.ReportingMode()`. A later quiet or verbose selection can change
|
||
presentation without clearing the debug input. Explicit `--debug=false` follows
|
||
native boolean binding semantics and disables capture.
|
||
|
||
| Configured directory | Invocation flags | Reporting selection | Capture |
|
||
| --- | --- | --- | --- |
|
||
| Any | No `--debug`, with any directory selection | Selected native mode | None. |
|
||
| Omitted | `--debug` | Debug | `/tmp/notarius-debug`. |
|
||
| `/var/tmp/notarius` | `--debug` | Debug | `/var/tmp/notarius`. |
|
||
| Any | `--debug --debug-dir ""` | Debug | None. |
|
||
| `""` | `--debug` | Debug | None. |
|
||
| `""` | `--debug --debug-dir /tmp/investigation` | Debug | `/tmp/investigation`. |
|
||
| Omitted | `--debug --quiet` | Quiet | `/tmp/notarius-debug`. |
|
||
| Omitted | `--quiet --debug` | Debug | `/tmp/notarius-debug`. |
|
||
| Any | `--debug --quiet --debug-dir ""` | Quiet | None. |
|
||
|
||
**Upstream ordering clarification:** CommandKit uses the last enabled reporting
|
||
mode. Thus `--debug --quiet` suppresses command output while retaining capture
|
||
when the effective directory is nonempty; `--quiet --debug` selects debug
|
||
verbosity. Preserve that native ordering rather than making quiet unconditional.
|
||
Quiet still shows failure explanations, and capture does not bypass result
|
||
suppression. On success the normal run report includes the bundle location when
|
||
capture was enabled; failure explanations include an allocated bundle path when
|
||
available.
|
||
|
||
### Native Evidence And A Small Application Recorder
|
||
|
||
Use native facilities directly, attaching application step/lane/chunk/validator
|
||
context where useful:
|
||
|
||
| Facility | Available evidence |
|
||
| --- | --- |
|
||
| `OnPrepared` | Prepared request details when execution reaches that point. |
|
||
| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
|
||
| Returned `RunResult` | Available model output and native validation result. |
|
||
| `Exchange.Details()` | Explicit inspection of captured conversation history. |
|
||
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and cache reuse. |
|
||
|
||
Compose capture and accounting observers. Remove the debug completion wrapper,
|
||
copied execution summaries, synthetic call identities, and duplicate terminal
|
||
reporting. Keep callbacks lightweight and synchronize shared capture state:
|
||
callbacks are synchronous, can overlap across operations, and `OnPrepared` runs
|
||
while admission is held. Retain needed details and persist them at normal
|
||
operation boundaries without requiring an entire pipeline's content in memory.
|
||
Native snapshots may repeat conversation prefixes when that is the convenient
|
||
representation; do not introduce a custom delta format merely to remove them.
|
||
|
||
Capture only available evidence. Preparation/admission failures may have no
|
||
`OnPrepared` notification; observer events have metadata, not response bodies;
|
||
operational failures may return no `RunResult`. Do not present the bundle as a
|
||
complete provider transcript. Use ordinary `Run` with native hooks, without
|
||
forcing split preparation/execution solely for recording. The cache's lookup
|
||
before execution is a separate reason to use native preparation; `OnPrepared`
|
||
notifications alone do not provide cache short-circuiting.
|
||
|
||
Conversation capture required for continuation remains in-process execution
|
||
state. Persisting its history requires enabled debug capture. Cached values are
|
||
accepted domain artifacts, not conversation state or provenance records. The
|
||
separate artifact-first output and completion-report contract is selected in item 4.
|
||
|
||
### Capture Lifecycle And Failure Policy
|
||
|
||
For enabled capture, allocate the bundle after application configuration loads
|
||
and the destination and run identity are known, before engine construction and
|
||
pipeline preflight. Argument and configuration-load failures use native
|
||
diagnostics and may have no bundle. Loaders remain free of storage allocation.
|
||
|
||
After creating the run directory, update `<debug-root>/latest` to a relative
|
||
symlink targeting that run ID before engine construction or preflight starts.
|
||
This points to the newly started debug session, including one that later fails;
|
||
it is not a last-successful-run selector. Runs without capture leave it alone.
|
||
Do not update it again at completion, so an older overlapping run cannot take
|
||
the link back when it finishes. Concurrent starts use the last successful link
|
||
update; individual run directories remain the actual storage locations.
|
||
Treat link-update failures under the requested-capture failure policy below.
|
||
|
||
One application-owned recorder records initial context, accepts native execution
|
||
evidence and application decisions, and finalizes once with the known pipeline,
|
||
publication, and capture outcomes. Debug remains separate from output/cache,
|
||
never supplies cache reuse, and is not automatically deleted.
|
||
|
||
Requested capture is required work:
|
||
|
||
- Allocation failure stops before expensive model work.
|
||
- Recording failure is surfaced at the next normal application boundary.
|
||
- Finalization failure is an operational command failure.
|
||
- An existing pipeline failure remains primary; capture failures are additional
|
||
information and preserve the underlying causes.
|
||
- Already-published artifacts and already-written diagnostic data remain
|
||
available. Capture failure does not cause model retries.
|
||
|
||
Observer callbacks cannot return errors. Retain a recording error for application
|
||
code to check at an ordinary boundary rather than adding an exception or retry
|
||
framework. Submit the successful run report only after publication and requested
|
||
capture finalization succeed. If finalization fails after publication, identify
|
||
the published output and available debug location in the native failure
|
||
explanation; preserve the successful publication as a fact.
|
||
|
||
### Debug Data And Operator Responsibility
|
||
|
||
Debug artifacts are expected to contain internal application data, including
|
||
source material, rendered requests, model output, and correction history where
|
||
useful for diagnosis. If the application data is sensitive, the debug artifacts
|
||
are sensitive in the same way. Operators secure them through the ordinary means
|
||
used for application inputs and outputs, such as filesystem permissions. Do not
|
||
require a special debug-only security or retention model, or use sensitivity
|
||
concerns to limit useful diagnostic content.
|
||
|
||
Do not directly print or serialize values explicitly marked as secrets by
|
||
ConfigKit, such as resolved API keys. Use native secret-aware configuration
|
||
inspection and preserve upstream credential omissions. This boundary does not
|
||
authorize scanning or rewriting LLM output, prompts, source content, or arbitrary
|
||
strings for suspected sensitive values. Do not carry forward heuristic content
|
||
scrubbing or generic secret-detection machinery as a migration requirement.
|
||
|
||
Separate debug/output/cache lifecycles and no automatic debug deletion remain
|
||
application organization choices. Choose capture contents for diagnostic value,
|
||
size, and duplication, rather than promising a shareable or sanitized bundle.
|
||
|
||
### Native Optional Diagnostics
|
||
|
||
Use the selected standard commands through the shared loaders and native inputs:
|
||
|
||
| Command family | Purpose |
|
||
| --- | --- |
|
||
| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. |
|
||
| `pipeline validate` | Check application composition and known dependencies offline. |
|
||
| `promptkit requirements show` | Discover request requirements. |
|
||
| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. |
|
||
| `promptkit test …` | Explicit provider diagnostics. |
|
||
|
||
Application request builders are added only for actual application needs.
|
||
Inspection/diagnostic commands use their native reports and do not automatically
|
||
allocate pipeline debug bundles; operators can redirect reports for retention.
|
||
Neither debug verbosity nor enabled capture initiates additional provider probes.
|
||
Smoke reports retain native support, applicability, and result distinctions;
|
||
status zero alone does not establish that a check passed.
|
||
|
||
### Selected Bundle Organization
|
||
|
||
**Guiding principle:** Capture the most convenient debug representation. Optimize
|
||
for straightforward capture and useful inspection, rather than maximum
|
||
conciseness, completeness, or deduplication. Native values, complete snapshots,
|
||
and self-contained operation records are appropriate even when fields or content
|
||
repeat.
|
||
|
||
"Record each fact once and reference it where needed" remains a practical
|
||
guideline for avoiding independently maintained decisions and accounting, not a
|
||
strict storage constraint. Use shared application outcome facts and native
|
||
execution evidence. Repeating that evidence in useful representations does not
|
||
require another source of truth or justify custom compression/reconstruction.
|
||
|
||
Organize the bundle around the run and its operations, replacing `summary/` and
|
||
`trace/` as separate categories. Use a compact overview for navigation and keep
|
||
detailed evidence with its operation. Summary/trace are not security classes.
|
||
|
||
Illustrative layout: `latest` is the selected link name; other file names and
|
||
exact serialized fields remain implementation choices:
|
||
|
||
```text
|
||
<debug-root>/
|
||
latest -> <run-id>/
|
||
<run-id>/
|
||
run.json
|
||
config/
|
||
application.yml
|
||
promptkit.yml
|
||
pipeline.json
|
||
source.json
|
||
operations/
|
||
<operation-id>/
|
||
...
|
||
```
|
||
|
||
| Component | Purpose |
|
||
| --- | --- |
|
||
| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. |
|
||
| Configuration snapshots | Preserve effective settings through native inspection. |
|
||
| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. |
|
||
| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. |
|
||
| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. |
|
||
|
||
The overview provides navigation, with terminal facts drawn from the same
|
||
application outcome used for the CLI report. Reference detailed records where
|
||
useful; include convenient summary fields without maintaining a second execution
|
||
ledger. Configuration and resolved composition normally need one snapshot each.
|
||
Use native execution IDs with application scope to connect records.
|
||
Reused work has a loaded-artifact snapshot and current cache-hit decision, not
|
||
a fabricated model execution. Historical provenance is not required in cache
|
||
files or reconstructed solely for debug capture.
|
||
|
||
Retain enough available context to investigate a run that fails before publishing
|
||
output. Do not depend entirely on successfully published artifacts or automatically
|
||
copy every original input and published artifact. Retain complete native
|
||
conversation snapshots for successive turns as selected below. Do not build a
|
||
competing transcript model or deduplication system to eliminate repeated history.
|
||
|
||
### Selected Capture Inventory And Depth
|
||
|
||
**Resolved:** Use one default capture depth preserving every application-visible
|
||
turn, its useful inputs and outputs, and the decisions made about it. An operator
|
||
should be able to inspect a failed operation without reconstructing deltas,
|
||
finding a cache entry, or extracting model text from escaped JSON strings.
|
||
Convenience governs both capture and inspection; direct native serialization
|
||
and a simple readable rendering of the same values can both be useful.
|
||
|
||
#### Run Context
|
||
|
||
Retain the run overview, native application and PromptKit configuration
|
||
snapshots, resolved pipeline, and parsed source identified above. Include
|
||
application version, run/pipeline identity, timing, outcome, publication location,
|
||
and navigation to operation records in the overview. Record effective invocation
|
||
selections so request overrides remain visible even when they do not change
|
||
engine configuration.
|
||
|
||
The parsed source snapshot retains the generic document's unit identities and
|
||
content independently of the original input file. Do not automatically archive
|
||
every original configuration, template, or input file; effective snapshots and
|
||
executed requests supply the default evidence. If input parsing fails, retain
|
||
the input location and available parsing diagnostics; a parsed-source snapshot
|
||
may not exist.
|
||
|
||
#### Operation Inputs
|
||
|
||
An operation is meaningful application work: chunk planning, extraction for one
|
||
lane/chunk, merging, normalization, or validation. Identify its stage, lane,
|
||
chunk, role, and relationships to other operations as applicable. Retain an input
|
||
snapshot containing the actual values consumed:
|
||
|
||
- Relevant source/chunk content.
|
||
- Loaded reference content used by the operation.
|
||
- Incoming artifacts.
|
||
- Fixed original candidates and candidate-handle mappings for normalization.
|
||
- Prompt input bindings and application variables where applicable.
|
||
|
||
Include this content even when it also appears elsewhere in the bundle. An
|
||
extraction operation should expose its source and reference context directly.
|
||
Use existing application values and artifact codecs at meaningful boundaries;
|
||
do not invent a parallel representation of the pipeline's entire internal state.
|
||
|
||
#### Model Executions And Conversations
|
||
|
||
For each initial execution, continuation, and LLM validator execution, retain
|
||
the following evidence when available:
|
||
|
||
| Evidence | Representation |
|
||
| --- | --- |
|
||
| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. |
|
||
| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. |
|
||
| Returned result | Native `RunResult` snapshot. |
|
||
| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. |
|
||
| Conversation | Full `Exchange.Details()` snapshot. |
|
||
| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. |
|
||
|
||
Use indented JSON for structured native snapshots and existing artifact formats
|
||
for domain payloads. Readable messages are a straightforward rendering of native
|
||
data, not an independently maintained transcript model. Use the completed
|
||
exchange when available and prepared messages when execution fails before
|
||
returning a result; label which representation is shown.
|
||
|
||
Keep full conversation snapshots for successive turns, including rejected turns.
|
||
Repeated history is acceptable and makes a later turn independently inspectable.
|
||
Enable native exchange capture for debug-enabled executions where needed,
|
||
including single-turn calls. Continuation's in-process exchange requirements
|
||
remain independent of whether durable debug capture is enabled.
|
||
|
||
#### Interpretation, Assessment, And Stage Artifacts
|
||
|
||
Retain beside each response:
|
||
|
||
- The interpreted candidate, serialized through the existing candidate codec
|
||
before semantic acceptance.
|
||
- Decode, interpretation, or safety-check failures when no candidate could be
|
||
constructed.
|
||
- Each configured validator's result, findings, and actionable correction guidance.
|
||
- The application's disposition: accepted, rejected, continued, failed, or
|
||
accepted under the selected fallback/incomplete-validation policy.
|
||
- Links to LLM validator executions that informed the assessment.
|
||
|
||
For normalization, retain the model's reconciliation plan and the candidate
|
||
produced by applying it to the fixed original inputs. Preserve constructed
|
||
correction inputs even if the next execution fails during preparation; successful
|
||
preparation captures the actual rendered correction in its native messages.
|
||
|
||
Retain deterministic stage outputs too: chunk plans and materialized chunks,
|
||
merged artifacts, and normalized artifacts. For cache reuse, retain the loaded
|
||
artifact and current hit decision without requiring historical provenance. These
|
||
records do not fabricate a current execution or new validation of reused work.
|
||
|
||
#### Failed Runs And Native Availability
|
||
|
||
Persist evidence as operations advance under the selected recorder lifecycle;
|
||
do not wait for final publication. On failure, retain available evidence and
|
||
record where execution stopped, the returned error, and the application outcome.
|
||
Distinguish no returned result from a returned result that failed validation.
|
||
A completed response returned through coordination's `Undelivered` field is
|
||
captured as unassessed, without implying acceptance.
|
||
|
||
Internal transport retries and structural repairs receive metadata capture at
|
||
the depth PromptKit exposes. Intermediate response bodies are not required by
|
||
this inventory. Operational errors may return no `RunResult`; record that
|
||
absence clearly. This scope does not require a provider wrapper or an upstream
|
||
feature request. Evaluate a later demonstrated gap on its actual diagnostic need.
|
||
|
||
Keep selected content in full, without arbitrary truncation or a second
|
||
capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and
|
||
native credential omissions; do not scan or scrub application content. Exact
|
||
filenames and serialized application fields remain implementation choices;
|
||
the inventory and capture depth are resolved.
|
||
|
||
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
|
||
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
|
||
[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run),
|
||
and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation).
|
||
The current [debug lifecycle](../internal/state.md#debug-bundles) and
|
||
[operator contract](../operations.md#debug-bundles) provide migration context.
|
||
|
||
## 6. Verification, Migration, And Roadmap Incorporation
|
||
|
||
**Discussion / decision:** Agreed. Complete one coordinated native cutover, with
|
||
CommandKit composition introduced early, verification focused on Notarius's
|
||
integration and domain responsibilities, and practical release migration guidance.
|
||
Incorporate these decisions into the existing master roadmap; another replacement
|
||
document is unnecessary. Implementation and master-roadmap incorporation are pending.
|
||
|
||
### Focused Verification Of Application Behavior
|
||
|
||
Use real CommandKit, ConfigKit, and PromptKit components in representative
|
||
assembled workflows, with a fake at the provider boundary. Keep the default
|
||
suite deterministic, offline, and independent of real credentials. Following the
|
||
[testing policy](../policy/testing.md), assign each behavior an appropriate test
|
||
owner and avoid duplicating upstream parser, renderer, or retry coverage.
|
||
|
||
Select a small set of workflows that collectively establish:
|
||
|
||
- Execution, inspection, and preflight resolve the same intended configuration
|
||
and prompt definitions, including application overrides and companion sources.
|
||
- Help, version, and completion work without application configuration; offline
|
||
preflight makes no provider calls.
|
||
- `run` and `pipeline run` reach the same application behavior, with the selected
|
||
flags, supplied-value distinctions, and overrides mapped correctly.
|
||
- Reports, diagnostics, and exit outcomes follow item 4, distinguishing domain
|
||
rejection, command failure, and presentation failure after publication.
|
||
- A downstream failure followed by another invocation reuses accepted artifacts
|
||
under the revised cache policy. Final corrected artifacts are stored under the
|
||
initial message hash; changed messages miss, settings-only changes can hit,
|
||
and hits neither make model calls nor repeat domain validation. Rejected or
|
||
validation-incomplete work is never stored as an accepted cache value.
|
||
- Debug capture follows item 5's directory selection, reporting-mode independence,
|
||
`latest` update timing, useful evidence, and allocation/write/finalization
|
||
failure behavior.
|
||
|
||
Use subprocess tests where the actual executable boundary adds protection:
|
||
stdout/stderr separation, process exit status, and representative machine output.
|
||
Test domain interpretation, validation, continuation, and recovery at their
|
||
appropriate package boundaries. Preserve meaningful native-execution acceptance
|
||
checks from the master roadmap without repeating them throughout CLI tests.
|
||
|
||
### Retire Tests With Superseded Machinery
|
||
|
||
Review tests of the private adapter, global scheduler, manual continuation
|
||
protocol, broad checkpoint fingerprints, and superseded CLI contracts for removal
|
||
as those implementations retire. Where an application requirement survives, test
|
||
that requirement through its new boundary. Do not mechanically reproduce old
|
||
parser, formatter, or wrapper tests against the new implementation.
|
||
|
||
Avoid freezing complete upstream help output, incidental diagnostic wording, or
|
||
every field of native debug snapshots. Debug verification should establish useful
|
||
evidence and correct failure handling while allowing representations to evolve
|
||
for convenience. Do not require exhaustive provider-attempt reconstruction or
|
||
strict serialization deduplication.
|
||
|
||
Reconcile older debug-sensitivity language, special-handling requirements, and
|
||
heuristic redaction behavior/tests with item 5. Avoid directly disclosing values
|
||
marked as secrets by ConfigKit and preserve native omissions. Do not carry forward
|
||
heuristic scrubbing of prompts, source content, or model output, or special
|
||
debug-data handling requirements. Operators secure debug artifacts as they do
|
||
ordinary application inputs and outputs.
|
||
|
||
### Migration Guidance And Documentation
|
||
|
||
Provide release guidance for operators and subprocess callers, with a practical
|
||
old-to-new migration map covering:
|
||
|
||
- Commands and flags, including the native `run` alias, configuration inspection,
|
||
and `--format`.
|
||
- Application and PromptKit configuration changes, source discovery, profiles,
|
||
and companion files.
|
||
- Native backend admission, independent worker settings, and revised retry controls.
|
||
- Accepted-artifact cache reuse, explicit refresh/bypass, and the selected
|
||
treatment of old checkpoint files.
|
||
- Machine reports, exit outcomes, and the distinction between successful command
|
||
execution and accepted domain results.
|
||
- Debug capture under `/tmp/notarius-debug`, `latest`, and the interaction between
|
||
capture and reporting modes.
|
||
|
||
CLI backward compatibility remains a non-goal; migration guidance does not imply
|
||
compatibility shims or preservation of old command behavior. Legacy checkpoint
|
||
handling and published-artifact compatibility remain explicit application
|
||
decisions, without assuming that old checkpoint envelopes must survive.
|
||
|
||
Maintain runnable examples for embedded defaults and operator overrides. Update
|
||
canonical CLI, configuration, operations, integration/consumer, and internal
|
||
documentation as the corresponding behavior lands. Record major architectural
|
||
changes in successor ADRs, preserving accepted decision text under the
|
||
[documentation policy](../policy/documentation.md#architecture-decision-records).
|
||
Keep future behavior in roadmap documents until implemented.
|
||
|
||
### Incorporation Into The Master Roadmap
|
||
|
||
Replace the deferred CommandKit Phase 7 with adoption throughout the existing
|
||
master roadmap. Use this phase sequence:
|
||
|
||
1. Establish the agreed dependency baseline and remaining contract decisions.
|
||
2. Build native application composition, shared loaders, standard commands, and
|
||
the dedicated preflight package alongside native source assembly.
|
||
3. Prove a representative native pipeline through the actual command surface,
|
||
including reporting and useful debug capture.
|
||
4. Migrate remaining roles and remove the adapter, scheduler, retry machinery,
|
||
and custom CLI infrastructure.
|
||
5. Complete cache migration, the selected output/report contract, and capture behavior.
|
||
6. Verify the assembled application and finish migration documentation.
|
||
|
||
Each phase delivers a reviewable result with appropriate verification. Tests and
|
||
documentation proceed throughout; the final phase checks integration and
|
||
completeness. The phases do not require separate releases or an intermediate
|
||
compatibility framework.
|
||
|
||
Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow
|
||
U9 to item 4's selected artifact-first output/completion contract and item 5's
|
||
debug implementation details within the resolved inventory, depth, and controls.
|
||
Retire earlier receipt/manifest field inventories rather than treating them as
|
||
unresolved requirements to recreate.
|
||
Update U1 for the upstream message-content hash and replace U8's checkpoint
|
||
identity/provenance plan with the accepted-artifact cache decision.
|
||
Reconcile removal inventory and acceptance criteria, including old
|
||
inspection-command and machine-output compatibility assumptions. Preserve
|
||
unresolved schema handling, role-specific retry/fallback policy, worker defaults,
|
||
checkpoint transition choices, and other concrete choices until actually settled.
|
||
|
||
Once incorporated, the master roadmap owns implementation sequencing and status;
|
||
both migration notepads remain discussion records. Do not maintain competing
|
||
implementation checklists in the notepads.
|
||
|
||
### Completion Checks
|
||
|
||
Use focused checks while implementing and the existing repository-wide
|
||
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` checks for the
|
||
completed cutover. Check the affected maintained examples, documentation links,
|
||
dependency attribution, and removal inventory as part of this migration.
|
||
|
||
Live provider smoke tests remain explicitly optional and outside the deterministic
|
||
default suite. These change-specific migration checks do not create additional
|
||
permanent release gates; publication continues to follow the
|
||
[release policy](../policy/release.md). Recording these decisions does not claim
|
||
that implementation acceptance checks have already passed.
|
||
|
||
## Evidence And Reading
|
||
|
||
These sources support the discussion and selected decisions; they do not
|
||
constitute a completed implementation audit. Sibling links assume the local
|
||
repositories remain available.
|
||
|
||
- [Notarius CLI implementation overview](../internal/cli.md) and
|
||
[configuration internals](../internal/configuration.md).
|
||
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md), including
|
||
module selection, typed inputs, raw/formatted output, reporting, and loaders.
|
||
- [ckconfig integration](../../../commandkit/integrations/ckconfig/README.md) and
|
||
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md).
|
||
- [Audita command composition](../../../audita/internal/cli/app_wiring.go), showing
|
||
application handlers and upstream integrations in one CommandKit application.
|