51 KiB
CommandKit Migration Discussion
Purpose And Status
Working notepad for the CommandKit decisions that will be incorporated into the master PromptKit-native migration roadmap. Record the proposal, open questions, evidence, and decisions here as we discuss them.
Direction agreed: Make a full cutover to a CommandKit-native application as part of the same migration as modern PromptKit, in separate, reviewable implementation steps. Adopt CommandKit's public-facing conventions and recommendations to the fullest extent. Bring its design decisions forward and complete adoption before the overall cutover.
The master roadmap currently places CommandKit in a subsequent Phase 7 and leaves its details in U10. That sequencing has not yet been revised. Once the relevant details below are settled, fold them into the master roadmap, including its dependency baseline, implementation phases, removal inventory, and acceptance criteria. No application or dependency changes have been implemented through this discussion.
Item 1 selects the dependency versions and standard capabilities, including
PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target.
Item 2 selects the application command structure and input conventions. Item 3
selects native composition, shared loaders, and application preflight. Item 4
selects native reporting, machine output, and outcomes. Item 5 selects conditional
debug capture through the shared --debug flag, native diagnostics, and a run/
operation bundle with a latest symlink. Capture depth, exact run-report fields,
and verification remain open below.
The PromptKit architecture and checkpoint decisions remain in force. This discussion develops the CommandKit portion of that migration and its interactions with U1 (dependencies), U9 (records/reporting), and U10 (CommandKit adoption).
Agreed Proposal
Guiding Rule: Full Native Cutover
Backward compatibility with the existing CLI conventions is a non-goal. Use CommandKit's native command, input, help/version, reporting, error, and outcome conventions as the design baseline. This mirrors the PromptKit-native direction.
Existing command names, flags, aliases, parsing behavior, report shapes, and exit conventions are migration evidence, not requirements to preserve. Do not add legacy aliases, compatibility shims, parallel parsers, or reporting wrappers solely to reproduce the old CLI. Any application-specific behavior should serve an actual Notarius need and fit the native conventions.
Provide migration guidance in release documentation, explaining the new commands and conventions and the actions operators or subprocess callers must take. Documenting the transition replaces a requirement to keep the old CLI working. Update canonical CLI and consumer documentation when the behavior lands.
Application responsibilities such as producing artifacts, reporting publication outcomes, and managing checkpoint/debug storage remain meaningful. Their existing CLI expression is not protected by an implicit compatibility exception. This guiding rule does not itself choose stored artifact or checkpoint formats; those remain separate application decisions in the master roadmap.
Why Adopt Together
- Source assembly and preflight are already changing. CommandKit integrations can consume the same application and native engine loaders.
- ckconfig and ckprompt can replace standard inspection commands directly, avoiding an intermediate adaptation of commands that would soon be replaced.
- Native execution facts and application outcomes can feed the intended reporting facilities while duplicate LLM bookkeeping is removed.
- Configuration and command migration can be documented and verified as one coherent transition.
The added work is principally CLI composition and contract migration: dispatch, flag parsing, configuration selection, reporting, and process-status mapping. Choose their native forms deliberately and document the transition, rather than spending migration effort recreating existing CLI conventions.
Ownership And Scope
| Concern | Selected owner |
|---|---|
| Command hosting, standard input/help mechanics, common report formatting and visibility | CommandKit. |
| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. |
| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. |
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, checkpoints, and domain outcomes | Notarius application handlers and framework. |
| Receipt content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
Adopt the standard capability suite selected in item 1, including upstream commands and recommendations beyond the existing Notarius surface. Items 2–4 select application inputs, composition, and reporting; the remaining discussion determines debug capture, verification, and concrete record details.
Pipeline and domain code should remain independent of CommandKit. Native PromptKit observations supply execution facts; Notarius attaches application meaning and selects report content. CommandKit supplies presentation mechanics.
Two application needs already identified for deliberate treatment are:
- The current
run --jsonpublishes a receipt after output publication. Item 4 replaces its presentation with one compact application report rendered natively for humans and machines. Exact fields/versioning remain with U9; reports must describe the actual publication and domain outcome. - Item 5 uses native
--debugfor diagnostic verbosity and for capture when an effective debug directory is available, including an application default. Capture selection remains independent of the final reporting mode.
Intended Sequencing
- Incorporate items 1–2's selected dependency baseline, standard capabilities, and native application command design, bringing U10 forward.
- Introduce item 3's CommandKit composition alongside native source assembly. Connect shared loaders, standard inspection, and the focused application preflight package.
- Develop native execution and accounting against the intended CLI boundary, with pipeline and domain tests independent of CommandKit.
- Complete command and reporting migration before final end-to-end verification and migration documentation.
This is the agreed sequencing direction, not a replacement phase plan. Concrete work units and their placement in the master roadmap follow the discussion below.
1. Adoption Scope And Dependency Baseline
Discussion / decision: Versions and standard capability scope agreed. Adopt the set below, including the move to PromptKit v0.23.3. Items 2–3 select application commands and composition; subsequent topics address reporting and debug capture. These are selected migration targets; the application dependencies have not changed.
Selected Versions
| Module | Version | Role |
|---|---|---|
go.maximumdirect.net/commandkit |
v0.7.0 |
Native command declarations, inputs, help/version, reporting, and completion. |
go.maximumdirect.net/ckconfig |
v0.9.0 |
Standard configuration commands and ConfigKit diagnostic classification. |
go.maximumdirect.net/ckprompt |
v0.10.0 |
Standard PromptKit inspection, validation, preparation, and smoke diagnostics. |
go.maximumdirect.net/configkit |
v1.2.1 |
Shared configuration dependency for the application and integrations. |
go.maximumdirect.net/promptkit |
v0.23.3 |
Native model execution, including corrected timeout/cancellation reporting. |
The CommandKit integration releases form a documented compatible set using PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout attribution and cancellation/error identity fixes. It retains the same dependencies and requires no configuration or request API migration from v0.23.2. The Go consumer minimum remains 1.27. Verify the assembled dependency set through the migration's normal integration checks and update attribution when the dependency changes land.
The version review used locally available release tags and tagged dependency manifests. Remote release lookup was unavailable, so it did not establish whether newer tags existed remotely. These are deliberate selected versions, not a claim that every module is the latest published release.
Evidence: integration version guidance and PromptKit v0.23.3 release notes.
Selected Standard Capabilities
| Area | Adopt in the cutover |
|---|---|
| Core CLI | Native help, version, typed flags and operands, supplied-value semantics, standard outcomes, and command declaration/dispatch. |
| Reporting | Native --format with text/YAML/JSON, --quiet, --verbose, --debug, and --progress, including upstream visibility and failure behavior. |
| Configuration | config scopes, validate, show, sources, explain, and diff, for application and PromptKit configuration. |
| Resource discovery | PromptKit listings for prompts, profiles, backends, namespaces, and collections, with native filters and selectors. |
| Resource inspection | Native prompt, profile, and backend reports, plus requirements discovery. |
| Concrete request checks | Requirements validation and prepared-request preview using native inputs, variables, profiles, and execution overrides. |
| Provider diagnostics | Reachability, authentication, text-inference, structured-output, and aggregate smoke tests. |
| Convenience features | Bash, Zsh, and Fish completion, useful declaration hints, and upstream PromptKit convenience aliases. |
Use the upstream CLI conventions directly. Providing the standard capability suite means supplying application loaders and callbacks, not recreating those commands locally. Upstream convenience aliases are selected native features; this does not reintroduce legacy Notarius aliases.
Wiring Boundaries And Follow-Up
- Keep configuration validation, requirements discovery, and concrete request validation distinct. Pipeline preflight remains application-owned. Concrete preparation may read inputs and schemas and check credentials even though it makes no model call.
- Install smoke commands for explicit invocation. Their availability does not introduce provider calls during startup, ordinary validation, or offline tests.
- Enable application configuration overrides through ConfigKit. The selected
PromptKit configuration scope does not support arbitrary
--setoverrides; use supported request-level controls rather than implementing a local equivalent. - Retain application handlers for pipeline execution and discovery, with native commands, inputs, and result presentation to be designed in items 2–5.
No upstream feature request has been identified as a blocker for this scope. Evaluate any later demonstrated gap on its actual application need. On roadmap reconciliation, incorporate these versions into U1, replace U10's open standard capability selection with this decision, and retain the unresolved application wiring and presentation details explicitly.
2. Native Command Surface And Inputs
Discussion / decision: Command structure and input conventions agreed. Use a
small application-owned pipeline family alongside the standard config and
promptkit families. These are migration targets, not implemented commands.
Command Structure
notarius pipeline list
notarius pipeline show <pipeline-id>
notarius pipeline validate <pipeline-id>
notarius pipeline run <pipeline-id> --input-file <path>
notarius run <pipeline-id> --input-file <path>
notarius config scopes
notarius config validate|show|sources|explain|diff ...
notarius promptkit list ...
notarius promptkit show ...
notarius promptkit requirements show|validate ...
notarius promptkit test <kind>
notarius version
notarius completion bash|zsh|fish
notarius help [command...]
The alternatives separated by | above summarize the installed operations;
they are not literal shell commands.
| Application command | Responsibility |
|---|---|
pipeline list |
Discover configured pipelines. |
pipeline show |
Describe the resolved ordered steps, lanes, modules, reference slots, and configured selections. |
pipeline validate |
Perform offline application preflight using the intended selections and overrides. |
pipeline run |
Execute the selected pipeline and publish artifacts. |
run |
Native convenience alias for pipeline run. |
Move pipeline-specific validation out of config validate --pipeline ....
Configuration validation, pipeline composition checks, and concrete PromptKit
request validation remain distinct. Pipeline preflight does not synthesize stage
requests or require provider credentials. PromptKit's concrete validation may
read inputs and schemas and check credentials without making a model call.
Naming decision: Keep pipeline run as the canonical command and install
run as a native convenience alias using
commandkit.Alias("run", "pipeline run"). Both routes use the same handler,
operands, flags, validation, and reporting. CommandKit supplies alias-aware help
and diagnostics; no forwarding handler or additional parser is needed.
The grouped family and root shortcut are application usability choices, not an
upstream naming requirement. The alias uses the new input conventions and does
not preserve legacy flags or behavior. Use notarius run in quick-start examples
and document both routes in the command reference when the migration lands.
See native alias behavior.
Input Conventions
Use a required pipeline-ID operand and a required --input-file PATH for runs.
Use native typed declarations and supplied-value semantics, including explicit
empty, false, and zero values. Apply defaults only when the relevant value is
omitted; application validation still decides whether a supplied value is valid.
| Concern | Selected convention |
|---|---|
| Configuration | Native --config and --overlay; name the application scope app. |
| Lane selection | Repeatable --lane ID; omission selects the normal pipeline lane set. Replace comma-separated --only. |
| Output placement | Application-owned --output-dir PATH. |
| Chunk-plan caching | --chunk-cache auto|bypass|refresh. |
| Checkpoint reuse | Keep --resume and --recompute-step ID with the agreed recovery policy. |
| Presentation | Native --format, --quiet, --verbose, --debug, and --progress. |
| Configuration-valued tuning | Supported application --set POINTER=JSON overrides, including worker and semantic-budget settings once their configuration design is settled. |
Adopt native parsing rules: flags follow the complete command path and may be
interspersed with operands; -- ends option parsing. Singleton flags use the
last successfully parsed occurrence, with every occurrence checked. Repeatable
flags preserve occurrence order and treat commas as ordinary value content.
Use filesystem and choice completion hints where applicable.
Keep worker counts and semantic continuation budgets in configuration rather than introducing additional dedicated CLI controls now. Their field names, defaults, and domain meaning remain master-roadmap decisions. Item 3 selects shared loading and override wiring across application and standard commands.
Native LLM Controls
Use ckprompt's reusable flag declarations for applicable profile, session, and
execution controls on pipeline run, including model, temperature, timeout,
reasoning effort, and transient retries. Run-wide overrides reach all LLM-backed
producers and validators, including continuation calls. Without an override,
normal binding/pipeline/prompt selection applies.
| Reasoning input | Meaning |
|---|---|
| Omitted | Inherit the selected settings. |
--llm-reasoning-effort high |
Replace the inherited setting. |
--llm-reasoning-effort "" |
Clear the inherited setting. |
Remove the separate --clear-reasoning-effort control. Native transient retries
remain distinct from application semantic continuation budgets. Individual
prompt selection and named prompt input injection belong on standard PromptKit
commands; pipeline modules construct their own requests and response contracts.
Use native --llm-profile and --llm-session-id spellings, without inventing
another LLM input parser.
Application Reference Inputs
Retain repeatable --reference SELECTOR=PATH and --without-reference SELECTOR.
Keep useful pipeline-wide, lane, chunk, and stage-specific selectors, such as
party=party.txt and npc-registry.extract.party=npc-party.txt. CommandKit owns
repeatable input parsing and typed conversion; Notarius owns selector expansion,
specificity, required slots, and external/generated-reference conflict checks.
These selectors describe pipeline relationships and are not interchangeable
with PromptKit's individual-request --llm-input-file bindings.
Preparing Pipeline Inspection Content
Follow-up clarification: Leverage ConfigKit for the composed, decoded configuration, defaults, imports, overlay, inheritance, overrides, and native provenance/redacted inspection. Feed that loaded configuration through the same application resolution used by validation and execution to obtain selected lanes, registered modules, validator chains, and reference relationships. CommandKit renders a small application report of that resolved composition.
ConfigKit does not know Notarius's module catalog or pipeline semantics. Reuse
the existing resolver rather than writing a second resolution path for show.
Generic effective configuration and source explanations remain available through
config show and config explain. ConfigKit's safe inspection facilities do not
automatically redact an arbitrary application report; select appropriate report
content rather than dumping the raw typed configuration. Item 3 selects loader
wiring; detailed report content remains with item 4.
Removals And Remaining Work
Retire custom dispatch, argument-count handling, assembled help, version-command
parsing, standard configuration/LLM flag parsing, comma-separated lane parsing,
and the separate reasoning-clearing machinery. Replace config effective with
native config show, config profile with promptkit show profile, and bespoke
--json switches with --format json.
Application handlers still validate meaningful input combinations and translate
typed inputs into pipeline selections. Native --debug supplies reporting
behavior and conditional artifact capture as selected in item 5.
Items 3–4 own loader behavior, exact report content, and outcome presentation.
Evidence: CommandKit command/input conventions, reusable PromptKit flags, ConfigKit loading and inspection, and the current Notarius configuration boundaries. Use the current CLI reference for migration guidance, not as a compatibility constraint.
3. Application Composition, Loaders, And Preflight
Discussion / decision: Agreed. Fully adopt native ConfigKit, PromptKit, and
CommandKit workflows through a small composition root, shared loaders, and
invocation-local state. Follow substantially the same composition/loading pattern
as Audita while retaining Notarius's pipeline, artifact, validation, recovery,
and publication responsibilities. Centralize application prompt/profile
dependency checks in a focused internal/preflight package. Implementation is
pending.
Native Application Composition
One application constructor installs the pipeline declarations and native run
alias, a ckconfig suite with app and promptkit scopes, the selected ckprompt
capabilities, and native help/version/completion/reporting. Supply ordinary
callbacks and typed inputs; do not introduce another CLI framework or LLM wrapper.
Constructing the application registers declarations without loading configuration, building an engine, or initializing run storage. Help, version, and completion remain independent of runtime configuration and provider work. Application handlers acquire only what their operation requires.
Shared Configuration And Engine Loaders
Use two shared loading responsibilities across application and standard commands:
- Application configuration: Select explicit
--config, then the existing environment/discovery policy; apply native--overlayand supported application--setoverrides. ConfigKit owns composition, imports, inheritance, presence, provenance, and inspection. Supply application defaults, normalization, and structural validation through its extension points. Retain the native ConfigKit result and selected main-source context; derive runtime values where needed without copying inspection machinery into another result abstraction. - PromptKit engine: Build a native
*promptkit.Enginefrom that loaded application's source context and the agreed native source assembly. Runtime, preflight, and inspection use the same builder and definitions.
Preserve supplied-value information throughout loading. An explicitly empty
configuration path receives deliberate validation rather than silently falling
through to discovery. Across application and PromptKit commands, --config
selects the Notarius application configuration; the PromptKit scope inspects the
engine assembled for that application selection.
Retain the master roadmap's source assembly: PromptKit's bundled underlay, embedded Notarius resources, then optional operator companions beside the main application source actually selected by ConfigKit. Use native imports and filesystem contexts. There is no second source search or working-directory fallback; strict composition, missing-companion handling, and load-error behavior remain as selected in the master roadmap. Do not reinterpret the application document as a standalone PromptKit configuration or add another inspection-only assembly path.
Application --set overrides flow through ConfigKit's load. Native request
execution overrides remain request controls; they do not mutate captured engine
configuration. Preserve the selected PromptKit configuration scope's lack of
arbitrary --set support.
Invocation State And Engine Ownership
Use CommandKit's supported invocation state to retain configuration and an engine when callbacks need them repeatedly. Application handlers use the same loader functions with small invocation-owned state. One pipeline execution shares one native engine across producers, validators, and continuations, preserving native backend admission limits.
Do not assume one invocation always means one configuration selection. Native
config diff acquires left and right selections with potentially different
configuration, overlays, and overrides while sharing invocation state. Load the
sides independently or distinguish selections in any local reuse. A single
unqualified cached result would be incorrect. No persistent loader cache or
cryptographic configuration identity is needed.
Shared Pipeline Resolution And Command Work
Reuse the existing application resolver with loaded configuration, selected pipeline/lanes/reference changes, applicable overrides, and the module catalog. It owns module/options validation, typed artifact compatibility, validator-chain selection, ordered dependencies, and reference-binding compatibility.
| Command | Required application work |
|---|---|
pipeline list |
Load application configuration and enumerate pipelines. |
pipeline show |
Resolve the selected pipeline and report its composition. |
pipeline validate |
Resolve the pipeline and check its known native resource dependencies offline. |
pipeline run / run |
Use the same resolution and preflight checks, then prepare and execute the pipeline. |
config … --scope app |
Supply the native ConfigKit result directly to the integration. |
| PromptKit inspection commands | Acquire the shared engine assembly and delegate to native inspection. |
Run handlers call shared application functions directly; they do not invoke the
validation command or parse its output. pipeline show can describe configured
prompt/profile selections without constructing an engine just to expand them
into execution settings or performing every execution-readiness check.
Prompt Dependencies And Offline Preflight
Notarius knows which operations need prompts; PromptKit knows whether those definitions resolve in the assembled engine. Check every statically known prompt dependency of the selected pipeline, including LLM-backed validators and any separately selected correction prompts. A deterministic stage may require none, an LLM-backed stage may require several, and multiple stages may share one. Configuration can select the required set. For data-dependent selections, check what is knowable before execution and resolve the final choice at runtime.
Share prompt identifiers and selection logic with execution through module or operation declarations. Do not maintain a separate hardcoded preflight inventory. These small dependency declarations identify required resources; they do not restore asset manifests, prompt filesystem registration, message assembly, or schema registries. Native sources remain the owners of prompt definitions.
Use the same engine and intended namespaces as execution to inspect known prompts and applicable profile selections, including a known prompt's default profile when effective. Partial profiles alone are not failures. Report missing dependencies with pipeline, step/lane, stage or validator, and resource context where applicable.
Offline preflight does not read the transcript, materialize reference contents,
synthesize stage requests, require credential values, or contact providers.
Engine assembly may read configuration and prompt resources. Availability does
not certify prompt domain correctness or guarantee that future requests pass
schema, credential, or provider checks. Concrete Prepare/ValidateRequest
remains the boundary for completed request validation, and smoke commands remain
explicit provider operations.
Dedicated Preflight Package
Add a small application-level internal/preflight package accepting the resolved
pipeline, relevant module declarations, and the native engine. It checks the
pipeline's known prompt/profile dependencies and returns contextual findings or
errors for the command boundary to present.
Keep configuration validation in ConfigKit/application definitions, structural resolution in the pipeline resolver, and concrete preparation in PromptKit. Preflight does not discover configuration, register production modules, construct engines, prepare synthetic requests, initialize storage, or own CLI reporting. The composition root supplies its collaborators. Module dependency declarations remain in existing module contracts; modules do not import the preflight package. Exact function signatures and finding representation remain implementation choices.
Request Construction, Storage, And Removals
Use native request builders directly for standard PromptKit commands where sufficient. Add an application callback only for a demonstrated application need, preserving supplied operator values. Pipeline modules construct their own domain requests; share concrete request-building functions when useful without simulating pipeline execution to populate an inspection command.
Output publication, checkpoints, debug capture, and execution records belong to the run path. Loaders do not initialize those resources; debug allocation timing remains with item 5. Remove command-specific loading paths, copied configuration and resource inspection, eager runtime setup, and duplicated preflight orchestration. No broader application/package reorganization is required by this decision.
Evidence: native suite and loader ownership, PromptKit integration callbacks, Audita composition, and the existing configuration ownership decision.
4. Reports, Machine Output, And Outcomes
Discussion / decision: Agreed. Use native CommandKit reports for command results, native outcomes for command completion, and a compact application-owned run report for artifact discovery and domain status. Distinguish pipeline results, command completion, and presentation delivery. Implementation is pending.
One Report Value For Human And Machine Output
Application commands submit ordinary values through Reporter.Report and use
native --format text|yaml|json rendering. Execution supplies one run-report
value with a concise text template for default human output; JSON and YAML render
the same structured value. Do not maintain separate human-summary and JSON-receipt
construction paths.
notarius run dnd-session --input-file transcript.json
notarius run dnd-session --input-file transcript.json --format json
Standard configuration and PromptKit commands retain their upstream report shapes
and behavior. Do not wrap them in a universal Notarius result envelope. No current
run-report need requires raw output; reserve native Output/RenderOutput for an
operation whose actual product needs a byte stream.
Compact Application Run Report
CommandKit owns presentation; Notarius owns the meaning and deliberate contract of the application run report. Retain these content categories:
| Content | Purpose |
|---|---|
| Application report version | Identify the report contract for consumers. |
| Run and pipeline identity | Identify completed work. |
| Published bundle directory and applicable index path | Locate artifacts without guessing filenames. |
| Accepted/normalized and rejected output counts | Summarize production results. |
| Validation completeness/status | Distinguish fully validated results from permitted incomplete validation. |
| Process-warning counts | Expose degradation independently of reporting verbosity. |
| Debug location, when applicable | Locate requested diagnostic artifacts. |
Detailed per-producer validation records, rejection reasons, and execution provenance belong in the published bundle rather than duplicated collections in the command report. Do not serialize the entire internal pipeline result or complete upstream runtime types as a durable contract. U9 will select exact fields, versioning, and durable record formats. The old CLI receipt's shape is migration evidence, not a requirement to reproduce it.
Publication, Streams, And Machine Consumption
The run command executes the pipeline, publishes the output bundle, completes required run finalization, submits its successful report, and returns a native outcome. The report describes an actual published result. Item 5 requires requested capture to finalize successfully before a successful run report is submitted and defines how capture failures preserve publication facts.
Use one primary run-result report, with progress and diagnostics separate. Wire
CommandKit's result destination to stdout and its diagnostic destination to
stderr. Subprocess consumers request --format json, capture streams separately,
check the exit status, and decode the successful run report. A report-write
failure may leave partial stdout even though publication already succeeded.
These run-consumption rules do not replace the native report behavior of other
commands, including configuration commands that can report failed validation.
Native Visibility And Message Categories
| Native facility | Application use |
|---|---|
Report |
Requested command result. |
Progress |
Pipeline progress. |
Warn |
Actionable process degradation. |
Info |
Supplementary operational information. |
Debug |
Diagnostic detail, distinct from durable artifact capture. |
| Outcome explanation | Terminal failure or usage guidance. |
Adopt native visibility rules directly: ordinary mode suppresses warning messages,
and quiet mode suppresses command results. Include validation status and
process-warning counts in the primary run report and its human summary so those
facts do not depend on warning-message visibility. Operators request --verbose
for warning details; machine callers expecting a report do not request --quiet.
Use native progress selection and failure visibility without custom bypasses.
Quiet suppression is not a delivery failure; native rendering errors can still
fail a quiet invocation.
Artifact-quality advisories retain their application classification. Do not reclassify them as process warnings merely to make them visible.
Native Outcomes And Domain Acceptance
Adopt the standard process-status conventions:
| Status | Meaning |
|---|---|
0 |
The command completed successfully. |
1 |
Operational failure, cancellation, publication failure, or presentation failure. |
2 |
Invalid invocation or identifiable invalid configuration input. |
3 |
Native config diff --check found differences. |
Use ckconfig's structured configuration-error classification. Do not classify every configuration failure as operational or every preflight failure as invalid input. Invalid operator selections and missing bundled application resources can have different causes; preserve that distinction and native error identity.
Command completion remains separate from domain acceptance. A successfully published run can contain recorded rejections or validation-incomplete results allowed by application policy; those facts appear in reports and artifacts and do not automatically fail the command. A terminal pipeline failure does fail the command. Native status zero also does not replace reading standard PromptKit reports for partial discovery or unsupported smoke checks.
Application handlers supply useful native outcome explanations and retain the underlying error through native cause handling. CommandKit presents the terminal result. Remove duplicate terminal-error printing and custom outcome wrappers.
Delivery Failures, Progress, And Accounting
Use CommandKit's native separation of operation causes from output and diagnostic delivery failures. If publication succeeds but stdout fails, the final command status reflects presentation failure while the published artifacts remain a fact. Do not rewrite pipeline history as failed execution or automatically rerun model work because report delivery failed.
Notarius owns stage/lane/chunk progress, domain acceptance, and checkpoint reuse.
PromptKit supplies native execution observations and ExecutionGroup totals;
CommandKit presents the selected facts. Explicitly enroll the relevant initial,
continuation, and validator executions without double counting. Checkpoint reuse
does not add historical token usage to current execution totals, and unavailable
usage remains distinguishable from zero.
Remove bespoke format selection, parallel receipt encoders, custom verbosity handling, duplicate execution accounting, and superseded terminal-reporting machinery. Retain only application report content, publication facts, domain decisions, and small deliberate durable projections.
Evidence: native reporting, visibility and delivery, and outcome classification. The current run-result contract and CLI failure mapping document migration impact. Item 6 of the PromptKit discussion record retains the bookkeeping boundary; U9 still owns concrete durable record choices.
5. Debug Capture And Optional Diagnostics
Discussion / decision: Controls, native capture hooks, lifecycle, failure
policy, and optional diagnostics agreed. Use --debug for native verbosity and
durable capture with /tmp/notarius-debug as the default root. Operator-owned
data handling and recording each fact once are agreed. Replace the summary/trace
hierarchy with a compact run index, configuration/composition snapshots, and
operation records; maintain a latest symlink at the debug root. Capture depth
and concrete serialized fields remain under discussion. Implementation is pending.
Shared Debug Control With Conditional Capture
Attach a native application boolean binding to CommandKit's shared --debug
option on the pipeline run command. Capture is enabled when that boolean is true
and an effective debug directory is available. --debug-dir PATH overrides
debug.directory in application configuration, whose default is
/tmp/notarius-debug. A directory alone does not enable capture. Do not add a
separate --capture-debug flag.
Default-directory decision: Use the absolute root /tmp/notarius-debug,
with each run's bundle beneath it. Temporary diagnostic output belongs under
/tmp or an operator-selected location under /var, rather than beside a
configuration file likely to live under /etc. Replace the old configuration-
relative ./notarius-debug default. Precedence is CLI override, configured value,
then application default. Explicit relative configuration paths retain their
source-relative interpretation; relative CLI paths use the invocation directory.
Omitting both directory overrides selects the default, so --debug enables
verbosity and capture without requiring a path. Detailed explicit-empty handling
and any demonstrated need for verbosity without capture remain separate choices.
Determine capture from the application boolean and destination, not from
Invocation.ReportingMode(). A later quiet or verbose selection can change
presentation without clearing the debug input. Explicit --debug=false follows
native boolean binding semantics and disables capture.
| Invocation flags | Reporting selection | Capture |
|---|---|---|
No --debug, with any directory selection |
Selected native mode | None. |
--debug, no directory override |
Debug | Enabled under /tmp/notarius-debug. |
--debug, with a directory override |
Debug | Enabled under the selected root. |
--debug --quiet, with any directory selection |
Quiet | Enabled. |
--quiet --debug, with any directory selection |
Debug | Enabled. |
Upstream ordering clarification: CommandKit uses the last enabled reporting
mode. Thus --debug --quiet suppresses command output while retaining capture;
--quiet --debug selects debug verbosity. Preserve that native ordering rather
than making quiet unconditional. Quiet still shows failure explanations, and
capture does not bypass result suppression. On success the normal run report
includes the bundle location; failure explanations include an allocated bundle
path when available.
Native Evidence And A Small Application Recorder
Use native facilities directly, attaching application step/lane/chunk/validator context where useful:
| Facility | Available evidence |
|---|---|
OnPrepared |
Prepared request details when execution reaches that point. |
Observer |
Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
Returned RunResult |
Available model output and native validation result. |
Exchange.Details() |
Explicit inspection of captured conversation history. |
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and checkpoint reuse. |
Compose capture and accounting observers. Remove the debug completion wrapper,
copied execution summaries, synthetic call identities, and duplicate terminal
reporting. Keep callbacks lightweight and synchronize shared capture state:
callbacks are synchronous, can overlap across operations, and OnPrepared runs
while admission is held. Retain needed details and persist them at normal
operation boundaries rather than buffering an entire pipeline or repeatedly
writing every accumulated conversation prefix.
Capture only available evidence. Preparation/admission failures may have no
OnPrepared notification; observer events have metadata, not response bodies;
operational failures may return no RunResult. Do not present the bundle as a
complete provider transcript. Use ordinary Run with native hooks, without
forcing split preparation/execution solely for recording.
Conversation capture required for continuation remains in-process execution state. Persisting its history requires enabled debug capture. Normal manifests, checkpoints, and run reports retain their minimal projections independently of debug selection.
Capture Lifecycle And Failure Policy
For enabled capture, allocate the bundle after application configuration loads and the destination and run identity are known, before engine construction and pipeline preflight. Argument and configuration-load failures use native diagnostics and may have no bundle. Loaders remain free of storage allocation.
After creating the run directory, update <debug-root>/latest to a relative
symlink targeting that run ID before engine construction or preflight starts.
This points to the newly started debug session, including one that later fails;
it is not a last-successful-run selector. Runs without capture leave it alone.
Do not update it again at completion, so an older overlapping run cannot take
the link back when it finishes. Concurrent starts use the last successful link
update; individual run directories remain the actual storage locations.
Treat link-update failures under the requested-capture failure policy below.
One application-owned recorder records initial context, accepts native execution evidence and application decisions, and finalizes once with the known pipeline, publication, and capture outcomes. Debug remains separate from output/cache, never supplies checkpoint reuse, and is not automatically deleted.
Requested capture is required work:
- Allocation failure stops before expensive model work.
- Recording failure is surfaced at the next normal application boundary.
- Finalization failure is an operational command failure.
- An existing pipeline failure remains primary; capture failures are additional information and preserve the underlying causes.
- Already-published artifacts and already-written diagnostic data remain available. Capture failure does not cause model retries.
Observer callbacks cannot return errors. Retain a recording error for application code to check at an ordinary boundary rather than adding an exception or retry framework. Submit the successful run report only after publication and requested capture finalization succeed. If finalization fails after publication, identify the published output and available debug location in the native failure explanation; preserve the successful publication as a fact.
Debug Data And Operator Responsibility
Debug artifacts are expected to contain internal application data, including source material, rendered requests, model output, and correction history where useful for diagnosis. If the application data is sensitive, the debug artifacts are sensitive in the same way. Operators secure them through the ordinary means used for application inputs and outputs, such as filesystem permissions. Do not require a special debug-only security or retention model, or use sensitivity concerns to limit useful diagnostic content.
Do not directly print or serialize values explicitly marked as secrets by ConfigKit, such as resolved API keys. Use native secret-aware configuration inspection and preserve upstream credential omissions. This boundary does not authorize scanning or rewriting LLM output, prompts, source content, or arbitrary strings for suspected sensitive values. Do not carry forward heuristic content scrubbing or generic secret-detection machinery as a migration requirement.
Separate debug/output/cache lifecycles and no automatic debug deletion remain application organization choices. Choose capture contents for diagnostic value, size, and duplication, rather than promising a shareable or sanitized bundle.
Native Optional Diagnostics
Use the selected standard commands through the shared loaders and native inputs:
| Command family | Purpose |
|---|---|
config …, promptkit list/show … |
Inspect configuration and stored resources. |
pipeline validate |
Check application composition and known dependencies offline. |
promptkit requirements show |
Discover request requirements. |
promptkit requirements validate, promptkit show prepared |
Validate or preview a concrete request without a model call. |
promptkit test … |
Explicit provider diagnostics. |
Application request builders are added only for actual application needs. Inspection/diagnostic commands use their native reports and do not automatically allocate pipeline debug bundles; operators can redirect reports for retention. Neither debug verbosity nor enabled capture initiates additional provider probes. Smoke reports retain native support, applicability, and result distinctions; status zero alone does not establish that a check passed.
Selected Bundle Organization
Agreed principle: Record each fact once and reference it where needed. Use shared application outcome facts and native execution evidence rather than parallel summary/trace accounting or repeated snapshots.
Organize the bundle around the run and its operations, replacing summary/ and
trace/ as separate categories. Use a compact overview for navigation and keep
detailed evidence with its operation. Summary/trace are not security classes.
Illustrative layout: latest is the selected link name; other file names and
exact serialized fields remain implementation choices:
<debug-root>/
latest -> <run-id>/
<run-id>/
run.json
config/
application.yml
promptkit.yml
pipeline.json
operations/
<operation-id>/
...
| Component | Purpose |
|---|---|
| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. |
| Configuration snapshots | Preserve effective settings through native inspection. |
| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. |
| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. |
The overview is an index rather than a second execution ledger. Its terminal facts come from the same application outcome used for the CLI report, and it references detailed records instead of duplicating requests, responses, findings, or error history. Capture configuration and resolved composition once when available. Use native execution IDs with application scope to connect records. Reused work has a checkpoint/reuse decision and reference, not a fabricated model execution.
Retain enough available context to investigate a run that fails before publishing output. Do not depend entirely on a successful output manifest or automatically copy every original input and published artifact. Select a conversation representation that avoids repeatedly saving the complete accumulated history after each turn or maintaining an independent competing transcript.
Open Discussion: Capture Depth
The organization is selected; precise evidence and granularity remain open:
- Which prepared request details, outputs, intermediate artifacts, validation decisions, and correction content to retain.
- How to represent conversation history without repeated prefixes while using PromptKit's native evidence.
- How much failed-run context to retain independently, and when references to existing inputs, checkpoints, or published artifacts suffice.
- Which desired evidence is available through native public hooks and which gaps would justify an upstream feature request.
Evidence: native shared debug binding, reporting-mode ordering, PromptKit observations, and exchange inspection. The current debug lifecycle and operator contract provide migration context.
6. Verification, Migration, And Roadmap Incorporation
Discussion / decision: Pending.
- Which focused command and subprocess tests protect the new native command, output, outcome, receipt, and debug behavior? Which tests of superseded CLI contracts and parser/formatter implementation can retire?
- Which representative assembled workflows prove that execution, inspection, and preflight use the same intended configuration without duplicating lower-level coverage? Keep the default suite offline and independent of real credentials.
- How should CommandKit steps replace the master roadmap's deferred Phase 7, and which migration examples, documentation, and ADRs need coordinated updates?
- What release-documentation guidance will show operators and subprocess callers how to migrate their workflows and integrations to the new conventions?
- Reconcile older debug-sensitivity language, special-handling requirements, and heuristic redaction behavior/tests with item 5's clarified boundary. Do not reintroduce them as implicit requirements while migrating documentation or code.
After discussion, reconcile the master roadmap's U1/U9/U10 entries, sequencing, removals, and acceptance criteria, including its treatment of existing inspection commands and machine-output compatibility. Carry forward the explicit non-goal of CLI backward compatibility and release migration guidance. Preserve other unresolved choices explicitly. Then mark this document as the discussion record and keep implementation status in the master roadmap.
Evidence And Reading
These are starting points for discussion, not a completed CommandKit migration audit. Sibling links assume the local repositories remain available.
- Notarius CLI implementation overview and configuration internals.
- CommandKit consumer guide, including module selection, typed inputs, raw/formatted output, reporting, and loaders.
- ckconfig integration and ckprompt integration.
- Audita command composition, showing application handlers and upstream integrations in one CommandKit application.