diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md index aecd744e..158dc1fa 100644 --- a/docs/roadmap/commandkit-migration-discussion.md +++ b/docs/roadmap/commandkit-migration-discussion.md @@ -23,8 +23,9 @@ Item 1 selects the dependency versions and standard capabilities, including PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target. Item 2 selects the application command structure and input conventions. Item 3 selects native composition, shared loaders, and application preflight. Item 4 -selects native reporting, machine output, and outcomes. Exact run-report fields, -debug capture, and verification choices remain open below. +selects native reporting, machine output, and outcomes. Item 5 selects conditional +debug capture through the shared `--debug` flag and native diagnostics. Debug +content/organization, exact run-report fields, and verification remain open below. The PromptKit architecture and checkpoint decisions remain in force. This discussion develops the CommandKit portion of that migration and its interactions @@ -96,9 +97,9 @@ Two application needs already identified for deliberate treatment are: replaces its presentation with one compact application report rendered natively for humans and machines. Exact fields/versioning remain with U9; reports must describe the actual publication and domain outcome. -- The current `--debug` requests durable artifacts. Retain the distinction between - artifact capture and reporting verbosity, while choosing native controls and - meanings rather than requiring that flag to retain its old behavior. +- Item 5 uses native `--debug` for diagnostic verbosity and for capture when a + debug directory is supplied through CLI or configuration. Capture selection + remains independent of the final reporting mode. ### Intended Sequencing @@ -329,7 +330,7 @@ native `config show`, `config profile` with `promptkit show profile`, and bespok Application handlers still validate meaningful input combinations and translate typed inputs into pipeline selections. Native `--debug` supplies reporting -behavior; a distinct artifact-capture control remains to be selected in item 5. +behavior and conditional artifact capture as selected in item 5. Items 3–4 own loader behavior, exact report content, and outcome presentation. Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax), @@ -549,8 +550,9 @@ migration evidence, not a requirement to reproduce it. The run command executes the pipeline, publishes the output bundle, completes required run finalization, submits its successful report, and returns a native -outcome. The report describes an actual published result. Item 5 will settle the -treatment of requested debug-finalization failures. +outcome. The report describes an actual published result. Item 5 requires +requested capture to finalize successfully before a successful run report is +submitted and defines how capture failures preserve publication facts. Use one primary run-result report, with progress and diagnostics separate. Wire CommandKit's result destination to stdout and its diagnostic destination to @@ -641,18 +643,160 @@ retains the bookkeeping boundary; U9 still owns concrete durable record choices. ## 5. Debug Capture And Optional Diagnostics -**Discussion / decision:** Pending. +**Discussion / decision:** Controls, native capture hooks, lifecycle, failure +policy, and optional diagnostics agreed. Use `--debug` for native verbosity and +conditional durable capture. The summary/trace distinction, concrete capture +inventory, and bundle organization remain under discussion. Implementation is +pending. -- How should artifact-capture controls fit alongside native quiet/verbose/debug - reporting selections without requiring the existing `--debug` behavior? -- How should requested debug artifact writes, failures, disclosure, and retention - connect to command completion and native callbacks? -- For selected preview, request-validation, or smoke capabilities, what native - request builders and explicit execution choices should the application supply? +### Shared Debug Control With Conditional Capture -Keep debug creation and physical storage in Notarius. Ordinary records exclude -sensitive response/history content, and provider smoke diagnostics remain -explicitly opt-in. Native inspection alone must not become a model call. +Attach a native application boolean binding to CommandKit's shared `--debug` +option on the pipeline run command. Capture is enabled when that boolean is true +and a debug directory is defined by `--debug-dir PATH` or `debug.directory` in +application configuration. The CLI directory overrides the configured directory. +A directory alone does not enable capture. Do not add a separate +`--capture-debug` flag. + +Make the configured directory optional, removing the implicit +`./notarius-debug` default. With no supplied destination, `--debug` supplies +native diagnostics without creating files. Preserve normal CLI/configuration +path resolution; detailed empty-value handling can be settled with the +configuration migration. + +Determine capture from the application boolean and destination, not from +`Invocation.ReportingMode()`. A later quiet or verbose selection can change +presentation without clearing the debug input. Explicit `--debug=false` follows +native boolean binding semantics and disables capture. + +| Invocation flags | Reporting selection | Capture | +| --- | --- | --- | +| `--debug`, no CLI/configured directory | Debug | None. | +| `--debug`, with CLI/configured directory | Debug | Enabled. | +| `--debug --quiet`, no CLI/configured directory | Quiet | None. | +| `--debug --quiet`, with CLI/configured directory | Quiet | Enabled. | +| `--quiet --debug`, with CLI/configured directory | Debug | Enabled. | + +**Upstream ordering clarification:** CommandKit uses the last enabled reporting +mode. Thus `--debug --quiet` suppresses command output while retaining capture; +`--quiet --debug` selects debug verbosity. Preserve that native ordering rather +than making quiet unconditional. Quiet still shows failure explanations, and +capture does not bypass result suppression. On success the normal run report +includes the bundle location; failure explanations include an allocated bundle +path when available. + +### Native Evidence And A Small Application Recorder + +Use native facilities directly, attaching application step/lane/chunk/validator +context where useful: + +| Facility | Available evidence | +| --- | --- | +| `OnPrepared` | Prepared request details when execution reaches that point. | +| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. | +| Returned `RunResult` | Available model output and native validation result. | +| `Exchange.Details()` | Explicit inspection of captured conversation history. | +| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and checkpoint reuse. | + +Compose capture and accounting observers. Remove the debug completion wrapper, +copied execution summaries, synthetic call identities, and duplicate terminal +reporting. Keep callbacks lightweight and synchronize shared capture state: +callbacks are synchronous, can overlap across operations, and `OnPrepared` runs +while admission is held. Retain needed details and persist them at normal +operation boundaries rather than buffering an entire pipeline or repeatedly +writing every accumulated conversation prefix. + +Capture only available evidence. Preparation/admission failures may have no +`OnPrepared` notification; observer events have metadata, not response bodies; +operational failures may return no `RunResult`. Do not present the bundle as a +complete provider transcript. Use ordinary `Run` with native hooks, without +forcing split preparation/execution solely for recording. + +Conversation capture required for continuation remains in-process execution +state. Persisting its history requires enabled debug capture. Normal manifests, +checkpoints, and run reports retain their minimal projections independently of +debug selection. + +### Capture Lifecycle And Failure Policy + +For enabled capture, allocate the bundle after application configuration loads +and the destination and run identity are known, before engine construction and +pipeline preflight. Argument and configuration-load failures use native +diagnostics and may have no bundle. Loaders remain free of storage allocation. + +One application-owned recorder records initial context, accepts native execution +evidence and application decisions, and finalizes once with the known pipeline, +publication, and capture outcomes. Debug remains separate from output/cache, +never supplies checkpoint reuse, and is not automatically deleted. + +Requested capture is required work: + +- Allocation failure stops before expensive model work. +- Recording failure is surfaced at the next normal application boundary. +- Finalization failure is an operational command failure. +- An existing pipeline failure remains primary; capture failures are additional + information and preserve the underlying causes. +- Already-published artifacts and already-written diagnostic data remain + available. Capture failure does not cause model retries. + +Observer callbacks cannot return errors. Retain a recording error for application +code to check at an ordinary boundary rather than adding an exception or retry +framework. Submit the successful run report only after publication and requested +capture finalization succeed. If finalization fails after publication, identify +the published output and available debug location in the native failure +explanation; preserve the successful publication as a fact. + +### Disclosure And Retention + +Retain the separate debug root, restrictive debug permissions, and explicit +operator retention policy. Captured content may include source material, rendered +requests, generated output, and correction history. Credential redaction does not +make that material safe to share. Do not record credential values, authorization +headers, or environment dumps. Use upstream safe inspection where available and +deliberate content selection elsewhere. No summary or directory label should +imply that arbitrary diagnostic content is non-sensitive. + +### Native Optional Diagnostics + +Use the selected standard commands through the shared loaders and native inputs: + +| Command family | Purpose | +| --- | --- | +| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. | +| `pipeline validate` | Check application composition and known dependencies offline. | +| `promptkit requirements show` | Discover request requirements. | +| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. | +| `promptkit test …` | Explicit provider diagnostics. | + +Application request builders are added only for actual application needs. +Inspection/diagnostic commands use their native reports and do not automatically +allocate pipeline debug bundles; operators can redirect reports for retention. +Neither debug verbosity nor enabled capture initiates additional provider probes. +Smoke reports retain native support, applicability, and result distinctions; +status zero alone does not establish that a check passed. + +### Open Discussion: Summary And Trace + +The current `summary/` and `trace/` layout is not selected as the new design. +Discuss what diagnostic questions the bundle should answer, which application +records and native evidence it needs, and how to avoid duplicate snapshots or +execution accounting. Keep these choices open: + +- Whether a separate summary is useful, and whether it is a compact index/final + outcome rather than another copy of execution records. +- Whether summary/trace are useful conceptual categories or directory boundaries. +- Which prepared details, outputs, conversation snapshots, and application + decisions to persist, and at what granularity. +- How self-contained a failed-run bundle should be, given that it may have no + published output bundle, and when to reference existing artifacts instead of + copying them. + +Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting), +[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility), +[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run), +and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation). +The current [debug lifecycle](../internal/state.md#debug-bundles) and +[operator contract](../operations.md#debug-bundles) provide migration context. ## 6. Verification, Migration, And Roadmap Incorporation