Record conditional debug capture and diagnostic design

This commit is contained in:
2026-11-19 18:27:51 +00:00
parent 27081bc0af
commit a299869fcc

View File

@@ -23,8 +23,9 @@ Item 1 selects the dependency versions and standard capabilities, including
PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target.
Item 2 selects the application command structure and input conventions. Item 3
selects native composition, shared loaders, and application preflight. Item 4
selects native reporting, machine output, and outcomes. Exact run-report fields,
debug capture, and verification choices remain open below.
selects native reporting, machine output, and outcomes. Item 5 selects conditional
debug capture through the shared `--debug` flag and native diagnostics. Debug
content/organization, exact run-report fields, and verification remain open below.
The PromptKit architecture and checkpoint decisions remain in force. This
discussion develops the CommandKit portion of that migration and its interactions
@@ -96,9 +97,9 @@ Two application needs already identified for deliberate treatment are:
replaces its presentation with one compact application report rendered natively
for humans and machines. Exact fields/versioning remain with U9; reports must
describe the actual publication and domain outcome.
- The current `--debug` requests durable artifacts. Retain the distinction between
artifact capture and reporting verbosity, while choosing native controls and
meanings rather than requiring that flag to retain its old behavior.
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when a
debug directory is supplied through CLI or configuration. Capture selection
remains independent of the final reporting mode.
### Intended Sequencing
@@ -329,7 +330,7 @@ native `config show`, `config profile` with `promptkit show profile`, and bespok
Application handlers still validate meaningful input combinations and translate
typed inputs into pipeline selections. Native `--debug` supplies reporting
behavior; a distinct artifact-capture control remains to be selected in item 5.
behavior and conditional artifact capture as selected in item 5.
Items 3–4 own loader behavior, exact report content, and outcome presentation.
Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax),
@@ -549,8 +550,9 @@ migration evidence, not a requirement to reproduce it.
The run command executes the pipeline, publishes the output bundle, completes
required run finalization, submits its successful report, and returns a native
outcome. The report describes an actual published result. Item 5 will settle the
treatment of requested debug-finalization failures.
outcome. The report describes an actual published result. Item 5 requires
requested capture to finalize successfully before a successful run report is
submitted and defines how capture failures preserve publication facts.
Use one primary run-result report, with progress and diagnostics separate. Wire
CommandKit's result destination to stdout and its diagnostic destination to
@@ -641,18 +643,160 @@ retains the bookkeeping boundary; U9 still owns concrete durable record choices.
## 5. Debug Capture And Optional Diagnostics
**Discussion / decision:** Pending.
**Discussion / decision:** Controls, native capture hooks, lifecycle, failure
policy, and optional diagnostics agreed. Use `--debug` for native verbosity and
conditional durable capture. The summary/trace distinction, concrete capture
inventory, and bundle organization remain under discussion. Implementation is
pending.
- How should artifact-capture controls fit alongside native quiet/verbose/debug
reporting selections without requiring the existing `--debug` behavior?
- How should requested debug artifact writes, failures, disclosure, and retention
connect to command completion and native callbacks?
- For selected preview, request-validation, or smoke capabilities, what native
request builders and explicit execution choices should the application supply?
### Shared Debug Control With Conditional Capture
Keep debug creation and physical storage in Notarius. Ordinary records exclude
sensitive response/history content, and provider smoke diagnostics remain
explicitly opt-in. Native inspection alone must not become a model call.
Attach a native application boolean binding to CommandKit's shared `--debug`
option on the pipeline run command. Capture is enabled when that boolean is true
and a debug directory is defined by `--debug-dir PATH` or `debug.directory` in
application configuration. The CLI directory overrides the configured directory.
A directory alone does not enable capture. Do not add a separate
`--capture-debug` flag.
Make the configured directory optional, removing the implicit
`./notarius-debug` default. With no supplied destination, `--debug` supplies
native diagnostics without creating files. Preserve normal CLI/configuration
path resolution; detailed empty-value handling can be settled with the
configuration migration.
Determine capture from the application boolean and destination, not from
`Invocation.ReportingMode()`. A later quiet or verbose selection can change
presentation without clearing the debug input. Explicit `--debug=false` follows
native boolean binding semantics and disables capture.
| Invocation flags | Reporting selection | Capture |
| --- | --- | --- |
| `--debug`, no CLI/configured directory | Debug | None. |
| `--debug`, with CLI/configured directory | Debug | Enabled. |
| `--debug --quiet`, no CLI/configured directory | Quiet | None. |
| `--debug --quiet`, with CLI/configured directory | Quiet | Enabled. |
| `--quiet --debug`, with CLI/configured directory | Debug | Enabled. |
**Upstream ordering clarification:** CommandKit uses the last enabled reporting
mode. Thus `--debug --quiet` suppresses command output while retaining capture;
`--quiet --debug` selects debug verbosity. Preserve that native ordering rather
than making quiet unconditional. Quiet still shows failure explanations, and
capture does not bypass result suppression. On success the normal run report
includes the bundle location; failure explanations include an allocated bundle
path when available.
### Native Evidence And A Small Application Recorder
Use native facilities directly, attaching application step/lane/chunk/validator
context where useful:
| Facility | Available evidence |
| --- | --- |
| `OnPrepared` | Prepared request details when execution reaches that point. |
| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
| Returned `RunResult` | Available model output and native validation result. |
| `Exchange.Details()` | Explicit inspection of captured conversation history. |
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and checkpoint reuse. |
Compose capture and accounting observers. Remove the debug completion wrapper,
copied execution summaries, synthetic call identities, and duplicate terminal
reporting. Keep callbacks lightweight and synchronize shared capture state:
callbacks are synchronous, can overlap across operations, and `OnPrepared` runs
while admission is held. Retain needed details and persist them at normal
operation boundaries rather than buffering an entire pipeline or repeatedly
writing every accumulated conversation prefix.
Capture only available evidence. Preparation/admission failures may have no
`OnPrepared` notification; observer events have metadata, not response bodies;
operational failures may return no `RunResult`. Do not present the bundle as a
complete provider transcript. Use ordinary `Run` with native hooks, without
forcing split preparation/execution solely for recording.
Conversation capture required for continuation remains in-process execution
state. Persisting its history requires enabled debug capture. Normal manifests,
checkpoints, and run reports retain their minimal projections independently of
debug selection.
### Capture Lifecycle And Failure Policy
For enabled capture, allocate the bundle after application configuration loads
and the destination and run identity are known, before engine construction and
pipeline preflight. Argument and configuration-load failures use native
diagnostics and may have no bundle. Loaders remain free of storage allocation.
One application-owned recorder records initial context, accepts native execution
evidence and application decisions, and finalizes once with the known pipeline,
publication, and capture outcomes. Debug remains separate from output/cache,
never supplies checkpoint reuse, and is not automatically deleted.
Requested capture is required work:
- Allocation failure stops before expensive model work.
- Recording failure is surfaced at the next normal application boundary.
- Finalization failure is an operational command failure.
- An existing pipeline failure remains primary; capture failures are additional
information and preserve the underlying causes.
- Already-published artifacts and already-written diagnostic data remain
available. Capture failure does not cause model retries.
Observer callbacks cannot return errors. Retain a recording error for application
code to check at an ordinary boundary rather than adding an exception or retry
framework. Submit the successful run report only after publication and requested
capture finalization succeed. If finalization fails after publication, identify
the published output and available debug location in the native failure
explanation; preserve the successful publication as a fact.
### Disclosure And Retention
Retain the separate debug root, restrictive debug permissions, and explicit
operator retention policy. Captured content may include source material, rendered
requests, generated output, and correction history. Credential redaction does not
make that material safe to share. Do not record credential values, authorization
headers, or environment dumps. Use upstream safe inspection where available and
deliberate content selection elsewhere. No summary or directory label should
imply that arbitrary diagnostic content is non-sensitive.
### Native Optional Diagnostics
Use the selected standard commands through the shared loaders and native inputs:
| Command family | Purpose |
| --- | --- |
| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. |
| `pipeline validate` | Check application composition and known dependencies offline. |
| `promptkit requirements show` | Discover request requirements. |
| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. |
| `promptkit test …` | Explicit provider diagnostics. |
Application request builders are added only for actual application needs.
Inspection/diagnostic commands use their native reports and do not automatically
allocate pipeline debug bundles; operators can redirect reports for retention.
Neither debug verbosity nor enabled capture initiates additional provider probes.
Smoke reports retain native support, applicability, and result distinctions;
status zero alone does not establish that a check passed.
### Open Discussion: Summary And Trace
The current `summary/` and `trace/` layout is not selected as the new design.
Discuss what diagnostic questions the bundle should answer, which application
records and native evidence it needs, and how to avoid duplicate snapshots or
execution accounting. Keep these choices open:
- Whether a separate summary is useful, and whether it is a compact index/final
outcome rather than another copy of execution records.
- Whether summary/trace are useful conceptual categories or directory boundaries.
- Which prepared details, outputs, conversation snapshots, and application
decisions to persist, and at what granularity.
- How self-contained a failed-run bundle should be, given that it may have no
published output bundle, and when to reference existing artifacts instead of
copying them.
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run),
and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation).
The current [debug lifecycle](../internal/state.md#debug-bundles) and
[operator contract](../operations.md#debug-bundles) provide migration context.
## 6. Verification, Migration, And Roadmap Incorporation