From 5618adea8e1bb3decf8bf992f20b28153c57463d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 19 Nov 2026 18:55:24 +0000 Subject: [PATCH] Record debug bundle organization and latest symlink --- .../commandkit-migration-discussion.md | 88 +++++++++++++++---- 1 file changed, 70 insertions(+), 18 deletions(-) diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md index 7cba4842..a3a2f47b 100644 --- a/docs/roadmap/commandkit-migration-discussion.md +++ b/docs/roadmap/commandkit-migration-discussion.md @@ -24,8 +24,9 @@ PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target. Item 2 selects the application command structure and input conventions. Item 3 selects native composition, shared loaders, and application preflight. Item 4 selects native reporting, machine output, and outcomes. Item 5 selects conditional -debug capture through the shared `--debug` flag and native diagnostics. Debug -content/organization, exact run-report fields, and verification remain open below. +debug capture through the shared `--debug` flag, native diagnostics, and a run/ +operation bundle with a `latest` symlink. Capture depth, exact run-report fields, +and verification remain open below. The PromptKit architecture and checkpoint decisions remain in force. This discussion develops the CommandKit portion of that migration and its interactions @@ -646,9 +647,10 @@ retains the bookkeeping boundary; U9 still owns concrete durable record choices. **Discussion / decision:** Controls, native capture hooks, lifecycle, failure policy, and optional diagnostics agreed. Use `--debug` for native verbosity and durable capture with `/tmp/notarius-debug` as the default root. Operator-owned -data handling and recording each fact once are agreed. The summary/trace -distinction, concrete capture inventory, and bundle organization remain under -discussion. Implementation is pending. +data handling and recording each fact once are agreed. Replace the summary/trace +hierarchy with a compact run index, configuration/composition snapshots, and +operation records; maintain a `latest` symlink at the debug root. Capture depth +and concrete serialized fields remain under discussion. Implementation is pending. ### Shared Debug Control With Conditional Capture @@ -731,6 +733,15 @@ and the destination and run identity are known, before engine construction and pipeline preflight. Argument and configuration-load failures use native diagnostics and may have no bundle. Loaders remain free of storage allocation. +After creating the run directory, update `/latest` to a relative +symlink targeting that run ID before engine construction or preflight starts. +This points to the newly started debug session, including one that later fails; +it is not a last-successful-run selector. Runs without capture leave it alone. +Do not update it again at completion, so an older overlapping run cannot take +the link back when it finishes. Concurrent starts use the last successful link +update; individual run directories remain the actual storage locations. +Treat link-update failures under the requested-capture failure policy below. + One application-owned recorder records initial context, accepts native execution evidence and application decisions, and finalizes once with the known pipeline, publication, and capture outcomes. Debug remains separate from output/cache, @@ -793,25 +804,66 @@ Neither debug verbosity nor enabled capture initiates additional provider probes Smoke reports retain native support, applicability, and result distinctions; status zero alone does not establish that a check passed. -### Open Discussion: Summary And Trace +### Selected Bundle Organization **Agreed principle:** Record each fact once and reference it where needed. Use shared application outcome facts and native execution evidence rather than parallel summary/trace accounting or repeated snapshots. -The current `summary/` and `trace/` layout is not selected as the new design. -Discuss what diagnostic questions the bundle should answer and which application -records and native evidence it needs. Summary/trace are not separate security -classes. Keep these choices open: +Organize the bundle around the run and its operations, replacing `summary/` and +`trace/` as separate categories. Use a compact overview for navigation and keep +detailed evidence with its operation. Summary/trace are not security classes. -- Whether a separate summary is useful, and whether it is a compact index/final - outcome rather than another copy of execution records. -- Whether summary/trace are useful conceptual categories or directory boundaries. -- Which prepared details, outputs, conversation snapshots, and application - decisions to persist, and at what granularity. -- How self-contained a failed-run bundle should be, given that it may have no - published output bundle, and when to reference existing artifacts instead of - copying them. +Illustrative layout: `latest` is the selected link name; other file names and +exact serialized fields remain implementation choices: + +```text +/ + latest -> / + / + run.json + config/ + application.yml + promptkit.yml + pipeline.json + operations/ + / + ... +``` + +| Component | Purpose | +| --- | --- | +| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. | +| Configuration snapshots | Preserve effective settings through native inspection. | +| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. | +| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. | + +The overview is an index rather than a second execution ledger. Its terminal +facts come from the same application outcome used for the CLI report, and it +references detailed records instead of duplicating requests, responses, findings, +or error history. Capture configuration and resolved composition once when +available. Use native execution IDs with application scope to connect records. +Reused work has a checkpoint/reuse decision and reference, not a fabricated model +execution. + +Retain enough available context to investigate a run that fails before publishing +output. Do not depend entirely on a successful output manifest or automatically +copy every original input and published artifact. Select a conversation +representation that avoids repeatedly saving the complete accumulated history +after each turn or maintaining an independent competing transcript. + +### Open Discussion: Capture Depth + +The organization is selected; precise evidence and granularity remain open: + +- Which prepared request details, outputs, intermediate artifacts, validation + decisions, and correction content to retain. +- How to represent conversation history without repeated prefixes while using + PromptKit's native evidence. +- How much failed-run context to retain independently, and when references to + existing inputs, checkpoints, or published artifacts suffice. +- Which desired evidence is available through native public hooks and which + gaps would justify an upstream feature request. Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting), [reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),