Settle debug capture inventory and depth

This commit is contained in:
2026-11-20 00:56:29 +00:00
parent cbf3d1acc2
commit 1601b535ff

View File

@@ -25,15 +25,15 @@ Item 2 selects the application command structure and input conventions. Item 3
selects native composition, shared loaders, and application preflight. Item 4
selects native reporting, machine output, and outcomes. Item 5 selects conditional
debug capture through the shared `--debug` flag, native diagnostics, and a run/
operation bundle with a `latest` symlink. Item 6 selects verification, migration
guidance, and incorporation into the master roadmap; that incorporation is still
pending.
operation bundle with a `latest` symlink. Its capture inventory and depth are
resolved below. Item 6 selects verification, migration guidance, and incorporation
into the master roadmap; that incorporation is still pending.
The major architectural decisions are settled. Remaining choices are the exact
capture inventory and depth (item 5), run-report fields/versioning and durable
record formats (item 4/U9), and explicit-empty debug-directory handling plus any
demonstrated need for verbosity without capture (item 5). Function signatures,
bundle filenames other than `latest`, and similar implementation details do not
The major architectural decisions are settled. Remaining choices are run-report
fields/versioning and durable record formats (item 4/U9), and explicit-empty
debug-directory handling plus any demonstrated need for verbosity without capture
(item 5). Function signatures, bundle filenames other than `latest`, and similar
implementation details do not
reopen the selected architecture. Broader PromptKit choices remain with the
master roadmap.
@@ -663,9 +663,12 @@ durable capture with `/tmp/notarius-debug` as the default root. Operator-owned
data handling and convenient diagnostic representation are agreed. Replace the
summary/trace hierarchy with a compact run index, configuration and composition
snapshots, and operation records; maintain a `latest` symlink at the debug root.
Convenience guides capture depth; the precise inventory and serialized fields
remain under discussion. Explicit-empty directory handling and any demonstrated
need for verbosity without capture also remain open. Implementation is pending.
The capture inventory and depth below are agreed: one default depth retaining
every application-visible turn, useful inputs and outputs, and application
assessments, with native snapshots and convenient readable views. Exact filenames
and serialized application fields remain implementation choices. Explicit-empty
directory handling and any demonstrated need for verbosity without capture
remain open. Implementation is pending.
### Shared Debug Control With Conditional Capture
@@ -850,6 +853,7 @@ exact serialized fields remain implementation choices:
application.yml
promptkit.yml
pipeline.json
source.json
operations/
<operation-id>/
...
@@ -860,6 +864,7 @@ exact serialized fields remain implementation choices:
| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. |
| Configuration snapshots | Preserve effective settings through native inspection. |
| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. |
| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. |
| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. |
The overview provides navigation, with terminal facts drawn from the same
@@ -867,29 +872,127 @@ application outcome used for the CLI report. Reference detailed records where
useful; include convenient summary fields without maintaining a second execution
ledger. Configuration and resolved composition normally need one snapshot each.
Use native execution IDs with application scope to connect records.
Reused work has a checkpoint/reuse decision and reference, not a fabricated model
execution.
Reused work has a restored-payload snapshot and reuse decision linked to original
provenance, not a fabricated model execution.
Retain enough available context to investigate a run that fails before publishing
output. Do not depend entirely on a successful output manifest or automatically
copy every original input and published artifact. Prefer the native conversation
representation when convenient, including complete per-turn snapshots if those
are easier to capture and inspect. Do not build a competing transcript model or
deduplication system to eliminate repeated history.
copy every original input and published artifact. Retain complete native
conversation snapshots for successive turns as selected below. Do not build a
competing transcript model or deduplication system to eliminate repeated history.
### Open Discussion: Capture Depth
### Selected Capture Inventory And Depth
The organization and emphasis on convenience are selected; precise evidence
and granularity remain open. Capture readily available, useful evidence without
making exhaustive reconstruction or every provider attempt a prerequisite:
**Resolved:** Use one default capture depth preserving every application-visible
turn, its useful inputs and outputs, and the decisions made about it. An operator
should be able to inspect a failed operation without reconstructing deltas,
finding a checkpoint, or extracting model text from escaped JSON strings.
Convenience governs both capture and inspection; direct native serialization
and a simple readable rendering of the same values can both be useful.
- Which prepared request details, outputs, intermediate artifacts, validation
decisions, and correction content to retain.
- Which native conversation snapshots make troubleshooting most convenient.
- How much failed-run context to retain independently, and when references to
existing inputs, checkpoints, or published artifacts suffice.
- Which desired evidence is available through native public hooks and which
gaps would justify an upstream feature request.
#### Run Context
Retain the run overview, native application and PromptKit configuration
snapshots, resolved pipeline, and parsed source identified above. Include
application version, run/pipeline identity, timing, outcome, publication location,
and navigation to operation records in the overview. Record effective invocation
selections so request overrides remain visible even when they do not change
engine configuration.
The parsed source snapshot retains the generic document's unit identities and
content independently of the original input file. Do not automatically archive
every original configuration, template, or input file; effective snapshots and
executed requests supply the default evidence. If input parsing fails, retain
the input location and available parsing diagnostics; a parsed-source snapshot
may not exist.
#### Operation Inputs
An operation is meaningful application work: chunk planning, extraction for one
lane/chunk, merging, normalization, or validation. Identify its stage, lane,
chunk, role, and relationships to other operations as applicable. Retain an input
snapshot containing the actual values consumed:
- Relevant source/chunk content.
- Loaded reference content used by the operation.
- Incoming artifacts.
- Fixed original candidates and candidate-handle mappings for normalization.
- Prompt input bindings and application variables where applicable.
Include this content even when it also appears elsewhere in the bundle. An
extraction operation should expose its source and reference context directly.
Use existing application values and artifact codecs at meaningful boundaries;
do not invent a parallel representation of the pipeline's entire internal state.
#### Model Executions And Conversations
For each initial execution, continuation, and LLM validator execution, retain
the following evidence when available:
| Evidence | Representation |
| --- | --- |
| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. |
| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. |
| Returned result | Native `RunResult` snapshot. |
| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. |
| Conversation | Full `Exchange.Details()` snapshot. |
| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. |
Use indented JSON for structured native snapshots and existing artifact formats
for domain payloads. Readable messages are a straightforward rendering of native
data, not an independently maintained transcript model. Use the completed
exchange when available and prepared messages when execution fails before
returning a result; label which representation is shown.
Keep full conversation snapshots for successive turns, including rejected turns.
Repeated history is acceptable and makes a later turn independently inspectable.
Enable native exchange capture for debug-enabled executions where needed,
including single-turn calls. Continuation's in-process exchange requirements
remain independent of whether durable debug capture is enabled.
#### Interpretation, Assessment, And Stage Artifacts
Retain beside each response:
- The interpreted candidate, serialized through the existing candidate codec
before semantic acceptance.
- Decode, interpretation, or safety-check failures when no candidate could be
constructed.
- Each configured validator's result, findings, and actionable correction guidance.
- The application's disposition: accepted, rejected, continued, failed, or
accepted under the selected fallback/incomplete-validation policy.
- Links to LLM validator executions that informed the assessment.
For normalization, retain the model's reconciliation plan and the candidate
produced by applying it to the fixed original inputs. Preserve constructed
correction inputs even if the next execution fails during preparation; successful
preparation captures the actual rendered correction in its native messages.
Retain deterministic stage outputs too: chunk plans and materialized chunks,
merged artifacts, and normalized artifacts. For checkpoint reuse, retain the
restored payload and reuse decision linked to original provenance. These records
do not fabricate a current execution or new validation of reused work.
#### Failed Runs And Native Availability
Persist evidence as operations advance under the selected recorder lifecycle;
do not wait for final publication. On failure, retain available evidence and
record where execution stopped, the returned error, and the application outcome.
Distinguish no returned result from a returned result that failed validation.
A completed response returned through coordination's `Undelivered` field is
captured as unassessed, without implying acceptance.
Internal transport retries and structural repairs receive metadata capture at
the depth PromptKit exposes. Intermediate response bodies are not required by
this inventory. Operational errors may return no `RunResult`; record that
absence clearly. This scope does not require a provider wrapper or an upstream
feature request. Evaluate a later demonstrated gap on its actual diagnostic need.
Keep selected content in full, without arbitrary truncation or a second
capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and
native credential omissions; do not scan or scrub application content. Exact
filenames and serialized application fields remain implementation choices;
the inventory and capture depth are resolved.
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
@@ -1008,10 +1111,12 @@ completeness. The phases do not require separate releases or an intermediate
compatibility framework.
Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow
U9 to remaining durable-field, accounting, and capture details within the selected
boundaries. Reconcile removal inventory and acceptance criteria, including old
inspection-command and machine-output compatibility assumptions. Preserve
unresolved schema handling, role-specific retry/fallback policy, worker defaults,
U9 to remaining durable-field and accounting choices, and debug implementation
details within item 5's resolved inventory and depth. Preserve its separate
open directory-control choices. Reconcile removal inventory and acceptance
criteria, including old inspection-command and machine-output compatibility
assumptions. Preserve unresolved schema handling, role-specific retry/fallback
policy, worker defaults,
checkpoint transition choices, and other concrete choices until actually settled.
Once incorporated, the master roadmap owns implementation sequencing and status;