Settle debug capture inventory and depth
This commit is contained in:
@@ -25,15 +25,15 @@ Item 2 selects the application command structure and input conventions. Item 3
|
||||
selects native composition, shared loaders, and application preflight. Item 4
|
||||
selects native reporting, machine output, and outcomes. Item 5 selects conditional
|
||||
debug capture through the shared `--debug` flag, native diagnostics, and a run/
|
||||
operation bundle with a `latest` symlink. Item 6 selects verification, migration
|
||||
guidance, and incorporation into the master roadmap; that incorporation is still
|
||||
pending.
|
||||
operation bundle with a `latest` symlink. Its capture inventory and depth are
|
||||
resolved below. Item 6 selects verification, migration guidance, and incorporation
|
||||
into the master roadmap; that incorporation is still pending.
|
||||
|
||||
The major architectural decisions are settled. Remaining choices are the exact
|
||||
capture inventory and depth (item 5), run-report fields/versioning and durable
|
||||
record formats (item 4/U9), and explicit-empty debug-directory handling plus any
|
||||
demonstrated need for verbosity without capture (item 5). Function signatures,
|
||||
bundle filenames other than `latest`, and similar implementation details do not
|
||||
The major architectural decisions are settled. Remaining choices are run-report
|
||||
fields/versioning and durable record formats (item 4/U9), and explicit-empty
|
||||
debug-directory handling plus any demonstrated need for verbosity without capture
|
||||
(item 5). Function signatures, bundle filenames other than `latest`, and similar
|
||||
implementation details do not
|
||||
reopen the selected architecture. Broader PromptKit choices remain with the
|
||||
master roadmap.
|
||||
|
||||
@@ -663,9 +663,12 @@ durable capture with `/tmp/notarius-debug` as the default root. Operator-owned
|
||||
data handling and convenient diagnostic representation are agreed. Replace the
|
||||
summary/trace hierarchy with a compact run index, configuration and composition
|
||||
snapshots, and operation records; maintain a `latest` symlink at the debug root.
|
||||
Convenience guides capture depth; the precise inventory and serialized fields
|
||||
remain under discussion. Explicit-empty directory handling and any demonstrated
|
||||
need for verbosity without capture also remain open. Implementation is pending.
|
||||
The capture inventory and depth below are agreed: one default depth retaining
|
||||
every application-visible turn, useful inputs and outputs, and application
|
||||
assessments, with native snapshots and convenient readable views. Exact filenames
|
||||
and serialized application fields remain implementation choices. Explicit-empty
|
||||
directory handling and any demonstrated need for verbosity without capture
|
||||
remain open. Implementation is pending.
|
||||
|
||||
### Shared Debug Control With Conditional Capture
|
||||
|
||||
@@ -850,6 +853,7 @@ exact serialized fields remain implementation choices:
|
||||
application.yml
|
||||
promptkit.yml
|
||||
pipeline.json
|
||||
source.json
|
||||
operations/
|
||||
<operation-id>/
|
||||
...
|
||||
@@ -860,6 +864,7 @@ exact serialized fields remain implementation choices:
|
||||
| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. |
|
||||
| Configuration snapshots | Preserve effective settings through native inspection. |
|
||||
| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. |
|
||||
| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. |
|
||||
| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. |
|
||||
|
||||
The overview provides navigation, with terminal facts drawn from the same
|
||||
@@ -867,29 +872,127 @@ application outcome used for the CLI report. Reference detailed records where
|
||||
useful; include convenient summary fields without maintaining a second execution
|
||||
ledger. Configuration and resolved composition normally need one snapshot each.
|
||||
Use native execution IDs with application scope to connect records.
|
||||
Reused work has a checkpoint/reuse decision and reference, not a fabricated model
|
||||
execution.
|
||||
Reused work has a restored-payload snapshot and reuse decision linked to original
|
||||
provenance, not a fabricated model execution.
|
||||
|
||||
Retain enough available context to investigate a run that fails before publishing
|
||||
output. Do not depend entirely on a successful output manifest or automatically
|
||||
copy every original input and published artifact. Prefer the native conversation
|
||||
representation when convenient, including complete per-turn snapshots if those
|
||||
are easier to capture and inspect. Do not build a competing transcript model or
|
||||
deduplication system to eliminate repeated history.
|
||||
copy every original input and published artifact. Retain complete native
|
||||
conversation snapshots for successive turns as selected below. Do not build a
|
||||
competing transcript model or deduplication system to eliminate repeated history.
|
||||
|
||||
### Open Discussion: Capture Depth
|
||||
### Selected Capture Inventory And Depth
|
||||
|
||||
The organization and emphasis on convenience are selected; precise evidence
|
||||
and granularity remain open. Capture readily available, useful evidence without
|
||||
making exhaustive reconstruction or every provider attempt a prerequisite:
|
||||
**Resolved:** Use one default capture depth preserving every application-visible
|
||||
turn, its useful inputs and outputs, and the decisions made about it. An operator
|
||||
should be able to inspect a failed operation without reconstructing deltas,
|
||||
finding a checkpoint, or extracting model text from escaped JSON strings.
|
||||
Convenience governs both capture and inspection; direct native serialization
|
||||
and a simple readable rendering of the same values can both be useful.
|
||||
|
||||
- Which prepared request details, outputs, intermediate artifacts, validation
|
||||
decisions, and correction content to retain.
|
||||
- Which native conversation snapshots make troubleshooting most convenient.
|
||||
- How much failed-run context to retain independently, and when references to
|
||||
existing inputs, checkpoints, or published artifacts suffice.
|
||||
- Which desired evidence is available through native public hooks and which
|
||||
gaps would justify an upstream feature request.
|
||||
#### Run Context
|
||||
|
||||
Retain the run overview, native application and PromptKit configuration
|
||||
snapshots, resolved pipeline, and parsed source identified above. Include
|
||||
application version, run/pipeline identity, timing, outcome, publication location,
|
||||
and navigation to operation records in the overview. Record effective invocation
|
||||
selections so request overrides remain visible even when they do not change
|
||||
engine configuration.
|
||||
|
||||
The parsed source snapshot retains the generic document's unit identities and
|
||||
content independently of the original input file. Do not automatically archive
|
||||
every original configuration, template, or input file; effective snapshots and
|
||||
executed requests supply the default evidence. If input parsing fails, retain
|
||||
the input location and available parsing diagnostics; a parsed-source snapshot
|
||||
may not exist.
|
||||
|
||||
#### Operation Inputs
|
||||
|
||||
An operation is meaningful application work: chunk planning, extraction for one
|
||||
lane/chunk, merging, normalization, or validation. Identify its stage, lane,
|
||||
chunk, role, and relationships to other operations as applicable. Retain an input
|
||||
snapshot containing the actual values consumed:
|
||||
|
||||
- Relevant source/chunk content.
|
||||
- Loaded reference content used by the operation.
|
||||
- Incoming artifacts.
|
||||
- Fixed original candidates and candidate-handle mappings for normalization.
|
||||
- Prompt input bindings and application variables where applicable.
|
||||
|
||||
Include this content even when it also appears elsewhere in the bundle. An
|
||||
extraction operation should expose its source and reference context directly.
|
||||
Use existing application values and artifact codecs at meaningful boundaries;
|
||||
do not invent a parallel representation of the pipeline's entire internal state.
|
||||
|
||||
#### Model Executions And Conversations
|
||||
|
||||
For each initial execution, continuation, and LLM validator execution, retain
|
||||
the following evidence when available:
|
||||
|
||||
| Evidence | Representation |
|
||||
| --- | --- |
|
||||
| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. |
|
||||
| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. |
|
||||
| Returned result | Native `RunResult` snapshot. |
|
||||
| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. |
|
||||
| Conversation | Full `Exchange.Details()` snapshot. |
|
||||
| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. |
|
||||
|
||||
Use indented JSON for structured native snapshots and existing artifact formats
|
||||
for domain payloads. Readable messages are a straightforward rendering of native
|
||||
data, not an independently maintained transcript model. Use the completed
|
||||
exchange when available and prepared messages when execution fails before
|
||||
returning a result; label which representation is shown.
|
||||
|
||||
Keep full conversation snapshots for successive turns, including rejected turns.
|
||||
Repeated history is acceptable and makes a later turn independently inspectable.
|
||||
Enable native exchange capture for debug-enabled executions where needed,
|
||||
including single-turn calls. Continuation's in-process exchange requirements
|
||||
remain independent of whether durable debug capture is enabled.
|
||||
|
||||
#### Interpretation, Assessment, And Stage Artifacts
|
||||
|
||||
Retain beside each response:
|
||||
|
||||
- The interpreted candidate, serialized through the existing candidate codec
|
||||
before semantic acceptance.
|
||||
- Decode, interpretation, or safety-check failures when no candidate could be
|
||||
constructed.
|
||||
- Each configured validator's result, findings, and actionable correction guidance.
|
||||
- The application's disposition: accepted, rejected, continued, failed, or
|
||||
accepted under the selected fallback/incomplete-validation policy.
|
||||
- Links to LLM validator executions that informed the assessment.
|
||||
|
||||
For normalization, retain the model's reconciliation plan and the candidate
|
||||
produced by applying it to the fixed original inputs. Preserve constructed
|
||||
correction inputs even if the next execution fails during preparation; successful
|
||||
preparation captures the actual rendered correction in its native messages.
|
||||
|
||||
Retain deterministic stage outputs too: chunk plans and materialized chunks,
|
||||
merged artifacts, and normalized artifacts. For checkpoint reuse, retain the
|
||||
restored payload and reuse decision linked to original provenance. These records
|
||||
do not fabricate a current execution or new validation of reused work.
|
||||
|
||||
#### Failed Runs And Native Availability
|
||||
|
||||
Persist evidence as operations advance under the selected recorder lifecycle;
|
||||
do not wait for final publication. On failure, retain available evidence and
|
||||
record where execution stopped, the returned error, and the application outcome.
|
||||
Distinguish no returned result from a returned result that failed validation.
|
||||
A completed response returned through coordination's `Undelivered` field is
|
||||
captured as unassessed, without implying acceptance.
|
||||
|
||||
Internal transport retries and structural repairs receive metadata capture at
|
||||
the depth PromptKit exposes. Intermediate response bodies are not required by
|
||||
this inventory. Operational errors may return no `RunResult`; record that
|
||||
absence clearly. This scope does not require a provider wrapper or an upstream
|
||||
feature request. Evaluate a later demonstrated gap on its actual diagnostic need.
|
||||
|
||||
Keep selected content in full, without arbitrary truncation or a second
|
||||
capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and
|
||||
native credential omissions; do not scan or scrub application content. Exact
|
||||
filenames and serialized application fields remain implementation choices;
|
||||
the inventory and capture depth are resolved.
|
||||
|
||||
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
|
||||
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
|
||||
@@ -1008,10 +1111,12 @@ completeness. The phases do not require separate releases or an intermediate
|
||||
compatibility framework.
|
||||
|
||||
Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow
|
||||
U9 to remaining durable-field, accounting, and capture details within the selected
|
||||
boundaries. Reconcile removal inventory and acceptance criteria, including old
|
||||
inspection-command and machine-output compatibility assumptions. Preserve
|
||||
unresolved schema handling, role-specific retry/fallback policy, worker defaults,
|
||||
U9 to remaining durable-field and accounting choices, and debug implementation
|
||||
details within item 5's resolved inventory and depth. Preserve its separate
|
||||
open directory-control choices. Reconcile removal inventory and acceptance
|
||||
criteria, including old inspection-command and machine-output compatibility
|
||||
assumptions. Preserve unresolved schema handling, role-specific retry/fallback
|
||||
policy, worker defaults,
|
||||
checkpoint transition choices, and other concrete choices until actually settled.
|
||||
|
||||
Once incorporated, the master roadmap owns implementation sequencing and status;
|
||||
|
||||
Reference in New Issue
Block a user