Simplify the planned artifact output and completion report

This commit is contained in:
2026-11-20 03:28:14 +00:00
parent 305a88e8f4
commit c03d0bb987
2 changed files with 203 additions and 134 deletions

View File

@@ -23,20 +23,22 @@ Item 1 selects the dependency versions and standard capabilities, including
PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target.
Item 2 selects the application command structure and input conventions. Item 3
selects native composition, shared loaders, and application preflight. Item 4
selects native reporting, machine output, and outcomes. Item 5 selects conditional
debug capture through the shared `--debug` flag, native diagnostics, and a run/
operation bundle with a `latest` symlink. Its capture inventory, depth, and
selects artifact-first output, a small native completion report, and outcomes.
Item 5 selects conditional debug capture through the shared `--debug` flag,
native diagnostics, and a run/operation bundle with a `latest` symlink. Its
capture inventory, depth, and
explicit-empty directory behavior are resolved below. Item 6 selects verification,
migration guidance, and incorporation into the master roadmap; that incorporation
is still pending.
The cache follow-up in item 4 replaces the earlier checkpoint design with an
accepted-artifact cache keyed by initial message content. The output/report
contract has been reopened around consumer needs; it is not merely waiting for
field names and versions. Function signatures, bundle filenames other than
`latest`, and similar implementation details do not
reopen the selected architecture. Broader PromptKit choices remain with the
master roadmap.
direction is settled: deliver artifacts as files and render a small structured
completion value through CommandKit, with a native text template for ordinary
human output. Detailed diagnostics belong in optional debug capture. Function
signatures, debug filenames other than `latest`, and similar implementation
details do not reopen the selected architecture. Broader PromptKit choices
remain with the master roadmap.
The PromptKit ownership direction remains in force. The revised
[cache decision](promptkit-architecture-discussion.md#5-accepted-artifact-cache)
@@ -93,13 +95,13 @@ spending migration effort recreating existing CLI conventions.
| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. |
| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. |
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, accepted-artifact caching, and domain outcomes | Notarius application handlers and framework. |
| Receipt content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
| Completion-summary content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
Adopt the standard capability suite selected in item 1, including upstream
commands and recommendations beyond the existing Notarius surface. Items 2–4
select application inputs, composition, and reporting; items 5–6 select debug
capture principles, verification, and migration. Concrete record details remain
open as identified below.
capture, verification, and migration. The earlier extensive receipt/manifest
proposal is superseded by item 4's artifact-first contract.
Pipeline and domain code should remain independent of CommandKit. Native
PromptKit observations supply execution facts; Notarius attaches application
@@ -108,9 +110,9 @@ meaning and selects report content. CommandKit supplies presentation mechanics.
Two application needs already identified for deliberate treatment are:
- The current `run --json` publishes a receipt after output publication. Item 4
records native reporting mechanics, but the output/report contract has been
reopened around artifact delivery and optional metadata. Its earlier mandatory
run-report proposal is subject to that discussion.
replaces that contract with artifact files and a small completion summary,
rendered natively as text, YAML, or JSON. No automatic metadata bundle or
separately persisted report is required.
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when an
effective debug directory is available, including an application default.
Capture selection remains independent of the final reporting mode.
@@ -202,8 +204,8 @@ implementing cache lookup; the version table remains the reviewed baseline, not
a claim that the new hash is already available. On roadmap
reconciliation, incorporate these versions into U1, replace U10's open standard
capability selection with this decision, and incorporate the application wiring
and presentation decisions from items 2–5. Preserve the remaining record choices
explicitly as directed by item 6.
and presentation decisions from items 2–5. Retire superseded durable-record
requirements as directed by item 6.
## 2. Native Command Surface And Inputs
@@ -336,8 +338,7 @@ Generic effective configuration and source explanations remain available through
`config show` and `config explain`. ConfigKit's safe inspection facilities do not
automatically redact an arbitrary application report; select appropriate report
content rather than dumping the raw typed configuration. Item 3 selects loader
wiring; item 4 selects reporting conventions and identifies remaining run-report
contract choices.
wiring; item 4 selects the artifact-first output and completion-report contract.
### Removals And Remaining Work
@@ -350,8 +351,9 @@ native `config show`, `config profile` with `promptkit show profile`, and bespok
Application handlers still validate meaningful input combinations and translate
typed inputs into pipeline selections. Native `--debug` supplies reporting
behavior and conditional artifact capture as selected in item 5.
Items 3–4 select loader behavior, report content categories, and outcome
presentation; exact run-report fields and durable formats remain with U9.
Items 3–4 select loader behavior, the small completion report, and native outcome
presentation. Reconcile U9 with these decisions rather than expanding durable
metadata.
Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax),
[reusable PromptKit flags](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading),
@@ -521,14 +523,12 @@ Evidence: [native suite and loader ownership](../../../commandkit/docs/consumers
## 4. Reports, Machine Output, And Outcomes
**Discussion / decision:** Native reporting and outcome mechanics remain selected.
The output/report contract is reopened: consumers primarily need resulting
artifacts, metadata may be optional, failures need useful nonzero outcomes, and
operators/developers use debug capture for detailed investigation. The earlier
run-report and durable-metadata proposals below remain discussion material for
that review, not requirements to preserve or expand. No new report/manifest
schemas have been selected. Finalize the cache decision first; return to the
consumer output/report contract afterward. Implementation is pending.
**Discussion / decision:** Agreed. Deliver artifacts as files and use CommandKit
to render a small structured completion report on stdout, with a native text
template for the ordinary successful case. Additional operational diagnostics
use stderr; detailed investigation uses optional debug capture. This supersedes
the earlier extensive receipt, manifest, and durable-execution-record proposals.
Keep native outcome and delivery-failure handling. Implementation is pending.
### Accepted-Artifact Cache And PromptKit Ownership
@@ -572,45 +572,82 @@ Concrete cache flags and legacy-checkpoint handling remain implementation-planni
choices. The output/report discussion must not reintroduce cache metadata solely
to populate a report. The master roadmap still needs this replacement policy.
### One Report Value For Human And Machine Output
### Artifacts Are The Deliverable
Application commands submit ordinary values through `Reporter.Report` and use
native `--format text|yaml|json` rendering. Execution supplies one run-report
value with a concise text template for default human output; JSON and YAML render
the same structured value. Do not maintain separate human-summary and JSON-receipt
construction paths.
Publish the accepted domain artifacts as files in the output directory. Their
contents and encoding are independent of CLI report formatting: `--format`
controls the completion summary, not the artifact payloads. Keep artifact
discovery understandable without requiring consumers to read execution metadata.
```sh
notarius run dnd-session --input-file transcript.json
notarius run dnd-session --input-file transcript.json --format json
Do not require a run manifest, metadata bundle, or persisted completion receipt
alongside the artifacts. Preserve information intrinsic to the domain artifacts
and their existing schemas; do not add model, retry, validation-history, or
execution inventories to those payloads merely because it is available.
Detailed operational evidence belongs in item 5's optional debug bundle.
### Small Structured Completion Report And Native Text Template
Submit one small application value through `Reporter.Report`. Use these fields
for an ordinary successful run:
| Field | Meaning |
| --- | --- |
| `pipeline` | Selected pipeline identifier. |
| `artifact_count` | Number of deliverable artifacts produced, including reused artifacts; excludes internal intermediate results, cache entries, and debug files. |
| `output_directory` | Absolute path to the directory containing the published artifacts. |
| `debug_directory` | Optional absolute path to the completed run-specific debug bundle; omitted when capture is disabled. |
For example, without debug capture:
```json
{
"pipeline": "dnd-session",
"artifact_count": 6,
"output_directory": "/work/results/run-123"
}
```
Standard configuration and PromptKit commands retain their upstream report shapes
and behavior. Do not wrap them in a universal Notarius result envelope. No current
run-report need requires raw output; reserve native `Output`/`RenderOutput` for an
operation whose actual product needs a byte stream.
Supply a short native `text/template` to render that same value as:
### Compact Application Run Report
```text
Success: dnd-session produced 6 artifacts in /work/results/run-123
```
CommandKit owns presentation; Notarius owns the meaning and deliberate contract
of the application run report. Retain these content categories:
CommandKit automatically renders a string value as plain text. Here the value
is structured so JSON/YAML callers receive named fields instead of a success
sentence they must parse. The native text template provides the concise human
presentation; without it, a structured value would default to YAML. Use native
`--format text|yaml|json` selection and serializers, without separate encoders or
independent human/machine report construction.
| Content | Purpose |
| --- | --- |
| Application report version | Identify the report contract for consumers. |
| Run and pipeline identity | Identify completed work. |
| Published bundle directory and applicable index path | Locate artifacts without guessing filenames. |
| Accepted/normalized and rejected output counts | Summarize production results. |
| Validation completeness/status | Distinguish fully validated results from permitted incomplete validation. |
| Process-warning counts | Expose degradation independently of reporting verbosity. |
| Debug location, when applicable | Locate requested diagnostic artifacts. |
When a completed run includes rejected work or permitted incomplete validation,
qualify the completion message clearly. Include the rejection count and/or
incomplete-validation fact in the same structured value when applicable. Do not
present unqualified success that conceals those outcomes, or make their visibility
depend on verbose/debug mode. These are small outcome qualifications, not
per-producer histories or a general metadata envelope.
Detailed per-producer validation records, rejection reasons, and execution
provenance belong in the published bundle rather than duplicated collections in
the command report. Do not serialize the entire internal pipeline result or
complete upstream runtime types as a durable contract. U9 will select exact
fields, versioning, and durable record formats. The old CLI receipt's shape is
migration evidence, not a requirement to reproduce it.
No report-version field, run-ID field, token totals, profile inventory, warning
statistics, validation-summary collection, or durable execution ledger is required
by this completion contract. The old receipt is migration context, not a shape
to preserve. Standard configuration and PromptKit commands retain their native
report values and behavior without a universal Notarius wrapper.
### Optional Machine Output And Saved Reports
Operators and subprocess callers select YAML or JSON through the native format
flag. The completion report is enough to find the artifacts and recognize any
qualified outcome. Consumers interested only in artifacts need no metadata file.
Save a report with ordinary shell redirection:
```sh
notarius run dnd-session --input-file transcript.json --format json > run-report.json
```
Do not add a separate report-file flag or automatically persist a duplicate report
for this design. Keep stdout as one completion report, with additional diagnostics
on stderr, so `--format json --verbose` still produces valid report JSON.
### Publication, Streams, And Machine Consumption
@@ -628,28 +665,37 @@ failure may leave partial stdout even though publication already succeeded.
These run-consumption rules do not replace the native report behavior of other
commands, including configuration commands that can report failed validation.
### Native Visibility And Message Categories
### Native Visibility, Diagnostics, And Debug Location
| Native facility | Application use |
| --- | --- |
| `Report` | Requested command result. |
| `Progress` | Pipeline progress. |
| `Warn` | Actionable process degradation. |
| `Info` | Supplementary operational information. |
| `Debug` | Diagnostic detail, distinct from durable artifact capture. |
| Outcome explanation | Terminal failure or usage guidance. |
Wire CommandKit's result destination to stdout and diagnostic destination to
stderr. Use `Info`, `Warn`, `Debug`, `Progress`, and native outcome explanations
for their respective categories; do not mix these messages into a JSON/YAML
completion report.
Adopt native visibility rules directly: ordinary mode suppresses warning messages,
and quiet mode suppresses command results. Include validation status and
process-warning counts in the primary run report and its human summary so those
facts do not depend on warning-message visibility. Operators request `--verbose`
for warning details; machine callers expecting a report do not request `--quiet`.
Use native progress selection and failure visibility without custom bypasses.
Quiet suppression is not a delivery failure; native rendering errors can still
fail a quiet invocation.
| Mode | Stdout | Stderr |
| --- | --- | --- |
| Ordinary | Brief completion summary. | Failure explanation, if needed. |
| `--verbose` | Same completion report. | Additional operational information, warnings, and any failure explanation. |
| `--quiet` | Nothing. | Failure explanation, if needed. |
| `--debug` | Completion report, including capture location when enabled. | Native debug diagnostics, information, warnings, progress, and any failure explanation. |
Artifact-quality advisories retain their application classification. Do not
reclassify them as process warnings merely to make them visible.
Adopt native reporting-mode ordering, explicit progress selection, and failure
visibility. The table describes mode defaults; `--progress` can explicitly enable
progress outside quiet mode. Machine callers expecting a report do not request
`--quiet`. Quiet suppression is not a delivery failure; native rendering errors
can still fail a quiet invocation. Artifact-quality advisories keep their domain
classification rather than becoming process warnings solely for visibility.
When capture completes, the text template adds a line such as:
```text
Debug capture: /tmp/notarius-debug/run-123
```
Use `debug_directory` in the structured value for the same location. Provide the
path rather than mirroring the debug bundle to stdout. An explicitly empty debug
directory preserves native debug diagnostics but creates no capture or capture
path in the completion report. Item 5 owns capture controls and lifecycle.
### Native Outcomes And Domain Acceptance
@@ -669,14 +715,18 @@ have different causes; preserve that distinction and native error identity.
Command completion remains separate from domain acceptance. A successfully
published run can contain recorded rejections or validation-incomplete results
allowed by application policy; those facts appear in reports and artifacts and
do not automatically fail the command. A terminal pipeline failure does fail the
allowed by application policy; those facts qualify the completion report and do
not automatically fail the command. A terminal pipeline failure does fail the
command. Native status zero also does not replace reading standard PromptKit
reports for partial discovery or unsupported smoke checks.
Application handlers supply useful native outcome explanations and retain the
underlying error through native cause handling. CommandKit presents the terminal
result. Remove duplicate terminal-error printing and custom outcome wrappers.
result. On failure, provide a useful explanation and nonzero status rather than
requiring a separate structured error-report protocol. `--format` controls the
completion report, not native diagnostics. If artifacts were published before a
later failure, include their location in the failure explanation. Remove duplicate
terminal-error printing and custom outcome wrappers.
### Delivery Failures, Progress, And Accounting
@@ -687,16 +737,21 @@ fact. Do not rewrite pipeline history as failed execution or automatically rerun
model work because report delivery failed.
Notarius owns stage/lane/chunk progress, domain acceptance, and cache reuse.
PromptKit supplies native execution observations and `ExecutionGroup` totals;
CommandKit presents the selected facts. Explicitly enroll the relevant initial,
continuation, and validator executions without double counting. Cache reuse
does not add historical token usage to current execution totals, and unavailable
usage remains distinguishable from zero.
PromptKit supplies native execution observations and `ExecutionGroup` totals.
These facts support native operational diagnostics and optional debug capture;
they do not require a permanent consumer execution report. Explicitly enroll the
relevant initial, continuation, and validator executions without double counting.
Cache reuse does not add historical token usage to current execution totals, and
unavailable usage remains distinguishable from zero.
Remove bespoke format selection, parallel receipt encoders, custom verbosity
handling, duplicate execution accounting, and superseded terminal-reporting
machinery. Retain only application report content, publication facts, domain
decisions, and small deliberate durable projections.
machinery. Retire automatic reporting-only metadata bundles, mandatory run
manifests, per-producer provenance inventories, and durable execution-ledger
requirements. Do not preserve or rebuild those structures to populate the small
completion value. Use existing application outcome facts for its contents and
the debug overview. Artifact schemas remain domain contracts; detailed debug
representations remain diagnostic data.
Evidence: [native reporting](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
[visibility and delivery](../../../commandkit/docs/cli.md#reporting), and
@@ -705,7 +760,8 @@ The current [run-result contract](../integrations/run-result.md) and
[CLI failure mapping](../internal/cli.md#failure-mapping-and-terminal-reporting)
document migration impact. Item 6 of the
[PromptKit discussion record](promptkit-architecture-discussion.md#6-inspection-and-execution-bookkeeping)
retains the bookkeeping boundary; U9 still owns concrete durable record choices.
retains the native bookkeeping boundary. Reconcile U9 with this simpler contract,
including retirement of superseded durable-record requirements.
## 5. Debug Capture And Optional Diagnostics
@@ -817,7 +873,7 @@ notifications alone do not provide cache short-circuiting.
Conversation capture required for continuation remains in-process execution
state. Persisting its history requires enabled debug capture. Cached values are
accepted domain artifacts, not conversation state or provenance records. The
separate output/report contract remains under discussion.
separate artifact-first output and completion-report contract is selected in item 4.
### Capture Lifecycle And Failure Policy
@@ -951,7 +1007,7 @@ a fabricated model execution. Historical provenance is not required in cache
files or reconstructed solely for debug capture.
Retain enough available context to investigate a run that fails before publishing
output. Do not depend entirely on a successful output manifest or automatically
output. Do not depend entirely on successfully published artifacts or automatically
copy every original input and published artifact. Retain complete native
conversation snapshots for successive turns as selected below. Do not build a
competing transcript model or deduplication system to eliminate repeated history.
@@ -1188,8 +1244,10 @@ completeness. The phases do not require separate releases or an intermediate
compatibility framework.
Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow
U9 to the reopened consumer output/report contract, and debug implementation
details within item 5's resolved inventory, depth, and directory controls.
U9 to item 4's selected artifact-first output/completion contract and item 5's
debug implementation details within the resolved inventory, depth, and controls.
Retire earlier receipt/manifest field inventories rather than treating them as
unresolved requirements to recreate.
Update U1 for the upstream message-content hash and replace U8's checkpoint
identity/provenance plan with the accepted-artifact cache decision.
Reconcile removal inventory and acceptance criteria, including old