Record inspection and execution bookkeeping ownership

This commit is contained in:
2026-11-19 14:59:56 +00:00
parent 3e1abc7fbb
commit 3fb8cb7392

View File

@@ -5,11 +5,9 @@
Working notepad for reviewing the ownership boundaries in the Working notepad for reviewing the ownership boundaries in the
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics [PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
below in order, recording questions, evidence, decisions, and follow-up work. below in order, recording questions, evidence, decisions, and follow-up work.
Items 1–5 record selected architectural directions. Detailed design, Items 1–6 record selected architectural directions. Detailed design and the
read-only filesystem and schema decisions, and the remaining topics are still read-only filesystem and schema decisions remain open. Item 7 records the
under discussion. completed Audita comparison and its implications for these decisions.
Item 7 records the completed Audita comparison and its implications for the
remaining decisions.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
capabilities as fully as practical. Application ownership of a policy does not capabilities as fully as practical. Application ownership of a policy does not
@@ -445,22 +443,119 @@ unusable or unaccepted stored artifacts. Continue with item 6.
## 6. Inspection And Execution Bookkeeping ## 6. Inspection And Execution Bookkeeping
Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations, **Discussion / decision:** Direction agreed: use native PromptKit inspection
and `ExecutionGroup` to reconsider surrounding wrappers and copied report types. and execution facts, with CommandKit owning standardized operator presentation.
Notarius retains stage correlation, artifact provenance, storage, and disclosure Remove duplicate LLM reporting and accounting machinery. Notarius retains
policy; CommandKit is the planned owner of standardized operator presentation. application context, domain decisions, durable artifacts, and their provenance.
No application changes have been implemented.
Questions: ### Ownership
- Which reports and accounting mechanisms can use upstream results directly? | Responsibility | Owner |
- Which application projections remain necessary for durable contracts? | --- | --- |
- Can the PromptKit migration avoid building temporary machinery that the | Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
CommandKit adoption would immediately replace? | Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
**Discussion / decision:** Pending. ### Native Inspection And Operator Presentation
**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit Remove the profile-inspector wrapper, copied effective-model-parameter types,
follow-up so report and adapter decisions are made once. and local reconstruction of upstream configuration. Use native inspection for
configured resources and native prepared/result data for the target actually
executed, including request overrides. Profile inspection can describe a partial
configured profile; it is not evidence of the completed execution target.
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
preview, and opt-in smoke commands. Notarius supplies configuration/engine
loaders and application request builders. Avoid building temporary custom
commands or report machinery that the planned CommandKit adoption would replace.
Retain application pipeline preflight: resolve modules, artifact compatibility,
references, and known prompt/profile selections. Use native definition checks
without simulating stage execution or requiring provider credentials. Concrete
request preparation remains the place to check the completed target, inputs,
schema, and credential requirements.
CommandKit supplies formatting, visibility, progress presentation, and delivery
error handling. Notarius supplies report content, domain outcomes, and exit-status
mapping. Preserve deliberate machine-output contracts rather than treating every
receipt as a generic formatted report.
### Native Execution Facts
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
for provenance or optional debug capture. Remove unconditional
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
- Use native observations for terminal execution facts, including failures that
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
preparation or admission failure; it cannot serve as a completion record.
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
Remove the private debug completion wrapper and duplicate synthetic call-ID
machinery where native identity and application attribution suffice.
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
cloning, and independent attempt/usage accumulators. Use native types internally
and project selected fields only at an application reporting or durable boundary.
- Replace the adapter's `LLMProfileRecorder` and provider-forwarding plumbing with
the application projection needed for actual-run provenance. Do not substitute
a catalog inventory for the profiles and targets actually used.
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
initial run, continuation runs, and nested validator runs as applicable.
Preserve terminal-coverage and token-usage completeness; unknown usage is not
zero. Do not count both attempt observations and terminal aggregates, or add
overlapping groups as though they were disjoint work.
Application execution code should attach scope and observers, run and assess the
native result, and record its disposition and execution references. This does
not require a replacement general-purpose LLM interface. A successful native
execution can still produce a domain-rejected candidate; provider success and
application acceptance remain separate facts.
### Application Records And Durable Boundaries
Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an
artifact, which validators assessed it, whether continuation or a safe fallback
was used, and whether the artifact was generated now or reused. PromptKit cannot
infer that meaning from provider execution alone.
Under item 5, a reused artifact retains its original execution and acceptance
provenance. The current invocation records the reuse decision without counting
the earlier execution's token usage as current work or claiming new validation.
Retain small, deliberate application projections for the
[published manifest](../integrations/json-output.md) and
[run-result receipt](../integrations/run-result.md), with explicit compatibility
decisions for changed fields. Durable contracts should not become raw upstream
type dumps or require parallel runtime representations of every native result.
Keep application debug artifact placement, opt-in capture, retention, and
disclosure handling. Native callbacks supply execution data; they do not own
Notarius's storage policy. Detailed prepared messages belong in the selected
debug capture path, while ordinary provenance can retain only the needed fields.
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
### Remaining Implementation Work
Select the minimal application scope and durable fields, observer/group
enrollment, and compatibility treatment for existing manifests, receipts, and
debug artifacts. Coordinate command migration with the roadmap's CommandKit
follow-up. The core ownership change needs no new PromptKit capability; any
remaining upstream request should identify a concrete unmet requirement.
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
significant boundary changes in successor ADRs. Update current-behavior
documentation when implemented. Verify failure reporting without a result,
continuation/validator accounting, unknown usage, actual-target provenance,
checkpoint reuse accounting, and optional debug capture during implementation.
## 7. Compare Audita's PromptKit Workflow ## 7. Compare Audita's PromptKit Workflow
@@ -619,9 +714,10 @@ flow and the remaining repair-prompt, response-contract, and policy details;
the comparison alone did not settle those details. the comparison alone did not settle those details.
Item 4 now records agreement to use native admission with bounded application Item 4 now records agreement to use native admission with bounded application
workers. Item 6 can draw on Audita's observations and CommandKit integration. workers. Item 5 selects simple application-owned recovery matching and removes
Item 5 now selects simple application-owned recovery matching and removes the the upstream execution-fingerprint prerequisite. Item 6 adopts native execution
upstream execution-fingerprint prerequisite. observations and CommandKit presentation, retaining application attribution,
domain outcomes, and durable provenance.
The two requested capabilities already work with v0.23.2; no new PromptKit The two requested capabilities already work with v0.23.2; no new PromptKit
feature is needed merely to provide replaceable defaults or domain-directed feature is needed merely to provide replaceable defaults or domain-directed
continuation. Any remaining upstream request should identify the specific unmet continuation. Any remaining upstream request should identify the specific unmet