Record inspection and execution bookkeeping ownership
This commit is contained in:
@@ -5,11 +5,9 @@
|
||||
Working notepad for reviewing the ownership boundaries in the
|
||||
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
||||
below in order, recording questions, evidence, decisions, and follow-up work.
|
||||
Items 1–5 record selected architectural directions. Detailed design,
|
||||
read-only filesystem and schema decisions, and the remaining topics are still
|
||||
under discussion.
|
||||
Item 7 records the completed Audita comparison and its implications for the
|
||||
remaining decisions.
|
||||
Items 1–6 record selected architectural directions. Detailed design and the
|
||||
read-only filesystem and schema decisions remain open. Item 7 records the
|
||||
completed Audita comparison and its implications for these decisions.
|
||||
|
||||
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
|
||||
capabilities as fully as practical. Application ownership of a policy does not
|
||||
@@ -445,22 +443,119 @@ unusable or unaccepted stored artifacts. Continue with item 6.
|
||||
|
||||
## 6. Inspection And Execution Bookkeeping
|
||||
|
||||
Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations,
|
||||
and `ExecutionGroup` to reconsider surrounding wrappers and copied report types.
|
||||
Notarius retains stage correlation, artifact provenance, storage, and disclosure
|
||||
policy; CommandKit is the planned owner of standardized operator presentation.
|
||||
**Discussion / decision:** Direction agreed: use native PromptKit inspection
|
||||
and execution facts, with CommandKit owning standardized operator presentation.
|
||||
Remove duplicate LLM reporting and accounting machinery. Notarius retains
|
||||
application context, domain decisions, durable artifacts, and their provenance.
|
||||
No application changes have been implemented.
|
||||
|
||||
Questions:
|
||||
### Ownership
|
||||
|
||||
- Which reports and accounting mechanisms can use upstream results directly?
|
||||
- Which application projections remain necessary for durable contracts?
|
||||
- Can the PromptKit migration avoid building temporary machinery that the
|
||||
CommandKit adoption would immediately replace?
|
||||
| Responsibility | Owner |
|
||||
| --- | --- |
|
||||
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
|
||||
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
|
||||
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
|
||||
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
|
||||
| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. |
|
||||
| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. |
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
### Native Inspection And Operator Presentation
|
||||
|
||||
**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit
|
||||
follow-up so report and adapter decisions are made once.
|
||||
Remove the profile-inspector wrapper, copied effective-model-parameter types,
|
||||
and local reconstruction of upstream configuration. Use native inspection for
|
||||
configured resources and native prepared/result data for the target actually
|
||||
executed, including request overrides. Profile inspection can describe a partial
|
||||
configured profile; it is not evidence of the completed execution target.
|
||||
|
||||
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
|
||||
preview, and opt-in smoke commands. Notarius supplies configuration/engine
|
||||
loaders and application request builders. Avoid building temporary custom
|
||||
commands or report machinery that the planned CommandKit adoption would replace.
|
||||
|
||||
Retain application pipeline preflight: resolve modules, artifact compatibility,
|
||||
references, and known prompt/profile selections. Use native definition checks
|
||||
without simulating stage execution or requiring provider credentials. Concrete
|
||||
request preparation remains the place to check the completed target, inputs,
|
||||
schema, and credential requirements.
|
||||
|
||||
CommandKit supplies formatting, visibility, progress presentation, and delivery
|
||||
error handling. Notarius supplies report content, domain outcomes, and exit-status
|
||||
mapping. Preserve deliberate machine-output contracts rather than treating every
|
||||
receipt as a generic formatted report.
|
||||
|
||||
### Native Execution Facts
|
||||
|
||||
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
|
||||
for provenance or optional debug capture. Remove unconditional
|
||||
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
|
||||
- Use native observations for terminal execution facts, including failures that
|
||||
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
|
||||
preparation or admission failure; it cannot serve as a completion record.
|
||||
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
|
||||
Remove the private debug completion wrapper and duplicate synthetic call-ID
|
||||
machinery where native identity and application attribution suffice.
|
||||
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
|
||||
cloning, and independent attempt/usage accumulators. Use native types internally
|
||||
and project selected fields only at an application reporting or durable boundary.
|
||||
- Replace the adapter's `LLMProfileRecorder` and provider-forwarding plumbing with
|
||||
the application projection needed for actual-run provenance. Do not substitute
|
||||
a catalog inventory for the profiles and targets actually used.
|
||||
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
|
||||
initial run, continuation runs, and nested validator runs as applicable.
|
||||
Preserve terminal-coverage and token-usage completeness; unknown usage is not
|
||||
zero. Do not count both attempt observations and terminal aggregates, or add
|
||||
overlapping groups as though they were disjoint work.
|
||||
|
||||
Application execution code should attach scope and observers, run and assess the
|
||||
native result, and record its disposition and execution references. This does
|
||||
not require a replacement general-purpose LLM interface. A successful native
|
||||
execution can still produce a domain-rejected candidate; provider success and
|
||||
application acceptance remain separate facts.
|
||||
|
||||
### Application Records And Durable Boundaries
|
||||
|
||||
Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an
|
||||
artifact, which validators assessed it, whether continuation or a safe fallback
|
||||
was used, and whether the artifact was generated now or reused. PromptKit cannot
|
||||
infer that meaning from provider execution alone.
|
||||
|
||||
Under item 5, a reused artifact retains its original execution and acceptance
|
||||
provenance. The current invocation records the reuse decision without counting
|
||||
the earlier execution's token usage as current work or claiming new validation.
|
||||
|
||||
Retain small, deliberate application projections for the
|
||||
[published manifest](../integrations/json-output.md) and
|
||||
[run-result receipt](../integrations/run-result.md), with explicit compatibility
|
||||
decisions for changed fields. Durable contracts should not become raw upstream
|
||||
type dumps or require parallel runtime representations of every native result.
|
||||
|
||||
Keep application debug artifact placement, opt-in capture, retention, and
|
||||
disclosure handling. Native callbacks supply execution data; they do not own
|
||||
Notarius's storage policy. Detailed prepared messages belong in the selected
|
||||
debug capture path, while ordinary provenance can retain only the needed fields.
|
||||
|
||||
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
|
||||
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
|
||||
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
|
||||
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
|
||||
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
|
||||
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
|
||||
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
|
||||
|
||||
### Remaining Implementation Work
|
||||
|
||||
Select the minimal application scope and durable fields, observer/group
|
||||
enrollment, and compatibility treatment for existing manifests, receipts, and
|
||||
debug artifacts. Coordinate command migration with the roadmap's CommandKit
|
||||
follow-up. The core ownership change needs no new PromptKit capability; any
|
||||
remaining upstream request should identify a concrete unmet requirement.
|
||||
|
||||
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
|
||||
significant boundary changes in successor ADRs. Update current-behavior
|
||||
documentation when implemented. Verify failure reporting without a result,
|
||||
continuation/validator accounting, unknown usage, actual-target provenance,
|
||||
checkpoint reuse accounting, and optional debug capture during implementation.
|
||||
|
||||
## 7. Compare Audita's PromptKit Workflow
|
||||
|
||||
@@ -619,9 +714,10 @@ flow and the remaining repair-prompt, response-contract, and policy details;
|
||||
the comparison alone did not settle those details.
|
||||
|
||||
Item 4 now records agreement to use native admission with bounded application
|
||||
workers. Item 6 can draw on Audita's observations and CommandKit integration.
|
||||
Item 5 now selects simple application-owned recovery matching and removes the
|
||||
upstream execution-fingerprint prerequisite.
|
||||
workers. Item 5 selects simple application-owned recovery matching and removes
|
||||
the upstream execution-fingerprint prerequisite. Item 6 adopts native execution
|
||||
observations and CommandKit presentation, retaining application attribution,
|
||||
domain outcomes, and durable provenance.
|
||||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||||
feature is needed merely to provide replaceable defaults or domain-directed
|
||||
continuation. Any remaining upstream request should identify the specific unmet
|
||||
|
||||
Reference in New Issue
Block a user