Record Audita PromptKit workflow review and architectural implications

This commit is contained in:
2026-11-18 19:10:53 +00:00
parent abecb349e4
commit 5a527c789d

View File

@@ -8,7 +8,8 @@ below in order, recording questions, evidence, decisions, and follow-up work.
Items 1 and 2 record selected architectural directions. Detailed design,
read-only filesystem and schema decisions, and the remaining topics are still
under discussion.
The Audita comparison is a seventh topic and may inform earlier decisions.
Item 7 records the completed Audita comparison and its implications for the
remaining decisions.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
capabilities as fully as practical. Application ownership of a policy does not
@@ -160,6 +161,9 @@ Questions:
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
replacement for the existing latest-response-only policy or full stage loop.
The Audita review in item 7 demonstrates application-owned validation inside
native continuation coordination. Evaluate adopting that model on its merits;
the existing Notarius history policy is subject to reconsideration under item 1.
## 4. Global Scheduling And Admission
@@ -221,23 +225,181 @@ follow-up so report and adapter decisions are made once.
## 7. Compare Audita's PromptKit Workflow
Review the sibling Audita checkout at `../audita` relative to the Notarius
repository root. Audita uses PromptKit for a workflow the user identifies as
very similar, potentially identical. Verify the similarities and differences
before drawing architectural conclusions.
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
not new Notarius contracts. The source links below assume the sibling checkout.
Questions:
### Embedded Defaults And Operator Replacement
- What are Audita's PromptKit version, execution flow, and ownership boundaries?
- How does it handle candidate assessment, correction, retries, scheduling,
configuration, inspection, and diagnostics?
- Which simpler patterns could Notarius adopt directly?
- Which differences arise from actual product requirements?
- Does duplication across the two applications reveal work that belongs in
PromptKit, ConfigKit, or CommandKit?
- One captured engine is assembled from PromptKit's default catalog, the
embedded Audita underlay, and a higher-priority operator underlay.
`FilesystemInputs` supplies eleven native prompt definitions in
`collections.audita`: four proposal prompts, four repair prompts, and three
validator prompts.
- Operator sources are optional `prompts.yml` and recursively selected YAML
under `prompts/`, anchored to the application main source's recorded context.
Both locations feed one operator underlay and are peers; conflicting peer
definitions fail rather than receiving filename-based precedence.
- Replacement is by resource identity, not filename. For example, a complete
prompt record with local ID `modules.grammar.repair` under
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
other embedded definitions remain available. Native same-ID replacement is
whole-record replacement, not recursive merging of individual prompt fields.
- Native imports and message-file context are retained. Tests show that an
existing engine retains captured message content after a file edit and a new
engine sees the edit. They also verify inherited message-file resolution,
main-relative discovery independent of the working directory, and peer
conflict rejection.
- A diagnostic embedded-only construction can classify a loading error; it
does not silently fall back to defaults after an operator-source failure.
**Discussion / decision:** Pending; Audita has not yet been reviewed for this
comparison.
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
**Follow-up:** Record the reviewed Audita revision and evidence, then feed the
findings back into items 1–6 and the adoption roadmap.
### Prompt Content And Schema Ownership Are Separate
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
small prompt registry records IDs and embedded paths; it does not reconstruct
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
supplied as template variables. Processing supplies the application-owned values
for those shared names, but an operator replacement decides whether its template
references them at all. They are not forcibly appended to replacement messages.
The processing request independently supplies a complete native output contract:
JSON Schema validation, the schema path selected for that operation, and its
structural-repair budget. The three schemas describe a correction set, a repair
decision, and validator decisions. PromptKit receives their canonical embedded
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
change the processing code's expected response shape. A test replaces a prompt
with a text/no-validation declaration and still exercises the processing
request's JSON-schema repair contract.
Audita still keeps a response-schema key/metadata/hash registry and application
decoding checks. Its use of native loading does not prove that every schema
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
a concrete option: module-owned response requirements expressed directly through
native per-request contracts. Standalone native preview/preparation can use the
prompt declaration's contract; processing supplies its own override.
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
`prepareCall` and `OutputContract` in the
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
### Native Continuation With Application Assessment
The implemented flow is:
1. Generate a section's proposal set, capturing a native immutable exchange when
correction repair is enabled. This retains the actual proposal messages and
response; it does not require debug recording.
2. Apply application validation. An actionable rejected proposal starts its own
branch from the original proposal exchange. Unavailable-only findings and
no-effect corrections do not start repair. Existing approvals remain separate.
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
Each follow-up selects a distinct repair prompt, repair profile, and repair
response schema. The request carries the original proposal reference, current
correction, ordered findings, and available before/after text previews.
4. PromptKit assembles history and progresses the conversation. A second turn
retains the first repair instruction and response. A sibling branch starts
from the original exchange and does not inherit another branch's repairs.
Audita does not rerender the original proposal or manually replay messages.
5. Audita decodes each delivered decision. Withdrawal ends the branch without
applying that correction. A revision undergoes the full application validation
chain again, including validators that approved its earlier version. Approval
stops the branch; actionable rejection can request another turn.
6. The final permitted response is assessed before exhaustion is determined.
Requesting another turn at that boundary does not allocate another logical
call. Recoverable provider/invalid-output failures finish the branch as
unavailable; fatal errors or parent cancellation interrupt processing.
Canceled, undelivered responses are diagnostic evidence only, never approvals.
Transport retries and structural repair operate inside each PromptKit execution;
they are separate from the semantic branch's additional-turn budget. Tests show
one semantic repair turn containing multiple provider attempts for structural
repair. The application retains acceptance, withdrawal, application ordering,
and reporting; PromptKit retains conversation and execution mechanics.
Audita's prompt roles support this model deliberately: system instructions carry
durable principles, the initial user message carries the initial module task,
and the repair prompt supplies a new user assignment. Adopting continuation in
Notarius would require reviewing the complete conversation and response-format
transitions, rather than mechanically appending to prompts written for fresh
replacement requests.
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
and [prompt guidance](../../../audita/docs/internal/prompts.md).
### Other Boundaries And Limits Of The Comparison
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
wrapper, and application diagnostic types. Adopt the demonstrated native
mechanisms without assuming Notarius should copy those wrappers; item 1's
adapter-removal direction remains in force.
- Proposal, validator, and repair roles share one engine and its per-backend
admission. The runner separately bounds section work. This demonstrates useful
separation of application work scheduling and provider capacity, but does not
supply Notarius's independent cross-backend ceiling.
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
feed a native `ExecutionGroup`; application records add logical-call and
stage/section/proposal attribution. Nested validator executions enroll
explicitly. Counts preserve terminal coverage and token-usage completeness.
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
commands using Audita's engine construction, including request preparation,
validation, and smoke diagnostics.
- Audita edits independently assessable transcript corrections; Notarius accepts
typed stage artifacts and has cross-stage dependencies. Audita branches at an
individual correction within a proposal response. Notarius must deliberately
choose its correction unit rather than assume per-entity branching.
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
reuse identity or persisted validation-provenance requirements.
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
[observations](../../../audita/internal/framework/llm/observations.go),
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
[component scope](../../../audita/docs/internal/overview.md).
### Implications For The Remaining Discussion
Item 2's native underlay direction has a working application precedent. Audita
also offers evidence for ordinary embedded filesystems and per-request schema
selection, while our filesystem and schema decisions remain open.
For item 3, native continuation is a practical alternative to the current
latest-response-only protocol. Keeping domain validation local does not require
keeping conversation assembly or the turn loop local. Decide whether Notarius
should adopt accumulated branch history, separate repair prompts, and revised
response contracts; none has been selected merely by this comparison.
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit
integration. Item 5 remains independent upstream/application identity work.
The two requested capabilities already work with v0.23.2; no new PromptKit
feature is needed merely to provide replaceable defaults or domain-directed
continuation. Any remaining upstream request should identify the specific unmet
requirement after reconsidering Notarius's historical behavior.
### Verification
Read the production composition, adapter, coordinator, domain repair callbacks,
prompt/schema definitions, and focused behavior tests. The following checks
passed from the Audita checkout:
```sh
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
```
The subprocess checks exercise assembled configuration, real native continuation
messages, operator replacement, final-turn assessment, sibling isolation,
repair-disabled behavior, and cancellation using a local fake provider. These
checks establish implementation behavior, not live-model quality. No live model
calls or application source changes were made. Broader migration decisions and
changes to the adoption roadmap remain for discussion.