Record Audita PromptKit workflow review and architectural implications
This commit is contained in:
@@ -8,7 +8,8 @@ below in order, recording questions, evidence, decisions, and follow-up work.
|
||||
Items 1 and 2 record selected architectural directions. Detailed design,
|
||||
read-only filesystem and schema decisions, and the remaining topics are still
|
||||
under discussion.
|
||||
The Audita comparison is a seventh topic and may inform earlier decisions.
|
||||
Item 7 records the completed Audita comparison and its implications for the
|
||||
remaining decisions.
|
||||
|
||||
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
|
||||
capabilities as fully as practical. Application ownership of a policy does not
|
||||
@@ -160,6 +161,9 @@ Questions:
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
|
||||
replacement for the existing latest-response-only policy or full stage loop.
|
||||
The Audita review in item 7 demonstrates application-owned validation inside
|
||||
native continuation coordination. Evaluate adopting that model on its merits;
|
||||
the existing Notarius history policy is subject to reconsideration under item 1.
|
||||
|
||||
## 4. Global Scheduling And Admission
|
||||
|
||||
@@ -221,23 +225,181 @@ follow-up so report and adapter decisions are made once.
|
||||
|
||||
## 7. Compare Audita's PromptKit Workflow
|
||||
|
||||
Review the sibling Audita checkout at `../audita` relative to the Notarius
|
||||
repository root. Audita uses PromptKit for a workflow the user identifies as
|
||||
very similar, potentially identical. Verify the similarities and differences
|
||||
before drawing architectural conclusions.
|
||||
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
|
||||
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
|
||||
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
|
||||
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
|
||||
not new Notarius contracts. The source links below assume the sibling checkout.
|
||||
|
||||
Questions:
|
||||
### Embedded Defaults And Operator Replacement
|
||||
|
||||
- What are Audita's PromptKit version, execution flow, and ownership boundaries?
|
||||
- How does it handle candidate assessment, correction, retries, scheduling,
|
||||
configuration, inspection, and diagnostics?
|
||||
- Which simpler patterns could Notarius adopt directly?
|
||||
- Which differences arise from actual product requirements?
|
||||
- Does duplication across the two applications reveal work that belongs in
|
||||
PromptKit, ConfigKit, or CommandKit?
|
||||
- One captured engine is assembled from PromptKit's default catalog, the
|
||||
embedded Audita underlay, and a higher-priority operator underlay.
|
||||
`FilesystemInputs` supplies eleven native prompt definitions in
|
||||
`collections.audita`: four proposal prompts, four repair prompts, and three
|
||||
validator prompts.
|
||||
- Operator sources are optional `prompts.yml` and recursively selected YAML
|
||||
under `prompts/`, anchored to the application main source's recorded context.
|
||||
Both locations feed one operator underlay and are peers; conflicting peer
|
||||
definitions fail rather than receiving filename-based precedence.
|
||||
- Replacement is by resource identity, not filename. For example, a complete
|
||||
prompt record with local ID `modules.grammar.repair` under
|
||||
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
|
||||
other embedded definitions remain available. Native same-ID replacement is
|
||||
whole-record replacement, not recursive merging of individual prompt fields.
|
||||
- Native imports and message-file context are retained. Tests show that an
|
||||
existing engine retains captured message content after a file edit and a new
|
||||
engine sees the edit. They also verify inherited message-file resolution,
|
||||
main-relative discovery independent of the working directory, and peer
|
||||
conflict rejection.
|
||||
- A diagnostic embedded-only construction can classify a loading error; it
|
||||
does not silently fall back to defaults after an operator-source failure.
|
||||
|
||||
**Discussion / decision:** Pending; Audita has not yet been reviewed for this
|
||||
comparison.
|
||||
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
|
||||
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
|
||||
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
|
||||
|
||||
**Follow-up:** Record the reviewed Audita revision and evidence, then feed the
|
||||
findings back into items 1–6 and the adoption roadmap.
|
||||
### Prompt Content And Schema Ownership Are Separate
|
||||
|
||||
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
|
||||
small prompt registry records IDs and embedded paths; it does not reconstruct
|
||||
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
|
||||
supplied as template variables. Processing supplies the application-owned values
|
||||
for those shared names, but an operator replacement decides whether its template
|
||||
references them at all. They are not forcibly appended to replacement messages.
|
||||
|
||||
The processing request independently supplies a complete native output contract:
|
||||
JSON Schema validation, the schema path selected for that operation, and its
|
||||
structural-repair budget. The three schemas describe a correction set, a repair
|
||||
decision, and validator decisions. PromptKit receives their canonical embedded
|
||||
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
|
||||
change the processing code's expected response shape. A test replaces a prompt
|
||||
with a text/no-validation declaration and still exercises the processing
|
||||
request's JSON-schema repair contract.
|
||||
|
||||
Audita still keeps a response-schema key/metadata/hash registry and application
|
||||
decoding checks. Its use of native loading does not prove that every schema
|
||||
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
|
||||
a concrete option: module-owned response requirements expressed directly through
|
||||
native per-request contracts. Standalone native preview/preparation can use the
|
||||
prompt declaration's contract; processing supplies its own override.
|
||||
|
||||
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
|
||||
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
|
||||
`prepareCall` and `OutputContract` in the
|
||||
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
|
||||
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
|
||||
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
|
||||
|
||||
### Native Continuation With Application Assessment
|
||||
|
||||
The implemented flow is:
|
||||
|
||||
1. Generate a section's proposal set, capturing a native immutable exchange when
|
||||
correction repair is enabled. This retains the actual proposal messages and
|
||||
response; it does not require debug recording.
|
||||
2. Apply application validation. An actionable rejected proposal starts its own
|
||||
branch from the original proposal exchange. Unavailable-only findings and
|
||||
no-effect corrections do not start repair. Existing approvals remain separate.
|
||||
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
|
||||
Each follow-up selects a distinct repair prompt, repair profile, and repair
|
||||
response schema. The request carries the original proposal reference, current
|
||||
correction, ordered findings, and available before/after text previews.
|
||||
4. PromptKit assembles history and progresses the conversation. A second turn
|
||||
retains the first repair instruction and response. A sibling branch starts
|
||||
from the original exchange and does not inherit another branch's repairs.
|
||||
Audita does not rerender the original proposal or manually replay messages.
|
||||
5. Audita decodes each delivered decision. Withdrawal ends the branch without
|
||||
applying that correction. A revision undergoes the full application validation
|
||||
chain again, including validators that approved its earlier version. Approval
|
||||
stops the branch; actionable rejection can request another turn.
|
||||
6. The final permitted response is assessed before exhaustion is determined.
|
||||
Requesting another turn at that boundary does not allocate another logical
|
||||
call. Recoverable provider/invalid-output failures finish the branch as
|
||||
unavailable; fatal errors or parent cancellation interrupt processing.
|
||||
Canceled, undelivered responses are diagnostic evidence only, never approvals.
|
||||
|
||||
Transport retries and structural repair operate inside each PromptKit execution;
|
||||
they are separate from the semantic branch's additional-turn budget. Tests show
|
||||
one semantic repair turn containing multiple provider attempts for structural
|
||||
repair. The application retains acceptance, withdrawal, application ordering,
|
||||
and reporting; PromptKit retains conversation and execution mechanics.
|
||||
|
||||
Audita's prompt roles support this model deliberately: system instructions carry
|
||||
durable principles, the initial user message carries the initial module task,
|
||||
and the repair prompt supplies a new user assignment. Adopting continuation in
|
||||
Notarius would require reviewing the complete conversation and response-format
|
||||
transitions, rather than mechanically appending to prompts written for fresh
|
||||
replacement requests.
|
||||
|
||||
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
|
||||
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
|
||||
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
|
||||
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
|
||||
and [prompt guidance](../../../audita/docs/internal/prompts.md).
|
||||
|
||||
### Other Boundaries And Limits Of The Comparison
|
||||
|
||||
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
|
||||
wrapper, and application diagnostic types. Adopt the demonstrated native
|
||||
mechanisms without assuming Notarius should copy those wrappers; item 1's
|
||||
adapter-removal direction remains in force.
|
||||
- Proposal, validator, and repair roles share one engine and its per-backend
|
||||
admission. The runner separately bounds section work. This demonstrates useful
|
||||
separation of application work scheduling and provider capacity, but does not
|
||||
supply Notarius's independent cross-backend ceiling.
|
||||
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
|
||||
feed a native `ExecutionGroup`; application records add logical-call and
|
||||
stage/section/proposal attribution. Nested validator executions enroll
|
||||
explicitly. Counts preserve terminal coverage and token-usage completeness.
|
||||
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
|
||||
commands using Audita's engine construction, including request preparation,
|
||||
validation, and smoke diagnostics.
|
||||
- Audita edits independently assessable transcript corrections; Notarius accepts
|
||||
typed stage artifacts and has cross-stage dependencies. Audita branches at an
|
||||
individual correction within a proposal response. Notarius must deliberately
|
||||
choose its correction unit rather than assume per-entity branching.
|
||||
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
|
||||
reuse identity or persisted validation-provenance requirements.
|
||||
|
||||
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
|
||||
[observations](../../../audita/internal/framework/llm/observations.go),
|
||||
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
|
||||
[component scope](../../../audita/docs/internal/overview.md).
|
||||
|
||||
### Implications For The Remaining Discussion
|
||||
|
||||
Item 2's native underlay direction has a working application precedent. Audita
|
||||
also offers evidence for ordinary embedded filesystems and per-request schema
|
||||
selection, while our filesystem and schema decisions remain open.
|
||||
|
||||
For item 3, native continuation is a practical alternative to the current
|
||||
latest-response-only protocol. Keeping domain validation local does not require
|
||||
keeping conversation assembly or the turn loop local. Decide whether Notarius
|
||||
should adopt accumulated branch history, separate repair prompts, and revised
|
||||
response contracts; none has been selected merely by this comparison.
|
||||
|
||||
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit
|
||||
integration. Item 5 remains independent upstream/application identity work.
|
||||
The two requested capabilities already work with v0.23.2; no new PromptKit
|
||||
feature is needed merely to provide replaceable defaults or domain-directed
|
||||
continuation. Any remaining upstream request should identify the specific unmet
|
||||
requirement after reconsidering Notarius's historical behavior.
|
||||
|
||||
### Verification
|
||||
|
||||
Read the production composition, adapter, coordinator, domain repair callbacks,
|
||||
prompt/schema definitions, and focused behavior tests. The following checks
|
||||
passed from the Audita checkout:
|
||||
|
||||
```sh
|
||||
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
|
||||
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
|
||||
```
|
||||
|
||||
The subprocess checks exercise assembled configuration, real native continuation
|
||||
messages, operator replacement, final-turn assessment, sibling isolation,
|
||||
repair-disabled behavior, and cancellation using a local fake provider. These
|
||||
checks establish implementation behavior, not live-model quality. No live model
|
||||
calls or application source changes were made. Broader migration decisions and
|
||||
changes to the adoption roadmap remain for discussion.
|
||||
|
||||
Reference in New Issue
Block a user