diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 14b486b4..6f6ed296 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -8,7 +8,8 @@ below in order, recording questions, evidence, decisions, and follow-up work. Items 1 and 2 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. -The Audita comparison is a seventh topic and may inform earlier decisions. +Item 7 records the completed Audita comparison and its implications for the +remaining decisions. The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not @@ -160,6 +161,9 @@ Questions: [ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct replacement for the existing latest-response-only policy or full stage loop. +The Audita review in item 7 demonstrates application-owned validation inside +native continuation coordination. Evaluate adopting that model on its merits; +the existing Notarius history policy is subject to reconsideration under item 1. ## 4. Global Scheduling And Admission @@ -221,23 +225,181 @@ follow-up so report and adapter decisions are made once. ## 7. Compare Audita's PromptKit Workflow -Review the sibling Audita checkout at `../audita` relative to the Notarius -repository root. Audita uses PromptKit for a workflow the user identifies as -very similar, potentially identical. Verify the similarities and differences -before drawing architectural conclusions. +**Review status:** Completed on 2026-11-18 against the clean sibling checkout +`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`. +Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0, +ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision, +not new Notarius contracts. The source links below assume the sibling checkout. -Questions: +### Embedded Defaults And Operator Replacement -- What are Audita's PromptKit version, execution flow, and ownership boundaries? -- How does it handle candidate assessment, correction, retries, scheduling, - configuration, inspection, and diagnostics? -- Which simpler patterns could Notarius adopt directly? -- Which differences arise from actual product requirements? -- Does duplication across the two applications reveal work that belongs in - PromptKit, ConfigKit, or CommandKit? +- One captured engine is assembled from PromptKit's default catalog, the + embedded Audita underlay, and a higher-priority operator underlay. + `FilesystemInputs` supplies eleven native prompt definitions in + `collections.audita`: four proposal prompts, four repair prompts, and three + validator prompts. +- Operator sources are optional `prompts.yml` and recursively selected YAML + under `prompts/`, anchored to the application main source's recorded context. + Both locations feed one operator underlay and are peers; conflicting peer + definitions fail rather than receiving filename-based precedence. +- Replacement is by resource identity, not filename. For example, a complete + prompt record with local ID `modules.grammar.repair` under + `collections.audita.prompts` replaces `audita/modules.grammar.repair` while + other embedded definitions remain available. Native same-ID replacement is + whole-record replacement, not recursive merging of individual prompt fields. +- Native imports and message-file context are retained. Tests show that an + existing engine retains captured message content after a file edit and a new + engine sees the edit. They also verify inherited message-file resolution, + main-relative discovery independent of the working directory, and peer + conflict rejection. +- A diagnostic embedded-only construction can classify a loading error; it + does not silently fall back to defaults after an operator-source failure. -**Discussion / decision:** Pending; Audita has not yet been reviewed for this -comparison. +Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go), +[source tests](../../../audita/internal/framework/llm/engine_sources_test.go), +and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go). -**Follow-up:** Record the reviewed Audita revision and evidence, then feed the -findings back into items 1–6 and the adoption roadmap. +### Prompt Content And Schema Ownership Are Separate + +Audita uses ordinary embedded filesystems for prompt and schema assets. Its +small prompt registry records IDs and embedded paths; it does not reconstruct +Notarius-style per-module virtual prompt trees. Shared instruction fragments are +supplied as template variables. Processing supplies the application-owned values +for those shared names, but an operator replacement decides whether its template +references them at all. They are not forcibly appended to replacement messages. + +The processing request independently supplies a complete native output contract: +JSON Schema validation, the schema path selected for that operation, and its +structural-repair budget. The three schemas describe a correction set, a repair +decision, and validator decisions. PromptKit receives their canonical embedded +filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not +change the processing code's expected response shape. A test replaces a prompt +with a text/no-validation declaration and still exercises the processing +request's JSON-schema repair contract. + +Audita still keeps a response-schema key/metadata/hash registry and application +decoding checks. Its use of native loading does not prove that every schema +wrapper is necessary, nor decide Notarius's deferred schema design. It supplies +a concrete option: module-owned response requirements expressed directly through +native per-request contracts. Standalone native preview/preparation can use the +prompt declaration's contract; processing supplies its own override. + +Evidence: [prompt registry](../../../audita/internal/prompts/registry.go), +[schema registry](../../../audita/internal/framework/responseschema/registry.go), +`prepareCall` and `OutputContract` in the +[processing adapter](../../../audita/internal/framework/llm/adapter.go), and +`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in +[adapter tests](../../../audita/internal/framework/llm/adapter_test.go). + +### Native Continuation With Application Assessment + +The implemented flow is: + +1. Generate a section's proposal set, capturing a native immutable exchange when + correction repair is enabled. This retains the actual proposal messages and + response; it does not require debug recording. +2. Apply application validation. An actionable rejected proposal starts its own + branch from the original proposal exchange. Unavailable-only findings and + no-effect corrections do not start repair. Existing approvals remain separate. +3. Call `Engine.Coordinate` with that exchange and an additional-turn budget. + Each follow-up selects a distinct repair prompt, repair profile, and repair + response schema. The request carries the original proposal reference, current + correction, ordered findings, and available before/after text previews. +4. PromptKit assembles history and progresses the conversation. A second turn + retains the first repair instruction and response. A sibling branch starts + from the original exchange and does not inherit another branch's repairs. + Audita does not rerender the original proposal or manually replay messages. +5. Audita decodes each delivered decision. Withdrawal ends the branch without + applying that correction. A revision undergoes the full application validation + chain again, including validators that approved its earlier version. Approval + stops the branch; actionable rejection can request another turn. +6. The final permitted response is assessed before exhaustion is determined. + Requesting another turn at that boundary does not allocate another logical + call. Recoverable provider/invalid-output failures finish the branch as + unavailable; fatal errors or parent cancellation interrupt processing. + Canceled, undelivered responses are diagnostic evidence only, never approvals. + +Transport retries and structural repair operate inside each PromptKit execution; +they are separate from the semantic branch's additional-turn budget. Tests show +one semantic repair turn containing multiple provider attempts for structural +repair. The application retains acceptance, withdrawal, application ordering, +and reporting; PromptKit retains conversation and execution mechanics. + +Audita's prompt roles support this model deliberately: system instructions carry +durable principles, the initial user message carries the initial module task, +and the repair prompt supplies a new user assignment. Adopting continuation in +Notarius would require reviewing the complete conversation and response-format +transitions, rather than mechanically appending to prompts written for fresh +replacement requests. + +Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), +[domain repair flow](../../../audita/internal/framework/runner/repair.go), +[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go), +[runner repair tests](../../../audita/internal/framework/runner/repair_test.go), +and [prompt guidance](../../../audita/docs/internal/prompts.md). + +### Other Boundaries And Limits Of The Comparison + +- Audita still has `RoleClient`, a structured-call interface, an opaque exchange + wrapper, and application diagnostic types. Adopt the demonstrated native + mechanisms without assuming Notarius should copy those wrappers; item 1's + adapter-removal direction remains in force. +- Proposal, validator, and repair roles share one engine and its per-backend + admission. The runner separately bounds section work. This demonstrates useful + separation of application work scheduling and provider capacity, but does not + supply Notarius's independent cross-backend ceiling. +- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations + feed a native `ExecutionGroup`; application records add logical-call and + stage/section/proposal attribution. Nested validator executions enroll + explicitly. Counts preserve terminal coverage and token-usage completeness. +- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt + commands using Audita's engine construction, including request preparation, + validation, and smoke diagnostics. +- Audita edits independently assessable transcript corrections; Notarius accepts + typed stage artifacts and has cross-stage dependencies. Audita branches at an + individual correction within a proposal response. Notarius must deliberately + choose its correction unit rather than assume per-entity branching. +- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's + reuse identity or persisted validation-provenance requirements. + +Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md), +[observations](../../../audita/internal/framework/llm/observations.go), +[CLI integrations](../../../audita/internal/cli/app_wiring.go), and +[component scope](../../../audita/docs/internal/overview.md). + +### Implications For The Remaining Discussion + +Item 2's native underlay direction has a working application precedent. Audita +also offers evidence for ordinary embedded filesystems and per-request schema +selection, while our filesystem and schema decisions remain open. + +For item 3, native continuation is a practical alternative to the current +latest-response-only protocol. Keeping domain validation local does not require +keeping conversation assembly or the turn loop local. Decide whether Notarius +should adopt accumulated branch history, separate repair prompts, and revised +response contracts; none has been selected merely by this comparison. + +Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit +integration. Item 5 remains independent upstream/application identity work. +The two requested capabilities already work with v0.23.2; no new PromptKit +feature is needed merely to provide replaceable defaults or domain-directed +continuation. Any remaining upstream request should identify the specific unmet +requirement after reconsidering Notarius's historical behavior. + +### Verification + +Read the production composition, adapter, coordinator, domain repair callbacks, +prompt/schema definitions, and focused behavior tests. The following checks +passed from the Audita checkout: + +```sh +go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation +go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1 +``` + +The subprocess checks exercise assembled configuration, real native continuation +messages, operator replacement, final-turn assessment, sibling isolation, +repair-disabled behavior, and cancellation using a local fake provider. These +checks establish implementation behavior, not live-model quality. No live model +calls or application source changes were made. Broader migration decisions and +changes to the adoption roadmap remain for discussion.