diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md new file mode 100644 index 00000000..0b977f19 --- /dev/null +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -0,0 +1,165 @@ +# PromptKit Architecture Discussion + +## Purpose And Status + +Working notepad for reviewing the ownership boundaries in the +[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics +below in order, recording questions, evidence, decisions, and follow-up work. +The six initial topics are proposals for discussion, not accepted changes. +The Audita comparison is a seventh topic and may inform earlier decisions. + +The central question is how much reusable LLM machinery Notarius should retain +as PromptKit, ConfigKit, and CommandKit take on more responsibility. A working +principle to evaluate is that application ownership of a policy does not +necessarily require application ownership of its implementation. + +No architectural changes have been agreed or implemented through this notepad. +The adoption roadmap remains the migration plan; reconcile it with decisions +from this discussion as they are made. Significant changes to accepted +architecture need successor ADRs under the +[documentation policy](../policy/documentation.md#architecture-decision-records). +Current boundaries are described in [Architecture](../policy/architecture.md) +and [LLM Runtime Internals](../internal/llm.md). + +## 1. LLM API And Diagnostic Boundary + +Consider using native PromptKit contracts in LLM-facing implementation packages +and reducing parallel completion, message, usage, execution-summary, and debug +types. Retain application contracts where they serve typed artifact handling, +stage context, durable output compatibility, or disclosure policy. + +Questions: + +- Which abstractions express Notarius behavior, and which merely copy PromptKit? +- Where should direct PromptKit dependencies be permitted? +- Can detailed debug material flow to an application-controlled writer without + being translated through generic pipeline contracts? +- Which persisted representations need compatibility or redaction adapters? + +**Discussion / decision:** Pending. Start here. + +**Follow-up:** Identify the smallest useful application boundary and any +architecture-policy changes it requires. + +## 2. Prompt Assets And Schema Assembly + +Consider replacing custom asset registration and virtual filesystem remounting +with native embedded sources, imports, collections, and source-relative assets. +Modules would continue to own prompt meaning and typed interpretation. + +Questions: + +- Can native sources represent the existing shared fragments and module assets? +- Which registry, remounting, and hashing helpers could disappear? +- What private schema identity is still needed for checkpoint reuse? +- How do we preserve message ordering and cache controls while simplifying + assembly? + +**Discussion / decision:** Pending. + +**Follow-up:** Separate private LLM schema plumbing from durable artifact +schemas; do not assume a captured prompt hash covers schema content. + +## 3. Retry And Semantic Correction Ownership + +Consider narrowing outer stage retries once PromptKit has exhausted its own +transport and structural-repair budgets. Notarius would retain application +assessment, semantic correction, fallback, and acceptance decisions. + +Questions: + +- Which operational failures justify repeating an entire stage? +- Should semantic correction preserve only the latest response or accumulate + conversation history? +- How should retries for non-LLM operations differ from LLM execution retries? +- Would a native orchestration API reduce machinery without obscuring typed + validation and terminal disposition? + +**Discussion / decision:** Pending. + +**Follow-up:** Evaluate behavior changes against +[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and +[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). +PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct +replacement for the existing latest-response-only policy or full stage loop. + +## 4. Global Scheduling And Admission + +Consider whether backend capacity limits and pipeline worker bounds meet the +operational need, or whether a separate cross-backend ceiling remains useful. +If it remains useful, consider implementing that reusable mechanism upstream. + +Questions: + +- What resource or operational budget does each limit protect? +- Should a global permit remain occupied during backend queueing and repair? +- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide + admission support? + +**Discussion / decision:** Pending. + +**Follow-up:** Account for the existing global-limit contract. PromptKit +v0.23.2's per-backend limits do not directly replace it. + +## 5. Execution Identity And Checkpoint Reuse + +Consider moving identity for PromptKit-owned execution semantics upstream. +Notarius would combine opaque upstream identities with its source, module, +reference, schema, and policy identities and retain checkpoint eligibility and +storage decisions. + +Questions: + +- Which identity can be computed before a stage runs, and which needs a concrete + prepared request? +- Which changes should invalidate reuse, and at what granularity? +- Should a conservative application fingerprint be an interim implementation + rather than the architectural destination? +- Does this justify upstream work and a later adoption target? + +**Discussion / decision:** Pending. + +**Follow-up:** Review PromptKit's deferred semantic execution-target fingerprint +proposal. This capability is not supplied by the v0.23.2 adoption target. + +## 6. Inspection And Execution Bookkeeping + +Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations, +and `ExecutionGroup` to reconsider surrounding wrappers and copied report types. +Notarius retains stage correlation, artifact provenance, storage, and disclosure +policy; CommandKit is the planned owner of standardized operator presentation. + +Questions: + +- Which reports and accounting mechanisms can use upstream results directly? +- Which application projections remain necessary for durable contracts? +- Can the PromptKit migration avoid building temporary machinery that the + CommandKit adoption would immediately replace? + +**Discussion / decision:** Pending. + +**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit +follow-up so report and adapter decisions are made once. + +## 7. Compare Audita's PromptKit Workflow + +Review the sibling Audita checkout at `../audita` relative to the Notarius +repository root. Audita uses PromptKit for a workflow the user identifies as +very similar, potentially identical. Verify the similarities and differences +before drawing architectural conclusions. + +Questions: + +- What are Audita's PromptKit version, execution flow, and ownership boundaries? +- How does it handle candidate assessment, correction, retries, scheduling, + configuration, inspection, and diagnostics? +- Which simpler patterns could Notarius adopt directly? +- Which differences arise from actual product requirements? +- Does duplication across the two applications reveal work that belongs in + PromptKit, ConfigKit, or CommandKit? + +**Discussion / decision:** Pending; Audita has not yet been reviewed for this +comparison. + +**Follow-up:** Record the reviewed Audita revision and evidence, then feed the +findings back into items 1–6 and the adoption roadmap. diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 6dc88a4f..6bf21311 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -1,5 +1,9 @@ # PromptKit v0.23.2 Adoption +Open questions about the retained application machinery are tracked in the +[architecture discussion notepad](promptkit-architecture-discussion.md). +Those proposals remain pending until reconciled into this roadmap. + ## Purpose Replace Notarius's PromptKit v0.12 integration with a native v0.23.2