Add PromptKit architecture discussion notepad
This commit is contained in:
165
docs/roadmap/promptkit-architecture-discussion.md
Normal file
165
docs/roadmap/promptkit-architecture-discussion.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# PromptKit Architecture Discussion
|
||||
|
||||
## Purpose And Status
|
||||
|
||||
Working notepad for reviewing the ownership boundaries in the
|
||||
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
||||
below in order, recording questions, evidence, decisions, and follow-up work.
|
||||
The six initial topics are proposals for discussion, not accepted changes.
|
||||
The Audita comparison is a seventh topic and may inform earlier decisions.
|
||||
|
||||
The central question is how much reusable LLM machinery Notarius should retain
|
||||
as PromptKit, ConfigKit, and CommandKit take on more responsibility. A working
|
||||
principle to evaluate is that application ownership of a policy does not
|
||||
necessarily require application ownership of its implementation.
|
||||
|
||||
No architectural changes have been agreed or implemented through this notepad.
|
||||
The adoption roadmap remains the migration plan; reconcile it with decisions
|
||||
from this discussion as they are made. Significant changes to accepted
|
||||
architecture need successor ADRs under the
|
||||
[documentation policy](../policy/documentation.md#architecture-decision-records).
|
||||
Current boundaries are described in [Architecture](../policy/architecture.md)
|
||||
and [LLM Runtime Internals](../internal/llm.md).
|
||||
|
||||
## 1. LLM API And Diagnostic Boundary
|
||||
|
||||
Consider using native PromptKit contracts in LLM-facing implementation packages
|
||||
and reducing parallel completion, message, usage, execution-summary, and debug
|
||||
types. Retain application contracts where they serve typed artifact handling,
|
||||
stage context, durable output compatibility, or disclosure policy.
|
||||
|
||||
Questions:
|
||||
|
||||
- Which abstractions express Notarius behavior, and which merely copy PromptKit?
|
||||
- Where should direct PromptKit dependencies be permitted?
|
||||
- Can detailed debug material flow to an application-controlled writer without
|
||||
being translated through generic pipeline contracts?
|
||||
- Which persisted representations need compatibility or redaction adapters?
|
||||
|
||||
**Discussion / decision:** Pending. Start here.
|
||||
|
||||
**Follow-up:** Identify the smallest useful application boundary and any
|
||||
architecture-policy changes it requires.
|
||||
|
||||
## 2. Prompt Assets And Schema Assembly
|
||||
|
||||
Consider replacing custom asset registration and virtual filesystem remounting
|
||||
with native embedded sources, imports, collections, and source-relative assets.
|
||||
Modules would continue to own prompt meaning and typed interpretation.
|
||||
|
||||
Questions:
|
||||
|
||||
- Can native sources represent the existing shared fragments and module assets?
|
||||
- Which registry, remounting, and hashing helpers could disappear?
|
||||
- What private schema identity is still needed for checkpoint reuse?
|
||||
- How do we preserve message ordering and cache controls while simplifying
|
||||
assembly?
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
|
||||
**Follow-up:** Separate private LLM schema plumbing from durable artifact
|
||||
schemas; do not assume a captured prompt hash covers schema content.
|
||||
|
||||
## 3. Retry And Semantic Correction Ownership
|
||||
|
||||
Consider narrowing outer stage retries once PromptKit has exhausted its own
|
||||
transport and structural-repair budgets. Notarius would retain application
|
||||
assessment, semantic correction, fallback, and acceptance decisions.
|
||||
|
||||
Questions:
|
||||
|
||||
- Which operational failures justify repeating an entire stage?
|
||||
- Should semantic correction preserve only the latest response or accumulate
|
||||
conversation history?
|
||||
- How should retries for non-LLM operations differ from LLM execution retries?
|
||||
- Would a native orchestration API reduce machinery without obscuring typed
|
||||
validation and terminal disposition?
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
|
||||
**Follow-up:** Evaluate behavior changes against
|
||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
|
||||
replacement for the existing latest-response-only policy or full stage loop.
|
||||
|
||||
## 4. Global Scheduling And Admission
|
||||
|
||||
Consider whether backend capacity limits and pipeline worker bounds meet the
|
||||
operational need, or whether a separate cross-backend ceiling remains useful.
|
||||
If it remains useful, consider implementing that reusable mechanism upstream.
|
||||
|
||||
Questions:
|
||||
|
||||
- What resource or operational budget does each limit protect?
|
||||
- Should a global permit remain occupied during backend queueing and repair?
|
||||
- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide
|
||||
admission support?
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
|
||||
**Follow-up:** Account for the existing global-limit contract. PromptKit
|
||||
v0.23.2's per-backend limits do not directly replace it.
|
||||
|
||||
## 5. Execution Identity And Checkpoint Reuse
|
||||
|
||||
Consider moving identity for PromptKit-owned execution semantics upstream.
|
||||
Notarius would combine opaque upstream identities with its source, module,
|
||||
reference, schema, and policy identities and retain checkpoint eligibility and
|
||||
storage decisions.
|
||||
|
||||
Questions:
|
||||
|
||||
- Which identity can be computed before a stage runs, and which needs a concrete
|
||||
prepared request?
|
||||
- Which changes should invalidate reuse, and at what granularity?
|
||||
- Should a conservative application fingerprint be an interim implementation
|
||||
rather than the architectural destination?
|
||||
- Does this justify upstream work and a later adoption target?
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
|
||||
**Follow-up:** Review PromptKit's deferred semantic execution-target fingerprint
|
||||
proposal. This capability is not supplied by the v0.23.2 adoption target.
|
||||
|
||||
## 6. Inspection And Execution Bookkeeping
|
||||
|
||||
Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations,
|
||||
and `ExecutionGroup` to reconsider surrounding wrappers and copied report types.
|
||||
Notarius retains stage correlation, artifact provenance, storage, and disclosure
|
||||
policy; CommandKit is the planned owner of standardized operator presentation.
|
||||
|
||||
Questions:
|
||||
|
||||
- Which reports and accounting mechanisms can use upstream results directly?
|
||||
- Which application projections remain necessary for durable contracts?
|
||||
- Can the PromptKit migration avoid building temporary machinery that the
|
||||
CommandKit adoption would immediately replace?
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
|
||||
**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit
|
||||
follow-up so report and adapter decisions are made once.
|
||||
|
||||
## 7. Compare Audita's PromptKit Workflow
|
||||
|
||||
Review the sibling Audita checkout at `../audita` relative to the Notarius
|
||||
repository root. Audita uses PromptKit for a workflow the user identifies as
|
||||
very similar, potentially identical. Verify the similarities and differences
|
||||
before drawing architectural conclusions.
|
||||
|
||||
Questions:
|
||||
|
||||
- What are Audita's PromptKit version, execution flow, and ownership boundaries?
|
||||
- How does it handle candidate assessment, correction, retries, scheduling,
|
||||
configuration, inspection, and diagnostics?
|
||||
- Which simpler patterns could Notarius adopt directly?
|
||||
- Which differences arise from actual product requirements?
|
||||
- Does duplication across the two applications reveal work that belongs in
|
||||
PromptKit, ConfigKit, or CommandKit?
|
||||
|
||||
**Discussion / decision:** Pending; Audita has not yet been reviewed for this
|
||||
comparison.
|
||||
|
||||
**Follow-up:** Record the reviewed Audita revision and evidence, then feed the
|
||||
findings back into items 1–6 and the adoption roadmap.
|
||||
@@ -1,5 +1,9 @@
|
||||
# PromptKit v0.23.2 Adoption
|
||||
|
||||
Open questions about the retained application machinery are tracked in the
|
||||
[architecture discussion notepad](promptkit-architecture-discussion.md).
|
||||
Those proposals remain pending until reconciled into this roadmap.
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace Notarius's PromptKit v0.12 integration with a native v0.23.2
|
||||
|
||||
Reference in New Issue
Block a user