Add PromptKit architecture discussion notepad
This commit is contained in:
165
docs/roadmap/promptkit-architecture-discussion.md
Normal file
165
docs/roadmap/promptkit-architecture-discussion.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
# PromptKit Architecture Discussion
|
||||||
|
|
||||||
|
## Purpose And Status
|
||||||
|
|
||||||
|
Working notepad for reviewing the ownership boundaries in the
|
||||||
|
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
||||||
|
below in order, recording questions, evidence, decisions, and follow-up work.
|
||||||
|
The six initial topics are proposals for discussion, not accepted changes.
|
||||||
|
The Audita comparison is a seventh topic and may inform earlier decisions.
|
||||||
|
|
||||||
|
The central question is how much reusable LLM machinery Notarius should retain
|
||||||
|
as PromptKit, ConfigKit, and CommandKit take on more responsibility. A working
|
||||||
|
principle to evaluate is that application ownership of a policy does not
|
||||||
|
necessarily require application ownership of its implementation.
|
||||||
|
|
||||||
|
No architectural changes have been agreed or implemented through this notepad.
|
||||||
|
The adoption roadmap remains the migration plan; reconcile it with decisions
|
||||||
|
from this discussion as they are made. Significant changes to accepted
|
||||||
|
architecture need successor ADRs under the
|
||||||
|
[documentation policy](../policy/documentation.md#architecture-decision-records).
|
||||||
|
Current boundaries are described in [Architecture](../policy/architecture.md)
|
||||||
|
and [LLM Runtime Internals](../internal/llm.md).
|
||||||
|
|
||||||
|
## 1. LLM API And Diagnostic Boundary
|
||||||
|
|
||||||
|
Consider using native PromptKit contracts in LLM-facing implementation packages
|
||||||
|
and reducing parallel completion, message, usage, execution-summary, and debug
|
||||||
|
types. Retain application contracts where they serve typed artifact handling,
|
||||||
|
stage context, durable output compatibility, or disclosure policy.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- Which abstractions express Notarius behavior, and which merely copy PromptKit?
|
||||||
|
- Where should direct PromptKit dependencies be permitted?
|
||||||
|
- Can detailed debug material flow to an application-controlled writer without
|
||||||
|
being translated through generic pipeline contracts?
|
||||||
|
- Which persisted representations need compatibility or redaction adapters?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending. Start here.
|
||||||
|
|
||||||
|
**Follow-up:** Identify the smallest useful application boundary and any
|
||||||
|
architecture-policy changes it requires.
|
||||||
|
|
||||||
|
## 2. Prompt Assets And Schema Assembly
|
||||||
|
|
||||||
|
Consider replacing custom asset registration and virtual filesystem remounting
|
||||||
|
with native embedded sources, imports, collections, and source-relative assets.
|
||||||
|
Modules would continue to own prompt meaning and typed interpretation.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- Can native sources represent the existing shared fragments and module assets?
|
||||||
|
- Which registry, remounting, and hashing helpers could disappear?
|
||||||
|
- What private schema identity is still needed for checkpoint reuse?
|
||||||
|
- How do we preserve message ordering and cache controls while simplifying
|
||||||
|
assembly?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending.
|
||||||
|
|
||||||
|
**Follow-up:** Separate private LLM schema plumbing from durable artifact
|
||||||
|
schemas; do not assume a captured prompt hash covers schema content.
|
||||||
|
|
||||||
|
## 3. Retry And Semantic Correction Ownership
|
||||||
|
|
||||||
|
Consider narrowing outer stage retries once PromptKit has exhausted its own
|
||||||
|
transport and structural-repair budgets. Notarius would retain application
|
||||||
|
assessment, semantic correction, fallback, and acceptance decisions.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- Which operational failures justify repeating an entire stage?
|
||||||
|
- Should semantic correction preserve only the latest response or accumulate
|
||||||
|
conversation history?
|
||||||
|
- How should retries for non-LLM operations differ from LLM execution retries?
|
||||||
|
- Would a native orchestration API reduce machinery without obscuring typed
|
||||||
|
validation and terminal disposition?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending.
|
||||||
|
|
||||||
|
**Follow-up:** Evaluate behavior changes against
|
||||||
|
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||||
|
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||||
|
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
|
||||||
|
replacement for the existing latest-response-only policy or full stage loop.
|
||||||
|
|
||||||
|
## 4. Global Scheduling And Admission
|
||||||
|
|
||||||
|
Consider whether backend capacity limits and pipeline worker bounds meet the
|
||||||
|
operational need, or whether a separate cross-backend ceiling remains useful.
|
||||||
|
If it remains useful, consider implementing that reusable mechanism upstream.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- What resource or operational budget does each limit protect?
|
||||||
|
- Should a global permit remain occupied during backend queueing and repair?
|
||||||
|
- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide
|
||||||
|
admission support?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending.
|
||||||
|
|
||||||
|
**Follow-up:** Account for the existing global-limit contract. PromptKit
|
||||||
|
v0.23.2's per-backend limits do not directly replace it.
|
||||||
|
|
||||||
|
## 5. Execution Identity And Checkpoint Reuse
|
||||||
|
|
||||||
|
Consider moving identity for PromptKit-owned execution semantics upstream.
|
||||||
|
Notarius would combine opaque upstream identities with its source, module,
|
||||||
|
reference, schema, and policy identities and retain checkpoint eligibility and
|
||||||
|
storage decisions.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- Which identity can be computed before a stage runs, and which needs a concrete
|
||||||
|
prepared request?
|
||||||
|
- Which changes should invalidate reuse, and at what granularity?
|
||||||
|
- Should a conservative application fingerprint be an interim implementation
|
||||||
|
rather than the architectural destination?
|
||||||
|
- Does this justify upstream work and a later adoption target?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending.
|
||||||
|
|
||||||
|
**Follow-up:** Review PromptKit's deferred semantic execution-target fingerprint
|
||||||
|
proposal. This capability is not supplied by the v0.23.2 adoption target.
|
||||||
|
|
||||||
|
## 6. Inspection And Execution Bookkeeping
|
||||||
|
|
||||||
|
Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations,
|
||||||
|
and `ExecutionGroup` to reconsider surrounding wrappers and copied report types.
|
||||||
|
Notarius retains stage correlation, artifact provenance, storage, and disclosure
|
||||||
|
policy; CommandKit is the planned owner of standardized operator presentation.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- Which reports and accounting mechanisms can use upstream results directly?
|
||||||
|
- Which application projections remain necessary for durable contracts?
|
||||||
|
- Can the PromptKit migration avoid building temporary machinery that the
|
||||||
|
CommandKit adoption would immediately replace?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending.
|
||||||
|
|
||||||
|
**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit
|
||||||
|
follow-up so report and adapter decisions are made once.
|
||||||
|
|
||||||
|
## 7. Compare Audita's PromptKit Workflow
|
||||||
|
|
||||||
|
Review the sibling Audita checkout at `../audita` relative to the Notarius
|
||||||
|
repository root. Audita uses PromptKit for a workflow the user identifies as
|
||||||
|
very similar, potentially identical. Verify the similarities and differences
|
||||||
|
before drawing architectural conclusions.
|
||||||
|
|
||||||
|
Questions:
|
||||||
|
|
||||||
|
- What are Audita's PromptKit version, execution flow, and ownership boundaries?
|
||||||
|
- How does it handle candidate assessment, correction, retries, scheduling,
|
||||||
|
configuration, inspection, and diagnostics?
|
||||||
|
- Which simpler patterns could Notarius adopt directly?
|
||||||
|
- Which differences arise from actual product requirements?
|
||||||
|
- Does duplication across the two applications reveal work that belongs in
|
||||||
|
PromptKit, ConfigKit, or CommandKit?
|
||||||
|
|
||||||
|
**Discussion / decision:** Pending; Audita has not yet been reviewed for this
|
||||||
|
comparison.
|
||||||
|
|
||||||
|
**Follow-up:** Record the reviewed Audita revision and evidence, then feed the
|
||||||
|
findings back into items 1–6 and the adoption roadmap.
|
||||||
@@ -1,5 +1,9 @@
|
|||||||
# PromptKit v0.23.2 Adoption
|
# PromptKit v0.23.2 Adoption
|
||||||
|
|
||||||
|
Open questions about the retained application machinery are tracked in the
|
||||||
|
[architecture discussion notepad](promptkit-architecture-discussion.md).
|
||||||
|
Those proposals remain pending until reconciled into this roadmap.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Replace Notarius's PromptKit v0.12 integration with a native v0.23.2
|
Replace Notarius's PromptKit v0.12 integration with a native v0.23.2
|
||||||
|
|||||||
Reference in New Issue
Block a user