Files
notarius/docs/roadmap/promptkit-architecture-discussion.md

166 lines
7.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PromptKit Architecture Discussion
## Purpose And Status
Working notepad for reviewing the ownership boundaries in the
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
below in order, recording questions, evidence, decisions, and follow-up work.
The six initial topics are proposals for discussion, not accepted changes.
The Audita comparison is a seventh topic and may inform earlier decisions.
The central question is how much reusable LLM machinery Notarius should retain
as PromptKit, ConfigKit, and CommandKit take on more responsibility. A working
principle to evaluate is that application ownership of a policy does not
necessarily require application ownership of its implementation.
No architectural changes have been agreed or implemented through this notepad.
The adoption roadmap remains the migration plan; reconcile it with decisions
from this discussion as they are made. Significant changes to accepted
architecture need successor ADRs under the
[documentation policy](../policy/documentation.md#architecture-decision-records).
Current boundaries are described in [Architecture](../policy/architecture.md)
and [LLM Runtime Internals](../internal/llm.md).
## 1. LLM API And Diagnostic Boundary
Consider using native PromptKit contracts in LLM-facing implementation packages
and reducing parallel completion, message, usage, execution-summary, and debug
types. Retain application contracts where they serve typed artifact handling,
stage context, durable output compatibility, or disclosure policy.
Questions:
- Which abstractions express Notarius behavior, and which merely copy PromptKit?
- Where should direct PromptKit dependencies be permitted?
- Can detailed debug material flow to an application-controlled writer without
being translated through generic pipeline contracts?
- Which persisted representations need compatibility or redaction adapters?
**Discussion / decision:** Pending. Start here.
**Follow-up:** Identify the smallest useful application boundary and any
architecture-policy changes it requires.
## 2. Prompt Assets And Schema Assembly
Consider replacing custom asset registration and virtual filesystem remounting
with native embedded sources, imports, collections, and source-relative assets.
Modules would continue to own prompt meaning and typed interpretation.
Questions:
- Can native sources represent the existing shared fragments and module assets?
- Which registry, remounting, and hashing helpers could disappear?
- What private schema identity is still needed for checkpoint reuse?
- How do we preserve message ordering and cache controls while simplifying
assembly?
**Discussion / decision:** Pending.
**Follow-up:** Separate private LLM schema plumbing from durable artifact
schemas; do not assume a captured prompt hash covers schema content.
## 3. Retry And Semantic Correction Ownership
Consider narrowing outer stage retries once PromptKit has exhausted its own
transport and structural-repair budgets. Notarius would retain application
assessment, semantic correction, fallback, and acceptance decisions.
Questions:
- Which operational failures justify repeating an entire stage?
- Should semantic correction preserve only the latest response or accumulate
conversation history?
- How should retries for non-LLM operations differ from LLM execution retries?
- Would a native orchestration API reduce machinery without obscuring typed
validation and terminal disposition?
**Discussion / decision:** Pending.
**Follow-up:** Evaluate behavior changes against
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
replacement for the existing latest-response-only policy or full stage loop.
## 4. Global Scheduling And Admission
Consider whether backend capacity limits and pipeline worker bounds meet the
operational need, or whether a separate cross-backend ceiling remains useful.
If it remains useful, consider implementing that reusable mechanism upstream.
Questions:
- What resource or operational budget does each limit protect?
- Should a global permit remain occupied during backend queueing and repair?
- Can the Notarius scheduler be removed, or should PromptKit gain engine-wide
admission support?
**Discussion / decision:** Pending.
**Follow-up:** Account for the existing global-limit contract. PromptKit
v0.23.2's per-backend limits do not directly replace it.
## 5. Execution Identity And Checkpoint Reuse
Consider moving identity for PromptKit-owned execution semantics upstream.
Notarius would combine opaque upstream identities with its source, module,
reference, schema, and policy identities and retain checkpoint eligibility and
storage decisions.
Questions:
- Which identity can be computed before a stage runs, and which needs a concrete
prepared request?
- Which changes should invalidate reuse, and at what granularity?
- Should a conservative application fingerprint be an interim implementation
rather than the architectural destination?
- Does this justify upstream work and a later adoption target?
**Discussion / decision:** Pending.
**Follow-up:** Review PromptKit's deferred semantic execution-target fingerprint
proposal. This capability is not supplied by the v0.23.2 adoption target.
## 6. Inspection And Execution Bookkeeping
Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations,
and `ExecutionGroup` to reconsider surrounding wrappers and copied report types.
Notarius retains stage correlation, artifact provenance, storage, and disclosure
policy; CommandKit is the planned owner of standardized operator presentation.
Questions:
- Which reports and accounting mechanisms can use upstream results directly?
- Which application projections remain necessary for durable contracts?
- Can the PromptKit migration avoid building temporary machinery that the
CommandKit adoption would immediately replace?
**Discussion / decision:** Pending.
**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit
follow-up so report and adapter decisions are made once.
## 7. Compare Audita's PromptKit Workflow
Review the sibling Audita checkout at `../audita` relative to the Notarius
repository root. Audita uses PromptKit for a workflow the user identifies as
very similar, potentially identical. Verify the similarities and differences
before drawing architectural conclusions.
Questions:
- What are Audita's PromptKit version, execution flow, and ownership boundaries?
- How does it handle candidate assessment, correction, retries, scheduling,
configuration, inspection, and diagnostics?
- Which simpler patterns could Notarius adopt directly?
- Which differences arise from actual product requirements?
- Does duplication across the two applications reveal work that belongs in
PromptKit, ConfigKit, or CommandKit?
**Discussion / decision:** Pending; Audita has not yet been reviewed for this
comparison.
**Follow-up:** Record the reviewed Audita revision and evidence, then feed the
findings back into items 1–6 and the adoption roadmap.