Files
notarius/docs/roadmap/promptkit-architecture-discussion.md

7.0 KiB
Raw Blame History

PromptKit Architecture Discussion

Purpose And Status

Working notepad for reviewing the ownership boundaries in the PromptKit v0.23.2 adoption roadmap. Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. The six initial topics are proposals for discussion, not accepted changes. The Audita comparison is a seventh topic and may inform earlier decisions.

The central question is how much reusable LLM machinery Notarius should retain as PromptKit, ConfigKit, and CommandKit take on more responsibility. A working principle to evaluate is that application ownership of a policy does not necessarily require application ownership of its implementation.

No architectural changes have been agreed or implemented through this notepad. The adoption roadmap remains the migration plan; reconcile it with decisions from this discussion as they are made. Significant changes to accepted architecture need successor ADRs under the documentation policy. Current boundaries are described in Architecture and LLM Runtime Internals.

1. LLM API And Diagnostic Boundary

Consider using native PromptKit contracts in LLM-facing implementation packages and reducing parallel completion, message, usage, execution-summary, and debug types. Retain application contracts where they serve typed artifact handling, stage context, durable output compatibility, or disclosure policy.

Questions:

  • Which abstractions express Notarius behavior, and which merely copy PromptKit?
  • Where should direct PromptKit dependencies be permitted?
  • Can detailed debug material flow to an application-controlled writer without being translated through generic pipeline contracts?
  • Which persisted representations need compatibility or redaction adapters?

Discussion / decision: Pending. Start here.

Follow-up: Identify the smallest useful application boundary and any architecture-policy changes it requires.

2. Prompt Assets And Schema Assembly

Consider replacing custom asset registration and virtual filesystem remounting with native embedded sources, imports, collections, and source-relative assets. Modules would continue to own prompt meaning and typed interpretation.

Questions:

  • Can native sources represent the existing shared fragments and module assets?
  • Which registry, remounting, and hashing helpers could disappear?
  • What private schema identity is still needed for checkpoint reuse?
  • How do we preserve message ordering and cache controls while simplifying assembly?

Discussion / decision: Pending.

Follow-up: Separate private LLM schema plumbing from durable artifact schemas; do not assume a captured prompt hash covers schema content.

3. Retry And Semantic Correction Ownership

Consider narrowing outer stage retries once PromptKit has exhausted its own transport and structural-repair budgets. Notarius would retain application assessment, semantic correction, fallback, and acceptance decisions.

Questions:

  • Which operational failures justify repeating an entire stage?
  • Should semantic correction preserve only the latest response or accumulate conversation history?
  • How should retries for non-LLM operations differ from LLM execution retries?
  • Would a native orchestration API reduce machinery without obscuring typed validation and terminal disposition?

Discussion / decision: Pending.

Follow-up: Evaluate behavior changes against ADR-0014 and ADR-0016. PromptKit v0.23.2's Coordinate continues an exchange; it is not a direct replacement for the existing latest-response-only policy or full stage loop.

4. Global Scheduling And Admission

Consider whether backend capacity limits and pipeline worker bounds meet the operational need, or whether a separate cross-backend ceiling remains useful. If it remains useful, consider implementing that reusable mechanism upstream.

Questions:

  • What resource or operational budget does each limit protect?
  • Should a global permit remain occupied during backend queueing and repair?
  • Can the Notarius scheduler be removed, or should PromptKit gain engine-wide admission support?

Discussion / decision: Pending.

Follow-up: Account for the existing global-limit contract. PromptKit v0.23.2's per-backend limits do not directly replace it.

5. Execution Identity And Checkpoint Reuse

Consider moving identity for PromptKit-owned execution semantics upstream. Notarius would combine opaque upstream identities with its source, module, reference, schema, and policy identities and retain checkpoint eligibility and storage decisions.

Questions:

  • Which identity can be computed before a stage runs, and which needs a concrete prepared request?
  • Which changes should invalidate reuse, and at what granularity?
  • Should a conservative application fingerprint be an interim implementation rather than the architectural destination?
  • Does this justify upstream work and a later adoption target?

Discussion / decision: Pending.

Follow-up: Review PromptKit's deferred semantic execution-target fingerprint proposal. This capability is not supplied by the v0.23.2 adoption target.

6. Inspection And Execution Bookkeeping

Extend the roadmap's use of native inspection, Run, OnPrepared, observations, and ExecutionGroup to reconsider surrounding wrappers and copied report types. Notarius retains stage correlation, artifact provenance, storage, and disclosure policy; CommandKit is the planned owner of standardized operator presentation.

Questions:

  • Which reports and accounting mechanisms can use upstream results directly?
  • Which application projections remain necessary for durable contracts?
  • Can the PromptKit migration avoid building temporary machinery that the CommandKit adoption would immediately replace?

Discussion / decision: Pending.

Follow-up: Coordinate this topic with item 1 and the roadmap's CommandKit follow-up so report and adapter decisions are made once.

7. Compare Audita's PromptKit Workflow

Review the sibling Audita checkout at ../audita relative to the Notarius repository root. Audita uses PromptKit for a workflow the user identifies as very similar, potentially identical. Verify the similarities and differences before drawing architectural conclusions.

Questions:

  • What are Audita's PromptKit version, execution flow, and ownership boundaries?
  • How does it handle candidate assessment, correction, retries, scheduling, configuration, inspection, and diagnostics?
  • Which simpler patterns could Notarius adopt directly?
  • Which differences arise from actual product requirements?
  • Does duplication across the two applications reveal work that belongs in PromptKit, ConfigKit, or CommandKit?

Discussion / decision: Pending; Audita has not yet been reviewed for this comparison.

Follow-up: Record the reviewed Audita revision and evidence, then feed the findings back into items 1–6 and the adoption roadmap.