Files
notarius/docs/roadmap/promptkit.md

14 KiB

PromptKit v0.23.2 Adoption

Purpose

Replace Notarius's PromptKit v0.12 integration with a native v0.23.2 integration. Delegate configuration composition, resource inspection, request preparation, and model execution to PromptKit and ConfigKit. Keep application machinery where it implements a distinct Notarius decision, such as typed artifact handling, stage acceptance, cross-backend scheduling, or checkpoint reuse.

This is a hard cutover for both the Notarius application configuration and PromptKit resource definitions. Provide migration guidance and maintained examples; support only the new formats in the resulting application. This draft describes the target, assuming the recommendations under Open Decisions. It does not select a release date or implementation sequence.

Reviewed Baseline

Notarius at 65b8e8f0a2d60041c18a29a3c33afc5c689468df uses application configuration version 5, Go 1.27, new-path ConfigKit v1.2.0, and old-path PromptKit v0.12.0. The latter brings in a separate old-path ConfigKit v0.5.0 and two external provider catalogs. Notarius and PromptKit currently share the YAML v4 release candidate.

The target is go.maximumdirect.net/promptkit@v0.23.2, published at 382ccfb14b9630fd8f48c8364dbe2b6159637d71. The tagged release notes, migration guide, format reference, and Go consumer guide are the primary upstream references.

The main baseline seams are:

  • internal/core/config resolves promptkit.profile_file and profile_dir and interprets the optional local_backend object. promptkit_profiles.go builds the operator layers for execution and inspection; asset_registry.go supplies the embedded layer.
  • assets/ and examples/llm-profiles/ use keyed resource definitions. Completion requests and diagnostics carry a PromptKit prompt version.
  • internal/framework/llm does split preparation and execution to retain debug material; it projects a backend-completed profile report and walks effective YAML to discover versioned prompts for checkpoint identity.
  • The pipeline's producer-attempt loop coordinates domain and stage policy. The scheduled LLM client enforces the application-wide limit independently of any PromptKit backend policy.

Target End State

Dependency And Application Configuration Cutover

Use the vanity PromptKit import path and shared ConfigKit v1.2.1, stable go-yaml v4, and the resulting compatible validation dependencies. Remove the old-path imports and catalog dependencies once unused. Use tagged modules without a compatibility fork or permanent replacement. The consumer minimum remains Go 1.27. Update attribution for the selected and removed components under the OSS policy.

Introduce application configuration version 6 as the only supported application version. Version 6 retains unrelated pipeline structure, discovery, and resolution behavior. ConfigKit continues to own application loading and composition under ADR-0020. Retire profile_file, profile_dir, and local_backend in favor of native PromptKit companion files. There is no separate promptkit.config_file selector. Preserve the ability to override transport retries through the PromptKit engine option; leave the upstream default in effect when the operator does not override it.

Companion Files And Embedded Defaults

Adopt Audita's companion-file convention. Use the directory of the main Notarius configuration actually selected by ConfigKit, whether explicit or discovered. Load the optional prompts.yml and recursive .yml and .yaml files under prompts/ beside it as one higher-priority operator underlay using native ConfigKit imports. Do not add a working-directory fallback for companion discovery. Configurations in the same directory share those companions. Missing companions are fine; loading errors follow upstream semantics rather than silently falling back to embedded assets. Native imports and source-relative resource paths govern any files reached from the companions. The two companion locations are strict peers; filenames do not set precedence between conflicting operator records.

Use one source assembly for runtime, preflight, and inspection: PromptKit's bundled OpenRouter underlay, Notarius's embedded underlay, then the operator underlay when present. Embed all Notarius default prompts and profiles under collections.notarius in the application underlay. Operator companion files can replace individual defaults by defining complete records with the same local IDs in that same collection. They can also define new resources and local backends using the native format.

The bundled OpenRouter inventory differs from the former external catalogs and supplies no backend-local concurrency limit. It does not implicitly select a model for any Notarius request. Migrate deployments that use removed profiles or backends to intentionally selected native records. The application-wide LLM ceiling remains independent of backend-local settings.

Native Prompt And Profile Resources

Convert maintained prompts, profiles, examples, and fixtures to identified lists, with config: for source controls and native sibling inheritance (extends: {id: parent}). Under native default composition, a later same-ID record replaces the earlier record in full. Explain complete replacements, distinct inherited variants, and the new backend fallback semantics for profile nulls. The optional PromptKit source specification header is version 2; it is not the Notarius application version. Do not translate legacy resource files, synthesize aliases, or reconstruct old recursive same-ID overlays.

Select prompts by ID alone and remove the obsolete PromptKit version selector and metadata from Notarius's request, debug, and reconciliation identity plumbing. Review affected diagnostic fields and document deliberate changes. Keep the current local resource IDs and use their qualified forms in lookups and references, for example notarius/dnd.spells and notarius/dnd-extraction. Update prompt profile references, application profile selections, and inspection examples to use the qualified IDs. The dnd-extraction profile retains its current model and settings and continues to reference the unqualified bundled openrouter backend. A profile that extends it belongs in the same collection and uses its local sibling ID in extends. Retain message roles, bytes, order, cache boundaries, and intended output/repair behavior while changing the declaration format. Keep module-owned asset selection and embedded filesystem contexts intact. Verify maintained private response schemas work under upstream's self-contained-schema rule; this is not a redesign of durable D&D artifacts.

Inspection And Reuse Identity

Use PromptKit's captured configuration reports for effective configuration and provenance. Use its resource inventory and inspection APIs for exact identities, presence-aware resource fields, and captured prompt hashes, rather than deriving an inventory from rendered configuration YAML. Change the config profile report to show the resolved profile and its selected backend, when present, as separate resources. Use upstream field semantics and retain the existing operator disclosure boundary, including redaction of credential-shaped provider parameters. Do not recreate backend default resolution to produce the former effective_model_params view.

Distinguish configuration and resource preflight from request readiness. Use InspectRequirements to explain recognized dependencies when helpful, not as proof that a request is executable. Use Prepare or ValidateRequest when concrete request inputs are available and their validation is actually needed. A routine config-only check should remain credential-free and offline, without synthetic requests.

Retain a private, versioned Notarius checkpoint identity based on the engine's captured state. Use native resource enumeration and prompt hashes, keeping the effective execution inputs needed to distinguish reuse safely. A diff of effective values alone cannot detect edits to a referenced message file. Version the fingerprint for the cutover so obsolete identity inputs cannot cause a false hit; do not reread mutable sources or expose the digest inputs. Keep run manifests based on the profiles actually used, not the inventory.

Native Execution And Retained Pipeline Policy

Use ordinary Engine.Run for structured completion and use OnPrepared where opt-in debug capture needs the actual prepared request. The callback runs after admission, before generation, and is not delivered for failed preparation or admission. Keep the callback lightweight and use native observations for safe terminal execution facts, including attempts, repairs, and available token usage. Where a diagnostic needs totals over selected executions, use ExecutionGroup rather than inventing a second accumulator. Use OperationTimeout where a completion calls for that scope of budget; do not add a new default deadline or replace application deadlines that also cover work outside the PromptKit call.

PromptKit owns provider admission, transport retries, and structural repair. Keep the Notarius cross-backend scheduler because it serves a different budget from backend or pipeline worker limits. Keep the adapter's typed decode, actual-run provenance, and provider-neutral error and diagnostic mapping. PromptKit execution types stay within that integration boundary. In particular, preserve the distinction between invalid structured output, provider failure, capacity rejection, and cancellation. A configured repair override still selects a complete upstream output contract; it must not accidentally drop the selected schema or validation mode.

Notarius owns stage attempts, validation and fallback decisions, and publication of accepted artifacts. Keep its latest-response-only correction policy using native appended messages, as defined by ADR-0014 and ADR-0016. Exchange and Coordinate are for workflows that benefit from retained conversation history. Adopting them for stage retries would change what the model sees and which work consumes an attempt. They are not needed to execute the existing retry policy.

Migration And Verification

Update the maintained minimal and complete D&D examples to distinguish embedded-only operation from companion configuration. Use separate example directories if necessary to demonstrate those deployments independently. Document how to move existing PromptKit sources into the companion layout or import them from it, declare a local backend, and replace or extend an embedded resource. Explain qualified IDs, retired catalog IDs, the changed profile report, and any new checkpoint cold misses. Update the canonical configuration, CLI, operations, and integration guides when behavior lands. Record the changed decisions about source precedence and inspection in a successor to ADR-0019 rather than rewriting its accepted decision text. Release preparation is separate.

Verify the dependency graph and offline examples after the cutover. Use focused behavior tests for the new format and intentional rejection of the old one, main-relative companion discovery, and native relative imports. Verify that a companion can replace a qualified embedded prompt or profile without changing unrelated defaults, and that preflight, inspection, and runtime use the same effective definitions. Exercise preparation, generation, structural repair, and semantic retries through the maintained fake-provider integration. Check that real message or target changes affect checkpoint reuse, and that session routing, cache controls, diagnostics, and durable outputs keep their existing meaning. Live provider checks are useful for deployment confidence, not a requirement of the offline suite.

Open Decisions

How Far Offline Preflight Goes

Recommendation: Check that the captured configuration is valid and that selected resource IDs resolve. Make clear that a valid profile can be partial and that concrete request readiness is checked during preparation. This preserves a useful offline preflight without duplicating target resolution or requiring credentials or source data.

Alternative: Preserve offline checking of completed targets by pursuing a dedicated upstream API that resolves targets without artifacts or credentials. That would catch more missing endpoint/model settings before a run and could serve other consumers, but it would add an upstream dependency to this work or require a subsequent release. Reimplementing completion rules in Notarius would risk a discrepancy with actual execution.

Operator Diagnostics In This Upgrade

Recommendation: Migrate existing inspection commands and use native inventory and observation APIs where they replace local work. Evaluate new commands for provider smoke checks or effective-config diffs separately, after identifying the operator workflows and output contracts they need.

Alternative: Include an explicitly invoked smoke command in the cutover. This would help operators verify credentials and structured inference after migrating a deployment, and it can delegate to Engine.SmokeTest. It adds a network-capable CLI surface, with cost and output handling to define. Keep it separate from ordinary offline validation in either case.