Files
notarius/docs/roadmap/promptkit.md

234 lines
13 KiB
Markdown

# PromptKit v0.23.2 Adoption
## Purpose
Replace Notarius's PromptKit v0.12 integration with a native v0.23.2
integration. Delegate configuration composition, resource inspection, request
preparation, and model execution to PromptKit and ConfigKit. Keep application
machinery where it implements a distinct Notarius decision, such as typed
artifact handling, stage acceptance, cross-backend scheduling, or checkpoint
reuse.
This is a hard cutover for both the Notarius application configuration and
PromptKit resource definitions. Provide migration guidance and maintained
examples; support only the new formats in the resulting application. This
draft describes the target, assuming the recommendations under
[Open Decisions](#open-decisions). It does not select a release date or
implementation sequence.
## Reviewed Baseline
Notarius at `65b8e8f0a2d60041c18a29a3c33afc5c689468df` uses application
configuration version 5, Go 1.27, new-path ConfigKit v1.2.0, and old-path
PromptKit v0.12.0. The latter brings in a separate old-path ConfigKit v0.5.0
and two external provider catalogs. Notarius and PromptKit currently share
the YAML v4 release candidate.
The target is `go.maximumdirect.net/promptkit@v0.23.2`, published at
`382ccfb14b9630fd8f48c8364dbe2b6159637d71`. The tagged [release
notes][release], [migration guide][migration], [format reference][formats],
and [Go consumer guide][consumer] are the primary upstream references.
The main baseline seams are:
- `internal/core/config` resolves `promptkit.profile_file` and
`profile_dir` and interprets the optional `local_backend` object.
`promptkit_profiles.go` builds the operator layers for execution and
inspection; `asset_registry.go` supplies the embedded layer.
- `assets/` and `examples/llm-profiles/` use keyed resource definitions.
Completion requests and diagnostics carry a PromptKit prompt version.
- `internal/framework/llm` does split preparation and execution to retain
debug material; it projects a backend-completed profile report and walks
effective YAML to discover versioned prompts for checkpoint identity.
- The pipeline's producer-attempt loop coordinates domain and stage policy.
The scheduled LLM client enforces the application-wide limit independently
of any PromptKit backend policy.
## Target End State
### Dependency And Application Configuration Cutover
Use the vanity PromptKit import path and shared ConfigKit v1.2.1, stable
go-yaml v4, and the resulting compatible validation dependencies. Remove the
old-path imports and catalog dependencies once unused. Use tagged modules
without a compatibility fork or permanent replacement. The consumer minimum
remains Go 1.27. Update attribution for the selected and removed components
under the [OSS policy](../policy/oss.md).
Introduce application configuration version 6 as the only supported application
version. Version 6 retains unrelated pipeline structure, discovery, and
resolution behavior. ConfigKit continues to own application loading and
composition under [ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
The proposed operator seam is an optional `promptkit.config_file` pointing to a
native PromptKit main source. Resolve this selector relative to the Notarius
file that declares it; leave imports and resource paths *inside* the selected
source to the upstream loader. Retire `profile_file`, `profile_dir`, and
`local_backend`: express related sources through native imports and local
endpoints through native backend records. Preserve the ability to override
transport retries through the PromptKit engine option; leave the upstream
default in effect when the operator does not override it.
Pass the selected main source and application asset underlay through one source
assembly used by runtime, preflight, and inspection. Use the bundled OpenRouter
underlay as supplied by PromptKit. That bundle is a different inventory from
the former external catalogs and supplies no backend-local concurrency limit.
It does not implicitly select a model for any Notarius request. Migrate
deployments that use removed profiles or backends to intentionally selected
native records. The application-wide LLM ceiling remains independent of
backend-local settings.
### Native Prompt And Profile Resources
Convert maintained prompts, profiles, examples, and fixtures to identified
lists, with `config:` for source controls and native sibling inheritance
(`extends: {id: parent}`). Under native default composition, a later same-ID
record replaces the earlier record in full. Explain complete replacements,
distinct inherited variants, and the new backend fallback semantics for
profile nulls.
The optional PromptKit source specification header is version 2; it is not the
Notarius application version. Keep source-local message resolution and use
native collections when deployments need namespaces. Do not translate
legacy resource files, synthesize aliases, or reconstruct old recursive
same-ID overlays.
Select prompts by ID alone and remove the obsolete PromptKit version selector
and metadata from Notarius's request, debug, and reconciliation identity
plumbing. Review affected diagnostic fields and document deliberate changes.
Retain logical workload IDs, including `dnd-extraction` and its current
explicit model/settings. Retain message roles, bytes, order, cache boundaries,
and intended output/repair behavior while changing the declaration format.
Keep module-owned asset selection and embedded filesystem contexts intact.
Verify maintained private response schemas work under upstream's
self-contained-schema rule; this is not a redesign of durable D&D artifacts.
### Inspection And Reuse Identity
Use PromptKit's captured configuration reports for effective configuration
and provenance. Use its resource inventory and inspection APIs for exact
identities, presence-aware resource fields, and captured prompt hashes, rather
than deriving an inventory from rendered configuration YAML. Change the
`config profile` report to show the resolved profile and its selected backend,
when present, as separate resources. Use upstream field semantics and retain
the existing operator disclosure boundary, including redaction of
credential-shaped provider parameters. Do not recreate backend default
resolution to produce the former `effective_model_params` view.
Distinguish configuration and resource preflight from request readiness.
Use `InspectRequirements` to explain recognized dependencies when helpful,
not as proof that a request is executable. Use `Prepare` or
`ValidateRequest` when concrete request inputs are available and their
validation is actually needed. A routine config-only check should remain
credential-free and offline, without synthetic requests.
Retain a private, versioned Notarius checkpoint identity based on the engine's
captured state. Use native resource enumeration and prompt hashes, keeping
the effective execution inputs needed to distinguish reuse safely. A diff of
effective values alone cannot detect edits to a referenced message file.
Version the fingerprint for the cutover so obsolete identity inputs cannot
cause a false hit; do not reread mutable sources or expose the digest inputs.
Keep run manifests based on the profiles actually used, not the inventory.
### Native Execution And Retained Pipeline Policy
Use ordinary `Engine.Run` for structured completion and use `OnPrepared`
where opt-in debug capture needs the actual prepared request. The callback
runs after admission, before generation, and is not delivered for failed
preparation or admission. Keep the callback lightweight and use native
observations for safe terminal execution facts, including attempts, repairs,
and available token usage. Where a diagnostic needs totals over selected
executions, use `ExecutionGroup` rather than inventing a second accumulator.
Use `OperationTimeout` where a completion calls for that scope of budget;
do not add a new default deadline or replace application deadlines that also
cover work outside the PromptKit call.
PromptKit owns provider admission, transport retries, and structural repair.
Keep the Notarius cross-backend scheduler because it serves a different
budget from backend or pipeline worker limits. Keep the adapter's typed
decode, actual-run provenance, and provider-neutral error and diagnostic
mapping. PromptKit execution types stay within that integration boundary.
In particular, preserve the distinction between invalid structured
output, provider failure, capacity rejection, and cancellation. A configured
repair override still selects a complete upstream output contract; it must not
accidentally drop the selected schema or validation mode.
Notarius owns stage attempts, validation and fallback decisions, and
publication of accepted artifacts. Keep its latest-response-only correction
policy using native appended messages, as defined by
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
`Exchange` and `Coordinate` are for workflows that benefit from retained
conversation history. Adopting them for stage retries would change what the
model sees and which work consumes an attempt. They are not needed to execute
the existing retry policy.
## Migration And Verification
Update the maintained minimal and complete D&D examples, including operator
profile inheritance, and document how to convert a directory deployment,
declare a local backend, and replace an embedded prompt or profile. Explain
retired catalog IDs, the changed profile report, and any new checkpoint cold
misses. Update the canonical configuration, CLI, operations, and integration
guides when behavior lands. Record the changed decisions about source
precedence and inspection in a successor to
[ADR-0019](../adr/0019-adopt-native-promptkit-configuration.md) rather than
rewriting its accepted decision text. Release preparation is separate.
Verify the dependency graph and offline examples after the cutover. Use
focused behavior tests for the new format and intentional rejection of the
old one, native same-ID replacement and relative imports, and shared source
selection across preflight, inspection, and runtime. Exercise preparation,
generation, structural repair, and semantic retries through the maintained
fake-provider integration. Check that real message or target changes affect
checkpoint reuse, and that session routing, cache controls, diagnostics, and
durable outputs keep their existing meaning. Live provider checks are useful
for deployment confidence, not a requirement of the offline suite.
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/releases/v0.23.2.md
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/consumers/configuration-migration.md
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/formats.md
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/consumers/pkg-promptkit.md
## Open Decisions
### Operator Source Selection
**Recommendation:** Expose one optional native main-file selector. It gives
operators a single entry point for native file and directory imports,
collection mounting, and relative resource resolution, while keeping source
composition out of Notarius.
**Alternative:** Expose an explicit list of file or directory inputs assembled
into native PromptKit source groups. This is useful if a caller must select
independent inputs without owning a common entry file. It also requires
Notarius to define group ordering, path resolution, and precedence for a
second composition surface. Prefer the main file unless that workflow is
needed.
### How Far Offline Preflight Goes
**Recommendation:** Check that the captured configuration is valid and that
selected resource IDs resolve. Make clear that a valid profile can be partial
and that concrete request readiness is checked during preparation. This
preserves a useful offline preflight without duplicating target resolution or
requiring credentials or source data.
**Alternative:** Preserve offline checking of completed targets by pursuing a
dedicated upstream API that resolves targets without artifacts or credentials.
That would catch more missing endpoint/model settings before a run and could
serve other consumers, but it would add an upstream dependency to this work or
require a subsequent release. Reimplementing completion rules in Notarius
would risk a discrepancy with actual execution.
### Operator Diagnostics In This Upgrade
**Recommendation:** Migrate existing inspection commands and use native
inventory and observation APIs where they replace local work. Evaluate new
commands for provider smoke checks or effective-config diffs separately,
after identifying the operator workflows and output contracts they need.
**Alternative:** Include an explicitly invoked smoke command in the cutover.
This would help operators verify credentials and structured inference after
migrating a deployment, and it can delegate to `Engine.SmokeTest`. It adds a
network-capable CLI surface, with cost and output handling to define. Keep it
separate from ordinary offline validation in either case.