Draft the PromptKit upgrade roadmap
This commit is contained in:
@@ -1,408 +1,233 @@
|
||||
# PromptKit 0.12 Configuration Migration
|
||||
# PromptKit v0.23.2 Adoption
|
||||
|
||||
## Purpose
|
||||
|
||||
Adopt PromptKit v0.12.0 after the completed ConfigKit upgrade, replacing the
|
||||
removed configuration APIs and asset formats with PromptKit's ConfigKit-backed
|
||||
configuration boundary. Make the new composition, inheritance, presence, and
|
||||
captured-configuration capabilities useful without introducing a second loader
|
||||
or changing Notarius's domain extraction and pipeline acceptance policies.
|
||||
Replace Notarius's PromptKit v0.12 integration with a native v0.23.2
|
||||
integration. Delegate configuration composition, resource inspection, request
|
||||
preparation, and model execution to PromptKit and ConfigKit. Keep application
|
||||
machinery where it implements a distinct Notarius decision, such as typed
|
||||
artifact handling, stage acceptance, cross-backend scheduling, or checkpoint
|
||||
reuse.
|
||||
|
||||
This is a feature roadmap, not an execution sequence. The operator-source and
|
||||
native-overlay decisions were accepted on 2026-10-04 and are incorporated below.
|
||||
Release preparation and publication are separate work.
|
||||
This is a hard cutover for both the Notarius application configuration and
|
||||
PromptKit resource definitions. Provide migration guidance and maintained
|
||||
examples; support only the new formats in the resulting application. This
|
||||
draft describes the target, assuming the recommendations under
|
||||
[Open Decisions](#open-decisions). It does not select a release date or
|
||||
implementation sequence.
|
||||
|
||||
## Reviewed Baseline
|
||||
|
||||
Reviewed on 2026-10-04 against Notarius commit
|
||||
`af62b3b`, with ConfigKit v0.5.0 and PromptKit v0.11.2 selected in `go.mod`.
|
||||
The remote PromptKit v0.12.0 tag resolves to full source commit
|
||||
`4b8169da286917fde7875425bb32b799b12c6af3`, matching the sibling checkout.
|
||||
Primary upstream references are the
|
||||
[release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md),
|
||||
[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api),
|
||||
and [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md).
|
||||
These were reviewed in the local checkout; the remote tag was verified directly.
|
||||
Notarius at `65b8e8f0a2d60041c18a29a3c33afc5c689468df` uses application
|
||||
configuration version 5, Go 1.27, new-path ConfigKit v1.2.0, and old-path
|
||||
PromptKit v0.12.0. The latter brings in a separate old-path ConfigKit v0.5.0
|
||||
and two external provider catalogs. Notarius and PromptKit currently share
|
||||
the YAML v4 release candidate.
|
||||
|
||||
The current integration has these migration-sensitive boundaries:
|
||||
The target is `go.maximumdirect.net/promptkit@v0.23.2`, published at
|
||||
`382ccfb14b9630fd8f48c8364dbe2b6159637d71`. The tagged [release
|
||||
notes][release], [migration guide][migration], [format reference][formats],
|
||||
and [Go consumer guide][consumer] are the primary upstream references.
|
||||
|
||||
- `internal/framework/llm/asset_registry.go` flattens module-owned prompt,
|
||||
schema, and fallback-profile filesystems and constructs `WithPromptFS`,
|
||||
`WithSchemaFS`, and `WithFallbackProfileFS` options.
|
||||
- `promptkit_profiles.go` shares profile-source construction between the
|
||||
execution client and credential-free explicit-profile inspection. It uses
|
||||
`Config.ProfileDir`, `WithProfileFile`, and the backend registration option.
|
||||
The inspection engine supplies `PromptDir: "."` without preparing a prompt.
|
||||
- `promptkit_client.go` constructs the engine and uses prepared execution,
|
||||
semantic correction messages, structural repair overrides, observations,
|
||||
direct sessions, reasoning overrides, and provider-neutral error adaptation.
|
||||
- `promptkit_profile_fingerprint.go` combines a pinned catalog marker with
|
||||
reread operator profile files and fallback asset bytes. The optional local
|
||||
endpoint contributes a separate identity; local concurrency is excluded.
|
||||
- Embedded prompt definitions throughout `assets/dnd/` and
|
||||
`assets/generic/normalize/deduplication/` use top-level `id` and `version`.
|
||||
`assets/dnd/profiles/dnd-extraction.yaml` and the maintained deployment file
|
||||
`examples/llm-profiles/dnd-extraction.yml` use top-level `id`.
|
||||
- The public profile contract says a matching operator profile completely
|
||||
replaces a fallback or catalog profile. It documents `base_profile`
|
||||
inheritance. These claims do not match v0.12.0's native semantics.
|
||||
The main baseline seams are:
|
||||
|
||||
The current focused LLM, CLI, and module-integration tests pass with
|
||||
`GOWORK=off`. This establishes the baseline only: no trial dependency upgrade,
|
||||
live inference, or production model-quality evaluation has been performed.
|
||||
|
||||
## Upstream Changes Relevant To Notarius
|
||||
|
||||
PromptKit v0.12.0 is an incompatible configuration/API release:
|
||||
|
||||
- `Config.Sources` accepts ordered ConfigKit underlays and an optional main
|
||||
input. Legacy prompt/profile directory fields and configuration-source
|
||||
options are removed. Schema and injected-runtime options remain available.
|
||||
- Documents use keyed `backends`, `profiles`, and `prompts` mappings, with a
|
||||
second version mapping beneath each prompt ID. Embedded `id`, `version`, and
|
||||
`base_profile` fields are invalid; profile inheritance uses `extends`.
|
||||
- Files within one source group compose strictly, not by filename precedence.
|
||||
Later groups overlay earlier groups recursively. Explicit null, zero,
|
||||
false, and empty values can differ from omission; `{}` does not clear a map.
|
||||
- Engine construction validates every final effective definition, including
|
||||
unused definitions, and captures message content and static templates.
|
||||
Ordinary execution and inspection do not reload them. Schemas, request
|
||||
artifacts, credentials, and provider operations retain their runtime timing.
|
||||
- `InspectConfiguration` exposes captured presence, sources, diagnostics,
|
||||
explanations, and redacted effective output. These reports are not
|
||||
automatically safe to publish: ordinary prompt/provider content remains data.
|
||||
- Maintained catalogs move to `/v3`, both at `v3.20260930.0`.
|
||||
PromptKit and Notarius already share the required ConfigKit v0.5.0 and Go
|
||||
1.25.5 baseline; no additional Go-version upgrade is required by this release.
|
||||
- `internal/core/config` resolves `promptkit.profile_file` and
|
||||
`profile_dir` and interprets the optional `local_backend` object.
|
||||
`promptkit_profiles.go` builds the operator layers for execution and
|
||||
inspection; `asset_registry.go` supplies the embedded layer.
|
||||
- `assets/` and `examples/llm-profiles/` use keyed resource definitions.
|
||||
Completion requests and diagnostics carry a PromptKit prompt version.
|
||||
- `internal/framework/llm` does split preparation and execution to retain
|
||||
debug material; it projects a backend-completed profile report and walks
|
||||
effective YAML to discover versioned prompts for checkpoint identity.
|
||||
- The pipeline's producer-attempt loop coordinates domain and stage policy.
|
||||
The scheduled LLM client enforces the application-wide limit independently
|
||||
of any PromptKit backend policy.
|
||||
|
||||
## Target End State
|
||||
|
||||
### One Native Configuration Boundary
|
||||
### Dependency And Application Configuration Cutover
|
||||
|
||||
PromptKit owns the effective backend/profile/prompt configuration and all
|
||||
upstream composition and inheritance. Notarius selects its sources and adapts
|
||||
the resulting engine to the existing provider-neutral completion contract.
|
||||
ConfigKit and PromptKit types remain inside the LLM adapter's explicit
|
||||
integration boundary, not in domain modules or pipeline contracts.
|
||||
Use the vanity PromptKit import path and shared ConfigKit v1.2.1, stable
|
||||
go-yaml v4, and the resulting compatible validation dependencies. Remove the
|
||||
old-path imports and catalog dependencies once unused. Use tagged modules
|
||||
without a compatibility fork or permanent replacement. The consumer minimum
|
||||
remains Go 1.27. Update attribution for the selected and removed components
|
||||
under the [OSS policy](../policy/oss.md).
|
||||
|
||||
Retain Notarius configuration `version: 5`, `promptkit.profile_file`,
|
||||
`promptkit.profile_dir`, their mutual exclusion and declaring-file-relative
|
||||
path handling, `transient_retries`, and the optional conventional local backend.
|
||||
Do not add replacement Notarius configuration fields merely because the
|
||||
upstream constructor has changed. Document the changed external file format.
|
||||
Introduce application configuration version 6 as the only supported application
|
||||
version. Version 6 retains unrelated pipeline structure, discovery, and
|
||||
resolution behavior. ConfigKit continues to own application loading and
|
||||
composition under [ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
|
||||
The proposed operator seam is an optional `promptkit.config_file` pointing to a
|
||||
native PromptKit main source. Resolve this selector relative to the Notarius
|
||||
file that declares it; leave imports and resource paths *inside* the selected
|
||||
source to the upstream loader. Retire `profile_file`, `profile_dir`, and
|
||||
`local_backend`: express related sources through native imports and local
|
||||
endpoints through native backend records. Preserve the ability to override
|
||||
transport retries through the PromptKit engine option; leave the upstream
|
||||
default in effect when the operator does not override it.
|
||||
|
||||
Use one shared source-construction path for execution and profile preflight.
|
||||
The source precedence is:
|
||||
Pass the selected main source and application asset underlay through one source
|
||||
assembly used by runtime, preflight, and inspection. Use the bundled OpenRouter
|
||||
underlay as supplied by PromptKit. That bundle is a different inventory from
|
||||
the former external catalogs and supplies no backend-local concurrency limit.
|
||||
It does not implicitly select a model for any Notarius request. Migrate
|
||||
deployments that use removed profiles or backends to intentionally selected
|
||||
native records. The application-wide LLM ceiling remains independent of
|
||||
backend-local settings.
|
||||
|
||||
1. PromptKit's maintained catalog layer, supplied by PromptKit itself.
|
||||
2. Registered application prompt and fallback-profile assets.
|
||||
3. The explicitly selected operator file or directory.
|
||||
4. A programmatic `local` backend contribution when Notarius's
|
||||
`promptkit.local_backend` is configured, so those explicit Notarius values
|
||||
win over conflicting file declarations of those fields.
|
||||
### Native Prompt And Profile Resources
|
||||
|
||||
The existing selected file/directory accepts native combined `profiles`,
|
||||
`backends`, and `prompts` documents. Enable explicit ConfigKit imports on these
|
||||
operator inputs. Describe the familiar `profile_file` and `profile_dir` fields
|
||||
as PromptKit configuration sources whose principal workflow is profile
|
||||
selection; do not add a section whitelist or a custom merge language.
|
||||
Domain modules remain responsible for interpreting and validating model output,
|
||||
even when an operator supplies a prompt overlay.
|
||||
Convert maintained prompts, profiles, examples, and fixtures to identified
|
||||
lists, with `config:` for source controls and native sibling inheritance
|
||||
(`extends: {id: parent}`). Under native default composition, a later same-ID
|
||||
record replaces the earlier record in full. Explain complete replacements,
|
||||
distinct inherited variants, and the new backend fallback semantics for
|
||||
profile nulls.
|
||||
The optional PromptKit source specification header is version 2; it is not the
|
||||
Notarius application version. Keep source-local message resolution and use
|
||||
native collections when deployments need namespaces. Do not translate
|
||||
legacy resource files, synthesize aliases, or reconstruct old recursive
|
||||
same-ID overlays.
|
||||
|
||||
Inputs within each group use native strict composition. Directory discovery is
|
||||
recursive over YAML files and has no hidden per-file priority. No unselected
|
||||
directory is scanned. Construct filesystem-backed inputs with usable resource
|
||||
contexts so referenced message files retain their declaring-source base.
|
||||
Do not combine host profile files with embedded content under a fabricated
|
||||
common filesystem root.
|
||||
Select prompts by ID alone and remove the obsolete PromptKit version selector
|
||||
and metadata from Notarius's request, debug, and reconciliation identity
|
||||
plumbing. Review affected diagnostic fields and document deliberate changes.
|
||||
Retain logical workload IDs, including `dnd-extraction` and its current
|
||||
explicit model/settings. Retain message roles, bytes, order, cache boundaries,
|
||||
and intended output/repair behavior while changing the declaration format.
|
||||
Keep module-owned asset selection and embedded filesystem contexts intact.
|
||||
Verify maintained private response schemas work under upstream's
|
||||
self-contained-schema rule; this is not a redesign of durable D&D artifacts.
|
||||
|
||||
Document import layouts that do not accidentally select the same definitions
|
||||
through both directory discovery and imports: a file-root import layout is
|
||||
often the simplest choice. Use native import/resource rules rather than
|
||||
inventing deduplication, filename override order, or broader host access.
|
||||
### Inspection And Reuse Identity
|
||||
|
||||
Use `FilesystemInputs` for the existing flattened embedded views,
|
||||
`DirectoryInputs` or an individual ConfigKit file input for the operator
|
||||
source, and `BackendSource` with the presence-aware local-backend helper for
|
||||
programmatic backend configuration. Keep schemas on the existing schema
|
||||
filesystem boundary. There is no need to deserialize catalog assets, register
|
||||
catalog backends individually, or recreate PromptKit inheritance.
|
||||
Use PromptKit's captured configuration reports for effective configuration
|
||||
and provenance. Use its resource inventory and inspection APIs for exact
|
||||
identities, presence-aware resource fields, and captured prompt hashes, rather
|
||||
than deriving an inventory from rendered configuration YAML. Change the
|
||||
`config profile` report to show the resolved profile and its selected backend,
|
||||
when present, as separate resources. Use upstream field semantics and retain
|
||||
the existing operator disclosure boundary, including redaction of
|
||||
credential-shaped provider parameters. Do not recreate backend default
|
||||
resolution to produce the former `effective_model_params` view.
|
||||
|
||||
Profile preflight does not need dummy prompt directories or synthetic prompts.
|
||||
Both construction paths include the same catalog, application prompt/fallback,
|
||||
operator, and local-backend layers. This matters for partial operator prompt
|
||||
overlays: inspecting a layer without its application base would validate a
|
||||
different effective definition. Call `InspectProfile` on the constructed engine
|
||||
without preparing a synthetic request. Construction still captures and
|
||||
validates the effective prompt definitions; preflight does not resolve
|
||||
credential values, prepare request artifacts, or contact providers.
|
||||
Distinguish configuration and resource preflight from request readiness.
|
||||
Use `InspectRequirements` to explain recognized dependencies when helpful,
|
||||
not as proof that a request is executable. Use `Prepare` or
|
||||
`ValidateRequest` when concrete request inputs are available and their
|
||||
validation is actually needed. A routine config-only check should remain
|
||||
credential-free and offline, without synthetic requests.
|
||||
|
||||
### Keyed Assets Without Prompt Behavior Changes
|
||||
Retain a private, versioned Notarius checkpoint identity based on the engine's
|
||||
captured state. Use native resource enumeration and prompt hashes, keeping
|
||||
the effective execution inputs needed to distinguish reuse safely. A diff of
|
||||
effective values alone cannot detect edits to a referenced message file.
|
||||
Version the fingerprint for the cutover so obsolete identity inputs cannot
|
||||
cause a false hit; do not reread mutable sources or expose the digest inputs.
|
||||
Keep run manifests based on the profiles actually used, not the inventory.
|
||||
|
||||
Every maintained prompt declaration becomes a keyed document of this shape:
|
||||
### Native Execution And Retained Pipeline Policy
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
dnd.spells:
|
||||
v1:
|
||||
# Existing definition body, without embedded id/version fields.
|
||||
```
|
||||
Use ordinary `Engine.Run` for structured completion and use `OnPrepared`
|
||||
where opt-in debug capture needs the actual prepared request. The callback
|
||||
runs after admission, before generation, and is not delivered for failed
|
||||
preparation or admission. Keep the callback lightweight and use native
|
||||
observations for safe terminal execution facts, including attempts, repairs,
|
||||
and available token usage. Where a diagnostic needs totals over selected
|
||||
executions, use `ExecutionGroup` rather than inventing a second accumulator.
|
||||
Use `OperationTimeout` where a completion calls for that scope of budget;
|
||||
do not add a new default deadline or replace application deadlines that also
|
||||
cover work outside the PromptKit call.
|
||||
|
||||
Profiles become keyed entries beneath `profiles`. Preserve literal prompt IDs,
|
||||
the `v1` version, logical profile IDs, selected response schemas, content paths,
|
||||
message order, roles, shared fragment bytes, cache-control metadata, and
|
||||
configured structural repair budgets. Do not conflate this migration with
|
||||
prompt rewriting, schema changes, or durable artifact version changes.
|
||||
PromptKit owns provider admission, transport retries, and structural repair.
|
||||
Keep the Notarius cross-backend scheduler because it serves a different
|
||||
budget from backend or pipeline worker limits. Keep the adapter's typed
|
||||
decode, actual-run provenance, and provider-neutral error and diagnostic
|
||||
mapping. PromptKit execution types stay within that integration boundary.
|
||||
In particular, preserve the distinction between invalid structured
|
||||
output, provider failure, capacity rejection, and cancellation. A configured
|
||||
repair override still selects a complete upstream output contract; it must not
|
||||
accidentally drop the selected schema or validation mode.
|
||||
|
||||
The application-owned `dnd-extraction` fallback continues to explicitly select
|
||||
`openrouter`, `openai/gpt-5.6-luna`, timeout 240, and service tier `flex`, with
|
||||
reasoning effort unset. Do not replace that choice with an upstream catalog
|
||||
alias or adopt unrelated model defaults. Migrate the maintained operator
|
||||
example and relevant inline Go/YAML test fixtures as well as production assets.
|
||||
Notarius owns stage attempts, validation and fallback decisions, and
|
||||
publication of accepted artifacts. Keep its latest-response-only correction
|
||||
policy using native appended messages, as defined by
|
||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
`Exchange` and `Coordinate` are for workflows that benefit from retained
|
||||
conversation history. Adopting them for stage retries would change what the
|
||||
model sees and which work consumes an attempt. They are not needed to execute
|
||||
the existing retry policy.
|
||||
|
||||
Module ownership, ordered asset manifests, scoped mounting, and selected-asset
|
||||
hashing remain intact. Check virtual filenames and source-relative
|
||||
`content_file` resolution after wrapping definitions; do not move business
|
||||
logic into the root `assets` package.
|
||||
## Migration And Verification
|
||||
|
||||
### Presence, Inheritance, And Operator Migration
|
||||
Update the maintained minimal and complete D&D examples, including operator
|
||||
profile inheritance, and document how to convert a directory deployment,
|
||||
declare a local backend, and replace an embedded prompt or profile. Explain
|
||||
retired catalog IDs, the changed profile report, and any new checkpoint cold
|
||||
misses. Update the canonical configuration, CLI, operations, and integration
|
||||
guides when behavior lands. Record the changed decisions about source
|
||||
precedence and inspection in a successor to
|
||||
[ADR-0019](../adr/0019-adopt-native-promptkit-configuration.md) rather than
|
||||
rewriting its accepted decision text. Release preparation is separate.
|
||||
|
||||
The adopted policy is native recursive overlay, not a compatibility shim
|
||||
for complete-definition replacement. Operators can override only the fields
|
||||
they want. Omitted fields retain lower-layer values, and `extends` uses native
|
||||
literal sibling-key lookup. Changing a backend or model does not erase
|
||||
explicit inherited settings; backend-derived defaults are applied to the
|
||||
final backend after inheritance.
|
||||
Verify the dependency graph and offline examples after the cutover. Use
|
||||
focused behavior tests for the new format and intentional rejection of the
|
||||
old one, native same-ID replacement and relative imports, and shared source
|
||||
selection across preflight, inspection, and runtime. Exercise preparation,
|
||||
generation, structural repair, and semantic retries through the maintained
|
||||
fake-provider integration. Check that real message or target changes affect
|
||||
checkpoint reuse, and that session routing, cache controls, diagnostics, and
|
||||
durable outputs keep their existing meaning. Live provider checks are useful
|
||||
for deployment confidence, not a requirement of the offline suite.
|
||||
|
||||
Document intentional clearing with `null`, including reasoning/service tier,
|
||||
sampling controls, credential lookup, and provider maps where supported.
|
||||
Distinguish null from zero/false/blank and from an empty mapping. For example,
|
||||
switching an operator's `dnd-extraction` overlay to a local model may also need
|
||||
`service_tier: null` to remove the fallback's explicit `flex` setting. Do not
|
||||
automatically scrub settings based on model-name heuristics.
|
||||
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/releases/v0.23.2.md
|
||||
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/consumers/configuration-migration.md
|
||||
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/formats.md
|
||||
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/consumers/pkg-promptkit.md
|
||||
|
||||
Translate `base_profile` to `extends` in maintained examples and explain that
|
||||
operators must migrate their external files. Do not retain legacy aliases or
|
||||
an alternate parser. Preserve exact PromptKit keys; do not introduce a new
|
||||
identifier normalization pass. Update tests that assumed whole replacement
|
||||
or selection-time validation deliberately, rather than treating those
|
||||
intentional upstream changes as regressions.
|
||||
## Open Decisions
|
||||
|
||||
Native eager validation applies to final effective values, not independently
|
||||
to each superseded contribution. Do not add a source-history audit or raw-layer
|
||||
validation that prevents legitimate overrides from repairing lower layers.
|
||||
### Operator Source Selection
|
||||
|
||||
### Native Operator Capabilities
|
||||
**Recommendation:** Expose one optional native main-file selector. It gives
|
||||
operators a single entry point for native file and directory imports,
|
||||
collection mounting, and relative resource resolution, while keeping source
|
||||
composition out of Notarius.
|
||||
|
||||
Support the capabilities of the unified format directly, without new Notarius
|
||||
configuration knobs or model-specific adaptation:
|
||||
**Alternative:** Expose an explicit list of file or directory inputs assembled
|
||||
into native PromptKit source groups. This is useful if a caller must select
|
||||
independent inputs without owning a common entry file. It also requires
|
||||
Notarius to define group ordering, path resolution, and precedence for a
|
||||
second composition surface. Prefer the main file unless that workflow is
|
||||
needed.
|
||||
|
||||
- Custom or multiple backend definitions, including endpoint, credential
|
||||
environment name, provider parameters, concurrency, and queue capacity.
|
||||
Operators may also overlay maintained backend settings, for example changing
|
||||
OpenRouter's admission limit. Backend IDs are configuration keys, not a
|
||||
closed registration list. Notarius's global scheduler still applies.
|
||||
- Profile inheritance, partial overrides, and recursive provider-parameter
|
||||
composition. Explicit null suppression and zero/false values follow native
|
||||
field rules. Native sibling inheritance is not limited by a Notarius
|
||||
profiles-only whitelist; prompt/backend mappings follow upstream rules too.
|
||||
- File-configured `api_key_required`, with PromptKit's exact credential
|
||||
precedence and runtime checks. In particular, true suppresses a backend's
|
||||
implicit environment source; an explicitly configured profile environment
|
||||
source remains usable. This is an operator choice, not a new mandatory
|
||||
credential policy or a reason to require credentials during preflight.
|
||||
- Deployment prompt overlays and declaring-source-relative message files,
|
||||
including imported content. Prompt version selection remains caller-owned;
|
||||
this upgrade does not infer a new module version from file order or add
|
||||
automatic latest-version selection.
|
||||
### How Far Offline Preflight Goes
|
||||
|
||||
The existing `local_backend` object remains a convenient highest-priority
|
||||
contribution for that conventional backend. Other backend settings remain
|
||||
available through the unified operator source; do not copy every upstream field
|
||||
into a parallel Notarius backend schema.
|
||||
**Recommendation:** Check that the captured configuration is valid and that
|
||||
selected resource IDs resolve. Make clear that a valid profile can be partial
|
||||
and that concrete request readiness is checked during preparation. This
|
||||
preserves a useful offline preflight without duplicating target resolution or
|
||||
requiring credentials or source data.
|
||||
|
||||
Keep the two maintained top-level configuration examples. Use the complete
|
||||
example's existing PromptKit deployment file and small canonical documentation
|
||||
snippets to demonstrate native inheritance/partial overrides, custom backend
|
||||
configuration, imports, null clearing, and prompt editing without adding a
|
||||
large unrelated example suite. Not every capability needs its own runnable
|
||||
pipeline.
|
||||
**Alternative:** Preserve offline checking of completed targets by pursuing a
|
||||
dedicated upstream API that resolves targets without artifacts or credentials.
|
||||
That would catch more missing endpoint/model settings before a run and could
|
||||
serve other consumers, but it would add an upstream dependency to this work or
|
||||
require a subsequent release. Reimplementing completion rules in Notarius
|
||||
would risk a discrepancy with actual execution.
|
||||
|
||||
### Captured Execution And Checkpoint Identity
|
||||
### Operator Diagnostics In This Upgrade
|
||||
|
||||
One engine represents one captured configuration for its lifetime. Editing
|
||||
operator definitions or message files requires constructing a new engine;
|
||||
there is no watcher or automatic hot reload in this scope. Preserve the normal
|
||||
CLI lifecycle without promising that independently constructed preflight and
|
||||
execution engines are one transactional filesystem snapshot.
|
||||
**Recommendation:** Migrate existing inspection commands and use native
|
||||
inventory and observation APIs where they replace local work. Evaluate new
|
||||
commands for provider smoke checks or effective-config diffs separately,
|
||||
after identifying the operator workflows and output contracts they need.
|
||||
|
||||
Compute and retain checkpoint identity from the configuration actually
|
||||
captured by the execution engine. Do not identify an old captured engine by
|
||||
files reread after it was constructed. Identity must account for effective
|
||||
profile/backend semantics and the pinned PromptKit/catalog revisions. It must
|
||||
also account for operator prompt definitions and their
|
||||
captured prompt content, not just `content_file` path strings.
|
||||
|
||||
Use public captured inspection and prompt-definition equality values where
|
||||
appropriate; do not reach into PromptKit internals. `InspectPrompt.PromptHash`
|
||||
includes captured content and is an opaque equality value. Inspection reports
|
||||
alone do not embed referenced message-file bodies. Preserve application
|
||||
module/schema fingerprints, actual-used profile provenance, and the existing
|
||||
exclusion of concurrency-only local settings from semantic checkpoint identity.
|
||||
A conservative identity across effective profiles is acceptable; per-selected-
|
||||
profile optimization is not a prerequisite for this upgrade.
|
||||
|
||||
Raw sources, configuration history, prompt bodies, endpoints, and credential
|
||||
values must not be added to ordinary manifests or checkpoint metadata. The
|
||||
new inspection API supports both private captured identity and the explicitly
|
||||
requested operator reports below. Do not automatically dump reports into run
|
||||
output or debug bundles. Hash-only identity is distinct from publishing an
|
||||
inspection report, and redaction is not a substitute for covering execution
|
||||
semantics.
|
||||
|
||||
### Operator-Facing PromptKit Inspection
|
||||
|
||||
Provide explicit CLI operations for three complementary views:
|
||||
|
||||
- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit
|
||||
configuration, including effective keyed backends, profiles, and prompts.
|
||||
Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing
|
||||
YAML from Notarius's profile declarations or runtime manifests.
|
||||
- **Setting explanation:** explain one canonical JSON Pointer in that captured
|
||||
configuration, including source history, layer overrides, and inheritance.
|
||||
Use the captured inspection's explanation API. Literal IDs containing `/`
|
||||
or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation.
|
||||
- **Resolved profile:** inspect one explicit profile ID with `InspectProfile`
|
||||
and present its backend-completed execution settings, including target,
|
||||
effective reasoning/provider controls, credential environment name and
|
||||
requirement, and selected backend identity. Endpoint-only profiles remain
|
||||
identifiable as such. Do not conflate configured profile fields with values
|
||||
derived from backend defaults.
|
||||
|
||||
Use the same selected Notarius configuration/overlay, registered application
|
||||
assets, operator sources, and local-backend contribution as execution and
|
||||
preflight. A report describes the engine captured for that invocation, not a
|
||||
live view of a separate running process. Constructing the engine performs
|
||||
native definition/content validation, but these commands do not need transcript
|
||||
input, execute pipeline stages, prepare requests, resolve credential values,
|
||||
admit provider capacity, or contact a model. Required-credential profiles remain
|
||||
inspectable when the named environment variable is unset.
|
||||
|
||||
Keep existing Notarius `config effective` and `config explain` behavior intact;
|
||||
the PromptKit view must be an explicit selection, not a replacement or an
|
||||
automatic second document appended to their output. Define the concrete command
|
||||
syntax in the implementation plan and maintain it in `docs/cli.md` when it
|
||||
lands. Standard output contains the selected report only; ordinary diagnostics
|
||||
and failures follow the existing CLI conventions. Unknown paths/profiles and
|
||||
invalid pointers must be distinguishable without leaking raw configuration
|
||||
through error messages.
|
||||
|
||||
Reports are operator-requested application data. Preserve upstream credential
|
||||
redaction and Notarius's existing secret-handling boundary; include credential
|
||||
variable names, never their values. Configuration explanations can contain
|
||||
overridden values and inline prompt/provider content, and effective YAML is not
|
||||
a transcript-free or generally sanitized report. Document these properties and
|
||||
operator-owned output handling without adding permission enforcement, new
|
||||
confirmation gates, automatic report persistence, or blanket content removal.
|
||||
|
||||
Effective configuration, backend-completed profile settings, and prepared
|
||||
request details are different views. These commands do not render a prompt
|
||||
with transcript inputs or return captured message-file bodies merely because
|
||||
they appear in the configuration as `content_file` references. Keep complete
|
||||
request/response inspection on the existing explicitly requested debug path.
|
||||
|
||||
### Existing Execution Policies Remain Unchanged
|
||||
|
||||
Preserve prepared execution, the exact latest-response semantic correction
|
||||
pair, prompt-prefix caching, direct sessions, reasoning overrides, cancellation,
|
||||
the shared scheduler, local/backend admission, observer accounting, error
|
||||
classification, and credential redaction. Structural repairs, transient HTTP
|
||||
retries, outer producer attempts, and validator execution budgets remain
|
||||
separate. The existing one-repair maintained prompt default remains unchanged.
|
||||
|
||||
The migration does not change extraction/normalization semantics, warning
|
||||
classification, artifact contracts, subprocess receipts, or reusable-state
|
||||
eligibility. Construction errors may occur earlier under native eager
|
||||
validation; adapt them through the existing safe adapter/CLI error boundary,
|
||||
not by exposing arbitrary upstream configuration content.
|
||||
|
||||
## Required Work
|
||||
|
||||
- Update `go.mod` and `go.sum` to PromptKit v0.12.0 and its selected `/v3`
|
||||
catalogs. Keep ConfigKit v0.5.0 and unrelated dependencies unless module
|
||||
resolution requires a change. Remove obsolete `/v2` requirements after tidy.
|
||||
- Replace removed constructor/source/backend APIs in the LLM adapter and
|
||||
shared profile-preflight construction; retain schema and injected-client
|
||||
options. Remove obsolete adapter helpers rather than layering a parallel
|
||||
configuration system beside them.
|
||||
- Migrate every registered prompt and fallback profile, the maintained
|
||||
deployment profile, and affected test fixtures to keyed definitions.
|
||||
- Adopt the unified source policy, explicit operator imports, presence/null
|
||||
behavior, native operator capabilities, and captured
|
||||
lifetime; replace live-file checkpoint fingerprinting with captured identity.
|
||||
- Add explicit effective-configuration, setting-explanation, and resolved-
|
||||
profile CLI inspection using the shared captured-engine construction and
|
||||
public PromptKit inspection APIs. Keep upstream inspection values within
|
||||
the adapter boundary and preserve ordinary Notarius configuration reports.
|
||||
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
|
||||
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
|
||||
loading and offline injected clients or local HTTP fixtures.
|
||||
- Update canonical documentation when implementation lands:
|
||||
`docs/config.md` owns selected sources, precedence, profile syntax and
|
||||
migration; `docs/operations.md` owns deployment and reload guidance;
|
||||
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
|
||||
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
|
||||
pinned links; `docs/cli.md` owns the new inspection operations and output/error
|
||||
behavior. Add a short troubleshooting workflow in `docs/operations.md` that
|
||||
links to those references. Revise consumer examples only where affected.
|
||||
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
|
||||
checking their actual notices and exact source commits. PromptKit remains
|
||||
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
|
||||
unchanged standard texts and preserve the project's own license grant.
|
||||
|
||||
Native layered overrides change an existing documented operator contract.
|
||||
Add a concise Notarius ADR recording the accepted native composition and
|
||||
unified-source ownership/compatibility decision rather than duplicating the
|
||||
upstream manual. Existing historical ADRs need not be rewritten.
|
||||
|
||||
## Verification And Acceptance
|
||||
|
||||
Acceptance is behavioral, not an inventory of private helper names:
|
||||
|
||||
- All registered D&D and generic prompts construct and prepare with migrated
|
||||
assets, resolving shared/module content in the correct source context.
|
||||
Request identities, message content/order, schemas, and cache controls retain
|
||||
their intended behavior. Do not add exact shared-prefix-length detector tests.
|
||||
- Operator overrides, inheritance, explicit zero/null, duplicate definitions
|
||||
within a group, local backend precedence, endpoint-only profiles, and invalid
|
||||
unused final definitions follow the native configuration policy. Cover
|
||||
representative imported combined definitions, custom/backend catalog
|
||||
overlays, partial prompt overlays with their embedded base, and optional
|
||||
versus explicitly required credentials. Credential-free preflight and
|
||||
execution resolve the same source ordering and application definitions.
|
||||
- Engine/file mutation tests demonstrate captured settings and content;
|
||||
fingerprints stay tied to that engine and change when a new engine captures
|
||||
different execution-relevant settings or operator prompt content.
|
||||
- CLI inspection reports effective values, setting origins, and backend-
|
||||
completed profiles from the same source policy without transcript input,
|
||||
credentials, or provider calls. Cover representative inheritance/null
|
||||
overrides, escaped literal IDs, required-credential inspection with an unset
|
||||
variable, and invalid/unknown selectors. Verify redaction and unchanged
|
||||
ordinary Notarius reports without pinning incidental report formatting.
|
||||
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
|
||||
transient errors, safe diagnostics, and observations retain existing
|
||||
provider-neutral behavior using deterministic boundary tests.
|
||||
- Maintained minimal, complete, and inherited development pipelines validate
|
||||
without credentials or provider calls. External legacy profile files receive
|
||||
clear migration guidance, not silent reinterpretation.
|
||||
|
||||
Use focused LLM, CLI, and module-integration tests during implementation, then
|
||||
`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, a temporary native build,
|
||||
and `GOWORK=off go mod tidy -diff` for this shared-boundary migration. Verify
|
||||
selected module versions, attribution links, maintained examples, and stale
|
||||
API/format claims. These are migration checks, not new permanent release gates.
|
||||
Live provider testing is optional follow-up evidence, not required acceptance.
|
||||
|
||||
This work warrants a minor Notarius release because the accepted external
|
||||
profile format and override behavior change. Keep Notarius configuration at
|
||||
version 5; select the actual release number during separate release preparation.
|
||||
**Alternative:** Include an explicitly invoked smoke command in the cutover.
|
||||
This would help operators verify credentials and structured inference after
|
||||
migrating a deployment, and it can delegate to `Engine.SmokeTest`. It adds a
|
||||
network-capable CLI surface, with cost and output handling to define. Keep it
|
||||
separate from ordinary offline validation in either case.
|
||||
|
||||
Reference in New Issue
Block a user