From 4aba4f5672dca2b7ff354507d79008647f241b2a Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 4 Oct 2026 14:12:11 +0000 Subject: [PATCH] Document PromptKit configuration ownership and capture --- ...-workload-oriented-llm-profile-defaults.md | 21 +++-- ...19-adopt-native-promptkit-configuration.md | 92 +++++++++++++++++++ docs/integrations/pkg-promptkit.md | 31 ++++++- docs/internal/cli.md | 44 +++++---- docs/internal/llm.md | 78 +++++++++------- docs/roadmap/implementation.md | 11 +++ 6 files changed, 215 insertions(+), 62 deletions(-) create mode 100644 docs/adr/0019-adopt-native-promptkit-configuration.md diff --git a/docs/adr/0010-workload-oriented-llm-profile-defaults.md b/docs/adr/0010-workload-oriented-llm-profile-defaults.md index cd6fe957..518dce05 100644 --- a/docs/adr/0010-workload-oriented-llm-profile-defaults.md +++ b/docs/adr/0010-workload-oriented-llm-profile-defaults.md @@ -1,6 +1,7 @@ # ADR-0010: Use workload-oriented LLM profile defaults -**Status:** Accepted +**Status:** Accepted; profile source replacement behavior superseded by +[ADR-0019](0019-adopt-native-promptkit-configuration.md) **Date:** 2026-08-03 ## Context @@ -23,11 +24,13 @@ intentional exceptions, and the run-wide CLI profile override has highest precedence. The D&D family owns an embedded fallback profile named `dnd-extraction`. -Operators may provide a complete profile with the same ID through a PromptKit -filesystem source. PromptKit selects the higher-precedence matching definition; -Notarius does not merge profile documents. Production, development, and local -deployments can therefore use different execution policy behind one unchanged -pipeline ID. +Operators may configure the same ID through a PromptKit filesystem source. The +original decision specified that an operator definition replaced the fallback +as a complete profile; that source-composition clause has been superseded by +[ADR-0019](0019-adopt-native-promptkit-configuration.md), which adopts native +recursive overlays. The workload-oriented identity and D&D ownership decision +remain in force. Production, development, and local deployments can therefore +use different execution policy behind one unchanged pipeline ID. ## Alternatives considered @@ -42,8 +45,8 @@ pipeline ID. ## Consequences Pipeline configuration expresses workload intent rather than a specific -provider or model. Operators can replace the complete execution policy without -editing bindings, while binding-level and run-wide exceptions remain available. -Profile changes affect resolved pipeline and checkpoint identity, so they may +provider or model. Operators can configure execution policy without editing +bindings, while binding-level and run-wide exceptions remain available. Profile +changes affect resolved pipeline and checkpoint identity, so they may intentionally cause work to be recomputed. The D&D fallback becomes a maintained application execution-policy asset. diff --git a/docs/adr/0019-adopt-native-promptkit-configuration.md b/docs/adr/0019-adopt-native-promptkit-configuration.md new file mode 100644 index 00000000..7d07d399 --- /dev/null +++ b/docs/adr/0019-adopt-native-promptkit-configuration.md @@ -0,0 +1,92 @@ +# ADR-0019: Adopt native PromptKit configuration sources + +**Status:** Accepted +**Date:** 2026-10-04 + +## Context + +PromptKit v0.12 replaces its former source options with ConfigKit-backed +configuration sources, keyed documents, recursive overlays, and native +inheritance. Notarius must choose how application-owned prompt and fallback +assets, operator configuration, and the optional local backend reach PromptKit +without recreating source parsing or inheritance in the adapter. Runtime, +preflight, and operator inspection must agree about the effective definitions. + +Notarius also captures checkpoint identity for the PromptKit configuration +used by an execution engine. That identity must follow the engine's captured +definition state rather than re-reading mutable source files, and must not +publish credentials, prompt bodies, provider parameters, or source paths. + +## Decision + +Use PromptKit's public v0.12 native source APIs as the only PromptKit +configuration boundary. Assemble PromptKit catalogs, registered application +prompt and fallback inputs, the optional operator file or recursively selected +directory, and the optional Notarius local backend registration in that order. +Use native ConfigKit imports and PromptKit/ConfigKit composition behavior; +retain each filesystem input's resource context. Inputs within a group compose +strictly, and later groups recursively overlay earlier ones. PromptKit owns +definition parsing, exact-key `extends` inheritance, and final effective-value +validation. Notarius does not provide a legacy parser, compatibility alias, +filename-priority merger, whole-profile replacement shim, or alternate +inheritance engine. + +Execution, explicit-profile preflight, and configuration inspection use the +same source construction. Each constructed engine captures its effective +definitions. Edits to source files take effect for a newly constructed engine, +not an existing one. Derive the private versioned +`promptkit_configuration` checkpoint identity from public captured inspection +and definition APIs, including effective configuration, profile targets and +credential-policy metadata, captured prompt hashes, and the PromptKit and +selected catalog identities. Retain only the digest as checkpoint metadata; +keep all digest inputs private. Exclude local-backend concurrency settings +because they affect scheduling rather than execution semantics. + +Keep PromptKit types and source assembly inside the LLM integration boundary. +The CLI may project copied profile fields into a Notarius-owned neutral JSON +contract. PromptKit's effective and explanation reports remain upstream +redacted configuration and source-history views. Neither inspection path reads +credential values or contacts a provider. + +## Compatibility + +The migration to v0.12 keyed configuration is an intentional external file +format change. Operators must migrate `id`/`version` embedded fields and +`base_profile` to native keyed documents and `extends`. Notarius does not +silently accept the former option-based API or file format. Notarius's own +version 5 configuration contract and loader decision in +[ADR-0018](0018-adopt-configkit.md) remain in force; this decision concerns +PromptKit definitions and source composition. + +The workload-oriented profile identity and D&D fallback ownership in +[ADR-0010](0010-workload-oriented-llm-profile-defaults.md) remain in force. +Only that ADR's former claim that a matching operator profile completely +replaces the fallback has been superseded: native recursive overlays retain +unspecified inherited fields. + +## Alternatives considered + +- Keep the removed PromptKit option-based configuration API or add a + compatibility parser. This would fork the supported upstream contract and + preserve legacy format machinery. +- Merge profile documents inside Notarius or restore whole-profile + replacement. This would duplicate upstream composition behavior and make + inspection, preflight, and runtime easier to diverge. +- Re-read and hash source files for checkpoint identity. This could describe + mutable files rather than the definitions captured by the engine and would + duplicate upstream discovery and composition. +- Expose upstream inspection structs directly as CLI output. This would bind + the public Notarius contract to dependency-owned Go types. + +## Consequences + +PromptKit remains the owner of its native configuration semantics, while +Notarius owns which registered and operator sources are assembled and when an +engine is reconstructed. A valid later overlay can supersede an earlier +definition; validation applies to the final effective definitions rather than +as a separate audit of every superseded value. Configuration reports can +contain ordinary prompt and provider parameters, so operators own the +destination and retention of explicitly requested reports even though +credential-shaped fields are redacted. PromptKit upgrades require review of +the source adapter, captured identity, reports, and the pinned upstream +contracts. diff --git a/docs/integrations/pkg-promptkit.md b/docs/integrations/pkg-promptkit.md index 8c187a9a..2e511792 100644 --- a/docs/integrations/pkg-promptkit.md +++ b/docs/integrations/pkg-promptkit.md @@ -93,16 +93,24 @@ application fallback profile assets, and optional backend configuration through the same construction path for inspection and execution. PromptKit owns source precedence, strict group composition, and recursive field overlay: omitted operator fields retain lower-layer values, while supplied fields -replace them. A malformed final effective definition fails construction. The -operator configuration and deployment workflow are defined in +replace them. The application group retains registered filesystem roots; the +operator may be one file with native imports or a recursively discovered +directory. Imports resolve resources from their declaring file. Prompt and +profile definitions are validated after source composition, so a malformed +lower-layer value superseded by a valid overlay is not separately audited. A +malformed final effective definition fails construction. The operator +configuration and deployment workflow are defined in [Configuration](../config.md#promptkit-profiles) and [Operations](../operations.md#promptkit-profile-deployment). PromptKit and ConfigKit own `extends` inheritance under the [pinned format rules](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md). Notarius records the selected leaf identity and resolved target without parsing -or merging inheritance. An unset filesystem `api_key_env` is optional and may -reach the provider without authorization, which can result in a 401 or 403. +or merging inheritance. `api_key_env` names the environment variable; inspection +does not check whether it is set. PromptKit's effective `api_key_required` +metadata is supported in file-backed profiles and reported as metadata, not +as an availability check. Whether a usable credential reaches the provider +remains an execution concern. PromptKit accepts only the `developer`, `system`, `user`, and `assistant` text-chat roles after normalizing case and surrounding whitespace. @@ -129,6 +137,11 @@ PromptKit obtains its maintained OpenRouter and Rakestrawhome backend and profile catalogs from independently versioned transitive modules. Notarius does not import or register either catalog; PromptKit retains catalog source, identity, precedence, credential, and capacity ownership. +The selected integration modules are +[`promptkit-backend-openrouter/v3`](../oss/attribution.md#promptkit-openrouter-catalog) +and +[`promptkit-backend-rakestrawhome/v3`](../oss/attribution.md#promptkit-rakestrawhome-catalog), +both pinned at `v3.20260930.0` for this compatibility boundary. External catalog profile names and settings can change between dependency releases, including removal without aliases. Review directly selected IDs and @@ -153,6 +166,16 @@ implicitly supported. A dependency upgrade requires reviewing the adapter, profile-source construction, and this compatibility statement against the pinned upstream documentation. +Captured inspection describes one engine's immutable effective view. Source +edits require constructing a new engine to become visible. Effective and +explanation reports retain PromptKit's redacted configuration and source +history contracts; the profile command is a separate Notarius-owned projection +of copied, provider-neutral metadata. None of these views reads credential +values or predicts provider defaults. See the public [CLI reference](../cli.md#config-effective), +[explanation](../cli.md#config-explain), and +[profile](../cli.md#config-profile) entries for their syntax and output +contracts. + ## Notarius Ownership [LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts diff --git a/docs/internal/cli.md b/docs/internal/cli.md index a6a0e084..7e73c64b 100644 --- a/docs/internal/cli.md +++ b/docs/internal/cli.md @@ -41,17 +41,26 @@ configured discovery and precedence contract is in [Configuration](../config.md), while the loading and resolution mechanics are in [Configuration Internals](configuration.md). -`config effective` and `config explain` only render the loaded ConfigKit view; -they do not resolve pipelines or construct providers. Configuration validation without a selected pipeline checks structural +`config effective` and `config explain` render the loaded ConfigKit view by +default and do not resolve pipelines or construct providers. With explicit +`--scope promptkit`, they construct the PromptKit inspection engine from the +same captured source groups used by execution. Effective output is PromptKit's +redacted YAML; explanation output shows its source history. `config profile` +uses that engine to emit one neutral Notarius-owned JSON summary for the exact +requested profile ID. These inspection commands load no run pipeline, prepare +no request, resolve no credential value, and make no provider call. Ordinary +prompt and provider-parameter content is not generally secret-free even though +credential-shaped provider parameter keys are redacted. + +Configuration validation without a selected pipeline checks structural configuration only. Validation with a selected pipeline also builds the effective catalog, resolves the pipeline, and verifies every explicit effective PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator -profiles are inspected -against the configured PromptKit source and backend registrations without -loading a prompt or performing generation, so an unknown or invalid profile -fails before pipeline preparation. Credential availability remains an -execution-time concern. Pipeline listing validates configuration before -returning normalized, sorted identifiers. +profiles are inspected against the configured PromptKit source and backend +registrations without loading a prompt or performing generation, so an unknown +or invalid final effective profile fails before pipeline preparation. +Credential availability remains an execution-time concern. Pipeline listing +validates configuration before returning normalized, sorted identifiers. ## Production Composition @@ -62,14 +71,17 @@ catalog used for resolution and the concrete constructors used for preparation. Tests may provide a catalog or registries instead; production code must not silently merge an injected partial catalog with production registrations. -The production LLM factory builds one PromptKit-backed client from the resolved -**promptkit.profile_dir** or **promptkit.profile_file** source, attaches the -profile-provenance recorder, creates one scheduler from the effective global -LLM limit, and wraps the client before it reaches modules. Registration and LLM -construction errors are returned before a pipeline is prepared. Configuration -field definitions remain in [Configuration](../config.md#promptkit-profiles); -the D&D registrar's fallback profile assets and the adapter mechanics remain in -[LLM Runtime](llm.md). +The production LLM factory builds one PromptKit-backed client from the native +source groups assembled from PromptKit catalogs, registered application prompt +and fallback assets, the optional operator file or recursively selected +directory, and the optional local backend registration. The same construction +path serves runtime, preflight, and configuration inspection. It attaches the +profile-provenance recorder, captures one immutable PromptKit configuration +identity, creates one scheduler from the effective global LLM limit, and wraps +the client before it reaches modules. Registration and LLM construction errors +are returned before a pipeline is prepared. Configuration fields remain in +[Configuration](../config.md#promptkit-profiles); source ownership, capture, +identity, and adapter mechanics remain in [LLM Runtime](llm.md). The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer preserves inherit, replace, and clear states across the composition boundary. diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 6c71401e..2c6489fa 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -83,35 +83,38 @@ retain an empty backend ID, which the published JSON omits. Successful completion responses and recorded profile manifests identify the adapter provider as `promptkit`. -The CLI's profile-inspection engine and the production adapter use the same -profile-source construction to apply the configured profile directory or file, -the optional registered fallback profile assets, and the optional conventional -`local` backend. Preflight therefore resolves the same profile sources and -backend membership as runtime without performing generation. Fallback assets -are mounted only when at least one source is registered. The production D&D -registrar contributes its `dnd-extraction` fallback, and the maintained D&D -prompts select that logical ID by default. PromptKit owns source precedence and -profile parsing and inheritance: an operator-provided matching profile takes precedence over a -fallback profile without Notarius merging either document. -When the registration is absent, a profile selecting `backend: local` fails -inspection instead of falling back to a built-in or endpoint-only target. +The CLI's profile-inspection engine and production adapter use the same source +assembly: PromptKit catalogs, one application group containing registered +prompts and fallback profiles, the selected operator file or directory, and +the optional final `local` backend registration. Execution, preflight, and +inspection therefore capture the same effective definitions. PromptKit and +ConfigKit compose recursively across groups while retaining each input's file +context; inputs within a group compose strictly. Operator imports are enabled +for the selected operator input, and imports retain their declaring file's +relative resource roots. PromptKit validates final effective definitions at +engine construction. It does not run a separate audit that rejects a malformed +definition superseded by a later overlay. The D&D registrar contributes its +`dnd-extraction` fallback, and maintained D&D prompts select that logical ID +by default. A profile selecting `backend: local` requires the optional local +registration. -Before execution, the adapter also contributes a non-secret checkpoint -fingerprint for the effective PromptKit profile source. It combines the -identity of PromptKit's compiled-in profile catalog with a deterministic digest -of every YAML profile in the configured profile directory, or of the configured -profile file, and a deterministic digest of the flattened fallback profile -assets. The fingerprint contains neither profile content nor source paths. It -covers inherited pipeline profiles, explicit binding profiles, and -prompt-selected defaults, so changing a model or other profile setting cannot -reuse checkpoints created under the -prior profile source. This cache identity is independent of durable -profile provenance: run manifests continue to list only profiles actually -observed during LLM calls. When the local backend is registered, a second -fingerprint hashes its trimmed endpoint behind a stable marker. Changing that -semantic execution target invalidates checkpoint reuse. The raw endpoint is not -stored in checkpoint identity, and the local concurrency limit is excluded -because it changes scheduling rather than execution semantics. +Each engine captures its definitions at construction. A later source-file edit +does not mutate an existing engine; constructing a new engine captures the +updated view. Before execution, the adapter computes one immutable, +versioned `promptkit_configuration` checkpoint identity from public captured +inspection APIs. The private digest covers captured effective configuration, +all effective profile targets and credential-policy metadata (including +provider parameters), and every effective prompt ID/version and captured +content hash. The identity also includes PromptKit and the selected OpenRouter +and Rakestrawhome catalog versions. Local-backend concurrency settings remain +excluded because they affect scheduling; its endpoint is represented only in +the private digest, never as checkpoint metadata. Neither the private digest +payload, raw sources, paths, endpoints, provider parameters, nor prompt bodies +are published. An existing engine retains its identity after files change; a +new engine gets the identity of its newly captured definitions. This replaces +the former live-file profile and local-endpoint fingerprints. Run manifests +remain independent and list only profile choices actually observed during LLM +calls. ## Shared Provider-Call Limit @@ -140,11 +143,16 @@ only by the Notarius scheduler. ## Prompt And Schema Assets An `AssetRegistry` collects prompt, schema, and optional fallback-profile -filesystems from production module families. It flattens registered roots into -the corresponding PromptKit filesystems and rejects invalid roots, unreadable -assets, duplicate paths, and missing prompt or schema files during preparation. -Fallback assets receive a safe content digest for checkpoint identity; raw -paths and bytes are never included. The framework’s `promptfs` helper combines +filesystems from production module families. It retains each registered root's +filesystem context when constructing PromptKit's application source group. +Schema files are supplied separately through the engine's schema filesystem; +PromptKit reads and validates a selected schema while preparing that request, +not while constructing the configuration snapshot. Prompt and fallback +definitions are validated after native source composition, and referenced +prompt content is captured with the engine. Checkpoint identity uses captured +prompt content hashes rather than rescanning live files or hashing only +content-file paths. Raw paths and bytes are never included. The framework’s +`promptfs` helper combines module-selected prompt files with reusable domain fragments without making the framework depend on D&D content. @@ -294,6 +302,10 @@ When debug recording is enabled, the pipeline decorates the shared client. The wrapper records prepared prompt and response material, timing, selected profile and backend, effective model parameters, and call identifiers in the run’s debug bundle, including material available from a failed structured completion. +The profile-inspection CLI projects copied values into its Notarius-owned +neutral JSON DTO; effective configuration and explanation remain PromptKit's +redacted YAML and source-history views. Credential names and `APIKeyRequired` +metadata are configuration metadata, not credential-availability checks. Effective parameters use PromptKit's stable lower-case JSON field names and may include `backend_id`. For a successful completion, a debug-write failure is surfaced; when the completion already failed, its call error remains the diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 084897f4..5edf4661 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -705,6 +705,17 @@ Validation: documentation/link review, scoped stale-term search, and `git diff --check`. Reuse code/example checks when unchanged; no new scanner or paid inference is required for prose. +> Completion: complete. Updated the internal LLM, CLI, and PromptKit integration +> guides to describe unified native sources, final-effective validation, +> captured engine lifetime, schema preparation timing, immutable configuration +> identity, inspection output ownership, and credential metadata. Added +> ADR-0019 and linked it from ADR-0010 while preserving the workload and fallback +> ownership decision; ADR-0018 remains unchanged. Scoped stale-claim review and +> local/pinned source and license-link review are complete; the browser could +> not access the private Gitea links, so pinned attribution targets and license +> identifiers were cross-checked against the locally cached pinned modules. +> `git diff --check` passes. + Prompt size: focused internal/integration references and one short ADR. ## Stage 15 — Verify Completeness And Remove Only Obsolete Machinery