base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document PromptKit catalogs and smoke testing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-17 19:15:58 +00:00
parent 8d1f33a40e
commit e653b7d6ec
15 changed files with 386 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -15,8 +15,8 @@ constructing them, and returns outcomes without exiting the process. Its
| Module | Package | Provides |
| --- | --- | --- |
| `gitea.maximumdirect.net/eric/CommandKit` | `commandkit` | Typed commands, bindings, execution, outcomes, help, and version. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | `ckconfig` | ConfigKit-backed `config` scopes and inspection commands. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | `ckprompt` | PromptKit engine configuration/comparison, prompt/profile catalogs, named inspection, requirements discovery, and concrete request validation. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | `ckconfig` | ConfigKit-backed `config` scopes, including configuration source evidence. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | `ckprompt` | PromptKit engine configuration/comparison, five catalogs, named inspection, requirements discovery, concrete request validation, and native smoke tests. |
Importing an integration does not install commands. Pass its declaration to
`commandkit.New` or `App.Register` to install it. The integrations depend on the

394
docs/cli.md
View File

@@ -175,14 +175,36 @@ sources:
kind: file
underlay_index: 0
imported_from: null
destination: /server
placement: mapping
destination_unresolved: false
version_check:
path: /version
version:
present: true
null: false
text: "1"
location:
source_id: source-1
line: 2
column: 1
redacted: false
```
Each source includes `id`, `label`, `path`, `layer`, `kind`, `underlay_index`,
and `imported_from`. The import location is null when absent; otherwise it has
`source_id`, `line`, and `column`. Owning values and source order are preserved,
including empty fields, zero coordinates, and synthetic sources without paths.
An empty inventory is `sources: []`. Text output begins with the scope heading
and then prints the same source fields, quoting labels and paths.
and `imported_from`, plus the native destination pointer, placement, unresolved
destination flag, and nullable `version_check`. Placement is `mapping` or
`sequence_item`. The import location is null when absent; otherwise it has
`source_id`, `line`, and `column`. Version evidence preserves `path`, `present`,
`null`, `text`, optional source `location`, and `redacted` as returned by
ConfigKit. Owning values and source order are preserved, including empty fields,
zero coordinates, false and null observations, and synthetic sources without
paths. An absent version check is null. An empty inventory is `sources: []`.
Text output begins with the scope heading and prints the same source fields,
quoting labels and paths. These fields project the selected ConfigKit source
records; CommandKit does not reload files or rerun version checks. See the
[ConfigKit v0.8.0 inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/137769f3f18a370c68444242fb5d88e620e307e5/inspection.go)
for native evidence definitions.
### Explain a configuration path
@@ -306,38 +328,35 @@ existing incomplete-report delivery outcome and can leave partial output.
## PromptKit definition inspection
Applications can install prompt and profile inspection independently with
`ckprompt.Command(suite)`, passing the resulting declaration to core. The
`ckprompt.FromEngine(loadEngine)` adapter supplies callbacks that acquire an
engine only after input checks pass and inspection is reached. The loader
receives invocation context, selection, input, and state; its implementation
controls whether an engine is constructed or reused. Direct suite callbacks
retain their own implementation and lifecycle.
`ckprompt.Command(suite)`. The `ckprompt.FromEngine(loadEngine)` adapter
supplies these callbacks alongside native requirements discovery and the five
catalogs. Its loader receives invocation context, configuration selection,
typed input, and shared state. Direct suite callbacks retain their own
implementation and lifecycle.
CommandKit's inspection path does not execute prompts, contact model providers,
or resolve provider credential values. Application acquisition and direct
callback effects remain application-owned. Prompt definitions and profile
resolution remain owned by
[PromptKit's inspection API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/engine.go).
Prompt inspection uses an ID only. PromptKit returns any descriptive version
metadata as an optional report field; there is no `--version` flag for selecting
a prompt version. The prompt format and native selection behavior remain owned by
[PromptKit v0.16.0](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md).
```text
example promptkit show prompt customer/summary --version 2 --format yaml
example promptkit show prompt customer/summary --format yaml
example promptkit show prompt customer/summary
example promptkit show profile customer-support --format yaml
```
Both commands require one ID operand and accept `--config`, `--overlay`, and
`--format text|yaml` (default `text`). Prompt inspection also accepts singleton
`--version`, defaulting to the empty string. PromptKit owns version selection:
an omitted version works for a single-version ID, while an ambiguous ID is an
operational error. These commands have no built-in `--scope` selection; their
namespace is `promptkit`.
`--format text|yaml` (default `text`). These commands have no built-in
`--scope` selection; their namespace is `promptkit`.
Prompt YAML has these fields. Input order is preserved and an empty input list is
`[]`. The prompt hash is an opaque owning-library equality value.
Prompt YAML preserves native input order and uses an optional `version` field.
The key is omitted when PromptKit returns no version and is retained when the
version is present, including an empty string. The prompt hash is an opaque
owning-library equality value.
```yaml
prompt_id: customer/summary
prompt_version: "2"
version: "2"
prompt_hash: opaque-value
default_profile_id: customer-support
inputs:
@@ -376,47 +395,41 @@ api_key_required: true
```
Text uses labeled fields and includes nested `extra_params` values. A missing ID
is a usage error. Blank, unknown, and ambiguous definition selections reach the
owning inspection API and produce operational results with supported public
error identity. Cancellation is also operational: standard handlers check the
caller's context at inspection boundaries, and PromptKit can return cancellation
from its own inspection methods. Supported cancellation identity is preserved.
is a usage error. Definition selection reaches the owning inspection API and
preserves supported error identity and cancellation.
## PromptKit requirements
When `Suite.InspectRequirements` is present,
`ckprompt.Command(suite)` installs `promptkit requirements show`. It accepts one
required prompt ID and uses PromptKit's native selection and discovery rules:
When `Suite.InspectRequirements` is present, `ckprompt.Command(suite)` installs
`promptkit requirements show`. It accepts one prompt ID and uses PromptKit's
native selection and discovery rules:
```text
example promptkit requirements show briefing --campaign spring
example promptkit requirements show briefing --version 2 --profile editorial --var mode=full
example promptkit requirements show briefing --profile editorial --var mode=full
example promptkit requirements show briefing --session-id session-42 --var note= --format yaml
```
The command accepts these flags:
The command accepts:
| Flag | Values | Default | Meaning |
| --- | --- | --- | --- |
| `--config` | path string | omitted | Select the configuration input; explicit presence is retained. |
| `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. |
| Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. |
| `--version` | string | empty | Pass the prompt version to PromptKit; empty uses its native version-selection rule. |
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native profile-selection rule. |
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. |
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. |
| `--format` | `yaml` | `yaml` | Select the requirements report format. Other formats are usage errors for this operation. |
The `promptkit` namespace identifies the PromptKit operation, so there is no
built-in `--scope` flag. The version, profile, and session fields use native
empty-string semantics: an explicit empty string has the same meaning as
omission. PromptKit owns their interpretation, including default version and
profile selection, session context, and selection errors. Requirements
inspection does not require a profile to be execution-ready. See the pinned
[PromptKit requirements API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/143c6e9b3a675872117b0471e2ff8bcb25055a54/requirements.go)
for the owning discovery contract. The built-in flags other than `--var` are
singleton bindings; when repeated, the last value is used. Suite selectors retain
their declared binding behavior.
The `promptkit` namespace identifies the operation, so there is no built-in
`--scope` flag. An explicitly empty profile or session value retains PromptKit's
native empty-string meaning. Requirements inspection does not require a profile
to be execution-ready. See the
[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md)
for native configuration, selection, and requirements behavior. The built-in
flags other than `--var` are singleton bindings; when repeated, the last value
is used. Suite selectors retain their declared binding behavior.
Each `--var` occurrence must contain `=`. CommandKit splits at the first
separator and preserves the exact key and remaining value. Empty keys and
@@ -424,20 +437,21 @@ values, whitespace, commas, and additional `=` characters are retained. All
occurrences are parsed; duplicate keys use the last value, while any malformed
occurrence is a usage error even if a later assignment is valid.
The successful report is YAML with selected identities, native completeness,
and the input, variable, and issue inventories. False values and empty
inventories are retained. Input metadata preserves null versus an empty string.
Variable entries contain `name`, `kind`, `supplied`, and `explanation`.
`supplied` reports native exact-key presence, including an explicitly empty
value, independently of `kind` and `complete`. Explanations and issue descriptions
are upstream-owned human-readable summaries, not machine-readable details.
The YAML report contains selected identities, native completeness, and input,
variable, and issue inventories. False values and empty inventories are
retained. Input metadata preserves null versus an empty string. Variable entries
contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native
exact-key presence, including an explicitly empty value, independently of
`kind` and `complete`. Explanations and issue descriptions are upstream-owned
human-readable summaries, not machine-readable details. The optional `version`
key is omitted when absent and retained when supplied by PromptKit.
For a prompt with the template `{{if .mode}}{{input "notes"}}{{end}}` and no
input declarations, omitting `mode` can produce this partial report:
input declarations, omitting `mode` can produce a partial report:
```yaml
prompt_id: briefing
prompt_version: "2"
version: "2"
profile_id: editorial
complete: false
inputs:
@@ -459,28 +473,24 @@ issues:
description: A missing variable prevents selecting a dependency-bearing branch.
```
PromptKit owns the inventory, classifications, explanations, completeness,
and structured issues. Completeness means discovery covered the relevant
dependencies; it does not mean inputs or variables are supplied, artifacts are
valid, credentials are available, or execution is ready. A complete report can
contain unsupplied required variables, and an incomplete report can still be
useful. Successfully acquired and delivered partial reports return status `0`;
structured issues remain in YAML and do not independently fail the command.
Provider errors, native selection errors, cancellation, and report delivery
failures use the operational path (status `1`) with supported cause identity.
Syntax, operand, variable-assignment, and format errors use usage status `2`.
PromptKit owns the inventory, classifications, explanations, completeness, and
structured issues. Successfully acquired and delivered partial reports return
status `0`; structured issues do not independently fail the command. Provider
errors, native selection errors, cancellation, and report delivery failures use
the operational path (status `1`) with supported cause identity. Syntax,
operand, variable-assignment, and format errors use usage status `2`.
### Concrete request validation
When `Suite.ValidateRequest` is present, `ckprompt.Command(suite)` also installs
`promptkit requirements validate`. It can be installed without discovery, and
discovery can remain installed without validation. The command accepts one
required prompt ID and passes a concrete request to PromptKit's native
prompt ID and passes a concrete request to PromptKit's native
`Engine.ValidateRequest` preparation path:
```text
example promptkit requirements validate briefing --campaign spring --input topic=topic.txt
example promptkit requirements validate briefing --version 2 --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt
example promptkit requirements validate briefing --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt
```
The command accepts:
@@ -490,7 +500,6 @@ The command accepts:
| `--config` | path string | omitted | Select the configuration input; explicit presence is retained. |
| `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. |
| Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. |
| `--version` | string | empty | Pass the prompt version to PromptKit. |
| `--profile` | string | empty | Pass the profile ID to PromptKit. |
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. |
@@ -498,24 +507,17 @@ The command accepts:
| `--format` | `yaml` | `yaml` | Select the success confirmation format. Other formats are usage errors. |
The built-in flags other than repeatable `--var` and `--input` are singleton
bindings; when repeated, the last value is used. Suite selectors retain their
declared binding behavior. Explicit empty `--config` or `--overlay` values keep
their supplied-presence information.
bindings; when repeated, the last value is used. Explicit empty `--config` or
`--overlay` values keep their supplied-presence information. The profile and
session fields use native empty-string semantics. PromptKit owns their
interpretation and selection errors.
The version, profile, and session fields use native empty-string semantics:
an explicit empty string has the same meaning as omission. PromptKit owns their
interpretation and selection errors. The `promptkit` namespace identifies the
operation, so there is no built-in `--scope` flag.
Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at
the first separator and preserves the exact key/name and remaining value/path.
Empty variable keys and values are passed to PromptKit. Empty input names and
paths are also passed to native validation. File paths are translated to
`promptkit.File(PATH)` references without reading, normalizing, expanding, or
rebasing them during argument binding. Duplicate keys or names use the last
value, but all occurrences are parsed, so any malformed occurrence is a usage
error even if a later assignment is valid. Syntax and path values after shell
processing are otherwise preserved.
Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at the
first separator and preserves the exact key/name and remaining value/path.
Empty variable keys and values and empty input names and paths are passed to
native validation. File paths become `promptkit.File(PATH)` references without
being read, normalized, expanded, or rebased during argument binding. Duplicate
keys or names use the last value, but every occurrence is parsed.
On success, the YAML confirmation contains only the requested prompt ID and
`valid: true`:
@@ -525,46 +527,64 @@ prompt_id: briefing
valid: true
```
Validation delegates concrete request preparation to PromptKit. Depending on
the request and configuration, native preparation may resolve prompt/profile
resources, load artifact and schema resources, render messages and session
context, and perform applicable configuration and credential checks. Native
errors are returned as encountered rather than as an exhaustive inventory.
Validation does not make a model call or retain a prepared execution snapshot.
A successful report means preparation passed for this invocation; it does not
certify provider acceptance or guarantee that later execution will succeed.
See PromptKit's [v0.15.0 `Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/engine.go)
for the native contract.
Validation delegates concrete request preparation to PromptKit. Native
preparation may resolve prompt/profile resources, load artifact and schema
resources, render messages and session context, and perform applicable
configuration and credential checks. It does not make a model call or retain a
prepared execution snapshot. A successful report describes this preparation
invocation; it does not certify provider acceptance or guarantee later
execution. See PromptKit's
[`Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go).
Builder, engine-loading, and native validation errors use the operational path
(status `1`) with supported error identity and acquired diagnostics retained.
Malformed syntax, missing operands, and unsupported formats use usage status
`2`. Input-check failures keep their existing shared handling. Successful
diagnostics and report-delivery failures follow the common delivery contract.
`2`. Successful diagnostics and report-delivery failures follow the common
delivery contract.
### PromptKit catalogs
An installed `ckprompt.Command(suite)` exposes the catalog commands supplied by
the suite:
An installed `ckprompt.Command(suite)` exposes each catalog whose callback is
present:
```text
example promptkit list prompts
example promptkit list profiles --config settings.yml --campaign spring
example promptkit list backends --namespace team
example promptkit list namespaces --kind prompt --kind profile
example promptkit list collections
```
Each command accepts `--config`, `--overlay`, and applicable selectors from
`Suite.Selectors`. It takes no ID or version operand and has no `--version` or
`--scope` selector. `--format` defaults to `yaml`; `--format yaml` is accepted,
and other formats are usage errors.
All catalogs accept `--config`, `--overlay`, applicable flags from
`Suite.Selectors`, and `--format yaml` (the default). Other formats are usage
errors. Prompt, profile, and backend listings accept namespace selection:
Each invocation writes one bare YAML sequence. Prompt entries have explicit
string `id` and `version` fields; profile entries are strings. Provider order and
exact identity values are preserved. For example:
| Binding | Effect |
| --- | --- |
| omitted `--namespace` and `--all-namespaces` | Forward PromptKit's native default namespace filter. |
| `--namespace team` | Forward the named namespace filter. |
| `--namespace ""` | Select the unqualified namespace. |
| `--all-namespaces` | Forward the all-namespaces filter. |
| `--all-namespaces=false` | Do not select an override; a supplied namespace still applies, otherwise native default is used. |
The namespace binding is a singleton and its last occurrence is used.
Supplying a namespace while `--all-namespaces=true` is active is a usage error
before provider work. Namespace inventory accepts repeatable `--kind KIND`;
omitting `--kind` forwards PromptKit's native default kind selection. Repeated
values are forwarded in order. Collections have no namespace or kind filter.
Each operation writes one bare YAML sequence, retaining provider order and
native identities:
- Prompt entries are `{id, version?}`. The optional `version` key is omitted
when absent and retained for a present empty or nonempty value.
- Profiles, backends, and namespaces are strings.
- Collection entries are `{namespace, version?}`, where `version` projects
PromptKit's optional release version.
- Nil and allocated-empty inventories both render as `[]`.
```yaml
- id: briefing
version: "1"
- id: briefing
- id: team/briefing
version: "2"
```
@@ -573,11 +593,155 @@ exact identity values are preserved. For example:
- spring
```
Successful nil and allocated-empty inventories both render as `[]`. Catalog
membership and ordering follow the selected provider, and a listed definition
is not a readiness check or execution result. PromptKit owns the meaning of its
native listing APIs; custom providers own their results. Applications retain
their own disclosure choices for catalog data.
A catalog result is an inventory, not a readiness check or execution result.
The owning provider controls membership and order. Namespace filters are passed
to PromptKit without local membership interpretation. See the
[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md)
for native resource and selection definitions.
### PromptKit smoke testing
When `Suite.SmokeTest` is present, `ckprompt.Command(suite)` installs
`promptkit test KIND`. The special kind `all` runs the four concrete operations
sequentially in this order: `reachability`, `authentication`, `text_inference`,
and `structured_output`. Both forms use the same single-operation callback;
each aggregate call changes the seed kind before invoking it. The command
accepts:
```text
example promptkit test text_inference --campaign spring --prompt briefing --profile spring
example promptkit test all --campaign spring --prompt briefing --profile spring
example promptkit test authentication --backend local
```
| Flag | Values | Default | Meaning |
| --- | --- | --- | --- |
| `--config`, `--overlay` | path strings | omitted | Select PromptKit configuration; supplied presence is retained. |
| Suite selectors | application-defined | per selector | Captured flags from `Suite.Selectors`. |
| `--backend` | string | empty | Native backend ID. |
| `--prompt` | string | empty | Native prompt ID. |
| `--profile` | string | empty | Native profile ID. |
| `--endpoint` | string | omitted | Override the execution endpoint. |
| `--model` | string | omitted | Override the execution model. |
| `--temperature` | float | omitted | Override execution temperature. |
| `--max-tokens` | integer | omitted | Override execution maximum tokens. |
| `--top-p` | float | omitted | Override execution top-p. |
| `--timeout-seconds` | integer | omitted | Override the execution timeout. |
| `--service-tier` | string | omitted | Override the execution service tier. |
| `--reasoning-effort` | string | omitted | Override execution reasoning effort. |
| `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. |
| `--transient-retries` | integer | omitted | Set the native inference transient retry count. |
| `--format` | `yaml` | `yaml` | Select the report format. Other formats are usage errors. |
The execution flags populate PromptKit's native execution override. The
`--api-key-env` value is a variable name, not a credential value. Application
selectors, loader policy, credentials, and client wiring remain application
owned. The `WithSmokeTesting` helper accepts an optional typed builder. A nil
builder forwards the seed; a supplied builder's complete native request is
authoritative. The helper retains a native report when PromptKit returns both a
report and an error.
A single report projects the native smoke report into YAML. It preserves native
false, unknown/null, and incomplete values and does not add a readiness score.
The report includes `kind`, `supported`, `applicable`, `completed`, `reason`,
`target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target
contains selected IDs, optional `version`, selection source, endpoint, effective
settings, and settings-presence flags. Attempt entries retain their own facts,
timing, failure, and token-usage evidence. Durations use Go duration strings;
token counts preserve `value`, `known`, and `complete`.
Here is a native projection as rendered by the command:
```yaml
kind: reachability
supported: false
applicable: null
completed: true
reason: http_diagnostics_unavailable
target:
backend_id: ""
profile_id: spring
prompt_id: briefing
version: "1"
default_source_backend_id: ""
selection_source: profile
endpoint: https://example.invalid/v1
effective_settings:
endpoint: https://example.invalid/v1
model: spring-model
temperature: 0
max_tokens: 0
top_p: 0
timeout_seconds: 600
service_tier: ""
reasoning_effort: ""
api_key_env: ""
extra_params:
campaign: spring
settings_presence:
temperature: false
max_tokens: false
top_p: false
timeout_seconds: false
facts:
credentials_supplied: null
endpoint_reachable: null
credentials_accepted: null
response_received: null
generation_completed: null
text_returned: null
expected_text_matched: null
structured_output_requested: null
valid_json: null
schema_matched: null
timing:
started_at: 2026-10-17T19:05:36Z
duration: 35.685µs
capacity_wait: null
request_duration: null
retry_delay: null
evaluation_duration: null
attempts: []
usage:
prompt_tokens: {value: 0, known: true, complete: true}
completion_tokens: {value: 0, known: true, complete: true}
total_tokens: {value: 0, known: true, complete: true}
cached_tokens: {value: 0, known: true, complete: true}
cache_write_tokens: {value: 0, known: true, complete: true}
failure: ""
```
Aggregate output is one YAML mapping with ordered `tests` entries. Each entry
has selected `kind`, nullable `report`, and optional `error` text. The inner
report keeps its native `kind`. This cancellation example shows two attempted
operations with no reports; later kinds are absent because cancellation stops
the sequence:
```yaml
tests:
- kind: reachability
report: null
error: "reachability operation failed"
- kind: authentication
report: null
error: "context canceled"
```
A successful delivered single operation returns status `0`. For `all`, all
four callbacks must return without error for status `0`. Native false, unknown,
unsupported, or incomplete observations do not independently change the
outcome. Operation errors use operational status `1`; aggregate runs continue
after individual errors, retain their reports and ordered error text, and join
causes for supported `errors.Is`/`errors.As` access. Cancellation stops later
operations and preserves results already acquired. Report encoding or delivery
failures use the common incomplete-report delivery outcome and retain operation
causes where applicable. Syntax, operand, unsupported format, and conflicting
namespace selection are usage status `2`.
Native configuration, selection, and diagnostic meanings remain owned by
PromptKit. See the [v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md),
[smoke API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/smoke_diagnostics.go),
and [engine API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go).
## Outcome status

51
docs/consumers/guide.md
View File

@@ -11,7 +11,7 @@ through `commandkit.New` or `App.Register`.
| --- | --- |
| `gitea.maximumdirect.net/eric/CommandKit` | The application needs typed command declarations and common CLI behavior. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | The application wants ConfigKit-backed configuration scopes and reports. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | The application wants PromptKit engine configuration/comparison, prompt/profile catalogs, named inspection, requirements discovery, or concrete request validation. |
| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | The application wants PromptKit engine configuration/comparison, five catalogs, named inspection, requirements discovery, request validation, or native smoke testing. |
The two integrations are independent optional imports. `ckprompt` uses the
public scope and error contracts from `ckconfig`; importing it does not add the
@@ -22,7 +22,7 @@ core-only, ConfigKit-only, and composed applications.
## Go compatibility
All three modules declare a consumer minimum of Go 1.26.0. The root module also
selects ReleaseKit v0.4.0 as a Go tool, which makes that tool's module graph part
selects ReleaseKit v0.4.2 as a Go tool, which makes that tool's module graph part
of the root consumer module requirements even though production packages do not
import ReleaseKit. The [release policy](../policy/release.md#consumer-minimum-and-release-toolchain)
defines the support assumption. Maintainers use Go 1.26.8 for validation and
@@ -126,17 +126,20 @@ engine's captured configuration. Add that scope to a `ckconfig.Suite` to expose
it under `config` with validate/show/sources/explain and native diff operations;
it does not install PromptKit catalog, definition, or requirements commands.
`ckprompt.FromEngine(loadEngine)` supplies five callbacks: prompt and profile
catalogs, named prompt and profile inspection, and native requirements
discovery. Concrete request validation is a sixth, independently optional
capability and is added explicitly with `ckprompt.WithRequestValidation`.
`ckprompt.Command(suite)` creates only operations whose callbacks are present;
discovery and validation share `promptkit requirements` but can be installed
independently. A composed suite can keep the standard five and add validation:
`ckprompt.FromEngine(loadEngine)` supplies eight callbacks: prompt, profile,
backend, namespace, and collection catalogs; named prompt and profile
inspection; and native requirements discovery. Concrete request validation and
smoke testing are separate capabilities, added explicitly with
`ckprompt.WithRequestValidation` and `ckprompt.WithSmokeTesting`. The suite has
ten independently optional callbacks in total. `ckprompt.Command(suite)` creates
only operations whose callbacks are present; discovery, validation, and smoke
testing can be installed independently. A composed suite can enable both
additional operations:
```go
suite := ckprompt.FromEngine(loadEngine)
suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest)
suite = ckprompt.WithSmokeTesting(suite, loadEngine, nil)
suite.NewInput = newInput
suite.NewState = newState
suite.Selectors = selectors
@@ -165,8 +168,9 @@ absent, so explicitly supplied empty values stay meaningful. Applications that
validate and then execute a request are encouraged to share their request
construction logic between those paths.
All six suite capabilities are independently optional. For catalogs without
named or requirements inspection, narrow the suite before command capture:
The eight catalog and inspection callbacks and the two explicit additions are
independently optional. For catalogs without named or requirements inspection,
narrow the suite before command capture:
```go
definitions := ckprompt.FromEngine(loadEngine)
@@ -176,7 +180,7 @@ definitions.InspectRequirements = nil
definitionCommand := ckprompt.Command(definitions)
```
Each selected inspection, requirements, validation, or catalog operation
Each selected inspection, requirements, validation, smoke, or catalog operation
prepares one input, applies its captured selectors, checks that input, and
initializes one state value when those steps are reached. The bundled validation
adapter delegates concrete request preparation to PromptKit and carries its
@@ -195,13 +199,19 @@ PromptKit preparation, which can resolve prompt/profile resources, load
artifacts and schemas, render messages, and perform applicable configuration
and credential checks. It does not make a model call or certify that a later
execution will succeed. Application loaders and direct callbacks retain their
own effects. The [CLI reference](../cli.md#promptkit-definition-inspection)
and [requirements contract](../cli.md#promptkit-requirements) own operator
syntax and reports.
own effects. Smoke testing explicitly invokes one native PromptKit diagnostic
or all four in order. Native inference checks use the configured public
`promptkit.LLMClient`; applications retain ownership of client and probe
configuration. A builder may customize the seeded native request, and its
complete return value is authoritative. The [CLI reference](../cli.md#promptkit-smoke-testing),
[definition inspection](../cli.md#promptkit-definition-inspection), and
[requirements contract](../cli.md#promptkit-requirements) own operator syntax
and reports.
The [composed example](../../integrations/ckprompt/examples/composed/README.md)
combines app and PromptKit config scopes, comparison and catalogs, named and
requirements inspection, and an application-owned `session restore` command.
requirements inspection, request validation, smoke testing, and an
application-owned `session restore` command.
## Outcomes and ownership
@@ -214,7 +224,8 @@ process.
Go declarations and GoDoc own exact exported contracts. ConfigKit owns
configuration decoding, validation, and provenance. PromptKit owns prompt
formats, profile resolution, and inspection details. See the
[ConfigKit inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/inspection.go),
[PromptKit engine methods](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/engine.go),
[configuration inspection](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/configuration_inspection.go),
and [inspection types](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/types.go).
[ConfigKit v0.8.0 inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/137769f3f18a370c68444242fb5d88e620e307e5/inspection.go),
[PromptKit engine methods](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go),
[configuration inspection](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/configuration_inspection.go),
[smoke diagnostics](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/smoke_diagnostics.go),
and [PromptKit v0.16.0 consumer guidance](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md).

8
docs/development.md
View File

@@ -131,7 +131,8 @@ or publishing.
## Current public examples
The maintained consumers compile with their owning module suites and run without
external services:
external services. The PromptKit smoke examples use local fixtures and the
example's configured client:
```sh
go run ./examples/core workspace greet --count=0 --tag=first --tag second
@@ -142,11 +143,16 @@ go run ./examples/core workspace greet --count=0 --tag=first --tag second
(cd integrations/ckprompt/examples/composed && go run . config diff --scope promptkit --config fixtures/before.yml --against-config fixtures/after.yml --campaign spring)
(cd integrations/ckprompt/examples/composed && go run . promptkit list prompts --config fixtures/before.yml --campaign spring)
(cd integrations/ckprompt/examples/composed && go run . promptkit list profiles --config fixtures/after.yml --campaign autumn --format yaml)
(cd integrations/ckprompt/examples/composed && go run . promptkit list backends --namespace team)
(cd integrations/ckprompt/examples/composed && go run . promptkit list namespaces --kind prompt --kind profile)
(cd integrations/ckprompt/examples/composed && go run . promptkit list collections)
(cd integrations/ckprompt/examples/composed && go run . promptkit show prompt briefing --campaign spring --format yaml)
(cd integrations/ckprompt/examples/composed && go run . promptkit requirements show briefing --campaign spring)
(cd integrations/ckprompt/examples/composed && go run . promptkit requirements show briefing --campaign spring --var mode=full --format yaml)
(cd integrations/ckprompt/examples/composed && go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt)
(cd integrations/ckprompt/examples/composed && go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt --var mode=full --input notes=fixtures/notes.txt --format yaml)
(cd integrations/ckprompt/examples/composed && go run . promptkit test reachability --campaign spring --prompt briefing --profile spring)
(cd integrations/ckprompt/examples/composed && go run . promptkit test all --campaign spring --prompt briefing --profile spring)
(cd integrations/ckprompt/examples/composed && go run . session restore session-42 --campaign spring)
```

16
docs/internal/overview.md
View File

@@ -47,10 +47,12 @@ initialized input, while one suite-owned state value is shared by an invocation'
acquisitions or comparison callback.
`integrations/ckprompt` owns PromptKit engine loading, configuration-scope
adaptation and native comparison, plus optional prompt/profile catalogs, named
inspection, native requirements discovery, and concrete request validation.
Validation delegates request preparation to PromptKit and is independently
opted into a suite. It uses public core and ckconfig contracts plus PromptKit's
adaptation and native comparison, plus optional prompt/profile/backend/namespace/
collection catalogs, named inspection, native requirements discovery,
concrete request validation, and native smoke testing. Validation and smoke
testing delegate to PromptKit and are independently opted into a suite. Smoke
operations use the application's native engine and its configured client or
probes. The integration uses public core and ckconfig contracts plus PromptKit's
public APIs. It does not add a configuration family when its declaration is
installed; callers compose `ckprompt.ConfigScope` with `ckconfig.Command`
separately when they want configuration operations alongside PromptKit
@@ -70,8 +72,10 @@ graph isolation.
ConfigKit owns configuration decoding, validation, source history, and path
explanations. PromptKit owns its captured engine configuration, prompt
definitions, profile inheritance, named inspection, requirements discovery,
and concrete request validation semantics. CommandKit adapts those public
contracts without maintaining parallel schemas or invoking model generation.
and concrete request validation, and native smoke-test semantics. CommandKit
adapts those public contracts without maintaining parallel schemas. Explicit
native smoke operations may use the configured client or probes; other
inspection, catalog, and request-validation paths do not make model calls.
Native request validation may load resources and artifacts, render messages,
and perform applicable configuration or credential checks through PromptKit.
Application loaders and domain behavior remain application owned.

3
docs/roadmap/deferred.md
View File

@@ -33,7 +33,8 @@ operator or consumer need and an appropriate ownership/behavior decision:
- JSON reports, shell completion, and output-file management or reloadable export.
- Configuration mutation, generic value assignment, and exhaustive field-origin
inventories.
- Credential diagnostics and live provider probes.
- Credential diagnostics beyond PromptKit's native smoke checks and additional
live provider probes.
- Additional component identity registration and domain artifact catalogs.
- Additional library families, including possible FeedKit integration; families
can define suitable verbs without requiring comparison or configuration scopes.

12
docs/roadmap/implementation.md
View File

@@ -1,6 +1,6 @@
# Dependency upgrades, discovery, and smoke testing implementation plan
Status: Stages 1–16 complete; Stage 17 is next.
Status: Stages 1–17 complete; Stage 18 is next.
## Authority and baseline
@@ -829,6 +829,16 @@ is required.
**Complete when:** current contracts describe implemented behavior under their
canonical owners and historical references remain correctly scoped.
**Completed:** 2026-10-17. GoDoc, module summaries, the README, CLI reference,
consumer guide, development commands, and internal inventory now describe the
adopted source evidence, eight `FromEngine` callbacks, five catalogs, optional
request validation, and native smoke behavior. Smoke examples and catalog
commands were run against the maintained composed consumer; `go doc` confirmed
the exported helper and request documentation. Local Markdown links/fragments,
stale upstream revision references, and whitespace checks passed. Deferred-work
scope and historical implementation-plan links were reconciled without changing
historical API examples or release notes.
## Stage 18: Verify the complete target and finish
1. Format all changed Go files and inspect the full diff for unrelated changes,

3
docs/roadmap/promptkit-validation.md
View File

@@ -3,7 +3,8 @@
Status: accepted adoption and feature roadmap; implementation complete.
Command placement, adapter composition, request construction, artifact syntax,
and initial presentation are settled. This document defines the target and
required work; the [implementation plan](implementation.md) sequences the work.
required work; the [implementation plan](https://gitea.maximumdirect.net/eric/CommandKit/src/commit/38b49f29fb3a8f22ff02528d2a0ed413e5c6ab1e/docs/roadmap/implementation.md)
sequences the historical implementation work.
## Purpose

3
docs/roadmap/remediation.md
View File

@@ -8,7 +8,8 @@ architectural redesign was identified as necessary.
Reviewed implementation commits after planning baseline `3218222`, through
`420483b2d4659862626422b0577e39bc0a4c1a2d`, against the
[accepted feature roadmap](promptkit-validation.md),
[implementation plan](implementation.md), applicable ADRs, and repository policy.
[implementation plan](https://gitea.maximumdirect.net/eric/CommandKit/src/commit/38b49f29fb3a8f22ff02528d2a0ed413e5c6ab1e/docs/roadmap/implementation.md),
applicable ADRs, and repository policy.
The review covered production changes, new tests, the maintained consumer,
dependency metadata/attribution, and updated documentation.

6
integrations/ckconfig/README.md
View File

@@ -27,8 +27,10 @@ scopes, selectors, input checks, and state shared by the selected acquisitions
or comparison in a config invocation. `ckconfig.FromConfigKit` adapts a typed
ConfigKit loader. The `config` command family exposes scope discovery and the
validate, show, sources, explain, and diff operations supported by the declared
scopes. Comparison returns ConfigKit's native records to the YAML report path;
the [CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#compare-configuration-selections)
scopes. Source reports include native destination, placement, unresolved
destination, and source version-check evidence. Comparison returns ConfigKit's
native records to the YAML report path; the
[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#compare-configuration-selections)
owns its operator details.
The maintained [simple example](examples/simple) shows a complete consumer.

4
integrations/ckconfig/doc.go
View File

@@ -1,4 +1,6 @@
// Package ckconfig provides CommandKit's optional ConfigKit integration. Its
// declarations install the standard config command family through the core
// commandkit package; importing ckconfig alone installs no commands.
// commandkit package. Source reports preserve ConfigKit's destination,
// placement, and version-check evidence; importing ckconfig alone installs no
// commands.
package ckconfig

25
integrations/ckprompt/README.md
View File

@@ -1,9 +1,9 @@
# CommandKit ckprompt
`ckprompt` is the optional PromptKit integration for CommandKit. It adapts
PromptKit engine configuration, native comparison, prompt/profile catalogs,
named inspection, and prompt requirements discovery. Importing it does not
install either command family.
PromptKit engine configuration, native comparison, five catalogs, named
inspection, requirements discovery and validation, and native smoke tests.
Importing it does not install either command family.
## Module
@@ -19,14 +19,17 @@ for callback, input, state, and acquisition contracts.
Add `ckprompt.ConfigScope(loadEngine)` to the scopes passed to
`ckconfig.Command` when the application wants PromptKit engine configuration
and native comparison in the `config` family. Install
`ckprompt.Command(ckprompt.FromEngine(loadEngine))` separately for prompt/profile
catalogs, named inspections, and requirements discovery. `FromEngine` supplies
all five PromptKit-backed callbacks; applications may narrow, wrap, or replace
them before capture or provide custom callbacks with `ckprompt.Suite`.
Requirements is independently optional. CommandKit-owned inspection and
listing code does not execute prompts or contact model providers; application
loaders and direct callbacks retain their own effects. See the
[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#promptkit-requirements)
`ckprompt.Command(ckprompt.FromEngine(loadEngine))` separately for catalogs,
named inspections, and requirements discovery. `FromEngine` supplies eight
PromptKit-backed callbacks: three inspections and five catalogs. Applications
may narrow, wrap, or replace them before capture or provide custom callbacks
with `ckprompt.Suite`. Add concrete request validation and smoke testing
explicitly with `WithRequestValidation` and `WithSmokeTesting`. Inspection,
requirements, and catalog operations do not execute prompts or contact model
providers; application loaders and callbacks retain their own effects. Smoke
testing invokes the selected native PromptKit operation; the
application retains ownership of client and configuration choices. See the
[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#promptkit-smoke-testing)
for operator commands and report semantics. The maintained
[composed example](examples/composed) shows complete application wiring.

22
integrations/ckprompt/adapters.go
View File

@@ -73,20 +73,20 @@ type RequirementsRequest struct {
// Factories are not invoked during construction, installation, help, or version,
// and need not run after failure.
type Suite[I, S any] struct {
// NewInput is called once when definition, requirements, validation, or
// catalog input preparation is reached, before applying selectors and
// NewInput is called once when inspection, requirements, validation, smoke,
// or catalog input preparation is reached, before applying selectors and
// calling CheckInput.
// If nil, input is the zero value of I, including nil for pointer types.
NewInput func() I
// NewState is called once when an inspection, validation, or catalog
// NewState is called once when an inspection, validation, smoke, or catalog
// callback is reached, after input checks pass. If nil, state is the zero
// value of S, including nil for pointer types.
NewState func() S
// Selectors declares the original typed flags accepted by definition,
// requirements, validation, and catalog commands.
// Selectors declares the original typed flags accepted by inspection,
// requirements, validation, smoke, and catalog commands.
Selectors []commandkit.Flag[I]
// CheckInput validates selected definition, requirements, validation, or catalog input
// before state creation.
// CheckInput validates selected inspection, requirements, validation, smoke,
// or catalog input before state creation.
CheckInput func(ckconfig.Selection, I) error
// InspectPrompt inspects the selected prompt after input and state preparation.
InspectPrompt func(context.Context, PromptRequest, I, S) (commandkit.Acquired[*promptkit.PromptInspection], error)
@@ -106,7 +106,8 @@ type Suite[I, S any] struct {
// SmokeTest runs one concrete native smoke-test operation after input and
// state preparation. It is optional; nil omits only the smoke-test command.
// The acquisition retains a native report even when the callback also
// returns an error.
// returns an error. For `test all`, CommandKit calls the captured callback
// sequentially for each selected kind after preparing input and state once.
SmokeTest func(context.Context, SmokeRequest, I, S) (commandkit.Acquired[*promptkit.SmokeTestReport], error)
// ListPrompts lists prompt identities after input and state preparation.
// It is optional independently of the other catalogs; nil omits only this
@@ -273,13 +274,14 @@ func promptKitContextError(ctx context.Context) error {
// The requirements callback forwards RequirementsRequest.Inspection to
// Engine.InspectRequirements. Concrete request validation is an independent
// capability and can be composed explicitly with WithRequestValidation. Smoke
// testing is also an explicit capability installed by its helper.
// testing is also an explicit capability installed with WithSmokeTesting.
// FromEngine does not call the loader during construction. Each selected
// operation passes its context, supplied selection, initialized input, and
// shared invocation state to the loader before calling PromptKit's public
// listing or inspection method. Applications can narrow, wrap, or replace these
// callbacks before Command captures them. A nil loader is retained as an
// adapter issue.
// adapter issue. Each callback returns loader diagnostics with the native
// operation's value and error.
func FromEngine[I, S any](load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error)) Suite[I, S] {
if load == nil {
return Suite[I, S]{issues: []error{errors.New("PromptKit engine loader is nil")}}

4
integrations/ckprompt/doc.go
View File

@@ -1,5 +1,5 @@
// Package ckprompt provides optional PromptKit configuration comparison,
// definition catalogs, named inspections, requirements discovery, and concrete
// request validation for CommandKit. Importing ckprompt installs neither
// prompt catalogs, named inspections, requirements discovery and validation,
// and native smoke testing for CommandKit. Importing ckprompt installs neither
// family; callers compose its declarations with core and ckconfig explicitly.
package ckprompt

8
integrations/ckprompt/smoke.go
View File

@@ -10,8 +10,9 @@ import (
promptkit "gitea.maximumdirect.net/eric/promptkit"
)
// SmokeRequest carries the configuration selection and seed request for one
// concrete PromptKit smoke-test operation.
// SmokeRequest pairs the configuration selection with the seed request for one
// concrete PromptKit smoke-test operation. It is passed to the optional
// builder and the Suite.SmokeTest callback.
type SmokeRequest struct {
// Selection identifies the configuration sources used to load the engine.
Selection ckconfig.Selection
@@ -25,7 +26,8 @@ type SmokeRequest struct {
// invocation context. Its complete returned request is passed to PromptKit;
// without a builder, the seed is forwarded unchanged. Reports and errors from
// PromptKit are both retained. A nil loader is kept as a suite wiring issue and
// does not install a smoke-test callback.
// does not install a smoke-test callback. The callback also retains engine
// loading diagnostics with the report and error.
func WithSmokeTesting[I, S any](
suite Suite[I, S],
load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error),