diff --git a/README.md b/README.md index 683d23b..eb3ea97 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,8 @@ constructing them, and returns outcomes without exiting the process. Its | Module | Package | Provides | | --- | --- | --- | | `gitea.maximumdirect.net/eric/CommandKit` | `commandkit` | Typed commands, bindings, execution, outcomes, help, and version. | -| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | `ckconfig` | ConfigKit-backed `config` scopes and inspection commands. | -| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | `ckprompt` | PromptKit engine configuration/comparison, prompt/profile catalogs, named inspection, requirements discovery, and concrete request validation. | +| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | `ckconfig` | ConfigKit-backed `config` scopes, including configuration source evidence. | +| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | `ckprompt` | PromptKit engine configuration/comparison, five catalogs, named inspection, requirements discovery, concrete request validation, and native smoke tests. | Importing an integration does not install commands. Pass its declaration to `commandkit.New` or `App.Register` to install it. The integrations depend on the diff --git a/docs/cli.md b/docs/cli.md index e13e64f..95dc071 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -175,14 +175,36 @@ sources: kind: file underlay_index: 0 imported_from: null + destination: /server + placement: mapping + destination_unresolved: false + version_check: + path: /version + version: + present: true + null: false + text: "1" + location: + source_id: source-1 + line: 2 + column: 1 + redacted: false ``` Each source includes `id`, `label`, `path`, `layer`, `kind`, `underlay_index`, -and `imported_from`. The import location is null when absent; otherwise it has -`source_id`, `line`, and `column`. Owning values and source order are preserved, -including empty fields, zero coordinates, and synthetic sources without paths. -An empty inventory is `sources: []`. Text output begins with the scope heading -and then prints the same source fields, quoting labels and paths. +and `imported_from`, plus the native destination pointer, placement, unresolved +destination flag, and nullable `version_check`. Placement is `mapping` or +`sequence_item`. The import location is null when absent; otherwise it has +`source_id`, `line`, and `column`. Version evidence preserves `path`, `present`, +`null`, `text`, optional source `location`, and `redacted` as returned by +ConfigKit. Owning values and source order are preserved, including empty fields, +zero coordinates, false and null observations, and synthetic sources without +paths. An absent version check is null. An empty inventory is `sources: []`. +Text output begins with the scope heading and prints the same source fields, +quoting labels and paths. These fields project the selected ConfigKit source +records; CommandKit does not reload files or rerun version checks. See the +[ConfigKit v0.8.0 inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/137769f3f18a370c68444242fb5d88e620e307e5/inspection.go) +for native evidence definitions. ### Explain a configuration path @@ -306,38 +328,35 @@ existing incomplete-report delivery outcome and can leave partial output. ## PromptKit definition inspection Applications can install prompt and profile inspection independently with -`ckprompt.Command(suite)`, passing the resulting declaration to core. The -`ckprompt.FromEngine(loadEngine)` adapter supplies callbacks that acquire an -engine only after input checks pass and inspection is reached. The loader -receives invocation context, selection, input, and state; its implementation -controls whether an engine is constructed or reused. Direct suite callbacks -retain their own implementation and lifecycle. +`ckprompt.Command(suite)`. The `ckprompt.FromEngine(loadEngine)` adapter +supplies these callbacks alongside native requirements discovery and the five +catalogs. Its loader receives invocation context, configuration selection, +typed input, and shared state. Direct suite callbacks retain their own +implementation and lifecycle. -CommandKit's inspection path does not execute prompts, contact model providers, -or resolve provider credential values. Application acquisition and direct -callback effects remain application-owned. Prompt definitions and profile -resolution remain owned by -[PromptKit's inspection API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/engine.go). +Prompt inspection uses an ID only. PromptKit returns any descriptive version +metadata as an optional report field; there is no `--version` flag for selecting +a prompt version. The prompt format and native selection behavior remain owned by +[PromptKit v0.16.0](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md). ```text -example promptkit show prompt customer/summary --version 2 --format yaml +example promptkit show prompt customer/summary --format yaml example promptkit show prompt customer/summary example promptkit show profile customer-support --format yaml ``` Both commands require one ID operand and accept `--config`, `--overlay`, and -`--format text|yaml` (default `text`). Prompt inspection also accepts singleton -`--version`, defaulting to the empty string. PromptKit owns version selection: -an omitted version works for a single-version ID, while an ambiguous ID is an -operational error. These commands have no built-in `--scope` selection; their -namespace is `promptkit`. +`--format text|yaml` (default `text`). These commands have no built-in +`--scope` selection; their namespace is `promptkit`. -Prompt YAML has these fields. Input order is preserved and an empty input list is -`[]`. The prompt hash is an opaque owning-library equality value. +Prompt YAML preserves native input order and uses an optional `version` field. +The key is omitted when PromptKit returns no version and is retained when the +version is present, including an empty string. The prompt hash is an opaque +owning-library equality value. ```yaml prompt_id: customer/summary -prompt_version: "2" +version: "2" prompt_hash: opaque-value default_profile_id: customer-support inputs: @@ -376,47 +395,41 @@ api_key_required: true ``` Text uses labeled fields and includes nested `extra_params` values. A missing ID -is a usage error. Blank, unknown, and ambiguous definition selections reach the -owning inspection API and produce operational results with supported public -error identity. Cancellation is also operational: standard handlers check the -caller's context at inspection boundaries, and PromptKit can return cancellation -from its own inspection methods. Supported cancellation identity is preserved. +is a usage error. Definition selection reaches the owning inspection API and +preserves supported error identity and cancellation. ## PromptKit requirements -When `Suite.InspectRequirements` is present, -`ckprompt.Command(suite)` installs `promptkit requirements show`. It accepts one -required prompt ID and uses PromptKit's native selection and discovery rules: +When `Suite.InspectRequirements` is present, `ckprompt.Command(suite)` installs +`promptkit requirements show`. It accepts one prompt ID and uses PromptKit's +native selection and discovery rules: ```text example promptkit requirements show briefing --campaign spring -example promptkit requirements show briefing --version 2 --profile editorial --var mode=full +example promptkit requirements show briefing --profile editorial --var mode=full example promptkit requirements show briefing --session-id session-42 --var note= --format yaml ``` -The command accepts these flags: +The command accepts: | Flag | Values | Default | Meaning | | --- | --- | --- | --- | | `--config` | path string | omitted | Select the configuration input; explicit presence is retained. | | `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. | | Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. | -| `--version` | string | empty | Pass the prompt version to PromptKit; empty uses its native version-selection rule. | -| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native profile-selection rule. | +| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. | | `--session-id` | string | empty | Pass session context to PromptKit. | | `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. | | `--format` | `yaml` | `yaml` | Select the requirements report format. Other formats are usage errors for this operation. | -The `promptkit` namespace identifies the PromptKit operation, so there is no -built-in `--scope` flag. The version, profile, and session fields use native -empty-string semantics: an explicit empty string has the same meaning as -omission. PromptKit owns their interpretation, including default version and -profile selection, session context, and selection errors. Requirements -inspection does not require a profile to be execution-ready. See the pinned -[PromptKit requirements API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/143c6e9b3a675872117b0471e2ff8bcb25055a54/requirements.go) -for the owning discovery contract. The built-in flags other than `--var` are -singleton bindings; when repeated, the last value is used. Suite selectors retain -their declared binding behavior. +The `promptkit` namespace identifies the operation, so there is no built-in +`--scope` flag. An explicitly empty profile or session value retains PromptKit's +native empty-string meaning. Requirements inspection does not require a profile +to be execution-ready. See the +[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md) +for native configuration, selection, and requirements behavior. The built-in +flags other than `--var` are singleton bindings; when repeated, the last value +is used. Suite selectors retain their declared binding behavior. Each `--var` occurrence must contain `=`. CommandKit splits at the first separator and preserves the exact key and remaining value. Empty keys and @@ -424,20 +437,21 @@ values, whitespace, commas, and additional `=` characters are retained. All occurrences are parsed; duplicate keys use the last value, while any malformed occurrence is a usage error even if a later assignment is valid. -The successful report is YAML with selected identities, native completeness, -and the input, variable, and issue inventories. False values and empty -inventories are retained. Input metadata preserves null versus an empty string. -Variable entries contain `name`, `kind`, `supplied`, and `explanation`. -`supplied` reports native exact-key presence, including an explicitly empty -value, independently of `kind` and `complete`. Explanations and issue descriptions -are upstream-owned human-readable summaries, not machine-readable details. +The YAML report contains selected identities, native completeness, and input, +variable, and issue inventories. False values and empty inventories are +retained. Input metadata preserves null versus an empty string. Variable entries +contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native +exact-key presence, including an explicitly empty value, independently of +`kind` and `complete`. Explanations and issue descriptions are upstream-owned +human-readable summaries, not machine-readable details. The optional `version` +key is omitted when absent and retained when supplied by PromptKit. For a prompt with the template `{{if .mode}}{{input "notes"}}{{end}}` and no -input declarations, omitting `mode` can produce this partial report: +input declarations, omitting `mode` can produce a partial report: ```yaml prompt_id: briefing -prompt_version: "2" +version: "2" profile_id: editorial complete: false inputs: @@ -459,28 +473,24 @@ issues: description: A missing variable prevents selecting a dependency-bearing branch. ``` -PromptKit owns the inventory, classifications, explanations, completeness, -and structured issues. Completeness means discovery covered the relevant -dependencies; it does not mean inputs or variables are supplied, artifacts are -valid, credentials are available, or execution is ready. A complete report can -contain unsupplied required variables, and an incomplete report can still be -useful. Successfully acquired and delivered partial reports return status `0`; -structured issues remain in YAML and do not independently fail the command. -Provider errors, native selection errors, cancellation, and report delivery -failures use the operational path (status `1`) with supported cause identity. -Syntax, operand, variable-assignment, and format errors use usage status `2`. +PromptKit owns the inventory, classifications, explanations, completeness, and +structured issues. Successfully acquired and delivered partial reports return +status `0`; structured issues do not independently fail the command. Provider +errors, native selection errors, cancellation, and report delivery failures use +the operational path (status `1`) with supported cause identity. Syntax, +operand, variable-assignment, and format errors use usage status `2`. ### Concrete request validation When `Suite.ValidateRequest` is present, `ckprompt.Command(suite)` also installs `promptkit requirements validate`. It can be installed without discovery, and discovery can remain installed without validation. The command accepts one -required prompt ID and passes a concrete request to PromptKit's native +prompt ID and passes a concrete request to PromptKit's native `Engine.ValidateRequest` preparation path: ```text example promptkit requirements validate briefing --campaign spring --input topic=topic.txt -example promptkit requirements validate briefing --version 2 --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt +example promptkit requirements validate briefing --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt ``` The command accepts: @@ -490,7 +500,6 @@ The command accepts: | `--config` | path string | omitted | Select the configuration input; explicit presence is retained. | | `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. | | Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. | -| `--version` | string | empty | Pass the prompt version to PromptKit. | | `--profile` | string | empty | Pass the profile ID to PromptKit. | | `--session-id` | string | empty | Pass session context to PromptKit. | | `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. | @@ -498,24 +507,17 @@ The command accepts: | `--format` | `yaml` | `yaml` | Select the success confirmation format. Other formats are usage errors. | The built-in flags other than repeatable `--var` and `--input` are singleton -bindings; when repeated, the last value is used. Suite selectors retain their -declared binding behavior. Explicit empty `--config` or `--overlay` values keep -their supplied-presence information. +bindings; when repeated, the last value is used. Explicit empty `--config` or +`--overlay` values keep their supplied-presence information. The profile and +session fields use native empty-string semantics. PromptKit owns their +interpretation and selection errors. -The version, profile, and session fields use native empty-string semantics: -an explicit empty string has the same meaning as omission. PromptKit owns their -interpretation and selection errors. The `promptkit` namespace identifies the -operation, so there is no built-in `--scope` flag. - -Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at -the first separator and preserves the exact key/name and remaining value/path. -Empty variable keys and values are passed to PromptKit. Empty input names and -paths are also passed to native validation. File paths are translated to -`promptkit.File(PATH)` references without reading, normalizing, expanding, or -rebasing them during argument binding. Duplicate keys or names use the last -value, but all occurrences are parsed, so any malformed occurrence is a usage -error even if a later assignment is valid. Syntax and path values after shell -processing are otherwise preserved. +Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at the +first separator and preserves the exact key/name and remaining value/path. +Empty variable keys and values and empty input names and paths are passed to +native validation. File paths become `promptkit.File(PATH)` references without +being read, normalized, expanded, or rebased during argument binding. Duplicate +keys or names use the last value, but every occurrence is parsed. On success, the YAML confirmation contains only the requested prompt ID and `valid: true`: @@ -525,46 +527,64 @@ prompt_id: briefing valid: true ``` -Validation delegates concrete request preparation to PromptKit. Depending on -the request and configuration, native preparation may resolve prompt/profile -resources, load artifact and schema resources, render messages and session -context, and perform applicable configuration and credential checks. Native -errors are returned as encountered rather than as an exhaustive inventory. -Validation does not make a model call or retain a prepared execution snapshot. -A successful report means preparation passed for this invocation; it does not -certify provider acceptance or guarantee that later execution will succeed. -See PromptKit's [v0.15.0 `Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/engine.go) -for the native contract. +Validation delegates concrete request preparation to PromptKit. Native +preparation may resolve prompt/profile resources, load artifact and schema +resources, render messages and session context, and perform applicable +configuration and credential checks. It does not make a model call or retain a +prepared execution snapshot. A successful report describes this preparation +invocation; it does not certify provider acceptance or guarantee later +execution. See PromptKit's +[`Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go). Builder, engine-loading, and native validation errors use the operational path (status `1`) with supported error identity and acquired diagnostics retained. Malformed syntax, missing operands, and unsupported formats use usage status -`2`. Input-check failures keep their existing shared handling. Successful -diagnostics and report-delivery failures follow the common delivery contract. +`2`. Successful diagnostics and report-delivery failures follow the common +delivery contract. ### PromptKit catalogs -An installed `ckprompt.Command(suite)` exposes the catalog commands supplied by -the suite: +An installed `ckprompt.Command(suite)` exposes each catalog whose callback is +present: ```text example promptkit list prompts example promptkit list profiles --config settings.yml --campaign spring +example promptkit list backends --namespace team +example promptkit list namespaces --kind prompt --kind profile +example promptkit list collections ``` -Each command accepts `--config`, `--overlay`, and applicable selectors from -`Suite.Selectors`. It takes no ID or version operand and has no `--version` or -`--scope` selector. `--format` defaults to `yaml`; `--format yaml` is accepted, -and other formats are usage errors. +All catalogs accept `--config`, `--overlay`, applicable flags from +`Suite.Selectors`, and `--format yaml` (the default). Other formats are usage +errors. Prompt, profile, and backend listings accept namespace selection: -Each invocation writes one bare YAML sequence. Prompt entries have explicit -string `id` and `version` fields; profile entries are strings. Provider order and -exact identity values are preserved. For example: +| Binding | Effect | +| --- | --- | +| omitted `--namespace` and `--all-namespaces` | Forward PromptKit's native default namespace filter. | +| `--namespace team` | Forward the named namespace filter. | +| `--namespace ""` | Select the unqualified namespace. | +| `--all-namespaces` | Forward the all-namespaces filter. | +| `--all-namespaces=false` | Do not select an override; a supplied namespace still applies, otherwise native default is used. | + +The namespace binding is a singleton and its last occurrence is used. +Supplying a namespace while `--all-namespaces=true` is active is a usage error +before provider work. Namespace inventory accepts repeatable `--kind KIND`; +omitting `--kind` forwards PromptKit's native default kind selection. Repeated +values are forwarded in order. Collections have no namespace or kind filter. + +Each operation writes one bare YAML sequence, retaining provider order and +native identities: + +- Prompt entries are `{id, version?}`. The optional `version` key is omitted + when absent and retained for a present empty or nonempty value. +- Profiles, backends, and namespaces are strings. +- Collection entries are `{namespace, version?}`, where `version` projects + PromptKit's optional release version. +- Nil and allocated-empty inventories both render as `[]`. ```yaml -- id: briefing - version: "1" -- id: briefing +- id: team/briefing version: "2" ``` @@ -573,11 +593,155 @@ exact identity values are preserved. For example: - spring ``` -Successful nil and allocated-empty inventories both render as `[]`. Catalog -membership and ordering follow the selected provider, and a listed definition -is not a readiness check or execution result. PromptKit owns the meaning of its -native listing APIs; custom providers own their results. Applications retain -their own disclosure choices for catalog data. +A catalog result is an inventory, not a readiness check or execution result. +The owning provider controls membership and order. Namespace filters are passed +to PromptKit without local membership interpretation. See the +[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md) +for native resource and selection definitions. + +### PromptKit smoke testing + +When `Suite.SmokeTest` is present, `ckprompt.Command(suite)` installs +`promptkit test KIND`. The special kind `all` runs the four concrete operations +sequentially in this order: `reachability`, `authentication`, `text_inference`, +and `structured_output`. Both forms use the same single-operation callback; +each aggregate call changes the seed kind before invoking it. The command +accepts: + +```text +example promptkit test text_inference --campaign spring --prompt briefing --profile spring +example promptkit test all --campaign spring --prompt briefing --profile spring +example promptkit test authentication --backend local +``` + +| Flag | Values | Default | Meaning | +| --- | --- | --- | --- | +| `--config`, `--overlay` | path strings | omitted | Select PromptKit configuration; supplied presence is retained. | +| Suite selectors | application-defined | per selector | Captured flags from `Suite.Selectors`. | +| `--backend` | string | empty | Native backend ID. | +| `--prompt` | string | empty | Native prompt ID. | +| `--profile` | string | empty | Native profile ID. | +| `--endpoint` | string | omitted | Override the execution endpoint. | +| `--model` | string | omitted | Override the execution model. | +| `--temperature` | float | omitted | Override execution temperature. | +| `--max-tokens` | integer | omitted | Override execution maximum tokens. | +| `--top-p` | float | omitted | Override execution top-p. | +| `--timeout-seconds` | integer | omitted | Override the execution timeout. | +| `--service-tier` | string | omitted | Override the execution service tier. | +| `--reasoning-effort` | string | omitted | Override execution reasoning effort. | +| `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. | +| `--transient-retries` | integer | omitted | Set the native inference transient retry count. | +| `--format` | `yaml` | `yaml` | Select the report format. Other formats are usage errors. | + +The execution flags populate PromptKit's native execution override. The +`--api-key-env` value is a variable name, not a credential value. Application +selectors, loader policy, credentials, and client wiring remain application +owned. The `WithSmokeTesting` helper accepts an optional typed builder. A nil +builder forwards the seed; a supplied builder's complete native request is +authoritative. The helper retains a native report when PromptKit returns both a +report and an error. + +A single report projects the native smoke report into YAML. It preserves native +false, unknown/null, and incomplete values and does not add a readiness score. +The report includes `kind`, `supported`, `applicable`, `completed`, `reason`, +`target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target +contains selected IDs, optional `version`, selection source, endpoint, effective +settings, and settings-presence flags. Attempt entries retain their own facts, +timing, failure, and token-usage evidence. Durations use Go duration strings; +token counts preserve `value`, `known`, and `complete`. + +Here is a native projection as rendered by the command: + +```yaml +kind: reachability +supported: false +applicable: null +completed: true +reason: http_diagnostics_unavailable +target: + backend_id: "" + profile_id: spring + prompt_id: briefing + version: "1" + default_source_backend_id: "" + selection_source: profile + endpoint: https://example.invalid/v1 + effective_settings: + endpoint: https://example.invalid/v1 + model: spring-model + temperature: 0 + max_tokens: 0 + top_p: 0 + timeout_seconds: 600 + service_tier: "" + reasoning_effort: "" + api_key_env: "" + extra_params: + campaign: spring + settings_presence: + temperature: false + max_tokens: false + top_p: false + timeout_seconds: false +facts: + credentials_supplied: null + endpoint_reachable: null + credentials_accepted: null + response_received: null + generation_completed: null + text_returned: null + expected_text_matched: null + structured_output_requested: null + valid_json: null + schema_matched: null +timing: + started_at: 2026-10-17T19:05:36Z + duration: 35.685µs + capacity_wait: null + request_duration: null + retry_delay: null + evaluation_duration: null +attempts: [] +usage: + prompt_tokens: {value: 0, known: true, complete: true} + completion_tokens: {value: 0, known: true, complete: true} + total_tokens: {value: 0, known: true, complete: true} + cached_tokens: {value: 0, known: true, complete: true} + cache_write_tokens: {value: 0, known: true, complete: true} +failure: "" +``` + +Aggregate output is one YAML mapping with ordered `tests` entries. Each entry +has selected `kind`, nullable `report`, and optional `error` text. The inner +report keeps its native `kind`. This cancellation example shows two attempted +operations with no reports; later kinds are absent because cancellation stops +the sequence: + +```yaml +tests: + - kind: reachability + report: null + error: "reachability operation failed" + - kind: authentication + report: null + error: "context canceled" +``` + +A successful delivered single operation returns status `0`. For `all`, all +four callbacks must return without error for status `0`. Native false, unknown, +unsupported, or incomplete observations do not independently change the +outcome. Operation errors use operational status `1`; aggregate runs continue +after individual errors, retain their reports and ordered error text, and join +causes for supported `errors.Is`/`errors.As` access. Cancellation stops later +operations and preserves results already acquired. Report encoding or delivery +failures use the common incomplete-report delivery outcome and retain operation +causes where applicable. Syntax, operand, unsupported format, and conflicting +namespace selection are usage status `2`. + +Native configuration, selection, and diagnostic meanings remain owned by +PromptKit. See the [v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md), +[smoke API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/smoke_diagnostics.go), +and [engine API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go). ## Outcome status diff --git a/docs/consumers/guide.md b/docs/consumers/guide.md index dd184b6..52344f3 100644 --- a/docs/consumers/guide.md +++ b/docs/consumers/guide.md @@ -11,7 +11,7 @@ through `commandkit.New` or `App.Register`. | --- | --- | | `gitea.maximumdirect.net/eric/CommandKit` | The application needs typed command declarations and common CLI behavior. | | `gitea.maximumdirect.net/eric/CommandKit/integrations/ckconfig` | The application wants ConfigKit-backed configuration scopes and reports. | -| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | The application wants PromptKit engine configuration/comparison, prompt/profile catalogs, named inspection, requirements discovery, or concrete request validation. | +| `gitea.maximumdirect.net/eric/CommandKit/integrations/ckprompt` | The application wants PromptKit engine configuration/comparison, five catalogs, named inspection, requirements discovery, request validation, or native smoke testing. | The two integrations are independent optional imports. `ckprompt` uses the public scope and error contracts from `ckconfig`; importing it does not add the @@ -22,7 +22,7 @@ core-only, ConfigKit-only, and composed applications. ## Go compatibility All three modules declare a consumer minimum of Go 1.26.0. The root module also -selects ReleaseKit v0.4.0 as a Go tool, which makes that tool's module graph part +selects ReleaseKit v0.4.2 as a Go tool, which makes that tool's module graph part of the root consumer module requirements even though production packages do not import ReleaseKit. The [release policy](../policy/release.md#consumer-minimum-and-release-toolchain) defines the support assumption. Maintainers use Go 1.26.8 for validation and @@ -126,17 +126,20 @@ engine's captured configuration. Add that scope to a `ckconfig.Suite` to expose it under `config` with validate/show/sources/explain and native diff operations; it does not install PromptKit catalog, definition, or requirements commands. -`ckprompt.FromEngine(loadEngine)` supplies five callbacks: prompt and profile -catalogs, named prompt and profile inspection, and native requirements -discovery. Concrete request validation is a sixth, independently optional -capability and is added explicitly with `ckprompt.WithRequestValidation`. -`ckprompt.Command(suite)` creates only operations whose callbacks are present; -discovery and validation share `promptkit requirements` but can be installed -independently. A composed suite can keep the standard five and add validation: +`ckprompt.FromEngine(loadEngine)` supplies eight callbacks: prompt, profile, +backend, namespace, and collection catalogs; named prompt and profile +inspection; and native requirements discovery. Concrete request validation and +smoke testing are separate capabilities, added explicitly with +`ckprompt.WithRequestValidation` and `ckprompt.WithSmokeTesting`. The suite has +ten independently optional callbacks in total. `ckprompt.Command(suite)` creates +only operations whose callbacks are present; discovery, validation, and smoke +testing can be installed independently. A composed suite can enable both +additional operations: ```go suite := ckprompt.FromEngine(loadEngine) suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest) +suite = ckprompt.WithSmokeTesting(suite, loadEngine, nil) suite.NewInput = newInput suite.NewState = newState suite.Selectors = selectors @@ -165,8 +168,9 @@ absent, so explicitly supplied empty values stay meaningful. Applications that validate and then execute a request are encouraged to share their request construction logic between those paths. -All six suite capabilities are independently optional. For catalogs without -named or requirements inspection, narrow the suite before command capture: +The eight catalog and inspection callbacks and the two explicit additions are +independently optional. For catalogs without named or requirements inspection, +narrow the suite before command capture: ```go definitions := ckprompt.FromEngine(loadEngine) @@ -176,7 +180,7 @@ definitions.InspectRequirements = nil definitionCommand := ckprompt.Command(definitions) ``` -Each selected inspection, requirements, validation, or catalog operation +Each selected inspection, requirements, validation, smoke, or catalog operation prepares one input, applies its captured selectors, checks that input, and initializes one state value when those steps are reached. The bundled validation adapter delegates concrete request preparation to PromptKit and carries its @@ -195,13 +199,19 @@ PromptKit preparation, which can resolve prompt/profile resources, load artifacts and schemas, render messages, and perform applicable configuration and credential checks. It does not make a model call or certify that a later execution will succeed. Application loaders and direct callbacks retain their -own effects. The [CLI reference](../cli.md#promptkit-definition-inspection) -and [requirements contract](../cli.md#promptkit-requirements) own operator -syntax and reports. +own effects. Smoke testing explicitly invokes one native PromptKit diagnostic +or all four in order. Native inference checks use the configured public +`promptkit.LLMClient`; applications retain ownership of client and probe +configuration. A builder may customize the seeded native request, and its +complete return value is authoritative. The [CLI reference](../cli.md#promptkit-smoke-testing), +[definition inspection](../cli.md#promptkit-definition-inspection), and +[requirements contract](../cli.md#promptkit-requirements) own operator syntax +and reports. The [composed example](../../integrations/ckprompt/examples/composed/README.md) combines app and PromptKit config scopes, comparison and catalogs, named and -requirements inspection, and an application-owned `session restore` command. +requirements inspection, request validation, smoke testing, and an +application-owned `session restore` command. ## Outcomes and ownership @@ -214,7 +224,8 @@ process. Go declarations and GoDoc own exact exported contracts. ConfigKit owns configuration decoding, validation, and provenance. PromptKit owns prompt formats, profile resolution, and inspection details. See the -[ConfigKit inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/inspection.go), -[PromptKit engine methods](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/engine.go), -[configuration inspection](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/configuration_inspection.go), -and [inspection types](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/types.go). +[ConfigKit v0.8.0 inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/137769f3f18a370c68444242fb5d88e620e307e5/inspection.go), +[PromptKit engine methods](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go), +[configuration inspection](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/configuration_inspection.go), +[smoke diagnostics](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/smoke_diagnostics.go), +and [PromptKit v0.16.0 consumer guidance](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md). diff --git a/docs/development.md b/docs/development.md index bfcc989..3cb586e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -131,7 +131,8 @@ or publishing. ## Current public examples The maintained consumers compile with their owning module suites and run without -external services: +external services. The PromptKit smoke examples use local fixtures and the +example's configured client: ```sh go run ./examples/core workspace greet --count=0 --tag=first --tag second @@ -142,11 +143,16 @@ go run ./examples/core workspace greet --count=0 --tag=first --tag second (cd integrations/ckprompt/examples/composed && go run . config diff --scope promptkit --config fixtures/before.yml --against-config fixtures/after.yml --campaign spring) (cd integrations/ckprompt/examples/composed && go run . promptkit list prompts --config fixtures/before.yml --campaign spring) (cd integrations/ckprompt/examples/composed && go run . promptkit list profiles --config fixtures/after.yml --campaign autumn --format yaml) +(cd integrations/ckprompt/examples/composed && go run . promptkit list backends --namespace team) +(cd integrations/ckprompt/examples/composed && go run . promptkit list namespaces --kind prompt --kind profile) +(cd integrations/ckprompt/examples/composed && go run . promptkit list collections) (cd integrations/ckprompt/examples/composed && go run . promptkit show prompt briefing --campaign spring --format yaml) (cd integrations/ckprompt/examples/composed && go run . promptkit requirements show briefing --campaign spring) (cd integrations/ckprompt/examples/composed && go run . promptkit requirements show briefing --campaign spring --var mode=full --format yaml) (cd integrations/ckprompt/examples/composed && go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt) (cd integrations/ckprompt/examples/composed && go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt --var mode=full --input notes=fixtures/notes.txt --format yaml) +(cd integrations/ckprompt/examples/composed && go run . promptkit test reachability --campaign spring --prompt briefing --profile spring) +(cd integrations/ckprompt/examples/composed && go run . promptkit test all --campaign spring --prompt briefing --profile spring) (cd integrations/ckprompt/examples/composed && go run . session restore session-42 --campaign spring) ``` diff --git a/docs/internal/overview.md b/docs/internal/overview.md index e0772a5..53019d5 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -47,10 +47,12 @@ initialized input, while one suite-owned state value is shared by an invocation' acquisitions or comparison callback. `integrations/ckprompt` owns PromptKit engine loading, configuration-scope -adaptation and native comparison, plus optional prompt/profile catalogs, named -inspection, native requirements discovery, and concrete request validation. -Validation delegates request preparation to PromptKit and is independently -opted into a suite. It uses public core and ckconfig contracts plus PromptKit's +adaptation and native comparison, plus optional prompt/profile/backend/namespace/ +collection catalogs, named inspection, native requirements discovery, +concrete request validation, and native smoke testing. Validation and smoke +testing delegate to PromptKit and are independently opted into a suite. Smoke +operations use the application's native engine and its configured client or +probes. The integration uses public core and ckconfig contracts plus PromptKit's public APIs. It does not add a configuration family when its declaration is installed; callers compose `ckprompt.ConfigScope` with `ckconfig.Command` separately when they want configuration operations alongside PromptKit @@ -70,8 +72,10 @@ graph isolation. ConfigKit owns configuration decoding, validation, source history, and path explanations. PromptKit owns its captured engine configuration, prompt definitions, profile inheritance, named inspection, requirements discovery, -and concrete request validation semantics. CommandKit adapts those public -contracts without maintaining parallel schemas or invoking model generation. +and concrete request validation, and native smoke-test semantics. CommandKit +adapts those public contracts without maintaining parallel schemas. Explicit +native smoke operations may use the configured client or probes; other +inspection, catalog, and request-validation paths do not make model calls. Native request validation may load resources and artifacts, render messages, and perform applicable configuration or credential checks through PromptKit. Application loaders and domain behavior remain application owned. diff --git a/docs/roadmap/deferred.md b/docs/roadmap/deferred.md index 6c31883..0f54575 100644 --- a/docs/roadmap/deferred.md +++ b/docs/roadmap/deferred.md @@ -33,7 +33,8 @@ operator or consumer need and an appropriate ownership/behavior decision: - JSON reports, shell completion, and output-file management or reloadable export. - Configuration mutation, generic value assignment, and exhaustive field-origin inventories. -- Credential diagnostics and live provider probes. +- Credential diagnostics beyond PromptKit's native smoke checks and additional + live provider probes. - Additional component identity registration and domain artifact catalogs. - Additional library families, including possible FeedKit integration; families can define suitable verbs without requiring comparison or configuration scopes. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 71b7f65..20026e6 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ # Dependency upgrades, discovery, and smoke testing implementation plan -Status: Stages 1–16 complete; Stage 17 is next. +Status: Stages 1–17 complete; Stage 18 is next. ## Authority and baseline @@ -829,6 +829,16 @@ is required. **Complete when:** current contracts describe implemented behavior under their canonical owners and historical references remain correctly scoped. +**Completed:** 2026-10-17. GoDoc, module summaries, the README, CLI reference, +consumer guide, development commands, and internal inventory now describe the +adopted source evidence, eight `FromEngine` callbacks, five catalogs, optional +request validation, and native smoke behavior. Smoke examples and catalog +commands were run against the maintained composed consumer; `go doc` confirmed +the exported helper and request documentation. Local Markdown links/fragments, +stale upstream revision references, and whitespace checks passed. Deferred-work +scope and historical implementation-plan links were reconciled without changing +historical API examples or release notes. + ## Stage 18: Verify the complete target and finish 1. Format all changed Go files and inspect the full diff for unrelated changes, diff --git a/docs/roadmap/promptkit-validation.md b/docs/roadmap/promptkit-validation.md index 1876cc0..4f1ab81 100644 --- a/docs/roadmap/promptkit-validation.md +++ b/docs/roadmap/promptkit-validation.md @@ -3,7 +3,8 @@ Status: accepted adoption and feature roadmap; implementation complete. Command placement, adapter composition, request construction, artifact syntax, and initial presentation are settled. This document defines the target and -required work; the [implementation plan](implementation.md) sequences the work. +required work; the [implementation plan](https://gitea.maximumdirect.net/eric/CommandKit/src/commit/38b49f29fb3a8f22ff02528d2a0ed413e5c6ab1e/docs/roadmap/implementation.md) +sequences the historical implementation work. ## Purpose diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 24be987..0eb43f4 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -8,7 +8,8 @@ architectural redesign was identified as necessary. Reviewed implementation commits after planning baseline `3218222`, through `420483b2d4659862626422b0577e39bc0a4c1a2d`, against the [accepted feature roadmap](promptkit-validation.md), -[implementation plan](implementation.md), applicable ADRs, and repository policy. +[implementation plan](https://gitea.maximumdirect.net/eric/CommandKit/src/commit/38b49f29fb3a8f22ff02528d2a0ed413e5c6ab1e/docs/roadmap/implementation.md), +applicable ADRs, and repository policy. The review covered production changes, new tests, the maintained consumer, dependency metadata/attribution, and updated documentation. diff --git a/integrations/ckconfig/README.md b/integrations/ckconfig/README.md index ebec781..fa6641b 100644 --- a/integrations/ckconfig/README.md +++ b/integrations/ckconfig/README.md @@ -27,8 +27,10 @@ scopes, selectors, input checks, and state shared by the selected acquisitions or comparison in a config invocation. `ckconfig.FromConfigKit` adapts a typed ConfigKit loader. The `config` command family exposes scope discovery and the validate, show, sources, explain, and diff operations supported by the declared -scopes. Comparison returns ConfigKit's native records to the YAML report path; -the [CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#compare-configuration-selections) +scopes. Source reports include native destination, placement, unresolved +destination, and source version-check evidence. Comparison returns ConfigKit's +native records to the YAML report path; the +[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#compare-configuration-selections) owns its operator details. The maintained [simple example](examples/simple) shows a complete consumer. diff --git a/integrations/ckconfig/doc.go b/integrations/ckconfig/doc.go index cd51fd6..5df3247 100644 --- a/integrations/ckconfig/doc.go +++ b/integrations/ckconfig/doc.go @@ -1,4 +1,6 @@ // Package ckconfig provides CommandKit's optional ConfigKit integration. Its // declarations install the standard config command family through the core -// commandkit package; importing ckconfig alone installs no commands. +// commandkit package. Source reports preserve ConfigKit's destination, +// placement, and version-check evidence; importing ckconfig alone installs no +// commands. package ckconfig diff --git a/integrations/ckprompt/README.md b/integrations/ckprompt/README.md index bf5c38c..6f1d7c3 100644 --- a/integrations/ckprompt/README.md +++ b/integrations/ckprompt/README.md @@ -1,9 +1,9 @@ # CommandKit ckprompt `ckprompt` is the optional PromptKit integration for CommandKit. It adapts -PromptKit engine configuration, native comparison, prompt/profile catalogs, -named inspection, and prompt requirements discovery. Importing it does not -install either command family. +PromptKit engine configuration, native comparison, five catalogs, named +inspection, requirements discovery and validation, and native smoke tests. +Importing it does not install either command family. ## Module @@ -19,14 +19,17 @@ for callback, input, state, and acquisition contracts. Add `ckprompt.ConfigScope(loadEngine)` to the scopes passed to `ckconfig.Command` when the application wants PromptKit engine configuration and native comparison in the `config` family. Install -`ckprompt.Command(ckprompt.FromEngine(loadEngine))` separately for prompt/profile -catalogs, named inspections, and requirements discovery. `FromEngine` supplies -all five PromptKit-backed callbacks; applications may narrow, wrap, or replace -them before capture or provide custom callbacks with `ckprompt.Suite`. -Requirements is independently optional. CommandKit-owned inspection and -listing code does not execute prompts or contact model providers; application -loaders and direct callbacks retain their own effects. See the -[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#promptkit-requirements) +`ckprompt.Command(ckprompt.FromEngine(loadEngine))` separately for catalogs, +named inspections, and requirements discovery. `FromEngine` supplies eight +PromptKit-backed callbacks: three inspections and five catalogs. Applications +may narrow, wrap, or replace them before capture or provide custom callbacks +with `ckprompt.Suite`. Add concrete request validation and smoke testing +explicitly with `WithRequestValidation` and `WithSmokeTesting`. Inspection, +requirements, and catalog operations do not execute prompts or contact model +providers; application loaders and callbacks retain their own effects. Smoke +testing invokes the selected native PromptKit operation; the +application retains ownership of client and configuration choices. See the +[CLI reference](https://gitea.maximumdirect.net/eric/CommandKit/src/branch/main/docs/cli.md#promptkit-smoke-testing) for operator commands and report semantics. The maintained [composed example](examples/composed) shows complete application wiring. diff --git a/integrations/ckprompt/adapters.go b/integrations/ckprompt/adapters.go index 9f4256c..a1606e5 100644 --- a/integrations/ckprompt/adapters.go +++ b/integrations/ckprompt/adapters.go @@ -73,20 +73,20 @@ type RequirementsRequest struct { // Factories are not invoked during construction, installation, help, or version, // and need not run after failure. type Suite[I, S any] struct { - // NewInput is called once when definition, requirements, validation, or - // catalog input preparation is reached, before applying selectors and + // NewInput is called once when inspection, requirements, validation, smoke, + // or catalog input preparation is reached, before applying selectors and // calling CheckInput. // If nil, input is the zero value of I, including nil for pointer types. NewInput func() I - // NewState is called once when an inspection, validation, or catalog + // NewState is called once when an inspection, validation, smoke, or catalog // callback is reached, after input checks pass. If nil, state is the zero // value of S, including nil for pointer types. NewState func() S - // Selectors declares the original typed flags accepted by definition, - // requirements, validation, and catalog commands. + // Selectors declares the original typed flags accepted by inspection, + // requirements, validation, smoke, and catalog commands. Selectors []commandkit.Flag[I] - // CheckInput validates selected definition, requirements, validation, or catalog input - // before state creation. + // CheckInput validates selected inspection, requirements, validation, smoke, + // or catalog input before state creation. CheckInput func(ckconfig.Selection, I) error // InspectPrompt inspects the selected prompt after input and state preparation. InspectPrompt func(context.Context, PromptRequest, I, S) (commandkit.Acquired[*promptkit.PromptInspection], error) @@ -106,7 +106,8 @@ type Suite[I, S any] struct { // SmokeTest runs one concrete native smoke-test operation after input and // state preparation. It is optional; nil omits only the smoke-test command. // The acquisition retains a native report even when the callback also - // returns an error. + // returns an error. For `test all`, CommandKit calls the captured callback + // sequentially for each selected kind after preparing input and state once. SmokeTest func(context.Context, SmokeRequest, I, S) (commandkit.Acquired[*promptkit.SmokeTestReport], error) // ListPrompts lists prompt identities after input and state preparation. // It is optional independently of the other catalogs; nil omits only this @@ -273,13 +274,14 @@ func promptKitContextError(ctx context.Context) error { // The requirements callback forwards RequirementsRequest.Inspection to // Engine.InspectRequirements. Concrete request validation is an independent // capability and can be composed explicitly with WithRequestValidation. Smoke -// testing is also an explicit capability installed by its helper. +// testing is also an explicit capability installed with WithSmokeTesting. // FromEngine does not call the loader during construction. Each selected // operation passes its context, supplied selection, initialized input, and // shared invocation state to the loader before calling PromptKit's public // listing or inspection method. Applications can narrow, wrap, or replace these // callbacks before Command captures them. A nil loader is retained as an -// adapter issue. +// adapter issue. Each callback returns loader diagnostics with the native +// operation's value and error. func FromEngine[I, S any](load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error)) Suite[I, S] { if load == nil { return Suite[I, S]{issues: []error{errors.New("PromptKit engine loader is nil")}} diff --git a/integrations/ckprompt/doc.go b/integrations/ckprompt/doc.go index 1f0edd8..a120228 100644 --- a/integrations/ckprompt/doc.go +++ b/integrations/ckprompt/doc.go @@ -1,5 +1,5 @@ // Package ckprompt provides optional PromptKit configuration comparison, -// definition catalogs, named inspections, requirements discovery, and concrete -// request validation for CommandKit. Importing ckprompt installs neither +// prompt catalogs, named inspections, requirements discovery and validation, +// and native smoke testing for CommandKit. Importing ckprompt installs neither // family; callers compose its declarations with core and ckconfig explicitly. package ckprompt diff --git a/integrations/ckprompt/smoke.go b/integrations/ckprompt/smoke.go index 23da56c..f915c53 100644 --- a/integrations/ckprompt/smoke.go +++ b/integrations/ckprompt/smoke.go @@ -10,8 +10,9 @@ import ( promptkit "gitea.maximumdirect.net/eric/promptkit" ) -// SmokeRequest carries the configuration selection and seed request for one -// concrete PromptKit smoke-test operation. +// SmokeRequest pairs the configuration selection with the seed request for one +// concrete PromptKit smoke-test operation. It is passed to the optional +// builder and the Suite.SmokeTest callback. type SmokeRequest struct { // Selection identifies the configuration sources used to load the engine. Selection ckconfig.Selection @@ -25,7 +26,8 @@ type SmokeRequest struct { // invocation context. Its complete returned request is passed to PromptKit; // without a builder, the seed is forwarded unchanged. Reports and errors from // PromptKit are both retained. A nil loader is kept as a suite wiring issue and -// does not install a smoke-test callback. +// does not install a smoke-test callback. The callback also retains engine +// loading diagnostics with the report and error. func WithSmokeTesting[I, S any]( suite Suite[I, S], load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error),