diff --git a/docs/roadmap/upgrades.md b/docs/roadmap/upgrades.md index b76616a..a7732f8 100644 --- a/docs/roadmap/upgrades.md +++ b/docs/roadmap/upgrades.md @@ -232,26 +232,145 @@ OD-3 records the accepted request/API and CLI filter choices. ### Explicit smoke testing -Expose PromptKit's `Engine.SmokeTest` through an independently optional command -in this roadmap. Forward a native `SmokeTestRequest` and present the returned -`SmokeTestReport`, including native reachability, authentication, text inference, -or structured-output observations, timing, and available usage as applicable. -Reuse the suite's typed input/state, configuration selection, and engine -acquisition paths. Keep request construction and report presentation close to -the native public API. +Expose PromptKit's `Engine.SmokeTest` through an independently optional +`promptkit test` command in this roadmap. Keep request construction and report +presentation close to the native public API, reusing the suite's typed +input/state, configuration selection, and engine acquisition paths. -Retain a returned report when the native call also returns an error, preserving -supported error identity and reporting output-delivery failures through the -existing outcome model. A nil native error and successful report delivery mean -success; native invocation errors receive the normal operational failure -classification. Present native observations without deriving an additional -readiness verdict or adding warnings about perceived upstream limitations. +#### Command vocabulary + +The selected syntax is: + +```text +app promptkit test KIND [options] +app promptkit test all [options] +``` + +One operand selects either a native kind or the aggregate `all` operation. +The native kind spellings are `reachability`, `authentication`, `text_inference`, +and `structured_output`. `all` is CommandKit-owned coordination syntax, not a +native `SmokeTestKind`. + +For example: + +```text +app promptkit test reachability --backend local +app promptkit test authentication --profile editorial +app promptkit test text_inference --prompt briefing +app promptkit test structured_output --profile editorial --model example-model +app promptkit test all --profile editorial +``` + +#### Request bindings + +Bind operator inputs directly to `promptkit.SmokeTestRequest` and its native +`ExecutionTargetOverride`: + +| CLI input | Native destination | Binding | +| --- | --- | --- | +| `KIND` | `Kind` | Native kind string; aggregate expansion is described below. | +| `--backend` | `BackendID` | String. | +| `--prompt` | `PromptID` | String. | +| `--profile` | `ProfileID` | String. | +| `--endpoint` | `Execution.Endpoint` | String. | +| `--model` | `Execution.Model` | String. | +| `--temperature` | `Execution.Temperature` | Optional floating-point value. | +| `--max-tokens` | `Execution.MaxTokens` | Optional integer. | +| `--top-p` | `Execution.TopP` | Optional floating-point value. | +| `--timeout-seconds` | `Execution.TimeoutSeconds` | Optional integer. | +| `--service-tier` | `Execution.ServiceTier` | String. | +| `--reasoning-effort` | `Execution.ReasoningEffort` | Optional string. | +| `--api-key-env` | `Execution.APIKeyEnv` | String. | +| `--transient-retries` | `TransientRetries` | Optional integer. | + +Also accept the existing `--config`, `--overlay`, and suite selectors. Reports +are YAML-only; `--format` defaults to `yaml` and accepts `yaml`. Built-in flags +are singleton bindings with the established last-occurrence behavior; suite +selectors retain their declared binding behavior. + +Preserve omission versus explicit zero or empty values for native pointer +fields. Leave `Execution` nil when no execution override was supplied. Backend, +prompt, and profile selectors may be supplied together: PromptKit owns their +precedence, resolution, defaults, ranges, and request validation. CommandKit +owns CLI syntax and primitive value conversion. + +Direct `APIKey` and structured `ExtraParams` remain available through the +optional application request builder. This scope uses that existing integration +pattern rather than adding a credential-source mechanism or generic override +language. + +#### Consumer API and request construction + +Use a narrow ckprompt-owned envelope: + +```go +type SmokeRequest struct { + Selection ckconfig.Selection + Request promptkit.SmokeTestRequest +} +``` + +An independently optional `Suite.SmokeTest` callback receives context, +`SmokeRequest`, typed application input, and shared state, and returns +`(commandkit.Acquired[*promptkit.SmokeTestReport], error)`. It represents one +concrete test and preserves both a returned native report and error. + +Provide `WithSmokeTesting(suite, load, build)` following the existing +request-validation adapter pattern. The loader uses the existing engine-loader +signature. The optional builder receives context, `SmokeRequest`, typed input, +and shared state, and returns a complete native `SmokeTestRequest` and error. +With no builder, forward the CLI-created native request. With a builder, its +returned request is authoritative; do not merge CLI values back over it. + +For aggregate execution, invoke the optional builder separately for each +concrete kind, seeding that kind into the otherwise common bound request. +Applications can supply per-kind credentials, structured parameters, or other +native request choices. The synthetic `all` value does not reach the native +request or the per-test builder. + +#### Aggregate execution + +`app promptkit test all` selects these four tests sequentially, in order: + +1. `reachability` +2. `authentication` +3. `text_inference` +4. `structured_output` + +Coordinate the concrete test callbacks within one CommandKit invocation, using +its configuration selection, prepared typed input, and shared state. Reuse +configuration acquisition and the engine where practical within the existing +loader model; this does not require a new caching framework or guarantee a +particular loader call count. + +Continue to the next kind after an individual test error so that remaining +results can be collected. When the invocation is canceled, stop scheduling +further tests and retain the results already available. PromptKit owns each +individual operation; CommandKit coordinates the selected sequence. + +#### Reports and invocation outcomes + +Present the native `SmokeTestReport`, including observations, timing, attempts, +and available usage. For `all`, deliver one YAML report containing the +individual results in execution order, with each selected kind's returned +report and associated error, when present. Retain useful partial reports +returned with errors. Preserve supported error identity through the invocation +outcome as well as presenting the errors. + +For a single test, a nil native error and successful report delivery mean +success. For `all`, success requires all selected calls to return without errors +and reporting to succeed. Native invocation errors use the normal operational +failure classification; cancellation and delivery failures retain their existing +outcome handling. Native negative or unknown findings remain reported facts; +CommandKit does not derive an additional readiness verdict or failure rule from +them. Use a focused report-delivery path that preserves a returned report alongside its native error, including cancellation. The existing success-only inspection writer cannot be reused unchanged for that case. Adapt delivery within the current outcome model; this work does not require a general diagnostics -framework. +framework. Describe returned data neutrally, without adding warnings about +perceived upstream limitations. Smoke testing runs when its command is explicitly invoked. Configuration, requirements, help, and release operations retain their own existing behavior. @@ -305,8 +424,9 @@ ID-only prompt requests, default/explicit listing filters, and smoke requests. Verify faithful representation of upstream-returned source evidence, optional labels, inventories, diagnostics, and errors, together with CommandKit's optional command availability, invocation lifetimes, outcome classification, and output -delivery. Include smoke report-plus-error delivery and error identity in this -boundary coverage. +delivery. Include smoke request bindings, per-kind builder forwarding, aggregate +order and continuation, cancellation, report-plus-error delivery, and error +identity in this boundary coverage. These checks establish CommandKit's access to the capabilities. They should not test whether ConfigKit correctly places or composes sources, checks versions, @@ -394,5 +514,7 @@ operational failures with supported error identity preserved. Native observation are reported neutrally. CommandKit does not add an aggregate readiness judgment or warnings about perceived upstream limitations. -The implementation plan must settle concrete command vocabulary and request -bindings within this scope. A separate smoke feature roadmap is not required. +The [explicit smoke testing target](#explicit-smoke-testing) records the accepted +`promptkit test KIND` and `promptkit test all` vocabulary, native request +bindings, per-kind builder, and aggregate execution/reporting semantics. A +separate smoke feature roadmap is not required.