base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record PromptKit test commands and aggregate behavior

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-17 16:34:07 +00:00
parent f173fdbc1b
commit 38b49f29fb
1 changed files with 140 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

158
docs/roadmap/upgrades.md
View File

@@ -232,26 +232,145 @@ OD-3 records the accepted request/API and CLI filter choices.
### Explicit smoke testing
Expose PromptKit's `Engine.SmokeTest` through an independently optional command
in this roadmap. Forward a native `SmokeTestRequest` and present the returned
`SmokeTestReport`, including native reachability, authentication, text inference,
or structured-output observations, timing, and available usage as applicable.
Reuse the suite's typed input/state, configuration selection, and engine
acquisition paths. Keep request construction and report presentation close to
the native public API.
Expose PromptKit's `Engine.SmokeTest` through an independently optional
`promptkit test` command in this roadmap. Keep request construction and report
presentation close to the native public API, reusing the suite's typed
input/state, configuration selection, and engine acquisition paths.
Retain a returned report when the native call also returns an error, preserving
supported error identity and reporting output-delivery failures through the
existing outcome model. A nil native error and successful report delivery mean
success; native invocation errors receive the normal operational failure
classification. Present native observations without deriving an additional
readiness verdict or adding warnings about perceived upstream limitations.
#### Command vocabulary
The selected syntax is:
```text
app promptkit test KIND [options]
app promptkit test all [options]
```
One operand selects either a native kind or the aggregate `all` operation.
The native kind spellings are `reachability`, `authentication`, `text_inference`,
and `structured_output`. `all` is CommandKit-owned coordination syntax, not a
native `SmokeTestKind`.
For example:
```text
app promptkit test reachability --backend local
app promptkit test authentication --profile editorial
app promptkit test text_inference --prompt briefing
app promptkit test structured_output --profile editorial --model example-model
app promptkit test all --profile editorial
```
#### Request bindings
Bind operator inputs directly to `promptkit.SmokeTestRequest` and its native
`ExecutionTargetOverride`:
| CLI input | Native destination | Binding |
| --- | --- | --- |
| `KIND` | `Kind` | Native kind string; aggregate expansion is described below. |
| `--backend` | `BackendID` | String. |
| `--prompt` | `PromptID` | String. |
| `--profile` | `ProfileID` | String. |
| `--endpoint` | `Execution.Endpoint` | String. |
| `--model` | `Execution.Model` | String. |
| `--temperature` | `Execution.Temperature` | Optional floating-point value. |
| `--max-tokens` | `Execution.MaxTokens` | Optional integer. |
| `--top-p` | `Execution.TopP` | Optional floating-point value. |
| `--timeout-seconds` | `Execution.TimeoutSeconds` | Optional integer. |
| `--service-tier` | `Execution.ServiceTier` | String. |
| `--reasoning-effort` | `Execution.ReasoningEffort` | Optional string. |
| `--api-key-env` | `Execution.APIKeyEnv` | String. |
| `--transient-retries` | `TransientRetries` | Optional integer. |
Also accept the existing `--config`, `--overlay`, and suite selectors. Reports
are YAML-only; `--format` defaults to `yaml` and accepts `yaml`. Built-in flags
are singleton bindings with the established last-occurrence behavior; suite
selectors retain their declared binding behavior.
Preserve omission versus explicit zero or empty values for native pointer
fields. Leave `Execution` nil when no execution override was supplied. Backend,
prompt, and profile selectors may be supplied together: PromptKit owns their
precedence, resolution, defaults, ranges, and request validation. CommandKit
owns CLI syntax and primitive value conversion.
Direct `APIKey` and structured `ExtraParams` remain available through the
optional application request builder. This scope uses that existing integration
pattern rather than adding a credential-source mechanism or generic override
language.
#### Consumer API and request construction
Use a narrow ckprompt-owned envelope:
```go
type SmokeRequest struct {
Selection ckconfig.Selection
Request promptkit.SmokeTestRequest
}
```
An independently optional `Suite.SmokeTest` callback receives context,
`SmokeRequest`, typed application input, and shared state, and returns
`(commandkit.Acquired[*promptkit.SmokeTestReport], error)`. It represents one
concrete test and preserves both a returned native report and error.
Provide `WithSmokeTesting(suite, load, build)` following the existing
request-validation adapter pattern. The loader uses the existing engine-loader
signature. The optional builder receives context, `SmokeRequest`, typed input,
and shared state, and returns a complete native `SmokeTestRequest` and error.
With no builder, forward the CLI-created native request. With a builder, its
returned request is authoritative; do not merge CLI values back over it.
For aggregate execution, invoke the optional builder separately for each
concrete kind, seeding that kind into the otherwise common bound request.
Applications can supply per-kind credentials, structured parameters, or other
native request choices. The synthetic `all` value does not reach the native
request or the per-test builder.
#### Aggregate execution
`app promptkit test all` selects these four tests sequentially, in order:
1. `reachability`
2. `authentication`
3. `text_inference`
4. `structured_output`
Coordinate the concrete test callbacks within one CommandKit invocation, using
its configuration selection, prepared typed input, and shared state. Reuse
configuration acquisition and the engine where practical within the existing
loader model; this does not require a new caching framework or guarantee a
particular loader call count.
Continue to the next kind after an individual test error so that remaining
results can be collected. When the invocation is canceled, stop scheduling
further tests and retain the results already available. PromptKit owns each
individual operation; CommandKit coordinates the selected sequence.
#### Reports and invocation outcomes
Present the native `SmokeTestReport`, including observations, timing, attempts,
and available usage. For `all`, deliver one YAML report containing the
individual results in execution order, with each selected kind's returned
report and associated error, when present. Retain useful partial reports
returned with errors. Preserve supported error identity through the invocation
outcome as well as presenting the errors.
For a single test, a nil native error and successful report delivery mean
success. For `all`, success requires all selected calls to return without errors
and reporting to succeed. Native invocation errors use the normal operational
failure classification; cancellation and delivery failures retain their existing
outcome handling. Native negative or unknown findings remain reported facts;
CommandKit does not derive an additional readiness verdict or failure rule from
them.
Use a focused report-delivery path that preserves a returned report alongside
its native error, including cancellation. The existing success-only inspection
writer cannot be reused unchanged for that case. Adapt delivery within the
current outcome model; this work does not require a general diagnostics
framework.
framework. Describe returned data neutrally, without adding warnings about
perceived upstream limitations.
Smoke testing runs when its command is explicitly invoked. Configuration,
requirements, help, and release operations retain their own existing behavior.
@@ -305,8 +424,9 @@ ID-only prompt requests, default/explicit listing filters, and smoke requests.
Verify faithful representation of upstream-returned source evidence, optional
labels, inventories, diagnostics, and errors, together with CommandKit's optional
command availability, invocation lifetimes, outcome classification, and output
delivery. Include smoke report-plus-error delivery and error identity in this
boundary coverage.
delivery. Include smoke request bindings, per-kind builder forwarding, aggregate
order and continuation, cancellation, report-plus-error delivery, and error
identity in this boundary coverage.
These checks establish CommandKit's access to the capabilities. They should not
test whether ConfigKit correctly places or composes sources, checks versions,
@@ -394,5 +514,7 @@ operational failures with supported error identity preserved. Native observation
are reported neutrally. CommandKit does not add an aggregate readiness judgment
or warnings about perceived upstream limitations.
The implementation plan must settle concrete command vocabulary and request
bindings within this scope. A separate smoke feature roadmap is not required.
The [explicit smoke testing target](#explicit-smoke-testing) records the accepted
`promptkit test KIND` and `promptkit test all` vocabulary, native request
bindings, per-kind builder, and aggregate execution/reporting semantics. A
separate smoke feature roadmap is not required.