base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Migrate PromptKit reports to shared rendering

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-07 04:10:47 +00:00
parent 5ff0dd4ea2
commit 6378ceb3c5
15 changed files with 249 additions and 265 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
docs/cli.md
View File

@@ -577,7 +577,7 @@ The command accepts:
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. | | `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. |
| `--session-id` | string | empty | Pass session context to PromptKit. | | `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. | | `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. |
| `--format` | `yaml` | `yaml` | Select the requirements report format. Other formats are usage errors for this operation. | | `--format` | `text`, `yaml`, `json` | automatic | Select the requirements report format; automatic and text use YAML for this structured report. |
The `promptkit` namespace identifies the operation, so there is no built-in The `promptkit` namespace identifies the operation, so there is no built-in
`--scope` flag. An explicitly empty profile or session value retains PromptKit's `--scope` flag. An explicitly empty profile or session value retains PromptKit's
@@ -594,7 +594,7 @@ values, whitespace, commas, and additional `=` characters are retained. All
occurrences are parsed; duplicate keys use the last value, while any malformed occurrences are parsed; duplicate keys use the last value, while any malformed
occurrence is a usage error even if a later assignment is valid. occurrence is a usage error even if a later assignment is valid.
The YAML report contains selected identities, native completeness, and input, The report contains selected identities, native completeness, and input,
variable, and issue inventories. False values and empty inventories are variable, and issue inventories. False values and empty inventories are
retained. Input metadata preserves null versus an empty string. Variable entries retained. Input metadata preserves null versus an empty string. Variable entries
contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native
@@ -659,7 +659,7 @@ The command accepts:
| `--session-id` | string | empty | Pass session context to PromptKit. | | `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. | | `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. |
| `--input` | repeatable `NAME=PATH` | none | Bind a file-backed PromptKit artifact. | | `--input` | repeatable `NAME=PATH` | none | Bind a file-backed PromptKit artifact. |
| `--format` | `yaml` | `yaml` | Select the success confirmation format. Other formats are usage errors. | | `--format` | `text`, `yaml`, `json` | automatic | Select the success confirmation format; automatic and text use YAML for this structured report. |
The built-in flags other than repeatable `--var` and `--input` are singleton The built-in flags other than repeatable `--var` and `--input` are singleton
bindings; when repeated, the last value is used. Explicit empty `--config` or bindings; when repeated, the last value is used. Explicit empty `--config` or
@@ -674,7 +674,7 @@ native validation. File paths become `promptkit.File(PATH)` references without
being read, normalized, expanded, or rebased during argument binding. Duplicate being read, normalized, expanded, or rebased during argument binding. Duplicate
keys or names use the last value, but every occurrence is parsed. keys or names use the last value, but every occurrence is parsed.
On success, the YAML confirmation contains only the requested prompt ID and On success, the confirmation contains only the requested prompt ID and
`valid: true`: `valid: true`:
```yaml ```yaml
@@ -712,8 +712,10 @@ example promptkit list collections
``` ```
All catalogs accept `--config`, `--overlay`, applicable flags from All catalogs accept `--config`, `--overlay`, applicable flags from
`Suite.Selectors`, and `--format yaml` (the default). Other formats are usage `Suite.Selectors`, and the shared `--format` flag. Automatic output uses text
errors. Prompt, profile, and backend listings accept namespace selection: for nonempty string lists and YAML for other inventories; explicit `text` uses
the same rendering for structured inventories. Prompt, profile, and backend
listings accept namespace selection:
| Binding | Effect | | Binding | Effect |
| --- | --- | | --- | --- |
@@ -729,8 +731,8 @@ before provider work. Namespace inventory accepts repeatable `--kind KIND`;
omitting `--kind` forwards PromptKit's native default kind selection. Repeated omitting `--kind` forwards PromptKit's native default kind selection. Repeated
values are forwarded in order. Collections have no namespace or kind filter. values are forwarded in order. Collections have no namespace or kind filter.
Each operation writes one bare YAML sequence, retaining provider order and Each operation reports one bare sequence, retaining provider order and native
native identities: identities. YAML examples follow:
- Prompt entries are `{id}`. - Prompt entries are `{id}`.
- Profiles, backends, and namespaces are strings. - Profiles, backends, and namespaces are strings.
@@ -785,7 +787,7 @@ example promptkit test authentication --backend local
| `--reasoning-effort` | string | omitted | Override execution reasoning effort. | | `--reasoning-effort` | string | omitted | Override execution reasoning effort. |
| `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. | | `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. |
| `--transient-retries` | integer | omitted | Set the native inference transient retry count. | | `--transient-retries` | integer | omitted | Set the native inference transient retry count. |
| `--format` | `yaml` | `yaml` | Select the report format. Other formats are usage errors. | | `--format` | `text`, `yaml`, `json` | automatic | Select the report format; automatic and text use YAML for these structured reports. |
The execution flags populate PromptKit's native execution override. The The execution flags populate PromptKit's native execution override. The
`--api-key-env` value is a variable name, not a credential value. Application `--api-key-env` value is a variable name, not a credential value. Application
@@ -795,7 +797,7 @@ builder forwards the seed; a supplied builder's complete native request is
authoritative. The helper retains a native report when PromptKit returns both a authoritative. The helper retains a native report when PromptKit returns both a
report and an error. report and an error.
A single report projects the native smoke report into YAML. It preserves native A single report projects the native smoke report. It preserves native
false, unknown/null, and incomplete values and does not add a readiness score. false, unknown/null, and incomplete values and does not add a readiness score.
The report includes `kind`, `supported`, `applicable`, `completed`, `reason`, The report includes `kind`, `supported`, `applicable`, `completed`, `reason`,
`target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target `target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target
@@ -865,7 +867,7 @@ usage:
failure: "" failure: ""
``` ```
Aggregate output is one YAML mapping with ordered `tests` entries. Each entry Aggregate output is one mapping with ordered `tests` entries. Each entry
has selected `kind`, nullable `report`, and optional `error` text. The inner has selected `kind`, nullable `report`, and optional `error` text. The inner
report keeps its native `kind`. This cancellation example shows two attempted report keeps its native `kind`. This cancellation example shows two attempted
operations with no reports; later kinds are absent because cancellation stops operations with no reports; later kinds are absent because cancellation stops

6
docs/roadmap/implementation.md
View File

@@ -1,6 +1,6 @@
# Shared reporting and PromptKit v0.20.0 implementation plan # Shared reporting and PromptKit v0.20.0 implementation plan
Status: implementation in progress; Stages 1–8 complete. Status: implementation in progress; Stages 1–9 complete.
## Scope, authority, and execution ## Scope, authority, and execution
@@ -323,6 +323,10 @@ native evidence rather than rechecking upstream inheritance or rendering.
## Stage 9: Consolidate remaining ckprompt reports ## Stage 9: Consolidate remaining ckprompt reports
**Complete.** Requirements, catalogs, and smoke reports use shared rendering.
The root, ckconfig, and ckprompt workspace suites passed with Go 1.27.0,
including smoke evidence, cancellation, quiet, and delivery-failure tests.
Move requirements show/validate, all catalogs, and smoke reports onto core Move requirements show/validate, all catalogs, and smoke reports onto core
rendering. Remove remaining local format bindings, allowlists, encoders, and rendering. Remove remaining local format bindings, allowlists, encoders, and
format-specific report paths. Preserve selected report content with JSON tags format-specific report paths. Preserve selected report content with JSON tags

146
integrations/ckprompt/command.go
View File

@@ -1,17 +1,14 @@
package ckprompt package ckprompt
import ( import (
"bytes"
"context" "context"
"errors" "errors"
"fmt" "fmt"
"io"
"strings" "strings"
"go.maximumdirect.net/ckconfig" "go.maximumdirect.net/ckconfig"
commandkit "go.maximumdirect.net/commandkit" commandkit "go.maximumdirect.net/commandkit"
promptkit "go.maximumdirect.net/promptkit" promptkit "go.maximumdirect.net/promptkit"
"go.yaml.in/yaml/v4"
) )
type definitionInput[I any] struct { type definitionInput[I any] struct {
@@ -27,7 +24,6 @@ type catalogInput[I any] struct {
allNamespaces commandkit.Supplied[bool] allNamespaces commandkit.Supplied[bool]
kinds []promptkit.ResourceKind kinds []promptkit.ResourceKind
filter promptkit.NamespaceFilter filter promptkit.NamespaceFilter
format string
captured commandkit.FlagValues[I] captured commandkit.FlagValues[I]
selected I selected I
} }
@@ -35,7 +31,6 @@ type catalogInput[I any] struct {
type requirementsInput[I any] struct { type requirementsInput[I any] struct {
selection ckconfig.Selection selection ckconfig.Selection
inspection promptkit.RequirementInspectionRequest inspection promptkit.RequirementInspectionRequest
format string
captured commandkit.FlagValues[I] captured commandkit.FlagValues[I]
selected I selected I
} }
@@ -138,8 +133,6 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Config = value }), func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Config = value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "overlay", Usage: "Configuration overlay path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString, commandkit.ValueFlag(commandkit.FlagMeta{Name: "overlay", Usage: "Configuration overlay path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Overlay = value }), func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Overlay = value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Catalog format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
) )
if name == "prompts" || name == "profiles" || name == "backends" { if name == "prompts" || name == "profiles" || name == "backends" {
flags = append(flags, flags = append(flags,
@@ -169,9 +162,6 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
if input == nil { if input == nil {
return errors.New("PromptKit catalog input must not be nil") return errors.New("PromptKit catalog input must not be nil")
} }
if err := checkCatalogFormat(input.format); err != nil {
return err
}
if err := prepareCatalogFilter(input, name); err != nil { if err := prepareCatalogFilter(input, name); err != nil {
return err return err
} }
@@ -185,21 +175,21 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
Run: func(invocation commandkit.Invocation, input *catalogInput[I]) commandkit.Outcome { Run: func(invocation commandkit.Invocation, input *catalogInput[I]) commandkit.Outcome {
switch name { switch name {
case "prompts": case "prompts":
return runCatalogList(invocation, suite, input, name, suite.ListPrompts, encodePromptCatalog) return runCatalogList(invocation, suite, input, name, suite.ListPrompts, promptCatalogReport)
case "profiles": case "profiles":
return runCatalogList(invocation, suite, input, name, suite.ListProfiles, encodeProfileCatalog) return runCatalogList(invocation, suite, input, name, suite.ListProfiles, stringCatalogReport)
case "backends": case "backends":
return runCatalogList(invocation, suite, input, name, suite.ListBackends, encodeProfileCatalog) return runCatalogList(invocation, suite, input, name, suite.ListBackends, stringCatalogReport)
case "namespaces": case "namespaces":
list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]string], error) { list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]string], error) {
return suite.ListNamespaces(ctx, NamespaceRequest{Selection: request.Selection, Kinds: input.kinds}, value, state) return suite.ListNamespaces(ctx, NamespaceRequest{Selection: request.Selection, Kinds: input.kinds}, value, state)
} }
return runCatalogList(invocation, suite, input, name, list, encodeProfileCatalog) return runCatalogList(invocation, suite, input, name, list, stringCatalogReport)
case "collections": case "collections":
list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]promptkit.CollectionInfo], error) { list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]promptkit.CollectionInfo], error) {
return suite.ListCollections(ctx, request.Selection, value, state) return suite.ListCollections(ctx, request.Selection, value, state)
} }
return runCatalogList(invocation, suite, input, name, list, encodeCollectionCatalog) return runCatalogList(invocation, suite, input, name, list, collectionCatalogReport)
default: default:
return definitionFailure("Unable to list the selected PromptKit catalog.", commandkit.Explanation{}, fmt.Errorf("unknown PromptKit catalog %q", name)) return definitionFailure("Unable to list the selected PromptKit catalog.", commandkit.Explanation{}, fmt.Errorf("unknown PromptKit catalog %q", name))
} }
@@ -230,20 +220,6 @@ func prepareCatalogFilter[I any](input *catalogInput[I], name string) error {
return nil return nil
} }
func checkCatalogFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported catalog format %q", format)
}
return nil
}
func checkRequirementsFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported requirements format %q; expected yaml", format)
}
return nil
}
func prepareCatalogInput[I, S any](suite Suite[I, S], input *catalogInput[I]) (I, error) { func prepareCatalogInput[I, S any](suite Suite[I, S], input *catalogInput[I]) (I, error) {
return prepareSuiteInput(suite, input.selection, input.captured) return prepareSuiteInput(suite, input.selection, input.captured)
} }
@@ -313,8 +289,6 @@ func newRequirementsCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **requirementsInput[I], value commandkit.Supplied[string]) { func(input **requirementsInput[I], value commandkit.Supplied[string]) {
(*input).inspection.SessionID = value.Value (*input).inspection.SessionID = value.Value
}), }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Report format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **requirementsInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment, commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment,
func(input **requirementsInput[I], value commandkit.Supplied[[]variableAssignment]) { func(input **requirementsInput[I], value commandkit.Supplied[[]variableAssignment]) {
variables := make(map[string]string, len(value.Value)) variables := make(map[string]string, len(value.Value))
@@ -339,9 +313,6 @@ func newRequirementsCommand[I, S any](suite Suite[I, S]) commandkit.Command {
}), }),
}, },
CheckInput: func(input *requirementsInput[I]) error { CheckInput: func(input *requirementsInput[I]) error {
if err := checkRequirementsFormat(input.format); err != nil {
return err
}
selected, err := prepareSuiteInput(suite, input.selection, input.captured) selected, err := prepareSuiteInput(suite, input.selection, input.captured)
if err != nil { if err != nil {
return err return err
@@ -382,7 +353,7 @@ func runCatalogList[T any, I, S any](
input *catalogInput[I], input *catalogInput[I],
catalog string, catalog string,
list func(context.Context, CatalogRequest, I, S) (commandkit.Acquired[T], error), list func(context.Context, CatalogRequest, I, S) (commandkit.Acquired[T], error),
encode func(T) ([]byte, error), project func(T) any,
) commandkit.Outcome { ) commandkit.Outcome {
if invocation.Context != nil { if invocation.Context != nil {
if err := invocation.Context.Err(); err != nil { if err := invocation.Context.Err(); err != nil {
@@ -409,9 +380,8 @@ func runCatalogList[T any, I, S any](
return definitionFailure("PromptKit catalog listing was canceled.", acquired.Explanation, err) return definitionFailure("PromptKit catalog listing was canceled.", acquired.Explanation, err)
} }
} }
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) { _ = invocation.Reporter().Report(commandkit.Report{Value: project(acquired.Value)})
return encode(acquired.Value) return commandkit.Success()
})
} }
func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *definitionInput[I], kind string) commandkit.Outcome { func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *definitionInput[I], kind string) commandkit.Outcome {
@@ -467,35 +437,35 @@ func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite S
} }
type requirementsInspectionReport struct { type requirementsInspectionReport struct {
PromptID string `yaml:"prompt_id"` PromptID string `json:"prompt_id"`
ProfileID string `yaml:"profile_id"` ProfileID string `json:"profile_id"`
Complete bool `yaml:"complete"` Complete bool `json:"complete"`
Inputs []requirementInputReport `yaml:"inputs"` Inputs []requirementInputReport `json:"inputs"`
Variables []requirementVariableReport `yaml:"variables"` Variables []requirementVariableReport `json:"variables"`
Issues []requirementIssueReport `yaml:"issues"` Issues []requirementIssueReport `json:"issues"`
} }
type requirementInputReport struct { type requirementInputReport struct {
Name string `yaml:"name"` Name string `json:"name"`
Kind string `yaml:"kind"` Kind string `json:"kind"`
Declared bool `yaml:"declared"` Declared bool `json:"declared"`
ContentType *string `yaml:"content_type"` ContentType *string `json:"content_type"`
Description *string `yaml:"description"` Description *string `json:"description"`
Explanation string `yaml:"explanation"` Explanation string `json:"explanation"`
} }
type requirementVariableReport struct { type requirementVariableReport struct {
Name string `yaml:"name"` Name string `json:"name"`
Kind string `yaml:"kind"` Kind string `json:"kind"`
Supplied bool `yaml:"supplied"` Supplied bool `json:"supplied"`
Explanation string `yaml:"explanation"` Explanation string `json:"explanation"`
} }
type requirementIssueReport struct { type requirementIssueReport struct {
Code string `yaml:"code"` Code string `json:"code"`
InputName string `yaml:"input_name"` InputName string `json:"input_name"`
VariableName string `yaml:"variable_name"` VariableName string `json:"variable_name"`
Description string `yaml:"description"` Description string `json:"description"`
} }
func requirementsReportFromInspection(inspection *promptkit.RequirementInspection) requirementsInspectionReport { func requirementsReportFromInspection(inspection *promptkit.RequirementInspection) requirementsInspectionReport {
@@ -566,76 +536,40 @@ func runRequirementsInspection[I, S any](invocation commandkit.Invocation, suite
return definitionFailure(fmt.Sprintf("Requirements inspection for PromptKit prompt %q was canceled.", input.inspection.PromptID), acquired.Explanation, err) return definitionFailure(fmt.Sprintf("Requirements inspection for PromptKit prompt %q was canceled.", input.inspection.PromptID), acquired.Explanation, err)
} }
} }
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) { _ = invocation.Reporter().Report(commandkit.Report{Value: requirementsReportFromInspection(acquired.Value)})
return encodePromptYAMLBytes(requirementsReportFromInspection(acquired.Value))
})
}
// These reports were already prepared before delivery. Keep that boundary
// inside managed rendering, with cancellation owned by the command outcome.
func renderPreparedPromptReport(invocation commandkit.Invocation, explanation commandkit.Explanation, prepare func() ([]byte, error)) commandkit.Outcome {
var canceled error
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error {
output, err := prepare()
if err != nil {
return err
}
if canceled = promptKitContextError(invocation.Context); canceled != nil {
return nil
}
_, err = writer.Write(output)
return err
})
if canceled != nil {
return definitionFailure("PromptKit report preparation was canceled.", explanation, canceled)
}
return commandkit.Success() return commandkit.Success()
} }
func encodePromptYAML(writer io.Writer, value any) error {
encoder := yaml.NewEncoder(writer)
encoder.SetIndent(2)
return encoder.Encode(value)
}
type promptCatalogEntry struct { type promptCatalogEntry struct {
ID string `yaml:"id"` ID string `json:"id"`
} }
func encodePromptCatalog(prompts []promptkit.PromptIdentity) ([]byte, error) { func promptCatalogReport(prompts []promptkit.PromptIdentity) any {
entries := make([]promptCatalogEntry, len(prompts)) entries := make([]promptCatalogEntry, len(prompts))
for index, prompt := range prompts { for index, prompt := range prompts {
entries[index] = promptCatalogEntry{ID: prompt.ID} entries[index] = promptCatalogEntry{ID: prompt.ID}
} }
return encodePromptYAMLBytes(entries) return entries
} }
func encodeProfileCatalog(profiles []string) ([]byte, error) { func stringCatalogReport(values []string) any {
if profiles == nil { if values == nil {
profiles = []string{} return []string{}
} }
return encodePromptYAMLBytes(profiles) return values
} }
type collectionCatalogEntry struct { type collectionCatalogEntry struct {
Namespace string `yaml:"namespace"` Namespace string `json:"namespace"`
Version *string `yaml:"version,omitempty"` Version *string `json:"version,omitempty"`
} }
func encodeCollectionCatalog(collections []promptkit.CollectionInfo) ([]byte, error) { func collectionCatalogReport(collections []promptkit.CollectionInfo) any {
entries := make([]collectionCatalogEntry, len(collections)) entries := make([]collectionCatalogEntry, len(collections))
for index, collection := range collections { for index, collection := range collections {
entries[index] = collectionCatalogEntry{Namespace: collection.Namespace, Version: collection.ReleaseVersion} entries[index] = collectionCatalogEntry{Namespace: collection.Namespace, Version: collection.ReleaseVersion}
} }
return encodePromptYAMLBytes(entries) return entries
}
func encodePromptYAMLBytes(value any) ([]byte, error) {
var output bytes.Buffer
if err := encodePromptYAML(&output, value); err != nil {
return nil, err
}
return output.Bytes(), nil
} }
func queuePromptDiagnostics(reporter *commandkit.Reporter, diagnostics []commandkit.Diagnostic) { func queuePromptDiagnostics(reporter *commandkit.Reporter, diagnostics []commandkit.Diagnostic) {

26
integrations/ckprompt/command_test.go
View File

@@ -202,7 +202,7 @@ func TestRequirementsCommandUsageFailuresPrecedeApplicationPreparation(t *testin
}{ }{
{name: "missing operand", args: []string{"promptkit", "requirements", "show"}}, {name: "missing operand", args: []string{"promptkit", "requirements", "show"}},
{name: "extra operand", args: []string{"promptkit", "requirements", "show", "briefing", "extra"}}, {name: "extra operand", args: []string{"promptkit", "requirements", "show", "briefing", "extra"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "show", "briefing", "--format", "text"}}, {name: "unsupported format", args: []string{"promptkit", "requirements", "show", "briefing", "--format", "invalid"}},
{name: "malformed variable followed by valid assignment", args: []string{"promptkit", "requirements", "show", "briefing", "--var", "broken", "--var", "mode=full"}}, {name: "malformed variable followed by valid assignment", args: []string{"promptkit", "requirements", "show", "briefing", "--var", "broken", "--var", "mode=full"}},
{name: "no built-in scope", args: []string{"promptkit", "requirements", "show", "briefing", "--scope", "all"}}, {name: "no built-in scope", args: []string{"promptkit", "requirements", "show", "briefing", "--scope", "all"}},
{name: "malformed variable on help", args: []string{"help", "promptkit", "requirements", "show", "--var", "broken"}}, {name: "malformed variable on help", args: []string{"help", "promptkit", "requirements", "show", "--var", "broken"}},
@@ -402,7 +402,7 @@ func TestDiscoveryCommandsForwardFiltersAndRenderCollections(t *testing.T) {
if outcome.Kind != commandkit.OutcomeSuccess || backendRequest.Filter.Mode != promptkit.NamespaceAll { if outcome.Kind != commandkit.OutcomeSuccess || backendRequest.Filter.Mode != promptkit.NamespaceAll {
t.Fatalf("backend list outcome/filter = (%#v, %#v)", outcome, backendRequest.Filter) t.Fatalf("backend list outcome/filter = (%#v, %#v)", outcome, backendRequest.Filter)
} }
outcome, output = run("promptkit", "list", "namespaces", "--kind", "prompt", "--kind", "backend") outcome, output = run("promptkit", "list", "namespaces", "--kind", "prompt", "--kind", "backend", "--format=yaml")
if outcome.Kind != commandkit.OutcomeSuccess || !reflect.DeepEqual(namespaceRequest.Kinds, []promptkit.ResourceKind{promptkit.ResourcePrompt, promptkit.ResourceBackend}) { if outcome.Kind != commandkit.OutcomeSuccess || !reflect.DeepEqual(namespaceRequest.Kinds, []promptkit.ResourceKind{promptkit.ResourcePrompt, promptkit.ResourceBackend}) {
t.Fatalf("namespace list outcome/kinds = (%#v, %#v)", outcome, namespaceRequest.Kinds) t.Fatalf("namespace list outcome/kinds = (%#v, %#v)", outcome, namespaceRequest.Kinds)
} }
@@ -460,7 +460,7 @@ func TestCatalogListUsesPreparedInputAndPreflightsFormat(t *testing.T) {
} }
app := newPromptTestApp(t, Command(suite)) app := newPromptTestApp(t, Command(suite))
var output bytes.Buffer var output bytes.Buffer
invalid := app.Execute(context.Background(), []string{"promptkit", "list", "prompts", "--format", "text"}, commandkit.Streams{Out: &output, Err: io.Discard}) invalid := app.Execute(context.Background(), []string{"promptkit", "list", "prompts", "--format", "invalid"}, commandkit.Streams{Out: &output, Err: io.Discard})
if invalid.Kind != commandkit.OutcomeUsage || newInputCalls != 0 || newStateCalls != 0 || providerCalls != 0 || output.Len() != 0 { if invalid.Kind != commandkit.OutcomeUsage || newInputCalls != 0 || newStateCalls != 0 || providerCalls != 0 || output.Len() != 0 {
t.Fatalf("invalid format preflight = %#v, factories/providers=%d/%d/%d output=%q", invalid, newInputCalls, newStateCalls, providerCalls, output.String()) t.Fatalf("invalid format preflight = %#v, factories/providers=%d/%d/%d output=%q", invalid, newInputCalls, newStateCalls, providerCalls, output.String())
} }
@@ -800,7 +800,7 @@ func TestCatalogAndInspectionRespectSuccessfulReportingModes(t *testing.T) {
if output.Len() != 0 { if output.Len() != 0 {
t.Fatalf("quiet inspection wrote result output %q", output.String()) t.Fatalf("quiet inspection wrote result output %q", output.String())
} }
} else if !strings.Contains(output.String(), "Prompt ID: \"briefing\"") { } else if !strings.Contains(output.String(), "id: briefing") {
t.Fatalf("%s inspection omitted result: %q", mode, output.String()) t.Fatalf("%s inspection omitted result: %q", mode, output.String())
} }
}) })
@@ -1011,7 +1011,7 @@ func TestPromptCatalogYAMLPreservesSchemaStringsAndOrder(t *testing.T) {
{ID: "001"}, {ID: "001"},
{ID: " briefing "}, {ID: " briefing "},
} }
output, err := encodePromptCatalog(want) output, err := renderTestYAML(promptCatalogReport(want))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -1038,10 +1038,12 @@ func TestCatalogYAMLEncodersRenderNilAndEmptyInventoriesAsSequences(t *testing.T
name string name string
data func() ([]byte, error) data func() ([]byte, error)
}{ }{
{name: "nil prompts", data: func() ([]byte, error) { return encodePromptCatalog(nil) }}, {name: "nil prompts", data: func() ([]byte, error) { return renderTestYAML(promptCatalogReport(nil)) }},
{name: "empty prompts", data: func() ([]byte, error) { return encodePromptCatalog(make([]promptkit.PromptIdentity, 0)) }}, {name: "empty prompts", data: func() ([]byte, error) {
{name: "nil profiles", data: func() ([]byte, error) { return encodeProfileCatalog(nil) }}, return renderTestYAML(promptCatalogReport(make([]promptkit.PromptIdentity, 0)))
{name: "empty profiles", data: func() ([]byte, error) { return encodeProfileCatalog(make([]string, 0)) }}, }},
{name: "nil profiles", data: func() ([]byte, error) { return renderTestYAML(stringCatalogReport(nil)) }},
{name: "empty profiles", data: func() ([]byte, error) { return renderTestYAML(stringCatalogReport(make([]string, 0))) }},
} { } {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
output, err := test.data() output, err := test.data()
@@ -1057,7 +1059,7 @@ func TestCatalogYAMLEncodersRenderNilAndEmptyInventoriesAsSequences(t *testing.T
func TestProfileCatalogYAMLPreservesStringValuesAndOrder(t *testing.T) { func TestProfileCatalogYAMLPreservesStringValuesAndOrder(t *testing.T) {
want := []string{" default ", "001", "same", "same"} want := []string{" default ", "001", "same", "same"}
output, err := encodeProfileCatalog(want) output, err := renderTestYAML(stringCatalogReport(want))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -1077,7 +1079,7 @@ func TestProfileCatalogYAMLPreservesStringValuesAndOrder(t *testing.T) {
} }
func TestRequirementsReportRendersEmptyInventoriesAndFalseValues(t *testing.T) { func TestRequirementsReportRendersEmptyInventoriesAndFalseValues(t *testing.T) {
output, err := encodePromptYAMLBytes(requirementsReportFromInspection(&promptkit.RequirementInspection{})) output, err := renderTestYAML(requirementsReportFromInspection(&promptkit.RequirementInspection{}))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -1135,7 +1137,7 @@ func TestRequirementsReportPreservesNativeFieldsAndMetadata(t *testing.T) {
for _, complete := range []bool{true, false} { for _, complete := range []bool{true, false} {
t.Run(map[bool]string{true: "complete", false: "partial"}[complete], func(t *testing.T) { t.Run(map[bool]string{true: "complete", false: "partial"}[complete], func(t *testing.T) {
inspection.Complete = complete inspection.Complete = complete
output, err := encodePromptYAMLBytes(requirementsReportFromInspection(inspection)) output, err := renderTestYAML(requirementsReportFromInspection(inspection))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

2
integrations/ckprompt/composition_test.go
View File

@@ -225,7 +225,7 @@ profiles:
output.Reset() output.Reset()
outcome = command.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--campaign", "spring", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard}) outcome = command.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--campaign", "spring", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || appAcquisitions != 1 || engineScopeAcquisitions != 1 || loaderCalls != 2 || !bytes.Contains(output.Bytes(), []byte("profile_id: default")) { if outcome.Kind != commandkit.OutcomeSuccess || appAcquisitions != 1 || engineScopeAcquisitions != 1 || loaderCalls != 2 || !bytes.Contains(output.Bytes(), []byte("id: default")) {
t.Fatalf("independent profile run = (%#v, output=%q, counts=%d/%d/%d)", outcome, output.String(), appAcquisitions, engineScopeAcquisitions, loaderCalls) t.Fatalf("independent profile run = (%#v, output=%q, counts=%d/%d/%d)", outcome, output.String(), appAcquisitions, engineScopeAcquisitions, loaderCalls)
} }
if newInputCalls != 3 || newStateCalls != 2 { if newInputCalls != 3 || newStateCalls != 2 {

2
integrations/ckprompt/examples/composed/main_test.go
View File

@@ -537,7 +537,7 @@ func TestComposedAliasesReachCanonicalCommands(t *testing.T) {
} { } {
output.Reset() output.Reset()
outcome = app.Execute(context.Background(), args, commandkit.Streams{Out: &output, Err: io.Discard}) outcome = app.Execute(context.Background(), args, commandkit.Streams{Out: &output, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "prompt_id: briefing") { if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "id: briefing") {
t.Fatalf("prompt inspection route %q outcome/report = (%#v, %q)", args, outcome, output.String()) t.Fatalf("prompt inspection route %q outcome/report = (%#v, %q)", args, outcome, output.String())
} }
} }

13
integrations/ckprompt/render_test.go Normal file
View File

@@ -0,0 +1,13 @@
package ckprompt
import (
"bytes"
commandkit "go.maximumdirect.net/commandkit"
)
func renderTestYAML(value any) ([]byte, error) {
var output bytes.Buffer
err := commandkit.RenderReport(&output, commandkit.Report{Value: value}, commandkit.FormatYAML)
return output.Bytes(), err
}

2
integrations/ckprompt/requirements_command_test.go
View File

@@ -71,7 +71,7 @@ func executeRequirementsAgainstNative(t *testing.T, engine *promptkit.Engine, re
if err != nil { if err != nil {
t.Fatalf("direct native inspection: %v", err) t.Fatalf("direct native inspection: %v", err)
} }
expectedBytes, err := encodePromptYAMLBytes(requirementsReportFromInspection(native)) expectedBytes, err := renderTestYAML(requirementsReportFromInspection(native))
if err != nil { if err != nil {
t.Fatalf("encode direct native report: %v", err) t.Fatalf("encode direct native report: %v", err)
} }

66
integrations/ckprompt/shared_smoke_test.go Normal file
View File

@@ -0,0 +1,66 @@
package ckprompt
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"testing"
commandkit "go.maximumdirect.net/commandkit"
promptkit "go.maximumdirect.net/promptkit"
)
func TestSmokeAggregateJSONRetainsPartialEvidence(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cause := errors.New("authentication interrupted")
calls := 0
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{SmokeTest: func(_ context.Context, request SmokeRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.SmokeTestReport], error) {
calls++
report := &promptkit.SmokeTestReport{Kind: request.Request.Kind, Supported: true, Reason: promptkit.SmokeTestReasonAuthenticationResponseLimitExceeded}
if calls == 2 {
cancel()
return commandkit.Acquired[*promptkit.SmokeTestReport]{Value: report}, cause
}
return commandkit.Acquired[*promptkit.SmokeTestReport]{Value: report}, nil
}}))
var output bytes.Buffer
result := app.Execute(ctx, []string{"promptkit", "test", "all", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
dec := json.NewDecoder(&output)
var report struct {
Tests []struct {
Kind string `json:"kind"`
Error *string `json:"error"`
Report struct {
Reason string `json:"reason"`
Facts struct {
CredentialsAccepted *bool `json:"credentials_accepted"`
EndpointReachable *bool `json:"endpoint_reachable"`
} `json:"facts"`
} `json:"report"`
} `json:"tests"`
}
if err := dec.Decode(&report); err != nil {
t.Fatal(err)
}
if err := dec.Decode(new(any)); err != io.EOF {
t.Fatalf("extra JSON: %v", err)
}
if calls != 2 || !errors.Is(result.Cause, cause) || !errors.Is(result.Cause, context.Canceled) || len(report.Tests) != 2 || report.Tests[1].Error == nil || report.Tests[1].Report.Reason != string(promptkit.SmokeTestReasonAuthenticationResponseLimitExceeded) || report.Tests[1].Report.Facts.CredentialsAccepted != nil || report.Tests[1].Report.Facts.EndpointReachable != nil {
t.Fatalf("result=%#v calls=%d report=%#v", result, calls, report)
}
}
func TestSmokePreparationErrorDoesNotInventReport(t *testing.T) {
cause := errors.New("request preparation failed")
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{SmokeTest: func(context.Context, SmokeRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.SmokeTestReport], error) {
return commandkit.Acquired[*promptkit.SmokeTestReport]{}, cause
}}))
var output bytes.Buffer
result := app.Execute(t.Context(), []string{"promptkit", "test", "authentication", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
if !errors.Is(result.Cause, cause) || output.Len() != 0 || result.Presentation.Output.State != commandkit.DeliveryNone {
t.Fatalf("result=%#v output=%q", result, output.String())
}
}

33
integrations/ckprompt/smoke_command.go
View File

@@ -15,7 +15,6 @@ type smokeCommandInput[I any] struct {
selection ckconfig.Selection selection ckconfig.Selection
kind string kind string
request promptkit.SmokeTestRequest request promptkit.SmokeTestRequest
format string
captured commandkit.FlagValues[I] captured commandkit.FlagValues[I]
selected I selected I
} }
@@ -36,8 +35,6 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) { func(input **smokeCommandInput[I], value commandkit.Supplied[string]) {
(*input).selection.Overlay = value (*input).selection.Overlay = value
}), }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Smoke report format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "backend", Usage: "PromptKit backend ID."}, (*string)(nil), parseDefinitionString, commandkit.ValueFlag(commandkit.FlagMeta{Name: "backend", Usage: "PromptKit backend ID."}, (*string)(nil), parseDefinitionString,
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) { func(input **smokeCommandInput[I], value commandkit.Supplied[string]) {
(*input).request.BackendID = value.Value (*input).request.BackendID = value.Value
@@ -134,7 +131,7 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
Name: "test", Name: "test",
Summary: "Run one PromptKit smoke test.", Summary: "Run one PromptKit smoke test.",
NewInput: func() *smokeCommandInput[I] { NewInput: func() *smokeCommandInput[I] {
return &smokeCommandInput[I]{format: "yaml"} return &smokeCommandInput[I]{}
}, },
Flags: flags, Flags: flags,
Arguments: []commandkit.Argument[*smokeCommandInput[I]]{ Arguments: []commandkit.Argument[*smokeCommandInput[I]]{
@@ -145,9 +142,6 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
if input == nil { if input == nil {
return errors.New("PromptKit smoke-test input must not be nil") return errors.New("PromptKit smoke-test input must not be nil")
} }
if err := checkSmokeFormat(input.format); err != nil {
return err
}
if input.kind != "all" { if input.kind != "all" {
input.request.Kind = promptkit.SmokeTestKind(input.kind) input.request.Kind = promptkit.SmokeTestKind(input.kind)
} }
@@ -175,13 +169,6 @@ func parseSmokeFloat(value string) (float64, error) {
return strconv.ParseFloat(value, 64) return strconv.ParseFloat(value, 64)
} }
func checkSmokeFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported smoke report format %q; expected yaml", format)
}
return nil
}
func runSmokeTest[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *smokeCommandInput[I]) commandkit.Outcome { func runSmokeTest[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *smokeCommandInput[I]) commandkit.Outcome {
if err := promptKitContextError(invocation.Context); err != nil { if err := promptKitContextError(invocation.Context); err != nil {
return definitionFailure("PromptKit smoke testing was canceled.", commandkit.Explanation{}, err) return definitionFailure("PromptKit smoke testing was canceled.", commandkit.Explanation{}, err)
@@ -214,14 +201,7 @@ func runSmokeTest[I, S any](invocation commandkit.Invocation, suite Suite[I, S],
return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr) return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr)
} }
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error { _ = invocation.Reporter().Report(commandkit.Report{Value: smokeReportFromNative(acquired.Value)})
output, err := encodeSmokeReport(acquired.Value)
if err != nil {
return err
}
_, err = writer.Write(output)
return err
})
if operationErr != nil { if operationErr != nil {
return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr) return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr)
} }
@@ -281,14 +261,7 @@ func runSmokeAggregate[I, S any](invocation commandkit.Invocation, suite Suite[I
if len(results) == 0 { if len(results) == 0 {
return definitionFailure("PromptKit aggregate smoke testing did not produce any results.", combineSmokeExplanations(explanations), errors.Join(operationErrors...)) return definitionFailure("PromptKit aggregate smoke testing did not produce any results.", combineSmokeExplanations(explanations), errors.Join(operationErrors...))
} }
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error { _ = invocation.Reporter().Report(commandkit.Report{Value: smokeResultsReport(results)})
output, err := encodeSmokeResults(results)
if err != nil {
return err
}
_, err = writer.Write(output)
return err
})
if len(operationErrors) > 0 { if len(operationErrors) > 0 {
return definitionFailure("One or more PromptKit smoke tests failed.", combineSmokeExplanations(explanations), errors.Join(operationErrors...)) return definitionFailure("One or more PromptKit smoke tests failed.", combineSmokeExplanations(explanations), errors.Join(operationErrors...))
} }

2
integrations/ckprompt/smoke_lifecycle_test.go
View File

@@ -78,7 +78,7 @@ func TestSmokeTestCommandUsageAndInputFailuresStopProviderWork(t *testing.T) {
{name: "malformed top-p", args: []string{"promptkit", "test", "authentication", "--top-p=high"}}, {name: "malformed top-p", args: []string{"promptkit", "test", "authentication", "--top-p=high"}},
{name: "malformed timeout", args: []string{"promptkit", "test", "authentication", "--timeout-seconds=soon"}}, {name: "malformed timeout", args: []string{"promptkit", "test", "authentication", "--timeout-seconds=soon"}},
{name: "malformed retries", args: []string{"promptkit", "test", "authentication", "--transient-retries=several"}}, {name: "malformed retries", args: []string{"promptkit", "test", "authentication", "--transient-retries=several"}},
{name: "unsupported format", args: []string{"promptkit", "test", "authentication", "--format=json"}}, {name: "unsupported format", args: []string{"promptkit", "test", "authentication", "--format=invalid"}},
{name: "unknown flag", args: []string{"promptkit", "test", "authentication", "--unknown=value"}}, {name: "unknown flag", args: []string{"promptkit", "test", "authentication", "--unknown=value"}},
} { } {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {

163
integrations/ckprompt/smoke_report.go
View File

@@ -7,100 +7,106 @@ import (
) )
type smokeReportDTO struct { type smokeReportDTO struct {
Kind promptkit.SmokeTestKind `yaml:"kind"` Kind promptkit.SmokeTestKind `json:"kind"`
Supported bool `yaml:"supported"` Supported bool `json:"supported"`
Applicable *bool `yaml:"applicable"` Applicable *bool `json:"applicable"`
Completed bool `yaml:"completed"` Completed bool `json:"completed"`
Reason promptkit.SmokeTestReason `yaml:"reason"` Reason promptkit.SmokeTestReason `json:"reason"`
Target smokeTargetDTO `yaml:"target"` Target smokeTargetDTO `json:"target"`
Facts smokeFactsDTO `yaml:"facts"` Facts smokeFactsDTO `json:"facts"`
Timing smokeTimingDTO `yaml:"timing"` Timing smokeTimingDTO `json:"timing"`
Attempts []smokeAttemptDTO `yaml:"attempts"` Attempts []smokeAttemptDTO `json:"attempts"`
Usage smokeUsageDTO `yaml:"usage"` Usage smokeUsageDTO `json:"usage"`
Failure promptkit.ExecutionFailureKind `yaml:"failure"` Failure promptkit.ExecutionFailureKind `json:"failure"`
} }
type smokeTargetDTO struct { type smokeTargetDTO struct {
BackendID string `yaml:"backend_id"` BackendID string `json:"backend_id"`
ProfileID string `yaml:"profile_id"` ProfileID string `json:"profile_id"`
PromptID string `yaml:"prompt_id"` PromptID string `json:"prompt_id"`
DefaultSourceBackendID string `yaml:"default_source_backend_id"` DefaultSourceBackendID string `json:"default_source_backend_id"`
SelectionSource string `yaml:"selection_source"` SelectionSource string `json:"selection_source"`
Endpoint string `yaml:"endpoint"` Endpoint string `json:"endpoint"`
EffectiveSettings smokeExecutionTargetDTO `yaml:"effective_settings"` EffectiveSettings smokeExecutionTargetDTO `json:"effective_settings"`
SettingsPresence smokeSettingsPresenceDTO `yaml:"settings_presence"` SettingsPresence smokeSettingsPresenceDTO `json:"settings_presence"`
} }
type smokeSettingsPresenceDTO struct { type smokeSettingsPresenceDTO struct {
Temperature bool `yaml:"temperature"` Temperature bool `json:"temperature"`
MaxTokens bool `yaml:"max_tokens"` MaxTokens bool `json:"max_tokens"`
TopP bool `yaml:"top_p"` TopP bool `json:"top_p"`
TimeoutSeconds bool `yaml:"timeout_seconds"` TimeoutSeconds bool `json:"timeout_seconds"`
} }
type smokeExecutionTargetDTO struct { type smokeExecutionTargetDTO struct {
BackendID string `yaml:"backend_id,omitempty"` BackendID string `json:"backend_id,omitempty"`
Endpoint string `yaml:"endpoint"` Endpoint string `json:"endpoint"`
Model string `yaml:"model"` Model string `json:"model"`
Temperature float64 `yaml:"temperature"` Temperature float64 `json:"temperature"`
MaxTokens int `yaml:"max_tokens"` MaxTokens int `json:"max_tokens"`
TopP float64 `yaml:"top_p"` TopP float64 `json:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"` TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `yaml:"service_tier"` ServiceTier string `json:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort"` ReasoningEffort string `json:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"` APIKeyEnv string `json:"api_key_env"`
ExtraParams map[string]any `yaml:"extra_params"` ExtraParams map[string]any `json:"extra_params"`
} }
type smokeFactsDTO struct { type smokeFactsDTO struct {
CredentialsSupplied *bool `yaml:"credentials_supplied"` CredentialsSupplied *bool `json:"credentials_supplied"`
EndpointReachable *bool `yaml:"endpoint_reachable"` EndpointReachable *bool `json:"endpoint_reachable"`
CredentialsAccepted *bool `yaml:"credentials_accepted"` CredentialsAccepted *bool `json:"credentials_accepted"`
ResponseReceived *bool `yaml:"response_received"` ResponseReceived *bool `json:"response_received"`
GenerationCompleted *bool `yaml:"generation_completed"` GenerationCompleted *bool `json:"generation_completed"`
TextReturned *bool `yaml:"text_returned"` TextReturned *bool `json:"text_returned"`
ExpectedTextMatched *bool `yaml:"expected_text_matched"` ExpectedTextMatched *bool `json:"expected_text_matched"`
StructuredOutputRequested *bool `yaml:"structured_output_requested"` StructuredOutputRequested *bool `json:"structured_output_requested"`
ValidJSON *bool `yaml:"valid_json"` ValidJSON *bool `json:"valid_json"`
SchemaMatched *bool `yaml:"schema_matched"` SchemaMatched *bool `json:"schema_matched"`
} }
type smokeTimingDTO struct { type smokeTimingDTO struct {
StartedAt time.Time `yaml:"started_at"` StartedAt time.Time `json:"started_at"`
Duration string `yaml:"duration"` Duration string `json:"duration"`
CapacityWait *string `yaml:"capacity_wait"` CapacityWait *string `json:"capacity_wait"`
RequestDuration *string `yaml:"request_duration"` RequestDuration *string `json:"request_duration"`
RetryDelay *string `yaml:"retry_delay"` RetryDelay *string `json:"retry_delay"`
EvaluationDuration *string `yaml:"evaluation_duration"` EvaluationDuration *string `json:"evaluation_duration"`
} }
type smokeAttemptDTO struct { type smokeAttemptDTO struct {
Dispatched bool `yaml:"dispatched"` Dispatched bool `json:"dispatched"`
InitialEndpoint string `yaml:"initial_endpoint"` InitialEndpoint string `json:"initial_endpoint"`
Endpoint string `yaml:"endpoint"` Endpoint string `json:"endpoint"`
HTTPStatus *int `yaml:"http_status"` HTTPStatus *int `json:"http_status"`
Facts smokeFactsDTO `yaml:"facts"` Facts smokeFactsDTO `json:"facts"`
Timing smokeTimingDTO `yaml:"timing"` Timing smokeTimingDTO `json:"timing"`
Failure promptkit.ExecutionFailureKind `yaml:"failure"` Failure promptkit.ExecutionFailureKind `json:"failure"`
Usage smokeUsageDTO `yaml:"usage"` Usage smokeUsageDTO `json:"usage"`
} }
type smokeUsageDTO struct { type smokeUsageDTO struct {
PromptTokens promptkit.ObservedTokenCount `yaml:"prompt_tokens"` PromptTokens smokeTokenCount `json:"prompt_tokens"`
CompletionTokens promptkit.ObservedTokenCount `yaml:"completion_tokens"` CompletionTokens smokeTokenCount `json:"completion_tokens"`
TotalTokens promptkit.ObservedTokenCount `yaml:"total_tokens"` TotalTokens smokeTokenCount `json:"total_tokens"`
CachedTokens promptkit.ObservedTokenCount `yaml:"cached_tokens"` CachedTokens smokeTokenCount `json:"cached_tokens"`
CacheWriteTokens promptkit.ObservedTokenCount `yaml:"cache_write_tokens"` CacheWriteTokens smokeTokenCount `json:"cache_write_tokens"`
}
type smokeTokenCount struct {
Value int `json:"value"`
Known bool `json:"known"`
Complete bool `json:"complete"`
} }
type smokeAggregateDTO struct { type smokeAggregateDTO struct {
Tests []smokeResultDTO `yaml:"tests"` Tests []smokeResultDTO `json:"tests"`
} }
type smokeResultDTO struct { type smokeResultDTO struct {
Kind promptkit.SmokeTestKind `yaml:"kind"` Kind promptkit.SmokeTestKind `json:"kind"`
Report *smokeReportDTO `yaml:"report"` Report *smokeReportDTO `json:"report"`
Error *string `yaml:"error,omitempty"` Error *string `json:"error,omitempty"`
} }
type smokeResult struct { type smokeResult struct {
@@ -109,23 +115,16 @@ type smokeResult struct {
Err error Err error
} }
func encodeSmokeReport(report *promptkit.SmokeTestReport) ([]byte, error) { func smokeResultsReport(results []smokeResult) smokeAggregateDTO {
return encodePromptYAMLBytes(smokeReportFromNative(report))
}
func encodeSmokeResults(results []smokeResult) ([]byte, error) {
entries := make([]smokeResultDTO, len(results)) entries := make([]smokeResultDTO, len(results))
for index, result := range results { for index, result := range results {
entries[index] = smokeResultDTO{ entries[index] = smokeResultDTO{Kind: result.Kind, Report: smokeReportFromNative(result.Report)}
Kind: result.Kind,
Report: smokeReportFromNative(result.Report),
}
if result.Err != nil { if result.Err != nil {
message := result.Err.Error() message := result.Err.Error()
entries[index].Error = &message entries[index].Error = &message
} }
} }
return encodePromptYAMLBytes(smokeAggregateDTO{Tests: entries}) return smokeAggregateDTO{Tests: entries}
} }
func smokeReportFromNative(report *promptkit.SmokeTestReport) *smokeReportDTO { func smokeReportFromNative(report *promptkit.SmokeTestReport) *smokeReportDTO {
@@ -224,10 +223,10 @@ func durationString(duration *time.Duration) *string {
func smokeUsageFromNative(usage promptkit.ObservedTokenUsage) smokeUsageDTO { func smokeUsageFromNative(usage promptkit.ObservedTokenUsage) smokeUsageDTO {
return smokeUsageDTO{ return smokeUsageDTO{
PromptTokens: usage.PromptTokens, PromptTokens: smokeTokenCount(usage.PromptTokens),
CompletionTokens: usage.CompletionTokens, CompletionTokens: smokeTokenCount(usage.CompletionTokens),
TotalTokens: usage.TotalTokens, TotalTokens: smokeTokenCount(usage.TotalTokens),
CachedTokens: usage.CachedTokens, CachedTokens: smokeTokenCount(usage.CachedTokens),
CacheWriteTokens: usage.CacheWriteTokens, CacheWriteTokens: smokeTokenCount(usage.CacheWriteTokens),
} }
} }

8
integrations/ckprompt/smoke_report_test.go
View File

@@ -84,7 +84,7 @@ func TestSmokeReportYAMLProjectsNativeFieldsAndPresence(t *testing.T) {
Failure: promptkit.ExecutionFailureTransport, Failure: promptkit.ExecutionFailureTransport,
} }
encoded, err := encodeSmokeReport(report) encoded, err := renderTestYAML(smokeReportFromNative(report))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -145,7 +145,7 @@ func TestSmokeReportYAMLProjectsNativeFieldsAndPresence(t *testing.T) {
} }
func TestSmokeReportYAMLEmitsNilAttemptsAsEmptySequence(t *testing.T) { func TestSmokeReportYAMLEmitsNilAttemptsAsEmptySequence(t *testing.T) {
encoded, err := encodeSmokeReport(&promptkit.SmokeTestReport{Attempts: nil}) encoded, err := renderTestYAML(smokeReportFromNative(&promptkit.SmokeTestReport{Attempts: nil}))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -161,11 +161,11 @@ func TestSmokeReportYAMLEmitsNilAttemptsAsEmptySequence(t *testing.T) {
func TestSmokeAggregateYAMLRetainsOrderErrorAndNativeReportKind(t *testing.T) { func TestSmokeAggregateYAMLRetainsOrderErrorAndNativeReportKind(t *testing.T) {
operationErr := errors.New("operation failed") operationErr := errors.New("operation failed")
report := &promptkit.SmokeTestReport{Kind: promptkit.SmokeTestReachability, Attempts: []promptkit.SmokeTestAttempt{}} report := &promptkit.SmokeTestReport{Kind: promptkit.SmokeTestReachability, Attempts: []promptkit.SmokeTestAttempt{}}
encoded, err := encodeSmokeResults([]smokeResult{ encoded, err := renderTestYAML(smokeResultsReport([]smokeResult{
{Kind: promptkit.SmokeTestTextInference, Report: report, Err: operationErr}, {Kind: promptkit.SmokeTestTextInference, Report: report, Err: operationErr},
{Kind: promptkit.SmokeTestAuthentication}, {Kind: promptkit.SmokeTestAuthentication},
{Kind: promptkit.SmokeTestStructuredOutput, Report: report, Err: errors.New("")}, {Kind: promptkit.SmokeTestStructuredOutput, Report: report, Err: errors.New("")},
}) }))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

19
integrations/ckprompt/validation_command.go
View File

@@ -11,14 +11,13 @@ import (
type validationCommandInput[I any] struct { type validationCommandInput[I any] struct {
promptID string promptID string
request ValidationRequest request ValidationRequest
format string
captured commandkit.FlagValues[I] captured commandkit.FlagValues[I]
selected I selected I
} }
type requestValidationReport struct { type requestValidationReport struct {
PromptID string `yaml:"prompt_id"` PromptID string `json:"prompt_id"`
Valid bool `yaml:"valid"` Valid bool `json:"valid"`
} }
func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command { func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
@@ -45,10 +44,6 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **validationCommandInput[I], value commandkit.Supplied[string]) { func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
(*input).request.Request.SessionID = value.Value (*input).request.Request.SessionID = value.Value
}), }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Confirmation format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
(*input).format = value.Value
}),
commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment, commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment,
func(input **validationCommandInput[I], value commandkit.Supplied[[]variableAssignment]) { func(input **validationCommandInput[I], value commandkit.Supplied[[]variableAssignment]) {
variables := make(map[string]string, len(value.Value)) variables := make(map[string]string, len(value.Value))
@@ -71,7 +66,7 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
Name: "validate", Name: "validate",
Summary: "Validate a concrete PromptKit request without executing it.", Summary: "Validate a concrete PromptKit request without executing it.",
NewInput: func() *validationCommandInput[I] { NewInput: func() *validationCommandInput[I] {
return &validationCommandInput[I]{format: "yaml"} return &validationCommandInput[I]{}
}, },
Flags: flags, Flags: flags,
Arguments: []commandkit.Argument[*validationCommandInput[I]]{ Arguments: []commandkit.Argument[*validationCommandInput[I]]{
@@ -85,9 +80,6 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
if input == nil { if input == nil {
return errors.New("PromptKit validation input must not be nil") return errors.New("PromptKit validation input must not be nil")
} }
if err := checkRequirementsFormat(input.format); err != nil {
return err
}
selected, err := prepareSuiteInput(suite, input.request.Selection, input.captured) selected, err := prepareSuiteInput(suite, input.request.Selection, input.captured)
if err != nil { if err != nil {
return err return err
@@ -121,7 +113,6 @@ func runRequestValidation[I, S any](invocation commandkit.Invocation, suite Suit
if err := promptKitContextError(invocation.Context); err != nil { if err := promptKitContextError(invocation.Context); err != nil {
return definitionFailure(fmt.Sprintf("Validation of the PromptKit request for prompt %q was canceled.", input.promptID), acquired.Explanation, err) return definitionFailure(fmt.Sprintf("Validation of the PromptKit request for prompt %q was canceled.", input.promptID), acquired.Explanation, err)
} }
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) { _ = invocation.Reporter().Report(commandkit.Report{Value: requestValidationReport{PromptID: input.promptID, Valid: true}})
return encodePromptYAMLBytes(requestValidationReport{PromptID: input.promptID, Valid: true}) return commandkit.Success()
})
} }

2
integrations/ckprompt/validation_command_test.go
View File

@@ -254,7 +254,7 @@ func TestValidationCommandUsageAndInputFailuresStopProviderWork(t *testing.T) {
}{ }{
{name: "missing operand", args: []string{"promptkit", "requirements", "validate"}}, {name: "missing operand", args: []string{"promptkit", "requirements", "validate"}},
{name: "extra operand", args: []string{"promptkit", "requirements", "validate", "briefing", "extra", "--campaign", "spring"}}, {name: "extra operand", args: []string{"promptkit", "requirements", "validate", "briefing", "extra", "--campaign", "spring"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--format", "text"}}, {name: "unsupported format", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--format", "invalid"}},
{name: "unknown flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--unknown", "value"}}, {name: "unknown flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--unknown", "value"}},
{name: "removed prompt version flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--version", "2"}}, {name: "removed prompt version flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--version", "2"}},
{name: "malformed variable before valid assignment", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--var", "broken", "--var", "mode=full"}}, {name: "malformed variable before valid assignment", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--var", "broken", "--var", "mode=full"}},