Add PromptKit definition inspection

This commit is contained in:
2026-10-07 14:40:41 +00:00
parent 6ad723cde1
commit 0af44a1172
7 changed files with 924 additions and 23 deletions

View File

@@ -8,7 +8,8 @@ and operand bindings, Cobra-backed routing, invocation outcomes, generated help,
application or CommandKit version reporting, configuration scope discovery, application or CommandKit version reporting, configuration scope discovery,
ConfigKit-backed validation, native effective YAML display, source inventory, ConfigKit-backed validation, native effective YAML display, source inventory,
and path explanation reports. Applications can register a PromptKit engine and path explanation reports. Applications can register a PromptKit engine
configuration scope independently of PromptKit definition commands. configuration scope independently of PromptKit definition commands, and can
install optional prompt and profile inspection commands.
## Documentation ## Documentation

View File

@@ -225,6 +225,76 @@ environment-variable name; effective and historical secret fragments remain
redacted. Empty location, inheritance, history, reason, and dependency lists are redacted. Empty location, inheritance, history, reason, and dependency lists are
serialized as `[]`. serialized as `[]`.
## PromptKit definition inspection
Applications can install prompt and profile inspection independently with
`RegisterPromptKit`. Each installed handler constructs the selected engine once
and asks PromptKit to inspect the named definition. Inspection reads the
engine's captured configuration; it does not generate a prompt, contact a
provider, or look up credentials.
Prompt definitions and profile resolution remain owned by
[PromptKit](https://pkg.go.dev/gitea.maximumdirect.net/eric/promptkit).
```text
example promptkit show prompt customer/summary --version 2 --format yaml
example promptkit show prompt customer/summary
example promptkit show profile customer-support --format yaml
```
Both commands require one ID operand and accept `--config`, `--overlay`, and
`--format text|yaml` (default `text`). Prompt inspection also accepts singleton
`--version`, defaulting to the empty string. PromptKit owns version selection:
an omitted version works for a single-version ID, while an ambiguous ID is an
operational error. These commands do not accept `--scope`; their namespace is
`promptkit`.
Prompt YAML has these fields. Input order is preserved and an empty input list is
`[]`. The prompt hash is an opaque owning-library equality value.
```yaml
prompt_id: customer/summary
prompt_version: "2"
prompt_hash: opaque-value
default_profile_id: customer-support
inputs:
- name: topic
required: true
content_type: text/plain
description: Topic to summarize
output_contract:
format: json
validation_mode: basic
schema_path: ""
repair_attempts: 2
```
Profile YAML preserves the effective values reported by PromptKit, including
zero values and nested provider parameters. `api_key_env` is only an
environment-variable name. `api_key_required` reports a direct credential
requirement; neither field reports credential availability. A nil provider
parameter map is displayed as `{}`.
```yaml
profile_id: customer-support
effective_model_params:
backend_id: local
endpoint: https://example.invalid/v1
model: support-model
temperature: 0
max_tokens: 0
top_p: 0
timeout_seconds: 600
service_tier: ""
reasoning_effort: ""
api_key_env: ""
extra_params: {}
api_key_required: true
```
Text uses labeled fields and includes nested `extra_params` values. A missing ID
is a usage error. Blank, unknown, ambiguous, and canceled selections are
operational inspection results from PromptKit and retain its error identity.
## Outcome status ## Outcome status
CommandKit usage failures use status `2`; operational failures use status `1`. CommandKit usage failures use status `2`; operational failures use status `1`.

View File

@@ -6,8 +6,8 @@ declarations, an instance-owned registry, a Cobra-backed routing and parsing
adapter, public `App.Execute` coordination, and generated help/version commands. adapter, public `App.Execute` coordination, and generated help/version commands.
Configuration suite registration, scope discovery, ConfigKit integration, Configuration suite registration, scope discovery, ConfigKit integration,
validation, native effective YAML display, source inventory, and path validation, native effective YAML display, source inventory, and path
explanation reports, and PromptKit engine configuration inspection are explanation reports, PromptKit engine configuration inspection, and optional
implemented; named prompt/profile inspection remains pending. The PromptKit prompt/profile inspection commands are implemented. The
[README](../README.md) provides project orientation; the [README](../README.md) provides project orientation; the
[documentation policy](policy/documentation.md#canonical-ownership) assigns [documentation policy](policy/documentation.md#canonical-ownership) assigns
canonical documentation owners. canonical documentation owners.

View File

@@ -1,6 +1,6 @@
# Initial CommandKit implementation plan # Initial CommandKit implementation plan
Status: implementation underway; Stages 1-12 complete and Stages 13-14 pending. Status: implementation underway; Stages 1-13 complete and Stage 14 pending.
Date: 2026-10-07. Date: 2026-10-07.
@@ -750,7 +750,7 @@ PromptKit/ConfigKit error identities. `go test ./...` passes.
## Stage 13: implement named prompt/profile inspection ## Stage 13: implement named prompt/profile inspection
**Status:** pending. **Prerequisite:** Stage 12. **Status:** complete. **Prerequisite:** Stage 12.
```go ```go
type PromptKitSuite[C any] struct { type PromptKitSuite[C any] struct {
@@ -811,6 +811,11 @@ formats, nil/empty parameters, independently installed prompt/profile capability
definition-only registration, and help without engine construction. Assert no definition-only registration, and help without engine construction. Assert no
handler enters generation rather than exercising paid/network execution paths. handler enters generation rather than exercising paid/network execution paths.
**Completion evidence:** `promptkit_test.go` exercises real captured prompt and
profile definitions, both report formats, inheritance and provider parameters,
selection errors, optional capabilities, and help without engine construction.
`go test ./...` passes.
## Stage 14: substantiate consumer fit and complete documentation ## Stage 14: substantiate consumer fit and complete documentation
**Status:** pending. **Prerequisite:** Stage 13. **Status:** pending. **Prerequisite:** Stage 13.

463
promptkit.go Normal file
View File

@@ -0,0 +1,463 @@
package commandkit
import (
"context"
"errors"
"fmt"
"io"
"strconv"
"strings"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/promptkit"
)
// PromptKitSuite declares optional PromptKit prompt and profile inspection
// commands for an App.
type PromptKitSuite[C any] struct {
// NewContext creates command-local application context when an inspection runs.
NewContext func() C
// Selectors declares context flags accepted by the installed commands. Each
// explicit scope list must include "promptkit"; an empty list applies here.
Selectors []Selector[C]
// CheckInput validates standard configuration selections and selector values.
CheckInput func(Selection, C) error
// InspectPrompt inspects an ID and exact or omitted version.
InspectPrompt func(context.Context, Selection, C, string, string) (Acquired[*promptkit.PromptInspection], error)
// InspectProfile inspects one profile ID.
InspectProfile func(context.Context, Selection, C, string) (Acquired[*promptkit.ProfileInspection], error)
}
// NewPromptKitSuite creates prompt and profile inspection callbacks backed by
// a PromptKit Engine loader. The loader runs once per selected command. Each
// callback captures configuration diagnostics before calling the corresponding
// PromptKit inspection method, and returns those diagnostics with lookup errors.
func NewPromptKitSuite[C any](
load func(context.Context, Selection, C) (*promptkit.Engine, error),
) (PromptKitSuite[C], error) {
if load == nil {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit engine loader is nil")
}
return PromptKitSuite[C]{
InspectPrompt: func(ctx context.Context, selection Selection, appContext C, id, version string) (Acquired[*promptkit.PromptInspection], error) {
engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load)
if err != nil {
return Acquired[*promptkit.PromptInspection]{Diagnostics: diagnostics}, err
}
inspection, err := engine.InspectPrompt(ctx, id, version)
return Acquired[*promptkit.PromptInspection]{Value: inspection, Diagnostics: diagnostics}, err
},
InspectProfile: func(ctx context.Context, selection Selection, appContext C, id string) (Acquired[*promptkit.ProfileInspection], error) {
engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load)
if err != nil {
return Acquired[*promptkit.ProfileInspection]{Diagnostics: diagnostics}, err
}
inspection, err := engine.InspectProfile(ctx, id)
return Acquired[*promptkit.ProfileInspection]{Value: inspection, Diagnostics: diagnostics}, err
},
}, nil
}
func loadPromptKitInspection[C any](ctx context.Context, selection Selection, appContext C, load func(context.Context, Selection, C) (*promptkit.Engine, error)) (*promptkit.Engine, promptkit.ConfigurationInspection, []Diagnostic, error) {
engine, err := load(ctx, selection, appContext)
if err != nil {
var loadErr *configkit.LoadError
if errors.As(err, &loadErr) {
return nil, promptkit.ConfigurationInspection{}, ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics()), err
}
return nil, promptkit.ConfigurationInspection{}, nil, err
}
if engine == nil {
return nil, promptkit.ConfigurationInspection{}, nil, fmt.Errorf("PromptKit configuration loader returned a nil engine")
}
inspection, err := engine.InspectConfiguration()
if err != nil {
return nil, promptkit.ConfigurationInspection{}, nil, err
}
sources := inspection.Sources()
diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics())
return engine, inspection, diagnostics, nil
}
// RegisterPromptKit installs the supplied PromptKit inspection commands under
// promptkit show. Each nonnil callback installs only its matching command. A
// suite with no inspection callbacks installs no PromptKit namespace.
func RegisterPromptKit[C any](app *App, suite PromptKitSuite[C]) error {
if suite.InspectPrompt == nil && suite.InspectProfile == nil {
return nil
}
if app == nil {
return fmt.Errorf("cannot register PromptKit commands on a nil app")
}
normalized, err := normalizePromptKitSuite(suite)
if err != nil {
return err
}
showCommands := make([]Command, 0, 2)
if normalized.InspectPrompt != nil {
command, err := newPromptKitPromptCommand(normalized)
if err != nil {
return err
}
showCommands = append(showCommands, command)
}
if normalized.InspectProfile != nil {
command, err := newPromptKitProfileCommand(normalized)
if err != nil {
return err
}
showCommands = append(showCommands, command)
}
show, err := Group("show", "Inspect PromptKit definitions.", showCommands...)
if err != nil {
return err
}
root, err := Group("promptkit", "Inspect PromptKit definitions.", show)
if err != nil {
return err
}
return app.Register(root)
}
func normalizePromptKitSuite[C any](suite PromptKitSuite[C]) (PromptKitSuite[C], error) {
normalized := suite
normalized.Selectors = append([]Selector[C](nil), suite.Selectors...)
seenNames := make(map[string]struct{}, len(normalized.Selectors))
seenShorts := make(map[string]struct{}, len(normalized.Selectors))
for i, selector := range normalized.Selectors {
binding := selector.Binding.binding
meta := binding.meta
if !binding.valid || binding.clear == nil || meta.Name == "" {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector %d has an invalid flag binding", i)
}
if !validFlagName(meta.Name) {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag name %q is invalid", meta.Name)
}
switch meta.Name {
case "config", "overlay", "scope", "format", "version", "help":
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag --%s conflicts with a standard flag", meta.Name)
}
if meta.Short == "h" {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short flag -%s conflicts with help", meta.Short)
}
if meta.Short != "" && (len(meta.Short) != 1 || meta.Short == "-") {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short spelling %q must be one byte", meta.Short)
}
if _, exists := seenNames[meta.Name]; exists {
return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector flag --%s", meta.Name)
}
seenNames[meta.Name] = struct{}{}
if meta.Short != "" {
if _, exists := seenShorts[meta.Short]; exists {
return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector short flag -%s", meta.Short)
}
seenShorts[meta.Short] = struct{}{}
}
applies := len(selector.Scopes) == 0
for _, scope := range selector.Scopes {
if scope == "promptkit" {
applies = true
break
}
}
if !applies {
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector --%s must apply to scope %q", meta.Name, "promptkit")
}
normalized.Selectors[i].Scopes = append([]string(nil), selector.Scopes...)
}
return normalized, nil
}
type promptKitSelectorState[C any] struct {
scopes []string
apply func(*C)
clear func(*C)
}
type promptKitInput[C any] struct {
Selection Selection
Format string
ID string
Version string
SelectorContext C
selectors []promptKitSelectorState[C]
}
func promptKitInputFlags[C any](suite PromptKitSuite[C], prompt bool) []Flag[promptKitInput[C]] {
formatDefault := "text"
flags := []Flag[promptKitInput[C]]{
ValueFlag[promptKitInput[C]](FlagMeta{Name: "format", Usage: "Output format."}, &formatDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.Format = value.Value
}),
ValueFlag[promptKitInput[C]](FlagMeta{Name: "config", Usage: "Configuration source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.Selection.Config = value
}),
ValueFlag[promptKitInput[C]](FlagMeta{Name: "overlay", Usage: "Overlay source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.Selection.Overlay = value
}),
}
if prompt {
versionDefault := ""
flags = append(flags, ValueFlag[promptKitInput[C]](FlagMeta{Name: "version", Usage: "Exact prompt version; empty selects the owning default."}, &versionDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.Version = value.Value
}))
}
for _, selector := range suite.Selectors {
flags = append(flags, adaptPromptKitSelector(selector))
}
return flags
}
func adaptPromptKitSelector[C any](selector Selector[C]) Flag[promptKitInput[C]] {
binding := selector.Binding.binding
meta := binding.meta
return Flag[promptKitInput[C]]{binding: flagBinding[promptKitInput[C]]{
meta: meta,
defaultText: binding.defaultText,
repeated: binding.repeated,
boolean: binding.boolean,
valid: binding.valid,
prepare: func(values []string) (func(*promptKitInput[C]), error) {
apply, err := binding.prepare(values)
if err != nil {
return nil, err
}
state := promptKitSelectorState[C]{scopes: append([]string(nil), selector.Scopes...), apply: apply, clear: binding.clear}
return func(input *promptKitInput[C]) {
apply(&input.SelectorContext)
input.selectors = append(input.selectors, state)
}, nil
},
}}
}
func checkPromptKitInput[C any](suite PromptKitSuite[C], input promptKitInput[C]) error {
if _, err := parseConfigFormat(input.Format); err != nil {
return err
}
if suite.CheckInput != nil {
return suite.CheckInput(input.Selection, input.SelectorContext)
}
return nil
}
func promptKitContext[C any](suite PromptKitSuite[C], input promptKitInput[C]) C {
var result C
if suite.NewContext != nil {
result = suite.NewContext()
}
for _, selector := range input.selectors {
if selectorApplies(selector.scopes, "promptkit") {
selector.apply(&result)
} else {
selector.clear(&result)
}
}
return result
}
func newPromptKitPromptCommand[C any](suite PromptKitSuite[C]) (Command, error) {
inputSpec := CommandSpec[promptKitInput[C]]{
Name: "prompt",
Summary: "Inspect one PromptKit prompt definition.",
Flags: promptKitInputFlags(suite, true),
Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.ID = value.Value
})},
CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) },
Run: func(invocation Invocation, input promptKitInput[C]) Outcome {
return runPromptKitPrompt(invocation, suite, input)
},
}
return NewCommand(inputSpec)
}
func newPromptKitProfileCommand[C any](suite PromptKitSuite[C]) (Command, error) {
inputSpec := CommandSpec[promptKitInput[C]]{
Name: "profile",
Summary: "Inspect one PromptKit profile definition.",
Flags: promptKitInputFlags(suite, false),
Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) {
input.ID = value.Value
})},
CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) },
Run: func(invocation Invocation, input promptKitInput[C]) Outcome {
return runPromptKitProfile(invocation, suite, input)
},
}
return NewCommand(inputSpec)
}
func runPromptKitPrompt[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome {
contextValue := promptKitContext(suite, input)
acquired, err := suite.InspectPrompt(invocation.Context, input.Selection, contextValue, input.ID, input.Version)
if err != nil {
return promptKitFailure(err, acquired.Diagnostics)
}
if acquired.Value == nil {
return promptKitFailure(fmt.Errorf("PromptKit prompt inspection returned no result"), acquired.Diagnostics)
}
report := promptReportFromInspection(acquired.Value)
return writePromptKitReport(invocation, input.Format, report, formatPromptText(report), acquired.Diagnostics)
}
func runPromptKitProfile[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome {
contextValue := promptKitContext(suite, input)
acquired, err := suite.InspectProfile(invocation.Context, input.Selection, contextValue, input.ID)
if err != nil {
return promptKitFailure(err, acquired.Diagnostics)
}
if acquired.Value == nil {
return promptKitFailure(fmt.Errorf("PromptKit profile inspection returned no result"), acquired.Diagnostics)
}
report := profileReportFromInspection(acquired.Value)
textReport, err := formatProfileText(report)
if err != nil {
return promptKitFailure(err, acquired.Diagnostics)
}
return writePromptKitReport(invocation, input.Format, report, textReport, acquired.Diagnostics)
}
func promptKitFailure(err error, diagnostics []Diagnostic) Outcome {
outcome := Failure(err)
outcome.Diagnostics = diagnostics
return outcome
}
func writePromptKitReport(invocation Invocation, format string, report any, textReport string, diagnostics []Diagnostic) Outcome {
var output strings.Builder
if format == "yaml" {
if err := encodeYAML(&output, report); err != nil {
return promptKitFailure(err, diagnostics)
}
} else if _, err := io.WriteString(&output, textReport); err != nil {
return promptKitFailure(err, diagnostics)
}
if err := writeReport(invocation.Streams.Out, []byte(output.String())); err != nil {
outcome := incompleteReport(err)
outcome.Diagnostics = diagnostics
return outcome
}
outcome := Success()
outcome.Diagnostics = diagnostics
return outcome
}
type promptInputReport struct {
Name string `yaml:"name"`
Required bool `yaml:"required"`
ContentType string `yaml:"content_type"`
Description string `yaml:"description"`
}
type promptOutputContractReport struct {
Format string `yaml:"format"`
ValidationMode string `yaml:"validation_mode"`
SchemaPath string `yaml:"schema_path"`
RepairAttempts int `yaml:"repair_attempts"`
}
type promptInspectionReport struct {
PromptID string `yaml:"prompt_id"`
PromptVersion string `yaml:"prompt_version"`
PromptHash string `yaml:"prompt_hash"`
DefaultProfileID string `yaml:"default_profile_id"`
Inputs []promptInputReport `yaml:"inputs"`
OutputContract promptOutputContractReport `yaml:"output_contract"`
}
func promptReportFromInspection(inspection *promptkit.PromptInspection) promptInspectionReport {
report := promptInspectionReport{
PromptID: inspection.PromptID,
PromptVersion: inspection.PromptVersion,
PromptHash: inspection.PromptHash,
DefaultProfileID: inspection.DefaultProfileID,
Inputs: make([]promptInputReport, 0, len(inspection.Inputs)),
OutputContract: promptOutputContractReport{
Format: string(inspection.OutputContract.Format),
ValidationMode: string(inspection.OutputContract.ValidationMode),
SchemaPath: inspection.OutputContract.SchemaPath,
RepairAttempts: inspection.OutputContract.RepairAttempts,
},
}
for _, input := range inspection.Inputs {
report.Inputs = append(report.Inputs, promptInputReport{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description})
}
return report
}
type modelParamsReport struct {
BackendID string `yaml:"backend_id"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`
Temperature float64 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"`
ServiceTier string `yaml:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]any `yaml:"extra_params"`
}
type profileInspectionReport struct {
ProfileID string `yaml:"profile_id"`
EffectiveModelParams modelParamsReport `yaml:"effective_model_params"`
APIKeyRequired bool `yaml:"api_key_required"`
}
func profileReportFromInspection(inspection *promptkit.ProfileInspection) profileInspectionReport {
params := inspection.EffectiveModelParams
extra := params.ExtraParams
if extra == nil {
extra = map[string]any{}
}
return profileInspectionReport{
ProfileID: inspection.ProfileID,
EffectiveModelParams: modelParamsReport{
BackendID: params.BackendID, Endpoint: params.Endpoint, Model: params.Model,
Temperature: params.Temperature, MaxTokens: params.MaxTokens, TopP: params.TopP,
TimeoutSeconds: params.TimeoutSeconds, ServiceTier: params.ServiceTier,
ReasoningEffort: params.ReasoningEffort, APIKeyEnv: params.APIKeyEnv, ExtraParams: extra,
},
APIKeyRequired: inspection.APIKeyRequired,
}
}
func formatPromptText(report promptInspectionReport) string {
var output strings.Builder
fmt.Fprintf(&output, "Prompt ID: %s\n", strconv.Quote(report.PromptID))
fmt.Fprintf(&output, "Prompt version: %s\n", strconv.Quote(report.PromptVersion))
fmt.Fprintf(&output, "Prompt hash: %s\n", strconv.Quote(report.PromptHash))
fmt.Fprintf(&output, "Default profile ID: %s\n", strconv.Quote(report.DefaultProfileID))
output.WriteString("Inputs:\n")
for _, input := range report.Inputs {
fmt.Fprintf(&output, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", strconv.Quote(input.Name), input.Required, strconv.Quote(input.ContentType), strconv.Quote(input.Description))
}
if len(report.Inputs) == 0 {
output.WriteString(" []\n")
}
fmt.Fprintf(&output, "Output contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", strconv.Quote(report.OutputContract.Format), strconv.Quote(report.OutputContract.ValidationMode), strconv.Quote(report.OutputContract.SchemaPath), report.OutputContract.RepairAttempts)
return output.String()
}
func formatProfileText(report profileInspectionReport) (string, error) {
params := report.EffectiveModelParams
var output strings.Builder
fmt.Fprintf(&output, "Profile ID: %s\n", strconv.Quote(report.ProfileID))
output.WriteString("Effective model params:\n")
fmt.Fprintf(&output, " backend_id: %s\n endpoint: %s\n model: %s\n temperature: %s\n max_tokens: %d\n top_p: %s\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n", strconv.Quote(params.BackendID), strconv.Quote(params.Endpoint), strconv.Quote(params.Model), strconv.FormatFloat(params.Temperature, 'g', -1, 64), params.MaxTokens, strconv.FormatFloat(params.TopP, 'g', -1, 64), params.TimeoutSeconds, strconv.Quote(params.ServiceTier), strconv.Quote(params.ReasoningEffort), strconv.Quote(params.APIKeyEnv))
output.WriteString(" extra_params:\n")
var nested strings.Builder
if err := encodeYAML(&nested, params.ExtraParams); err != nil {
return "", err
}
for _, line := range strings.Split(strings.TrimSuffix(nested.String(), "\n"), "\n") {
output.WriteString(" ")
output.WriteString(line)
output.WriteByte('\n')
}
fmt.Fprintf(&output, "API key required: %t\n", report.APIKeyRequired)
return output.String(), nil
}

View File

@@ -2,10 +2,8 @@ package commandkit
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/promptkit" "gitea.maximumdirect.net/eric/promptkit"
) )
@@ -23,24 +21,10 @@ func NewPromptKitScope[C any](
Summary: "PromptKit engine configuration.", Summary: "PromptKit engine configuration.",
Operations: []ConfigOperation{ConfigValidate, ConfigShow, ConfigSources, ConfigExplain}, Operations: []ConfigOperation{ConfigValidate, ConfigShow, ConfigSources, ConfigExplain},
Acquire: func(ctx context.Context, request ConfigRequest, appContext C) (Acquired[ConfigView], error) { Acquire: func(ctx context.Context, request ConfigRequest, appContext C) (Acquired[ConfigView], error) {
engine, err := load(ctx, request.Selection, appContext) _, inspection, diagnostics, err := loadPromptKitInspection(ctx, request.Selection, appContext, load)
if err != nil { if err != nil {
var loadErr *configkit.LoadError return Acquired[ConfigView]{Diagnostics: diagnostics}, err
if errors.As(err, &loadErr) {
return Acquired[ConfigView]{Diagnostics: ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics())}, err
}
return Acquired[ConfigView]{}, err
} }
if engine == nil {
return Acquired[ConfigView]{}, fmt.Errorf("PromptKit configuration loader returned a nil engine")
}
inspection, err := engine.InspectConfiguration()
if err != nil {
return Acquired[ConfigView]{}, err
}
sources := inspection.Sources()
diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics())
return Acquired[ConfigView]{ return Acquired[ConfigView]{
Value: ConfigView{ Value: ConfigView{
Effective: inspection, Effective: inspection,

378
promptkit_test.go Normal file
View File

@@ -0,0 +1,378 @@
package commandkit_test
import (
"bytes"
"context"
"errors"
"io"
"reflect"
"strings"
"testing"
"testing/fstest"
commandkit "gitea.maximumdirect.net/eric/CommandKit"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/promptkit"
"go.yaml.in/yaml/v4"
)
const promptKitInspectionFixture = `prompts:
"team/summary":
"2":
default_profile: child
inputs:
- name: topic
required: true
content_type: text/plain
description: "Topic to summarize"
- name: audience
required: false
content_type: text/plain
description: ""
messages:
- role: user
content: "Summarize {{.topic}}"
output:
format: json
validation_mode: basic
repair_attempts: 2
"3":
messages:
- role: user
content: "Version three"
output:
format: text
validation_mode: none
empty:
"1":
messages:
- role: user
content: "No inputs"
output:
format: text
validation_mode: none
`
func promptKitInspectionEngine(t *testing.T) *promptkit.Engine {
t.Helper()
inputs, err := promptkit.FilesystemInputs("prompt fixture", fstest.MapFS{
"config/prompts.yml": &fstest.MapFile{Data: []byte(promptKitInspectionFixture)},
}, "config")
if err != nil {
t.Fatal(err)
}
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{
Underlays: []configkit.Underlay{
{Inputs: inputs},
{Inputs: []configkit.Input{{Source: configkit.Mapping("profile fixture", map[string]any{
"backends": map[string]any{"local": map[string]any{"endpoint": "https://example.invalid/v1"}},
"profiles": map[string]any{
"parent": map[string]any{
"backend": "local", "model": "base-model", "temperature": 0.7,
"max_tokens": 100, "top_p": 0.8,
"extra_params": map[string]any{
"token": "provider-private-sentinel",
"nested": map[string]any{"null_value": nil, "visible": "kept"},
"sequence": []any{"first", nil, 3},
},
},
"child": map[string]any{"extends": "parent", "temperature": 0, "max_tokens": 0, "api_key_env": "PROMPTKIT_TEST_KEY"},
"requires-key": map[string]any{"backend": "local", "model": "key-model", "api_key_required": true},
"no-extra": map[string]any{"backend": "local", "model": "empty-params"},
},
})}}},
},
}})
if err != nil {
t.Fatalf("construct PromptKit fixture engine: %v", err)
}
return engine
}
func TestPromptKitInspectionCommandsReportRealCapturedDefinitions(t *testing.T) {
t.Setenv("PROMPTKIT_TEST_KEY", "credential-private-sentinel")
engine := promptKitInspectionEngine(t)
loadCalls := 0
suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) {
loadCalls++
return engine, nil
})
if err != nil {
t.Fatal(err)
}
app, err := commandkit.New(commandkit.Identity{Name: "example"})
if err != nil {
t.Fatal(err)
}
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
t.Fatal(err)
}
var promptYAML bytes.Buffer
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2", "--format", "yaml"}, commandkit.Streams{Out: &promptYAML, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 1 {
t.Fatalf("prompt inspection outcome=%#v calls=%d report=%q", outcome, loadCalls, promptYAML.String())
}
var promptReport map[string]any
if err := yaml.Unmarshal(promptYAML.Bytes(), &promptReport); err != nil {
t.Fatalf("decode prompt YAML: %v", err)
}
if !reflect.DeepEqual(sortedKeys(promptReport), []string{"default_profile_id", "inputs", "output_contract", "prompt_hash", "prompt_id", "prompt_version"}) {
t.Fatalf("prompt report keys = %v", sortedKeys(promptReport))
}
if promptReport["prompt_id"] != "team/summary" || promptReport["prompt_version"] != "2" || promptReport["default_profile_id"] != "child" || promptReport["prompt_hash"] == "" {
t.Fatalf("prompt report identity = %#v", promptReport)
}
inputs := promptReport["inputs"].([]any)
if len(inputs) != 2 || inputs[0].(map[string]any)["name"] != "topic" || inputs[1].(map[string]any)["name"] != "audience" {
t.Fatalf("prompt inputs/order = %#v", inputs)
}
outputContract := promptReport["output_contract"].(map[string]any)
if outputContract["format"] != "json" || outputContract["validation_mode"] != "basic" || outputContract["repair_attempts"] != 2 {
t.Fatalf("output contract = %#v", outputContract)
}
var promptText bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2"}, commandkit.Streams{Out: &promptText, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(promptText.String(), `Prompt hash: "`) || !strings.Contains(promptText.String(), `- name: "topic"`) || !strings.Contains(promptText.String(), `validation_mode: "basic"`) {
t.Fatalf("prompt text = (%#v, %q)", outcome, promptText.String())
}
var omittedVersion bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--format", "yaml"}, commandkit.Streams{Out: &omittedVersion, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(omittedVersion.String(), "prompt_version: \"1\"") {
t.Fatalf("omitted singleton prompt version = (%#v, %q)", outcome, omittedVersion.String())
}
var profileYAML bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child", "--format", "yaml"}, commandkit.Streams{Out: &profileYAML, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess {
t.Fatalf("profile inspection outcome=%#v report=%q", outcome, profileYAML.String())
}
var profileReport map[string]any
if err := yaml.Unmarshal(profileYAML.Bytes(), &profileReport); err != nil {
t.Fatalf("decode profile YAML: %v", err)
}
if !reflect.DeepEqual(sortedKeys(profileReport), []string{"api_key_required", "effective_model_params", "profile_id"}) || profileReport["profile_id"] != "child" || profileReport["api_key_required"] != false {
t.Fatalf("profile report = %#v", profileReport)
}
params := profileReport["effective_model_params"].(map[string]any)
if !reflect.DeepEqual(sortedKeys(params), []string{"api_key_env", "backend_id", "endpoint", "extra_params", "max_tokens", "model", "reasoning_effort", "service_tier", "temperature", "timeout_seconds", "top_p"}) {
t.Fatalf("model parameter keys = %v", sortedKeys(params))
}
if params["temperature"] != 0 || params["max_tokens"] != 0 || params["api_key_env"] != "PROMPTKIT_TEST_KEY" {
t.Fatalf("effective owning values = %#v", params)
}
extra := params["extra_params"].(map[string]any)
nested := extra["nested"].(map[string]any)
sequence := extra["sequence"].([]any)
if nested["null_value"] != nil || nested["visible"] != "kept" || len(sequence) != 3 || sequence[1] != nil || sequence[2] != 3 {
t.Fatalf("nested provider parameters changed: %#v", extra)
}
if extra["token"] != "provider-private-sentinel" || strings.Contains(profileYAML.String(), "credential-private-sentinel") {
t.Fatalf("provider parameters or credential reference changed: %q", profileYAML.String())
}
var profileText bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: &profileText, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(profileText.String(), "temperature: 0\n") || !strings.Contains(profileText.String(), "api_key_env: \"PROMPTKIT_TEST_KEY\"") || !strings.Contains(profileText.String(), "null_value: null") {
t.Fatalf("profile text = (%#v, %q)", outcome, profileText.String())
}
var requiredKey bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "requires-key", "--format", "yaml"}, commandkit.Streams{Out: &requiredKey, Err: io.Discard})
var requiredReport map[string]any
if outcome.Kind != commandkit.OutcomeSuccess || yaml.Unmarshal(requiredKey.Bytes(), &requiredReport) != nil || requiredReport["api_key_required"] != true {
t.Fatalf("credential requirement report = (%#v, %q)", outcome, requiredKey.String())
}
var emptyPrompt bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--version", "1", "--format", "yaml"}, commandkit.Streams{Out: &emptyPrompt, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(emptyPrompt.String(), "inputs: []") {
t.Fatalf("empty inputs report = (%#v, %q)", outcome, emptyPrompt.String())
}
var zeroExtra bytes.Buffer
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "no-extra", "--format", "yaml"}, commandkit.Streams{Out: &zeroExtra, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(zeroExtra.String(), "extra_params: {}") {
t.Fatalf("nil extra params report = (%#v, %q)", outcome, zeroExtra.String())
}
}
func TestPromptKitInspectionSelectionErrorsHelpAndOptionalCapabilities(t *testing.T) {
engine := promptKitInspectionEngine(t)
loaderCalls := 0
suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) {
loaderCalls++
return engine, nil
})
if err != nil {
t.Fatal(err)
}
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
t.Fatal(err)
}
badTarget := suite
badTarget.Selectors = []commandkit.Selector[struct{}]{{
Binding: commandkit.ValueFlag[struct{}](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(*struct{}, commandkit.Supplied[string]) {}),
Scopes: []string{"app"},
}}
invalidApp, _ := commandkit.New(commandkit.Identity{Name: "invalid"})
if err := commandkit.RegisterPromptKit(invalidApp, badTarget); err == nil {
t.Fatal("selector for another scope was accepted")
}
for _, tc := range []struct {
args []string
want error
}{
{args: []string{"promptkit", "show", "prompt", "team/summary"}, want: promptkit.ErrPromptLoad},
{args: []string{"promptkit", "show", "prompt", "missing", "--version", "1"}, want: promptkit.ErrPromptNotFound},
{args: []string{"promptkit", "show", "profile", "missing"}, want: promptkit.ErrProfileNotFound},
{args: []string{"promptkit", "show", "prompt", "", "--version", "2"}, want: promptkit.ErrInvalidRequest},
} {
outcome := app.Execute(context.Background(), tc.args, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, tc.want) {
t.Errorf("execute %q = (%#v), want operational error matching %v", tc.args, outcome, tc.want)
}
}
cancelled, cancel := context.WithCancel(context.Background())
cancel()
outcome := app.Execute(cancelled, []string{"promptkit", "show", "prompt", "empty", "--version", "1"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, context.Canceled) {
t.Fatalf("cancelled prompt inspection = %#v", outcome)
}
beforeHelp := loaderCalls
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "--help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != beforeHelp {
t.Fatalf("help acquired engine: outcome=%#v calls=%d", outcome, loaderCalls)
}
beforeMissing := loaderCalls
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeUsage || loaderCalls != beforeMissing {
t.Fatalf("missing ID = (%#v, calls=%d), want usage before acquisition", outcome, loaderCalls)
}
profileOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "profile-only"})
if err := commandkit.RegisterPromptKit(profileOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectProfile: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id string) (commandkit.Acquired[*promptkit.ProfileInspection], error) {
value, err := engine.InspectProfile(ctx, id)
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: value}, err
}}); err != nil {
t.Fatal(err)
}
outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess {
t.Fatalf("profile-only capability was not installed: %#v", outcome)
}
outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeUsage {
t.Fatalf("profile-only suite installed prompt inspection: %#v", outcome)
}
promptOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "prompt-only"})
if err := commandkit.RegisterPromptKit(promptOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id, version string) (commandkit.Acquired[*promptkit.PromptInspection], error) {
value, err := engine.InspectPrompt(ctx, id, version)
return commandkit.Acquired[*promptkit.PromptInspection]{Value: value}, err
}}); err != nil {
t.Fatal(err)
}
outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess {
t.Fatalf("prompt-only capability was not installed: %#v", outcome)
}
outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeUsage {
t.Fatalf("prompt-only suite installed profile inspection: %#v", outcome)
}
emptyApp, _ := commandkit.New(commandkit.Identity{Name: "empty"})
if err := commandkit.RegisterPromptKit(emptyApp, commandkit.PromptKitSuite[struct{}]{}); err != nil {
t.Fatalf("empty PromptKit suite = %v", err)
}
outcome = emptyApp.Execute(context.Background(), []string{"help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess {
t.Fatalf("empty PromptKit suite changed command family: %#v", outcome)
}
outcome = emptyApp.Execute(context.Background(), []string{"help", "promptkit"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeUsage {
t.Fatalf("empty PromptKit suite installed its namespace: %#v", outcome)
}
}
func TestPromptKitSuitePassesSelectionAndProjectedContext(t *testing.T) {
type appContext struct {
Campaign commandkit.Supplied[string]
Runtime string
}
type contextKey struct{}
engine := promptKitInspectionEngine(t)
ctx := context.WithValue(context.Background(), contextKey{}, "caller")
wantSelection := commandkit.Selection{
Config: commandkit.Supplied[string]{Value: "main.yml", Set: true},
Overlay: commandkit.Supplied[string]{Value: "dev", Set: true},
}
loaderCalls, contexts, checks := 0, 0, 0
var gotSelection commandkit.Selection
var gotContext appContext
suite, err := commandkit.NewPromptKitSuite(func(gotCtx context.Context, selection commandkit.Selection, input appContext) (*promptkit.Engine, error) {
loaderCalls++
gotSelection, gotContext = selection, input
if gotCtx != ctx || gotCtx.Value(contextKey{}) != "caller" {
t.Fatal("engine loader did not receive invocation context")
}
return engine, nil
})
if err != nil {
t.Fatal(err)
}
suite.NewContext = func() appContext { contexts++; return appContext{Runtime: "invocation cache"} }
suite.Selectors = []commandkit.Selector[appContext]{{
Binding: commandkit.ValueFlag[appContext](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(input *appContext, value commandkit.Supplied[string]) { input.Campaign = value }),
Scopes: []string{"promptkit"},
}}
suite.CheckInput = func(selection commandkit.Selection, input appContext) error {
checks++
if selection != wantSelection || input.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) {
t.Errorf("preflight inputs = (%+v, %+v)", selection, input)
}
return nil
}
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
t.Fatal(err)
}
outcome := app.Execute(ctx, []string{"promptkit", "show", "profile", "child", "--config", "main.yml", "--overlay", "dev", "--campaign", "spring"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != 1 || contexts != 1 || checks != 1 {
t.Fatalf("inspection = (%#v), loader/context/check calls = %d/%d/%d", outcome, loaderCalls, contexts, checks)
}
if gotSelection != wantSelection || gotContext.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) || gotContext.Runtime != "invocation cache" {
t.Fatalf("loader selection/context = (%+v, %+v)", gotSelection, gotContext)
}
}
func TestPromptKitDirectSuiteRetainsDiagnosticsOnNamedLookupFailure(t *testing.T) {
warning := commandkit.Diagnostic{Severity: configkit.SeverityWarning, Write: func(w io.Writer) error { _, err := io.WriteString(w, "configuration warning\n"); return err }}
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
err := commandkit.RegisterPromptKit(app, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(context.Context, commandkit.Selection, struct{}, string, string) (commandkit.Acquired[*promptkit.PromptInspection], error) {
return commandkit.Acquired[*promptkit.PromptInspection]{Diagnostics: []commandkit.Diagnostic{warning}}, promptkit.ErrPromptNotFound
}})
if err != nil {
t.Fatal(err)
}
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "missing"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, promptkit.ErrPromptNotFound) || len(outcome.Diagnostics) < 1 || outcome.Diagnostics[0].Severity != configkit.SeverityWarning {
t.Fatalf("named lookup outcome did not retain acquisition diagnostic: %#v", outcome)
}
}
func sortedKeys(values map[string]any) []string {
keys := make([]string, 0, len(values))
for key := range values {
keys = append(keys, key)
}
for i := 0; i < len(keys); i++ {
for j := i + 1; j < len(keys); j++ {
if keys[j] < keys[i] {
keys[i], keys[j] = keys[j], keys[i]
}
}
}
return keys
}