Add PromptKit definition inspection
This commit is contained in:
@@ -8,7 +8,8 @@ and operand bindings, Cobra-backed routing, invocation outcomes, generated help,
|
|||||||
application or CommandKit version reporting, configuration scope discovery,
|
application or CommandKit version reporting, configuration scope discovery,
|
||||||
ConfigKit-backed validation, native effective YAML display, source inventory,
|
ConfigKit-backed validation, native effective YAML display, source inventory,
|
||||||
and path explanation reports. Applications can register a PromptKit engine
|
and path explanation reports. Applications can register a PromptKit engine
|
||||||
configuration scope independently of PromptKit definition commands.
|
configuration scope independently of PromptKit definition commands, and can
|
||||||
|
install optional prompt and profile inspection commands.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
|
|||||||
70
docs/cli.md
70
docs/cli.md
@@ -225,6 +225,76 @@ environment-variable name; effective and historical secret fragments remain
|
|||||||
redacted. Empty location, inheritance, history, reason, and dependency lists are
|
redacted. Empty location, inheritance, history, reason, and dependency lists are
|
||||||
serialized as `[]`.
|
serialized as `[]`.
|
||||||
|
|
||||||
|
## PromptKit definition inspection
|
||||||
|
|
||||||
|
Applications can install prompt and profile inspection independently with
|
||||||
|
`RegisterPromptKit`. Each installed handler constructs the selected engine once
|
||||||
|
and asks PromptKit to inspect the named definition. Inspection reads the
|
||||||
|
engine's captured configuration; it does not generate a prompt, contact a
|
||||||
|
provider, or look up credentials.
|
||||||
|
Prompt definitions and profile resolution remain owned by
|
||||||
|
[PromptKit](https://pkg.go.dev/gitea.maximumdirect.net/eric/promptkit).
|
||||||
|
|
||||||
|
```text
|
||||||
|
example promptkit show prompt customer/summary --version 2 --format yaml
|
||||||
|
example promptkit show prompt customer/summary
|
||||||
|
example promptkit show profile customer-support --format yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Both commands require one ID operand and accept `--config`, `--overlay`, and
|
||||||
|
`--format text|yaml` (default `text`). Prompt inspection also accepts singleton
|
||||||
|
`--version`, defaulting to the empty string. PromptKit owns version selection:
|
||||||
|
an omitted version works for a single-version ID, while an ambiguous ID is an
|
||||||
|
operational error. These commands do not accept `--scope`; their namespace is
|
||||||
|
`promptkit`.
|
||||||
|
|
||||||
|
Prompt YAML has these fields. Input order is preserved and an empty input list is
|
||||||
|
`[]`. The prompt hash is an opaque owning-library equality value.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
prompt_id: customer/summary
|
||||||
|
prompt_version: "2"
|
||||||
|
prompt_hash: opaque-value
|
||||||
|
default_profile_id: customer-support
|
||||||
|
inputs:
|
||||||
|
- name: topic
|
||||||
|
required: true
|
||||||
|
content_type: text/plain
|
||||||
|
description: Topic to summarize
|
||||||
|
output_contract:
|
||||||
|
format: json
|
||||||
|
validation_mode: basic
|
||||||
|
schema_path: ""
|
||||||
|
repair_attempts: 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Profile YAML preserves the effective values reported by PromptKit, including
|
||||||
|
zero values and nested provider parameters. `api_key_env` is only an
|
||||||
|
environment-variable name. `api_key_required` reports a direct credential
|
||||||
|
requirement; neither field reports credential availability. A nil provider
|
||||||
|
parameter map is displayed as `{}`.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
profile_id: customer-support
|
||||||
|
effective_model_params:
|
||||||
|
backend_id: local
|
||||||
|
endpoint: https://example.invalid/v1
|
||||||
|
model: support-model
|
||||||
|
temperature: 0
|
||||||
|
max_tokens: 0
|
||||||
|
top_p: 0
|
||||||
|
timeout_seconds: 600
|
||||||
|
service_tier: ""
|
||||||
|
reasoning_effort: ""
|
||||||
|
api_key_env: ""
|
||||||
|
extra_params: {}
|
||||||
|
api_key_required: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Text uses labeled fields and includes nested `extra_params` values. A missing ID
|
||||||
|
is a usage error. Blank, unknown, ambiguous, and canceled selections are
|
||||||
|
operational inspection results from PromptKit and retain its error identity.
|
||||||
|
|
||||||
## Outcome status
|
## Outcome status
|
||||||
|
|
||||||
CommandKit usage failures use status `2`; operational failures use status `1`.
|
CommandKit usage failures use status `2`; operational failures use status `1`.
|
||||||
|
|||||||
@@ -6,8 +6,8 @@ declarations, an instance-owned registry, a Cobra-backed routing and parsing
|
|||||||
adapter, public `App.Execute` coordination, and generated help/version commands.
|
adapter, public `App.Execute` coordination, and generated help/version commands.
|
||||||
Configuration suite registration, scope discovery, ConfigKit integration,
|
Configuration suite registration, scope discovery, ConfigKit integration,
|
||||||
validation, native effective YAML display, source inventory, and path
|
validation, native effective YAML display, source inventory, and path
|
||||||
explanation reports, and PromptKit engine configuration inspection are
|
explanation reports, PromptKit engine configuration inspection, and optional
|
||||||
implemented; named prompt/profile inspection remains pending. The
|
PromptKit prompt/profile inspection commands are implemented. The
|
||||||
[README](../README.md) provides project orientation; the
|
[README](../README.md) provides project orientation; the
|
||||||
[documentation policy](policy/documentation.md#canonical-ownership) assigns
|
[documentation policy](policy/documentation.md#canonical-ownership) assigns
|
||||||
canonical documentation owners.
|
canonical documentation owners.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Initial CommandKit implementation plan
|
# Initial CommandKit implementation plan
|
||||||
|
|
||||||
Status: implementation underway; Stages 1-12 complete and Stages 13-14 pending.
|
Status: implementation underway; Stages 1-13 complete and Stage 14 pending.
|
||||||
|
|
||||||
Date: 2026-10-07.
|
Date: 2026-10-07.
|
||||||
|
|
||||||
@@ -750,7 +750,7 @@ PromptKit/ConfigKit error identities. `go test ./...` passes.
|
|||||||
|
|
||||||
## Stage 13: implement named prompt/profile inspection
|
## Stage 13: implement named prompt/profile inspection
|
||||||
|
|
||||||
**Status:** pending. **Prerequisite:** Stage 12.
|
**Status:** complete. **Prerequisite:** Stage 12.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
type PromptKitSuite[C any] struct {
|
type PromptKitSuite[C any] struct {
|
||||||
@@ -811,6 +811,11 @@ formats, nil/empty parameters, independently installed prompt/profile capability
|
|||||||
definition-only registration, and help without engine construction. Assert no
|
definition-only registration, and help without engine construction. Assert no
|
||||||
handler enters generation rather than exercising paid/network execution paths.
|
handler enters generation rather than exercising paid/network execution paths.
|
||||||
|
|
||||||
|
**Completion evidence:** `promptkit_test.go` exercises real captured prompt and
|
||||||
|
profile definitions, both report formats, inheritance and provider parameters,
|
||||||
|
selection errors, optional capabilities, and help without engine construction.
|
||||||
|
`go test ./...` passes.
|
||||||
|
|
||||||
## Stage 14: substantiate consumer fit and complete documentation
|
## Stage 14: substantiate consumer fit and complete documentation
|
||||||
|
|
||||||
**Status:** pending. **Prerequisite:** Stage 13.
|
**Status:** pending. **Prerequisite:** Stage 13.
|
||||||
|
|||||||
463
promptkit.go
Normal file
463
promptkit.go
Normal file
@@ -0,0 +1,463 @@
|
|||||||
|
package commandkit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// PromptKitSuite declares optional PromptKit prompt and profile inspection
|
||||||
|
// commands for an App.
|
||||||
|
type PromptKitSuite[C any] struct {
|
||||||
|
// NewContext creates command-local application context when an inspection runs.
|
||||||
|
NewContext func() C
|
||||||
|
// Selectors declares context flags accepted by the installed commands. Each
|
||||||
|
// explicit scope list must include "promptkit"; an empty list applies here.
|
||||||
|
Selectors []Selector[C]
|
||||||
|
// CheckInput validates standard configuration selections and selector values.
|
||||||
|
CheckInput func(Selection, C) error
|
||||||
|
// InspectPrompt inspects an ID and exact or omitted version.
|
||||||
|
InspectPrompt func(context.Context, Selection, C, string, string) (Acquired[*promptkit.PromptInspection], error)
|
||||||
|
// InspectProfile inspects one profile ID.
|
||||||
|
InspectProfile func(context.Context, Selection, C, string) (Acquired[*promptkit.ProfileInspection], error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewPromptKitSuite creates prompt and profile inspection callbacks backed by
|
||||||
|
// a PromptKit Engine loader. The loader runs once per selected command. Each
|
||||||
|
// callback captures configuration diagnostics before calling the corresponding
|
||||||
|
// PromptKit inspection method, and returns those diagnostics with lookup errors.
|
||||||
|
func NewPromptKitSuite[C any](
|
||||||
|
load func(context.Context, Selection, C) (*promptkit.Engine, error),
|
||||||
|
) (PromptKitSuite[C], error) {
|
||||||
|
if load == nil {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit engine loader is nil")
|
||||||
|
}
|
||||||
|
return PromptKitSuite[C]{
|
||||||
|
InspectPrompt: func(ctx context.Context, selection Selection, appContext C, id, version string) (Acquired[*promptkit.PromptInspection], error) {
|
||||||
|
engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load)
|
||||||
|
if err != nil {
|
||||||
|
return Acquired[*promptkit.PromptInspection]{Diagnostics: diagnostics}, err
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectPrompt(ctx, id, version)
|
||||||
|
return Acquired[*promptkit.PromptInspection]{Value: inspection, Diagnostics: diagnostics}, err
|
||||||
|
},
|
||||||
|
InspectProfile: func(ctx context.Context, selection Selection, appContext C, id string) (Acquired[*promptkit.ProfileInspection], error) {
|
||||||
|
engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load)
|
||||||
|
if err != nil {
|
||||||
|
return Acquired[*promptkit.ProfileInspection]{Diagnostics: diagnostics}, err
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectProfile(ctx, id)
|
||||||
|
return Acquired[*promptkit.ProfileInspection]{Value: inspection, Diagnostics: diagnostics}, err
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadPromptKitInspection[C any](ctx context.Context, selection Selection, appContext C, load func(context.Context, Selection, C) (*promptkit.Engine, error)) (*promptkit.Engine, promptkit.ConfigurationInspection, []Diagnostic, error) {
|
||||||
|
engine, err := load(ctx, selection, appContext)
|
||||||
|
if err != nil {
|
||||||
|
var loadErr *configkit.LoadError
|
||||||
|
if errors.As(err, &loadErr) {
|
||||||
|
return nil, promptkit.ConfigurationInspection{}, ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics()), err
|
||||||
|
}
|
||||||
|
return nil, promptkit.ConfigurationInspection{}, nil, err
|
||||||
|
}
|
||||||
|
if engine == nil {
|
||||||
|
return nil, promptkit.ConfigurationInspection{}, nil, fmt.Errorf("PromptKit configuration loader returned a nil engine")
|
||||||
|
}
|
||||||
|
inspection, err := engine.InspectConfiguration()
|
||||||
|
if err != nil {
|
||||||
|
return nil, promptkit.ConfigurationInspection{}, nil, err
|
||||||
|
}
|
||||||
|
sources := inspection.Sources()
|
||||||
|
diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics())
|
||||||
|
return engine, inspection, diagnostics, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterPromptKit installs the supplied PromptKit inspection commands under
|
||||||
|
// promptkit show. Each nonnil callback installs only its matching command. A
|
||||||
|
// suite with no inspection callbacks installs no PromptKit namespace.
|
||||||
|
func RegisterPromptKit[C any](app *App, suite PromptKitSuite[C]) error {
|
||||||
|
if suite.InspectPrompt == nil && suite.InspectProfile == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if app == nil {
|
||||||
|
return fmt.Errorf("cannot register PromptKit commands on a nil app")
|
||||||
|
}
|
||||||
|
normalized, err := normalizePromptKitSuite(suite)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
showCommands := make([]Command, 0, 2)
|
||||||
|
if normalized.InspectPrompt != nil {
|
||||||
|
command, err := newPromptKitPromptCommand(normalized)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
showCommands = append(showCommands, command)
|
||||||
|
}
|
||||||
|
if normalized.InspectProfile != nil {
|
||||||
|
command, err := newPromptKitProfileCommand(normalized)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
showCommands = append(showCommands, command)
|
||||||
|
}
|
||||||
|
show, err := Group("show", "Inspect PromptKit definitions.", showCommands...)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
root, err := Group("promptkit", "Inspect PromptKit definitions.", show)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return app.Register(root)
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizePromptKitSuite[C any](suite PromptKitSuite[C]) (PromptKitSuite[C], error) {
|
||||||
|
normalized := suite
|
||||||
|
normalized.Selectors = append([]Selector[C](nil), suite.Selectors...)
|
||||||
|
seenNames := make(map[string]struct{}, len(normalized.Selectors))
|
||||||
|
seenShorts := make(map[string]struct{}, len(normalized.Selectors))
|
||||||
|
for i, selector := range normalized.Selectors {
|
||||||
|
binding := selector.Binding.binding
|
||||||
|
meta := binding.meta
|
||||||
|
if !binding.valid || binding.clear == nil || meta.Name == "" {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector %d has an invalid flag binding", i)
|
||||||
|
}
|
||||||
|
if !validFlagName(meta.Name) {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag name %q is invalid", meta.Name)
|
||||||
|
}
|
||||||
|
switch meta.Name {
|
||||||
|
case "config", "overlay", "scope", "format", "version", "help":
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag --%s conflicts with a standard flag", meta.Name)
|
||||||
|
}
|
||||||
|
if meta.Short == "h" {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short flag -%s conflicts with help", meta.Short)
|
||||||
|
}
|
||||||
|
if meta.Short != "" && (len(meta.Short) != 1 || meta.Short == "-") {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short spelling %q must be one byte", meta.Short)
|
||||||
|
}
|
||||||
|
if _, exists := seenNames[meta.Name]; exists {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector flag --%s", meta.Name)
|
||||||
|
}
|
||||||
|
seenNames[meta.Name] = struct{}{}
|
||||||
|
if meta.Short != "" {
|
||||||
|
if _, exists := seenShorts[meta.Short]; exists {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector short flag -%s", meta.Short)
|
||||||
|
}
|
||||||
|
seenShorts[meta.Short] = struct{}{}
|
||||||
|
}
|
||||||
|
applies := len(selector.Scopes) == 0
|
||||||
|
for _, scope := range selector.Scopes {
|
||||||
|
if scope == "promptkit" {
|
||||||
|
applies = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !applies {
|
||||||
|
return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector --%s must apply to scope %q", meta.Name, "promptkit")
|
||||||
|
}
|
||||||
|
normalized.Selectors[i].Scopes = append([]string(nil), selector.Scopes...)
|
||||||
|
}
|
||||||
|
return normalized, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitSelectorState[C any] struct {
|
||||||
|
scopes []string
|
||||||
|
apply func(*C)
|
||||||
|
clear func(*C)
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptKitInput[C any] struct {
|
||||||
|
Selection Selection
|
||||||
|
Format string
|
||||||
|
ID string
|
||||||
|
Version string
|
||||||
|
SelectorContext C
|
||||||
|
selectors []promptKitSelectorState[C]
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitInputFlags[C any](suite PromptKitSuite[C], prompt bool) []Flag[promptKitInput[C]] {
|
||||||
|
formatDefault := "text"
|
||||||
|
flags := []Flag[promptKitInput[C]]{
|
||||||
|
ValueFlag[promptKitInput[C]](FlagMeta{Name: "format", Usage: "Output format."}, &formatDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.Format = value.Value
|
||||||
|
}),
|
||||||
|
ValueFlag[promptKitInput[C]](FlagMeta{Name: "config", Usage: "Configuration source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.Selection.Config = value
|
||||||
|
}),
|
||||||
|
ValueFlag[promptKitInput[C]](FlagMeta{Name: "overlay", Usage: "Overlay source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.Selection.Overlay = value
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
if prompt {
|
||||||
|
versionDefault := ""
|
||||||
|
flags = append(flags, ValueFlag[promptKitInput[C]](FlagMeta{Name: "version", Usage: "Exact prompt version; empty selects the owning default."}, &versionDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.Version = value.Value
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
for _, selector := range suite.Selectors {
|
||||||
|
flags = append(flags, adaptPromptKitSelector(selector))
|
||||||
|
}
|
||||||
|
return flags
|
||||||
|
}
|
||||||
|
|
||||||
|
func adaptPromptKitSelector[C any](selector Selector[C]) Flag[promptKitInput[C]] {
|
||||||
|
binding := selector.Binding.binding
|
||||||
|
meta := binding.meta
|
||||||
|
return Flag[promptKitInput[C]]{binding: flagBinding[promptKitInput[C]]{
|
||||||
|
meta: meta,
|
||||||
|
defaultText: binding.defaultText,
|
||||||
|
repeated: binding.repeated,
|
||||||
|
boolean: binding.boolean,
|
||||||
|
valid: binding.valid,
|
||||||
|
prepare: func(values []string) (func(*promptKitInput[C]), error) {
|
||||||
|
apply, err := binding.prepare(values)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
state := promptKitSelectorState[C]{scopes: append([]string(nil), selector.Scopes...), apply: apply, clear: binding.clear}
|
||||||
|
return func(input *promptKitInput[C]) {
|
||||||
|
apply(&input.SelectorContext)
|
||||||
|
input.selectors = append(input.selectors, state)
|
||||||
|
}, nil
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkPromptKitInput[C any](suite PromptKitSuite[C], input promptKitInput[C]) error {
|
||||||
|
if _, err := parseConfigFormat(input.Format); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if suite.CheckInput != nil {
|
||||||
|
return suite.CheckInput(input.Selection, input.SelectorContext)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitContext[C any](suite PromptKitSuite[C], input promptKitInput[C]) C {
|
||||||
|
var result C
|
||||||
|
if suite.NewContext != nil {
|
||||||
|
result = suite.NewContext()
|
||||||
|
}
|
||||||
|
for _, selector := range input.selectors {
|
||||||
|
if selectorApplies(selector.scopes, "promptkit") {
|
||||||
|
selector.apply(&result)
|
||||||
|
} else {
|
||||||
|
selector.clear(&result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func newPromptKitPromptCommand[C any](suite PromptKitSuite[C]) (Command, error) {
|
||||||
|
inputSpec := CommandSpec[promptKitInput[C]]{
|
||||||
|
Name: "prompt",
|
||||||
|
Summary: "Inspect one PromptKit prompt definition.",
|
||||||
|
Flags: promptKitInputFlags(suite, true),
|
||||||
|
Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.ID = value.Value
|
||||||
|
})},
|
||||||
|
CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) },
|
||||||
|
Run: func(invocation Invocation, input promptKitInput[C]) Outcome {
|
||||||
|
return runPromptKitPrompt(invocation, suite, input)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return NewCommand(inputSpec)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newPromptKitProfileCommand[C any](suite PromptKitSuite[C]) (Command, error) {
|
||||||
|
inputSpec := CommandSpec[promptKitInput[C]]{
|
||||||
|
Name: "profile",
|
||||||
|
Summary: "Inspect one PromptKit profile definition.",
|
||||||
|
Flags: promptKitInputFlags(suite, false),
|
||||||
|
Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) {
|
||||||
|
input.ID = value.Value
|
||||||
|
})},
|
||||||
|
CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) },
|
||||||
|
Run: func(invocation Invocation, input promptKitInput[C]) Outcome {
|
||||||
|
return runPromptKitProfile(invocation, suite, input)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return NewCommand(inputSpec)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runPromptKitPrompt[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome {
|
||||||
|
contextValue := promptKitContext(suite, input)
|
||||||
|
acquired, err := suite.InspectPrompt(invocation.Context, input.Selection, contextValue, input.ID, input.Version)
|
||||||
|
if err != nil {
|
||||||
|
return promptKitFailure(err, acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
if acquired.Value == nil {
|
||||||
|
return promptKitFailure(fmt.Errorf("PromptKit prompt inspection returned no result"), acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
report := promptReportFromInspection(acquired.Value)
|
||||||
|
return writePromptKitReport(invocation, input.Format, report, formatPromptText(report), acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runPromptKitProfile[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome {
|
||||||
|
contextValue := promptKitContext(suite, input)
|
||||||
|
acquired, err := suite.InspectProfile(invocation.Context, input.Selection, contextValue, input.ID)
|
||||||
|
if err != nil {
|
||||||
|
return promptKitFailure(err, acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
if acquired.Value == nil {
|
||||||
|
return promptKitFailure(fmt.Errorf("PromptKit profile inspection returned no result"), acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
report := profileReportFromInspection(acquired.Value)
|
||||||
|
textReport, err := formatProfileText(report)
|
||||||
|
if err != nil {
|
||||||
|
return promptKitFailure(err, acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
return writePromptKitReport(invocation, input.Format, report, textReport, acquired.Diagnostics)
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptKitFailure(err error, diagnostics []Diagnostic) Outcome {
|
||||||
|
outcome := Failure(err)
|
||||||
|
outcome.Diagnostics = diagnostics
|
||||||
|
return outcome
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePromptKitReport(invocation Invocation, format string, report any, textReport string, diagnostics []Diagnostic) Outcome {
|
||||||
|
var output strings.Builder
|
||||||
|
if format == "yaml" {
|
||||||
|
if err := encodeYAML(&output, report); err != nil {
|
||||||
|
return promptKitFailure(err, diagnostics)
|
||||||
|
}
|
||||||
|
} else if _, err := io.WriteString(&output, textReport); err != nil {
|
||||||
|
return promptKitFailure(err, diagnostics)
|
||||||
|
}
|
||||||
|
if err := writeReport(invocation.Streams.Out, []byte(output.String())); err != nil {
|
||||||
|
outcome := incompleteReport(err)
|
||||||
|
outcome.Diagnostics = diagnostics
|
||||||
|
return outcome
|
||||||
|
}
|
||||||
|
outcome := Success()
|
||||||
|
outcome.Diagnostics = diagnostics
|
||||||
|
return outcome
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptInputReport struct {
|
||||||
|
Name string `yaml:"name"`
|
||||||
|
Required bool `yaml:"required"`
|
||||||
|
ContentType string `yaml:"content_type"`
|
||||||
|
Description string `yaml:"description"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptOutputContractReport struct {
|
||||||
|
Format string `yaml:"format"`
|
||||||
|
ValidationMode string `yaml:"validation_mode"`
|
||||||
|
SchemaPath string `yaml:"schema_path"`
|
||||||
|
RepairAttempts int `yaml:"repair_attempts"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptInspectionReport struct {
|
||||||
|
PromptID string `yaml:"prompt_id"`
|
||||||
|
PromptVersion string `yaml:"prompt_version"`
|
||||||
|
PromptHash string `yaml:"prompt_hash"`
|
||||||
|
DefaultProfileID string `yaml:"default_profile_id"`
|
||||||
|
Inputs []promptInputReport `yaml:"inputs"`
|
||||||
|
OutputContract promptOutputContractReport `yaml:"output_contract"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptReportFromInspection(inspection *promptkit.PromptInspection) promptInspectionReport {
|
||||||
|
report := promptInspectionReport{
|
||||||
|
PromptID: inspection.PromptID,
|
||||||
|
PromptVersion: inspection.PromptVersion,
|
||||||
|
PromptHash: inspection.PromptHash,
|
||||||
|
DefaultProfileID: inspection.DefaultProfileID,
|
||||||
|
Inputs: make([]promptInputReport, 0, len(inspection.Inputs)),
|
||||||
|
OutputContract: promptOutputContractReport{
|
||||||
|
Format: string(inspection.OutputContract.Format),
|
||||||
|
ValidationMode: string(inspection.OutputContract.ValidationMode),
|
||||||
|
SchemaPath: inspection.OutputContract.SchemaPath,
|
||||||
|
RepairAttempts: inspection.OutputContract.RepairAttempts,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, input := range inspection.Inputs {
|
||||||
|
report.Inputs = append(report.Inputs, promptInputReport{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description})
|
||||||
|
}
|
||||||
|
return report
|
||||||
|
}
|
||||||
|
|
||||||
|
type modelParamsReport struct {
|
||||||
|
BackendID string `yaml:"backend_id"`
|
||||||
|
Endpoint string `yaml:"endpoint"`
|
||||||
|
Model string `yaml:"model"`
|
||||||
|
Temperature float64 `yaml:"temperature"`
|
||||||
|
MaxTokens int `yaml:"max_tokens"`
|
||||||
|
TopP float64 `yaml:"top_p"`
|
||||||
|
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||||
|
ServiceTier string `yaml:"service_tier"`
|
||||||
|
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||||
|
APIKeyEnv string `yaml:"api_key_env"`
|
||||||
|
ExtraParams map[string]any `yaml:"extra_params"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type profileInspectionReport struct {
|
||||||
|
ProfileID string `yaml:"profile_id"`
|
||||||
|
EffectiveModelParams modelParamsReport `yaml:"effective_model_params"`
|
||||||
|
APIKeyRequired bool `yaml:"api_key_required"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func profileReportFromInspection(inspection *promptkit.ProfileInspection) profileInspectionReport {
|
||||||
|
params := inspection.EffectiveModelParams
|
||||||
|
extra := params.ExtraParams
|
||||||
|
if extra == nil {
|
||||||
|
extra = map[string]any{}
|
||||||
|
}
|
||||||
|
return profileInspectionReport{
|
||||||
|
ProfileID: inspection.ProfileID,
|
||||||
|
EffectiveModelParams: modelParamsReport{
|
||||||
|
BackendID: params.BackendID, Endpoint: params.Endpoint, Model: params.Model,
|
||||||
|
Temperature: params.Temperature, MaxTokens: params.MaxTokens, TopP: params.TopP,
|
||||||
|
TimeoutSeconds: params.TimeoutSeconds, ServiceTier: params.ServiceTier,
|
||||||
|
ReasoningEffort: params.ReasoningEffort, APIKeyEnv: params.APIKeyEnv, ExtraParams: extra,
|
||||||
|
},
|
||||||
|
APIKeyRequired: inspection.APIKeyRequired,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatPromptText(report promptInspectionReport) string {
|
||||||
|
var output strings.Builder
|
||||||
|
fmt.Fprintf(&output, "Prompt ID: %s\n", strconv.Quote(report.PromptID))
|
||||||
|
fmt.Fprintf(&output, "Prompt version: %s\n", strconv.Quote(report.PromptVersion))
|
||||||
|
fmt.Fprintf(&output, "Prompt hash: %s\n", strconv.Quote(report.PromptHash))
|
||||||
|
fmt.Fprintf(&output, "Default profile ID: %s\n", strconv.Quote(report.DefaultProfileID))
|
||||||
|
output.WriteString("Inputs:\n")
|
||||||
|
for _, input := range report.Inputs {
|
||||||
|
fmt.Fprintf(&output, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", strconv.Quote(input.Name), input.Required, strconv.Quote(input.ContentType), strconv.Quote(input.Description))
|
||||||
|
}
|
||||||
|
if len(report.Inputs) == 0 {
|
||||||
|
output.WriteString(" []\n")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&output, "Output contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", strconv.Quote(report.OutputContract.Format), strconv.Quote(report.OutputContract.ValidationMode), strconv.Quote(report.OutputContract.SchemaPath), report.OutputContract.RepairAttempts)
|
||||||
|
return output.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatProfileText(report profileInspectionReport) (string, error) {
|
||||||
|
params := report.EffectiveModelParams
|
||||||
|
var output strings.Builder
|
||||||
|
fmt.Fprintf(&output, "Profile ID: %s\n", strconv.Quote(report.ProfileID))
|
||||||
|
output.WriteString("Effective model params:\n")
|
||||||
|
fmt.Fprintf(&output, " backend_id: %s\n endpoint: %s\n model: %s\n temperature: %s\n max_tokens: %d\n top_p: %s\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n", strconv.Quote(params.BackendID), strconv.Quote(params.Endpoint), strconv.Quote(params.Model), strconv.FormatFloat(params.Temperature, 'g', -1, 64), params.MaxTokens, strconv.FormatFloat(params.TopP, 'g', -1, 64), params.TimeoutSeconds, strconv.Quote(params.ServiceTier), strconv.Quote(params.ReasoningEffort), strconv.Quote(params.APIKeyEnv))
|
||||||
|
output.WriteString(" extra_params:\n")
|
||||||
|
var nested strings.Builder
|
||||||
|
if err := encodeYAML(&nested, params.ExtraParams); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
for _, line := range strings.Split(strings.TrimSuffix(nested.String(), "\n"), "\n") {
|
||||||
|
output.WriteString(" ")
|
||||||
|
output.WriteString(line)
|
||||||
|
output.WriteByte('\n')
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&output, "API key required: %t\n", report.APIKeyRequired)
|
||||||
|
return output.String(), nil
|
||||||
|
}
|
||||||
@@ -2,10 +2,8 @@ package commandkit
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
|
||||||
"gitea.maximumdirect.net/eric/promptkit"
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -23,24 +21,10 @@ func NewPromptKitScope[C any](
|
|||||||
Summary: "PromptKit engine configuration.",
|
Summary: "PromptKit engine configuration.",
|
||||||
Operations: []ConfigOperation{ConfigValidate, ConfigShow, ConfigSources, ConfigExplain},
|
Operations: []ConfigOperation{ConfigValidate, ConfigShow, ConfigSources, ConfigExplain},
|
||||||
Acquire: func(ctx context.Context, request ConfigRequest, appContext C) (Acquired[ConfigView], error) {
|
Acquire: func(ctx context.Context, request ConfigRequest, appContext C) (Acquired[ConfigView], error) {
|
||||||
engine, err := load(ctx, request.Selection, appContext)
|
_, inspection, diagnostics, err := loadPromptKitInspection(ctx, request.Selection, appContext, load)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
var loadErr *configkit.LoadError
|
return Acquired[ConfigView]{Diagnostics: diagnostics}, err
|
||||||
if errors.As(err, &loadErr) {
|
|
||||||
return Acquired[ConfigView]{Diagnostics: ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics())}, err
|
|
||||||
}
|
|
||||||
return Acquired[ConfigView]{}, err
|
|
||||||
}
|
}
|
||||||
if engine == nil {
|
|
||||||
return Acquired[ConfigView]{}, fmt.Errorf("PromptKit configuration loader returned a nil engine")
|
|
||||||
}
|
|
||||||
|
|
||||||
inspection, err := engine.InspectConfiguration()
|
|
||||||
if err != nil {
|
|
||||||
return Acquired[ConfigView]{}, err
|
|
||||||
}
|
|
||||||
sources := inspection.Sources()
|
|
||||||
diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics())
|
|
||||||
return Acquired[ConfigView]{
|
return Acquired[ConfigView]{
|
||||||
Value: ConfigView{
|
Value: ConfigView{
|
||||||
Effective: inspection,
|
Effective: inspection,
|
||||||
|
|||||||
378
promptkit_test.go
Normal file
378
promptkit_test.go
Normal file
@@ -0,0 +1,378 @@
|
|||||||
|
package commandkit_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
|
||||||
|
commandkit "gitea.maximumdirect.net/eric/CommandKit"
|
||||||
|
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit"
|
||||||
|
"go.yaml.in/yaml/v4"
|
||||||
|
)
|
||||||
|
|
||||||
|
const promptKitInspectionFixture = `prompts:
|
||||||
|
"team/summary":
|
||||||
|
"2":
|
||||||
|
default_profile: child
|
||||||
|
inputs:
|
||||||
|
- name: topic
|
||||||
|
required: true
|
||||||
|
content_type: text/plain
|
||||||
|
description: "Topic to summarize"
|
||||||
|
- name: audience
|
||||||
|
required: false
|
||||||
|
content_type: text/plain
|
||||||
|
description: ""
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "Summarize {{.topic}}"
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 2
|
||||||
|
"3":
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "Version three"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
empty:
|
||||||
|
"1":
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "No inputs"
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
`
|
||||||
|
|
||||||
|
func promptKitInspectionEngine(t *testing.T) *promptkit.Engine {
|
||||||
|
t.Helper()
|
||||||
|
inputs, err := promptkit.FilesystemInputs("prompt fixture", fstest.MapFS{
|
||||||
|
"config/prompts.yml": &fstest.MapFile{Data: []byte(promptKitInspectionFixture)},
|
||||||
|
}, "config")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{
|
||||||
|
Underlays: []configkit.Underlay{
|
||||||
|
{Inputs: inputs},
|
||||||
|
{Inputs: []configkit.Input{{Source: configkit.Mapping("profile fixture", map[string]any{
|
||||||
|
"backends": map[string]any{"local": map[string]any{"endpoint": "https://example.invalid/v1"}},
|
||||||
|
"profiles": map[string]any{
|
||||||
|
"parent": map[string]any{
|
||||||
|
"backend": "local", "model": "base-model", "temperature": 0.7,
|
||||||
|
"max_tokens": 100, "top_p": 0.8,
|
||||||
|
"extra_params": map[string]any{
|
||||||
|
"token": "provider-private-sentinel",
|
||||||
|
"nested": map[string]any{"null_value": nil, "visible": "kept"},
|
||||||
|
"sequence": []any{"first", nil, 3},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"child": map[string]any{"extends": "parent", "temperature": 0, "max_tokens": 0, "api_key_env": "PROMPTKIT_TEST_KEY"},
|
||||||
|
"requires-key": map[string]any{"backend": "local", "model": "key-model", "api_key_required": true},
|
||||||
|
"no-extra": map[string]any{"backend": "local", "model": "empty-params"},
|
||||||
|
},
|
||||||
|
})}}},
|
||||||
|
},
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("construct PromptKit fixture engine: %v", err)
|
||||||
|
}
|
||||||
|
return engine
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitInspectionCommandsReportRealCapturedDefinitions(t *testing.T) {
|
||||||
|
t.Setenv("PROMPTKIT_TEST_KEY", "credential-private-sentinel")
|
||||||
|
engine := promptKitInspectionEngine(t)
|
||||||
|
loadCalls := 0
|
||||||
|
suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) {
|
||||||
|
loadCalls++
|
||||||
|
return engine, nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
app, err := commandkit.New(commandkit.Identity{Name: "example"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var promptYAML bytes.Buffer
|
||||||
|
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2", "--format", "yaml"}, commandkit.Streams{Out: &promptYAML, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 1 {
|
||||||
|
t.Fatalf("prompt inspection outcome=%#v calls=%d report=%q", outcome, loadCalls, promptYAML.String())
|
||||||
|
}
|
||||||
|
var promptReport map[string]any
|
||||||
|
if err := yaml.Unmarshal(promptYAML.Bytes(), &promptReport); err != nil {
|
||||||
|
t.Fatalf("decode prompt YAML: %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(sortedKeys(promptReport), []string{"default_profile_id", "inputs", "output_contract", "prompt_hash", "prompt_id", "prompt_version"}) {
|
||||||
|
t.Fatalf("prompt report keys = %v", sortedKeys(promptReport))
|
||||||
|
}
|
||||||
|
if promptReport["prompt_id"] != "team/summary" || promptReport["prompt_version"] != "2" || promptReport["default_profile_id"] != "child" || promptReport["prompt_hash"] == "" {
|
||||||
|
t.Fatalf("prompt report identity = %#v", promptReport)
|
||||||
|
}
|
||||||
|
inputs := promptReport["inputs"].([]any)
|
||||||
|
if len(inputs) != 2 || inputs[0].(map[string]any)["name"] != "topic" || inputs[1].(map[string]any)["name"] != "audience" {
|
||||||
|
t.Fatalf("prompt inputs/order = %#v", inputs)
|
||||||
|
}
|
||||||
|
outputContract := promptReport["output_contract"].(map[string]any)
|
||||||
|
if outputContract["format"] != "json" || outputContract["validation_mode"] != "basic" || outputContract["repair_attempts"] != 2 {
|
||||||
|
t.Fatalf("output contract = %#v", outputContract)
|
||||||
|
}
|
||||||
|
|
||||||
|
var promptText bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2"}, commandkit.Streams{Out: &promptText, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(promptText.String(), `Prompt hash: "`) || !strings.Contains(promptText.String(), `- name: "topic"`) || !strings.Contains(promptText.String(), `validation_mode: "basic"`) {
|
||||||
|
t.Fatalf("prompt text = (%#v, %q)", outcome, promptText.String())
|
||||||
|
}
|
||||||
|
var omittedVersion bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--format", "yaml"}, commandkit.Streams{Out: &omittedVersion, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(omittedVersion.String(), "prompt_version: \"1\"") {
|
||||||
|
t.Fatalf("omitted singleton prompt version = (%#v, %q)", outcome, omittedVersion.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var profileYAML bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child", "--format", "yaml"}, commandkit.Streams{Out: &profileYAML, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess {
|
||||||
|
t.Fatalf("profile inspection outcome=%#v report=%q", outcome, profileYAML.String())
|
||||||
|
}
|
||||||
|
var profileReport map[string]any
|
||||||
|
if err := yaml.Unmarshal(profileYAML.Bytes(), &profileReport); err != nil {
|
||||||
|
t.Fatalf("decode profile YAML: %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(sortedKeys(profileReport), []string{"api_key_required", "effective_model_params", "profile_id"}) || profileReport["profile_id"] != "child" || profileReport["api_key_required"] != false {
|
||||||
|
t.Fatalf("profile report = %#v", profileReport)
|
||||||
|
}
|
||||||
|
params := profileReport["effective_model_params"].(map[string]any)
|
||||||
|
if !reflect.DeepEqual(sortedKeys(params), []string{"api_key_env", "backend_id", "endpoint", "extra_params", "max_tokens", "model", "reasoning_effort", "service_tier", "temperature", "timeout_seconds", "top_p"}) {
|
||||||
|
t.Fatalf("model parameter keys = %v", sortedKeys(params))
|
||||||
|
}
|
||||||
|
if params["temperature"] != 0 || params["max_tokens"] != 0 || params["api_key_env"] != "PROMPTKIT_TEST_KEY" {
|
||||||
|
t.Fatalf("effective owning values = %#v", params)
|
||||||
|
}
|
||||||
|
extra := params["extra_params"].(map[string]any)
|
||||||
|
nested := extra["nested"].(map[string]any)
|
||||||
|
sequence := extra["sequence"].([]any)
|
||||||
|
if nested["null_value"] != nil || nested["visible"] != "kept" || len(sequence) != 3 || sequence[1] != nil || sequence[2] != 3 {
|
||||||
|
t.Fatalf("nested provider parameters changed: %#v", extra)
|
||||||
|
}
|
||||||
|
if extra["token"] != "provider-private-sentinel" || strings.Contains(profileYAML.String(), "credential-private-sentinel") {
|
||||||
|
t.Fatalf("provider parameters or credential reference changed: %q", profileYAML.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var profileText bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: &profileText, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(profileText.String(), "temperature: 0\n") || !strings.Contains(profileText.String(), "api_key_env: \"PROMPTKIT_TEST_KEY\"") || !strings.Contains(profileText.String(), "null_value: null") {
|
||||||
|
t.Fatalf("profile text = (%#v, %q)", outcome, profileText.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var requiredKey bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "requires-key", "--format", "yaml"}, commandkit.Streams{Out: &requiredKey, Err: io.Discard})
|
||||||
|
var requiredReport map[string]any
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || yaml.Unmarshal(requiredKey.Bytes(), &requiredReport) != nil || requiredReport["api_key_required"] != true {
|
||||||
|
t.Fatalf("credential requirement report = (%#v, %q)", outcome, requiredKey.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
var emptyPrompt bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--version", "1", "--format", "yaml"}, commandkit.Streams{Out: &emptyPrompt, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(emptyPrompt.String(), "inputs: []") {
|
||||||
|
t.Fatalf("empty inputs report = (%#v, %q)", outcome, emptyPrompt.String())
|
||||||
|
}
|
||||||
|
var zeroExtra bytes.Buffer
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "no-extra", "--format", "yaml"}, commandkit.Streams{Out: &zeroExtra, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(zeroExtra.String(), "extra_params: {}") {
|
||||||
|
t.Fatalf("nil extra params report = (%#v, %q)", outcome, zeroExtra.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitInspectionSelectionErrorsHelpAndOptionalCapabilities(t *testing.T) {
|
||||||
|
engine := promptKitInspectionEngine(t)
|
||||||
|
loaderCalls := 0
|
||||||
|
suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) {
|
||||||
|
loaderCalls++
|
||||||
|
return engine, nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
|
||||||
|
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
badTarget := suite
|
||||||
|
badTarget.Selectors = []commandkit.Selector[struct{}]{{
|
||||||
|
Binding: commandkit.ValueFlag[struct{}](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(*struct{}, commandkit.Supplied[string]) {}),
|
||||||
|
Scopes: []string{"app"},
|
||||||
|
}}
|
||||||
|
invalidApp, _ := commandkit.New(commandkit.Identity{Name: "invalid"})
|
||||||
|
if err := commandkit.RegisterPromptKit(invalidApp, badTarget); err == nil {
|
||||||
|
t.Fatal("selector for another scope was accepted")
|
||||||
|
}
|
||||||
|
for _, tc := range []struct {
|
||||||
|
args []string
|
||||||
|
want error
|
||||||
|
}{
|
||||||
|
{args: []string{"promptkit", "show", "prompt", "team/summary"}, want: promptkit.ErrPromptLoad},
|
||||||
|
{args: []string{"promptkit", "show", "prompt", "missing", "--version", "1"}, want: promptkit.ErrPromptNotFound},
|
||||||
|
{args: []string{"promptkit", "show", "profile", "missing"}, want: promptkit.ErrProfileNotFound},
|
||||||
|
{args: []string{"promptkit", "show", "prompt", "", "--version", "2"}, want: promptkit.ErrInvalidRequest},
|
||||||
|
} {
|
||||||
|
outcome := app.Execute(context.Background(), tc.args, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, tc.want) {
|
||||||
|
t.Errorf("execute %q = (%#v), want operational error matching %v", tc.args, outcome, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cancelled, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
outcome := app.Execute(cancelled, []string{"promptkit", "show", "prompt", "empty", "--version", "1"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, context.Canceled) {
|
||||||
|
t.Fatalf("cancelled prompt inspection = %#v", outcome)
|
||||||
|
}
|
||||||
|
beforeHelp := loaderCalls
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "--help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != beforeHelp {
|
||||||
|
t.Fatalf("help acquired engine: outcome=%#v calls=%d", outcome, loaderCalls)
|
||||||
|
}
|
||||||
|
beforeMissing := loaderCalls
|
||||||
|
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeUsage || loaderCalls != beforeMissing {
|
||||||
|
t.Fatalf("missing ID = (%#v, calls=%d), want usage before acquisition", outcome, loaderCalls)
|
||||||
|
}
|
||||||
|
|
||||||
|
profileOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "profile-only"})
|
||||||
|
if err := commandkit.RegisterPromptKit(profileOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectProfile: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id string) (commandkit.Acquired[*promptkit.ProfileInspection], error) {
|
||||||
|
value, err := engine.InspectProfile(ctx, id)
|
||||||
|
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: value}, err
|
||||||
|
}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess {
|
||||||
|
t.Fatalf("profile-only capability was not installed: %#v", outcome)
|
||||||
|
}
|
||||||
|
outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeUsage {
|
||||||
|
t.Fatalf("profile-only suite installed prompt inspection: %#v", outcome)
|
||||||
|
}
|
||||||
|
promptOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "prompt-only"})
|
||||||
|
if err := commandkit.RegisterPromptKit(promptOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id, version string) (commandkit.Acquired[*promptkit.PromptInspection], error) {
|
||||||
|
value, err := engine.InspectPrompt(ctx, id, version)
|
||||||
|
return commandkit.Acquired[*promptkit.PromptInspection]{Value: value}, err
|
||||||
|
}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess {
|
||||||
|
t.Fatalf("prompt-only capability was not installed: %#v", outcome)
|
||||||
|
}
|
||||||
|
outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeUsage {
|
||||||
|
t.Fatalf("prompt-only suite installed profile inspection: %#v", outcome)
|
||||||
|
}
|
||||||
|
|
||||||
|
emptyApp, _ := commandkit.New(commandkit.Identity{Name: "empty"})
|
||||||
|
if err := commandkit.RegisterPromptKit(emptyApp, commandkit.PromptKitSuite[struct{}]{}); err != nil {
|
||||||
|
t.Fatalf("empty PromptKit suite = %v", err)
|
||||||
|
}
|
||||||
|
outcome = emptyApp.Execute(context.Background(), []string{"help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess {
|
||||||
|
t.Fatalf("empty PromptKit suite changed command family: %#v", outcome)
|
||||||
|
}
|
||||||
|
outcome = emptyApp.Execute(context.Background(), []string{"help", "promptkit"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeUsage {
|
||||||
|
t.Fatalf("empty PromptKit suite installed its namespace: %#v", outcome)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitSuitePassesSelectionAndProjectedContext(t *testing.T) {
|
||||||
|
type appContext struct {
|
||||||
|
Campaign commandkit.Supplied[string]
|
||||||
|
Runtime string
|
||||||
|
}
|
||||||
|
type contextKey struct{}
|
||||||
|
engine := promptKitInspectionEngine(t)
|
||||||
|
ctx := context.WithValue(context.Background(), contextKey{}, "caller")
|
||||||
|
wantSelection := commandkit.Selection{
|
||||||
|
Config: commandkit.Supplied[string]{Value: "main.yml", Set: true},
|
||||||
|
Overlay: commandkit.Supplied[string]{Value: "dev", Set: true},
|
||||||
|
}
|
||||||
|
loaderCalls, contexts, checks := 0, 0, 0
|
||||||
|
var gotSelection commandkit.Selection
|
||||||
|
var gotContext appContext
|
||||||
|
suite, err := commandkit.NewPromptKitSuite(func(gotCtx context.Context, selection commandkit.Selection, input appContext) (*promptkit.Engine, error) {
|
||||||
|
loaderCalls++
|
||||||
|
gotSelection, gotContext = selection, input
|
||||||
|
if gotCtx != ctx || gotCtx.Value(contextKey{}) != "caller" {
|
||||||
|
t.Fatal("engine loader did not receive invocation context")
|
||||||
|
}
|
||||||
|
return engine, nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
suite.NewContext = func() appContext { contexts++; return appContext{Runtime: "invocation cache"} }
|
||||||
|
suite.Selectors = []commandkit.Selector[appContext]{{
|
||||||
|
Binding: commandkit.ValueFlag[appContext](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(input *appContext, value commandkit.Supplied[string]) { input.Campaign = value }),
|
||||||
|
Scopes: []string{"promptkit"},
|
||||||
|
}}
|
||||||
|
suite.CheckInput = func(selection commandkit.Selection, input appContext) error {
|
||||||
|
checks++
|
||||||
|
if selection != wantSelection || input.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) {
|
||||||
|
t.Errorf("preflight inputs = (%+v, %+v)", selection, input)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
|
||||||
|
if err := commandkit.RegisterPromptKit(app, suite); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
outcome := app.Execute(ctx, []string{"promptkit", "show", "profile", "child", "--config", "main.yml", "--overlay", "dev", "--campaign", "spring"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != 1 || contexts != 1 || checks != 1 {
|
||||||
|
t.Fatalf("inspection = (%#v), loader/context/check calls = %d/%d/%d", outcome, loaderCalls, contexts, checks)
|
||||||
|
}
|
||||||
|
if gotSelection != wantSelection || gotContext.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) || gotContext.Runtime != "invocation cache" {
|
||||||
|
t.Fatalf("loader selection/context = (%+v, %+v)", gotSelection, gotContext)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptKitDirectSuiteRetainsDiagnosticsOnNamedLookupFailure(t *testing.T) {
|
||||||
|
warning := commandkit.Diagnostic{Severity: configkit.SeverityWarning, Write: func(w io.Writer) error { _, err := io.WriteString(w, "configuration warning\n"); return err }}
|
||||||
|
app, _ := commandkit.New(commandkit.Identity{Name: "example"})
|
||||||
|
err := commandkit.RegisterPromptKit(app, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(context.Context, commandkit.Selection, struct{}, string, string) (commandkit.Acquired[*promptkit.PromptInspection], error) {
|
||||||
|
return commandkit.Acquired[*promptkit.PromptInspection]{Diagnostics: []commandkit.Diagnostic{warning}}, promptkit.ErrPromptNotFound
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "missing"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, promptkit.ErrPromptNotFound) || len(outcome.Diagnostics) < 1 || outcome.Diagnostics[0].Severity != configkit.SeverityWarning {
|
||||||
|
t.Fatalf("named lookup outcome did not retain acquisition diagnostic: %#v", outcome)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func sortedKeys(values map[string]any) []string {
|
||||||
|
keys := make([]string, 0, len(values))
|
||||||
|
for key := range values {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
for i := 0; i < len(keys); i++ {
|
||||||
|
for j := i + 1; j < len(keys); j++ {
|
||||||
|
if keys[j] < keys[i] {
|
||||||
|
keys[i], keys[j] = keys[j], keys[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return keys
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user