Add prepared request preview
This commit is contained in:
23
docs/cli.md
23
docs/cli.md
@@ -555,6 +555,26 @@ performs a network probe. See the [PromptKit resource inspection contracts](http
|
|||||||
A missing ID is a usage error; native lookup errors and cancellation retain
|
A missing ID is a usage error; native lookup errors and cancellation retain
|
||||||
supported cause identity.
|
supported cause identity.
|
||||||
|
|
||||||
|
## PromptKit prepared-request preview
|
||||||
|
|
||||||
|
When `Suite.PrepareRequest` is present, `ckprompt.Command(suite)` installs
|
||||||
|
`promptkit show prepared PROMPT_ID` independently of validation and other
|
||||||
|
inspection commands. It accepts the same configuration, selector, profile,
|
||||||
|
session, variable, input, and execution flags as
|
||||||
|
[request validation](#concrete-request-validation):
|
||||||
|
|
||||||
|
```text
|
||||||
|
example promptkit show prepared briefing --llm-input-text topic=roadmap --format json
|
||||||
|
```
|
||||||
|
|
||||||
|
The command returns PromptKit's native prepared-run report using the shared
|
||||||
|
formats; automatic and explicit text use YAML. Preparation can read resources
|
||||||
|
and report rendered application content. It makes no model call and does not
|
||||||
|
guarantee a subsequent execution will succeed. Application callbacks and
|
||||||
|
loaders retain their own effects. Native preparation errors retain supported
|
||||||
|
cause identity. The [PromptKit Prepare API](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/8ff9aa3efdc9f16b9d2729e0a29ea083eb3c3eaf/engine.go)
|
||||||
|
owns the prepared value and upstream preparation behavior.
|
||||||
|
|
||||||
## PromptKit requirements
|
## PromptKit requirements
|
||||||
|
|
||||||
When `Suite.InspectRequirements` is present, `ckprompt.Command(suite)` installs
|
When `Suite.InspectRequirements` is present, `ckprompt.Command(suite)` installs
|
||||||
@@ -676,7 +696,8 @@ contain `=`. CommandKit splits at the first separator and preserves the exact
|
|||||||
key/name and remaining value. Empty names and values are passed to native
|
key/name and remaining value. Empty names and values are passed to native
|
||||||
validation. Text becomes a `promptkit.Inline` reference; file paths become
|
validation. Text becomes a `promptkit.Inline` reference; file paths become
|
||||||
`promptkit.File` references without being read, normalized, expanded, or
|
`promptkit.File` references without being read, normalized, expanded, or
|
||||||
rebased during argument binding. Every occurrence is parsed.
|
rebased during argument binding. Every occurrence is parsed; for each input
|
||||||
|
name the last text or file occurrence wins, regardless of form.
|
||||||
|
|
||||||
On success, the confirmation contains only the requested prompt ID and
|
On success, the confirmation contains only the requested prompt ID and
|
||||||
`valid: true`:
|
`valid: true`:
|
||||||
|
|||||||
@@ -260,17 +260,18 @@ it does not install PromptKit catalog, definition, or requirements commands.
|
|||||||
|
|
||||||
`ckprompt.FromEngine(loadEngine)` supplies nine callbacks: prompt, profile,
|
`ckprompt.FromEngine(loadEngine)` supplies nine callbacks: prompt, profile,
|
||||||
backend, namespace, and collection catalogs; named prompt, profile, and backend
|
backend, namespace, and collection catalogs; named prompt, profile, and backend
|
||||||
inspection; and native requirements discovery. Concrete request validation and
|
inspection; and native requirements discovery. Concrete request validation,
|
||||||
smoke testing are separate capabilities, added explicitly with
|
prepared-request preview, and smoke testing are separate capabilities, added
|
||||||
`ckprompt.WithRequestValidation` and `ckprompt.WithSmokeTesting`. The suite has
|
with `ckprompt.WithRequestValidation`, `ckprompt.WithRequestPreparation`, and
|
||||||
eleven independently optional callbacks in total. `ckprompt.Command(suite)` creates
|
`ckprompt.WithSmokeTesting`. The suite has twelve independently optional
|
||||||
only operations whose callbacks are present; discovery, validation, and smoke
|
callbacks in total. `ckprompt.Command(suite)` creates only operations whose
|
||||||
testing can be installed independently. A composed suite can enable both
|
callbacks are present; these capabilities can be installed independently. A
|
||||||
additional operations:
|
composed suite can enable the additional operations:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
suite := ckprompt.FromEngine(loadEngine)
|
suite := ckprompt.FromEngine(loadEngine)
|
||||||
suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest)
|
suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest)
|
||||||
|
suite = ckprompt.WithRequestPreparation(suite, loadEngine, nil)
|
||||||
suite = ckprompt.WithSmokeTesting(suite, loadEngine, nil)
|
suite = ckprompt.WithSmokeTesting(suite, loadEngine, nil)
|
||||||
suite.NewInput = newInput
|
suite.NewInput = newInput
|
||||||
suite.NewState = newState
|
suite.NewState = newState
|
||||||
@@ -300,7 +301,7 @@ absent, so explicitly supplied empty values stay meaningful. Applications that
|
|||||||
validate and then execute a request are encouraged to share their request
|
validate and then execute a request are encouraged to share their request
|
||||||
construction logic between those paths.
|
construction logic between those paths.
|
||||||
|
|
||||||
The nine catalog and inspection callbacks and the two explicit additions are
|
The catalog and inspection callbacks and the explicit additions are
|
||||||
independently optional. For catalogs without named or requirements inspection,
|
independently optional. For catalogs without named or requirements inspection,
|
||||||
narrow the suite before command capture:
|
narrow the suite before command capture:
|
||||||
|
|
||||||
@@ -313,7 +314,7 @@ definitions.InspectRequirements = nil
|
|||||||
definitionCommand := ckprompt.Command(definitions)
|
definitionCommand := ckprompt.Command(definitions)
|
||||||
```
|
```
|
||||||
|
|
||||||
Each selected inspection, requirements, validation, smoke, or catalog operation
|
Each selected inspection, requirements, validation, preparation, smoke, or catalog operation
|
||||||
prepares one input, applies its captured selectors, checks that input, and
|
prepares one input, applies its captured selectors, checks that input, and
|
||||||
initializes one state value when those steps are reached. The bundled validation
|
initializes one state value when those steps are reached. The bundled validation
|
||||||
adapter delegates concrete request preparation to PromptKit and carries its
|
adapter delegates concrete request preparation to PromptKit and carries its
|
||||||
@@ -326,6 +327,12 @@ and state. `FromEngine` uses PromptKit's public listing and inspection methods,
|
|||||||
including `Engine.InspectRequirements`; validation must be opted into separately
|
including `Engine.InspectRequirements`; validation must be opted into separately
|
||||||
and uses `Engine.ValidateRequest`.
|
and uses `Engine.ValidateRequest`.
|
||||||
|
|
||||||
|
`ckprompt.WithRequestPreparation` adds a separate callback backed by
|
||||||
|
`Engine.Prepare`. Its optional builder receives the seeded request and returns
|
||||||
|
the complete request to prepare. The resulting native report can contain
|
||||||
|
rendered application content. See the [CLI reference](../cli.md#promptkit-prepared-request-preview)
|
||||||
|
for operator behavior.
|
||||||
|
|
||||||
Application-owned commands can use `ckprompt.NewExecutionFlags`,
|
Application-owned commands can use `ckprompt.NewExecutionFlags`,
|
||||||
`ckprompt.NewResourceFlags`, and `ckprompt.NewPromptInputFlags` to bind the
|
`ckprompt.NewResourceFlags`, and `ckprompt.NewPromptInputFlags` to bind the
|
||||||
same canonical request flags into their own input types. Each collection exposes
|
same canonical request flags into their own input types. Each collection exposes
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Shared reporting and PromptKit v0.20.0 implementation plan
|
# Shared reporting and PromptKit v0.20.0 implementation plan
|
||||||
|
|
||||||
Status: implementation in progress; Stages 1–10 complete.
|
Status: implementation in progress; Stages 1–11 complete.
|
||||||
|
|
||||||
## Scope, authority, and execution
|
## Scope, authority, and execution
|
||||||
|
|
||||||
@@ -398,6 +398,12 @@ forwarding, JSON object parsing, and builder replacement at the boundary.
|
|||||||
|
|
||||||
## Stage 11: Ordered literal/file inputs and concrete preview
|
## Stage 11: Ordered literal/file inputs and concrete preview
|
||||||
|
|
||||||
|
**Complete.** Text and file inputs follow command-line occurrence order, and
|
||||||
|
prepared-request preview is independently installable. The root, ckconfig, and
|
||||||
|
ckprompt workspace suites passed with Go 1.27.0; focused tests cover request
|
||||||
|
forwarding, native results and errors, builder replacement, and no model calls.
|
||||||
|
Scoped aliases are verified with their installation in Stage 13.
|
||||||
|
|
||||||
Use `OccurrenceFlag` for named text/file assignments. Split at the first `=`,
|
Use `OccurrenceFlag` for named text/file assignments. Split at the first `=`,
|
||||||
preserve literal empty text, construct native `Inline` or `File` references, and
|
preserve literal empty text, construct native `Inline` or `File` references, and
|
||||||
assign into the same `PromptInputs.Inputs` map as occurrences are replayed.
|
assign into the same `PromptInputs.Inputs` map as occurrences are replayed.
|
||||||
|
|||||||
@@ -71,8 +71,8 @@ type RequirementsRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Suite describes application-owned input, shared state, and optional catalog,
|
// Suite describes application-owned input, shared state, and optional catalog,
|
||||||
// named inspection, requirements discovery, concrete request validation, and
|
// named inspection, requirements discovery, request validation and preparation,
|
||||||
// smoke testing capabilities. Its private issues retain adapter construction
|
// and smoke testing capabilities. Its private issues retain adapter construction
|
||||||
// failures.
|
// failures.
|
||||||
// Factory results and callback effects remain caller-owned: Command does not
|
// Factory results and callback effects remain caller-owned: Command does not
|
||||||
// deep-copy arbitrary objects or repair aliases returned by factories.
|
// deep-copy arbitrary objects or repair aliases returned by factories.
|
||||||
@@ -81,19 +81,19 @@ type RequirementsRequest struct {
|
|||||||
// Factories are not invoked during construction, installation, help, or version,
|
// Factories are not invoked during construction, installation, help, or version,
|
||||||
// and need not run after failure.
|
// and need not run after failure.
|
||||||
type Suite[I, S any] struct {
|
type Suite[I, S any] struct {
|
||||||
// NewInput is called once when inspection, requirements, validation, smoke,
|
// NewInput is called once when inspection, requirements, validation, preparation, smoke,
|
||||||
// or catalog input preparation is reached, before applying selectors and
|
// or catalog input preparation is reached, before applying selectors and
|
||||||
// calling CheckInput.
|
// calling CheckInput.
|
||||||
// If nil, input is the zero value of I, including nil for pointer types.
|
// If nil, input is the zero value of I, including nil for pointer types.
|
||||||
NewInput func() I
|
NewInput func() I
|
||||||
// NewState is called once when an inspection, validation, smoke, or catalog
|
// NewState is called once when an inspection, validation, preparation, smoke, or catalog
|
||||||
// callback is reached, after input checks pass. If nil, state is the zero
|
// callback is reached, after input checks pass. If nil, state is the zero
|
||||||
// value of S, including nil for pointer types.
|
// value of S, including nil for pointer types.
|
||||||
NewState func() S
|
NewState func() S
|
||||||
// Selectors declares the original typed flags accepted by inspection,
|
// Selectors declares the original typed flags accepted by inspection,
|
||||||
// requirements, validation, smoke, and catalog commands.
|
// requirements, validation, preparation, smoke, and catalog commands.
|
||||||
Selectors []commandkit.Flag[I]
|
Selectors []commandkit.Flag[I]
|
||||||
// CheckInput validates selected inspection, requirements, validation, smoke,
|
// CheckInput validates selected inspection, requirements, validation, preparation, smoke,
|
||||||
// or catalog input before state creation.
|
// or catalog input before state creation.
|
||||||
CheckInput func(ckconfig.Selection, I) error
|
CheckInput func(ckconfig.Selection, I) error
|
||||||
// InspectPrompt inspects the selected prompt after input and state
|
// InspectPrompt inspects the selected prompt after input and state
|
||||||
@@ -119,6 +119,9 @@ type Suite[I, S any] struct {
|
|||||||
// context, and an empty value on success; PromptKit errors are returned
|
// context, and an empty value on success; PromptKit errors are returned
|
||||||
// unchanged where supported.
|
// unchanged where supported.
|
||||||
ValidateRequest func(context.Context, ValidationRequest, I, S) (commandkit.Acquired[struct{}], error)
|
ValidateRequest func(context.Context, ValidationRequest, I, S) (commandkit.Acquired[struct{}], error)
|
||||||
|
// PrepareRequest independently enables `promptkit show prepared`, returning
|
||||||
|
// the native prepared request and any acquisition diagnostics.
|
||||||
|
PrepareRequest func(context.Context, PreparationRequest, I, S) (commandkit.Acquired[*promptkit.PreparedRun], error)
|
||||||
// SmokeTest runs one concrete native smoke-test operation after input and
|
// SmokeTest runs one concrete native smoke-test operation after input and
|
||||||
// state preparation. It is optional; nil omits only the smoke-test command.
|
// state preparation. It is optional; nil omits only the smoke-test command.
|
||||||
// The acquisition retains a native report, independent diagnostics, and
|
// The acquisition retains a native report, independent diagnostics, and
|
||||||
|
|||||||
@@ -43,12 +43,12 @@ type variableAssignment struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type inputAssignment struct {
|
type inputAssignment struct {
|
||||||
name string
|
name string
|
||||||
path string
|
value string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Command captures the optional PromptKit catalogs, named inspection,
|
// Command captures the optional PromptKit catalogs, named inspection,
|
||||||
// requirements discovery, concrete request validation, and smoke-test
|
// requirements discovery, concrete request validation and preparation, and smoke-test
|
||||||
// callbacks and adds only supplied operations. Discovery and validation share the
|
// callbacks and adds only supplied operations. Discovery and validation share the
|
||||||
// `requirements` group and remain independently optional. An absent suite
|
// `requirements` group and remain independently optional. An absent suite
|
||||||
// claims no namespace; attached construction issues remain reportable. It adds
|
// claims no namespace; attached construction issues remain reportable. It adds
|
||||||
@@ -61,7 +61,7 @@ type inputAssignment struct {
|
|||||||
// Named resource inspections return native reports through shared formatting.
|
// Named resource inspections return native reports through shared formatting.
|
||||||
// Catalog commands accept config/overlay selection and suite selectors.
|
// Catalog commands accept config/overlay selection and suite selectors.
|
||||||
func Command[I, S any](suite Suite[I, S]) commandkit.Command {
|
func Command[I, S any](suite Suite[I, S]) commandkit.Command {
|
||||||
if suite.InspectPrompt == nil && suite.InspectProfile == nil && suite.InspectBackend == nil && suite.InspectRequirements == nil && suite.ValidateRequest == nil && suite.SmokeTest == nil && suite.ListPrompts == nil && suite.ListProfiles == nil && suite.ListBackends == nil && suite.ListNamespaces == nil && suite.ListCollections == nil {
|
if suite.InspectPrompt == nil && suite.InspectProfile == nil && suite.InspectBackend == nil && suite.InspectRequirements == nil && suite.ValidateRequest == nil && suite.PrepareRequest == nil && suite.SmokeTest == nil && suite.ListPrompts == nil && suite.ListProfiles == nil && suite.ListBackends == nil && suite.ListNamespaces == nil && suite.ListCollections == nil {
|
||||||
command := commandkit.NoCommand()
|
command := commandkit.NoCommand()
|
||||||
for _, issue := range suite.issues {
|
for _, issue := range suite.issues {
|
||||||
command = commandkit.WithError(command, issue)
|
command = commandkit.WithError(command, issue)
|
||||||
@@ -81,6 +81,9 @@ func Command[I, S any](suite Suite[I, S]) commandkit.Command {
|
|||||||
if suite.InspectBackend != nil {
|
if suite.InspectBackend != nil {
|
||||||
showCommands = append(showCommands, newDefinitionCommand(suite, "backend"))
|
showCommands = append(showCommands, newDefinitionCommand(suite, "backend"))
|
||||||
}
|
}
|
||||||
|
if suite.PrepareRequest != nil {
|
||||||
|
showCommands = append(showCommands, newPreparationCommand(suite))
|
||||||
|
}
|
||||||
if len(showCommands) != 0 {
|
if len(showCommands) != 0 {
|
||||||
rootCommands = append(rootCommands, commandkit.Group("show", "Show a named PromptKit definition.", showCommands...))
|
rootCommands = append(rootCommands, commandkit.Group("show", "Show a named PromptKit definition.", showCommands...))
|
||||||
}
|
}
|
||||||
@@ -606,11 +609,9 @@ func parseVariableAssignment(raw string) (variableAssignment, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func parseInputAssignment(raw string) (inputAssignment, error) {
|
func parseInputAssignment(raw string) (inputAssignment, error) {
|
||||||
name, path, found := strings.Cut(raw, "=")
|
name, value, found := strings.Cut(raw, "=")
|
||||||
if !found {
|
if !found {
|
||||||
return inputAssignment{}, fmt.Errorf("input assignment %q must contain =", raw)
|
return inputAssignment{}, fmt.Errorf("input assignment %q must contain =", raw)
|
||||||
}
|
}
|
||||||
return inputAssignment{name: name, path: path}, nil
|
return inputAssignment{name: name, value: value}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func stringDefault(value string) *string { return &value }
|
|
||||||
|
|||||||
98
integrations/ckprompt/preparation.go
Normal file
98
integrations/ckprompt/preparation.go
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
package ckprompt
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"go.maximumdirect.net/ckconfig"
|
||||||
|
commandkit "go.maximumdirect.net/commandkit"
|
||||||
|
promptkit "go.maximumdirect.net/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// PreparationRequest carries the configuration selection and seed native request
|
||||||
|
// supplied to preparation callbacks and builders.
|
||||||
|
type PreparationRequest struct {
|
||||||
|
// Selection identifies the configuration sources used to load the engine.
|
||||||
|
Selection ckconfig.Selection
|
||||||
|
// Request contains the native seed request populated by command inputs.
|
||||||
|
Request promptkit.RunRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithRequestPreparation attaches native prepared-request inspection. The
|
||||||
|
// optional builder may replace the complete seed request before engine loading.
|
||||||
|
// Preparation may read resources and returns rendered application content; it
|
||||||
|
// does not call the model or retain an execution handle.
|
||||||
|
func WithRequestPreparation[I, S any](
|
||||||
|
suite Suite[I, S],
|
||||||
|
load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error),
|
||||||
|
build func(context.Context, PreparationRequest, I, S) (promptkit.RunRequest, error),
|
||||||
|
) Suite[I, S] {
|
||||||
|
if load == nil {
|
||||||
|
issues := append([]error(nil), suite.issues...)
|
||||||
|
suite.issues = append(issues, errors.New("PromptKit engine loader is nil"))
|
||||||
|
return suite
|
||||||
|
}
|
||||||
|
suite.PrepareRequest = func(ctx context.Context, request PreparationRequest, input I, state S) (commandkit.Acquired[*promptkit.PreparedRun], error) {
|
||||||
|
if err := promptKitContextError(ctx); err != nil {
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{}, fmt.Errorf("PromptKit request preparation before building: %w", err)
|
||||||
|
}
|
||||||
|
concrete := request.Request
|
||||||
|
if build != nil {
|
||||||
|
built, err := build(ctx, request, input, state)
|
||||||
|
if err != nil {
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{}, err
|
||||||
|
}
|
||||||
|
concrete = built
|
||||||
|
}
|
||||||
|
if err := promptKitContextError(ctx); err != nil {
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{}, fmt.Errorf("PromptKit request preparation after building: %w", err)
|
||||||
|
}
|
||||||
|
engine, _, messages, err := loadPromptKitEngine(ctx, request.Selection, input, state, "", load)
|
||||||
|
if err != nil {
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{Diagnostics: messages.diagnostics, Explanation: messages.explanation}, err
|
||||||
|
}
|
||||||
|
if err := promptKitContextError(ctx); err != nil {
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{Diagnostics: messages.diagnostics, Explanation: messages.explanation}, fmt.Errorf("PromptKit request preparation before native preparation: %w", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(ctx, concrete)
|
||||||
|
messages = messages.withOperationError(err)
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{Value: prepared, Diagnostics: messages.diagnostics, Explanation: messages.explanation}, err
|
||||||
|
}
|
||||||
|
return suite
|
||||||
|
}
|
||||||
|
|
||||||
|
func newPreparationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
|
||||||
|
spec := newConcreteRequestSpec(suite)
|
||||||
|
spec.Name = "prepared"
|
||||||
|
spec.Summary = "Show a prepared PromptKit request without calling the model."
|
||||||
|
spec.Description = "Prepares the request using resource I/O and reports rendered application content without calling the model."
|
||||||
|
spec.Run = func(invocation commandkit.Invocation, input *concreteRequestInput[I]) commandkit.Outcome {
|
||||||
|
return runRequestPreparation(invocation, suite, input)
|
||||||
|
}
|
||||||
|
return commandkit.NewCommand(spec)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runRequestPreparation[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *concreteRequestInput[I]) commandkit.Outcome {
|
||||||
|
if err := promptKitContextError(invocation.Context); err != nil {
|
||||||
|
return definitionFailure("PromptKit request preparation was canceled.", commandkit.Explanation{}, err)
|
||||||
|
}
|
||||||
|
var state S
|
||||||
|
if suite.NewState != nil {
|
||||||
|
state = suite.NewState()
|
||||||
|
}
|
||||||
|
if err := promptKitContextError(invocation.Context); err != nil {
|
||||||
|
return definitionFailure("PromptKit request preparation was canceled.", commandkit.Explanation{}, err)
|
||||||
|
}
|
||||||
|
request := PreparationRequest{Selection: input.request.Selection, Request: input.request.Request}
|
||||||
|
acquired, err := suite.PrepareRequest(invocation.Context, request, input.selected, state)
|
||||||
|
queuePromptDiagnostics(invocation.Reporter(), acquired.Diagnostics)
|
||||||
|
if err != nil {
|
||||||
|
return definitionFailure(fmt.Sprintf("Unable to prepare the PromptKit request for prompt %q.", input.promptID), acquired.Explanation, err)
|
||||||
|
}
|
||||||
|
if err := promptKitContextError(invocation.Context); err != nil {
|
||||||
|
return definitionFailure(fmt.Sprintf("Preparation of the PromptKit request for prompt %q was canceled.", input.promptID), acquired.Explanation, err)
|
||||||
|
}
|
||||||
|
_ = invocation.Reporter().Report(commandkit.Report{Value: acquired.Value})
|
||||||
|
return commandkit.Success()
|
||||||
|
}
|
||||||
107
integrations/ckprompt/preparation_test.go
Normal file
107
integrations/ckprompt/preparation_test.go
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
package ckprompt
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.maximumdirect.net/ckconfig"
|
||||||
|
commandkit "go.maximumdirect.net/commandkit"
|
||||||
|
promptkit "go.maximumdirect.net/promptkit"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPreparationCommandForwardsOrderedInputsAndNativeReport(t *testing.T) {
|
||||||
|
type key struct{}
|
||||||
|
ctx := context.WithValue(context.Background(), key{}, "context")
|
||||||
|
var received PreparationRequest
|
||||||
|
calls := 0
|
||||||
|
native := &promptkit.PreparedRun{PromptID: "native", Messages: []promptkit.RenderedMessage{{Content: "rendered"}}}
|
||||||
|
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{PrepareRequest: func(got context.Context, req PreparationRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.PreparedRun], error) {
|
||||||
|
if got.Value(key{}) != "context" {
|
||||||
|
t.Fatal("invocation context lost")
|
||||||
|
}
|
||||||
|
calls++
|
||||||
|
received = req
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{Value: native}, nil
|
||||||
|
}}))
|
||||||
|
var output bytes.Buffer
|
||||||
|
result := app.Execute(ctx, []string{"promptkit", "show", "prepared", "briefing", "--config=chosen", "--llm-profile=editorial", "--llm-input-file=x=first.txt", "--llm-input-text=x=a=b", "--llm-input-text=y=", "--llm-input-file=z=path", "--llm-input-text=z=last", "--llm-input-file=z=final", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||||
|
if result.Kind != commandkit.OutcomeSuccess || calls != 1 || received.Request.PromptID != "briefing" || received.Request.ProfileID != "editorial" || received.Selection.Config.Value != "chosen" || !received.Selection.Config.Set {
|
||||||
|
t.Fatalf("outcome/request = %#v %#v", result, received)
|
||||||
|
}
|
||||||
|
want := map[string]promptkit.ArtifactRef{"x": promptkit.Inline("a=b"), "y": promptkit.Inline(""), "z": promptkit.File("final")}
|
||||||
|
if !reflect.DeepEqual(received.Request.Inputs, want) {
|
||||||
|
t.Fatalf("inputs = %#v", received.Request.Inputs)
|
||||||
|
}
|
||||||
|
var report, expected any
|
||||||
|
encoded, err := json.Marshal(native)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(encoded, &expected); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(output.Bytes(), &report); err != nil || !reflect.DeepEqual(report, expected) {
|
||||||
|
t.Fatalf("report = %s (%v)", output.String(), err)
|
||||||
|
}
|
||||||
|
output.Reset()
|
||||||
|
result = app.Execute(ctx, []string{"promptkit", "show", "prepared", "fresh", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||||
|
if result.Kind != commandkit.OutcomeSuccess || received.Request.Inputs != nil {
|
||||||
|
t.Fatalf("fresh request = %#v %#v", result, received)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPreparationErrorsAndHelpDoNotInvokeUnrelatedCallbacks(t *testing.T) {
|
||||||
|
cause := errors.New("preparation failure")
|
||||||
|
calls := 0
|
||||||
|
suite := Suite[struct{}, struct{}]{PrepareRequest: func(context.Context, PreparationRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.PreparedRun], error) {
|
||||||
|
calls++
|
||||||
|
return commandkit.Acquired[*promptkit.PreparedRun]{}, cause
|
||||||
|
}}
|
||||||
|
app := newPromptTestApp(t, Command(suite))
|
||||||
|
var help bytes.Buffer
|
||||||
|
result := app.Execute(context.Background(), []string{"promptkit", "show", "prepared", "--help"}, commandkit.Streams{Out: &help, Err: io.Discard})
|
||||||
|
if result.Kind != commandkit.OutcomeSuccess || calls != 0 || !bytes.Contains(help.Bytes(), []byte("resource I/O")) || !bytes.Contains(help.Bytes(), []byte("rendered application content")) {
|
||||||
|
t.Fatalf("help = %#v %q", result, help.String())
|
||||||
|
}
|
||||||
|
result = app.Execute(context.Background(), []string{"promptkit", "show", "prepared", "missing"}, commandkit.Streams{Out: io.Discard, Err: io.Discard})
|
||||||
|
if result.Kind != commandkit.OutcomeOperational || !errors.Is(result.Cause, cause) || calls != 1 {
|
||||||
|
t.Fatalf("failure = %#v", result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNativePreparationBuilderAndErrorIdentity(t *testing.T) {
|
||||||
|
client := &validationFailOnGenerate{}
|
||||||
|
engine := validationTestEngine(t, adapterPrompt, promptkit.WithLLMClient(client))
|
||||||
|
built := promptkit.RunRequest{PromptID: "briefing", ProfileID: "default", Inputs: map[string]promptkit.ArtifactRef{"topic": promptkit.Inline("builder value")}}
|
||||||
|
ctx := context.Background()
|
||||||
|
want, err := engine.Prepare(ctx, built)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
calls := 0
|
||||||
|
suite := WithRequestPreparation(Suite[struct{}, struct{}]{}, func(context.Context, ckconfig.Selection, struct{}, struct{}) (*promptkit.Engine, error) {
|
||||||
|
return engine, nil
|
||||||
|
}, func(_ context.Context, seed PreparationRequest, _, _ struct{}) (promptkit.RunRequest, error) {
|
||||||
|
calls++
|
||||||
|
if seed.Request.PromptID != "seed" {
|
||||||
|
t.Fatalf("seed = %#v", seed)
|
||||||
|
}
|
||||||
|
return built, nil
|
||||||
|
})
|
||||||
|
got, err := suite.PrepareRequest(ctx, PreparationRequest{Request: promptkit.RunRequest{PromptID: "seed"}}, struct{}{}, struct{}{})
|
||||||
|
if err != nil || calls != 1 || got.Value == nil || got.Value.PromptID != want.PromptID || !reflect.DeepEqual(got.Value.Messages, want.Messages) || client.calls != 0 {
|
||||||
|
t.Fatalf("prepared = %#v, err = %v, calls = %d, model calls = %d", got, err, calls, client.calls)
|
||||||
|
}
|
||||||
|
withoutBuilder := WithRequestPreparation(Suite[struct{}, struct{}]{}, func(context.Context, ckconfig.Selection, struct{}, struct{}) (*promptkit.Engine, error) {
|
||||||
|
return engine, nil
|
||||||
|
}, nil)
|
||||||
|
_, err = withoutBuilder.PrepareRequest(ctx, PreparationRequest{Request: promptkit.RunRequest{PromptID: "missing"}}, struct{}{}, struct{}{})
|
||||||
|
if !errors.Is(err, promptkit.ErrPromptNotFound) {
|
||||||
|
t.Fatalf("native error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -175,17 +175,12 @@ func NewPromptInputFlags[I any](destination func(*I) *PromptInputs) PromptInputF
|
|||||||
destination(i).Vars = values
|
destination(i).Vars = values
|
||||||
})
|
})
|
||||||
inputFlag := func(meta commandkit.FlagMeta, makeRef func(string) promptkit.ArtifactRef) commandkit.Flag[I] {
|
inputFlag := func(meta commandkit.FlagMeta, makeRef func(string) promptkit.ArtifactRef) commandkit.Flag[I] {
|
||||||
return commandkit.RepeatFlag(meta, []inputAssignment(nil), parseInputAssignment, func(i *I, v commandkit.Supplied[[]inputAssignment]) {
|
return commandkit.OccurrenceFlag(meta, parseInputAssignment, func(i *I, v commandkit.Supplied[inputAssignment]) {
|
||||||
if !v.Set {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
input := destination(i)
|
input := destination(i)
|
||||||
if input.Inputs == nil {
|
if input.Inputs == nil {
|
||||||
input.Inputs = make(map[string]promptkit.ArtifactRef)
|
input.Inputs = make(map[string]promptkit.ArtifactRef)
|
||||||
}
|
}
|
||||||
for _, assignment := range v.Value {
|
input.Inputs[v.Value.name] = makeRef(v.Value.value)
|
||||||
input.Inputs[assignment.name] = makeRef(assignment.path)
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return PromptInputFlags[I]{Variable: variable, Text: inputFlag(requestFlags.InputText.meta, promptkit.Inline), File: inputFlag(requestFlags.InputFile.meta, promptkit.File)}
|
return PromptInputFlags[I]{Variable: variable, Text: inputFlag(requestFlags.InputText.meta, promptkit.Inline), File: inputFlag(requestFlags.InputFile.meta, promptkit.File)}
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
commandkit "go.maximumdirect.net/commandkit"
|
commandkit "go.maximumdirect.net/commandkit"
|
||||||
)
|
)
|
||||||
|
|
||||||
type validationCommandInput[I any] struct {
|
type concreteRequestInput[I any] struct {
|
||||||
promptID string
|
promptID string
|
||||||
request ValidationRequest
|
request ValidationRequest
|
||||||
execution ExecutionInput
|
execution ExecutionInput
|
||||||
@@ -22,8 +22,8 @@ type requestValidationReport struct {
|
|||||||
Valid bool `json:"valid"`
|
Valid bool `json:"valid"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
|
func newConcreteRequestSpec[I, S any](suite Suite[I, S]) commandkit.CommandSpec[*concreteRequestInput[I]] {
|
||||||
flags := commandkit.CaptureFlags(suite.Selectors, func(input **validationCommandInput[I]) *commandkit.FlagValues[I] {
|
flags := commandkit.CaptureFlags(suite.Selectors, func(input **concreteRequestInput[I]) *commandkit.FlagValues[I] {
|
||||||
if input == nil || *input == nil {
|
if input == nil || *input == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -31,38 +31,36 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
|
|||||||
})
|
})
|
||||||
flags = append(flags,
|
flags = append(flags,
|
||||||
commandkit.ValueFlag(commandkit.FlagMeta{Name: "config", Usage: "Configuration input path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
|
commandkit.ValueFlag(commandkit.FlagMeta{Name: "config", Usage: "Configuration input path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
|
||||||
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
|
func(input **concreteRequestInput[I], value commandkit.Supplied[string]) {
|
||||||
(*input).request.Selection.Config = value
|
(*input).request.Selection.Config = value
|
||||||
}),
|
}),
|
||||||
commandkit.ValueFlag(commandkit.FlagMeta{Name: "overlay", Usage: "Configuration overlay path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
|
commandkit.ValueFlag(commandkit.FlagMeta{Name: "overlay", Usage: "Configuration overlay path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
|
||||||
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
|
func(input **concreteRequestInput[I], value commandkit.Supplied[string]) {
|
||||||
(*input).request.Selection.Overlay = value
|
(*input).request.Selection.Overlay = value
|
||||||
}),
|
}),
|
||||||
)
|
)
|
||||||
|
|
||||||
resources := NewResourceFlags(func(input **validationCommandInput[I]) *ResourceInput { return &(*input).resources })
|
resources := NewResourceFlags(func(input **concreteRequestInput[I]) *ResourceInput { return &(*input).resources })
|
||||||
execution := NewExecutionFlags(func(input **validationCommandInput[I]) *ExecutionInput { return &(*input).execution })
|
execution := NewExecutionFlags(func(input **concreteRequestInput[I]) *ExecutionInput { return &(*input).execution })
|
||||||
inputs := NewPromptInputFlags(func(input **validationCommandInput[I]) *PromptInputs { return &(*input).inputs })
|
inputs := NewPromptInputFlags(func(input **concreteRequestInput[I]) *PromptInputs { return &(*input).inputs })
|
||||||
flags = append(flags, resources.Profile, resources.SessionID, inputs.Variable, inputs.Text, inputs.File)
|
flags = append(flags, resources.Profile, resources.SessionID, inputs.Variable, inputs.Text, inputs.File)
|
||||||
flags = append(flags, execution.All()...)
|
flags = append(flags, execution.All()...)
|
||||||
|
|
||||||
return commandkit.NewCommand(commandkit.CommandSpec[*validationCommandInput[I]]{
|
return commandkit.CommandSpec[*concreteRequestInput[I]]{
|
||||||
Name: "validate",
|
NewInput: func() *concreteRequestInput[I] {
|
||||||
Summary: "Validate a concrete PromptKit request without executing it.",
|
return &concreteRequestInput[I]{}
|
||||||
NewInput: func() *validationCommandInput[I] {
|
|
||||||
return &validationCommandInput[I]{}
|
|
||||||
},
|
},
|
||||||
Flags: flags,
|
Flags: flags,
|
||||||
Arguments: []commandkit.Argument[*validationCommandInput[I]]{
|
Arguments: []commandkit.Argument[*concreteRequestInput[I]]{
|
||||||
commandkit.ValueArgument(commandkit.ArgumentMeta{Name: "id", Usage: "Prompt ID."}, true, parseDefinitionString,
|
commandkit.ValueArgument(commandkit.ArgumentMeta{Name: "id", Usage: "Prompt ID."}, true, parseDefinitionString,
|
||||||
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
|
func(input **concreteRequestInput[I], value commandkit.Supplied[string]) {
|
||||||
(*input).promptID = value.Value
|
(*input).promptID = value.Value
|
||||||
(*input).request.Request.PromptID = value.Value
|
(*input).request.Request.PromptID = value.Value
|
||||||
}),
|
}),
|
||||||
},
|
},
|
||||||
CheckInput: func(input *validationCommandInput[I]) error {
|
CheckInput: func(input *concreteRequestInput[I]) error {
|
||||||
if input == nil {
|
if input == nil {
|
||||||
return errors.New("PromptKit validation input must not be nil")
|
return errors.New("PromptKit request input must not be nil")
|
||||||
}
|
}
|
||||||
input.request.Request.ProfileID = input.resources.Profile.Value
|
input.request.Request.ProfileID = input.resources.Profile.Value
|
||||||
input.request.Request.SessionID = input.resources.SessionID.Value
|
input.request.Request.SessionID = input.resources.SessionID.Value
|
||||||
@@ -77,13 +75,20 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
|
|||||||
input.selected = selected
|
input.selected = selected
|
||||||
return nil
|
return nil
|
||||||
},
|
},
|
||||||
Run: func(invocation commandkit.Invocation, input *validationCommandInput[I]) commandkit.Outcome {
|
}
|
||||||
return runRequestValidation(invocation, suite, input)
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func runRequestValidation[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *validationCommandInput[I]) commandkit.Outcome {
|
func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
|
||||||
|
spec := newConcreteRequestSpec(suite)
|
||||||
|
spec.Name = "validate"
|
||||||
|
spec.Summary = "Validate a concrete PromptKit request without executing it."
|
||||||
|
spec.Run = func(invocation commandkit.Invocation, input *concreteRequestInput[I]) commandkit.Outcome {
|
||||||
|
return runRequestValidation(invocation, suite, input)
|
||||||
|
}
|
||||||
|
return commandkit.NewCommand(spec)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runRequestValidation[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *concreteRequestInput[I]) commandkit.Outcome {
|
||||||
if err := promptKitContextError(invocation.Context); err != nil {
|
if err := promptKitContextError(invocation.Context); err != nil {
|
||||||
return definitionFailure("PromptKit request validation was canceled.", commandkit.Explanation{}, err)
|
return definitionFailure("PromptKit request validation was canceled.", commandkit.Explanation{}, err)
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user