base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Settle PromptKit request validation roadmap

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-12 21:53:52 +00:00
parent 87391903f4
commit 1119bda6b7
1 changed files with 179 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files

311
docs/roadmap/promptkit-validation.md
View File

@@ -1,8 +1,9 @@
# PromptKit concrete request validation
Status: proposed adoption and feature roadmap. The command placement is agreed;
the recommendations under Open Decisions await review. This document defines
the target and required work, rather than a staged implementation plan.
Status: accepted adoption and feature roadmap; implementation has not started.
Command placement, adapter composition, request construction, artifact syntax,
and initial presentation are settled. This document defines the target and
required work; a later implementation plan will sequence the work.
## Purpose
@@ -95,10 +96,31 @@ current CLI contract. PromptKit owns semantic interpretation of the values.
The namespace identifies the library; configuration scope selection belongs
to the separate `config` family.
The artifact-binding syntax and success presentation are proposed in OD-2 and
OD-3. Application-specific settings continue to use typed application input
and callbacks. The command need not offer a built-in flag for every field in
`RunRequest` to permit applications to use those fields through their builder.
Add repeatable `--input NAME=PATH` for file-backed bindings, translated to native
`promptkit.File(PATH)` references without eagerly reading files. Split at the
first `=`, preserve exact names and paths, and use the last binding for a
duplicate name. Every occurrence is parsed; a later valid binding does not
erase an earlier malformed one.
| Invocation fragment | Result |
| --- | --- |
| `--input notes=notes.txt` | Bind `notes` to `promptkit.File("notes.txt")`. |
| `--input notes=dir/a=b.txt` | Preserve the path `dir/a=b.txt`. |
| `--input notes=` | Forward an empty file path to native validation. |
| `--input =notes.txt` | Forward the empty input name to native validation. |
| `--input notes=first.txt --input notes=second.txt` | Bind `notes` to `second.txt`. |
| `--input notes` | Usage error because the separator is missing. |
PromptKit owns semantic artifact validation and loading. CommandKit adds no
input-name whitelist, path restrictions, normalization, shell expansion, or
configuration-relative rebasing. Paths are preserved after shell processing.
Use `--format yaml` for the initial success report, with YAML as the default.
Application-specific settings continue to use typed application input and
callbacks. Applications can supply inline and domain-derived artifacts through
their builder; a standard inline-input flag is outside this delivery. The
command need not offer a built-in flag for every field in `RunRequest` to permit
applications to use those fields through their builder.
### Public integration and request construction
@@ -108,23 +130,96 @@ continue to install only discovery. Both operations belong to the same
`requirements` group when present. The existing installed-family ownership,
absent-suite behavior, and retained wiring errors apply.
Provide an engine-backed adapter that combines an application engine loader
with an application-owned concrete request builder. The builder receives the
parsed standard request context and bindings, selected application input, and
invocation state, and produces a native `promptkit.RunRequest`. It can supply
application artifacts, inline content, execution/output-contract overrides,
appended messages, credentials, or other supported native settings without
CommandKit defining parallel domain abstractions.
Use a ckprompt-owned envelope containing configuration selection and the native
seed request populated from the standard command inputs:
The adapter should make it straightforward to honor the selected prompt and
operator values while adding application defaults. Document how standard
values reach the builder; do not hide a second merging layer inside the adapter
or introduce independent native-field validation. Reusing a request builder
between validation and application execution is recommended consumer guidance,
not a requirement governing downstream application internals.
```go
type ValidationRequest struct {
Selection ckconfig.Selection
Request promptkit.RunRequest
}
```
The public API shape and wiring recommendation are in OD-1. Exact declarations
will be finalized before preparing the staged implementation plan.
Add this optional callback field to `Suite[I, S]`:
```go
ValidateRequest func(
context.Context, ValidationRequest, I, S,
) (commandkit.Acquired[struct{}], error)
```
The acquisition container carries diagnostics; validation has no native result
value to invent. Custom callbacks remain available for application-owned runtime
arrangements.
Provide a suite-returning `WithRequestValidation` helper that takes a suite, the
existing engine-loader shape, and an optional typed builder:
```go
func WithRequestValidation[I, S any](
suite Suite[I, S],
load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error),
build func(context.Context, ValidationRequest, I, S) (promptkit.RunRequest, error),
) Suite[I, S]
```
The helper attaches the engine-backed validation callback and retains construction
errors, including a nil loader, in the returned suite for installation to report.
Existing suite capabilities and attached issues are preserved. Narrowing the
capabilities afterward does not discard retained wiring errors. Returning a suite
gives these errors the same treatment as existing adapters.
For application-defined `loadEngine` and `buildRequest` callbacks, composition is:
```go
suite := ckprompt.FromEngine(loadEngine)
suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest)
```
A validation-only consumer can start from an empty suite. Given its application
types `Input` and `State`, it can use the standard command-populated request
directly:
```go
suite := ckprompt.WithRequestValidation(
ckprompt.Suite[Input, State]{}, loadEngine, nil,
)
```
A nil builder passes the seed request unchanged to native validation. Standard
flags already supply useful prompt context, variables, and file bindings;
application mapping is optional. `FromEngine` alone continues to supply its
existing inspection callbacks, so validation remains an explicit wiring choice.
When present, the builder receives invocation context, the selection and seed
request, selected application input, and shared invocation state. It returns
the concrete native request. It can supply application artifacts, inline content,
execution/output-contract overrides, appended messages, credentials, or other
supported native settings. The adapter forwards that result without reapplying
or merging fields afterward.
Builders should start from the seed, preserve the selected prompt identity and
operator values, and add application defaults deliberately. Map defaults respect
key presence, including a supplied empty value. For example, within a builder:
```go
request := seed.Request
if request.Vars == nil {
request.Vars = make(map[string]string)
}
if _, supplied := request.Vars["audience"]; !supplied {
request.Vars["audience"] = "internal"
}
return request, nil
```
Here `seed` is the builder's `ValidationRequest`; `--var audience=` remains an
explicit empty value. Version, profile, and session fields retain their native
empty-string semantics, including equivalence between omitted and explicitly
empty scalar values. Configuration selection retains its existing presence
information. CommandKit documents these responsibilities without adding a second
native-field validation layer. Reusing a request builder between validation and
application execution is recommended consumer guidance.
### Invocation lifecycle and outcomes
@@ -134,17 +229,28 @@ outcome/delivery machinery. Construction, installation, help, and version must
leave acquisition and request-building callbacks uninvoked, under the existing
[architecture policy](../policy/architecture.md).
Complete applicable CLI preflight and application input checks before shared
state creation and acquisition. Pass the invocation context through request
construction, engine loading, and native validation. Preserve supported native
error identity and configuration diagnostics, including failures during engine
loading. Request building must be able to fail without a fabricated validation
success result.
For the bundled adapter, use this order:
1. Complete CLI preflight and application input checks.
2. Create shared invocation state.
3. Invoke the builder, if supplied; otherwise use the seed request unchanged.
4. Load the engine using the selected configuration, application input, and state.
5. Invoke native validation with the concrete request.
6. Deliver diagnostics and the success report through the established paths.
Both builder and loader receive the same invocation state and context.
Applications can explicitly share configuration acquisition through that state.
A builder failure prevents engine loading. Check cancellation between stages,
preserve diagnostics already acquired, and leave later stages uninvoked after
failure. Preserve supported native error identity and configuration diagnostics,
including failures during engine loading.
Use the existing classification model: malformed CLI syntax and missing
operands are usage failures; engine loading and native request validation
failures are operational failures. Application input and builder errors retain
the supported classification behavior of CommandKit's error APIs. Preserve
operands are usage failures. Application input-check failures retain their
existing handling. Builder, engine-loader, and native-validation failures are
operational failures with supported error identity preserved. Returned provider
errors do not carry arbitrary usage or domain outcome classifications;
application-owned handlers can return their own outcomes. Preserve
native cancellation error categories; do not add a promise that every native
failure matches `ctx.Err()`.
@@ -153,6 +259,29 @@ after it succeeds. Diagnostics and output failures follow the established
delivery contract. The presentation reports the validation outcome without
dumping the request, artifact content, credential values, or rendered messages.
### Initial success presentation
Emit one small YAML report, defaulting to YAML and accepting explicit
`--format yaml` as the only format for this delivery:
```yaml
prompt_id: briefing
valid: true
```
The ID identifies the command's requested prompt. `valid` means that validation
of the concrete request returned nil at this invocation; with the bundled
adapter, this is the result of native preparation. Builder guidance preserves
the selected prompt identity so the confirmation describes the operator's
selection. CommandKit adds no independent identity-validation layer.
Native validation supplies neither resolved version/profile metadata nor a
prepared snapshot. The report therefore contains no such claims or discovery
inventories. Failures use the existing diagnostic/error outcome path. Output
delivery is part of invocation success. This initial presentation provides an
explicit confirmation for operators and automation; additional formats can
follow concrete needs.
### Package and dependency ownership
All new provider contracts, request adaptation, standard handlers, and
@@ -177,21 +306,24 @@ where the selected graph or incorporated material actually changes.
### Integration API and native adapter
Resolve OD-1, then add the optional validation callback and engine-backed
adapter. Reuse existing engine-loading diagnostics and supported error handling.
Add `ValidationRequest`, the optional suite callback, and the suite-returning
`WithRequestValidation` helper. Reuse existing engine-loading diagnostics and
supported error handling.
Support concrete request construction from typed application input and standard
CLI values, without copying PromptKit's preparation logic. Account for builder
failure and nil/invalid wiring using the established integration conventions.
failure and invalid loader wiring using the established integration conventions;
a nil builder uses the seed unchanged.
Keep current discovery and configuration adapters available independently.
`FromEngine` alone cannot invent application artifact bindings or concrete
request defaults; enabling validation must be an explicit wiring choice.
`FromEngine` keeps its current capabilities. Validation is explicitly enabled
through the helper or a custom suite callback, with an application builder only
when additional mapping or defaults are needed.
### Command assembly, bindings, and presentation
Extend `ckprompt.Command` to assemble the requirements group from its supplied
show and validate capabilities. Add the validate handler and the selected
OD-2/OD-3 bindings and presentation. Share common binding/report mechanics where
show and validate capabilities. Add the validate handler, file-binding syntax,
and YAML success report. Share common binding/report mechanics where
that simplifies the code, preserving the delivered discovery contract.
### Maintained consumer and documentation
@@ -201,6 +333,8 @@ offline validation examples for a successful request and a missing required
binding. Demonstrate the existing typed domain selectors alongside the new
standard bindings. The example should make the different questions answered
by show and validate clear without requiring a model service or real credential.
Document the nil-builder path and a builder that preserves supplied map entries
while adding defaults.
When implemented, update Go declarations/GoDoc, the CLI reference, consumer
guide, and affected orientation/internal descriptions under their canonical
@@ -209,16 +343,17 @@ statements about metadata inspection do not accidentally claim that concrete
validation performs no rendering, artifact loading, or credential checks.
This capability fits the accepted optional-family and owning-library boundaries.
An API or presentation addition alone does not require rewriting accepted ADRs.
Reassess ADR/policy alignment if resolving an open decision changes an
architectural boundary rather than simply adding the capability within it.
No new ADR or revision to accepted ADRs is needed. The selected API and CLI
details belong in their canonical documentation when implemented.
### Completion evidence for implementation
Use proportionate public-boundary evidence for independent capability
availability, help without acquisition, standard/typed input mapping, fresh
invocation state, builder and loader failures, native validation error identity,
and diagnostic/output delivery. Use a real PromptKit engine for representative
invocation state, nil-builder forwarding, builder default precedence, retained
wiring errors, builder-before-loader behavior, native validation error identity,
and diagnostic/output delivery. Cover builder and loader failures and cancellation
between stages. Use a real PromptKit engine for representative
successful validation and a missing binding, with a model-call double that
detects accidental execution. Protect continued successful discovery of a
partial inventory; do not turn it into a validation failure.
@@ -231,92 +366,4 @@ workspace and disposable-module practices in the
[developer guide](../development.md). Implementation evidence must not rely
solely on a workspace that silently substitutes a sibling PromptKit checkout.
## Open Decisions
### OD-1: Validation provider and engine-adapter API
**Recommendation:** Add a separate `Suite.ValidateRequest` callback receiving a
CommandKit request envelope containing configuration selection and a native
`RunRequest` populated from standard command inputs, plus `I` and `S` as with
existing callbacks. Return diagnostics and error through
`commandkit.Acquired[struct{}]` and `error`; validation has no native value to
invent. Provide a separate adapter factory taking the existing engine-loader
shape and a typed request-builder callback, whose result can be assigned to
`Suite.ValidateRequest`. The builder transforms the command's seed request
into the concrete request passed to `Engine.ValidateRequest`; the adapter does
not reapply or merge fields afterward.
This keeps capability opt-in explicit, reuses diagnostics machinery, and lets
applications share concrete request construction without requiring them to
write engine acquisition/error adaptation again. The builder's application
defaults must respect the selected prompt and native empty-value semantics;
document that responsibility rather than adding duplicate domain checks.
The factory's final name and declarations should be settled with a small
consumer mockup before implementation planning.
**Alternative:** Provide only the suite callback, and let applications load
the engine, construct the request, invoke native validation, and return
diagnostics themselves. This has the smallest public adapter surface and
supports unusual runtime arrangements. It leaves repeated engine-loading and
diagnostic adaptation in consumers, weakening the infrastructure-reduction
goal. Custom callbacks remain useful even if the recommended helper exists.
**Alternative:** Extend `FromEngine` with an explicit validation option carrying
the builder. This makes one constructor supply all engine-backed capabilities.
It adds a constructor option and makes validation availability conditional on
an extra argument, whereas a separate helper expresses the independent
capability directly. It is viable if the consumer mockup shows materially
simpler wiring.
### OD-2: Standard artifact-binding inputs
**Recommendation:** Add repeatable `--input NAME=PATH` for file-backed bindings,
translated to native `promptkit.File(PATH)` references without eagerly reading
the files. Reuse first-separator assignment parsing, preserve exact names and
paths, and use the last binding for a duplicate name. Let native validation
decide whether the resulting references are valid and loadable. Applications
can use their typed input and builder to supply inline or domain-derived
artifacts. No independent input-name whitelist or path restrictions are needed.
This supplies a useful common CLI workflow with simple semantics, while leaving
PromptKit responsible for artifact preparation. It avoids defining a new
serialization format for every native request field. Syntax missing `=` is a
usage error; semantic artifact failures belong to native validation.
**Alternative:** Also add `--input-inline NAME=CONTENT`. This gives all consumers
a direct inline workflow without an application flag. It adds another binding
source and requires an explicit duplicate-name precedence rule across the two
flags; it is justified if representative consumers need that common syntax.
Selecting it would require settling that precedence before implementation.
**Alternative:** Provide no standard artifact flag and accept bindings only
through application selectors and the builder. This fits applications whose
artifacts come entirely from domain state, but makes an ordinary file-validation
workflow less consistent and recreates avoidable CLI plumbing in consumers.
### OD-3: Successful validation presentation
**Recommendation:** Emit a small YAML report with `prompt_id` and `valid: true`,
using YAML as the default and only supported `--format` value for this operation.
The ID identifies the command's requested prompt; `valid` means native
preparation returned nil. Do not claim a resolved version/profile or return
discovery inventories: native validation supplies neither a report nor a
prepared snapshot. Failures use the existing diagnostic/error outcome path,
not a newly invented aggregate validation report.
This complements requirements show's YAML output and gives operators and
automation an explicit success result. Its limited meaning must be clear in
the CLI reference, especially when an application builder supplies additional
context. Output delivery becomes part of invocation success, as for other
CommandKit reports.
**Alternative:** Produce no success output and rely on exit status and
diagnostics. This closely follows the native error-only API, needs no output
schema or format flag, and is suitable for scripting. Interactive operators
receive less confirmation, and it differs from the existing configuration
validation reporting convention.
**Alternative:** Emit a short text confirmation, with optional YAML. This
matches `config validate`'s text-oriented default and can be easier to scan.
It introduces two presentations where the requirements command's current
YAML convention may already satisfy both interactive and automated use.
The decisions needed for an implementation plan are settled.