From 1119bda6b720171c708f3e6d03c5c8690b65ea81 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 12 Oct 2026 21:53:52 +0000 Subject: [PATCH] Settle PromptKit request validation roadmap --- docs/roadmap/promptkit-validation.md | 311 +++++++++++++++------------ 1 file changed, 179 insertions(+), 132 deletions(-) diff --git a/docs/roadmap/promptkit-validation.md b/docs/roadmap/promptkit-validation.md index ee4e719..6188723 100644 --- a/docs/roadmap/promptkit-validation.md +++ b/docs/roadmap/promptkit-validation.md @@ -1,8 +1,9 @@ # PromptKit concrete request validation -Status: proposed adoption and feature roadmap. The command placement is agreed; -the recommendations under Open Decisions await review. This document defines -the target and required work, rather than a staged implementation plan. +Status: accepted adoption and feature roadmap; implementation has not started. +Command placement, adapter composition, request construction, artifact syntax, +and initial presentation are settled. This document defines the target and +required work; a later implementation plan will sequence the work. ## Purpose @@ -95,10 +96,31 @@ current CLI contract. PromptKit owns semantic interpretation of the values. The namespace identifies the library; configuration scope selection belongs to the separate `config` family. -The artifact-binding syntax and success presentation are proposed in OD-2 and -OD-3. Application-specific settings continue to use typed application input -and callbacks. The command need not offer a built-in flag for every field in -`RunRequest` to permit applications to use those fields through their builder. +Add repeatable `--input NAME=PATH` for file-backed bindings, translated to native +`promptkit.File(PATH)` references without eagerly reading files. Split at the +first `=`, preserve exact names and paths, and use the last binding for a +duplicate name. Every occurrence is parsed; a later valid binding does not +erase an earlier malformed one. + +| Invocation fragment | Result | +| --- | --- | +| `--input notes=notes.txt` | Bind `notes` to `promptkit.File("notes.txt")`. | +| `--input notes=dir/a=b.txt` | Preserve the path `dir/a=b.txt`. | +| `--input notes=` | Forward an empty file path to native validation. | +| `--input =notes.txt` | Forward the empty input name to native validation. | +| `--input notes=first.txt --input notes=second.txt` | Bind `notes` to `second.txt`. | +| `--input notes` | Usage error because the separator is missing. | + +PromptKit owns semantic artifact validation and loading. CommandKit adds no +input-name whitelist, path restrictions, normalization, shell expansion, or +configuration-relative rebasing. Paths are preserved after shell processing. + +Use `--format yaml` for the initial success report, with YAML as the default. +Application-specific settings continue to use typed application input and +callbacks. Applications can supply inline and domain-derived artifacts through +their builder; a standard inline-input flag is outside this delivery. The +command need not offer a built-in flag for every field in `RunRequest` to permit +applications to use those fields through their builder. ### Public integration and request construction @@ -108,23 +130,96 @@ continue to install only discovery. Both operations belong to the same `requirements` group when present. The existing installed-family ownership, absent-suite behavior, and retained wiring errors apply. -Provide an engine-backed adapter that combines an application engine loader -with an application-owned concrete request builder. The builder receives the -parsed standard request context and bindings, selected application input, and -invocation state, and produces a native `promptkit.RunRequest`. It can supply -application artifacts, inline content, execution/output-contract overrides, -appended messages, credentials, or other supported native settings without -CommandKit defining parallel domain abstractions. +Use a ckprompt-owned envelope containing configuration selection and the native +seed request populated from the standard command inputs: -The adapter should make it straightforward to honor the selected prompt and -operator values while adding application defaults. Document how standard -values reach the builder; do not hide a second merging layer inside the adapter -or introduce independent native-field validation. Reusing a request builder -between validation and application execution is recommended consumer guidance, -not a requirement governing downstream application internals. +```go +type ValidationRequest struct { + Selection ckconfig.Selection + Request promptkit.RunRequest +} +``` -The public API shape and wiring recommendation are in OD-1. Exact declarations -will be finalized before preparing the staged implementation plan. +Add this optional callback field to `Suite[I, S]`: + +```go +ValidateRequest func( + context.Context, ValidationRequest, I, S, +) (commandkit.Acquired[struct{}], error) +``` + +The acquisition container carries diagnostics; validation has no native result +value to invent. Custom callbacks remain available for application-owned runtime +arrangements. + +Provide a suite-returning `WithRequestValidation` helper that takes a suite, the +existing engine-loader shape, and an optional typed builder: + +```go +func WithRequestValidation[I, S any]( + suite Suite[I, S], + load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error), + build func(context.Context, ValidationRequest, I, S) (promptkit.RunRequest, error), +) Suite[I, S] +``` + +The helper attaches the engine-backed validation callback and retains construction +errors, including a nil loader, in the returned suite for installation to report. +Existing suite capabilities and attached issues are preserved. Narrowing the +capabilities afterward does not discard retained wiring errors. Returning a suite +gives these errors the same treatment as existing adapters. + +For application-defined `loadEngine` and `buildRequest` callbacks, composition is: + +```go +suite := ckprompt.FromEngine(loadEngine) +suite = ckprompt.WithRequestValidation(suite, loadEngine, buildRequest) +``` + +A validation-only consumer can start from an empty suite. Given its application +types `Input` and `State`, it can use the standard command-populated request +directly: + +```go +suite := ckprompt.WithRequestValidation( + ckprompt.Suite[Input, State]{}, loadEngine, nil, +) +``` + +A nil builder passes the seed request unchanged to native validation. Standard +flags already supply useful prompt context, variables, and file bindings; +application mapping is optional. `FromEngine` alone continues to supply its +existing inspection callbacks, so validation remains an explicit wiring choice. + +When present, the builder receives invocation context, the selection and seed +request, selected application input, and shared invocation state. It returns +the concrete native request. It can supply application artifacts, inline content, +execution/output-contract overrides, appended messages, credentials, or other +supported native settings. The adapter forwards that result without reapplying +or merging fields afterward. + +Builders should start from the seed, preserve the selected prompt identity and +operator values, and add application defaults deliberately. Map defaults respect +key presence, including a supplied empty value. For example, within a builder: + +```go +request := seed.Request +if request.Vars == nil { + request.Vars = make(map[string]string) +} +if _, supplied := request.Vars["audience"]; !supplied { + request.Vars["audience"] = "internal" +} +return request, nil +``` + +Here `seed` is the builder's `ValidationRequest`; `--var audience=` remains an +explicit empty value. Version, profile, and session fields retain their native +empty-string semantics, including equivalence between omitted and explicitly +empty scalar values. Configuration selection retains its existing presence +information. CommandKit documents these responsibilities without adding a second +native-field validation layer. Reusing a request builder between validation and +application execution is recommended consumer guidance. ### Invocation lifecycle and outcomes @@ -134,17 +229,28 @@ outcome/delivery machinery. Construction, installation, help, and version must leave acquisition and request-building callbacks uninvoked, under the existing [architecture policy](../policy/architecture.md). -Complete applicable CLI preflight and application input checks before shared -state creation and acquisition. Pass the invocation context through request -construction, engine loading, and native validation. Preserve supported native -error identity and configuration diagnostics, including failures during engine -loading. Request building must be able to fail without a fabricated validation -success result. +For the bundled adapter, use this order: + +1. Complete CLI preflight and application input checks. +2. Create shared invocation state. +3. Invoke the builder, if supplied; otherwise use the seed request unchanged. +4. Load the engine using the selected configuration, application input, and state. +5. Invoke native validation with the concrete request. +6. Deliver diagnostics and the success report through the established paths. + +Both builder and loader receive the same invocation state and context. +Applications can explicitly share configuration acquisition through that state. +A builder failure prevents engine loading. Check cancellation between stages, +preserve diagnostics already acquired, and leave later stages uninvoked after +failure. Preserve supported native error identity and configuration diagnostics, +including failures during engine loading. Use the existing classification model: malformed CLI syntax and missing -operands are usage failures; engine loading and native request validation -failures are operational failures. Application input and builder errors retain -the supported classification behavior of CommandKit's error APIs. Preserve +operands are usage failures. Application input-check failures retain their +existing handling. Builder, engine-loader, and native-validation failures are +operational failures with supported error identity preserved. Returned provider +errors do not carry arbitrary usage or domain outcome classifications; +application-owned handlers can return their own outcomes. Preserve native cancellation error categories; do not add a promise that every native failure matches `ctx.Err()`. @@ -153,6 +259,29 @@ after it succeeds. Diagnostics and output failures follow the established delivery contract. The presentation reports the validation outcome without dumping the request, artifact content, credential values, or rendered messages. +### Initial success presentation + +Emit one small YAML report, defaulting to YAML and accepting explicit +`--format yaml` as the only format for this delivery: + +```yaml +prompt_id: briefing +valid: true +``` + +The ID identifies the command's requested prompt. `valid` means that validation +of the concrete request returned nil at this invocation; with the bundled +adapter, this is the result of native preparation. Builder guidance preserves +the selected prompt identity so the confirmation describes the operator's +selection. CommandKit adds no independent identity-validation layer. + +Native validation supplies neither resolved version/profile metadata nor a +prepared snapshot. The report therefore contains no such claims or discovery +inventories. Failures use the existing diagnostic/error outcome path. Output +delivery is part of invocation success. This initial presentation provides an +explicit confirmation for operators and automation; additional formats can +follow concrete needs. + ### Package and dependency ownership All new provider contracts, request adaptation, standard handlers, and @@ -177,21 +306,24 @@ where the selected graph or incorporated material actually changes. ### Integration API and native adapter -Resolve OD-1, then add the optional validation callback and engine-backed -adapter. Reuse existing engine-loading diagnostics and supported error handling. +Add `ValidationRequest`, the optional suite callback, and the suite-returning +`WithRequestValidation` helper. Reuse existing engine-loading diagnostics and +supported error handling. Support concrete request construction from typed application input and standard CLI values, without copying PromptKit's preparation logic. Account for builder -failure and nil/invalid wiring using the established integration conventions. +failure and invalid loader wiring using the established integration conventions; +a nil builder uses the seed unchanged. Keep current discovery and configuration adapters available independently. -`FromEngine` alone cannot invent application artifact bindings or concrete -request defaults; enabling validation must be an explicit wiring choice. +`FromEngine` keeps its current capabilities. Validation is explicitly enabled +through the helper or a custom suite callback, with an application builder only +when additional mapping or defaults are needed. ### Command assembly, bindings, and presentation Extend `ckprompt.Command` to assemble the requirements group from its supplied -show and validate capabilities. Add the validate handler and the selected -OD-2/OD-3 bindings and presentation. Share common binding/report mechanics where +show and validate capabilities. Add the validate handler, file-binding syntax, +and YAML success report. Share common binding/report mechanics where that simplifies the code, preserving the delivered discovery contract. ### Maintained consumer and documentation @@ -201,6 +333,8 @@ offline validation examples for a successful request and a missing required binding. Demonstrate the existing typed domain selectors alongside the new standard bindings. The example should make the different questions answered by show and validate clear without requiring a model service or real credential. +Document the nil-builder path and a builder that preserves supplied map entries +while adding defaults. When implemented, update Go declarations/GoDoc, the CLI reference, consumer guide, and affected orientation/internal descriptions under their canonical @@ -209,16 +343,17 @@ statements about metadata inspection do not accidentally claim that concrete validation performs no rendering, artifact loading, or credential checks. This capability fits the accepted optional-family and owning-library boundaries. -An API or presentation addition alone does not require rewriting accepted ADRs. -Reassess ADR/policy alignment if resolving an open decision changes an -architectural boundary rather than simply adding the capability within it. +No new ADR or revision to accepted ADRs is needed. The selected API and CLI +details belong in their canonical documentation when implemented. ### Completion evidence for implementation Use proportionate public-boundary evidence for independent capability availability, help without acquisition, standard/typed input mapping, fresh -invocation state, builder and loader failures, native validation error identity, -and diagnostic/output delivery. Use a real PromptKit engine for representative +invocation state, nil-builder forwarding, builder default precedence, retained +wiring errors, builder-before-loader behavior, native validation error identity, +and diagnostic/output delivery. Cover builder and loader failures and cancellation +between stages. Use a real PromptKit engine for representative successful validation and a missing binding, with a model-call double that detects accidental execution. Protect continued successful discovery of a partial inventory; do not turn it into a validation failure. @@ -231,92 +366,4 @@ workspace and disposable-module practices in the [developer guide](../development.md). Implementation evidence must not rely solely on a workspace that silently substitutes a sibling PromptKit checkout. -## Open Decisions - -### OD-1: Validation provider and engine-adapter API - -**Recommendation:** Add a separate `Suite.ValidateRequest` callback receiving a -CommandKit request envelope containing configuration selection and a native -`RunRequest` populated from standard command inputs, plus `I` and `S` as with -existing callbacks. Return diagnostics and error through -`commandkit.Acquired[struct{}]` and `error`; validation has no native value to -invent. Provide a separate adapter factory taking the existing engine-loader -shape and a typed request-builder callback, whose result can be assigned to -`Suite.ValidateRequest`. The builder transforms the command's seed request -into the concrete request passed to `Engine.ValidateRequest`; the adapter does -not reapply or merge fields afterward. - -This keeps capability opt-in explicit, reuses diagnostics machinery, and lets -applications share concrete request construction without requiring them to -write engine acquisition/error adaptation again. The builder's application -defaults must respect the selected prompt and native empty-value semantics; -document that responsibility rather than adding duplicate domain checks. -The factory's final name and declarations should be settled with a small -consumer mockup before implementation planning. - -**Alternative:** Provide only the suite callback, and let applications load -the engine, construct the request, invoke native validation, and return -diagnostics themselves. This has the smallest public adapter surface and -supports unusual runtime arrangements. It leaves repeated engine-loading and -diagnostic adaptation in consumers, weakening the infrastructure-reduction -goal. Custom callbacks remain useful even if the recommended helper exists. - -**Alternative:** Extend `FromEngine` with an explicit validation option carrying -the builder. This makes one constructor supply all engine-backed capabilities. -It adds a constructor option and makes validation availability conditional on -an extra argument, whereas a separate helper expresses the independent -capability directly. It is viable if the consumer mockup shows materially -simpler wiring. - -### OD-2: Standard artifact-binding inputs - -**Recommendation:** Add repeatable `--input NAME=PATH` for file-backed bindings, -translated to native `promptkit.File(PATH)` references without eagerly reading -the files. Reuse first-separator assignment parsing, preserve exact names and -paths, and use the last binding for a duplicate name. Let native validation -decide whether the resulting references are valid and loadable. Applications -can use their typed input and builder to supply inline or domain-derived -artifacts. No independent input-name whitelist or path restrictions are needed. - -This supplies a useful common CLI workflow with simple semantics, while leaving -PromptKit responsible for artifact preparation. It avoids defining a new -serialization format for every native request field. Syntax missing `=` is a -usage error; semantic artifact failures belong to native validation. - -**Alternative:** Also add `--input-inline NAME=CONTENT`. This gives all consumers -a direct inline workflow without an application flag. It adds another binding -source and requires an explicit duplicate-name precedence rule across the two -flags; it is justified if representative consumers need that common syntax. -Selecting it would require settling that precedence before implementation. - -**Alternative:** Provide no standard artifact flag and accept bindings only -through application selectors and the builder. This fits applications whose -artifacts come entirely from domain state, but makes an ordinary file-validation -workflow less consistent and recreates avoidable CLI plumbing in consumers. - -### OD-3: Successful validation presentation - -**Recommendation:** Emit a small YAML report with `prompt_id` and `valid: true`, -using YAML as the default and only supported `--format` value for this operation. -The ID identifies the command's requested prompt; `valid` means native -preparation returned nil. Do not claim a resolved version/profile or return -discovery inventories: native validation supplies neither a report nor a -prepared snapshot. Failures use the existing diagnostic/error outcome path, -not a newly invented aggregate validation report. - -This complements requirements show's YAML output and gives operators and -automation an explicit success result. Its limited meaning must be clear in -the CLI reference, especially when an application builder supplies additional -context. Output delivery becomes part of invocation success, as for other -CommandKit reports. - -**Alternative:** Produce no success output and rely on exit status and -diagnostics. This closely follows the native error-only API, needs no output -schema or format flag, and is suitable for scripting. Interactive operators -receive less confirmation, and it differs from the existing configuration -validation reporting convention. - -**Alternative:** Emit a short text confirmation, with optional YAML. This -matches `config validate`'s text-oriented default and can be easier to scan. -It introduces two presentations where the requirements command's current -YAML convention may already satisfy both interactive and automated use. +The decisions needed for an implementation plan are settled.