base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan PromptKit request validation implementation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-12 22:05:16 +00:00
parent 1119bda6b7
commit 3218222e53
2 changed files with 449 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

448
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,448 @@
# PromptKit request validation implementation plan
Status: ready for implementation; no implementation stages have been completed.
## Authority, scope, and baseline
Follow the stages below in order. Each stage is sized for one Luna implementation
prompt and identifies its changes, evidence, and completion condition. The
[accepted feature roadmap](promptkit-validation.md) owns the target state and
user choices. The [architecture policy](../policy/architecture.md), accepted
[ADRs](../adr/), and existing public contracts remain authoritative.
The planning baseline is commit `1119bda6b720171c708f3e6d03c5c8690b65ea81`.
The implementation still requires PromptKit v0.14.0; the accepted roadmap adds
v0.15.0 adoption and `promptkit requirements validate PROMPT_ID`.
The deliverable is the complete accepted feature, including the independently
optional callback, suite-returning helper, nil-builder path, file bindings,
YAML confirmation, maintained consumer, and canonical documentation. Existing
discovery, catalogs, definitions, configuration operations, and application
commands keep their current contracts.
## Working instructions
- Start with [development guidance](../development.md) and applicable policies.
Prefer the repository knowledge graph for code discovery; verify snippets
against current source when graph evidence is incomplete or stale.
- Implement in `integrations/ckprompt`, using public core, ckconfig, ConfigKit,
and PromptKit APIs. Core and ckconfig need no new public API or PromptKit
dependency for this feature.
- Use `Engine.ValidateRequest` directly. Discovery completeness is not a
validation gate. Avoid parallel native validation rules, identity enforcement,
request remerging, automatic execution, or prepared-run retention.
- Preserve caller ownership of callback objects, input, and state. Omitted
factories supply zero values, including nil pointers. No arbitrary deep
copies, automatic pointer repair, or concurrent-dispatch guarantee is added.
- Keep machinery proportionate. Private file and helper names below identify
a concrete implementation route; equivalent simple organization within the
owning package is acceptable. Public names and signatures are settled.
- Apply `gofmt` to changed Go files. Add focused tests alongside the stage that
owns each behavior; reuse existing helpers where useful without duplicating
PromptKit's suite. Current tests are evidence, not a source of new policy.
- Preserve the ignored local workspace and any existing replacements. Do not
commit workspace files or sibling-directory replacements. The selected
upstream PromptKit must be the released v0.15.0 module.
- Commit each completed nonempty stage under the
[agent commit policy](../policy/agents.md#git-commit-policy). Record concise
completion markers and actual validation evidence beneath the corresponding
stage here. Never mark a future stage complete merely because its tests pass
incidentally. Do not push, tag, publish, or rewrite historical release notes
as part of this sprint.
The validation commands in this plan are for the implementation sprint. This
planning pass creates documentation only.
## Stage 1: Adopt the released PromptKit dependency and notices
**Primary files:** `integrations/ckprompt/go.mod`, its `go.sum`, and
`integrations/ckprompt/docs/oss/attribution.md`.
1. Upgrade only the selected PromptKit requirement to v0.15.0. Keep Go 1.26.0,
ConfigKit v0.7.1, and the existing CommandKit prerequisite version selections.
Review any actual transitive graph changes rather than upgrading unrelated
dependencies.
2. Resolve module metadata with the real released PromptKit. If unpublished
CommandKit prerequisite versions prevent workspace-off operations, use
temporary copies of ckprompt's module and sum files as described under
Stage 9. Add replacements only for core and ckconfig in those copies, run
the targeted upgrade/tidy there, then remove those temporary replacements
before transferring the resulting metadata to the owning checked-in files.
Preserve unrelated existing checksum entries unless normal tidy removes them.
3. Update the **integration's** PromptKit attribution entry from v0.14.0 to
v0.15.0 and pin its source/notice links to
`635674c066d0b432187f96e72b9c03d085443ac5`. The root attribution document is
not the owner of this entry. Review released copyright/license material;
update other entries or texts only for actual changes in selected material.
4. Confirm the released dependency provides `Engine.ValidateRequest` and that
no committed manifest includes a sibling PromptKit replacement.
**Validation:** Run ckprompt's existing suite with the upgraded dependency;
inspect its selected module version and metadata diff. Use the module practices
in Stage 9 if required. Prose and attribution checks follow
[OSS](../policy/oss.md) and [documentation](../policy/documentation.md) policy.
**Complete when:** ckprompt selects released PromptKit v0.15.0, its current
suite passes, notices identify that release, and module boundaries are unchanged.
## Stage 2: Add the public capability and bundled validation adapter
**Primary files:** `integrations/ckprompt/adapters.go`, a new owning-package
`validation.go`, and focused adapter tests.
1. Add these exact public contracts, with GoDoc:
```go
type ValidationRequest struct {
Selection ckconfig.Selection
Request promptkit.RunRequest
}
// Additional field in Suite[I, S]:
ValidateRequest func(
context.Context, ValidationRequest, I, S,
) (commandkit.Acquired[struct{}], error)
func WithRequestValidation[I, S any](
suite Suite[I, S],
load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error),
build func(context.Context, ValidationRequest, I, S) (promptkit.RunRequest, error),
) Suite[I, S]
```
2. Return the supplied suite with its engine-backed `ValidateRequest` callback
attached. Preserve other callbacks, factories, selectors, checks, and
attached issues. With a nil loader, retain a wiring error on the returned
suite and return without constructing a usable new validation callback.
When adding an issue, use an independent issues slice so the operation does
not overwrite a source suite's retained metadata through slice capacity.
Do not deep-copy caller-owned callback objects or selector bindings here.
3. Leave `FromEngine`'s existing five capabilities unchanged. A nil builder is
valid and means forwarding the seed request unchanged.
4. The attached callback checks cancellation before building, uses the builder
if present, checks cancellation again after successful building, acquires
the engine through `loadPromptKitEngine`, and invokes
`engine.ValidateRequest(ctx, concreteRequest)`. Pass the same `I`, `S`, and
invocation context to builder and loader. Use the envelope's original
configuration selection for loading.
5. A builder failure returns its supported error identity and prevents engine
loading. A loader failure returns the acquired configuration diagnostics and
error. A native failure returns that error with the acquired diagnostics.
Check operation errors before substituting a later context error, preserving
PromptKit's native cancellation category. Reuse `promptKitContextError` and
the existing engine-loading diagnostic adaptation where applicable.
6. Treat `Acquired[struct{}]` as diagnostics plus an empty value, not a Boolean
result or nil-result protocol. Builder output is the authoritative concrete
request: forward it without reapplying seed fields, merging maps, or checking
its prompt identity independently. Successful builder calls have no separate
diagnostic-return field in this selected API.
**Validation:** Add direct adapter tests for composition preserving existing
suite members, nil-builder success, builder failure before loading, and retained
nil-loader issues. Confirm installation reports retained issues even after all
capabilities are cleared. Run the focused tests and ckprompt suite.
**Complete when:** the public API and adapter work without command dispatch,
with explicit capability opt-in and the selected wiring/error behavior.
## Stage 3: Verify native delegation, request construction, and cancellation
**Primary files:** a focused `integrations/ckprompt/validation_adapters_test.go`
or equivalent owning adapter tests.
1. Use a real offline PromptKit engine with synthetic prompts/profiles and a
counting or fail-on-call model client. Demonstrate successful concrete
validation and a missing required binding, asserting zero model calls.
2. Compare representative adapter failures with native `ValidateRequest` using
the same concrete request, or assert the documented native category with
`errors.Is`. Cover an invalid selection and missing/unloadable artifact
without reproducing every native preparation rule.
3. Demonstrate that a builder receives the seed, context, selection, typed input,
and shared state, then can add inline bindings and request settings. Use an
observable native effect, such as supplying a needed variable/artifact or
invalid appended message, to detect discarded builder output. The adapter
must not silently restore seed fields afterward.
4. Cover a nil engine result and a loader's real `*configkit.LoadError`, including
its retained diagnostic records and error identity. Reuse the existing
fixture and diagnostic helpers rather than a new abstraction framework.
5. Cover cancellation before building, after successful building, and during
or after loading. Verify later work is skipped and already acquired
diagnostics survive. For cancellation returned during native artifact
loading, assert PromptKit's applicable operation category; do not require
a general `errors.Is(err, context.Canceled)` relationship for native errors.
**Validation:** Run these focused adapter tests and the ckprompt suite. Prefer
behavioral assertions; exact ordering assertions belong only to the explicitly
selected builder-before-loader lifecycle.
**Complete when:** the adapter's native authority, builder forwarding, failure
handling, and cancellation boundaries have meaningful public-boundary evidence.
## Stage 4: Implement the validate command and requirements group composition
**Primary files:** `integrations/ckprompt/command.go`, a new private
`validation_command.go` in the same package, and basic command tests.
1. Include `Suite.ValidateRequest` in `Command`'s capability/absence check.
Assemble one `requirements` group from the independently supplied show and
validate declarations. Continue attaching retained suite issues and using
the existing root-only family declaration. Update help summaries to describe
inspection and validation without changing catalog/definition routing.
2. Introduce a private per-invocation validation input holding `ValidationRequest`,
the format, captured selector values, and prepared application input. Build
a normal typed `CommandSpec` named `validate`, with one required prompt ID.
Use `CaptureFlags` and `prepareSuiteInput` for application selectors/checks.
3. Bind the following fields; defaults and repetition follow the accepted
roadmap and current requirements bindings:
| CLI input | Destination |
| --- | --- |
| Prompt ID operand | `Request.PromptID` |
| `--config`, `--overlay` | Presence-aware `Selection.Config`, `Selection.Overlay` |
| `--version` | `Request.PromptVersion` |
| `--profile` | `Request.ProfileID` |
| `--session-id` | `Request.SessionID` |
| Repeatable `--var KEY=VALUE` | `Request.Vars` |
| Repeatable `--input NAME=PATH` | `Request.Inputs[NAME] = promptkit.File(PATH)` |
| `--format` | Private format, default `yaml` |
| Suite selectors | Captured bindings applied to initialized `I` |
4. Parse file assignments at the first `=` and preserve the name and path
exactly. Parse every occurrence, then let the last duplicate name win.
Missing separators are usage errors; empty names/paths go to native
validation. Do not read files, normalize paths, expand them, or rebase them
while binding. Reuse variable-assignment mechanics where this is simple,
keeping their existing behavior unchanged.
5. Reject unsupported formats before application input preparation. Use the
existing YAML-only format check where appropriate. Missing operands,
malformed bindings, and unknown flags retain existing usage handling.
6. In the handler, check cancellation, initialize state using the existing
zero-value convention, check cancellation again, and invoke the captured
validation callback. On error, return `definitionFailure` with diagnostics.
After successful validation, check cancellation before encoding/delivery.
A successful empty acquisition value is valid; there is no inspection value
to require or dereference.
7. Encode a private YAML DTO with explicit `prompt_id` and `valid` fields. Use
the command's requested ID and `valid: true`, with no resolved metadata or
request dump. Reuse `encodePromptYAMLBytes` and `writePromptReport`, returning
diagnostics for core delivery. Do not change established report/diagnostic
delivery order, short-write handling, or failure bookkeeping.
**Validation:** Add a basic assembled command test with a capturing custom
callback and semantic YAML decoding. Verify the correct seed reaches the
callback, and both show and validate can be installed in one requirements group.
Run focused tests and the ckprompt suite.
**Complete when:** the new command invokes the selected provider and produces
the agreed YAML confirmation using existing core mechanics.
## Stage 5: Cover command grammar, capability availability, and input lifetimes
**Primary files:** a focused `integrations/ckprompt/validation_command_test.go`
and existing requirements command tests where an extension is simpler.
1. Exercise validation-only, discovery-only, both, and absent suites. Check
installed help/availability and the existing namespace collision/root-only
behavior for the new validation-only declaration, reusing core coverage
rather than retesting the registry in full.
2. Verify construction, installation, help, and version leave input/state
factories, builders, and loaders uninvoked. Capture a command, then mutate
source callbacks/selector metadata to confirm the established eager-capture
behavior includes the new callback. Callback-owned referenced objects stay
caller-owned.
3. Capture standard request fields and typed selectors in a custom provider.
Cover explicit empty config/overlay presence, empty scalar native semantics,
singleton repetition, and repeated variable/file assignments. Include paths
containing `=`, empty names/values, whitespace/commas, duplicate names, and
a malformed earlier occurrence followed by a valid one. Inspect native file
references without requiring nonexistent paths to pass actual validation.
4. Cover missing/extra operands, unsupported format, unknown flags, malformed
`--var`/`--input`, and required suite selectors. Confirm applicable syntax
failures stop before application preparation/acquisition. Input-check
failures stop state creation and provider work using current classifications.
5. Run multiple invocations on one App and verify parsed Vars/Inputs, selection,
and selector presence do not leak into later calls. Verify one reached input
preparation/check and state creation per selected validation invocation,
and omitted factories supply zero values including nil pointers. Do not
promise isolation for aliases created by application factories.
**Validation:** Run the focused command tests and ckprompt suite. Decode each
YAML output into a fresh value; avoid output-prefix or key-order assertions.
**Complete when:** the new capability obeys the selected standard grammar,
availability, capture, and invocation-lifetime contracts.
## Stage 6: Cover native command outcomes and report delivery
**Primary files:** validation command tests and small real-engine fixtures.
1. Exercise `WithRequestValidation` through `Command` and `App.Execute`, with
real native success, missing required binding, and one selection failure.
Include a useful nil-builder request built entirely from standard flags.
Assert no model call and preserve native error identity; compare categories,
not exact human-readable error wording.
2. Cover custom-provider diagnostics on success and error. Builder, loader,
and native failures are operational outcomes/status 1. CLI usage failures
are status 2. Retain warning diagnostics without turning warning severity
alone into failure. Reuse existing input-check handling.
3. Verify cancellation before acquisition and after a successful callback
prevents later work/report output. Preserve callback errors and native error
categories rather than overwriting them with a general context error.
4. Reuse existing writer doubles to cover report errors, short writes, and
diagnostic writer errors. Preserve `ReportIncomplete`, `DeliveryError`,
diagnostic-delivery state, and supported cause identity as established by
the shared delivery contract. Do not add rollback or blocking-writer
interruption guarantees.
5. Decode the success report and assert only the agreed confirmation fields
are emitted, without values/artifact contents/rendered messages. Keep the
existing partial-discovery-success tests as regression coverage; do not
duplicate their whole matrix for validation.
**Validation:** Run focused tests and the complete ckprompt suite, including
its maintained composed consumer tests.
**Complete when:** assembled validation follows native semantics and existing
outcome/report delivery contracts under representative successes and failures.
## Stage 7: Extend the maintained composed consumer
**Primary files:** `integrations/ckprompt/examples/composed/main.go`, its tests,
README, and synthetic fixtures under its existing `fixtures/` directory.
1. Compose `WithRequestValidation` into the existing `FromEngine` suite, keeping
its factories, campaign selector, catalogs, definitions, discovery, and
separately installed config/session commands.
2. Add a typed builder that starts from the seed and uses the existing
application configuration loader and invocation cache. Load the selected
campaign through that shared state, then default a missing `topic` variable
to a synthetic campaign-specific topic and missing `mode` to `""`. Use map
membership, preserving supplied empty values. Keep seed identity, scalar
context, and operator artifact bindings. The engine loader can then reuse
that application load through the same cache. This is example behavior,
not a CommandKit cache/defaults policy.
3. Keep the existing synthetic prompt template and required `topic` artifact.
Add secret-free `fixtures/topic.txt` and `fixtures/notes.txt`. Do not supply
the required topic artifact implicitly in the builder: its absence is the
useful validation failure. The existing profile uses an inert endpoint and
no real credentials; validation remains offline.
4. Add README commands, from the composed directory:
```sh
go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt
go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt --var mode=full --input notes=fixtures/notes.txt --format yaml
go run . promptkit requirements validate briefing --campaign spring
```
The first two succeed; the last intentionally fails for the missing required
topic binding. Preserve the existing show examples, explaining their distinct
discovery result. Describe success as passing preparation at that invocation.
5. Add focused composed-consumer evidence for successful file-backed validation,
missing-topic failure, and the builder preserving explicitly empty variables
while supplying defaults only for absent keys. Test builder behavior directly
where needed to observe precedence without inventing a library enforcement
rule. Preserve current campaign and discovery tests.
**Validation:** Run composed tests and the documented successful/failing
commands. Record the expected classifications and semantic reports rather than
making the intentionally failing example a failed completion check.
**Complete when:** the maintained consumer demonstrates the accepted feature,
typed selection, shared state, and application-owned defaults without execution.
## Stage 8: Update canonical documentation for implemented behavior
**Primary files:** GoDoc, `docs/cli.md`, `docs/consumers/guide.md`, README,
`docs/development.md`, and `docs/internal/overview.md` as affected.
1. Complete GoDoc for `ValidationRequest`, `Suite.ValidateRequest`,
`WithRequestValidation`, and command factory behavior. Update the existing
`InspectRequirements` comment so omission refers to `requirements show`,
rather than implying the entire requirements group is absent. State optionality,
nil-builder forwarding, retained wiring issues, builder-before-loader order,
callback ownership, and native error/diagnostic treatment.
2. Extend the CLI requirements section with separate show/validate descriptions,
exact flags and repetition rules, file-binding semantics, YAML confirmation,
and status meanings. Scope existing discovery-only statements appropriately:
concrete validation loads resources and renders through native preparation;
it returns encountered errors and no snapshot/model output.
3. Update consumer guidance from five independently optional callbacks to six,
distinguishing `FromEngine`'s five supplied callbacks from explicit validation
composition. Include minimal full-suite and validation-only/nil-builder
wiring, and link to the maintained consumer for the complete builder example.
Recommend shared validation/execution request construction without prescribing
downstream architecture.
4. Update module summaries and implemented internal inventory to include concrete
validation. Audit statements about no rendering/resource loading/credential
resolution so metadata-inspection descriptions do not accidentally cover the
new capability. Keep native semantics linked to adopted upstream contracts.
5. Leave accepted ADRs, policy boundaries, historical releases, and unrelated
deferred proposals intact. No new ADR or policy revision is required by this
accepted feature. Current API/CLI documentation becomes the canonical owner
when implementation lands.
**Validation:** Review examples against implemented signatures and commands;
check local links/anchors and Markdown whitespace. Reuse Stage 7's unchanged
example evidence. Check prose for accidental stronger guarantees or consumer
mandates.
**Complete when:** current documentation accurately describes the implemented
capability under its assigned owners, with no stale five-callback or
metadata-only claims covering request validation.
## Stage 9: Validate the complete end state and finish
1. Format changed Go files and inspect the final diff for scope, generated junk,
local replacements, and unrelated changes. Keep `deferred.md` intact unless
a statement actually becomes stale through this feature.
2. Run the complete workspace suites from all three module directories:
```sh
GOTOOLCHAIN=local go test -count=1 ./...
(cd integrations/ckconfig && GOTOOLCHAIN=local go test -count=1 ./...)
(cd integrations/ckprompt && GOTOOLCHAIN=local go test -count=1 ./...)
```
Root `./...` does not traverse nested modules. These ordinary suites do not
authorize a release publication exercise or live model service tests.
3. Separately verify each module with `GOWORK=off`, `GOTOOLCHAIN=local`, and
`-mod=readonly`. For unpublished CommandKit prerequisites, create temporary
absolute-path copies of each affected module's `go.mod` and `go.sum`, keeping
paired names such as `ckprompt.mod` and `ckprompt.sum`. Run from that module's
real directory with `-modfile=/absolute/temp/path/ckprompt.mod`.
- Core: use its actual manifest; no optional-library replacements.
- ckconfig: replace only its required core version with the current root.
- ckprompt: replace only its required core and ckconfig versions with their
current local directories.
- Use version-specific replacements for the versions in the manifests.
PromptKit and ConfigKit remain real released dependencies. Temporary
replacements are validation scaffolding, never committed metadata.
- Prepare any needed cache/checksums in the temporary copies, then record
their hashes before running readonly tests. Verify those files are unchanged
afterward. A workspace-only pass does not establish isolated resolution.
In each module, use `go test -mod=readonly -count=1 ./...` with the applicable
`-modfile` and environment above. Inspect `go list -m all` and
`go list -deps -test ./...` with the same isolated manifest: core has no optional
integration dependency; ckconfig has no PromptKit/ckprompt dependency;
ckprompt selects released PromptKit v0.15.0. Do not change checked-in
prerequisite versions merely to make isolated checks easier. Remove temporary
module files after retaining the relevant validation evidence.
4. Review the Stage 7 commands and canonical docs together. Check that the
nil-builder path and typed builder are both covered, show still permits
partial discovery, errors preserve native categories, and only the agreed
YAML confirmation is emitted on validation success.
5. Run relevant prose whitespace/link checks, including staged documentation.
Confirm the integration attribution matches the selected released dependency.
No new coverage threshold, benchmark, platform matrix, release gate, or
threat scope is introduced by this plan.
6. Mark completed stages with actual evidence, update the feature roadmap's
implementation status, and record any genuine remaining limitation accurately.
Commit outstanding sprint changes using a concise plain-English message,
inspect status afterward, and report the final commit(s) and validation.
Retain planning documents until their cleanup is requested.
**Complete when:** every accepted roadmap capability is implemented, all
applicable evidence above passes, public documentation and module metadata are
aligned, and the workspace is clean. No unresolved product or API decisions
were identified in preparing this plan.

2
docs/roadmap/promptkit-validation.md
View File

@@ -3,7 +3,7 @@
Status: accepted adoption and feature roadmap; implementation has not started.
Command placement, adapter composition, request construction, artifact syntax,
and initial presentation are settled. This document defines the target and
required work; a later implementation plan will sequence the work.
required work; the [implementation plan](implementation.md) sequences the work.
## Purpose