promptkit/docs/roadmap/implementation.md

Go-Based Example Testing Implementation Plan

Implement example-testing.md as one end-to-end sprint. Each sequential stage below is a single prompt-sized assignment for a Luna coding agent. The feature roadmap owns intent and scope; this document owns the implementation steps and local design choices.

Baseline And Working Rules

The reviewed baseline is 79dfd97afa2acc3cbf1ba6936b3a11b116702cf4: Go 1.27.0, module go.maximumdirect.net/promptkit, ConfigKit v1.0.0, ReleaseKit v0.8.0, and three maintained programs under examples/go-library/. There are no Go tests alongside those programs. scripts/check-examples.py executes them and checks JSON results through the sole custom ReleaseKit hook. scripts/test-release.py is a separate development-only diagnostic with a synthetic replacement for that hook.

This plan replaces the completed ConfigKit migration plan. Its historical instructions remain available with git show 79dfd97:docs/roadmap/implementation.md; they are not assignments for this sprint.

Read docs/development.md and the applicable agent, architecture, testing, documentation, and release policies before work. Preserve unrelated changes. Use the module-compatible local Go toolchain and prepare caches before offline validation. No dependency, Go-minimum, consumer API, or source/hosted-release workflow change is needed.

Intermediate stages may be incomplete or temporarily inconsistent: there is no requirement to introduce compatibility scaffolding, compile every intermediate state, or stop for human review. Finish all stages before claiming completion. Run focused checks when useful; if a check depends on later work, record that dependency rather than claiming it passed. Do not publish or contact providers. Follow the agent policy for focused local commits.

Implementation Choices

• Put tests in main_test.go with package main beside each example. Test actual workflow and encoding paths directly, not reconstructed engine setups.
• For preparation and execution, extract run(ctx context.Context, exampleDir string, out io.Writer) error. exampleDir is that example's resource directory, not an application-wide configuration root. Use filepath.Join to locate its existing files. Each main passes its existing repository-relative example directory and os.Stdout; tests pass "." and a buffer from the package directory.
• Observation already has run() (summary, error) and embedded assets. Keep that workflow and add a small execute(out io.Writer) error which calls it and encodes its result. Both main and its test use execute.
• Keep stderr reporting and nonzero exit handling in the thin main functions. Workflow helpers return errors instead of exiting. Preserve current successful summaries and existing example-side semantic checks.
• Use one representative happy-path workflow test per example to carry the existing useful assertions. Decode JSON independently of production summary tags: test-local structs or maps are sufficient. Check presence where a missing field could masquerade as an expected false, zero, or null value. Do not reject harmless extra fields, require exact JSON formatting, or add whole-document golden snapshots.
• Use buffers and explicit resource directories rather than redirecting global stdout or changing the process working directory. These are local design choices for these examples, not new library contracts.
• No subprocess example test is needed for this baseline: the small extractions cover the workflow and output without complicating the examples. Verify the actual go run commands once during integration instead of adding redundant executable tests to every default run.

Stage 1: Test The Execution Example In Go

Scope: examples/go-library/run/main.go and a new main_test.go there.

1. Extract its current engine construction, injected deterministic client, request, and summary encoding into run(ctx, exampleDir, out). Resolve prompt.yaml relative to exampleDir. Pass the supplied context to engine.Run instead of creating another background context there.
2. Return construction, execution, and encoding errors. Keep the existing indented JSON output. Make main call this helper with a background context, "examples/go-library/run", and stdout; retain stderr/exit handling at that boundary. Remove obsolete error-exit helper code if no longer used.
3. Add TestRunExample calling the helper with t.Context(), ".", and a bytes.Buffer. Decode and assert the current output:
   • output: Ada finished the migration review.;
   • is_valid: true;
   • validation_status: passed;
   • model: offline-model;
   • total_tokens: 18.
4. Keep this local to the example: no new shared test framework or public API.

Validation: format changed Go files and run go test ./examples/go-library/run. Confirm the test executes the workflow, not just the summary type. These expected values describe this fixture only.

Completion: extracted the execution workflow and JSON encoding into a helper with explicit context, resource directory, and writer. The focused Go test executes the workflow and checks its summary; it passed on Go 1.27.0.

Stage 2: Test Preparation With Explicit Resource Context

Scope: examples/go-library/prepare/main.go and its new main_test.go.

1. Extract the current composition, inspection, validation, preparation, and summary encoding into the chosen run(ctx, exampleDir, out) helper. Replace the three hard-coded file prefixes with paths under exampleDir; retain the existing collections, source ordering, imports, and later override.
2. Preserve the inline checks for the discovered required note input, the override explanation, and the effective timeout. Return their errors. Keep those checks readable in the example instead of duplicating their complete machinery in the test. Use the supplied context for engine calls.
3. Keep main callable through the existing repository-root command by passing "examples/go-library/prepare" and stdout. Retain current indented encoding and error presentation while removing exits from the workflow helper.
4. Add TestPrepareExample using the helper, package-local resources, and a buffer. Assert these retained JSON results:
   • prompt_id: support/prepare; selected_profile: support/fast;
   • model: offline-model; message_count greater than zero;
   • profile_timeout_seconds: 30;
   • namespaces: review, then support;
   • collections: review with an explicit null release_version, followed by support with release_version equal to 1.4.0;
   • scoped_prompts: support/prepare; review_prompts: review/check. Preserve the distinction between an omitted member and the explicit null currently checked for the review collection.

Validation: format and run go test ./examples/go-library/prepare. Ensure imports and relative content still resolve from their contributing sources; do not rewrite configuration fixtures merely to bypass path handling.

Completion: the preparation workflow accepts an explicit context, example directory, and writer while preserving its inline checks and source contexts. Its Go test verifies the retained summary, including explicit null collection metadata, and passed on Go 1.27.0.

Stage 3: Test The Observation Example And Its Output

Scope: examples/go-library/observe/main.go and its new main_test.go.

1. Retain its existing run, embedded files, scripted HTTP transport, event recorder, and application-owned assessment logic.
2. Extract only the workflow/encoding coordination into execute(out) as specified above. It returns workflow or encoding errors. Make main call it with stdout and retain the current stderr/exit behavior. Do not redesign the transport or add a new client abstraction.
3. Add TestObserveExample calling execute with a buffer. Decode and assert:
   • proposal_attempts: 2; proposal_transient_retries: 1;
   • proposal_usage_complete: false;
   • assessment_attempts: 1; assessment_transient_retries: 0;
   • assessment_outcome: accepted. Missing false/zero fields must not silently pass. Run the workflow once per test rather than separately recomputing typed and encoded results.

Validation: format and run go test ./examples/go-library/observe, followed by go test ./examples/go-library/.... Existing deterministic collaborators remain the network boundary; no live provider or credential is needed.

Stage 4: Remove The Custom Release Hook And Adapt Its Diagnostic

Scope: scripts/check-examples.py, releasekit.json, and scripts/test-release.py.

1. Delete the Python example runner after the Go tests carry its useful coverage. Remove the profile's checks property and its example-specific environment overlay. Retain every existing built-in validator, delivery setting, and token-file protection. Do not add a replacement custom command.
2. In the release-wiring fixture, remove the generated example-check stub, FIXTURE_CHECK_MARKER, FIXTURE_CHECK_FAILURE, related environment setup, and assertions about invocation of that removed hook. Update its docstring to describe launcher/profile wiring, not product-check dispatch.
3. Retain bootstrap/version/build-info checks, native usage status 2, successful publication of the intended local fixture candidate without delivery credentials, selected race/local-Go verification, token-path selection and ignore checks, and module/worktree preservation.
4. Preserve a small status-1 wrapper check without manufacturing a product hook: create only the successful version's note in the fixture, then invoke the publication wrapper with FAILED_VERSION, whose note is absent. Expect status 1 and retain the remote-tag absence check. This exercises forwarding a native candidate failure, not a new publication algorithm.
5. Keep this diagnostic development-only and in Python. Do not add it to the default Go suite or the release profile. The native launchers are unchanged.

Validation: run python3 scripts/test-release.py using the selected Go on PATH and prepared caches. Its remotes must remain disposable local bare repositories. Check the profile diff to confirm only the obsolete custom hook was removed. Do not test by publishing a real tag or hosted page.

Stage 5: Align Documentation And Verify The Complete Sprint

Scope: the development guide, local testing/release policies, release runbook, affected example guidance, and integration verification.

1. Update docs/development.md to use normal Go testing for example validation. Offer go test ./examples/go-library/... for targeted development without making it a separate required release pass. Keep the Python release-wiring command documented as a focused optional diagnostic.

2. Update the Promptkit-specific sections of docs/policy/testing.md and docs/policy/release.md: example assertions now run inside the existing Go suite, not after it as a custom check. Keep the examples' distinct coverage ownership, all other selected checks, and the publication completion rules.

3. Remove Python from normal release prerequisites in docs/release.md. Preserve Go selection, credentials, source publication, hosted delivery, and recovery. Update any other current guidance pointing to the deleted runner. Historical release notes remain unchanged.

4. Update the feature roadmap's completion/baseline text when recording the implemented result. Remove its now-dead Markdown link to the deleted Python file, using a plain historical path if still useful. Avoid leaving a broken link in a completed coordination document. Do not retire unrelated roadmaps.

5. Format changed Go files, review the full diff, and run the integrated checks below with the same compatible local Go toolchain:

   go test -race -vet=off ./...
   go vet ./...
   go run ./examples/go-library/prepare
   go run ./examples/go-library/run
   go run ./examples/go-library/observe
   python3 scripts/test-release.py
   git diff --check
   

   Run the three programs from the repository root. Confirm their outputs retain the meanings checked by the new tests; JSON spacing is not a new contract. Reuse successful focused checks for unchanged content rather than repeating them solely to finish a stage. The separate full vet invocation above pairs with -vet=off to avoid duplicate vet work.

6. As a one-time proof, temporarily make one meaningful expected value wrong in an example test, run its package test, and confirm failure. Restore the value and rerun the affected package before final review. Do not commit that probe or add a permanent self-test of the testing framework.

7. Search current code/configuration/guidance for obsolete runner and hook references, verify local documentation links, and confirm go.mod, go.sum, public library code, and the native release launchers were not changed by this tooling migration. Commit the focused completed work under agent policy.

Completion requires all three workflows to have real Go test coverage, successful ordinary suite discovery, preserved useful assertions, removal of the runner and custom hook, and accurate current documentation. No performance, coverage-percentage, platform, or release-service gate is added. There are no unresolved product decisions or upstream publication dependencies for this sprint.