promptkit/docs/roadmap/implementation.md

# Go-Based Example Testing Implementation Plan

Implement [example-testing.md](example-testing.md) as one end-to-end sprint.
Each sequential stage below is a single prompt-sized assignment for a Luna
coding agent. The feature roadmap owns intent and scope; this document owns
the implementation steps and local design choices.

## Baseline And Working Rules

The reviewed baseline is `79dfd97afa2acc3cbf1ba6936b3a11b116702cf4`:
Go 1.27.0, module `go.maximumdirect.net/promptkit`, ConfigKit v1.0.0,
ReleaseKit v0.8.0, and three maintained programs under `examples/go-library/`.
There are no Go tests alongside those programs. `scripts/check-examples.py`
executes them and checks JSON results through the sole custom ReleaseKit hook.
`scripts/test-release.py` is a separate development-only diagnostic with a
synthetic replacement for that hook.

This plan replaces the completed ConfigKit migration plan. Its historical
instructions remain available with
`git show 79dfd97:docs/roadmap/implementation.md`; they are not assignments for
this sprint.

Read `docs/development.md` and the applicable agent, architecture, testing,
documentation, and release policies before work. Preserve unrelated changes.
Use the module-compatible local Go toolchain and prepare caches before offline
validation. No dependency, Go-minimum, consumer API, or source/hosted-release
workflow change is needed.

Intermediate stages may be incomplete or temporarily inconsistent: there is no
requirement to introduce compatibility scaffolding, compile every intermediate
state, or stop for human review. Finish all stages before claiming completion.
Run focused checks when useful; if a check depends on later work, record that
dependency rather than claiming it passed. Do not publish or contact providers.
Follow the agent policy for focused local commits.

## Implementation Choices

- Put tests in `main_test.go` with `package main` beside each example. Test
  actual workflow and encoding paths directly, not reconstructed engine setups.
- For preparation and execution, extract
  `run(ctx context.Context, exampleDir string, out io.Writer) error`.
  `exampleDir` is that example's resource directory, not an application-wide
  configuration root. Use `filepath.Join` to locate its existing files.
  Each `main` passes its existing repository-relative example directory and
  `os.Stdout`; tests pass `"."` and a buffer from the package directory.
- Observation already has `run() (summary, error)` and embedded assets. Keep
  that workflow and add a small `execute(out io.Writer) error` which calls it
  and encodes its result. Both `main` and its test use `execute`.
- Keep stderr reporting and nonzero exit handling in the thin `main` functions.
  Workflow helpers return errors instead of exiting. Preserve current successful
  summaries and existing example-side semantic checks.
- Use one representative happy-path workflow test per example to carry the
  existing useful assertions. Decode JSON independently of production summary
  tags: test-local structs or maps are sufficient. Check presence where a
  missing field could masquerade as an expected false, zero, or null value.
  Do not reject harmless extra fields, require exact JSON formatting, or add
  whole-document golden snapshots.
- Use buffers and explicit resource directories rather than redirecting global
  stdout or changing the process working directory. These are local design
  choices for these examples, not new library contracts.
- No subprocess example test is needed for this baseline: the small extractions
  cover the workflow and output without complicating the examples. Verify the
  actual `go run` commands once during integration instead of adding redundant
  executable tests to every default run.

## Stage 1: Test The Execution Example In Go

Scope: `examples/go-library/run/main.go` and a new `main_test.go` there.

1. Extract its current engine construction, injected deterministic client,
   request, and summary encoding into `run(ctx, exampleDir, out)`. Resolve
   `prompt.yaml` relative to `exampleDir`. Pass the supplied context to
   `engine.Run` instead of creating another background context there.
2. Return construction, execution, and encoding errors. Keep the existing
   indented JSON output. Make `main` call this helper with a background context,
   `"examples/go-library/run"`, and stdout; retain stderr/exit handling at that
   boundary. Remove obsolete error-exit helper code if no longer used.
3. Add `TestRunExample` calling the helper with `t.Context()`, `"."`, and a
   `bytes.Buffer`. Decode and assert the current output:
   - `output`: `Ada finished the migration review.`;
   - `is_valid`: true;
   - `validation_status`: `passed`;
   - `model`: `offline-model`;
   - `total_tokens`: 18.
4. Keep this local to the example: no new shared test framework or public API.

Validation: format changed Go files and run
`go test ./examples/go-library/run`. Confirm the test executes the workflow,
not just the summary type. These expected values describe this fixture only.

Completion: extracted the execution workflow and JSON encoding into a helper
with explicit context, resource directory, and writer. The focused Go test
executes the workflow and checks its summary; it passed on Go 1.27.0.

## Stage 2: Test Preparation With Explicit Resource Context

Scope: `examples/go-library/prepare/main.go` and its new `main_test.go`.

1. Extract the current composition, inspection, validation, preparation, and
   summary encoding into the chosen `run(ctx, exampleDir, out)` helper. Replace
   the three hard-coded file prefixes with paths under `exampleDir`; retain
   the existing collections, source ordering, imports, and later override.
2. Preserve the inline checks for the discovered required `note` input, the
   override explanation, and the effective timeout. Return their errors.
   Keep those checks readable in the example instead of duplicating their
   complete machinery in the test. Use the supplied context for engine calls.
3. Keep `main` callable through the existing repository-root command by passing
   `"examples/go-library/prepare"` and stdout. Retain current indented encoding
   and error presentation while removing exits from the workflow helper.
4. Add `TestPrepareExample` using the helper, package-local resources, and a
   buffer. Assert these retained JSON results:
   - `prompt_id`: `support/prepare`; `selected_profile`: `support/fast`;
   - `model`: `offline-model`; `message_count` greater than zero;
   - `profile_timeout_seconds`: 30;
   - `namespaces`: `review`, then `support`;
   - `collections`: review with an explicit null `release_version`, followed
     by support with `release_version` equal to `1.4.0`;
   - `scoped_prompts`: `support/prepare`; `review_prompts`: `review/check`.
   Preserve the distinction between an omitted member and the explicit null
   currently checked for the review collection.

Validation: format and run `go test ./examples/go-library/prepare`. Ensure
imports and relative content still resolve from their contributing sources;
do not rewrite configuration fixtures merely to bypass path handling.

Completion: the preparation workflow accepts an explicit context, example
directory, and writer while preserving its inline checks and source contexts.
Its Go test verifies the retained summary, including explicit null collection
metadata, and passed on Go 1.27.0.

## Stage 3: Test The Observation Example And Its Output

Scope: `examples/go-library/observe/main.go` and its new `main_test.go`.

1. Retain its existing `run`, embedded files, scripted HTTP transport, event
   recorder, and application-owned assessment logic.
2. Extract only the workflow/encoding coordination into `execute(out)` as
   specified above. It returns workflow or encoding errors. Make `main` call
   it with stdout and retain the current stderr/exit behavior. Do not redesign
   the transport or add a new client abstraction.
3. Add `TestObserveExample` calling `execute` with a buffer. Decode and assert:
   - `proposal_attempts`: 2; `proposal_transient_retries`: 1;
   - `proposal_usage_complete`: false;
   - `assessment_attempts`: 1; `assessment_transient_retries`: 0;
   - `assessment_outcome`: `accepted`.
   Missing false/zero fields must not silently pass. Run the workflow once per
   test rather than separately recomputing typed and encoded results.

Validation: format and run `go test ./examples/go-library/observe`, followed
by `go test ./examples/go-library/...`. Existing deterministic collaborators
remain the network boundary; no live provider or credential is needed.

## Stage 4: Remove The Custom Release Hook And Adapt Its Diagnostic

Scope: `scripts/check-examples.py`, `releasekit.json`, and
`scripts/test-release.py`.

1. Delete the Python example runner after the Go tests carry its useful
   coverage. Remove the profile's `checks` property and its example-specific
   environment overlay. Retain every existing built-in validator, delivery
   setting, and token-file protection. Do not add a replacement custom command.
2. In the release-wiring fixture, remove the generated example-check stub,
   `FIXTURE_CHECK_MARKER`, `FIXTURE_CHECK_FAILURE`, related environment setup,
   and assertions about invocation of that removed hook. Update its docstring
   to describe launcher/profile wiring, not product-check dispatch.
3. Retain bootstrap/version/build-info checks, native usage status 2, successful
   publication of the intended local fixture candidate without delivery
   credentials, selected race/local-Go verification, token-path selection and
   ignore checks, and module/worktree preservation.
4. Preserve a small status-1 wrapper check without manufacturing a product
   hook: create only the successful version's note in the fixture, then invoke
   the publication wrapper with `FAILED_VERSION`, whose note is absent.
   Expect status 1 and retain the remote-tag absence check. This exercises
   forwarding a native candidate failure, not a new publication algorithm.
5. Keep this diagnostic development-only and in Python. Do not add it to the
   default Go suite or the release profile. The native launchers are unchanged.

Validation: run `python3 scripts/test-release.py` using the selected Go on
`PATH` and prepared caches. Its remotes must remain disposable local bare
repositories. Check the profile diff to confirm only the obsolete custom hook
was removed. Do not test by publishing a real tag or hosted page.

## Stage 5: Align Documentation And Verify The Complete Sprint

Scope: the development guide, local testing/release policies, release runbook,
affected example guidance, and integration verification.

1. Update `docs/development.md` to use normal Go testing for example validation.
   Offer `go test ./examples/go-library/...` for targeted development without
   making it a separate required release pass. Keep the Python release-wiring
   command documented as a focused optional diagnostic.
2. Update the Promptkit-specific sections of `docs/policy/testing.md` and
   `docs/policy/release.md`: example assertions now run inside the existing Go
   suite, not after it as a custom check. Keep the examples' distinct coverage
   ownership, all other selected checks, and the publication completion rules.
3. Remove Python from normal release prerequisites in `docs/release.md`.
   Preserve Go selection, credentials, source publication, hosted delivery, and
   recovery. Update any other current guidance pointing to the deleted runner.
   Historical release notes remain unchanged.
4. Update the feature roadmap's completion/baseline text when recording the
   implemented result. Remove its now-dead Markdown link to the deleted Python
   file, using a plain historical path if still useful. Avoid leaving a broken
   link in a completed coordination document. Do not retire unrelated roadmaps.
5. Format changed Go files, review the full diff, and run the integrated checks
   below with the same compatible local Go toolchain:

   ```sh
   go test -race -vet=off ./...
   go vet ./...
   go run ./examples/go-library/prepare
   go run ./examples/go-library/run
   go run ./examples/go-library/observe
   python3 scripts/test-release.py
   git diff --check
   ```

   Run the three programs from the repository root. Confirm their outputs
   retain the meanings checked by the new tests; JSON spacing is not a new
   contract. Reuse successful focused checks for unchanged content rather than
   repeating them solely to finish a stage. The separate full vet invocation
   above pairs with `-vet=off` to avoid duplicate vet work.
6. As a one-time proof, temporarily make one meaningful expected value wrong in
   an example test, run its package test, and confirm failure. Restore the value
   and rerun the affected package before final review. Do not commit that probe
   or add a permanent self-test of the testing framework.
7. Search current code/configuration/guidance for obsolete runner and hook
   references, verify local documentation links, and confirm `go.mod`, `go.sum`,
   public library code, and the native release launchers were not changed by
   this tooling migration. Commit the focused completed work under agent policy.

Completion requires all three workflows to have real Go test coverage,
successful ordinary suite discovery, preserved useful assertions, removal of
the runner and custom hook, and accurate current documentation. No performance,
coverage-percentage, platform, or release-service gate is added. There are no
unresolved product decisions or upstream publication dependencies for this sprint.