6.1 KiB
Go-Based Testing For Maintained Examples
Status: complete. Example assertions now run in the ordinary Go suite; the custom release check and its Python runner have been removed.
Purpose
Move the maintained consumer-example checks into Promptkit's ordinary Go test suite. Developers and ReleaseKit should exercise these workflows through the same testing entry point, without a separate Python example runner or custom release-validation command.
Keep the examples useful as readable, copyable consumer guidance. The objective is simpler ownership and execution of tests, not a language-only translation or a particular reduction in line count. This is a tooling change, not a new public library API or a new release gate.
The companion implementation plan defines the staged work for reaching this target.
Previous Baseline
The preparation, execution, and observation examples are Go programs. Their deterministic clients and transport keep them independent of live providers. Preparation and execution use repository-relative configuration paths; observation embeds its assets and already has a workflow helper returning a summary and error.
The former scripts/check-examples.py launched all three programs and checked
their JSON summaries. Ordinary go test ./... then had no example tests, and
the ReleaseKit profile selected the Python script as a
custom check after shared validation.
The separate release-wiring test is also Python, but is a development-only diagnostic, not a release gate. Its disposable fixture previously assumed the example script was a configured product check.
Target State
Ordinary Go tests alongside the examples exercise the actual example workflows.
They are discovered by go test ./... and can be selected with
go test ./examples/go-library/.... ReleaseKit's existing standard suite runs
them with its selected race and vet settings; no second example-validation pass
is needed.
Preserve the useful coverage currently supplied by the Python runner:
- Preparation: collection composition and overrides, scoped selection, inspection/preparation, and the resulting identities and inventories.
- Execution: injected-client execution, expected output and structural validity, model selection, and usage reporting.
- Observation: bounded transient retries, attempt metadata, incomplete usage reporting, and application-owned interpretation of proposal/assessment results.
Assertions should remain tied to the synthetic examples and meaningful results. Their output formatting and fixture values do not become new public API guarantees. Preserve relevant output coverage without freezing complete JSON documents or introducing broad golden snapshots.
Example And Test Structure
Prefer small, unexported workflow helpers that return errors or results and,
where useful, accept an output writer. Keep main responsible for process-level
output and exit handling. Reuse the observation example's existing separation
where it already meets the need; uniform helper signatures are not an objective.
Tests should normally call the workflow directly, using local output buffers
and the existing deterministic collaborators. Handle resource paths explicitly
so package-directory test execution and the documented repository-root
go run commands both work. A small resource-base or filesystem input is
preferable to changing process-wide working directories or redirecting global
standard output merely to reach the workflow.
Use a Go subprocess test only where launching the executable protects a distinct risk or avoids making an example materially harder to understand. Do not add one for every example automatically. Subprocesses need their own consideration of toolchain, offline execution, and instrumentation; parent race or coverage settings do not automatically instrument a separately launched program.
Keep tests deterministic, offline, and independent of real credentials under the testing policy. Avoid a new public package, shared example framework, or expanded failure-case inventory solely for this migration. Examples remain consumer programs rather than release artifacts or library implementation dependencies, consistent with the architecture policy.
Integration And Documentation Work
The Go tests cover the retained behavior; the former scripts/check-examples.py
and its custom command and environment overlay have been removed. The existing
built-in validation selections and source/hosted publication workflow remain,
without a ReleaseKit dependency upgrade.
The development-only release-wiring fixture no longer expects the removed script or product-check marker. It retains checks for the actual launcher and profile without manufacturing an unused custom hook. The diagnostic remains Python. Removing Python from every repository tool is outside this scope.
The development guide, testing policy, release policy, and release procedure describe Go test coverage as part of the standard suite and retain Python only for the optional release-wiring test. The existing consumer quickstart commands remain available.
Verification
Verify the example packages through ordinary Go tests and the selected
race-enabled suite and vet. Confirm the documented go run entry points still
work from the repository root, and exercise the adapted release-wiring test in
disposable repositories. These are migration checks, not new recurring release
steps. Check that a meaningful example assertion failure fails the Go suite
rather than relying only on successful compilation.
Completion means useful example behavior remains covered, normal test execution includes that coverage, and the obsolete runner, profile hook, and documentation references are removed. No performance thresholds, coverage percentage targets, new platform commitments, hosted CI requirements, or changes to Promptkit's consumer contracts are introduced by this work.