120 lines
6.3 KiB
Markdown
120 lines
6.3 KiB
Markdown
# Go-Based Testing For Maintained Examples
|
|
|
|
## Purpose
|
|
|
|
Move the maintained consumer-example checks into Promptkit's ordinary Go test
|
|
suite. Developers and ReleaseKit should exercise these workflows through the
|
|
same testing entry point, without a separate Python example runner or custom
|
|
release-validation command.
|
|
|
|
Keep the examples useful as readable, copyable consumer guidance. The objective
|
|
is simpler ownership and execution of tests, not a language-only translation or
|
|
a particular reduction in line count. This is a tooling change, not a new public
|
|
library API or a new release gate.
|
|
|
|
## Current Baseline
|
|
|
|
The [preparation](../../examples/go-library/prepare/main.go),
|
|
[execution](../../examples/go-library/run/main.go), and
|
|
[observation](../../examples/go-library/observe/main.go) examples are Go programs.
|
|
Their deterministic clients and transport keep them independent of live
|
|
providers. Preparation and execution use repository-relative configuration
|
|
paths; observation embeds its assets and already has a workflow helper returning
|
|
a summary and error.
|
|
|
|
[scripts/check-examples.py](../../scripts/check-examples.py) launches all three
|
|
programs, decodes their JSON summaries, and checks meaningful results. Ordinary
|
|
`go test ./...` compiles the example packages but currently has no example tests
|
|
to execute. The [ReleaseKit profile](../../releasekit.json) runs the Python
|
|
script as its one custom check after shared validation.
|
|
|
|
The separate [release-wiring test](../../scripts/test-release.py) is also Python,
|
|
but is a development-only diagnostic, not a release gate. Its disposable fixture
|
|
currently assumes the example script is a configured product check.
|
|
|
|
## Target State
|
|
|
|
Ordinary Go tests alongside the examples exercise the actual example workflows.
|
|
They are discovered by `go test ./...` and can be selected with
|
|
`go test ./examples/go-library/...`. ReleaseKit's existing standard suite runs
|
|
them with its selected race and vet settings; no second example-validation pass
|
|
is needed.
|
|
|
|
Preserve the useful coverage currently supplied by the Python runner:
|
|
|
|
- Preparation: collection composition and overrides, scoped selection,
|
|
inspection/preparation, and the resulting identities and inventories.
|
|
- Execution: injected-client execution, expected output and structural validity,
|
|
model selection, and usage reporting.
|
|
- Observation: bounded transient retries, attempt metadata, incomplete usage
|
|
reporting, and application-owned interpretation of proposal/assessment results.
|
|
|
|
Assertions should remain tied to the synthetic examples and meaningful results.
|
|
Their output formatting and fixture values do not become new public API
|
|
guarantees. Preserve relevant output coverage without freezing complete JSON
|
|
documents or introducing broad golden snapshots.
|
|
|
|
## Example And Test Structure
|
|
|
|
Prefer small, unexported workflow helpers that return errors or results and,
|
|
where useful, accept an output writer. Keep `main` responsible for process-level
|
|
output and exit handling. Reuse the observation example's existing separation
|
|
where it already meets the need; uniform helper signatures are not an objective.
|
|
|
|
Tests should normally call the workflow directly, using local output buffers
|
|
and the existing deterministic collaborators. Handle resource paths explicitly
|
|
so package-directory test execution and the documented repository-root
|
|
`go run` commands both work. A small resource-base or filesystem input is
|
|
preferable to changing process-wide working directories or redirecting global
|
|
standard output merely to reach the workflow.
|
|
|
|
Use a Go subprocess test only where launching the executable protects a distinct
|
|
risk or avoids making an example materially harder to understand. Do not add
|
|
one for every example automatically. Subprocesses need their own consideration
|
|
of toolchain, offline execution, and instrumentation; parent race or coverage
|
|
settings do not automatically instrument a separately launched program.
|
|
|
|
Keep tests deterministic, offline, and independent of real credentials under the
|
|
[testing policy](../policy/testing.md). Avoid a new public package, shared
|
|
example framework, or expanded failure-case inventory solely for this migration.
|
|
Examples remain consumer programs rather than release artifacts or library
|
|
implementation dependencies, consistent with the
|
|
[architecture policy](../policy/architecture.md).
|
|
|
|
## Integration And Documentation Work
|
|
|
|
Once the Go tests cover the retained behavior, remove `scripts/check-examples.py`
|
|
and its custom command and environment overlay from `releasekit.json`. Keep the
|
|
existing built-in validation selections and source/hosted publication workflow.
|
|
There is no need for new ReleaseKit functionality or a dependency upgrade.
|
|
|
|
Adapt the development-only release-wiring fixture so it no longer expects the
|
|
removed script or product-check marker. Retain checks relevant to the actual
|
|
launcher and profile; do not manufacture an unused custom hook merely to retain
|
|
old fixture assertions. The Python release-wiring diagnostic can remain Python.
|
|
Removing Python from every repository tool is outside this scope.
|
|
|
|
Update the [development guide](../development.md),
|
|
[testing policy](../policy/testing.md), [release policy](../policy/release.md),
|
|
and [release procedure](../release.md) when implementation lands. Replace the
|
|
standalone example command with Go test guidance, describe example coverage as
|
|
part of the standard suite, and remove Python as a normal release prerequisite.
|
|
Continue documenting Python where the optional release-wiring test needs it.
|
|
Check other incoming references to the removed runner and retain the existing
|
|
consumer quickstart commands.
|
|
|
|
## Verification
|
|
|
|
Verify the example packages through ordinary Go tests and the selected
|
|
race-enabled suite and vet. Confirm the documented `go run` entry points still
|
|
work from the repository root, and exercise the adapted release-wiring test in
|
|
disposable repositories. These are migration checks, not new recurring release
|
|
steps. Check that a meaningful example assertion failure fails the Go suite
|
|
rather than relying only on successful compilation.
|
|
|
|
Completion means useful example behavior remains covered, normal test execution
|
|
includes that coverage, and the obsolete runner, profile hook, and documentation
|
|
references are removed. No performance thresholds, coverage percentage targets,
|
|
new platform commitments, hosted CI requirements, or changes to Promptkit's
|
|
consumer contracts are introduced by this work.
|