Files
scriptorium/docs/roadmap/implementation.md

453 lines
26 KiB
Markdown
Raw Normal View History

2026-11-17 20:40:57 +00:00
# ConfigKit v1.2.0 Interim Implementation Plan
## Status, Baseline, And Authority
2026-11-17 20:40:57 +00:00
Planned on 2026-11-17 for a Luna coding agent against Scriptorium
`a5d8394313c4cbdee8e8618844e65364c70189e9`. Stage completion is recorded below.
2026-11-17 20:40:57 +00:00
The [feature roadmap](configkit.md) owns the target, rationale, native-semantics
policy, and confirmed interim sequencing decision. This document owns the
ordered implementation work.
This replaces the completed PromptKit v0.12.0 implementation plan, preserved at
`a5d8394313c4cbdee8e8618844e65364c70189e9:docs/roadmap/implementation.md`.
Do not execute that earlier plan again.
Adopt application `go.maximumdirect.net/configkit v1.2.0` at
`c1c7e87f2cdf77a0cbf3327831ac05578e756b9e`. Retain
`gitea.maximumdirect.net/eric/ConfigKit v0.5.0` at
`dc09959cca4e105f49ec316c6d4ba585bd2340bd` where the unchanged
`gitea.maximumdirect.net/eric/promptkit v0.12.0` public API needs its types.
Keep go-yaml v4.0.0-rc.6 and the existing PromptKit catalog selections.
Use ConfigKit's pinned
[consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e/docs/consumers/README.md),
[configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e/docs/config.md),
and public root-package declarations. The sibling checkout currently contains
v1.2.1; read the selected tag or downloaded v1.2.0 module, not unqualified sibling
HEAD. Keep PromptKit behavior tied to its existing tagged contract.
Planning evidence: a temporary module resolved both ConfigKit identities,
PromptKit v0.12.0, and the existing YAML prerelease. Package compilation with
`go test -run '^$'` passed for all three libraries on Go 1.27.4. This does not
constitute an application migration or test pass. Stable YAML cannot be selected
in this interim graph because old ConfigKit imports its removed
`plugin/limit` package.
## Sprint Execution Rules
Implement Stage 1 through Stage 7 in order as one end-to-end sprint. Intermediate
stages may fail compilation or tests and need not be independently deployable.
Do not add compatibility layers, temporary alternate loaders, skips, or weakened
tests just to make an intermediate stage green. Use focused checks as useful
feedback, record failures accurately, and continue when the remaining work is
expressly assigned to later stages. Investigate unrelated failures. Only the
completed end state must pass the full validation.
Start with working-tree inspection and the task reading guides in
`docs/development.md`. Follow the agent policy and applicable architecture,
testing, documentation, OSS, and release policies. Use the graph for code
discovery. Keep completed-stage evidence here concise and accurate; commit
focused completed units under the agent policy.
The work is native ConfigKit adoption, not preservation of old processing
semantics. Keep application schema, domain defaults, normalization, validation,
path interpretation, and CLI selection choices with Scriptorium under ADR 0005.
Imports, composition, inheritance, decoding, inspection, and processing order
belong to the selected library. Do not introduce new restrictions, limits,
identity rules, sensitivity heuristics, or guarantees because they are convenient
to assert.
This sprint does not upgrade PromptKit, migrate its module path, select ConfigKit
v1.2.1 or stable YAML, add configuration commands/endpoints, change resource paths
to source-relative interpretation, or adopt ReleaseKit. HTTP request independence,
shared backend capacity, artifact handling, credential disclosure, execution
controls, and existing request/response contracts are not redesign targets.
The existing restriction on committed workspaces, replacements, vendoring, and
framework internals remains applicable.
Use real temporary files and the existing assembled collaborators for tests;
provider interactions use existing fakes or loopback HTTP servers. Default tests
stay offline and credential-independent. Prefer typed errors, diagnostic codes,
effective values, and meaningful history over native prose snapshots or fixed
source IDs. Reuse unchanged successful checks; do not run the entire baseline
after every intermediate edit. Final validation is in Stage 7.
## Stage 1: Select The Interim Dependencies And Go Toolchain
**Complete.** Go 1.27.4 and the selected dependency graph were provisioned
from the local cache. Offline `go list -m all` confirmed both ConfigKit
versions, PromptKit v0.12.0, the existing catalogs, and YAML v4.0.0-rc.6.
2026-11-17 20:40:57 +00:00
**Work**
1. Select Go 1.27.4 for release validation and builds. It is available from the
[official Go downloads](https://go.dev/dl/), and the official
`golang:1.27.4` image supports the existing Linux targets.
Provision the toolchain and dependency cache before offline validation.
2. Set the consumer directive in `go.mod` to `go 1.27.0`, as required by
ConfigKit v1.2.0. Add the exact vanity module version without removing the
old ConfigKit requirement or changing PromptKit, catalogs, or YAML.
The application module path remains unchanged.
3. Use ordinary Go module commands to update required sums. Defer final
`go mod tidy` until source imports are migrated, so the newly selected
dependency is not removed merely because it is not used yet. Do not add a
`tool` entry for upstream maintainer tools or require Go 1.27.4 from all
consumers through a new toolchain directive.
4. Change only the Go build image in `.woodpecker/release.yml` from 1.26.11 to
1.27.4. Retain `GOTOOLCHAIN=local`, build targets, publication configuration,
and asynchronous delivery behavior. No release-script gate or workflow
redesign is needed.
**Verification focus**
Inspect `go list -m all` for the exact selected modules above. Tool-only graph
requirements may appear without being runtime imports; inspect actual selection
rather than delete them or upgrade unrelated modules speculatively. Compile
failures from the remaining application migration belong to subsequent stages.
Do not publish, tag, push, or invoke the release script.
## Stage 2: Move Application Loading And Resolution To Native v1.2.0
**Complete.** Application loading uses native search and v1.2.0 resolution;
`go build ./internal/config` passed with Go 1.27.4. Adapter and test migration
remain assigned below.
2026-11-17 20:40:57 +00:00
**Primary files:** `internal/config/document.go`, `load.go`, and `settings.go`.
**Work**
1. Change application ConfigKit imports to `go.maximumdirect.net/configkit`.
Keep `Document`, its field schema, `LoadOptions`, `AppSettings`, and
`Settings` responsibilities. Preserve fresh defaults, selector order,
nullable retry presence, domain validation, and working-directory-relative
application resource strings. Add no framework-resource acquisition to
application validation.
2. Replace the `os.Stat` loop in `Load` with source selection only:
- explicit selection: `configkit.File(options.Path)`, even for a blank
explicit argument, leaving its failure to ConfigKit;
- implicit selection:
`configkit.SearchConfig("scriptorium", configkit.SearchPaths(defaultConfigSearchPaths...))`.
Retain the existing ordered two-location slice and test substitution seam.
Construct the search for each load. Remove the `fileBacked` bookkeeping,
filesystem probing, and hand-written discovery error formatting.
3. Call ordinary `configkit.Load` once with the existing definition, selected
overlay, explicit overrides, and `Imports: true`. Leave import defaults,
filesystem context, source-version policy, and sequence registrations unset.
Optional discovery exhaustion follows native absent-main behavior.
4. Keep the existing contextual load-error wrapper with `%w` so the new
`*configkit.LoadError` remains discoverable. Do not recover by searching
another file after a selected candidate fails.
5. In `resolveDocument`, keep field-level `Normalize` calls. Remove the
`sourcePathsChanged` aggregation and redundant generic whole-list
`resolution.Record` call. Simplify the local path-normalization helper's
boolean result if it has no other caller. Use `RecordNormalization` only
for actual direct mutations requiring it; no replacement container record
is necessary for today's same-length list-field edits.
6. Preserve ordinary native control and collection handling. Add neither
legacy control aliases nor application filtering of declared imports,
inference of collection identity, or a second composition pass.
**Verification focus**
Review the loader against the selected public lifecycle and the resolver against
v1.1+ leaf recording. Existing unit tests still require import/syntax changes in
Stage 4. Do not broaden `LoadOptions` or introduce a filesystem/merge framework
to preserve the old test setup.
## Stage 3: Separate Application And Framework Types In CLI Adapters
**Primary files:** `internal/adapter/cli/application_config.go`, `config.go`,
`run.go`, `inspect.go`, and `framework_config.go`.
Use `configkit` for the new application package and `legacyconfigkit` for
the old package where both are imported.
| Boundary | ConfigKit owner |
| --- | --- |
| Application overrides, paths, results, stored diagnostics, and writers | Vanity v1.2.0 |
| `newEngine` input/underlay construction and `promptkit.DirectoryInputs` values | Old-path v0.5.0 |
| Captured framework paths, native source records, diagnostics, and writers | Old-path v0.5.0 |
**Work**
1. Migrate `applicationOverrides`, `loadApplicationSettings`, and
`writeApplicationWarnings` to new public types and writers. Migrate stored
application diagnostics in run, serve, common-command, prompt/profile
inspection, and framework-command selection structs. The latter's
`overrides` and pre-engine `diagnostics` are still application-owned.
2. Keep `newEngine`'s `Input`, `Underlay`, and `File` calls old-typed.
Continue passing `promptkit.DirectoryInputs` output directly to that old
boundary. Keep source ordering, underlay grouping, retry defaults, and shared
server engine construction unchanged.
3. `configCommand` uses new paths and writers. Its existing
`configurationSource` DTO uses the new `Layer` type, yielding `main`
for application file sources. Keep its JSON shape and empty-array behavior.
4. Framework `explain` parses paths with `legacyconfigkit.ParsePath` and
passes them to the existing view. Keep `frameworkConfigurationSource`,
`frameworkConfigurationSources`, `Layer`, and `SourceKind` old-typed;
it is already a separate DTO, so no new shared conversion type is needed.
5. In `inspectFrameworkConfigurationCommand`, retain the new application
warning call for `cfg.diagnostics`. Send `view.Diagnostics()` directly
to `legacyconfigkit.WriteDiagnostics`, propagating writer failure through
the existing runtime-error exit. Keep native framework output and its
established scoped redaction.
6. Preserve structured error presentation for both generations. In the existing
common `writeApplicationError` helper, check for the new `*LoadError`
and then the old `*legacyconfigkit.LoadError` with `errors.As`, sending
each one's diagnostics to its matching writer. Retain current handling of
diagnostic-write failure and the fallback for non-load errors. This small
explicit dispatch preserves errors from engine construction as well as
application loading; it is not a cross-version data-conversion layer.
7. Audit actual ConfigKit references by owner rather than globally replacing
imports. No casts, YAML/JSON round trips, or duplicate configuration result
representations are needed between these generations.
**Verification focus**
Check both native writer signatures and all CLI construction paths. Compilation
provides useful feedback, but stale tests/fixtures remain assigned below. Neither
successful application configuration validation nor its inspection should
construct a PromptKit engine.
## Stage 4: Migrate And Extend Application Configuration Tests
**Primary files:** `internal/config/document_test.go` and `load_test.go`.
**Work**
1. Migrate their imports, error assertions, paths, and `LayerBase` references
to the new package and `LayerMain`. Change application input controls from
`configkit` to `config`, marking indispensable imported fixtures
`required: true`. Do not edit framework-definition fixtures as if they
were application documents.
2. Retain and adjust existing discovery/defaults tests. Cover the first
candidate, a missing first candidate selecting the second, optional
exhaustion, explicit missing/blank selection, and a malformed or nonregular
selected candidate that does not fall through to a valid second file.
Reuse `useConfigSearchPaths`; tests that mutate that package state remain
nonparallel and restore it through cleanup.
3. Update `TestLoadDiscoveryErrorQuotesPath` to inspect a typed load error's
rendered diagnostics, not demand that `LoadError.Error()` contain a path.
Keep the real protection: terminal control characters in a failing path are
escaped by the native writer. A deterministic nonregular candidate can be
used instead of requiring host symlink support.
4. Rename/reframe `TestLoadRejectsOverlayWithoutBase` around this application's
absent-main/no-context selection. It should still fail for an active overlay,
while `default` selects none. It must not claim ConfigKit universally
prohibits overlays without a main or imports on an absent main.
5. Add focused import coverage: missing optional input succeeds with omitted
requiredness, the corresponding required input fails, and the removed
`optional` field fails. Use one `dir` import with an authored YAML
pattern and `destination: [server]` to mount a fragment containing `addr`.
Verify final application values and an applicable origin, not a complete
upstream traversal/filter matrix.
6. Demonstrate native collection policy on the existing selector list with
YAML `config.sequences`, `destination: [promptkit_sources]`, `key: file`,
and `mode: upsert`. Use only file-selector records in that fixture.
Distinct imported peers combine; a duplicate authored identity conflicts;
an overlay appends a new identity while retaining lower records.
Contrast unkeyed whole-list replacement and explicit CLI whole-list
replacement. These are authored policies, not Go identity registrations.
7. Cover changed CLI timing with a paired case: a source value with a decoding
type error still fails despite a replacing override; a decodable value
failing application validation can be corrected by an override.
Retain explicit zero/false/empty/allowed-null tests where applicable to the
schema and callbacks; create no new fields solely to assert all categories.
8. Keep effective-value, source-order, snapshot, and normalization coverage.
For a changed selector leaf, assert its normalized result and recorded
normalization history; for an unchanged leaf, assert no invented operation.
Avoid testing incidental history counts or private callback choreography.
**Verification focus**
Run `go test ./internal/config` when useful. Preserve distinct regression
protection already present; consolidate overlapping assertions rather than
clone ConfigKit's complete format suite.
## Stage 5: Migrate CLI Integration Fixtures And Maintain Runnable Examples
**Primary files:** CLI configuration/execution/inspection tests and
`examples/`. Review HTTP/report test helpers only for their ConfigKit owner.
**Work**
1. In CLI tests, use new ConfigKit for application-load errors/diagnostics and
old ConfigKit for framework engine setup and captured view records.
Specifically review `run_test.go`, `framework_sources_test.go`, and
`framework_config_test.go`. Keep the old imports in the HTTP
`newHandlerEngineFromDirectories` helper and report engine-construction
test, because they construct PromptKit sources rather than load application
settings.
2. Migrate application control literals in `config_test.go` and
`run_test.go` to `config.imports`, with explicit requiredness when
absence should fail. Retain framework controls in the generation selected
by PromptKit v0.12.0. Keep pre-existing backend-inheritance coverage in the
framework, where those backend definitions now live.
3. Exercise the Stage 3 dispatch using real failed new application loads and
failed old framework loads. Keep stdout clean on failures, native diagnostics
on stderr, and diagnostic/output writer-failure coverage. Existing framework
`show`, `sources`, and `explain` tests protect the old path, records, and
redaction; update only expectations actually owned by new application loading.
4. Protect an assembled path that loads new application imports and an overlay,
then renders through unchanged PromptKit definitions. Include a framework
file with its supported old import controls so blanket control renaming would
be detected. Continue application-only inspection coverage with nonexistent
framework resources to prove it does not acquire them.
5. Change `examples/config.composed.yml` to root `config.imports` and add
`required: true` to its source-selection fragment. Other ordinary application
and framework examples change only when their actual owner requires it.
6. Add this concrete, small collection example using existing framework files:
- `examples/config.collections.yml`: ordinary application main; import
directory `config-fragments/collections` with `pattern: "*.yml"` and
`required: true`; declare `promptkit_sources` identity `file` and
`mode: upsert` under root `config.sequences`; set
`schema_dir: ./examples/schemas`.
- In that imported directory, `10-backends.yml`, `20-profile.yml`, and
`30-prompt.yml` each contribute one file selector, respectively
`./examples/framework/backends.yml`,
`./examples/profiles/local-fast.yaml`, and
`./examples/prompts/generic.markdown_summary.yaml`.
- `examples/overlays/collection.yml` adds one file selector for
`./examples/profiles/local-gpu.yml`. The inherited upsert policy retains
the three lower selectors and appends the new one.
No new `id` field, default registration, runtime merge logic, or copied
framework definition is needed. Application path strings remain relative to
the repository invocation directory.
7. Exercise the maintained collection main and overlay through the application
load and at least one no-generation render with the transcript and glossary
fixtures. Compare effective selector order/retention and a real prepared
result, not a giant native-text golden. Reuse tests from Stage 4 where their
risk is already covered.
8. Finish `go mod tidy` after the imports are in place. Confirm the exact
interim versions and distinguish tool-only module requirements from the
runtime graph. Leave implementation of v1.2.1 and the upcoming PromptKit
roadmap to the separate work set.
**Verification focus**
Run affected CLI tests and the existing HTTP/report suites as warranted by
changed imports. All fixtures must eventually compile with both identities.
New source-list composition must reach the old framework boundary in order;
do not sort or deduplicate it to repair a test.
## Stage 6: Update Canonical Documentation And Both ConfigKit Attributions
**Work**
1. Update `docs/config.md` for application ConfigKit v1.2.0 and native
discovery. Describe the actual processing order, optional import default,
explicit `required`, expanded selectors/placement, authored collection
policy, and CLI timing. Correct unconditional statements that overlays always
replace lists and defaults-only loading cannot enable imports. Explain that
this CLI supplies no independent overlay root; the context limitation is
not a universal library prohibition.
2. Clearly distinguish application root `config` controls from the unchanged
PromptKit v0.12.0 framework contract. Keep working-directory-relative resource
values, strict application schema, and current credential handling. Link to
the pinned upstream contract for detailed grammar instead of copying it.
3. Update `docs/cli.md` for application `main` provenance and changed
application loading behavior. Keep source JSON fields, command flags,
streams, and exits otherwise unchanged; framework inspection continues to use
its own old native types and writers. Add no new source-detail or diff command.
4. Update README example navigation and the owned configuration examples.
Add copyable inspection and render invocations for the collection example.
Use `--config-overlay collection` for its added selector and both existing
input fixtures for `generic.markdown_summary`; no provider connection or
credential is needed for these render checks.
5. In `docs/development.md`, document Go 1.27.0 as the consumer floor because
of application ConfigKit, and point release builds to Go 1.27.4. Update
`docs/release.md`'s selected toolchain, installation path example, and version
output to 1.27.4, matching the image from Stage 1. Leave release commands,
gates, footer conventions, and publication completion unchanged.
6. Update `docs/internal/sources.md`, `docs/internal/adapters.md`, and the
overview as affected: native discovery, two ConfigKit generations, leaf
resolution recording, warning/error dispatch, and actual selector conversion.
Remove the obsolete claim that application backend IDs are sorted.
Update only the architecture policy's affected dependency/boundary description
and local date; preserve its base version, upstream provenance, and adopted
decisions. Accepted ADR content stays unchanged.
7. In `docs/oss/attribution.md`, retain a distinct entry for old-path ConfigKit
v0.5.0 used by PromptKit and add application ConfigKit v1.2.0. Use factual
headings/identification to distinguish them, the actual copyright and
`LGPL-3.0-or-later` grant, and shared local LGPL/GPL text links.
New ConfigKit source/notice links use forge commit
`c1c7e87f2cdf77a0cbf3327831ac05578e756b9e`; legacy ConfigKit uses its
retained commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd` at that same
canonical ConfigKit repository. Verify notice files at both revisions.
8. Preserve unchanged PromptKit, catalog, YAML prerelease, and other dependency
notices. Inspect newly selected runtime material and add notices only if
actually incorporated or distributed; upstream ReleaseKit, Goldmark, and
Go module tooling requirements alone are not runtime inclusion. No project
relicense, new standard-text copies, or tool adoption follows automatically.
9. Keep the feature roadmap as target/context rather than a duplicate step log.
Preserve the explicit v1.2.1/stable-YAML/old-module-removal handoff to the next
PromptKit roadmap. Do not overwrite the completed historical PromptKit
roadmap or invent a release version to create a versioned release note.
**Verification focus**
Verify current facts against implementation, affected local links and pinned
source paths, license-text fidelity, and changed example commands. Preserve
published release notes and tags. The eventual new minor release and its note
remain separate preparation; this stage does not authorize publication.
## Stage 7: Validate The End State And Record The Handoff
**Work**
1. Review the entire diff for unintended contract strengthening or obsolete
compatibility logic: native search rather than probes; default optional
imports; no default selector identity; post-decode CLI assignment; leaf
normalization; no new redaction, filesystem policy, or framework processing.
Confirm all imports, paths, diagnostics, and writers belong to their owner.
2. Inspect `go list -m all` and `go list -deps ./cmd/scriptorium` under the
selected release toolchain. Verify both ConfigKit identities at their exact
versions, unchanged PromptKit/catalogs/YAML, no replacements/workspaces, and
the intended runtime versus tool-only dependency distinction. Do not freeze
the entire dependency inventory into a new permanent test.
3. Format changed Go code and run the existing development baseline once against
the final source: `go test ./...`, `go test -race ./...`,
`go vet ./...`, and a native `go build` of `./cmd/scriptorium` with
output outside the repository. Use existing offline/provider-independent
fixtures; loopback test servers are local test infrastructure, not live probes.
4. Check both release targets with the existing build settings. With Go 1.27.4
already selected and the dependency cache provisioned, the following are
validation commands, not publication:
```sh
2026-11-17 20:40:57 +00:00
validation_dir=$(mktemp -d)
export GOWORK=off GOTOOLCHAIN=local GOFLAGS=-mod=readonly
export GOPROXY=off GONOPROXY=none GOSUMDB=off
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags '-s -w' \
-o "$validation_dir/scriptorium-linux-amd64" ./cmd/scriptorium
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags '-s -w' \
-o "$validation_dir/scriptorium-linux-arm64" ./cmd/scriptorium
```
2026-11-17 20:40:57 +00:00
Keep products outside the worktree. Do not invoke `scripts/release.sh`,
push, tag, poll hosted delivery, or add a new release check.
5. Run the maintained ordinary, composed, inheritance, and new collection
no-generation examples; check collection overlay retention, representative
application inspection, and old framework inspection. Reuse equivalent
unchanged successful checks. Run `git diff --check` and validate affected
documentation links and workflow YAML/shell syntax.
6. Audit every roadmap completion criterion against code, tests, examples,
metadata, and documentation. Fix remaining gaps instead of marking a partly
migrated but green subset complete. Do not carry old PromptKit stage records
forward as evidence for this sprint.
7. Record concise actual validation results and mark stages complete only when
their work is done. Update the feature roadmap status after the complete
audit, retaining its scope and later-upgrade handoff. Commit remaining scoped
changes with a plain-English message, review status, and report residual
uncommitted work or blocked validation accurately.
Completion means application ConfigKit v1.2.0 is native and operator-visible
where already supported, PromptKit v0.12.0 still works through old ConfigKit,
the shared YAML prerelease remains selected, and all seven stages are delivered.
It does not mean ConfigKit v1.2.1, stable YAML, or a new Scriptorium release has
been delivered. No significant product questions remain open for this sprint.