21 KiB
Implementation Review Remediation
ConfigKit v1.2.0 Interim Adoption Review
Reviewed on 2026-11-17 at
eec4fa2d91beca68b176dcf20026a0c7e5a7b7bf, covering the seven implementation
commits after 222f405. The review followed each stage of the
implementation plan against the code, tests, examples,
documentation, and feature roadmap.
The runtime architecture matches the selected interim target. Application configuration uses public ConfigKit v1.2.0 with native discovery, composition, post-decode CLI assignment, recorded leaf normalization, and final domain validation. PromptKit v0.12.0 retains old-path ConfigKit v0.5.0 for framework sources and captured inspection. Each generation's paths, diagnostics, and writers remain separate; the shared YAML dependency stays at v4.0.0-rc.6. No runtime correctness defect, unnecessary compatibility layer, or unrequested runtime restriction was found. One documentation contract overstatement remains.
| Stage | Review result |
|---|---|
| 1: Dependencies and toolchain | Both ConfigKit identities, unchanged PromptKit/catalogs/YAML, Go 1.27.0 consumer minimum, and Go 1.27.4 release image are selected as planned. |
| 2: Native application loading | One native load owns discovery and processing; application callbacks retain schema/domain responsibilities and leaf recording. |
| 3: Adapter type separation | Application and framework values use their respective public types; structured errors and warnings reach the matching native writer. |
| 4: Configuration coverage | Tests cover discovery, optional/required imports, placement, authored collections, CLI timing, presence, normalization, and inspection without new application identity rules. |
| 5: Integration and examples | New application controls coexist with old framework controls; ordinary, composed, inheritance, and collection examples work, including overlay retention. |
| 6: Documentation and attribution | Dependency/toolchain ownership and both notices are aligned; selector-clearing language still overstates the native contract (R6). |
| 7: End-state validation | Independent tests, builds, example commands, dependency checks, and document checks passed; R6 must be resolved for complete documentation alignment. |
R6: Qualify Selector Clearing By The Effective Collection Policy
Priority: P3 (low). Status: Open.
Locations: docs/config.md:62–64 and docs/config.md:108–109.
The selector reference says that omitting promptkit_sources or setting it to
[] or null selects no application-supplied framework sources. The overlay
description also says that explicit nulls replace without qualifying collection
operations. Those statements retain the former simple-list contract even
though the upgrade now exposes native authored collection policies. The later
general statement that policies may combine lists does not explain the
different empty/null behavior.
Evidence: with this application main document:
config:
sequences:
- destination: [promptkit_sources]
key: file
mode: upsert
promptkit_sources:
- file: ./base-framework.yml
an overlay containing promptkit_sources: [] makes config show succeed and
retain base-framework.yml. An overlay containing promptkit_sources: null
instead exits 1 with sequence.invalid: append and upsert require a sequence
contribution. These checks used the reviewed binary and temporary application
files; no framework file or provider was needed. The results agree with
ConfigKit's pinned collection contract.
Required change: distinguish an empty final resolved selector list from an
authored source contribution. Under ordinary replacement, an empty list clears
the selection and an allowed null produces no selectors. Under append/upsert,
[] contributes no records and null is invalid; omission preserves lower
contributions. Explain or link to the native way to clear an identified
collection: select mode: replace at that destination and supply [].
Qualify the overlay null statement accordingly.
This is a documentation fix. Preserve ConfigKit's current behavior and the existing CLI whole-list override semantics; do not add special-case clearing, new validation, or a compatibility merge pass to satisfy the old wording.
Acceptance: the selector and overlay descriptions agree with the pinned native contract and the reproduced empty/null cases. Check the affected links and any added example. No production-code change or new runtime guarantee is required.
Independent Review Verification
go test -count=1 ./...,go test -race -count=1 ./..., andgo vet ./...passed with Go 1.27.4 and a provisioned offline cache.- A native binary and Linux amd64/arm64 binaries built outside the worktree; cross-builds used the existing cgo-disabled, trimpath, stripped settings.
- All six maintained configuration/overlay selections passed application validate/show/sources/explain, framework validate/sources, and JSON render. Framework show/explain also passed. Selected credential variables were unset; no generation command or live provider was invoked.
- The selected module and executable dependency graphs,
go mod tidy -diff, changed-Go formatting, diff whitespace, affected local documentation links, workflow YAML/shell syntax, both ConfigKit notice revisions, and shared LGPL/GPL text fidelity were checked. No workspace or vendor tree is committed. - The only repository change made by this review is this remediation record. Historical findings below are retained unchanged.
PromptKit v0.12.0 Review
Reviewed on 2026-10-04 at
58cb02f9732b09740ff7e22096d2a4072ddfddca, covering the thirteen implementation
commits after b1bb55a. The runtime architecture matches the accepted
feature roadmap and implementation plan.
No runtime regression, missing framework capability, or newly imposed runtime
restriction was identified. Three low-priority documentation/help gaps were
identified; their status and resolutions are recorded below. The first retained
a loading guarantee that was incompatible with
the deliberately selected full-source engine construction.
R3: Correct The Definition-Inspection Loading Contract
Priority: P3 (low). Status: Complete.
Locations: docs/cli.md:208–224; related orientation in
docs/internal/adapters.md:49–56. The CLI wording was retained in b913ef0
despite the engine-wiring change in 0df2aad.
The profile-inspection reference still says it never loads a prompt. Both
definition-inspection commands now construct the same complete selected engine
as execution. PromptKit resolves and validates all final definitions and
captures their message content before the selected inspection runs. Consequently,
an unused prompt with invalid syntax or an unreadable content_file can prevent
inspection of an otherwise valid profile. Prompt inspection's unqualified
claim that it does not resolve a profile also needs to distinguish engine
assembly from request-specific target resolution.
Evidence: inspecting a valid profile in a selected combined file with an
unused prompt referring to a missing message file exits 1 before emitting a
profile report. The existing
TestProfileInspectionUsesCombinedSourcesAndValidatesUnusedPrompts also
protects this intentional eager-validation boundary.
Required change: explain that both commands construct and validate the full selected configuration, including unused definitions and message resources, before inspecting the chosen definition. Preserve the distinction that they do not prepare an execution request, load request-time schemas/artifacts, resolve credential values, reserve capacity, or call a provider. Clarify that no prompt collection is mandatory, but selected prompts are not ignored. Align the internal summary with the CLI contract rather than restating a stronger independence guarantee. Do not restore the empty-prompt-filesystem workaround or weaken eager validation to satisfy the stale wording.
Acceptance: the owning contract and internal summary agree with the roadmap, shared engine construction, and existing tests. Review the documentation diff and links; no production-code change or new runtime guarantee is needed.
Resolution: the CLI contract now distinguishes full-source engine construction from inspecting a selected definition, explains eager validation/content capture and unused-definition failures, and retains the request-time exclusions. It states that no prompt collection is mandatory but selected prompts are not ignored. The internal summary links to that contract without promising a separate no-prompt-loading path. Existing definition-inspection tests and documentation link/diff checks passed; no production code changed.
R4: Update The Composed-Example Description
Priority: P3 (low). Status: Complete.
Location: docs/config.md:95–96, after the example migration in 2027b80.
The reference says examples/config.composed.yml imports two application
fragments. It now imports only config-fragments/sources.yml; the former
backend fragment became examples/framework/backends.yml, selected through
promptkit_sources rather than imported into the application document.
The example and code are correct, but the explanation confuses the two source
boundaries introduced by this migration.
Required change: describe the actual source-selection fragment and separately selected framework backend file, with links where useful. Do not reintroduce an application backend mapping or add an unnecessary fragment merely to match the old prose. The old backend fragment had no application defaults to preserve.
Acceptance: the description matches the maintained files and the existing composed-overlay render command still succeeds offline.
Resolution: the configuration reference now links to the one imported application source-selection fragment and the separately selected framework backend file, retaining the local-overlay example. The existing composed-overlay render command passed offline with credential environment variables unset; the linked files and diff whitespace were checked. No configuration files or loading behavior changed.
R5: Include Framework Configuration Inspection In Built-In Usage
Priority: P3 (low). Status: Complete.
Location: internal/adapter/cli/run.go:742–749 (printUsage), following the
new command-family implementation in 2ba2ce1 and 411a67b.
The top-level usage lists inspect prompt, inspect profile, and application
config, but omits the implemented inspect configuration family entirely.
Invoking the binary without arguments reproduces this omission, so users
relying on its built-in command summary cannot discover the new framework
validation/provenance interface or distinguish it from application inspection.
Required change: add a concise usage line identifying all four operations,
the supported selection flags, and the --path requirement for explain.
Preserve existing exit behavior and command parsing; this does not require a
new help subsystem or aliases. Protect discoverability with a small CLI-entry
assertion for the family/operations, not a snapshot of the entire usage text
or its incidental ordering.
Acceptance: built-in usage advertises the actual family and remains consistent
with docs/cli.md; the focused CLI check and development baseline pass.
Resolution: built-in usage now advertises all four framework operations, the
supported selection flags, and that --path is required for explain.
TestUsageAdvertisesFrameworkConfigurationInspection exercises the CLI entry
point and checks the family, operations, flags, exit status, and output stream
without snapshotting the whole usage text or fixing operation order. It failed
before the usage change and passed afterward. Full tests, race tests, static
checks, a native build outside the repository, formatting, and diff checks
passed; command parsing and exit behavior are unchanged.
Remediation Completion
R3, R4, and R5 are complete. R1's earlier resolution was verified in the current HTTP timeout test: assertions concern deadlines and cancellation, not context identity. R2 remains superseded; the current plan and feature roadmap accurately record completed implementation and separate release preparation. No active finding remains, and no historical plan or release was rewritten.
Review Verification
go test -count=1 ./...,go test -race -count=1 ./..., andgo vet ./...passed. A native binary built outside the repository.- Changed Go files are formatted; diff whitespace checks passed.
- All four maintained application configurations passed both application and framework validation with provider credentials unset. Both render scripts, composed-overlay render, and inheritance render passed offline.
- All four captured framework inspection operations passed against the complete example. Focused tests cover native layering/imports/resource capture, literal IDs, presence/null inheritance, partial-profile completion, scoped effective/history redaction, and unchanged existing inspection behavior.
- The selected graph is PromptKit v0.12.0, ConfigKit v0.5.0, and the two
/v3catalogs at v3.20260930.0, without a local replacement or workspace. Selected PromptKit contract and notice links resolved. - The broader redaction in the new framework-report family, removed legacy source fields/flags, and eager definition capture are explicitly approved changes, not accidental strengthening. Shared server capacity, request independence, HTTP wire behavior, and lexical artifact containment remain intact. No implementation fixes were made during this review.
ConfigKit v0.5.0 Review
Reviewed on 2026-10-03 at bcf2ddf, covering the five implementation
commits after e1ce526. The ConfigKit adoption and runtime ownership
boundaries match the accepted roadmap. No runtime defect, missing application
feature, or unrequested strengthening of contracts was identified. One
documentation handoff issue was identified; its current status is recorded below.
R2: Correct The Completed Plan's Introductory Status
Priority: P3 (low). Status: Superseded; no action against the current plan.
Location: docs/roadmap/implementation.md:5–9.
The introduction still says "All five stages are pending" and directs the reader to implement Stage 1 through Stage 5. Each stage is actually marked Complete with completion evidence, and the feature roadmap identifies the adoption as delivered. These contradictory directions leave the overall handoff inaccurate and could lead a later coding agent to repeat completed work.
This is a completion-documentation issue, not an unfinished runtime feature. The documentation policy assigns implementation status to roadmaps; the overall status should agree with the stage evidence.
Required change: update the introduction to state that all five stages are complete. Describe the retained steps as implementation history and acceptance evidence rather than instructions to start the work again. Preserve the original planning baseline, completed stage records, deferred scope, and separate release handoff. Do not change application code or rerun the feature implementation to repair the wording.
Acceptance: the overall status, individual stages, and feature roadmap agree
that implementation is complete, while release preparation remains separate.
Review the documentation diff and run git diff --check; this correction
does not require new tests.
The ConfigKit plan was replaced by the PromptKit plan in b1bb55a. Its
historical snapshot remains in Git, while the current introduction correctly
states that all thirteen PromptKit stages are complete. This old finding does
not direct changes to the new plan or repetition of either implementation.
Review Verification
go test -race -count=1 ./...passed, including the new configuration, inspection, and engine-adoption tests and existing shared-capacity coverage.go vet ./..., a native build outside the repository, and diff checks passed.- All five maintained inheritance example commands succeeded with
LOCAL_GPU_API_KEYabsent: validate, show, sources, explain, and render. The child selected its own backend ID with the inherited endpoint, zero queue capacity and false parameter remained explicit, and explanation retained the parent/destination/directive relationship. - Changed upstream documentation and notice links, affected local paths, and the historical policy-plan reference resolved.
- The work set changes dependency pins, tests, examples, and documentation; no production Go implementation was changed. ConfigKit v0.5.0 owns inheritance through the existing single-load integration; Promptkit remains v0.11.2.
- The
extendsreservation is the explicitly accepted upstream compatibility change, not a new Scriptorium restriction. Parameter and credential-reference visibility, working-directory-relative paths, ordinary parent registration, and existing shared server capacity remain as selected.
Earlier Promptkit Upgrade Review (Resolved)
Scope And Status
Review baseline: e177d7c, covering implementation commits after 5c3b0cf.
This document records a resolved test-contract issue; it does not authorize
unrelated implementation changes. No runtime defect or missing planned feature
was identified in this review. Release preparation remains separate work.
R1: Remove The Unrequested Context-Identity Invariant
Priority: P3 (low). Status: Complete.
Original location: internal/adapter/http/handler_test.go:334–349,
TestHandlerRunTimeoutValidationAndContextMapping, introduced in 7de2b02.
The test's child field and (ctx != parent) != tc.child assertion required
the exact parent context object for omitted/null/zero timeouts and a distinct
object for a positive timeout. The requested contract concerns effective
deadlines and cancellation propagation, not Go interface identity. In
particular, a harmless derived context that preserves the parent's deadline,
values, and cancellation would fail the omission cases despite satisfying the
application contract.
This elevated an incidental implementation choice into a test-enforced invariant, contrary to the agent policy and testing policy. The current handler's context behavior is correct; this is a maintainability issue, not a runtime cancellation bug.
Required Change
Remove the identity comparison and the identity-only table field. Retain behavioral assertions that an earlier parent deadline is not extended and parent cancellation reaches execution. For omitted/null/zero values, verify that no deadline is introduced when the parent has none. Retain coverage of positive deadlines and the distinction between a canceled execution context and a provider-attempt timeout while the execution context remains live.
Use the existing runner seam and focused cases; do not introduce a context wrapper framework, timer abstraction, new public guarantee, or production refactor solely to change this test.
Completion Criteria
- Tests protect effective deadline/cancellation behavior without requiring context object identity.
- A semantically equivalent derived context does not fail merely because it is a different object.
- Existing timeout validation and HTTP outcome coverage remains intact.
Resolution And Validation
Removed the identity-only table field and context comparisons. The test now checks effective deadlines, including omitted/null/zero requests with no parent deadline, and explicitly checks that parent cancellation reaches the runner. Existing positive-deadline integration tests and live-context provider-timeout mapping remain intact. No production code changed.
Focused HTTP timeout tests, go test ./..., go test -race ./...,
go vet ./..., a native build outside the repository, and git diff --check
all passed after remediation.
Original Review Verification
go test -race -count=1 ./...passed, including retry, shared-capacity, deadline, report, definition-conformance, and architecture-boundary tests.go vet ./...passed.- A native
cmd/scriptoriumbuild passed, with output outside the repository. - The selected dependency is Promptkit v0.11.0 with its intended v2 catalogs.
- Application code, tests, release automation, and historical releases were not changed during this review.
The runtime ownership boundaries remain as intended: Promptkit owns framework formats, execution, retries, and backend admission; Scriptorium owns executable adapters, application settings, HTTP artifact policy, deadlines, and terminal report presentation. No additional runtime restrictions, durable workflow state, or extra release compliance gates were identified in the reviewed diff.