Files
scriptorium/docs/roadmap/remediation.md

393 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Implementation Review Remediation
## ConfigKit v1.2.0 Interim Adoption Review
Reviewed on 2026-11-17 at
`eec4fa2d91beca68b176dcf20026a0c7e5a7b7bf`, covering the seven implementation
commits after `222f405`. The review followed each stage of the
[implementation plan](implementation.md) against the code, tests, examples,
documentation, and [feature roadmap](configkit.md).
The runtime architecture matches the selected interim target. Application
configuration uses public ConfigKit v1.2.0 with native discovery, composition,
post-decode CLI assignment, recorded leaf normalization, and final domain
validation. PromptKit v0.12.0 retains old-path ConfigKit v0.5.0 for framework
sources and captured inspection. Each generation's paths, diagnostics, and
writers remain separate; the shared YAML dependency stays at v4.0.0-rc.6.
No runtime correctness defect, unnecessary compatibility layer, or unrequested
runtime restriction was found. One documentation contract overstatement was
identified; its remediation is complete below.
| Stage | Review result |
| --- | --- |
| 1: Dependencies and toolchain | Both ConfigKit identities, unchanged PromptKit/catalogs/YAML, Go 1.27.0 consumer minimum, and Go 1.27.4 release image are selected as planned. |
| 2: Native application loading | One native load owns discovery and processing; application callbacks retain schema/domain responsibilities and leaf recording. |
| 3: Adapter type separation | Application and framework values use their respective public types; structured errors and warnings reach the matching native writer. |
| 4: Configuration coverage | Tests cover discovery, optional/required imports, placement, authored collections, CLI timing, presence, normalization, and inspection without new application identity rules. |
| 5: Integration and examples | New application controls coexist with old framework controls; ordinary, composed, inheritance, and collection examples work, including overlay retention. |
| 6: Documentation and attribution | Dependency/toolchain ownership and both notices are aligned; selector-clearing wording is now corrected (R6). |
| 7: End-state validation | Independent tests, builds, example commands, dependency checks, and document checks passed; R6's documentation alignment is now complete. |
### R6: Qualify Selector Clearing By The Effective Collection Policy
Priority: P3 (low). Status: Complete.
Locations: `docs/config.md:62–64` and `docs/config.md:108–109`.
The selector reference says that omitting `promptkit_sources` or setting it to
`[]` or `null` selects no application-supplied framework sources. The overlay
description also says that explicit nulls replace without qualifying collection
operations. Those statements retain the former simple-list contract even
though the upgrade now exposes native authored collection policies. The later
general statement that policies may combine lists does not explain the
different empty/null behavior.
Evidence: with this application main document:
```yaml
config:
sequences:
- destination: [promptkit_sources]
key: file
mode: upsert
promptkit_sources:
- file: ./base-framework.yml
```
an overlay containing `promptkit_sources: []` makes `config show` succeed and
retain `base-framework.yml`. An overlay containing `promptkit_sources: null`
instead exits 1 with `sequence.invalid`: append and upsert require a sequence
contribution. These checks used the reviewed binary and temporary application
files; no framework file or provider was needed. The results agree with
[ConfigKit's pinned collection contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e/docs/config.md#yaml-declarations-and-policy-lifetime).
Required change: distinguish an empty final resolved selector list from an
authored source contribution. Under ordinary replacement, an empty list clears
the selection and an allowed null produces no selectors. Under append/upsert,
`[]` contributes no records and null is invalid; omission preserves lower
contributions. Explain or link to the native way to clear an identified
collection: select `mode: replace` at that destination and supply `[]`.
Qualify the overlay null statement accordingly.
This is a documentation fix. Preserve ConfigKit's current behavior and the
existing CLI whole-list override semantics; do not add special-case clearing,
new validation, or a compatibility merge pass to satisfy the old wording.
Acceptance: the selector and overlay descriptions agree with the pinned native
contract and the reproduced empty/null cases. Check the affected links and any
added example. No production-code change or new runtime guarantee is required.
Resolution (2026-11-18): the configuration reference now distinguishes the final
resolved selector list from individual source contributions. It explains
omission, empty lists, nulls, and explicit replacement under the effective
collection policy, qualifies overlay null handling, and links to the pinned
upstream policy-lifetime contract. Runtime behavior and CLI whole-list overrides
are unchanged. All 15 temporary selector-policy checks passed against a fresh
build: omitted/empty/null/explicit-replace overlays under ordinary replacement,
append, and upsert, plus CLI whole-list replacement and no-generation rendering
under each policy. The pinned document and anchor and diff whitespace were
checked. No provider was contacted; no production code or tests changed.
### Independent Review Verification
- `go test -count=1 ./...`, `go test -race -count=1 ./...`, and `go vet ./...`
passed with Go 1.27.4 and a provisioned offline cache.
- A native binary and Linux amd64/arm64 binaries built outside the worktree;
cross-builds used the existing cgo-disabled, trimpath, stripped settings.
- All six maintained configuration/overlay selections passed application
validate/show/sources/explain, framework validate/sources, and JSON render.
Framework show/explain also passed. Selected credential variables were unset;
no generation command or live provider was invoked.
- The selected module and executable dependency graphs, `go mod tidy -diff`,
changed-Go formatting, diff whitespace, affected local documentation links,
workflow YAML/shell syntax, both ConfigKit notice revisions, and shared
LGPL/GPL text fidelity were checked. No workspace or vendor tree is committed.
- The only repository change made by this review is this remediation record.
Historical findings below are retained unchanged.
## PromptKit v0.12.0 Review
Reviewed on 2026-10-04 at
`58cb02f9732b09740ff7e22096d2a4072ddfddca`, covering the thirteen implementation
commits after `b1bb55a`. The runtime architecture matches the accepted
[feature roadmap](promptkit.md) and [implementation plan](implementation.md).
No runtime regression, missing framework capability, or newly imposed runtime
restriction was identified. Three low-priority documentation/help gaps were
identified; their status and resolutions are recorded below. The first retained
a loading guarantee that was incompatible with
the deliberately selected full-source engine construction.
### R3: Correct The Definition-Inspection Loading Contract
Priority: P3 (low). Status: Complete.
Locations: `docs/cli.md:208–224`; related orientation in
`docs/internal/adapters.md:49–56`. The CLI wording was retained in `b913ef0`
despite the engine-wiring change in `0df2aad`.
The profile-inspection reference still says it never loads a prompt. Both
definition-inspection commands now construct the same complete selected engine
as execution. PromptKit resolves and validates all final definitions and
captures their message content before the selected inspection runs. Consequently,
an unused prompt with invalid syntax or an unreadable `content_file` can prevent
inspection of an otherwise valid profile. Prompt inspection's unqualified
claim that it does not resolve a profile also needs to distinguish engine
assembly from request-specific target resolution.
Evidence: inspecting a valid profile in a selected combined file with an
unused prompt referring to a missing message file exits 1 before emitting a
profile report. The existing
`TestProfileInspectionUsesCombinedSourcesAndValidatesUnusedPrompts` also
protects this intentional eager-validation boundary.
Required change: explain that both commands construct and validate the full
selected configuration, including unused definitions and message resources,
before inspecting the chosen definition. Preserve the distinction that they
do not prepare an execution request, load request-time schemas/artifacts,
resolve credential values, reserve capacity, or call a provider. Clarify that
no prompt collection is mandatory, but selected prompts are not ignored.
Align the internal summary with the CLI contract rather than restating a
stronger independence guarantee. Do not restore the empty-prompt-filesystem
workaround or weaken eager validation to satisfy the stale wording.
Acceptance: the owning contract and internal summary agree with the roadmap,
shared engine construction, and existing tests. Review the documentation diff
and links; no production-code change or new runtime guarantee is needed.
Resolution: the CLI contract now distinguishes full-source engine construction
from inspecting a selected definition, explains eager validation/content capture
and unused-definition failures, and retains the request-time exclusions. It
states that no prompt collection is mandatory but selected prompts are not
ignored. The internal summary links to that contract without promising a
separate no-prompt-loading path. Existing definition-inspection tests and
documentation link/diff checks passed; no production code changed.
### R4: Update The Composed-Example Description
Priority: P3 (low). Status: Complete.
Location: `docs/config.md:95–96`, after the example migration in `2027b80`.
The reference says `examples/config.composed.yml` imports two application
fragments. It now imports only `config-fragments/sources.yml`; the former
backend fragment became `examples/framework/backends.yml`, selected through
`promptkit_sources` rather than imported into the application document.
The example and code are correct, but the explanation confuses the two source
boundaries introduced by this migration.
Required change: describe the actual source-selection fragment and separately
selected framework backend file, with links where useful. Do not reintroduce
an application backend mapping or add an unnecessary fragment merely to match
the old prose. The old backend fragment had no application defaults to preserve.
Acceptance: the description matches the maintained files and the existing
composed-overlay render command still succeeds offline.
Resolution: the configuration reference now links to the one imported
application source-selection fragment and the separately selected framework
backend file, retaining the local-overlay example. The existing composed-overlay
render command passed offline with credential environment variables unset;
the linked files and diff whitespace were checked. No configuration files or
loading behavior changed.
### R5: Include Framework Configuration Inspection In Built-In Usage
Priority: P3 (low). Status: Complete.
Location: `internal/adapter/cli/run.go:742–749` (`printUsage`), following the
new command-family implementation in `2ba2ce1` and `411a67b`.
The top-level usage lists `inspect prompt`, `inspect profile`, and application
`config`, but omits the implemented `inspect configuration` family entirely.
Invoking the binary without arguments reproduces this omission, so users
relying on its built-in command summary cannot discover the new framework
validation/provenance interface or distinguish it from application inspection.
Required change: add a concise usage line identifying all four operations,
the supported selection flags, and the `--path` requirement for `explain`.
Preserve existing exit behavior and command parsing; this does not require a
new help subsystem or aliases. Protect discoverability with a small CLI-entry
assertion for the family/operations, not a snapshot of the entire usage text
or its incidental ordering.
Acceptance: built-in usage advertises the actual family and remains consistent
with `docs/cli.md`; the focused CLI check and development baseline pass.
Resolution: built-in usage now advertises all four framework operations, the
supported selection flags, and that `--path` is required for `explain`.
`TestUsageAdvertisesFrameworkConfigurationInspection` exercises the CLI entry
point and checks the family, operations, flags, exit status, and output stream
without snapshotting the whole usage text or fixing operation order. It failed
before the usage change and passed afterward. Full tests, race tests, static
checks, a native build outside the repository, formatting, and diff checks
passed; command parsing and exit behavior are unchanged.
### Remediation Completion
R3, R4, and R5 are complete. R1's earlier resolution was verified in the current
HTTP timeout test: assertions concern deadlines and cancellation, not context
identity. R2 remains superseded; the current plan and feature roadmap accurately
record completed implementation and separate release preparation. No active
finding remains, and no historical plan or release was rewritten.
### Review Verification
- `go test -count=1 ./...`, `go test -race -count=1 ./...`, and
`go vet ./...` passed. A native binary built outside the repository.
- Changed Go files are formatted; diff whitespace checks passed.
- All four maintained application configurations passed both application and
framework validation with provider credentials unset. Both render scripts,
composed-overlay render, and inheritance render passed offline.
- All four captured framework inspection operations passed against the complete
example. Focused tests cover native layering/imports/resource capture,
literal IDs, presence/null inheritance, partial-profile completion, scoped
effective/history redaction, and unchanged existing inspection behavior.
- The selected graph is PromptKit v0.12.0, ConfigKit v0.5.0, and the two `/v3`
catalogs at v3.20260930.0, without a local replacement or workspace. Selected
PromptKit contract and notice links resolved.
- The broader redaction in the new framework-report family, removed legacy
source fields/flags, and eager definition capture are explicitly approved
changes, not accidental strengthening. Shared server capacity, request
independence, HTTP wire behavior, and lexical artifact containment remain
intact. No implementation fixes were made during this review.
## ConfigKit v0.5.0 Review
Reviewed on 2026-10-03 at `bcf2ddf`, covering the five implementation
commits after `e1ce526`. The ConfigKit adoption and runtime ownership
boundaries match the accepted roadmap. No runtime defect, missing application
feature, or unrequested strengthening of contracts was identified. One
documentation handoff issue was identified; its current status is recorded below.
### R2: Correct The Completed Plan's Introductory Status
Priority: P3 (low). Status: Superseded; no action against the current plan.
Location: `docs/roadmap/implementation.md:5–9`.
The introduction still says "All five stages are pending" and directs the
reader to implement Stage 1 through Stage 5. Each stage is actually marked
Complete with completion evidence, and the feature roadmap identifies the
adoption as delivered. These contradictory directions leave the overall
handoff inaccurate and could lead a later coding agent to repeat completed
work.
This is a completion-documentation issue, not an unfinished runtime feature.
The [documentation policy](../policy/documentation.md) assigns implementation
status to roadmaps; the overall status should agree with the stage evidence.
Required change: update the introduction to state that all five stages are
complete. Describe the retained steps as implementation history and acceptance
evidence rather than instructions to start the work again. Preserve the
original planning baseline, completed stage records, deferred scope, and
separate release handoff. Do not change application code or rerun the feature
implementation to repair the wording.
Acceptance: the overall status, individual stages, and feature roadmap agree
that implementation is complete, while release preparation remains separate.
Review the documentation diff and run `git diff --check`; this correction
does not require new tests.
The ConfigKit plan was replaced by the PromptKit plan in `b1bb55a`. Its
historical snapshot remains in Git, while the current introduction correctly
states that all thirteen PromptKit stages are complete. This old finding does
not direct changes to the new plan or repetition of either implementation.
### Review Verification
- `go test -race -count=1 ./...` passed, including the new configuration,
inspection, and engine-adoption tests and existing shared-capacity coverage.
- `go vet ./...`, a native build outside the repository, and diff checks passed.
- All five maintained inheritance example commands succeeded with
`LOCAL_GPU_API_KEY` absent: validate, show, sources, explain, and render.
The child selected its own backend ID with the inherited endpoint, zero queue
capacity and false parameter remained explicit, and explanation retained
the parent/destination/directive relationship.
- Changed upstream documentation and notice links, affected local paths,
and the historical policy-plan reference resolved.
- The work set changes dependency pins, tests, examples, and documentation;
no production Go implementation was changed. ConfigKit v0.5.0 owns inheritance
through the existing single-load integration; Promptkit remains v0.11.2.
- The `extends` reservation is the explicitly accepted upstream compatibility
change, not a new Scriptorium restriction. Parameter and credential-reference
visibility, working-directory-relative paths, ordinary parent registration,
and existing shared server capacity remain as selected.
## Earlier Promptkit Upgrade Review (Resolved)
### Scope And Status
Review baseline: `e177d7c`, covering implementation commits after `5c3b0cf`.
This document records a resolved test-contract issue; it does not authorize
unrelated implementation changes. No runtime defect or missing planned feature
was identified in this review. Release preparation remains separate work.
### R1: Remove The Unrequested Context-Identity Invariant
Priority: P3 (low). Status: Complete.
Original location: `internal/adapter/http/handler_test.go:334–349`,
`TestHandlerRunTimeoutValidationAndContextMapping`, introduced in `7de2b02`.
The test's `child` field and `(ctx != parent) != tc.child` assertion required
the exact parent context object for omitted/null/zero timeouts and a distinct
object for a positive timeout. The requested contract concerns effective
deadlines and cancellation propagation, not Go interface identity. In
particular, a harmless derived context that preserves the parent's deadline,
values, and cancellation would fail the omission cases despite satisfying the
application contract.
This elevated an incidental implementation choice into a test-enforced
invariant, contrary to the [agent policy](../policy/agents.md) and
[testing policy](../policy/testing.md). The current handler's context behavior
is correct; this is a maintainability issue, not a runtime cancellation bug.
#### Required Change
Remove the identity comparison and the identity-only table field. Retain
behavioral assertions that an earlier parent deadline is not extended and
parent cancellation reaches execution. For omitted/null/zero values, verify
that no deadline is introduced when the parent has none. Retain coverage of
positive deadlines and the distinction between a canceled execution context
and a provider-attempt timeout while the execution context remains live.
Use the existing runner seam and focused cases; do not introduce a context
wrapper framework, timer abstraction, new public guarantee, or production
refactor solely to change this test.
#### Completion Criteria
- Tests protect effective deadline/cancellation behavior without requiring
context object identity.
- A semantically equivalent derived context does not fail merely because it
is a different object.
- Existing timeout validation and HTTP outcome coverage remains intact.
#### Resolution And Validation
Removed the identity-only table field and context comparisons. The test now
checks effective deadlines, including omitted/null/zero requests with no
parent deadline, and explicitly checks that parent cancellation reaches the
runner. Existing positive-deadline integration tests and live-context
provider-timeout mapping remain intact. No production code changed.
Focused HTTP timeout tests, `go test ./...`, `go test -race ./...`,
`go vet ./...`, a native build outside the repository, and `git diff --check`
all passed after remediation.
### Original Review Verification
- `go test -race -count=1 ./...` passed, including retry, shared-capacity,
deadline, report, definition-conformance, and architecture-boundary tests.
- `go vet ./...` passed.
- A native `cmd/scriptorium` build passed, with output outside the repository.
- The selected dependency is Promptkit v0.11.0 with its intended v2 catalogs.
- Application code, tests, release automation, and historical releases were
not changed during this review.
The runtime ownership boundaries remain as intended: Promptkit owns framework
formats, execution, retries, and backend admission; Scriptorium owns executable
adapters, application settings, HTTP artifact policy, deadlines, and terminal
report presentation. No additional runtime restrictions, durable workflow
state, or extra release compliance gates were identified in the reviewed diff.