Files
promptkit/docs/roadmap/remediation.md

163 lines
8.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Configuration Handoff And Repair Review
Reviewed implementation stages 1–17 against
[the implementation plan](implementation.md),
[the feature roadmap](next-release-considerations.md), and accepted ADRs
[0001](../adr/0001-configuration-boundary.md) and
[0002](../adr/0002-shared-configuration-data-model.md).
Implementation baseline: `40f8a6825359b4923645256e230a050dfdb00710`;
reviewed changes begin after `248d0db`.
## Findings
### 1. Narrow The Pre-Handoff Mutation Wording To The Accepted Expectation
Status: open. Priority: P3. Implementation stage: 16.
Locations: `engine.go:280` and `docs/consumers/pkg-promptkit.md:132`.
`NewFromConfig` says callers **must not mutate** selected settings between loading
and handoff; the consumer guide repeats that blanket instruction. The accepted
roadmap instead asks callers to hand off before changes that invalidate the
frozen inspection/provenance evidence, and expressly treats this as a usage
expectation rather than a new enforcement mechanism. The published wording
prohibits even value-preserving reassignment or copying, which needs no such
restriction.
Required change: describe the narrower consistency expectation in both owners.
Explain that construction consumes the resolved settings while inspection retains
the completed load's frozen evidence, so callers should keep those consistent at
handoff. Preserve the existing protection against mutation after construction.
No mutation detection, extra validation pass, or new error is needed.
Verification: review both passages against the feature roadmap's ownership
paragraph; ensure they no longer impose an unconditional mutation prohibition.
This is a documentation correction and does not require a new runtime test.
### 2. Keep Application-Field Sensitivity Policy With The Application
Status: open. Priority: P3. Implementation stage: 16.
Location: `docs/formats.md:235–236`.
The backend format reference says an unrecognized backend field is application
data and **must not contain secrets**. The accepted extension-point contract
assigns interpretation, validation, and sensitivity rules for those fields to
the application. The implementation follows that contract: inline fields remain
in native inspection, application sensitivity predicates compose with PromptKit's,
and those fields are omitted from resource reports and provider parameters.
The added prohibition unnecessarily narrows the application-owned data contract.
Required change: retain the useful warning that a raw `api_key` extension is not
a PromptKit credential input and is not automatically redacted by PromptKit.
State that the application owns sensitivity declarations and disclosure policy
for its extension fields, rather than prohibiting their contents. Review the
adjacent profile wording for the same distinction between PromptKit credentials
and application-owned data. Do not add automatic credential behavior or a new
validator for application fields.
Verification: compare the revised text with the feature roadmap's
application-specific-fields contract and existing focused-inspection sensitivity
coverage. No change to execution or redaction implementation is needed.
### 3. Preserve Effective Input Metadata Supplied By Application Defaults
Status: open. Priority: P2. Implementation stages: 5–6 and 13.
Locations: `internal/configuration/load.go:208–216`,
`internal/configuration/prompts.go:174–180`, and
`internal/configuration/resources.go:81–87`.
The new application-owned loading path preserves ConfigKit `Defaults` and
application resolution callbacks, but the existing input-inspection machinery
still equates `FieldInfo.Explicit == false` with an absent effective value.
`finalFieldState` consequently discards metadata for values supplied by Go
defaults or recorded derived defaults. These are valid final values, not missing
configuration.
Reproduced through the public API with a completed ConfigKit load whose
`Definition.Defaults` returns a prompt input equivalent to:
```go
promptkit.PromptInputSettings{
Name: "doc", Required: true,
ContentType: new("text/plain"), Description: new("default description"),
}
```
With a valid prompt and default profile, `NewFromConfig` succeeds. The final
typed input retains those values; native metadata describes their defaulted,
nonnull presence (`Explicit: false`, `Null: false`, default layer), but:
- `InspectPrompt` returns `Required: nil` instead of the effective true value.
- `InspectRequirements` correctly classifies `doc` as required but returns nil
content type and description, contrary to its effective-metadata contract.
- `ValidateRequest` without `doc` fails with `ErrPromptRender` and a missing
required-input error, confirming that execution uses the true value that
resource inspection fails to report.
A second public-API reproduction using an application `Resolve` callback and
`RecordDerivedDefault` for an omitted `required` and `content_type` has the same
result. Source-loaded inputs ordinarily have explicit metadata, which is why
the migrated source fixtures and current handoff tests do not expose this gap.
Required change: capture effective input metadata from the final typed values
and native final metadata without treating source explicitness as a universal
presence test. Preserve the existing ordinary omission/null and explicit
false/empty-string behavior. Keep source provenance in native inspection;
do not inspect raw sources, replay history, reject application defaults, or
rerun resolution to repair the report.
Verification: add public integration regressions for both Go defaults and
recorded application-derived defaults. Compare `InspectPrompt`,
`InspectRequirements`, and preparation/preflight for the same captured input,
and retain existing source-presence cases. Include a collection case if the fix
introduces separate root/collection handling; the current capture logic is shared.
## Initial Review Results And Validation
- Stage 1: ConfigKit v1.4.0, unchanged Go minimum, and commit-pinned attribution
match the selected module and its notices.
- Stages 2–3: configuration and runtime inputs resolve mode-dependent repair
budgets; explicit zero and null/omission, whole-contract replacement, prepared
retention, and existing repair accounting are covered.
- Stages 4–9: shared settings aliases, composed stateless rules, final-metadata
resource capture, native focused inspection, explicit source mounting, and
shared constructor/assembly paths match the target architecture.
- Stages 10–15: migrated tests and maintained offline examples exercise the
changed boundaries, including concurrent loads, cross-filesystem inheritance,
application extensions, ownership, and standalone/handoff equivalence.
- Stages 16–17: canonical documentation is updated, subject to the two wording
corrections above. No additional functional or architectural defect was found.
Fresh checks passed with Go 1.27.4, `GOTOOLCHAIN=local`, and `GOWORK=off`:
`go test -race ./...` and `go vet ./...`. Changed Go files have no `gofmt`
differences, and the implementation diff passes `git diff --check`.
This review changes only this findings document; it does not implement fixes.
## Integration-Focused Follow-Up
The second review revisited stages 1–17 at their interaction boundaries: the
ConfigKit upgrade and native metadata semantics; repair-input conversion and
prepared budgets; shared settings and application callbacks; capture, inspection,
mounting, and constructor wiring; migrated tests and examples; and their contracts
with the existing runtime. Finding 3 records the newly reproduced integration
gap. Findings 1 and 2 remain unchanged.
Additional disposable external-package checks, outside the repository, passed
under the race detector for both `NewEngine` and `NewFromConfig`. They combined
configured default repair with continuation, appended messages, direct session
selection, namespace views, frozen prepared execution, structured output,
numeric-presence settings, usage/observer accounting, capacity release, and
single-pass smoke diagnostics. Both default-metadata reproductions failed their
expected inspection assertions, including under the race detector; those failures
are the evidence for finding 3, not failures in the existing test suite.
Fresh targeted race-enabled tests passed for the public handoff and runtime
contracts, configuration and use-case packages, maintained examples, capacity,
generation, observation, rendering, and validation. No other integration defect
was found. The prior full-suite and vet results remain applicable to unchanged
implementation code. No repository implementation or test files were changed by
this review.