Files
promptkit/docs/roadmap/remediation.md

15 KiB
Raw Blame History

Configuration Handoff And Repair Review

Reviewed implementation stages 1–17 against the implementation plan, the feature roadmap, and accepted ADRs 0001 and 0002. Implementation baseline: 40f8a6825359b4923645256e230a050dfdb00710; reviewed changes begin after 248d0db.

Findings

1. Narrow The Pre-Handoff Mutation Wording To The Accepted Expectation

Status: complete. Priority: P3. Implementation stage: 16.

Completion: narrowed constructor GoDoc and consumer guidance to the consistency expectation; reviewed against the roadmap. No runtime behavior changed.

Locations: engine.go:280 and docs/consumers/pkg-promptkit.md:132.

NewFromConfig says callers must not mutate selected settings between loading and handoff; the consumer guide repeats that blanket instruction. The accepted roadmap instead asks callers to hand off before changes that invalidate the frozen inspection/provenance evidence, and expressly treats this as a usage expectation rather than a new enforcement mechanism. The published wording prohibits even value-preserving reassignment or copying, which needs no such restriction.

Required change: describe the narrower consistency expectation in both owners. Explain that construction consumes the resolved settings while inspection retains the completed load's frozen evidence, so callers should keep those consistent at handoff. Preserve the existing protection against mutation after construction. No mutation detection, extra validation pass, or new error is needed.

Verification: review both passages against the feature roadmap's ownership paragraph; ensure they no longer impose an unconditional mutation prohibition. This is a documentation correction and does not require a new runtime test.

2. Keep Application-Field Sensitivity Policy With The Application

Status: complete. Priority: P3. Implementation stage: 16.

Completion: backend and profile guidance now assigns extension sensitivity and disclosure to applications, retaining the credential/redaction warning. Reviewed against the accepted extension contract; no runtime changes were needed.

Location: docs/formats.md:235–236.

The backend format reference says an unrecognized backend field is application data and must not contain secrets. The accepted extension-point contract assigns interpretation, validation, and sensitivity rules for those fields to the application. The implementation follows that contract: inline fields remain in native inspection, application sensitivity predicates compose with PromptKit's, and those fields are omitted from resource reports and provider parameters. The added prohibition unnecessarily narrows the application-owned data contract.

Required change: retain the useful warning that a raw api_key extension is not a PromptKit credential input and is not automatically redacted by PromptKit. State that the application owns sensitivity declarations and disclosure policy for its extension fields, rather than prohibiting their contents. Review the adjacent profile wording for the same distinction between PromptKit credentials and application-owned data. Do not add automatic credential behavior or a new validator for application fields.

Verification: compare the revised text with the feature roadmap's application-specific-fields contract and existing focused-inspection sensitivity coverage. No change to execution or redaction implementation is needed.

3. Preserve Effective Input Metadata Supplied By Application Defaults

Status: open. Priority: P2. Implementation stages: 5–6 and 13.

Locations: internal/configuration/load.go:208–216, internal/configuration/prompts.go:174–180, and internal/configuration/resources.go:81–87.

The new application-owned loading path preserves ConfigKit Defaults and application resolution callbacks, but the existing input-inspection machinery still equates FieldInfo.Explicit == false with an absent effective value. finalFieldState consequently discards metadata for values supplied by Go defaults or recorded derived defaults. These are valid final values, not missing configuration.

Reproduced through the public API with a completed ConfigKit load whose Definition.Defaults returns a prompt input equivalent to:

promptkit.PromptInputSettings{
    Name: "doc", Required: true,
    ContentType: new("text/plain"), Description: new("default description"),
}

With a valid prompt and default profile, NewFromConfig succeeds. The final typed input retains those values; native metadata describes their defaulted, nonnull presence (Explicit: false, Null: false, default layer), but:

  • InspectPrompt returns Required: nil instead of the effective true value.
  • InspectRequirements correctly classifies doc as required but returns nil content type and description, contrary to its effective-metadata contract.
  • ValidateRequest without doc fails with ErrPromptRender and a missing required-input error, confirming that execution uses the true value that resource inspection fails to report.

A second public-API reproduction using an application Resolve callback and RecordDerivedDefault for an omitted required and content_type has the same result. Source-loaded inputs ordinarily have explicit metadata, which is why the migrated source fixtures and current handoff tests do not expose this gap.

Required change: capture effective input metadata from the final typed values and native final metadata without treating source explicitness as a universal presence test. Preserve the existing ordinary omission/null and explicit false/empty-string behavior. Keep source provenance in native inspection; do not inspect raw sources, replay history, reject application defaults, or rerun resolution to repair the report.

Verification: add public integration regressions for both Go defaults and recorded application-derived defaults. Compare InspectPrompt, InspectRequirements, and preparation/preflight for the same captured input, and retain existing source-presence cases. Include a collection case if the fix introduces separate root/collection handling; the current capture logic is shared.

4. Avoid Expanding Inheritance Provenance For Input-Presence Flags

Status: open for consideration. Priority: P3. Implementation stage: 6. Related to finding 3; consider addressing them together.

Locations: internal/configuration/load.go:173–178 and 208–216.

Capture calls Metadata.Lookup three times per declared input, then retains only the Explicit and Null flags. In the selected ConfigKit v1.4.0 implementation, Lookup also materializes and copies the field's complete inheritance-step list. ConfigKit keeps that evidence compact internally; the full projection is useful for operator inspection, but unnecessary for these flags. Its public Metadata.IsExplicit method performs the effective-field query without that projection.

Complexity: for I input declarations and inheritance depths h_i, these presence queries perform O(I + sum(h_i)) provenance work, apart from path processing. A chain of N prompts inheriting a constant-sized input declaration has depths 0..N-1, producing quadratic ancestry-step work and cumulative temporary allocations merely to capture linear-sized presence data. Reading one metadata snapshot, as selected in the plan, does not eliminate this per-query expansion.

Refactoring opportunity: combine effective typed values/pointer presence with Metadata.IsExplicit where source explicitness is actually needed. Nullable content-type and description pointers already distinguish retained values from absent/null metadata for their public reports. Avoid requesting a complete origin solely to classify presence, while correcting the default-value handling in finding 3. Presence classification can be linear in the number of declarations, plus ordinary path-query costs, independent of inheritance-chain length.

Keep full Lookup operations where source origins are genuinely needed for resource binding. This finding does not propose a different inheritance algorithm, metadata snapshot policy, inspection contract, or source-origin fallback.

Verification: preserve inherited, omitted/null, explicit-false, empty-string, and application-default metadata outcomes. Confirm from the call path that the presence-only phase no longer projects inheritance steps. No timing threshold or real-world benchmark is needed to establish that algorithmic change.

5. Share Immutable Backend Defaults Across Profiles Without Overrides

Status: open for consideration. Priority: P3. Implementation stage: 6.

Locations: internal/configuration/targets.go:27–29 and 68–78, internal/backend/registry.go:39–48, and internal/domain/smoke_diagnostics.go:124–128.

Every captured profile selecting a backend calls the registry's defensive-copy lookup. That lookup deep-copies the backend's provider parameters, and the profile retains the copied map even when it has no provider-parameter overrides. Many profiles selecting one backend therefore retain many identical immutable trees. This is an inherited cost reused by the new capture path, not a newly introduced regression. The plan selects precomputed execution profiles and ownership boundaries, not duplication of identical defaults within the owned snapshot.

Complexity: for P profiles sharing one backend with S provider-value nodes and no profile-level provider maps, this step costs Theta(P*S) copying time and retained parameter storage, despite only O(P+S) distinct input data. The cost is quadratic when both dimensions grow together. Existing ID sorting is additional work and is not the target of this finding.

Refactoring opportunity: obtain an owned backend-default snapshot once per backend and share its provider map read-only among profiles without overrides. Keep separate merged maps when a profile needs its own provider parameters. For the no-override case, default-map copying and storage can become O(P+S); the existing sorting and public result-copy costs remain unchanged. A small construction-local cache or use of already owned immutable definitions is sufficient; a general persistent-map framework is unnecessary.

Preserve defensive GetBackend semantics and copies at runtime/caller/client boundaries. GetProfile already detaches the selected target's provider map, so sharing construction-owned defaults need not expose shared mutable state to consumers or merge caller requests into retained configuration.

Verification: review copy placement to establish the reduction, and retain backend/profile mutation isolation, override-merge, prepared-state ownership, and injected-client mutation coverage. Do not make internal pointer sharing a public guarantee or introduce benchmark thresholds.

Initial Review Results And Validation

  • Stage 1: ConfigKit v1.4.0, unchanged Go minimum, and commit-pinned attribution match the selected module and its notices.
  • Stages 2–3: configuration and runtime inputs resolve mode-dependent repair budgets; explicit zero and null/omission, whole-contract replacement, prepared retention, and existing repair accounting are covered.
  • Stages 4–9: shared settings aliases, composed stateless rules, final-metadata resource capture, native focused inspection, explicit source mounting, and shared constructor/assembly paths match the target architecture.
  • Stages 10–15: migrated tests and maintained offline examples exercise the changed boundaries, including concurrent loads, cross-filesystem inheritance, application extensions, ownership, and standalone/handoff equivalence.
  • Stages 16–17: canonical documentation is updated, subject to the two wording corrections above. No additional functional or architectural defect was found.

Fresh checks passed with Go 1.27.4, GOTOOLCHAIN=local, and GOWORK=off: go test -race ./... and go vet ./.... Changed Go files have no gofmt differences, and the implementation diff passes git diff --check.

This review changes only this findings document; it does not implement fixes.

Integration-Focused Follow-Up

The second review revisited stages 1–17 at their interaction boundaries: the ConfigKit upgrade and native metadata semantics; repair-input conversion and prepared budgets; shared settings and application callbacks; capture, inspection, mounting, and constructor wiring; migrated tests and examples; and their contracts with the existing runtime. Finding 3 records the newly reproduced integration gap. Findings 1 and 2 remain unchanged.

Additional disposable external-package checks, outside the repository, passed under the race detector for both NewEngine and NewFromConfig. They combined configured default repair with continuation, appended messages, direct session selection, namespace views, frozen prepared execution, structured output, numeric-presence settings, usage/observer accounting, capacity release, and single-pass smoke diagnostics. Both default-metadata reproductions failed their expected inspection assertions, including under the race detector; those failures are the evidence for finding 3, not failures in the existing test suite.

Fresh targeted race-enabled tests passed for the public handoff and runtime contracts, configuration and use-case packages, maintained examples, capacity, generation, observation, rendering, and validation. No other integration defect was found. The prior full-suite and vet results remain applicable to unchanged implementation code. No repository implementation or test files were changed by this review.

Algorithmic Review

The third review considered stages 1–17 in order, excluding algorithm choices expressly selected by the plan. Findings 4 and 5 record two local optimization opportunities; findings 1–3 remain unchanged. No additional ordinary correctness bug was found.

  • Stages 1–3: checked the selected metadata API costs, constant-time budget defaulting/conversion, and the unchanged bounded repair/preparation paths.
  • Stages 4–6: checked shared data declarations, per-resource normalization and validation, known-value ownership traversal, presence capture, and profile construction. Nested resource/input loops traverse distinct data; the two findings concern hidden work in helper calls rather than loop syntax.
  • Stages 7–9: native reporting remains delegated; diagnostic copying is linear in returned records/locations, and the constructors add no repeated load or configuration-sized work to ordinary execution.
  • Stages 10–15: checked migrated test and example helpers for additional passes or algorithm changes; they do not introduce another production loader or runtime resource scan.
  • Stages 16–17: checked the findings against the accepted ownership, inspection, ordering, and configuration-boundary decisions. Deliberate lexical inventories, native ConfigKit algorithms, and full-prompt repair remain outside reconsideration.

This was static complexity analysis of PromptKit and the relevant calls into the selected ConfigKit release. No performance measurements or benchmarks were run. Implementation code is unchanged, so prior functional validation remains applicable; this pass validates the findings-document diff only.