promptkit/docs/roadmap/implementation.md

# Recorded Field Normalization Implementation Plan

Implement [field-normalization.md](field-normalization.md) in the sequential
stages below. Each stage is sized for one Luna coding-agent prompt. The feature
roadmap owns intent and target behavior; this plan owns implementation steps.

## Baseline And Sprint Rules

The baseline is `fb7ebe6`: ConfigKit v1.0.0, Go 1.27.0, six registered resource
list identities, and execution-owned interpretations in `internal/configuration`.
`configurationDefinition` captures presence in `Resolve`, then populates
`state.interpreted` in `Validate`. `load.go`, `targets.go`, and `prompts.go`
consume that second configuration. Message roles/cache controls are normalized
in `materializePrompt`.

Read `docs/development.md` and the applicable agent, architecture, testing,
documentation, and OSS policies. Follow ADR
[0001](../adr/0001-configuration-boundary.md). Use ConfigKit's released public
API and documentation, including its v1.1.0 resolution-recording rules.

This is one end-to-end sprint. Intermediate stages need not compile or be
coherent; do not add compatibility scaffolding, human approval gates, or external
publication dependencies to make them independently deployable. Keep dependency
adoption in a focused commit for diagnosis, not as a review gate. Run checks
when their dependencies are present and record incomplete verification honestly.
Reuse successful checks for unchanged content. Do not publish a release.

## Implementation Decisions

- Keep the existing identity registrations, configuration specification version,
  public APIs, and normalization rules. Do not normalize resource IDs, namespaces,
  references, model names, content, or `content_file` paths.
- Capture existing backend/profile field states and prompt-input metadata from
  `Resolution.EntryInfo` before any assignment. Preserve omission, inherited
  explicitness, null, empty strings, false, and zero independently of normalized
  values. Do not retain callback contexts beyond their lifetime.
- Perform changed-field assignments in `Resolve` and record exact final-index
  paths. Compare old/new scalar values before recording; preserve nil pointers
  and absent optional containers. Use concise value-free reasons. A small typed
  helper for optional strings is appropriate; a reflective traversal framework
  or a second configuration tree is not needed.
- Resource lists inside `Collections` need map-entry copy/writeback. Modify the
  actual slices/pointers and write collection values back; helper targets and
  recorded paths must agree with the final typed document.
- Visit records and nested fields directly. Build a uniqueness set once per
  resource list for the eligibility check below; avoid a metadata snapshot or
  full-resource search per scalar. Retain the existing lazy shared origin lookup
  for content materialization.
- Before changing a registered list, check its final typed IDs once using the
  existing local-ID predicate and an exact-ID uniqueness set. If any ID is invalid
  or duplicated, leave that entire resource list unmodified by normalization and
  defaulting. ConfigKit's final identity validation and PromptKit's existing
  domain validation then reject it at their ordinary boundaries. This is an
  internal mutation-eligibility check on final values, not source prevalidation,
  identity repair, or a new acceptance rule. Test the invalid-ID-plus-padding
  combination so it cannot become an unrecorded-list-mutation error.
- Successful endpoint normalization is assigned and recorded. If normalization
  fails, leave the original field unchanged; pure semantic validation reports
  the endpoint error using the existing validation boundary. Do not use a generic
  returned resolver error that hides the useful field diagnostic. Shared domain
  functions may be used for pure checks without assigning in validation.
- Queue defaulting remains `concurrency_limit != nil && > 0 && queue_capacity ==
  nil`. Assign 1024 and call `RecordDerivedDefault` at `queue_capacity`, with
  `concurrency_limit` as dependency. Do not substitute `Default`: explicit null
  currently receives this default too. Explicit zero remains zero.
- `Validate` checks the resolved document without mutation. Domain construction,
  prompt compilation, and file loading remain at their current appropriate
  boundaries. Preserve public error identities where applicable, not exact text,
  diagnostic ordering, or callback timing.
- Retain request-time normalization, backend-to-profile completion, runtime-owned
  copies, and detached public results. They serve different purposes from the
  duplicate interpreted configuration being removed.

## Stage 1: Adopt Released ConfigKit v1.1.0

Scope: module metadata and applicable attribution only.

1. Update `go.maximumdirect.net/configkit` to v1.1.0 using the compatible local
   Go installation. Update sums without unrelated dependency upgrades or a
   committed filesystem replacement. Keep Go 1.27.0 unless the actual dependency
   metadata requires a change; no increase is planned.
2. Review the selected release's license and notices. Update its House Format
   entry in `docs/oss/attribution.md`, including version and commit-pinned links.
   The local upstream v1.1.0 tag currently resolves to
   `191015e289d7f2b91abb9fdd9c4f2f5cf75b4de9`; verify against the selected source.
3. Run `go test ./...` with caches prepared. Resolve only actual compatibility
   failures; do not start the normalization refactor in this dependency unit.

Acceptance: module and notices select the release, the existing suite works,
and no local replacement or unrelated upgrade was introduced.

Completion: ConfigKit v1.1.0 and its commit-pinned notices are selected; the
complete Go test suite passed with Go 1.27.0.

## Stage 2: Record Backend And Profile Transformations

Scope: `internal/configuration/load.go`, focused normalization helpers if useful,
and backend/profile loading tests.

1. Keep presence capture first in `Resolve`. Add the final-list eligibility check
   and in-place traversal for root and collection backends/profiles.
2. Replace detached backend interpretation with field records for base endpoint,
   API-key environment name, smoke reachability/authentication endpoints, and
   authentication kind. Preserve existing endpoint helper behavior, including
   diagnostic endpoints' distinct rules. Authentication kind trims/lowercases;
   environment names only trim.
3. Record the conditional queue assignment exactly as specified above. Do not
   manufacture absent smoke-test containers or fields.
4. Normalize profiles' nonempty endpoint overrides and supplied service-tier,
   reasoning-effort, and API-key environment strings. An exactly empty endpoint
   override remains distinct from whitespace which fails endpoint normalization.
5. Move endpoint failure checks into pure validation on effective fields.
   Temporarily remaining interpretation helpers must not repeat mutation or
   maintain a second backend/profile normalization policy.
6. Adapt `TestLoadInterpretsBackendNormalizationAndNullDefault` and the target
   case in `interpretation_test.go`: inspection now contains normalized values
   and defaulted capacity, while execution settings remain as before. Add exact
   field-operation assertions and one unchanged-field/no-new-event assertion.

Validation: focused `go test ./internal/configuration`, once the intermediate
callback wiring supports it. Cover absent/null/zero capacity and missing/zero/
positive concurrency, optional endpoint overrides, and invalid endpoints.

## Stage 3: Record Prompt Metadata And Configured Messages

Scope: prompt resolution helpers, `prompts.go`, and prompt loading tests.

1. Traverse root/collection prompt lists with the same eligibility handling.
   Move description/session-template trimming, input-name trimming, optional
   input content-type/description trimming, and optional output schema-path
   trimming from `interpretPrompts` into exact field records.
2. For configured messages, reuse `domain.NormalizeMessageRole` and
   `domain.NormalizeCacheControl`. Write back only changed supplied scalar
   fields at `messages/<index>/role` or `cache_control/type` and `cache_control/ttl`.
   Preserve required-field errors for absent role/type. Do not allocate an
   omitted cache control or TTL, lowercase cache-control types, trim content,
   or record an entire message/list merely to cover a scalar change.
3. If a message normalizer rejects a value, leave that field/group unchanged and
   let existing message validation/construction reject it with useful context.
   Keep shared domain checks where needed for validation; remove redundant
   configuration-specific assignment logic from materialization. Caller-appended
   messages continue using their runtime normalizers.
4. Keep content-file opening, template compilation, output-contract construction,
   duplicate normalized input-name checks, and hash construction in their existing
   owning paths. A trimmed schema path still addresses the selected schema source.
5. Adapt prompt interpretation tests to the new inspection view. Add focused
   cases for padded roles, trim-only cache-control spelling, optional/null input
   metadata, and normalized input-name collisions. Compare padded and canonical
   definitions with the same identities for equivalent materialized definitions
   and hashes; do not freeze a new hash encoding.

Validation: prompt/configuration tests plus existing domain-message tests. Invalid
roles, cache-control values, and missing required message fields still fail;
explicit empty inline content retains its existing behavior.

## Stage 4: Use The Single Resolved Configuration

Scope: loader/target/materialization wiring and affected tests.

1. Change `Validate` to validate its supplied resolved document directly.
   Remove `loaded.interpreted`, `interpretConfiguration`, and obsolete detached
   interpretation helpers/copies after their transformations have moved.
2. Route backend inventory/index construction, execution-profile construction,
   and prompt materialization through `state.result.Config`. Keep the captured
   entry field-state maps and prompt metadata needed for runtime presence/null
   semantics. Do not derive those states from newly defaulted pointer values.
3. Preserve domain/runtime copying and public detachment. Keep the existing
   source-origin lookup for `content_file`; none of the selected transformations
   changes that field. No history scan or origin fallback is needed.
4. Remove or rename tests whose old assertion was preservation of authored
   whitespace in configuration inspection. Retain their meaningful execution,
   ownership, and rejection assertions rather than weakening coverage wholesale.
5. Check existing partial-profile completion, inherited endpoint/key settings,
   prepared execution, requirement metadata, and scoped inventories through the
   new single-configuration path.

Validation: `go test ./internal/configuration ./internal/engine` and the affected
root API tests. Search for remaining interpreted-state consumers and confirm
validation does not mutate the working configuration.

## Stage 5: Verify Provenance And Inspection Integration

Scope: focused configuration tests and existing public inspection/diff tests.

1. Add representative root and collection cases covering field normalization
   operations and queue-derived-default dependency evidence. Check operation,
   path, effective value, and relevant retained source evidence without freezing
   whole histories, reason wording, source ordering, or upstream internals.
2. Extend the inherited/imported content fixtures so a message role/cache-control
   edit occurs beside a `content_file` originating in another source directory
   or filesystem. Include later retained fields from a different contributor.
   Assert the expected file's captured content and unchanged effective origin;
   do not substitute a common directory that hides wrong-origin resolution.
3. Combine invalid/duplicate final resource identities with fields needing
   normalization/defaulting. Assert useful rejection and absence of an accidental
   unrecorded-mutation failure. Retain existing tests that replaced lower-layer
   invalid values are not independently validated.
4. Verify configuration diffs omit a value change when two otherwise identical
   definitions normalize to the same value, while genuinely changed list values
   still use ConfigKit's atomic list diff. Exercise the public passthrough rather
   than implementing PromptKit-specific diff equivalence.
5. Extend existing sensitive-provider inspection coverage to normalized records
   and explanations as relevant. Reasons must not interpolate configuration
   values. Keep exact input metadata presence distinctions and public detachment.
6. Reuse existing runtime/appended-message tests; add only missing boundary cases.
   Do not reproduce ConfigKit's generic identity-correspondence suite, add live
   provider tests, or introduce performance thresholds.

Validation: focused configuration, public inspection/diff, preparation, and
execution tests. All new tests use deterministic offline sources/collaborators.

## Stage 6: Align Documentation And Verify The Sprint

Scope: current contract/implementation documentation and final integration.

1. Update `docs/formats.md`, `docs/consumers/pkg-promptkit.md`,
   `docs/consumers/configuration-migration.md`, and `docs/internal/sources.md`
   where they describe authored/composed values excluding PromptKit normalization.
   Explain recorded effective normalization and defaults, retained provenance,
   normalized-value diff behavior, and the boundary with request completion.
2. Review affected public GoDoc and architecture wording against the new callback
   flow. Change only stale claims; no new public API, YAML version, normalizer
   policy, origin guarantee, or execution-readiness promise is needed.
3. Document the inspection/dependency change and its limits in the next release's
   `docs/releases/v0.18.0.md`, creating the note if needed under the release
   procedure. Keep canonical detail in the format/inspection guides. v0.17.0
   has been published: preserve its note, including its historical two-view
   normalization description. Do not publish, retag, or edit published notes
   for this work.
4. Format changed Go files and run the integrated checks with the compatible
   local toolchain and prepared caches:

   ```sh
   go test -race -vet=off ./...
   go vet ./...
   go run ./examples/go-library/prepare
   go run ./examples/go-library/run
   go run ./examples/go-library/observe
   git diff --check
   ```

   The examples' successful output semantics should remain unchanged. No release
   launcher/profile change is planned, so the separate Python release-wiring
   diagnostic need not be rerun merely for this refactor.
5. Review the full diff for presence loss, mutation outside resolution, broad
   list records, duplicate configuration state, stale docs, and unintended API
   changes. Record completion in the feature roadmap; retire it and its catalog
   pointer during normal completed-roadmap cleanup. Follow focused commit policy.

Completion means the one resolved configuration drives inspection and runtime
construction; all selected transformations have field-level records; unchanged
resource origins and runtime semantics are preserved; tests and documentation
cover the intentional inspection change. There are no unresolved product
decisions or upstream publication steps required by this plan.