Plan field-level configuration normalization

docs/roadmap/field-normalization.md

@@ -0,0 +1,218 @@
+# Field-Level Configuration Normalization
+
+## Status And Purpose
+
+Planned; implementation has not started. ConfigKit v1.1.0 is released, while
+PromptKit currently depends on v1.0.0.
+
+Move PromptKit's existing configuration-time normalization and derived defaults
+into ConfigKit's recorded application resolution. Configuration inspection should
+then show the normalized values used to construct execution settings, with
+explanations of the changes, while unchanged resource references retain the
+source context needed to read their files.
+
+The benefit is a more useful inspection view and removal of the parallel
+interpreted configuration maintained for execution. This is a focused loading
+change, not a new configuration format, normalization policy, or execution API.
+
+## Current Baseline
+
+PromptKit already uses identified lists for backends, profiles, and prompts at
+the root and within named collections. Six ConfigKit identity registrations
+cover those destinations. This roadmap does not repeat that migration.
+
+The configuration adapter's `Definition.Resolve` currently captures input
+presence information. During validation, `interpretConfiguration` constructs
+detached backend, profile, and prompt interpretations and applies normalization
+and a conditional backend queue default. Runtime construction uses those
+interpretations; configuration inspection retains ConfigKit's composed values.
+Configured message roles and cache-control values are normalized later during
+prompt materialization.
+
+This separation avoided recording whole resource lists as normalized under
+ConfigKit v1.0.0. Such a record would replace the effective origins of unchanged
+fields, including `content_file`, whose origin selects the filesystem and
+directory used to load message text.
+
+The current behavior is documented in the [format reference](../formats.md) and
+[source-loading implementation](../internal/sources.md). The
+[configuration ownership decision](../adr/0001-configuration-boundary.md)
+continues to govern the division of responsibility: ConfigKit owns composition;
+PromptKit interprets effective values and supplies domain rules.
+
+## Released Upstream Capability
+
+[ConfigKit v1.1.0](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.1.0/docs/releases/v1.1.0.md)
+allows individual field records inside lists through the existing `Normalize`,
+`RecordNormalization`, and derived-default APIs:
+
+- Equal-length ordinary lists are compared recursively by position. This covers
+  nested prompt messages and inputs without registering new identities for them.
+- Registered collections with valid, unique entry and final identities support
+  matched field comparison. PromptKit already registers its resource lists.
+- List length changes, and registered identity order or membership changes,
+  still need list or ancestor coverage. Matching is conservative when identities
+  are unsuitable. Final identity validation remains ConfigKit's responsibility.
+- Entry queries retain their fixed resolution-entry meaning; records address
+  final positions. Public effective-configuration diffs remain atomic for lists.
+
+These capabilities support the proposed in-place field transformations. No
+additional ConfigKit API is a prerequisite. Historical source evidence retained
+for a matched record is distinct from its effective origin: a whole-list
+resolution record can still replace descendant effective origins. The migration
+should use individual field records for individual changes.
+
+## Target End State
+
+### One Normalized Configuration For Loading And Inspection
+
+PromptKit's `Definition.Resolve` captures the entry metadata needed for existing
+presence/null semantics, then applies configuration-time transformations to the
+working configuration and records their effects at the affected fields.
+`Definition.Validate` examines the resulting values without mutating them.
+Runtime indexes and prompt materialization consume that same normalized
+configuration, producing the execution-owned domain values they need.
+
+Remove `state.interpreted` and the detached interpretation copies once their
+normalization work has moved. Retain copies needed for runtime ownership and
+public detached results; removing the duplicate configuration is not a reason
+to expose shared mutable runtime state.
+
+PromptKit continues to own normalization rules. ConfigKit records their effects
+and provenance. Composition, inheritance, precedence, and source acquisition
+remain upstream responsibilities; this work adds no source-level validation or
+history-based domain enforcement.
+
+### Transformation Scope
+
+Use the existing transformations as the migration inventory, rather than
+introducing general trimming or case-folding across the configuration:
+
+| Area | Existing behavior to move into recorded resolution |
+| --- | --- |
+| Backends | Normalize base endpoints and diagnostic endpoints; trim API-key environment names; trim and lowercase smoke-test authentication kinds. |
+| Backend capacity | Apply the existing conditional queue-capacity default. |
+| Profiles | Normalize configured nonempty endpoint overrides; trim service tier, reasoning effort, and API-key environment names. |
+| Prompt metadata | Trim descriptions and session-ID templates. |
+| Prompt inputs | Trim names and supplied content-type and description values. |
+| Prompt output | Trim a supplied schema path. |
+| Configured messages | Normalize roles and supplied cache-control type/TTL values using the shared domain rules. |
+
+Preserve the rules' existing distinctions. For example, roles are lowercased,
+but cache-control type normalization only trims; the move should not make new
+spellings acceptable. Preserve optional pointers and existing empty/null
+interpretation rather than creating fields merely to record an event.
+
+The backend queue default currently applies when concurrency is enabled and
+the decoded queue-capacity pointer is nil, including explicit null. ConfigKit's
+`Default` helper skips explicit input, so substituting it mechanically would
+change behavior. Retain the existing condition and record the resulting
+assignment with `RecordDerivedDefault`, including its concurrency dependency.
+
+Resource IDs, namespaces, and resource references keep their existing exact
+matching rules. Message content and `content_file` values are not new
+normalization targets. Schema paths continue to address the selected schema
+source, rather than acquiring a configuration-file-relative meaning.
+
+Request overrides, profile completion from backend settings, and caller-appended
+messages still need their existing runtime processing. Share domain normalizers
+where appropriate; avoid moving request-dependent work into loading merely to
+make inspection look fully executable.
+
+### Provenance And Resource Handling
+
+Record changed scalar fields at their exact paths with concise, value-free
+reasons. A role edit, for example, belongs at the role field within the message,
+not at the message or prompt list. Avoid recording unchanged values needlessly.
+
+Keep the current origin-based lookup for unchanged `content_file` values,
+including values inherited or retained from another source. Normalizing a sibling
+role or cache-control field should not change which file is opened.
+
+A normalized field acquires a resolution origin; its earlier source evidence is
+available for inspection. This does not promise that every normalized value
+retains a file-backed effective origin. If implementation discovers a transformed
+field that itself requires a source-relative resource lookup, preserve the
+relevant entry context explicitly before changing that field, using ConfigKit's
+public context APIs. Do not infer a runtime resource base from arbitrary history.
+
+The planned normalizer does not need to reorder, add, or remove resource or
+message entries. That describes this implementation's scope, not a new
+restriction on operators' ConfigKit composition workflows.
+
+### Observable Changes
+
+Configuration inspection will show normalized configuration values and recorded
+derived defaults. Explanations will expose the associated resolution operations
+and retained source evidence through ConfigKit's existing inspection behavior.
+This remains configuration inspection, not a rendering of a request after
+profile completion, variable substitution, or caller overrides.
+
+Configuration diffs will compare these normalized effective values. Two inputs
+that differ only in a normalized spelling may no longer produce a value diff;
+list changes that remain are still reported atomically. Existing redaction
+continues to apply to inspection and explanations.
+
+The intended execution behavior, accepted configuration values, resource
+selection, and presence/null handling remain the same. Preserve public error
+classification where applicable while integrating normalization failures with
+the existing validation path. Do not add a promise of identical diagnostic
+wording, ordering, or internal callback timing. No YAML syntax or collection
+specification-version change is planned.
+
+## Implementation Stages
+
+### 1. Adopt ConfigKit v1.1.0
+
+Update the released module dependency and sums, review applicable attribution,
+and run the existing suite against the release. Use the published module rather
+than a committed local replacement. Keep this dependency change separable from
+the normalization refactor so failures can be attributed clearly.
+
+### 2. Move Existing Configuration Transformations
+
+Move backend, profile, and prompt transformations into recorded resolution,
+including the conditional queue default. Preserve the entry metadata consumed
+by presence-sensitive behavior. Update runtime construction to use the resolved
+configuration and remove redundant interpretation state.
+
+Keep domain validation on effective values. Account for ConfigKit's conservative
+audit of lists with invalid identities so an ordinary invalid resource does not
+accidentally become an unrecorded-mutation failure merely because a sibling
+field needed normalization. This calls for integration coverage, not additional
+identity rules or a separate source audit.
+
+### 3. Include Configured Message Fields
+
+Move configured role and cache-control normalization into the same lifecycle,
+recording nested fields individually. Reuse the domain normalizers that serve
+runtime messages. Simplify prompt materialization where work has become
+redundant, while retaining template compilation, content-file reading, and
+domain construction at their appropriate boundaries.
+
+### 4. Verify The Boundary And Update Documentation
+
+Add focused tests where behavior crosses the loading/inspection boundary:
+
+- Normalized effective values and explanations for representative backend,
+  profile, prompt, and nested message fields at root and collection destinations.
+- Relative content-file loading after sibling normalization, including inherited
+  values and values retained from a different contributing source.
+- Existing absence, explicit null, empty, and zero handling, especially the queue
+  default and optional input/message metadata.
+- Existing invalid-value and identity rejection through useful public errors,
+  without broadening accepted values or adding new restrictions.
+- Execution-relevant equivalence for normalized definitions, including prompt
+  hashes, and continued normalization of caller-appended messages.
+- Existing redaction, normalized-value diff behavior, and atomic list diffs.
+
+Use targeted integration cases instead of reproducing ConfigKit's correspondence
+test suite. Run the relevant configuration, inspection, and execution tests,
+then the ordinary repository checks for the completed change.
+
+Update the format reference, consumer inspection guidance, source-loading
+documentation, and affected GoDoc when behavior lands. Explain the inspection
+change and its limits in the release notes. Review architecture wording against
+the implemented callback flow without converting implementation details into
+new contracts. Retire this roadmap and its incoming catalog link once the work
+is complete.

docs/roadmap/future.md

@@ -40,28 +40,9 @@ consumers.
 
 ### Field-Level Normalization With Preserved List Provenance
 
-Reevaluate normalizing resource fields directly in ConfigKit's effective
-configuration after upstream ConfigKit supports recording changes within a
-list without replacing the source attribution of unchanged fields.
-
-This could let configuration inspection show PromptKit-normalized values and
-field-level change history while preserving the origins needed to resolve
-relative resources such as `content_file`. Assess whether that inspection
-benefit justifies changing the execution-owned normalization approach.
-
-Include normalization of roles and cache-control values in configured messages
-in this reevaluation. This could make inspection reflect the values used for
-execution and reduce separate normalization paths. Upstream support would need
-to preserve untouched field origins within nested message lists as well as outer
-resource lists. PromptKit would continue owning the normalization rules;
-ConfigKit would record their effects and provenance. Caller-appended messages
-would still need request-time normalization because they do not pass through
-ConfigKit.
-Consider moving these transformations only if it improves inspection consistency
-and simplifies implementation without changing execution behavior.
-
-Upstream support is the reconsideration trigger, not a dependency of the
-current ConfigKit migration or a commitment to adopt the enhancement.
+Selected for planning following ConfigKit v1.1.0. The
+[field-level normalization roadmap](field-normalization.md) owns the target
+behavior, scope, and implementation stages.
 
 ## Entry Format