Plan null simplification after ConfigKit resolution

docs/roadmap/inspect-backend.md

@@ -8,8 +8,9 @@ presentation; Promptkit exposes the selected resource's resolved configuration.
 
 The direction is accepted for planning; implementation has not begun. Selected
 work includes fuller prompt inspection, profile inspection of the resolved
-profile itself, a new `InspectBackend`, `BackendConfig.Extends`, and typed prompt
+profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt
-construction through `PromptConfig`/`PromptSource`. Execution-policy reporting
+construction through `PromptConfig`/`PromptSource`, and simplification of
+Promptkit's post-handoff null handling. Execution-policy reporting
 and prepared-request enhancements have been removed from this roadmap. The
 decisions below record the accepted scope and representation approach.
 
@@ -72,12 +73,12 @@ guidelines, not prohibitions on a small helper or type when it serves the API.
 
 **Selected presentation:** Omit absent optional settings. Preserve explicit
 values, including zero. Absence is not displayed as zero or `unspecified`.
-Treat null like omission where they have the same meaning after ConfigKit
+Treat null like omission for nullable resource fields after ConfigKit resolution,
-resolution. Preserve their distinction only where it changes resource behavior;
+under the selected post-handoff semantics below. Literal nulls inside provider
-the mere presence of a null key is not enough to justify another public state.
+data retain their meaning; they are not optional resource-field presence states.
 The previous draft's rule to omit suppressed outbound controls belonged to an
 execution report and does not define resource inspection. Accepted decision 4
-selects typed results with this limited preservation of null semantics.
+selects typed results without a third state for nullable resource fields.
 
 Inspection reports effective resource values rather than original YAML layout,
 comments, or the history of each inherited field. `InspectConfiguration` remains
@@ -118,9 +119,9 @@ Report fields supplied by the profile or inherited through ConfigKit, including
 backend reference, endpoint/model where present, generation controls, credential
 metadata, and provider parameters. Do not merge in backend provider defaults or
 resolve an environment variable. Profiles may be partial, and their references
-remain references. Distinguish omitted values, explicit zero, and meaningful
+remain references. Distinguish omitted settings from explicit values, including
-explicit null using the resolved resource's information, rather than inferring
+zero, using the resolved resource's information rather than inferring presence
-presence from an execution target.
+from an execution target. Null uses the same absent representation as omission.
 
 Execution target presence and prepared metadata describe a later resolution
 stage and are not substitutes for profile configuration presence. Use the
@@ -128,9 +129,48 @@ existing configuration capture path to expose the needed information without
 introducing a second configuration interpreter. Follow accepted decision 4 for
 typed results and migration from `EffectiveModelParams`.
 
-This roadmap changes no `Prepare`, `PreparedRun`, `PrepareExecution`, or
+This roadmap adds no preparation API surface. Execution continues to apply
-`PreparedExecution.Details` contract. Execution continues to apply backend
+backend completion, framework defaults, and request overrides through its
-completion, framework defaults, and request overrides through its existing path.
+existing path, with the deliberate configuration behavior change below.
+
+## Post-Handoff Null Semantics
+
+**Selected behavior:** For Promptkit's interpretation of nullable resource
+fields, null and omission mean the same thing. Apply this consistently to
+resource inspection and execution, rather than hiding a runtime distinction
+only in inspection results.
+
+ConfigKit owns everything before the handoff of its resolved configuration,
+including source composition, inheritance, and null's effects during those
+operations. This work makes no upstream change or request, removes no nulls
+from input sources, and does not reinterpret or repeat ConfigKit's composition.
+The retained ConfigKit result and its provenance remain available through
+`InspectConfiguration`; resource projection does not rewrite that tree.
+
+Remove the existing special treatment of null in these profile fields:
+
+| Field | Selected meaning of either omission or null after handoff |
+| --- | --- |
+| `endpoint` | Permit the selected backend's endpoint fallback. |
+| `api_key_env` | Permit the ordinary backend credential-source fallback, subject to the existing `api_key_required` rule. |
+| Whole-map `extra_params` | Permit the backend's configured provider parameters. |
+
+Retain explicit values, including zero, false, and the existing empty-string
+clearing behavior for `endpoint` and `api_key_env`. Retain the ordinary map
+overlay behavior: `extra_params: {}` does not clear backend defaults, and
+`extra_params: ""` is not a valid mapping. Whole-map suppression of backend
+parameters is deliberately removed; do not replace it with another clearing
+option without a concrete requirement.
+
+Null entries within arbitrary provider maps and arrays remain literal provider
+data. This is not recursive null removal. Required-field checks and fields that
+currently reject null remain governed by their existing validation rules; this
+decision does not broaden accepted configuration shapes.
+
+For example, ConfigKit may resolve a child's null over a parent's endpoint.
+Promptkit then treats the resulting unset profile endpoint like omission and
+uses the selected backend's endpoint, if available. The parent's endpoint is
+not restored or recomputed by Promptkit.
 
 ## Backend Inspection
 
@@ -179,6 +219,13 @@ not presented as merely adding more fields. Document the Go/serialization
 migration for the typed result selected in decision 4 and follow the existing
 release policy for compatibility changes.
 
+Document the post-handoff null change as a configuration compatibility change
+as well. Existing null endpoint/credential-source clearing can use the existing
+explicit empty-string form when that behavior is desired. There is no new
+replacement for whole-map suppression of backend provider parameters. Update
+the [format reference](../formats.md#defaults-and-overrides) when implementation
+lands; it describes the current behavior until then.
+
 Existing preparation APIs remain available for separate concrete-request uses;
 this roadmap adds neither preview commands nor execution-policy report fields.
 Transport timeouts, engine retry settings, caller deadlines, and injected-client
@@ -193,8 +240,9 @@ Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
 when nil. ConfigKit owns sibling inheritance and its diagnostics. The source
 helper does not look up parents or populate inherited fields itself.
 
-Keep existing root/collection identity, replacement, and null semantics. Raw
+Keep existing root/collection identity and ConfigKit composition semantics,
-mappings remain available for generic controls outside the convenience helper.
+including its handling of null. Raw mappings remain available for generic
+controls outside the convenience helper.
 See [resource inheritance](../formats.md#resource-inheritance) and the existing
 [profile helper](../../profiles.go).
 
@@ -219,8 +267,9 @@ under a `prompts` list, then delegate to `configkit.Mapping`. Optional nil field
 mean omission; explicit values, including zero, false, and empty strings where
 supported, must remain representable. Preserve nil-versus-empty collections
 where that distinction matters to composition. Raw ConfigKit mappings remain
-the escape hatch for explicit-null configuration operations and other advanced
+the escape hatch for ConfigKit's explicit-null composition operations and other
-composition controls outside the typed convenience surface.
+advanced controls outside the typed convenience surface. The post-handoff
+interpretation above does not change how source helpers supply values to ConfigKit.
 
 The helper does not resolve inheritance, apply execution defaults, render
 messages, read content files, or independently validate a prompt. Existing
@@ -245,9 +294,9 @@ source round-tripping are not selected deliverables.
 
 ## Decisions
 
-All five decisions below are resolved. Numbering is retained for continuity with
+All five decisions below are resolved, including the post-handoff null
-the discussion; implementation is pending. The follow-up review of field-specific
+simplification in decision 4. Numbering is retained for continuity with the
-null semantics in decision 4 does not select a configuration behavior change.
+discussion; implementation is pending.
 
 ### 1. How Should An Unknown Backend Be Classified?
 
@@ -320,27 +369,21 @@ the resolved profile rather than retaining a second execution-settings view for
 compatibility. Apply the same resource semantics to prompt/backend additions.
 
 Represent absent settings and explicit values, including zero, false, and empty
-strings where valid. Preserve an omitted/null distinction only when it has a
+strings where valid. Treat nullable resource fields set to null as absent in
-meaningful effect after ConfigKit has finished resolution. Otherwise, treat null
+the result and adopt the post-handoff behavior described above. This supersedes
-as absent in the resource result. Clearing a parent's value during ConfigKit
+the earlier proposal to preserve three profile-specific null distinctions.
-inheritance is not, by itself, a reason to retain a third state afterward.
+ConfigKit composition and literal nulls within provider data retain their
-Use the smallest field-specific representation needed; a universal three-state
+existing ownership and meaning. No third resource-field presence state or
-wrapper or generic resource framework is not selected. Execution-target presence
+generic resource framework is needed. Execution-target presence does not
-does not represent the uncompleted profile and must not substitute for it.
+represent the uncompleted profile and must not substitute for it.
-
-The current cases needing this distinction are profile `endpoint`, `api_key_env`,
-and whole-map `extra_params`: they control whether later backend completion may
-provide a value. Arbitrary nulls inside provider data remain provider data.
-The [format reference](../formats.md#defaults-and-overrides) owns those semantics.
-Their continuing value is a separate follow-up discussion; accepting typed
-inspection does not endorse every existing null rule as permanently necessary
-or authorize changing configuration behavior in this work.
 
 **Rationale:** Typed results provide discoverable Go fields without making
-consumers interpret a generic configuration mapping. Limiting additional
+consumers interpret a generic configuration mapping. Endpoint and credential
-presence handling to meaningful cases avoids carrying source syntax and
+overrides already take precedence without first clearing a value, and explicit
-inheritance history into every field. Existing implementation machinery is not
+empty strings already support clearing those settings. No concrete requirement
-itself justification for a public distinction.
+justifies retaining whole-map suppression of backend provider parameters.
+Removing these exceptions simplifies interpretation and avoids carrying source
+syntax and inheritance history into every field.
 
 **Alternative considered:** A caller-owned resource mapping naturally represents
 missing keys and null and can reduce per-field projection work, but sacrifices
@@ -368,25 +411,32 @@ in resource inspection.
 
 1. Define the typed results and migration selected in decision 4. Source-input
    helper types and inspection results have distinct purposes; select their
-   shapes accordingly, with additional null handling only where meaningful.
+   shapes accordingly, preserving explicit values without a null presence state.
-2. Expose expanded prompt content and resolved profile fields through existing
+2. Simplify Promptkit's post-handoff handling of nullable fields and remove
-   capture/lookup paths, before execution completion. Preserve configured
+   presence machinery made unnecessary by that change. Retain presence needed
-   presence/null distinctions only where meaningful, without another composition
+   to distinguish omission from explicit zero, false, or empty values. Expose
-   pass.
+   expanded prompt content and resolved profile fields through existing
+   capture/lookup paths, before execution completion, without another
+   composition pass or changes to ConfigKit's retained result.
 3. Add backend inspection and the accepted public missing-backend error,
    returning configured diagnostic metadata without endpoint derivation.
 4. Add typed backend inheritance and the prompt source/helper types, delegating
    both to the ordinary ConfigKit mapping and loading path.
-5. Update GoDoc, consumer/internal guidance, and release notes for the selected
+5. Update GoDoc, formats, consumer/internal guidance, and release notes for the
-   behavior and profile compatibility change. Coordinate CommandKit reporting.
+   selected behavior, profile-result migration, and null compatibility change.
+   Coordinate CommandKit reporting.
 
 Use focused offline tests for actual boundary risks:
 
 - Child resources show the resolved inherited fields and overrides, not an
-  unresolved `extends` directive; explicit zero survives and absent fields
+  unresolved `extends` directive; explicit zero survives. Omitted/null nullable
-  remain distinguishable from configured null where meaningful. Equivalent
+  settings use the same absent representation; provider-data nulls retain their
-  omitted/null settings use the same absent representation; provider-data nulls
+  literal meaning.
-  retain their literal meaning.
+- Null and omitted profile endpoint, credential-source, and provider-map fields
+  permit the same backend fallback. Explicit empty-string clearing and empty-map
+  overlay behavior remain intact. Include a ConfigKit inheritance case where
+  null clears a parent value before Promptkit applies backend defaults, and
+  verify that resource inspection does not rewrite the retained ConfigKit tree.
 - Profile inspection excludes settings supplied only by a referenced backend,
   framework execution defaults, or a request. Existing execution still applies
   those settings through its ordinary resolution path.
@@ -410,5 +460,5 @@ Completion means operators can inspect the resolved selected prompt, profile,
 or backend accurately without assembling execution settings or navigating raw
 configuration paths. All three resource kinds have typed source helpers,
 including inheritance support, and consumers have migration guidance for changed
-profile inspection.
+profile inspection and post-handoff null semantics.
 No engine-policy or prepared-request additions are part of completion.