ConfigKit/docs/roadmap/interface-remediation.md

# Interface Contract Remediation Roadmap

Status: target settled, including shared history ownership; implementation has
not begun.
Planning baseline: `1f64673` (v0.10.2 implementation plus documentation changes).

This roadmap covers the four findings in the [interface audit](interfaces.md).
The audit owns reproductions, evidence, and finding numbers. This document owns
the combined target, required changes, and general dependency order. A subsequent
implementation plan should own concrete stages, private implementation choices,
validation commands, and progress.

## Purpose And Scope

Restore agreement between ConfigKit's presence, metadata, history, and diagnostic
interfaces under their existing contracts. The first three findings concern
semantic paths that share a JSON Pointer spelling; the fourth concerns a
composition claimant order that is lost during diagnostic canonicalization.

Build one shared internal history model from which metadata history and
explanation are derived. It will own the historical evidence and semantic
attribution used by both views, replacing their independent reconstruction of
that attribution. This is the selected resolution of OD-1.

The [Path declarations](../../path.go), [metadata contract](../config.md#layered-metadata),
[resolution contract](../config.md#recorded-application-resolution), and
[strict-composition contract](../config.md#strict-document-composition) remain
authoritative. This work changes the implementation of those contracts without
selecting a new public API, path syntax, precedence rule, or configuration
restriction. The existing root-owned facade and engine responsibilities remain.

## Current Baseline And Targeted End State

An explicit mapping-key path and an explicit sequence-index path can both render
as `/shape/0`. `ParsePath` deliberately leaves numeric tokens contextual, whereas
`Keys` and `Index` preserve the caller's explicit choice. The engine already has
shape-aware effective/historical path indexes and a frozen resolution-entry view,
but some queries and metadata updates still use pointer spelling alone.

History currently reconstructs definition and previous-origin records, while
explanation replays source contributions, inheritance, decoding checkpoints, CLI
assignments, and grouped resolution snapshots. Neither public representation
contains everything needed by the other. The target is shared underlying evidence
with distinct projections for each interface, preserving their existing behavior.

| Finding | Current discrepancy | Target |
| --- | --- | --- |
| 1: resolution presence | `IsExplicit` and `Default` consult string-keyed presence after schema validation, which cannot determine actual shape beneath dynamic values. | Presence decisions use the matching semantic path in the fixed resolution-entry state. |
| 2: effective explicitness | Resolution retains old field information when pointer spelling survives a container-shape change. | Input explicitness carries forward for the same semantic path; a newly introduced key or index does not inherit another path's explicitness. |
| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket; explanation selects evidence separately. | Both interfaces derive attribution from one shared history model, selecting exact definitions, previous origins, and applicable ancestor operations by semantic path. |
| 4: claimant order | Composition constructs related locations in assembly order, then canonicalization sorts them by location. | Structured and rendered composition conflicts preserve the selected claimant order. |

For example, if input contains `shape: {'0': old}` and recorded application
resolution changes `shape` to a sequence, the old key remains an input path in
the callback's entry view. The new index is effective in final metadata but was
not explicitly supplied by that input. Its history must not present the old key's
definition as its own. This distinction concerns container shape, not identity
tracking for records that move within a sequence.

## Required Changes

### 1. Use Entry-State Shape For Resolution Presence

In [resolution.go](../../internal/engine/resolution.go), align `IsExplicit` and
the explicitness decision in `Default` with the frozen, shape-aware entry metadata
already used by `InputInfo` in
[load_metadata.go](../../internal/engine/load_metadata.go).

Reuse the existing semantic-path matching machinery. A shared private lookup can
avoid divergent presence decisions without requiring every public query to have
identical error behavior. In particular:

- Preserve `IsExplicit`'s schema-path validation and its existing diagnostics.
  `InputInfo` remains a query that returns false for missing or mismatched paths
  without those diagnostics.
- Preserve `Default`'s helper-target validation, assignment, recording, and final
  helper-value checks. Correct its presence gate; do not bypass the other checks.
- Use entry-state shape even after application code changes the working value.
  Explicit false, zero, empty, and null retain their existing presence meaning.
- Keep callback lifetime and detached metadata behavior intact.

A shape mismatch beneath a schema-valid dynamic path must not become explicit
merely because its pointer spelling matches an input path. Contextually parsed
numeric tokens continue to resolve against the entry state's actual containers.

### 2. Preserve Explicitness By Semantic Path During Resolution

Update resolution metadata application and field refresh in
[load_metadata.go](../../internal/engine/load_metadata.go) to compare the relevant
entry and final shapes before reusing presence information.

The comparison must account for container kinds along the full path. A mapping
key becoming a sequence index at an ancestor can change a descendant's semantic
path even when its last segment and full pointer spelling stay unchanged.

Keep explicitness for matching surviving paths, including descendants covered by
an ancestor normalization record. New semantic paths introduced by resolution
remain non-explicit, and retain the applicable recorded resolution origin.
Removed paths leave effective metadata while remaining available to historical
queries under the existing contract. Do not clear every descendant's presence
just because its parent was normalized.

Use the existing before/after projection and entry metadata where possible.
This work does not require tracking every transient mutation inside a callback;
the existing net-change and coalesced-record model continues to apply.

### 3. Build A Shared Internal History Model

#### Evidence And Ownership

Give historical evidence one private owner in the engine. Composition,
inheritance, decoding, CLI assignment, and resolution continue to perform their
existing work and supply evidence to that owner. The model does not drive a
second configuration load or repeat application callbacks.

Organize the evidence into ordered groups with the phase boundaries required by
the existing inspection contracts:

| Evidence group | Information to preserve |
| --- | --- |
| Default baseline | Canonical typed application defaults or Go-zero values and their default attribution. |
| Source contributions | Underlay, main-tier, and selected-overlay contributions in their existing order; exact definitions, composite origins, displaced definitions, raw policy-validated values, and sequence-position projections. |
| Inheritance | Parent transfers, child contributions including equal-valued replacements, controlling directives and their consumption, identity-field preservation, and contribution origins/dependencies. |
| Typed checkpoints | Pre-CLI and post-CLI typed projections and attribution needed to reconcile retained defaults and raw contributions with decoded values. |
| CLI contributions | Explicit assignments, including equal-valued assignments, raw supplied representation, source attribution, and affected paths. |
| Resolution | Original recorded operations, reasons, dependencies, and before/after evidence, with the grouping needed to explain overlapping records as net subtree changes. |

The logical order remains baseline, source contributions, inheritance, pre-CLI
decoding reconciliation, CLI contributions, post-CLI typed reconciliation, then
recorded resolution. The two typed checkpoints belong on their respective sides
of the CLI contribution phase.

The groups describe logical inspection chronology, not a transcript of every
physical mutation. Retain the existing net-change and coalescing rules for
resolution. Preserve superseded raw input that need not decode into the final
application type; typed checkpoints do not replace that evidence.

Use a small set of ordinary private structures and existing document/path
mechanisms. Exact types, indexes, and storage layout belong in the implementation
plan. Complete evidence means enough for the existing history, explanation, and
internal attribution consumers; it does not add persistence, a public event API,
or recording of transient application effects.

#### Shared Selection And Attribution

Retain concrete semantic-path evidence when it is available. A query over the
model must distinguish exact definitions from operations affecting an ancestor
or descendant, determine the relevant previous origins, and apply sequence
contribution coordinates at their assigned positions.

The affected semantic path and the operation's reported path are distinct facts.
For example, a replacement at `/shape` can remove a mapping key and introduce a
sequence index with the same pointer spelling. Preserve enough before/after shape
evidence to answer each explicit query correctly. Merely filtering `Change.Path`
after concrete segment kinds have been discarded is insufficient.

Make this selection and attribution common to both projections, including
inheritance integration and `Previous` reconstruction. A false definition must
not survive as a previous origin after it has been excluded from the query's
history. Preserve genuine ancestor replacement/removal effects and chronology.
A path that never existed does not acquire history solely because an ancestor
changed.

Keep the contextual meaning of parsed numeric tokens across historical states
and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`.
Indexes and compact derived records may remain, but their attribution must come
from the shared evidence. Metadata queries should not require full configuration
replay or YAML rendering to obtain definition and origin records.

#### Distinct Public Projections

`Metadata.History` produces detached definition, inheritance, and resolution
records from the model. `Explain` reconstructs and safely renders values from
the same evidence, retaining its existing decoding and grouping behavior. Neither
public API should be implemented by translating the other's public result.

Preserve these intentional differences:

| Case | Metadata history | Explanation |
| --- | --- | --- |
| Several strict contributions to a mapping | Composite definition/origin information. | Each contribution's effect on the mapping. |
| Raw input differs from canonical typed values | Definition and attribution records. | The applicable decoding reconciliation transition. |
| Overlapping resolution records | Specific recorded attribution under the existing coverage rules. | One net subtree event with combined reasons and dependencies. |
| Equal-valued input or inheritance operations | Explicit definition and operation attribution remains visible. | Relevant operations remain visible even when before and after values are equal. |

Agreement means consistent semantic identity and attribution, not identical
record lists. Retain original resolution records as evidence from which net
replay groups are derived; a grouped explanation alone loses information needed
by metadata.

#### Replace Existing Attribution Machinery

The common model must replace independent historical attribution, rather than
become another representation alongside it. The implementation plan should map
the following owners to their replacements:

| Current owner | Required change |
| --- | --- |
| History-building portions of `metadataFromDecoded`, `applyOverlayMetadata`, and `appendPostDecodeCLIHistory` in [load_metadata.go](../../internal/engine/load_metadata.go) and [override_context.go](../../internal/engine/override_context.go) | Produce or project shared evidence instead of independently reconstructing pointer-keyed histories. Retain effective-field construction where it is still needed. |
| Previous-origin reconstruction and inheritance insertion in [inheritance_history.go](../../internal/engine/inheritance_history.go) | Use common semantic selection and attribution while preserving the existing inheritance projection rules. |
| Relevance and chronology orchestration in [explain.go](../../internal/engine/explain.go) | Consume the shared model; retain useful value reconstruction and redaction helpers. |
| Separate published inheritance evidence for metadata and explanation in [load_result.go](../../internal/engine/load_result.go) | Consolidate under the frozen history owner where independent copies are unnecessary; keep required output detachment. |
| Original and coalesced resolution evidence in [history.go](../../internal/engine/history.go) | Own original records in the model and derive the replay grouping without losing specific attribution or maintaining a second independently interpreted history. |

Existing source, sequence-combination, and inheritance evidence producers may
remain useful. Their algorithms need not be rewritten merely to contribute to
the model. Likewise, final effective fields, the frozen resolution-entry view,
source records, and final configuration snapshots retain their responsibilities.
Findings 1 and 2 still require explicit presence corrections.

#### Other Consumers, Lifetime, And Resource Accounting

Move historical dependency reads in
[secret classification](../../internal/engine/secret.go) and historical location
fallbacks in [override_context.go](../../internal/engine/override_context.go) to
the shared evidence. Do not retain the old history machinery solely for those
readers. Preserve existing sensitivity propagation and historical classification;
correcting attribution does not authorize narrowing redaction for paths that
share a pointer spelling.

Evidence must be available at the lifecycle stages where these internal readers
need it, including diagnostic attribution before publication. Keep the callback's
entry metadata fixed while later resolution evidence is added. Finalize the
model for published queries without introducing lazy mutation of shared state.
Private frozen evidence may be shared by the two views while their returned
records remain detached under the existing ownership contract.

Reuse retained trees and existing lookup machinery where practical. Review
retained-history accounting against the resulting ownership graph, including
typed checkpoints and inheritance/resolution evidence. The model does not require
a full document copy per event, a new quota, or a new concurrency guarantee.
Reduced duplication is a design benefit to evaluate, not a promised memory or
speed improvement.

### 4. Preserve Composition Claimant Order Through Diagnostics

Keep the order produced by `compositionConflictDiagnostic` in
[compose.go](../../internal/engine/compose.go) through accumulation, `LoadError`,
facade conversion, and `WriteDiagnostics` in
[diagnostic.go](../../internal/engine/diagnostic.go).

Prefer a narrow internal canonicalization rule based on the diagnostic semantics
already carried across these boundaries. Identify the composition diagnostic
producers governed by the claimant-order contract rather than changing all
related-location lists. Preserve canonical ordering of diagnostic records and
the existing sorting of related locations for other diagnostics.

The root renderer converts public diagnostic records back to engine records and
canonicalizes them again. An engine-only marker discarded by facade conversion
would therefore be insufficient. No public ordering option is needed for this
correction. Keep primary selection, conflict detection, acquisition order, and
dependency-ready scheduling unchanged.

## General Implementation Order

1. Prepare the concrete shared-evidence design and focused regression cases.
   Map existing producers and consumers to the new owner, distinguish common
   attribution from presentation rules, and identify the old paths to remove.
   Select representations and indexes that preserve existing query and lifetime
   needs without adding public contracts.
2. Repair entry-state presence queries and `Default` (finding 1). This is a local
   correction that can proceed independently of the larger history work.
3. Repair final explicitness preservation (finding 2), retaining the shape evidence
   needed by metadata updates. Coordinate this work with history construction so
   the two fixes do not introduce competing definitions of path identity.
4. Build the shared model and migrate metadata history, explanation, sensitivity
   dependencies, and diagnostic-location readers. Preserve phase evidence and
   derive both public projections through common semantic attribution. Remove
   the superseded reconstruction paths and reconcile retained-history accounting.
   Verify finding 3 and its interaction with final presence metadata; repairing
   pointer buckets alone does not complete the selected architecture.
5. Complete composition diagnostic ordering (finding 4) as an independent change;
   it may proceed alongside the path work. Check rendering as well as the
   structured error before considering it complete.
6. Review the integrated result, run applicable development checks, and update
   affected implementation documentation and finding status.

These are high-level dependencies, not a requirement to combine unrelated fixes
in one commit. The implementation plan should select concrete edits and test
commands while keeping the four findings individually traceable.

## Validation And Completion

Follow the [testing policy](../policy/testing.md) and use the
[development guide](../development.md#baseline-validation) for checks. Extend
existing test owners rather than recreating full internal matrices at the facade.
The focused cases should cover:

- Both mapping-key/index mismatch directions beneath dynamic values, including
  explicit paths and contextual parsed pointers. Check callback presence and
  `Default` behavior, including the fixed entry view after working-value changes.
- Both container-shape transitions during recorded resolution, a descendant
  affected by an ancestor shape change, and a same-shape normalization control.
  Check explicitness and recorded origins through final metadata and explanation.
- Shape-changing CLI and resolution history, checking exact definitions,
  `Previous`, and genuine ancestor replacement/removal. Include contextual
  enumeration/query coverage and preserve existing inheritance and overlay cases;
  add another case only where those integrations expose an unprotected risk.
- Shared-model projections across composite mapping origins, typed decoding and
  default reconciliation, equal-valued writes, inheritance controls/transfers,
  mapped sequence replacements and removed descendants, and overlapping
  resolution records. Reuse existing cases where they protect those semantics;
  check the intentional differences between the public projections as well as
  agreement on attribution.
- Historical sensitivity/dependency propagation, diagnostic source-location
  fallback, detached returned records, and complete retained-history accounting
  under the new ownership graph. Use small controlled budget cases and existing
  disclosure fixtures rather than memory or timing thresholds.
- A composition conflict whose acquisition and assembly orders differ, using the
  audit fixture. Check both published diagnostics and rendered related-location
  order, while retaining coverage for ordinary diagnostic canonicalization.

Retain existing checks for invalid schema paths, callback lifetime, detached
records, safe historical rendering, and default/null presence semantics. Test
observable behavior rather than a particular private index or number of copies.
No benchmark campaign, broader audit, or new release gate is part of this roadmap.

Completion requires all four findings to have implementation and regression
evidence, appropriate development checks, and both history interfaces deriving
semantic attribution from the shared model. Review the final producer/consumer
map to confirm superseded independent reconstruction has been removed and
internal attribution readers have migrated. Passing the four defect regressions
while retaining the old parallel attribution machinery is not completion.

The audit's previous passing suite is not validation of these future changes.
Scope here does not promise identical output from different inspection interfaces
or a universal guarantee about unrelated path-processing code.

## Decision Status

OD-1 is settled: build one shared internal history model with distinct metadata
and explanation projections. The design investigation found that both views
require evidence the other's public representation does not preserve. Shared
ownership can replace independent attribution and redundant retained evidence,
while keeping the current lifecycle and public contracts.

The narrower repair of stored history events would leave the separate replay
attribution owner in place. Promoting current explanation output to authoritative
history would lose composite-definition and specific resolution information.
The selected common underlying model addresses both concerns. This accepts a
larger internal refactor than a local shape-filtering fix.

No significant open decision remains at roadmap level. Private representation,
indexing, and helper choices belong in the implementation plan. This roadmap
records future implementation ownership; it does not revise accepted ADRs or
declare the model implemented.