configkit/docs/roadmap/implementation.md

# Sequence resolution and inspection implementation plan

## Scope and execution

Implement the agreed [feature roadmap](resolution-sequences.md) from baseline
`9adcf14`. The target is field-level resolution auditing for ordinary lists and
identity-aware resolution history for application-registered collections. This
plan is for a Luna coding agent executing the numbered stages in order during
one end-to-end sprint. Intermediate stages may leave integration incomplete or
temporarily fail to compile; do not build compatibility adapters merely to make
each intermediate state green. The completed sprint must satisfy the whole
roadmap, including history, metadata, redaction, and removed-record inspection.

Read [agent policy](../policy/agents.md), [architecture
policy](../policy/architecture.md), [testing policy](../policy/testing.md), and
[documentation policy](../policy/documentation.md), using the
[development guide](../development.md) for orientation and commands. Read the
roadmap as the target contract; current [resolution
documentation](../config.md#recorded-application-resolution) describes the
baseline that this work deliberately extends. Preserve unrelated changes.

Use the existing toolchain required by `go.mod` and acquired dependencies.
This sprint does not change module versions, release machinery, configuration
syntax, public path types, collection composition, or identity selector scope.
There are no benchmarks, timing thresholds, CI waits, or additional release gates
required by this plan. Prefer focused tests during implementation and the final
checks in Stage 13. Commit completed nonempty units under the agent policy.

## Implementation decisions

- Keep `compareConfigurationValues` atomic for sequences. Add a separate private
  resolution comparison operation; validation and public effective-value diff
  retain their existing comparison semantics.
- Ordinary equal-length lists compare recursively by position. A length change
  requires coverage at the list or an ancestor. Registered lists use matching
  only when both endpoint sequences have valid, unique identities under the
  applicable application registration. A registered collection that cannot be
  matched uses whole-list auditing when changed; equality still requires no
  record. Do not emit matching-eligibility errors or move final validation earlier.
- Match within the same concrete registered collection destination, using the
  already compiled application policy. Use payload identity rules, including
  ordinary late `extends` fields. YAML-only declarations and matching across
  different collection destinations are outside this correspondence.
- A valid registered collection's changed identity order or membership requires
  whole-list coverage. Continue matching surviving records for inspection even
  when that structural delta already covers all field changes.
- Records and helper verification address final positions for surviving values;
  entry queries retain entry coordinates. A final record's missing descendant
  may still refer to a removed value on its matched entry record. A wholly removed
  position retains the existing historical meaning. When an address is reused,
  the surviving record takes precedence; use the container to record removal of
  the former occupant. Helpers never follow moves automatically.
- Use one private correspondence result throughout resolution and inspection.
  Store concrete entry/final record-prefix pairs and explicit unmatched cases,
  not a mutable identity registry or a full tree copy per field. Source
  coordinates, original input payloads, and callback entry metadata remain fixed.
- A surviving record query selects its own entry lineage. A newly added identity
  has no prior occupant's lineage. Collection queries retain physical before/after
  order and removed evidence; absent historical positions remain positional.
  Reuse existing source, inheritance, CLI, and decoding history semantics within
  the selected entry lineage; do not rematch identities across earlier phases.
- Record reasons, dependencies, equal-valued intent, resolution origins, and
  coalescing retain their existing semantics. Identity correspondence supplies
  earlier evidence; it does not replace the recorded application operation.
- Reuse the shared history model and snapshot masking boundary. Preserve
  historical sensitivity and dependency closure, including conservative masking
  already required by the current contract. This feature promises no reduction
  in masking and adds no unredacted inspection mode.

Suggested new private file names below are implementation locations, not public
interfaces. Keep responsibilities cohesive and use existing helpers where they
already express the needed behavior. Do not introduce a general comparison
framework, new public selectors, caches, or mutation logging for this feature.

## Stage 1 — Record the architectural extension and inspect the baseline

**Scope:** decision documentation and a focused starting check.

1. Add `docs/adr/0021-sequence-resolution-correspondence.md` with Accepted status
   using the documentation policy's ADR format. Record the approved positional
   and registered-identity rules, the entry-to-final boundary, and the distinction
   between resolution auditing and public atomic diff presentation. Include
   removed/reused-position inspection and final-only registration validation.
   Keep discussion of alternatives in Alternatives Considered; focus on ConfigKit.
2. Relate the decision to ADRs 0003, 0010, 0017, and 0018. This extends resolution
   inspection and preserves their source-assembly, diff, and lifecycle choices;
   do not rewrite accepted decision bodies or mark them superseded wholesale.
3. Read `resolution.go`, `load_metadata.go`, `load_result.go`, `override_context.go`,
   `history_model.go`, `history_resolution.go`, `history_select.go`, `history.go`,
   `explain.go`, `secret.go`, and `snapshot.go` in `internal/engine`. Note that
   engine `Resolution.InputInfo` backs public `Resolution.EntryInfo`.
4. Run the existing focused engine tests for resolution, history, sequence
   identity, and diff to establish the starting state. Do not change existing
   expectations simply because the new contract will later allow more precision.

**Completion:** one architectural decision records the scope without claiming
implementation completion. No current-behavior reference is updated yet.

## Stage 2 — Build endpoint collection correspondence

**Scope:** private, pure matching logic, preferably in
`internal/engine/resolution_correspondence.go`.

1. Build a correspondence object from the canonical post-CLI entry tree, final
   projected tree, and compiled `sequencePolicy`. Reuse `identityFieldsAt` and
   `validateNormalizedSequenceIdentityFrom` with `sequencePayloadRecord`, or
   extract a small shared non-reporting helper if necessary. Reuse exact tuple
   equality; do not implement another numeric or compound-key comparison.
2. Distinguish ordinary, eligible registered, and ineligible registered sequences.
   For eligible pairs, retain entry-to-final and final-to-entry item indexes and
   whether ordered identity membership changed. Distinguish an unmatched final
   item from a matched item at the same numeric index.
3. Use tuple lookup tables and ordered sequence traversal. Resolve each record's
   identity once per endpoint. Reuse mapping indexes rather than scanning every
   record for every match. Keep identity tuples private and out of diagnostics.
4. Provide shape-aware prefix translation for final record descendants to their
   corresponding entry paths, with an explicit no-entry result for additions.
   Unregistered paths remain positional. Handle missing descendants without
   borrowing the previous occupant's field; distinguish numeric keys and indexes.
5. Honor existing selector traversal: mappings and dynamic mapping namespaces
   can lead to registered lists; selectors do not traverse list indexes. Nested
   ordinary lists still receive recursive positional comparison later.

**Verification:** focused pure behavior tests for reorder, insertion/removal,
identity change, empty collections, malformed/duplicate endpoints, compound and
typed scalar identities, and exact versus dynamic registrations. Reuse existing
identity equality tests rather than repeating their entire scalar matrix.

**Completion:** correspondence is deterministic and detached from callback state;
invalid entry identities remain repairable and no new load failures are added.

## Stage 3 — Compare ordinary lists recursively during resolution

**Scope:** resolution-only delta production and its initial integration.

1. Add a private resolution comparator, preferably in `resolution_compare.go`.
   Reuse canonical scalar equality and existing mapping comparison behavior.
   Compare equal-length ordinary sequences element by element recursively.
2. Emit a list-level replacement for ordinary length changes. Keep additions,
   removals, null, and kind replacements at the appropriate affected path.
   Until Stage 4 connects identity matching, registered sequences can remain
   conservative; do not change the shared atomic comparator.
3. Preserve canonical diagnostic ordering and existing exact/ancestor coverage
   through `resolutionCoverageIndex`. Never let a descendant record cover a
   container delta. Avoid comparing a complete subtree before recursively
   comparing it again at every level.
4. Use the new comparator in `runResolution`. Review `resolutionResult.deltas`
   consumers; metadata integration is completed in Stage 6.

**Verification:** field-only recording inside an ordinary list succeeds; a second
unrecorded field fails at its own path; length changes still need a container
record. Cover nested mappings/lists and explicit zero, false, and null through
representative cases. Existing public diff/comparator tests must remain atomic.

## Stage 4 — Integrate registered comparison and paired record snapshots

**Scope:** auditing, record validation, and before/after capture.

1. Pass `request.sequencePolicy` through `resolveLoadInput` to `runResolution`;
   reuse the preflight-compiled application policy. Build correspondence once
   after final projection, before coverage and snapshot capture.
2. In eligible collections, recursively compare matched records at final paths.
   Emit a structural list delta for membership/order changes independently of
   field deltas. Preserve pairs even when no additional field delta is needed.
   Ineligible changed registered lists use an atomic delta; unchanged ones do not.
3. Validate each record against the correctly paired entry and final target.
   A helper must still equal the final value at its literal declared path. For
   additions, the before side is absent; do not fall back to an old occupant.
   For removals, use the addressing decisions above and existing schema checks.
4. Extend private resolution snapshots/results with the paired path information
   needed downstream. Capture a field's before value from its matching entry
   record; a collection/root snapshot still contains the complete physical tree.
   Preserve cloned snapshots, record coalescing, reason/dependency order,
   normalization dominance, and equal-valued intent.

**Verification:** registered field edits succeed; reorder plus only a field
record fails; container coverage succeeds; identity changes require container
coverage. Check invalid-entry repair and unchanged invalid collections without
inventing an early identity error. Final invalid values still fail at the
existing final identity validation boundary. Include a helper whose target was
moved afterward and no longer matches its declared final address.

## Stage 5 — Retain correspondence in the shared history owner

**Scope:** evidence transport, ownership, and retained-node accounting.

1. Extend `historyResolutionPayload` to own the compact correspondence and paired
   snapshot coordinates. Transfer them from the audited result through
   `resolveLoadInput` and `observeResolutionEvidence`; retain no callback or
   mutable configuration references.
2. Keep original record observations and `resolutionReplayGroup` coalescing.
   Index the actual concrete before and after shapes, using paired paths for
   fields and the physical paths for container snapshots. Do not globally
   rewrite earlier observations or duplicate all source evidence under each
   moved record's final prefix.
3. Keep one scratch replay tree. Paired before paths describe evidence, not
   instructions to move scratch nodes one assignment at a time. Apply a
   structural container's final subtree once, as its existing replay group does.
4. Include any newly retained document roots in `historyModel.documentRoots`
   and the existing admission check before secret lookup and validation. Prefer
   borrowing already immutable checkpoint/snapshot roots. Release temporary
   identity tables unless needed by frozen queries; retain positional pairs,
   not identity text, where sufficient.

**Verification:** extend the focused history-resolution tests to check behavior
with paired sides and overlapping records. Reuse history-budget tests to prove
that new retained roots cannot escape accounting and shared roots count once.
Do not assert an incidental struct layout or a fixed number of internal calls.

## Stage 6 — Refresh effective metadata using the matched entry record

**Scope:** presence, origins, and post-resolution diagnostic locations.

1. In `applyResolutionMetadata` and `refreshResolutionFields`, seed final fields
   from their corresponding entry fields rather than the previous occupant of
   the final index. Use shape-aware entry lookup for exact descendants. Added
   identities and newly created fields have no invented input explicitness.
2. Apply recorded resolution origins using the existing covering-record and
   finer-record semantics. A structural normalization remains a resolution
   operation, including when an unchanged record moves beneath it. Source
   lineage remains available through history; do not suppress the operation to
   make a field appear to come directly from its original file.
3. Remove effective fields absent from the final tree and rebuild effective path
   shapes. Preserve positional behavior outside eligible registered collections.
4. Ensure `documentWithMetadataLocations` and `metadataFieldLocation` can obtain
   the matched record's source evidence for later secret/validation diagnostics.
   Preserve source IDs and passive contexts. Do not mutate the fixed entry view,
   move source-document coordinates, or fabricate sources for additions.

**Verification:** moved records with different explicit/defaulted descendants
retain their own presence; added records do not inherit old explicitness. Test
deleted fields and zero/null states, resolution origins, and source attribution.
During the callback, `EntryInfo` and `IsExplicit` must keep their old coordinate
meanings. History-based location fallback becomes fully usable in Stage 7.

## Stage 7 — Select history through the correct entry coordinate

**Scope:** shared history query selection and metadata history projection.

1. Add a query-local coordinate description to the existing history selection
   machinery. For a final item or a descendant under an eligible collection,
   distinguish matched entry coordinates, no-entry additions, and unchanged
   positional queries. A removed descendant of a surviving record uses that
   record's correspondence even if the descendant is absent finally.
2. For baseline, sources, inheritance, decoding, and CLI, select the matched
   entry path's existing evidence. Do not select the prior occupant of the final
   index. For an added identity, suppress that prior occupant's earlier evidence.
   Preserve the existing chronological and source-composition behavior at the
   selected entry path; this does not infer identity across historical tiers.
3. For resolution, select paired observations and their covering operation.
   Keep original-record attribution and reason/dependency lists available even
   when explanation uses a coalesced replay event.
4. Project `Change.Path` into the queried final coordinate where lineage is
   followed, retaining a meaningful affected path or container ancestor. Keep
   original source `Location` coordinates and recorded dependency paths intact.
   Derive `Previous` from the selected lineage, not unrelated positional records.
5. Keep collection/ancestor queries on physical history coordinates so they retain
   the complete operation. Use existing indexes for selection; do not rescan
   every historical fact separately for every final item.

**Verification:** `Metadata.History` and historical location lookup follow the
matched record through a swap with distinct source locations. Added records do
not claim the previous occupant's definitions. Existing key/index shape and
ancestor-replacement selection tests remain valid. Stages 8 and 9 complete value
replay and removed-position handling using this same selection owner.

## Stage 8 — Replay earlier explanation values at the entry coordinate

**Scope:** pre-resolution explanation replay, including inheritance and CLI.

1. Extend the query-local replay selection context in `explain.go` to use the
   coordinate description from Stage 7. Keep `Explanation.Path` and its effective
   snapshot at the requested final address; read earlier matched values at the
   entry address instead of the final numeric slot.
2. Apply source/CLI payloads to the existing physical scratch tree unchanged.
   Use the entry coordinate for before/after snapshots, contribution selection,
   baseline, and decoding checkpoints. For additions with no entry lineage,
   report absence before resolution rather than another record's value.
3. Thread the same coordinate through `appendInheritanceExplanation` and its
   helpers. Preserve inherited source representation, original locations,
   consumed controls, chronological order, and the existing typed-decoding
   transition. Do not create a second independently retained history model.
4. Retain snapshot-side path information so Stage 11 can mask values using the
   coordinates from which they were actually read. Do not rebase sensitive
   values and then mask them solely at their new index.

**Verification:** a record moved from index 0 to 1 shows its own earlier field
value, including selected overlay, inheritance, typed defaults, and CLI evidence
where applicable. Use focused representative fixtures rather than multiplying
every source form across every case. Collection queries keep original list order.

## Stage 9 — Explain net resolution and preserve removed evidence

**Scope:** resolution replay groups, structural changes, and reused addresses.

1. For matched-record queries, read a group's before side using the entry path
   and after side using the final path. Apply the physical covering subtree once
   to scratch; do not replay overlapping child assignments or guess callback
   order. Preserve combined reasons/dependencies and explicit equal writes.
2. A pure move may have equal before/after record values but still includes the
   recorded collection operation. A field edit plus move shows the matched old
   value and correct final value, with the application operation retained.
3. For newly added records, begin their individual lineage with absent before
   values. For identity changes, use removal/addition, not a rename transfer.
4. Implement the roadmap's removal example: `[reports, invoices]` becomes
   `[invoices]`. `/resources/0` follows invoices; `/resources` retains reports'
   removal, earlier values, and available attribution; absent `/resources/1`
   remains explainable using the established historical-position rules.
5. Ensure collection-level selection returns the necessary removed definition
   evidence, not just a generic parent normalization. Do not expose a removed
   occupant as part of a surviving record's lineage. Preserve historical-path
   enumeration of actual observed shapes without inventing projected positions
   for values that never existed there. Surviving record prefixes take precedence
   when a missing descendant is queried; wholly absent positions stay historical.

**Verification:** swap, deletion followed by compaction, insertion with shifted
survivors, identity replacement, removed field on a moved record, and queries of
reused versus absent positions. Assert metadata attribution and explanation
values agree, parent history retains removed evidence, and unknown paths remain
unknown. Use both whole-list-only and overlapping list/field records.

## Stage 10 — Classify matched and removed sensitivity evidence

**Scope:** existing classification and dependency closure.

1. Extend history sensitivity candidates to include actual entry/final shapes
   and mapped surviving evidence. Reuse schema tags, projected sensitive nodes,
   predicates, and recorded dependencies; classify historical values even when
   removed. Keep identity tuples and resolved credential bytes out of evidence.
2. Transfer sensitivity from matched entry descendants to final descendants
   using the endpoint correspondence. Use the entry evidence as the source for
   this projection, not an in-place loop that confuses indexes during swaps.
   Preserve historical-coordinate masks as required for collection and removed
   queries. Avoid modeling item matching as arbitrary transitive mutation edges.
3. Preserve supplied dependency paths and their existing directional semantics.
   A dependency path does not follow later moves as a public identity selector.
   Include dependency-derived sensitivity from retained matched evidence and
   original resolution records, including children hidden by a replay group.
4. Keep typed secret lookup and final identity validation at their current
   phases. Preserve source-version evidence classification at its original
   coordinates. Do not introduce callbacks, credential reads, or new failure
   modes while collecting correspondence sensitivity.

**Verification:** focused cases for sensitivity transferred during a swap,
removed evidence retained at collection level, and dependency-derived sensitivity
from records hidden by a replay group. Reuse existing dependency-closure tests.
Stage 11 verifies the resulting disclosure through the safe rendering boundary.

## Stage 11 — Mask paired snapshots and preserve safe disclosure

**Scope:** snapshot rendering and inspection integration.

1. Mask a moved explanation's before and after snapshots using their respective
   coordinates and required historical/dependency closure. Reuse `snapshot.go`;
   support paired query-local mask information if needed rather than creating
   another renderer. Conservative historical masking remains acceptable under
   the existing contract; do not weaken masking to make examples look narrower.
2. Preserve effective rendering, public diff masking from both results, source
   version evidence redaction, and effective secret-reference disclosure rules.
   A positional historical query and an identity-following surviving query must
   both mask all sensitive values they actually render.
3. Check the same safe-snapshot operation for whole-collection before/after
   values, including removed records. Container queries retain physical order
   and their original source evidence; per-record queries use paired positions.

**Verification:** disposable canaries for sensitive moved fields, sensitive
removed records at collection level, dynamic sensitive tags, predicates, and
derived dependencies. Verify absence of canaries in safe inspection and failure
output. Include a dependency hidden by parent/child coalescing and a swap, so
masking cannot accidentally protect only the previous numeric position. Reuse
existing tests for ordinary dependency closure and credential non-retention.

## Stage 12 — Verify the complete feature through the public facade

**Scope:** representative integration and ownership regressions.

1. Add a focused public fixture, for example
   `facade_resolution_sequences_test.go`, using real `Load`/`LoadLayered`, public
   `Definition.SequenceIdentities`, resolution records/helpers, metadata, and
   explanation. Reuse engine coverage for the detailed algorithm matrix.
2. Cover the motivating ordinary-list field edit, a registered reorder plus
   normalization, and removal/position reuse with distinct source evidence.
   Include one dynamic registration and one ordinary/layered equivalence case.
   Check codes and semantic paths for uncovered structural changes, rather than
   snapshotting entire diagnostic messages.
3. Verify fixed callback entry queries versus final identity-aware queries;
   correct final values, explicitness, origins, history, and safe explanation;
   and a public `Diff` that still reports the containing list atomically.
4. Mutate returned configuration and query records after loading and show that
   frozen explanation and other loads are unaffected. Reuse existing callback
   lifetime and concurrency tests; add only correspondence-specific ownership
   coverage where those tests would miss an alias.
5. Confirm whole-list-recording consumers still work, YAML-only identities do
   not gain final matching semantics, and final registration checks can still
   validate or reject defaults/CLI/resolution output at the established boundary.

**Completion:** the feature is observable through the existing public API;
no exported type, selector, or callback lifecycle change is needed.

## Stage 13 — Document the implemented contract and validate the sprint

**Scope:** canonical documentation and final integration checks.

1. Update `docs/config.md` resolution, layered metadata, historical explanation,
   identity-registration, and private-comparison sections. Explain ordinary
   positional comparison, eligible registered correspondence, atomic structural
   coverage, final versus entry addresses, additions/removals, and reused-position
   history. Keep public effective diff atomic. Include a short field-edit and
   remove/compact example and link the ADR for rationale.
2. Update public comments in `lifecycle.go`, `metadata.go`, `inspection.go`, and
   `sequence_identity.go` where their current descriptions need qualification.
   Check the engine-facing counterparts as well. Preserve fixed `EntryInfo` and
   `IsExplicit` semantics and avoid promising identity-addressed public paths.
3. Update only relevant architecture-policy provisions to recognize registered
   identity correspondence during resolution inspection. Preserve composition,
   final-only identity validation, public facade ownership, resource lifetimes,
   and secret boundaries. Update `docs/internal/overview.md` for actual new
   responsibilities and `docs/development.md` for the ADR and test owners.
   Keep detailed current behavior in its canonical owner.
4. Update roadmap/plan completion status after reviewing the full target, without
   deleting unrelated pending roadmaps. Do not prepare release notes or publish
   a release as part of this sprint.
5. Format changed Go files and review whitespace. Run the checks below with the
   installed compatible Go toolchain. These are development checks, not new
   release gates; do not wait for CI.

```sh
go test ./internal/engine -run 'Resolution|History|Explain|Sequence|Collection|Secret|Sensitivity|Diff|Independent|Provenance'
go test . -run 'Resolution|SequenceIdentity|FacadeInspection|Diff|PublicFacadeBoundary|DependencyImportBoundary|LocalDocumentationLinks'
go test ./examples/consumers
go test -race ./...
./scripts/check.sh
```

Use narrower applicable commands during earlier stages. On failure, resolve the
actual defect or deliberately changed contract rather than weakening unrelated
assertions. Do not add traversal-count, benchmark-threshold, or configuration
snapshot tests. No new network or credential requirement belongs in the suite.

**Final acceptance review:** verify all roadmap examples, ordinary and registered
list rules, fallback without early rejection, complete field coverage, unchanged
helper/entry semantics, matched source/presence/history, added-record absence,
removed/reused-position evidence, safe moved/removed values, detached ownership,
existing history admission, and atomic public diff. A passing coverage audit
alone is insufficient. Review the final diff and commit the completed nonempty
work with a plain-English message, leaving the workspace clean and reporting
validation and any material limitations.