diff --git a/docs/adr/0021-sequence-resolution-correspondence.md b/docs/adr/0021-sequence-resolution-correspondence.md index 0ba9c63..18cad38 100644 --- a/docs/adr/0021-sequence-resolution-correspondence.md +++ b/docs/adr/0021-sequence-resolution-correspondence.md @@ -60,5 +60,6 @@ and lifecycle ordering in [ADR 0018](0018-cli-after-structural-resolution.md). Resolution auditing and inspection must share entry-to-final correspondence, including explicit unmatched records, while preserving historical coordinates and conservative sensitivity masking. Source assembly and public path types do -not change. The [implementation plan](../roadmap/implementation.md) tracks the -work; the [configuration contract](../config.md) owns implemented behavior. +not change. The completed implementation plan is preserved in Git history at +`7e53863:docs/roadmap/implementation.md`; the +[configuration contract](../config.md) owns implemented behavior. diff --git a/docs/development.md b/docs/development.md index 9539164..f5371a9 100644 --- a/docs/development.md +++ b/docs/development.md @@ -93,6 +93,19 @@ and current facade ownership belongs to the [component overview](internal/overvi ## Baseline validation +The completed field-level sequence resolution roadmap, implementation plan, and +resolved audit findings are preserved in Git history at `7e53863` under +`docs/roadmap/`. For example: + +```sh +git show 7e53863:docs/roadmap/implementation.md +git show 7e53863:docs/roadmap/remediation.md +``` + +The [configuration contract](config.md#recorded-application-resolution) owns +current resolution behavior, and [ADR 0021](adr/0021-sequence-resolution-correspondence.md) +records the correspondence decision. + The completed filesystem-source and underlay roadmap, original feature request, implementation plan, and resolved remediation record are preserved in Git history at `b604aa1` under `docs/roadmap/`. Current behavior belongs to the diff --git a/docs/releases/v1.1.0.md b/docs/releases/v1.1.0.md new file mode 100644 index 0000000..9fe3447 --- /dev/null +++ b/docs/releases/v1.1.0.md @@ -0,0 +1,79 @@ +# ConfigKit v1.1.0 + +ConfigKit now accepts field-level resolution records inside lists and follows +registered records across position changes when reporting their provenance and +history. This release also standardizes more release validation on ReleaseKit. + +## Field-level resolution in lists + +Applications can normalize or derive a field inside a list and record that +field directly. For example, changing `/resources/0/path` can be covered by +`RecordNormalization` at that path, without recording the entire `/resources` +list when no other changes require coverage. + +For ordinary lists of equal length, resolution auditing compares corresponding +positions recursively. Positions are addresses; ConfigKit does not infer an +element's identity from similar values. A length change still requires a record +for the list or an ancestor. + +For collections registered through `Definition.SequenceIdentities`, ConfigKit +matches records between the resolution-entry and final snapshots when both +endpoints have valid, unique identities. Field edits can be recorded individually. +Changes to identity order or membership still require list or ancestor coverage; +changing an identity is treated as removal and addition. When matching is +unavailable, a changed registered collection is audited as a whole. Existing +final identity validation remains in place, allowing resolution to repair entry +values. + +See [recorded application resolution](../config.md#recorded-application-resolution) +for the complete rules. + +## Matched history and inspection + +For eligible registered collections, inspection at a surviving record's final +position follows its own entry evidence, including earlier values, input +explicitness, and source attribution. A record no longer inherits the evidence +of a different record that previously occupied its final position. Original +source coordinates remain intact, while reported operation paths follow the +matched final record. + +Collection inspection retains removed records' evidence. Removed fields of +surviving moved records remain available through metadata history and +explanation, including deep inherited fields. Sensitivity propagation protects +matched and removed evidence and recorded dependencies; detached before/after +snapshots remain conservatively masked. + +Correspondence lookup, sensitivity propagation, and resolution-change ordering +reuse indexes, a work queue, and prepared pointer strings to avoid repeated +prefix construction, full-map passes, and formatting during sorting. No +end-to-end performance improvement is claimed. + +## Compatibility + +This is a backward-compatible extension of resolution auditing and inspection. +Existing callbacks that record a whole list remain supported. Records and helper +targets for surviving values refer to final positions; helpers do not follow +later moves automatically. `Resolution.EntryInfo` and `Resolution.IsExplicit` +retain their fixed entry-snapshot meaning. + +Public effective-configuration diffs continue to report list changes atomically. +Source composition, collection merge operations, configuration syntax, and public +path addressing are unchanged. No new public Go symbols are required. + +The consumer minimum remains Go 1.27.0, and the YAML dependency is unchanged. +Linux amd64 and arm64 remain release-validated; macOS remains best-effort. + +## Release tooling + +ReleaseKit advances from v0.6.0 to v0.8.0. The runbook uses the module-selected +`go tool releasekit` command and documents saved publication attempts and +reconciliation after failed or uncertain tag pushes. + +ReleaseKit now checks committed Markdown links and rejects filesystem dependency +replacements during candidate validation. The custom documentation-link Go test +is removed, so automated link feedback occurs during candidate validation rather +than ordinary Go tests. Source publication and hosted Gitea delivery remain +separate outcomes; CI remains outside release gates. See the +[release procedure](../release.md). + +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.1.0/COPYRIGHT) | [LICENSE](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.1.0/LICENSE) | [ATTRIBUTION](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.1.0/docs/oss/attribution.md) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 35cf9d8..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,475 +0,0 @@ -# Sequence resolution and inspection implementation plan - -Status: complete. - -## Scope and execution - -Implement the agreed [feature roadmap](resolution-sequences.md) from baseline -`9adcf14`. The target is field-level resolution auditing for ordinary lists and -identity-aware resolution history for application-registered collections. This -plan is for a Luna coding agent executing the numbered stages in order during -one end-to-end sprint. Intermediate stages may leave integration incomplete or -temporarily fail to compile; do not build compatibility adapters merely to make -each intermediate state green. The completed sprint must satisfy the whole -roadmap, including history, metadata, redaction, and removed-record inspection. - -Read [agent policy](../policy/agents.md), [architecture -policy](../policy/architecture.md), [testing policy](../policy/testing.md), and -[documentation policy](../policy/documentation.md), using the -[development guide](../development.md) for orientation and commands. Read the -roadmap as the target contract; current [resolution -documentation](../config.md#recorded-application-resolution) describes the -baseline that this work deliberately extends. Preserve unrelated changes. - -Use the existing toolchain required by `go.mod` and acquired dependencies. -This sprint does not change module versions, release machinery, configuration -syntax, public path types, collection composition, or identity selector scope. -There are no benchmarks, timing thresholds, CI waits, or additional release gates -required by this plan. Prefer focused tests during implementation and the final -checks in Stage 13. Commit completed nonempty units under the agent policy. - -## Implementation decisions - -- Keep `compareConfigurationValues` atomic for sequences. Add a separate private - resolution comparison operation; validation and public effective-value diff - retain their existing comparison semantics. -- Ordinary equal-length lists compare recursively by position. A length change - requires coverage at the list or an ancestor. Registered lists use matching - only when both endpoint sequences have valid, unique identities under the - applicable application registration. A registered collection that cannot be - matched uses whole-list auditing when changed; equality still requires no - record. Do not emit matching-eligibility errors or move final validation earlier. -- Match within the same concrete registered collection destination, using the - already compiled application policy. Use payload identity rules, including - ordinary late `extends` fields. YAML-only declarations and matching across - different collection destinations are outside this correspondence. -- A valid registered collection's changed identity order or membership requires - whole-list coverage. Continue matching surviving records for inspection even - when that structural delta already covers all field changes. -- Records and helper verification address final positions for surviving values; - entry queries retain entry coordinates. A final record's missing descendant - may still refer to a removed value on its matched entry record. A wholly removed - position retains the existing historical meaning. When an address is reused, - the surviving record takes precedence; use the container to record removal of - the former occupant. Helpers never follow moves automatically. -- Use one private correspondence result throughout resolution and inspection. - Store concrete entry/final record-prefix pairs and explicit unmatched cases, - not a mutable identity registry or a full tree copy per field. Source - coordinates, original input payloads, and callback entry metadata remain fixed. -- A surviving record query selects its own entry lineage. A newly added identity - has no prior occupant's lineage. Collection queries retain physical before/after - order and removed evidence; absent historical positions remain positional. - Reuse existing source, inheritance, CLI, and decoding history semantics within - the selected entry lineage; do not rematch identities across earlier phases. -- Record reasons, dependencies, equal-valued intent, resolution origins, and - coalescing retain their existing semantics. Identity correspondence supplies - earlier evidence; it does not replace the recorded application operation. -- Reuse the shared history model and snapshot masking boundary. Preserve - historical sensitivity and dependency closure, including conservative masking - already required by the current contract. This feature promises no reduction - in masking and adds no unredacted inspection mode. - -Suggested new private file names below are implementation locations, not public -interfaces. Keep responsibilities cohesive and use existing helpers where they -already express the needed behavior. Do not introduce a general comparison -framework, new public selectors, caches, or mutation logging for this feature. - -## Stage 1 — Record the architectural extension and inspect the baseline - -**Status:** Complete. The focused engine baseline and documentation-link checks passed. - -**Scope:** decision documentation and a focused starting check. - -1. Add `docs/adr/0021-sequence-resolution-correspondence.md` with Accepted status - using the documentation policy's ADR format. Record the approved positional - and registered-identity rules, the entry-to-final boundary, and the distinction - between resolution auditing and public atomic diff presentation. Include - removed/reused-position inspection and final-only registration validation. - Keep discussion of alternatives in Alternatives Considered; focus on ConfigKit. -2. Relate the decision to ADRs 0003, 0010, 0017, and 0018. This extends resolution - inspection and preserves their source-assembly, diff, and lifecycle choices; - do not rewrite accepted decision bodies or mark them superseded wholesale. -3. Read `resolution.go`, `load_metadata.go`, `load_result.go`, `override_context.go`, - `history_model.go`, `history_resolution.go`, `history_select.go`, `history.go`, - `explain.go`, `secret.go`, and `snapshot.go` in `internal/engine`. Note that - engine `Resolution.InputInfo` backs public `Resolution.EntryInfo`. -4. Run the existing focused engine tests for resolution, history, sequence - identity, and diff to establish the starting state. Do not change existing - expectations simply because the new contract will later allow more precision. - -**Completion:** one architectural decision records the scope without claiming -implementation completion. No current-behavior reference is updated yet. - -## Stage 2 — Build endpoint collection correspondence - -**Status:** Complete. Focused correspondence tests cover identity matching, fallback, and dynamic selectors. - -**Scope:** private, pure matching logic, preferably in -`internal/engine/resolution_correspondence.go`. - -1. Build a correspondence object from the canonical post-CLI entry tree, final - projected tree, and compiled `sequencePolicy`. Reuse `identityFieldsAt` and - `validateNormalizedSequenceIdentityFrom` with `sequencePayloadRecord`, or - extract a small shared non-reporting helper if necessary. Reuse exact tuple - equality; do not implement another numeric or compound-key comparison. -2. Distinguish ordinary, eligible registered, and ineligible registered sequences. - For eligible pairs, retain entry-to-final and final-to-entry item indexes and - whether ordered identity membership changed. Distinguish an unmatched final - item from a matched item at the same numeric index. -3. Use tuple lookup tables and ordered sequence traversal. Resolve each record's - identity once per endpoint. Reuse mapping indexes rather than scanning every - record for every match. Keep identity tuples private and out of diagnostics. -4. Provide shape-aware prefix translation for final record descendants to their - corresponding entry paths, with an explicit no-entry result for additions. - Unregistered paths remain positional. Handle missing descendants without - borrowing the previous occupant's field; distinguish numeric keys and indexes. -5. Honor existing selector traversal: mappings and dynamic mapping namespaces - can lead to registered lists; selectors do not traverse list indexes. Nested - ordinary lists still receive recursive positional comparison later. - -**Verification:** focused pure behavior tests for reorder, insertion/removal, -identity change, empty collections, malformed/duplicate endpoints, compound and -typed scalar identities, and exact versus dynamic registrations. Reuse existing -identity equality tests rather than repeating their entire scalar matrix. - -**Completion:** correspondence is deterministic and detached from callback state; -invalid entry identities remain repairable and no new load failures are added. - -## Stage 3 — Compare ordinary lists recursively during resolution - -**Status:** Complete. Focused resolution and public diff checks passed. - -**Scope:** resolution-only delta production and its initial integration. - -1. Add a private resolution comparator, preferably in `resolution_compare.go`. - Reuse canonical scalar equality and existing mapping comparison behavior. - Compare equal-length ordinary sequences element by element recursively. -2. Emit a list-level replacement for ordinary length changes. Keep additions, - removals, null, and kind replacements at the appropriate affected path. - Until Stage 4 connects identity matching, registered sequences can remain - conservative; do not change the shared atomic comparator. -3. Preserve canonical diagnostic ordering and existing exact/ancestor coverage - through `resolutionCoverageIndex`. Never let a descendant record cover a - container delta. Avoid comparing a complete subtree before recursively - comparing it again at every level. -4. Use the new comparator in `runResolution`. Review `resolutionResult.deltas` - consumers; metadata integration is completed in Stage 6. - -**Verification:** field-only recording inside an ordinary list succeeds; a second -unrecorded field fails at its own path; length changes still need a container -record. Cover nested mappings/lists and explicit zero, false, and null through -representative cases. Existing public diff/comparator tests must remain atomic. - -## Stage 4 — Integrate registered comparison and paired record snapshots - -**Status:** Complete. Focused resolution and identity tests passed, including repair and helper-target checks. - -**Scope:** auditing, record validation, and before/after capture. - -1. Pass `request.sequencePolicy` through `resolveLoadInput` to `runResolution`; - reuse the preflight-compiled application policy. Build correspondence once - after final projection, before coverage and snapshot capture. -2. In eligible collections, recursively compare matched records at final paths. - Emit a structural list delta for membership/order changes independently of - field deltas. Preserve pairs even when no additional field delta is needed. - Ineligible changed registered lists use an atomic delta; unchanged ones do not. -3. Validate each record against the correctly paired entry and final target. - A helper must still equal the final value at its literal declared path. For - additions, the before side is absent; do not fall back to an old occupant. - For removals, use the addressing decisions above and existing schema checks. -4. Extend private resolution snapshots/results with the paired path information - needed downstream. Capture a field's before value from its matching entry - record; a collection/root snapshot still contains the complete physical tree. - Preserve cloned snapshots, record coalescing, reason/dependency order, - normalization dominance, and equal-valued intent. - -**Verification:** registered field edits succeed; reorder plus only a field -record fails; container coverage succeeds; identity changes require container -coverage. Check invalid-entry repair and unchanged invalid collections without -inventing an early identity error. Final invalid values still fail at the -existing final identity validation boundary. Include a helper whose target was -moved afterward and no longer matches its declared final address. - -## Stage 5 — Retain correspondence in the shared history owner - -**Status:** Complete. Focused history, correspondence, and retained-budget checks passed. - -**Scope:** evidence transport, ownership, and retained-node accounting. - -1. Extend `historyResolutionPayload` to own the compact correspondence and paired - snapshot coordinates. Transfer them from the audited result through - `resolveLoadInput` and `observeResolutionEvidence`; retain no callback or - mutable configuration references. -2. Keep original record observations and `resolutionReplayGroup` coalescing. - Index the actual concrete before and after shapes, using paired paths for - fields and the physical paths for container snapshots. Do not globally - rewrite earlier observations or duplicate all source evidence under each - moved record's final prefix. -3. Keep one scratch replay tree. Paired before paths describe evidence, not - instructions to move scratch nodes one assignment at a time. Apply a - structural container's final subtree once, as its existing replay group does. -4. Include any newly retained document roots in `historyModel.documentRoots` - and the existing admission check before secret lookup and validation. Prefer - borrowing already immutable checkpoint/snapshot roots. Release temporary - identity tables unless needed by frozen queries; retain positional pairs, - not identity text, where sufficient. - -**Verification:** extend the focused history-resolution tests to check behavior -with paired sides and overlapping records. Reuse history-budget tests to prove -that new retained roots cannot escape accounting and shared roots count once. -Do not assert an incidental struct layout or a fixed number of internal calls. - -## Stage 6 — Refresh effective metadata using the matched entry record - -**Status:** Complete. Focused resolution, metadata, validation, and secret checks passed. - -**Scope:** presence, origins, and post-resolution diagnostic locations. - -1. In `applyResolutionMetadata` and `refreshResolutionFields`, seed final fields - from their corresponding entry fields rather than the previous occupant of - the final index. Use shape-aware entry lookup for exact descendants. Added - identities and newly created fields have no invented input explicitness. -2. Apply recorded resolution origins using the existing covering-record and - finer-record semantics. A structural normalization remains a resolution - operation, including when an unchanged record moves beneath it. Source - lineage remains available through history; do not suppress the operation to - make a field appear to come directly from its original file. -3. Remove effective fields absent from the final tree and rebuild effective path - shapes. Preserve positional behavior outside eligible registered collections. -4. Ensure `documentWithMetadataLocations` and `metadataFieldLocation` can obtain - the matched record's source evidence for later secret/validation diagnostics. - Preserve source IDs and passive contexts. Do not mutate the fixed entry view, - move source-document coordinates, or fabricate sources for additions. - -**Verification:** moved records with different explicit/defaulted descendants -retain their own presence; added records do not inherit old explicitness. Test -deleted fields and zero/null states, resolution origins, and source attribution. -During the callback, `EntryInfo` and `IsExplicit` must keep their old coordinate -meanings. History-based location fallback becomes fully usable in Stage 7. - -## Stage 7 — Select history through the correct entry coordinate - -**Status:** Complete. Focused history and matched-location checks passed. - -**Scope:** shared history query selection and metadata history projection. - -1. Add a query-local coordinate description to the existing history selection - machinery. For a final item or a descendant under an eligible collection, - distinguish matched entry coordinates, no-entry additions, and unchanged - positional queries. A removed descendant of a surviving record uses that - record's correspondence even if the descendant is absent finally. -2. For baseline, sources, inheritance, decoding, and CLI, select the matched - entry path's existing evidence. Do not select the prior occupant of the final - index. For an added identity, suppress that prior occupant's earlier evidence. - Preserve the existing chronological and source-composition behavior at the - selected entry path; this does not infer identity across historical tiers. -3. For resolution, select paired observations and their covering operation. - Keep original-record attribution and reason/dependency lists available even - when explanation uses a coalesced replay event. -4. Project `Change.Path` into the queried final coordinate where lineage is - followed, retaining a meaningful affected path or container ancestor. Keep - original source `Location` coordinates and recorded dependency paths intact. - Derive `Previous` from the selected lineage, not unrelated positional records. -5. Keep collection/ancestor queries on physical history coordinates so they retain - the complete operation. Use existing indexes for selection; do not rescan - every historical fact separately for every final item. - -**Verification:** `Metadata.History` and historical location lookup follow the -matched record through a swap with distinct source locations. Added records do -not claim the previous occupant's definitions. Existing key/index shape and -ancestor-replacement selection tests remain valid. Stages 8 and 9 complete value -replay and removed-position handling using this same selection owner. - -## Stage 8 — Replay earlier explanation values at the entry coordinate - -**Status:** Complete. Focused source, inheritance, CLI, history, and explanation checks passed. - -**Scope:** pre-resolution explanation replay, including inheritance and CLI. - -1. Extend the query-local replay selection context in `explain.go` to use the - coordinate description from Stage 7. Keep `Explanation.Path` and its effective - snapshot at the requested final address; read earlier matched values at the - entry address instead of the final numeric slot. -2. Apply source/CLI payloads to the existing physical scratch tree unchanged. - Use the entry coordinate for before/after snapshots, contribution selection, - baseline, and decoding checkpoints. For additions with no entry lineage, - report absence before resolution rather than another record's value. -3. Thread the same coordinate through `appendInheritanceExplanation` and its - helpers. Preserve inherited source representation, original locations, - consumed controls, chronological order, and the existing typed-decoding - transition. Do not create a second independently retained history model. -4. Retain snapshot-side path information so Stage 11 can mask values using the - coordinates from which they were actually read. Do not rebase sensitive - values and then mask them solely at their new index. - -**Verification:** a record moved from index 0 to 1 shows its own earlier field -value, including selected overlay, inheritance, typed defaults, and CLI evidence -where applicable. Use focused representative fixtures rather than multiplying -every source form across every case. Collection queries keep original list order. - -## Stage 9 — Explain net resolution and preserve removed evidence - -**Status:** Complete. Focused history and explanation checks passed for moves, removals, additions, and reused positions. - -**Scope:** resolution replay groups, structural changes, and reused addresses. - -1. For matched-record queries, read a group's before side using the entry path - and after side using the final path. Apply the physical covering subtree once - to scratch; do not replay overlapping child assignments or guess callback - order. Preserve combined reasons/dependencies and explicit equal writes. -2. A pure move may have equal before/after record values but still includes the - recorded collection operation. A field edit plus move shows the matched old - value and correct final value, with the application operation retained. -3. For newly added records, begin their individual lineage with absent before - values. For identity changes, use removal/addition, not a rename transfer. -4. Implement the roadmap's removal example: `[reports, invoices]` becomes - `[invoices]`. `/resources/0` follows invoices; `/resources` retains reports' - removal, earlier values, and available attribution; absent `/resources/1` - remains explainable using the established historical-position rules. -5. Ensure collection-level selection returns the necessary removed definition - evidence, not just a generic parent normalization. Do not expose a removed - occupant as part of a surviving record's lineage. Preserve historical-path - enumeration of actual observed shapes without inventing projected positions - for values that never existed there. Surviving record prefixes take precedence - when a missing descendant is queried; wholly absent positions stay historical. - -**Verification:** swap, deletion followed by compaction, insertion with shifted -survivors, identity replacement, removed field on a moved record, and queries of -reused versus absent positions. Assert metadata attribution and explanation -values agree, parent history retains removed evidence, and unknown paths remain -unknown. Use both whole-list-only and overlapping list/field records. - -## Stage 10 — Classify matched and removed sensitivity evidence - -**Status:** Complete. Focused sensitivity, secret, and dependency checks passed. - -**Scope:** existing classification and dependency closure. - -1. Extend history sensitivity candidates to include actual entry/final shapes - and mapped surviving evidence. Reuse schema tags, projected sensitive nodes, - predicates, and recorded dependencies; classify historical values even when - removed. Keep identity tuples and resolved credential bytes out of evidence. -2. Transfer sensitivity from matched entry descendants to final descendants - using the endpoint correspondence. Use the entry evidence as the source for - this projection, not an in-place loop that confuses indexes during swaps. - Preserve historical-coordinate masks as required for collection and removed - queries. Avoid modeling item matching as arbitrary transitive mutation edges. -3. Preserve supplied dependency paths and their existing directional semantics. - A dependency path does not follow later moves as a public identity selector. - Include dependency-derived sensitivity from retained matched evidence and - original resolution records, including children hidden by a replay group. -4. Keep typed secret lookup and final identity validation at their current - phases. Preserve source-version evidence classification at its original - coordinates. Do not introduce callbacks, credential reads, or new failure - modes while collecting correspondence sensitivity. - -**Verification:** focused cases for sensitivity transferred during a swap, -removed evidence retained at collection level, and dependency-derived sensitivity -from records hidden by a replay group. Reuse existing dependency-closure tests. -Stage 11 verifies the resulting disclosure through the safe rendering boundary. - -## Stage 11 — Mask paired snapshots and preserve safe disclosure - -**Status:** Complete. Focused moved, removed, dynamic-tag, dependency, secret, and diff checks passed. - -**Scope:** snapshot rendering and inspection integration. - -1. Mask a moved explanation's before and after snapshots using their respective - coordinates and required historical/dependency closure. Reuse `snapshot.go`; - support paired query-local mask information if needed rather than creating - another renderer. Conservative historical masking remains acceptable under - the existing contract; do not weaken masking to make examples look narrower. -2. Preserve effective rendering, public diff masking from both results, source - version evidence redaction, and effective secret-reference disclosure rules. - A positional historical query and an identity-following surviving query must - both mask all sensitive values they actually render. -3. Check the same safe-snapshot operation for whole-collection before/after - values, including removed records. Container queries retain physical order - and their original source evidence; per-record queries use paired positions. - -**Verification:** disposable canaries for sensitive moved fields, sensitive -removed records at collection level, dynamic sensitive tags, predicates, and -derived dependencies. Verify absence of canaries in safe inspection and failure -output. Include a dependency hidden by parent/child coalescing and a swap, so -masking cannot accidentally protect only the previous numeric position. Reuse -existing tests for ordinary dependency closure and credential non-retention. - -## Stage 12 — Verify the complete feature through the public facade - -**Status:** Complete. Public resolution, identity, diff, disclosure, and boundary checks passed. - -**Scope:** representative integration and ownership regressions. - -1. Add a focused public fixture, for example - `facade_resolution_sequences_test.go`, using real `Load`/`LoadLayered`, public - `Definition.SequenceIdentities`, resolution records/helpers, metadata, and - explanation. Reuse engine coverage for the detailed algorithm matrix. -2. Cover the motivating ordinary-list field edit, a registered reorder plus - normalization, and removal/position reuse with distinct source evidence. - Include one dynamic registration and one ordinary/layered equivalence case. - Check codes and semantic paths for uncovered structural changes, rather than - snapshotting entire diagnostic messages. -3. Verify fixed callback entry queries versus final identity-aware queries; - correct final values, explicitness, origins, history, and safe explanation; - and a public `Diff` that still reports the containing list atomically. -4. Mutate returned configuration and query records after loading and show that - frozen explanation and other loads are unaffected. Reuse existing callback - lifetime and concurrency tests; add only correspondence-specific ownership - coverage where those tests would miss an alias. -5. Confirm whole-list-recording consumers still work, YAML-only identities do - not gain final matching semantics, and final registration checks can still - validate or reject defaults/CLI/resolution output at the established boundary. - -**Completion:** the feature is observable through the existing public API; -no exported type, selector, or callback lifecycle change is needed. - -## Stage 13 — Document the implemented contract and validate the sprint - -**Status:** Complete. Focused engine, facade, and consumer checks, `go test -race ./...`, and `./scripts/check.sh` passed with Go 1.27.0 on Linux. - -**Scope:** canonical documentation and final integration checks. - -1. Update `docs/config.md` resolution, layered metadata, historical explanation, - identity-registration, and private-comparison sections. Explain ordinary - positional comparison, eligible registered correspondence, atomic structural - coverage, final versus entry addresses, additions/removals, and reused-position - history. Keep public effective diff atomic. Include a short field-edit and - remove/compact example and link the ADR for rationale. -2. Update public comments in `lifecycle.go`, `metadata.go`, `inspection.go`, and - `sequence_identity.go` where their current descriptions need qualification. - Check the engine-facing counterparts as well. Preserve fixed `EntryInfo` and - `IsExplicit` semantics and avoid promising identity-addressed public paths. -3. Update only relevant architecture-policy provisions to recognize registered - identity correspondence during resolution inspection. Preserve composition, - final-only identity validation, public facade ownership, resource lifetimes, - and secret boundaries. Update `docs/internal/overview.md` for actual new - responsibilities and `docs/development.md` for the ADR and test owners. - Keep detailed current behavior in its canonical owner. -4. Update roadmap/plan completion status after reviewing the full target, without - deleting unrelated pending roadmaps. Do not prepare release notes or publish - a release as part of this sprint. -5. Format changed Go files and review whitespace. Run the checks below with the - installed compatible Go toolchain. These are development checks, not new - release gates; do not wait for CI. - -```sh -go test ./internal/engine -run 'Resolution|History|Explain|Sequence|Collection|Secret|Sensitivity|Diff|Independent|Provenance' -go test . -run 'Resolution|SequenceIdentity|FacadeInspection|Diff|PublicFacadeBoundary|DependencyImportBoundary|LocalDocumentationLinks' -go test ./examples/consumers -go test -race ./... -./scripts/check.sh -``` - -Use narrower applicable commands during earlier stages. On failure, resolve the -actual defect or deliberately changed contract rather than weakening unrelated -assertions. Do not add traversal-count, benchmark-threshold, or configuration -snapshot tests. No new network or credential requirement belongs in the suite. - -**Final acceptance review:** verify all roadmap examples, ordinary and registered -list rules, fallback without early rejection, complete field coverage, unchanged -helper/entry semantics, matched source/presence/history, added-record absence, -removed/reused-position evidence, safe moved/removed values, detached ownership, -existing history admission, and atomic public diff. A passing coverage audit -alone is insufficient. Review the final diff and commit the completed nonempty -work with a plain-English message, leaving the workspace clean and reporting -validation and any material limitations. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index 2eda4cc..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,290 +0,0 @@ -# Sequence resolution implementation review - -Reviewed on 2026-11-02 against the [feature roadmap](resolution-sequences.md), -[implementation plan](implementation.md), and -[ADR 0021](../adr/0021-sequence-resolution-correspondence.md). - -Scope: commits `1cee127..b066090` plus the pre-existing working-tree changes, -including final documentation, an additional sensitivity regression, and the -sensitivity fixed-point loop in `internal/engine/secret.go`. The latter loop is -not present in `b066090`; Finding 3 specifically concerns the reviewed working -tree. Findings 1 and 2 also affect the committed implementation. Existing work -was preserved; this review changes no production code or tests. - -The architecture now separates resolution comparison from the atomic comparator -used by public diff. A private entry-to-final correspondence for registered -collections feeds auditing, snapshots, metadata refresh, and shared history -selection/replay. All five findings below have now been addressed. Original -audit observations are retained for context; completion notes describe the fixes. - -A follow-up integration and algorithm audit at `27be6ba`, including the same -pre-existing working-tree changes, added Findings 4 and 5. It preserved the -decided comparison, identity, ordering, and history policies. Findings 1–3 were -still open at that audit; Finding 3 covers the principal collection-size quadratic loop and -is not duplicated below. - -## Finding 1 — Metadata history rejects a valid matched historical field - -**Priority:** P2. **Status:** Complete. - -Completion: metadata history now delegates correspondence-aware eligibility to -the shared selector. A regression verifies agreement with explanation for a -removed field absent from the previous final-slot occupant, while effective -lookup, unknown paths, and explicit key/index distinctions remain correct. -The regression failed before the fix; focused tests and `./scripts/check.sh` -passed after it. - -**Code:** [engine metadata](../../internal/engine/metadata.go), -`Metadata.History` at lines 50–55; compare -[shared history selection](../../internal/engine/history_select.go), -`selectHistory` at lines 43–49. - -`selectHistory` accepts a query whose matched entry coordinate has an observed -shape, even if its final coordinate never had that field. `Metadata.History` -still requires the requested coordinate itself to have been observed before -calling that selector. Consequently metadata history and explanation disagree -on a removed descendant of a moved record. - -Reproduction using `Items []map[string]any`, with an application registration -of `id` at `/items`: - -```yaml -items: [{id: a, extra: own}, {id: b}] -``` - -During resolution, delete `extra` from `a`, swap the records, and record a -normalization at `/items`. Query `/items/1/extra`, which follows surviving -record `a` back to entry position 0. - -**Expected:** both inspection APIs retain `a`'s earlier field evidence and its -removal. Effective lookup remains absent. The roadmap and plan explicitly cover -removed descendants of surviving records through correspondence. - -**Actual, verified through the public API:** `Explain` succeeds with two history -events, including a resolution transition from `own` to absence, while -`Metadata.History` returns an empty slice. The requested physical path did not -exist on either endpoint, although its matched entry path did. - -**Remediation:** make metadata history use the shared selector's correspondence- -aware eligibility rather than its stronger preliminary shape check. Preserve -unknown-path and explicit key/index-shape handling. Add a focused regression -where the old occupant of the final position never had the removed field; -the existing removed-field test gives both original records that field and does -not expose this inconsistency. This requires no synthetic historical-path -enumeration or new public addressing mechanism. - -## Finding 2 — Deep descendant history reports the old record's operation path - -**Priority:** P2. **Status:** Complete. - -Completion: pre-resolution operation paths now use the matched record-prefix -translation, preserving ancestor scope and suffixes. Resolution records already -in final coordinates are left unchanged. A regression covers deep inherited -fields, record/container queries, original source coordinates, and overlapping -final-position records. Focused history checks and `./scripts/check.sh` passed. - -**Code:** [shared history selection](../../internal/engine/history_select.go), -`selectHistory`, especially the reported-path translation at lines 116–120. - -Selection follows the correct entry record, but reported-path translation uses -the queried leaf and a depth heuristic rather than the matched record prefix. -An operation reported at an entry record or other ancestor more than one level -above the queried leaf can retain its old index. That `Change.Path` is neither -the final queried path nor its ancestor. This diverges from Stage 7's requirement -to project affected paths into the final coordinate while preserving original -source locations separately. - -Verified reproduction with registered `id` identity: - -```yaml -items: - - id: parent - nested: {more: {value: old}} - - id: child - extends: {id: parent} -``` - -Swap the records during resolution, record `/items`, then inspect -`Metadata.History(/items/0/nested/more/value)`. The child is now at position 0. - -**Expected:** the inherited operation's reported path identifies the affected -region under final `/items/0`, or its actual common container ancestor; source -`Location` coordinates continue to describe the original input. - -**Actual:** history contains a main-layer inherited operation reported at -`/items/1`, followed by the resolution operation at `/items`. The first reported -path now identifies the other record. Source/value matching alone does not -repair this public path inconsistency. - -**Remediation:** translate reported paths using the actual entry/final item-prefix -pair, preserving the operation's relative suffix and ancestor scope. Distinguish -entry-coordinate source/inheritance paths from resolution records that already -use final coordinates, so the latter are not translated twice. Verify a deep -descendant of a moved inherited record, a containing operation, and an unchanged -common collection ancestor. Avoid substituting the queried leaf for every -operation path solely to satisfy an ancestor check. - -## Finding 3 — Sensitivity propagation rescans the whole collection quadratically - -**Priority:** P2. **Status:** Complete. - -Completion: matching edges and real dependencies now share one indexed work -queue. Candidate coordinates are registered once, newly reached historical -coordinates are registered on demand, and each trigger fires at most once; -classification no longer rebuilds closure in full-map passes. Conservative -historical masking is retained. The existing earlier-matched-value regression -is retained in this fix, and a cyclic rotation regression covers removed fields -and dependency-derived values. Focused sensitivity tests and -`./scripts/check.sh` passed. - -**Code:** [secret classification](../../internal/engine/secret.go), -`classifySecretPathsWithEvidence`, working-tree lines 83–122; associated -[correspondence translations](../../internal/engine/resolution_correspondence.go). - -The working-tree fix repeatedly projects every currently sensitive path in both -directions through the permutation, merges those paths into the same positional -mask, and rebuilds dependency closure until no mask changes. Each pass scans all -candidate paths and reruns `closeSecretDependencies` from scratch. - -For an `n`-record cyclic rotation with one initially sensitive field and no -dependency edges, the first pass reaches neighboring positions. Subsequent -passes treat those newly projected positions as fresh entry/final seeds and -advance around the cycle. There are Theta(n) passes over Theta(n) paths, giving -Theta(n²) work at fixed record width and depth. This does not require a dense -dependency graph or unusually deep configuration. The plan called for endpoint -projection from entry evidence and explicitly cautioned against treating item -matching as arbitrary transitive mutation edges. - -A public-API probe with entry identities `[a, b, c, d]`, final identities -`[d, a, b, c]`, and only `/items/0/value` predicate-classified confirms that all -four final values become masked. Some masking at reused historical coordinates -is deliberately conservative; the issue is reusing projected positional masks -as further correspondence seeds and rescanning the complete population. This -finding does not introduce a promise to disclose more values or weaken the -existing historical sensitivity contract. - -**Remediation:** retain the protection provided by the additional -`TestMovedDependencyProtectsEarlierMatchedValue` regression, including backward -protection of earlier matched values after dependency classification. Represent -entry/final sensitivity correspondence explicitly and process newly classified -states with indexed dependency/matching edges or another bounded worklist, -rather than rebuilding full closure on every projection pass. Keep original -historical-coordinate masks and required conservative redaction. Check cyclic -rotations and interactions with dependencies without performance thresholds or -tests of incidental traversal counts. Reverting to a single incomplete -projection pass would lose the dependency-history protection the working-tree -change was added to provide. - -## Finding 4 — Correspondence lookup repeatedly serializes growing path prefixes - -**Priority:** P2. **Status:** Complete. - -Completion: eligible collection prefixes now use the existing prefix index. -Translation formats the requested pointer once, retains concrete path-kind -checks, and copies the resulting path independently. No-eligible cases return -positionally without formatting prefixes; no-registration loads skip matching -construction entirely. Cloning rebuilds an independently owned index. Existing -matching/history/ownership coverage and a numeric-key/escaped-key regression -passed, along with `./scripts/check.sh`. - -**Code:** [correspondence translations](../../internal/engine/resolution_correspondence.go), -`entryPath` at lines 96–125 and `finalPath` at lines 129–156; consumers include -[metadata refresh](../../internal/engine/load_metadata.go), resolution record -validation/snapshots, shared history queries, diagnostic locations, and sensitivity. - -Both translation methods search for a containing collection by constructing and -serializing successively longer prefixes of the requested path. For an unmatched -path with `d` constant-width segments, prefix serialization performs work -proportional to `0 + 1 + ... + (d-1)`, or Theta(d²), per lookup. Each nonempty -serialization also allocates a pointer string. The final owned path copy only -needs Theta(d) work. This prefix search is not an algorithm selected by the -roadmap or plan. - -This also affects ordinary configurations with no registered collections: -`entryPath` still formats every prefix even when `collections` is empty. -`applyResolutionMetadata` calls it while seeding every final field and again -while refreshing explicitness. Thus a resolution callback introduces this cost -across ordinary metadata, not only when records actually move. On a chain of -depth `d`, the lookups at all levels contribute Theta(d³) prefix-serialization -work, compared with Theta(d²) for linear lookup at each level. Existing depth -limits bound the magnitude; this is an avoidable extra depth factor within those -limits, not a claim of unbounded supported input depth. - -**Remediation:** return an owned positional path immediately when there is no -eligible correspondence. For registered scopes, use a segment-prefix index, or -serialize the requested pointer once and query the existing `pathPrefixIndex` -for matching ancestors. Retain concrete path-kind verification so numeric keys -and sequence indexes remain distinct, and preserve additions, removals, and -wholly absent historical-position behavior. Because registration selectors do -not pass through sequence indexes, the relevant matching record prefix does -not require an exhaustive scan of every collection. This can make translation -linear in encoded path length while retaining the current compact pair tables. -Also avoid the preparatory correspondence tree walk when the compiled -application policy contains no registrations. Verify behavioral equivalence -through existing translation, shape, history, and ownership tests rather than -adding timing thresholds or exact allocation assertions. - -## Finding 5 — Delta sorting formats pointers inside every comparison - -**Priority:** P3. **Status:** Complete. - -Completion: both comparison operations share the existing decorated delta sort, -which constructs one pointer per delta and preserves canonical pointer ordering -and the kind tie-break. Existing comparison and diff coverage passed, along with -`./scripts/check.sh` and the final `go test -race ./...` run. No performance -thresholds, persistent cache, or new public contract were introduced. - -**Code:** [resolution comparison](../../internal/engine/resolution_compare.go), -`compareResolutionValues` at lines 64–69; compare the existing decorated sort in -[shared structural comparison](../../internal/engine/document_compare.go), -`compareConfigurationValues`. - -The new resolution comparator invokes `Path.String()` on both operands during -every sorting comparison. With `k` deltas, comparison sorting can invoke this -Theta(k log k) times. For pointer length `L`, that contributes Theta(k L log k) -serialization work and Theta(k log k) non-root pointer allocations in the worst -case. This is newly avoidable overhead; the existing shared comparator already -formats one pointer per delta before sorting. - -**Remediation:** follow the existing decorate/sort/unwrap pattern: compute each -delta's pointer once, sort by the retained pointer and existing kind tie-break, -then return the original semantic paths. Pointer construction becomes Theta(k L) -with Theta(k) pointer allocations. Sorting still performs O(k log k) comparisons -with normal string-comparison costs; this does not claim a better worst-case -bound for string sorting itself. Keep canonical ordering as expressly selected -in the plan. Reuse ordering tests; no alternate sort policy, persistent cache, -benchmark gate, or API change is needed. - -## Validation performed - -- The current working tree passed `./scripts/check.sh` with Go 1.27.0 on Linux. -- `go test -race ./...` passed with the same toolchain. -- Separate disposable programs outside the repository reproduced Findings 1 - and 2 through the public facade and confirmed the cyclic masking behavior - described in Finding 3. Its complexity follows from the implemented loop; - no timing benchmarks were collected. -- The review found no additional public API restrictions or earlier identity - validation requirements. Shared atomic diff comparison, callback entry views, - registered selector scope, and YAML-only identity lifetime remain preserved. -- The follow-up traced identity-table construction, resolution comparison, - coverage and snapshot handoffs, metadata refresh, history selection/replay, - and sensitivity propagation. Endpoint matching uses tuple lookup tables rather - than pairwise record searches. Existing whole-tree explanation replay and - canonical ordering were retained as planned choices, not reconsidered. -- Findings 4 and 5 follow directly from the loops, `Path.String` implementation, - and existing prefix-index/decorated-sort alternatives. No new tests or - benchmarks were needed; prior successful checks apply to the unchanged - production and test files. The amended document's links and whitespace were - checked separately. - -## Remediation validation - -All five issues were addressed sequentially in separate nonempty commits, each -with its completion note and relevant verification. Final validation passed -`./scripts/check.sh` and `go test -race ./...` with Go 1.27.0 on Linux. The -development check includes formatting, whitespace, tests, local documentation -links, vet, and build checks. No benchmark thresholds or release gates were added. - -The directly related pending sensitivity regression was retained with Finding 3. -Unrelated pre-existing documentation and public-comment changes were preserved -outside the remediation commits. diff --git a/docs/roadmap/resolution-sequences.md b/docs/roadmap/resolution-sequences.md deleted file mode 100644 index b698052..0000000 --- a/docs/roadmap/resolution-sequences.md +++ /dev/null @@ -1,252 +0,0 @@ -# Field-level resolution and identity-aware sequence history - -Status: implemented. This roadmap records field-level -resolution auditing and identity-aware inspection of registered collections. -The [implementation plan](implementation.md) owns execution order and verification. - -## Purpose - -Allow applications to normalize or derive individual fields inside lists and -record those fields directly. Preserve the requirement that every net change -made during resolution has an exact or ancestor record, together with coherent -final values, provenance, history, and sensitivity metadata. - -For a collection with an applicable application registration, the declared key -defines a record across positions. Use that correspondence in auditing and -inspection. For ordinary lists, position is the only available correspondence; -compare positions as addresses without inferring which application object an -element represents. - -## Prior baseline - -At the planning baseline, the resolution contract audited the typed configuration -before and after `Definition.Resolve`. Mappings were compared recursively, while -the private structural comparator reported any list change at the whole-list -path. A field-level record inside a changed list therefore failed to cover that -list-level delta. Recording the whole list worked. - -For example, changing `/resources/0/path` and recording only that path previously -produced an unrecorded-change diagnostic at `/resources`. That was established -behavior rather than a new regression associated with v1.0.0. - -The comparator also serves other inspection and metadata operations. Making -resolution auditing more precise must account for those callers rather than -silently changing the public diff contract. - -The prior resolution implementation captured before and after values at the -same numeric path and refreshed metadata at that path. Following a registered -record through a move requires the matching entry and final record evidence. - -Application registrations already define identities for collections and constrain -their final values. YAML-only declarations have a shorter lifetime. The -[identity contract](../config.md#application-registrations-and-identity-details) -and [ADR 0017](../adr/0017-collection-sequence-policy.md) own those distinctions. - -## Target behavior - -### Mappings and ordinary lists - -Mapping comparison remains recursive. For lists without applicable application -identity registrations: - -- If the lists have equal lengths, compare corresponding positions recursively. - A position is an address, not an assertion of element identity. -- If the length changes, report a structural change at the list path, requiring - a record for that list or an ancestor. -- For equal-length lists, a reorder manifests as changes at the affected - positions. Every detected change needs coverage; a whole-list record remains - a convenient way to cover the operation. - -Apply these rules recursively to nested values. Preserve existing treatment of -additions, removals, null, scalar differences, and changes of value kind except -where finer sequence comparison deliberately changes the reported path. -Ordinary-list inspection remains positional; similar field values do not justify -moving provenance or history between positions. - -### Lists with registered identity - -Use applicable `Definition.SequenceIdentities` registrations to match records -between the entry and final snapshots. Reuse existing identity-field selection, -compound-identity, scalar-equality, and duplicate-detection semantics. Do not infer -an identity from a field merely named `name` or `id`. - -| Observed change | Proposed auditing behavior | -| --- | --- | -| An ordinary field changes on a matched record | Compare recursively and allow a record covering that field. | -| Identity order changes | Report a structural change at the list path; require a list or ancestor record. | -| An identity is added or removed | Report a structural change at the list path; require a list or ancestor record. | -| An identity changes | Treat it as removal and addition, without inferring a rename. | - -Compare matched records by identity even when their numeric positions differ. -Structural changes still require whole-list coverage. A field record by itself -does not cover an accompanying reorder or membership change. - -Identity matching serves both audit coverage and inspection. Although a -whole-list record covers every edit in a moved collection, matching still -identifies which record's earlier values and source evidence belong to each -final record. Stopping at a whole-list delta is insufficient for that purpose. - -Match only the resolution-entry and final snapshots. Identity changes are -removal and addition; temporary identities, intermediate moves, and the order of -callback mutations are not reconstructed. Matching does not introduce a new -collection merge operation or change whole-record `upsert`. - -Use the existing registration selectors and scope. They traverse mapping keys, -not sequence indexes; recursive comparison of nested lists does not expand the -registration API. YAML-only declarations do not acquire a new final-resolution -guarantee or become identity selectors for this work. - -### Matching eligibility - -Use identity matching when both snapshots provide valid, unique identities under -the applicable registration. Otherwise use conservative whole-list auditing for -a changed collection, without inventing correspondence. An unchanged collection -does not require a resolution record merely because matching is unavailable. - -Preserve existing final identity validation. Resolution may repair invalid entry -values; this feature does not reject them earlier or require valid identities -throughout the callback. When matching is unavailable, the recorded container -operation describes the change without claiming that individual records survived. - -### Example - -Assume the application registers `name` as the identity for `resources`: - -```yaml -resources: - - name: reports - path: ./reports - - name: invoices - path: ./invoices -``` - -Changing the first path to `/srv/reports` can be covered by a normalization record -at `/resources/0/path`. The unchanged `invoices` record requires no record. - -If resolution also swaps the records, the final collection is: - -```yaml -resources: - - name: invoices - path: ./invoices - - name: reports - path: /srv/reports -``` - -Recording only `/resources/1/path` does not cover the reorder. Recording -`/resources` covers that structural operation and any field edits within the -list. A field record can additionally explain the path normalization. - -Inspection at the final `reports` position includes that record's earlier -evidence from position 0, with its original source coordinates, and the recorded -collection operation explaining the move. Its previous path is `./reports`, -not the `./invoices` value that previously occupied position 1. The `invoices` -record moved without a field change. An ordinary unregistered list instead -describes changes at positions 0 and 1, without identifying records across them. - -### Recording, inspection, and ownership - -Resolution records and helper targets for surviving values describe final -positions. Preserve the existing ability to record removed values. -`Resolution.EntryInfo` and `Resolution.IsExplicit` retain their existing fixed -entry-snapshot meaning. Numeric paths and helper targets are not automatically -rebound to an identity when the application later moves an element. A helper -assignment must still agree with the final value at its declared target. - -Identity establishes correspondence internally; this feature adds no public -identity-addressed path or mutation-tracking API. An application that restructures -a list can record its container rather than maintain individual helper targets -through moves. - -Preserve helper-value consistency checks, reason and dependency recording, -presence distinctions, detached query results, and independent-load ownership. -Fine-grained accepted changes must appear consistently in the returned values, -metadata, history, explanation, and sensitivity propagation. For eligible -registered collections, use the same entry-to-final correspondence for before -values, source attribution, input explicitness, and sensitivity. Do not attach -the previous occupant's evidence to a different record merely because it now -occupies that index. Identity matching alone does not make a newly added record -explicit input or give it an invented source. - -A history query at a final record's position includes the matched record's -earlier evidence from its entry position. Preserve original source coordinates -and include the recorded collection operation explaining the move. Recorded -normalization and derivation retain their resolution origins; following prior -evidence does not erase the application operation. Removed evidence remains -available as described below. - -Explicit structural records remain the explanation of application-owned list -restructuring. Dependency paths retain their existing recording semantics; -identity matching does not turn supplied paths into selectors that follow later -mutations. Check sensitivity propagation through both the matched evidence and -recorded dependencies, including moves and removed values. The proposal does not -reconstruct the callback's intermediate mutations. - -Public effective-configuration diff output remains atomic for lists. Resolution -comparison may need a dedicated internal operation or comparison mode so that -its new behavior does not alter unrelated callers. - -### Removed records and reused positions - -For eligible registered collections, inspection distinguishes a surviving -record's lineage from evidence of records removed during resolution: - -- Queries at surviving positions describe the surviving record's lineage, - including its matched entry evidence and the recorded collection operation. - A removed record that previously occupied that position does not become part - of the survivor's individual history. -- Collection-level history and explanation retain the removal and the removed - record's evidence, including available source attribution and before values. -- Historical positions that no longer exist remain inspectable under the - existing historical-path rules. Their availability does not give a reused - position a second, identity-addressed meaning. - -For example, if `[reports, invoices]` becomes `[invoices]`, a query at -`/resources/0` follows `invoices` back to its entry position 1. The removal of -`reports` and its earlier evidence remain inspectable at `/resources`, rather -than being mixed into the individual history of `invoices`. The now-absent -position `/resources/1` remains available under the historical-path contract. -Original source coordinates are preserved, and existing sensitivity rules apply -to all retained evidence. - -## Work needed from the baseline - -- Trace resolution comparison, record coverage, helper verification, metadata - updates, history replay, and sensitivity handling to identify where each needs - finer sequence changes or explicit before/after correspondence. -- Add positional recursive comparison for ordinary equal-length lists and - identity-aware comparison for eligible registered lists, while retaining - structural coverage for the cases above. -- Reuse collection identity machinery and validate matching eligibility without - strengthening the existing final-only registration guarantee. -- Carry matched entry and final positions into resolution snapshots, metadata, - history, and sensitivity handling through the existing shared history model. - A coverage-only implementation does not complete this feature. Preserve - source coordinates and distinguish a record's lineage from the prior occupant - of its final position. -- Verify field edits, nested lists, zero/false/null values, multiple simultaneous - changes, length changes, reorders, identity changes, helper targets, and - metadata behavior across the public loading boundary. Include cases in which - a field record must not conceal another unrecorded change. -- Exercise moved records with distinct source evidence, explicitness, and - sensitivity; combined reorders and field edits; removed values; repaired entry - identities; and dynamic registration selectors. Check agreement between final - metadata and explanation while retaining entry-query behavior and atomic public - diff output. Verify the removal-and-position-reuse example above: surviving - record queries follow their own lineage, collection queries retain removed - evidence, and absent historical positions remain inspectable. -- Update the resolution contract and relevant API comments when implemented. - Review ADR and policy follow-up for registered identity as the correspondence - used during resolution inspection, as well as the distinction between - resolution comparison and public diff presentation. Preserve accepted ADR - decision bodies. - -Use identity lookup tables for expected linear matching work in collection size, -plus recursive value-comparison work. Preserve deterministic output and existing -equality semantics; avoid pairwise searches for matching records. This is a -design objective, not a new public performance guarantee. - -No configuration syntax, dependency, release gate, or new mutation-history system -is proposed. Existing callbacks that record a whole list should continue to work. -The matching-eligibility, final-position-addressing, and removed-record inspection -decisions are settled and integrated above.