From 1cee127300c94779bd28f82ed5609b14871a9c56 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 2 Nov 2026 14:32:15 +0000 Subject: [PATCH] Plan field-level sequence resolution and history --- docs/roadmap/implementation.md | 447 +++++++++++++++++++++++++++ docs/roadmap/resolution-sequences.md | 2 +- 2 files changed, 448 insertions(+), 1 deletion(-) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..c96d055 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,447 @@ +# Sequence resolution and inspection implementation plan + +## Scope and execution + +Implement the agreed [feature roadmap](resolution-sequences.md) from baseline +`9adcf14`. The target is field-level resolution auditing for ordinary lists and +identity-aware resolution history for application-registered collections. This +plan is for a Luna coding agent executing the numbered stages in order during +one end-to-end sprint. Intermediate stages may leave integration incomplete or +temporarily fail to compile; do not build compatibility adapters merely to make +each intermediate state green. The completed sprint must satisfy the whole +roadmap, including history, metadata, redaction, and removed-record inspection. + +Read [agent policy](../policy/agents.md), [architecture +policy](../policy/architecture.md), [testing policy](../policy/testing.md), and +[documentation policy](../policy/documentation.md), using the +[development guide](../development.md) for orientation and commands. Read the +roadmap as the target contract; current [resolution +documentation](../config.md#recorded-application-resolution) describes the +baseline that this work deliberately extends. Preserve unrelated changes. + +Use the existing toolchain required by `go.mod` and acquired dependencies. +This sprint does not change module versions, release machinery, configuration +syntax, public path types, collection composition, or identity selector scope. +There are no benchmarks, timing thresholds, CI waits, or additional release gates +required by this plan. Prefer focused tests during implementation and the final +checks in Stage 13. Commit completed nonempty units under the agent policy. + +## Implementation decisions + +- Keep `compareConfigurationValues` atomic for sequences. Add a separate private + resolution comparison operation; validation and public effective-value diff + retain their existing comparison semantics. +- Ordinary equal-length lists compare recursively by position. A length change + requires coverage at the list or an ancestor. Registered lists use matching + only when both endpoint sequences have valid, unique identities under the + applicable application registration. A registered collection that cannot be + matched uses whole-list auditing when changed; equality still requires no + record. Do not emit matching-eligibility errors or move final validation earlier. +- Match within the same concrete registered collection destination, using the + already compiled application policy. Use payload identity rules, including + ordinary late `extends` fields. YAML-only declarations and matching across + different collection destinations are outside this correspondence. +- A valid registered collection's changed identity order or membership requires + whole-list coverage. Continue matching surviving records for inspection even + when that structural delta already covers all field changes. +- Records and helper verification address final positions for surviving values; + entry queries retain entry coordinates. A final record's missing descendant + may still refer to a removed value on its matched entry record. A wholly removed + position retains the existing historical meaning. When an address is reused, + the surviving record takes precedence; use the container to record removal of + the former occupant. Helpers never follow moves automatically. +- Use one private correspondence result throughout resolution and inspection. + Store concrete entry/final record-prefix pairs and explicit unmatched cases, + not a mutable identity registry or a full tree copy per field. Source + coordinates, original input payloads, and callback entry metadata remain fixed. +- A surviving record query selects its own entry lineage. A newly added identity + has no prior occupant's lineage. Collection queries retain physical before/after + order and removed evidence; absent historical positions remain positional. + Reuse existing source, inheritance, CLI, and decoding history semantics within + the selected entry lineage; do not rematch identities across earlier phases. +- Record reasons, dependencies, equal-valued intent, resolution origins, and + coalescing retain their existing semantics. Identity correspondence supplies + earlier evidence; it does not replace the recorded application operation. +- Reuse the shared history model and snapshot masking boundary. Preserve + historical sensitivity and dependency closure, including conservative masking + already required by the current contract. This feature promises no reduction + in masking and adds no unredacted inspection mode. + +Suggested new private file names below are implementation locations, not public +interfaces. Keep responsibilities cohesive and use existing helpers where they +already express the needed behavior. Do not introduce a general comparison +framework, new public selectors, caches, or mutation logging for this feature. + +## Stage 1 — Record the architectural extension and inspect the baseline + +**Scope:** decision documentation and a focused starting check. + +1. Add `docs/adr/0021-sequence-resolution-correspondence.md` with Accepted status + using the documentation policy's ADR format. Record the approved positional + and registered-identity rules, the entry-to-final boundary, and the distinction + between resolution auditing and public atomic diff presentation. Include + removed/reused-position inspection and final-only registration validation. + Keep discussion of alternatives in Alternatives Considered; focus on ConfigKit. +2. Relate the decision to ADRs 0003, 0010, 0017, and 0018. This extends resolution + inspection and preserves their source-assembly, diff, and lifecycle choices; + do not rewrite accepted decision bodies or mark them superseded wholesale. +3. Read `resolution.go`, `load_metadata.go`, `load_result.go`, `override_context.go`, + `history_model.go`, `history_resolution.go`, `history_select.go`, `history.go`, + `explain.go`, `secret.go`, and `snapshot.go` in `internal/engine`. Note that + engine `Resolution.InputInfo` backs public `Resolution.EntryInfo`. +4. Run the existing focused engine tests for resolution, history, sequence + identity, and diff to establish the starting state. Do not change existing + expectations simply because the new contract will later allow more precision. + +**Completion:** one architectural decision records the scope without claiming +implementation completion. No current-behavior reference is updated yet. + +## Stage 2 — Build endpoint collection correspondence + +**Scope:** private, pure matching logic, preferably in +`internal/engine/resolution_correspondence.go`. + +1. Build a correspondence object from the canonical post-CLI entry tree, final + projected tree, and compiled `sequencePolicy`. Reuse `identityFieldsAt` and + `validateNormalizedSequenceIdentityFrom` with `sequencePayloadRecord`, or + extract a small shared non-reporting helper if necessary. Reuse exact tuple + equality; do not implement another numeric or compound-key comparison. +2. Distinguish ordinary, eligible registered, and ineligible registered sequences. + For eligible pairs, retain entry-to-final and final-to-entry item indexes and + whether ordered identity membership changed. Distinguish an unmatched final + item from a matched item at the same numeric index. +3. Use tuple lookup tables and ordered sequence traversal. Resolve each record's + identity once per endpoint. Reuse mapping indexes rather than scanning every + record for every match. Keep identity tuples private and out of diagnostics. +4. Provide shape-aware prefix translation for final record descendants to their + corresponding entry paths, with an explicit no-entry result for additions. + Unregistered paths remain positional. Handle missing descendants without + borrowing the previous occupant's field; distinguish numeric keys and indexes. +5. Honor existing selector traversal: mappings and dynamic mapping namespaces + can lead to registered lists; selectors do not traverse list indexes. Nested + ordinary lists still receive recursive positional comparison later. + +**Verification:** focused pure behavior tests for reorder, insertion/removal, +identity change, empty collections, malformed/duplicate endpoints, compound and +typed scalar identities, and exact versus dynamic registrations. Reuse existing +identity equality tests rather than repeating their entire scalar matrix. + +**Completion:** correspondence is deterministic and detached from callback state; +invalid entry identities remain repairable and no new load failures are added. + +## Stage 3 — Compare ordinary lists recursively during resolution + +**Scope:** resolution-only delta production and its initial integration. + +1. Add a private resolution comparator, preferably in `resolution_compare.go`. + Reuse canonical scalar equality and existing mapping comparison behavior. + Compare equal-length ordinary sequences element by element recursively. +2. Emit a list-level replacement for ordinary length changes. Keep additions, + removals, null, and kind replacements at the appropriate affected path. + Until Stage 4 connects identity matching, registered sequences can remain + conservative; do not change the shared atomic comparator. +3. Preserve canonical diagnostic ordering and existing exact/ancestor coverage + through `resolutionCoverageIndex`. Never let a descendant record cover a + container delta. Avoid comparing a complete subtree before recursively + comparing it again at every level. +4. Use the new comparator in `runResolution`. Review `resolutionResult.deltas` + consumers; metadata integration is completed in Stage 6. + +**Verification:** field-only recording inside an ordinary list succeeds; a second +unrecorded field fails at its own path; length changes still need a container +record. Cover nested mappings/lists and explicit zero, false, and null through +representative cases. Existing public diff/comparator tests must remain atomic. + +## Stage 4 — Integrate registered comparison and paired record snapshots + +**Scope:** auditing, record validation, and before/after capture. + +1. Pass `request.sequencePolicy` through `resolveLoadInput` to `runResolution`; + reuse the preflight-compiled application policy. Build correspondence once + after final projection, before coverage and snapshot capture. +2. In eligible collections, recursively compare matched records at final paths. + Emit a structural list delta for membership/order changes independently of + field deltas. Preserve pairs even when no additional field delta is needed. + Ineligible changed registered lists use an atomic delta; unchanged ones do not. +3. Validate each record against the correctly paired entry and final target. + A helper must still equal the final value at its literal declared path. For + additions, the before side is absent; do not fall back to an old occupant. + For removals, use the addressing decisions above and existing schema checks. +4. Extend private resolution snapshots/results with the paired path information + needed downstream. Capture a field's before value from its matching entry + record; a collection/root snapshot still contains the complete physical tree. + Preserve cloned snapshots, record coalescing, reason/dependency order, + normalization dominance, and equal-valued intent. + +**Verification:** registered field edits succeed; reorder plus only a field +record fails; container coverage succeeds; identity changes require container +coverage. Check invalid-entry repair and unchanged invalid collections without +inventing an early identity error. Final invalid values still fail at the +existing final identity validation boundary. Include a helper whose target was +moved afterward and no longer matches its declared final address. + +## Stage 5 — Retain correspondence in the shared history owner + +**Scope:** evidence transport, ownership, and retained-node accounting. + +1. Extend `historyResolutionPayload` to own the compact correspondence and paired + snapshot coordinates. Transfer them from the audited result through + `resolveLoadInput` and `observeResolutionEvidence`; retain no callback or + mutable configuration references. +2. Keep original record observations and `resolutionReplayGroup` coalescing. + Index the actual concrete before and after shapes, using paired paths for + fields and the physical paths for container snapshots. Do not globally + rewrite earlier observations or duplicate all source evidence under each + moved record's final prefix. +3. Keep one scratch replay tree. Paired before paths describe evidence, not + instructions to move scratch nodes one assignment at a time. Apply a + structural container's final subtree once, as its existing replay group does. +4. Include any newly retained document roots in `historyModel.documentRoots` + and the existing admission check before secret lookup and validation. Prefer + borrowing already immutable checkpoint/snapshot roots. Release temporary + identity tables unless needed by frozen queries; retain positional pairs, + not identity text, where sufficient. + +**Verification:** extend the focused history-resolution tests to check behavior +with paired sides and overlapping records. Reuse history-budget tests to prove +that new retained roots cannot escape accounting and shared roots count once. +Do not assert an incidental struct layout or a fixed number of internal calls. + +## Stage 6 — Refresh effective metadata using the matched entry record + +**Scope:** presence, origins, and post-resolution diagnostic locations. + +1. In `applyResolutionMetadata` and `refreshResolutionFields`, seed final fields + from their corresponding entry fields rather than the previous occupant of + the final index. Use shape-aware entry lookup for exact descendants. Added + identities and newly created fields have no invented input explicitness. +2. Apply recorded resolution origins using the existing covering-record and + finer-record semantics. A structural normalization remains a resolution + operation, including when an unchanged record moves beneath it. Source + lineage remains available through history; do not suppress the operation to + make a field appear to come directly from its original file. +3. Remove effective fields absent from the final tree and rebuild effective path + shapes. Preserve positional behavior outside eligible registered collections. +4. Ensure `documentWithMetadataLocations` and `metadataFieldLocation` can obtain + the matched record's source evidence for later secret/validation diagnostics. + Preserve source IDs and passive contexts. Do not mutate the fixed entry view, + move source-document coordinates, or fabricate sources for additions. + +**Verification:** moved records with different explicit/defaulted descendants +retain their own presence; added records do not inherit old explicitness. Test +deleted fields and zero/null states, resolution origins, and source attribution. +During the callback, `EntryInfo` and `IsExplicit` must keep their old coordinate +meanings. History-based location fallback becomes fully usable in Stage 7. + +## Stage 7 — Select history through the correct entry coordinate + +**Scope:** shared history query selection and metadata history projection. + +1. Add a query-local coordinate description to the existing history selection + machinery. For a final item or a descendant under an eligible collection, + distinguish matched entry coordinates, no-entry additions, and unchanged + positional queries. A removed descendant of a surviving record uses that + record's correspondence even if the descendant is absent finally. +2. For baseline, sources, inheritance, decoding, and CLI, select the matched + entry path's existing evidence. Do not select the prior occupant of the final + index. For an added identity, suppress that prior occupant's earlier evidence. + Preserve the existing chronological and source-composition behavior at the + selected entry path; this does not infer identity across historical tiers. +3. For resolution, select paired observations and their covering operation. + Keep original-record attribution and reason/dependency lists available even + when explanation uses a coalesced replay event. +4. Project `Change.Path` into the queried final coordinate where lineage is + followed, retaining a meaningful affected path or container ancestor. Keep + original source `Location` coordinates and recorded dependency paths intact. + Derive `Previous` from the selected lineage, not unrelated positional records. +5. Keep collection/ancestor queries on physical history coordinates so they retain + the complete operation. Use existing indexes for selection; do not rescan + every historical fact separately for every final item. + +**Verification:** `Metadata.History` and historical location lookup follow the +matched record through a swap with distinct source locations. Added records do +not claim the previous occupant's definitions. Existing key/index shape and +ancestor-replacement selection tests remain valid. Stages 8 and 9 complete value +replay and removed-position handling using this same selection owner. + +## Stage 8 — Replay earlier explanation values at the entry coordinate + +**Scope:** pre-resolution explanation replay, including inheritance and CLI. + +1. Extend the query-local replay selection context in `explain.go` to use the + coordinate description from Stage 7. Keep `Explanation.Path` and its effective + snapshot at the requested final address; read earlier matched values at the + entry address instead of the final numeric slot. +2. Apply source/CLI payloads to the existing physical scratch tree unchanged. + Use the entry coordinate for before/after snapshots, contribution selection, + baseline, and decoding checkpoints. For additions with no entry lineage, + report absence before resolution rather than another record's value. +3. Thread the same coordinate through `appendInheritanceExplanation` and its + helpers. Preserve inherited source representation, original locations, + consumed controls, chronological order, and the existing typed-decoding + transition. Do not create a second independently retained history model. +4. Retain snapshot-side path information so Stage 11 can mask values using the + coordinates from which they were actually read. Do not rebase sensitive + values and then mask them solely at their new index. + +**Verification:** a record moved from index 0 to 1 shows its own earlier field +value, including selected overlay, inheritance, typed defaults, and CLI evidence +where applicable. Use focused representative fixtures rather than multiplying +every source form across every case. Collection queries keep original list order. + +## Stage 9 — Explain net resolution and preserve removed evidence + +**Scope:** resolution replay groups, structural changes, and reused addresses. + +1. For matched-record queries, read a group's before side using the entry path + and after side using the final path. Apply the physical covering subtree once + to scratch; do not replay overlapping child assignments or guess callback + order. Preserve combined reasons/dependencies and explicit equal writes. +2. A pure move may have equal before/after record values but still includes the + recorded collection operation. A field edit plus move shows the matched old + value and correct final value, with the application operation retained. +3. For newly added records, begin their individual lineage with absent before + values. For identity changes, use removal/addition, not a rename transfer. +4. Implement the roadmap's removal example: `[reports, invoices]` becomes + `[invoices]`. `/resources/0` follows invoices; `/resources` retains reports' + removal, earlier values, and available attribution; absent `/resources/1` + remains explainable using the established historical-position rules. +5. Ensure collection-level selection returns the necessary removed definition + evidence, not just a generic parent normalization. Do not expose a removed + occupant as part of a surviving record's lineage. Preserve historical-path + enumeration of actual observed shapes without inventing projected positions + for values that never existed there. Surviving record prefixes take precedence + when a missing descendant is queried; wholly absent positions stay historical. + +**Verification:** swap, deletion followed by compaction, insertion with shifted +survivors, identity replacement, removed field on a moved record, and queries of +reused versus absent positions. Assert metadata attribution and explanation +values agree, parent history retains removed evidence, and unknown paths remain +unknown. Use both whole-list-only and overlapping list/field records. + +## Stage 10 — Classify matched and removed sensitivity evidence + +**Scope:** existing classification and dependency closure. + +1. Extend history sensitivity candidates to include actual entry/final shapes + and mapped surviving evidence. Reuse schema tags, projected sensitive nodes, + predicates, and recorded dependencies; classify historical values even when + removed. Keep identity tuples and resolved credential bytes out of evidence. +2. Transfer sensitivity from matched entry descendants to final descendants + using the endpoint correspondence. Use the entry evidence as the source for + this projection, not an in-place loop that confuses indexes during swaps. + Preserve historical-coordinate masks as required for collection and removed + queries. Avoid modeling item matching as arbitrary transitive mutation edges. +3. Preserve supplied dependency paths and their existing directional semantics. + A dependency path does not follow later moves as a public identity selector. + Include dependency-derived sensitivity from retained matched evidence and + original resolution records, including children hidden by a replay group. +4. Keep typed secret lookup and final identity validation at their current + phases. Preserve source-version evidence classification at its original + coordinates. Do not introduce callbacks, credential reads, or new failure + modes while collecting correspondence sensitivity. + +**Verification:** focused cases for sensitivity transferred during a swap, +removed evidence retained at collection level, and dependency-derived sensitivity +from records hidden by a replay group. Reuse existing dependency-closure tests. +Stage 11 verifies the resulting disclosure through the safe rendering boundary. + +## Stage 11 — Mask paired snapshots and preserve safe disclosure + +**Scope:** snapshot rendering and inspection integration. + +1. Mask a moved explanation's before and after snapshots using their respective + coordinates and required historical/dependency closure. Reuse `snapshot.go`; + support paired query-local mask information if needed rather than creating + another renderer. Conservative historical masking remains acceptable under + the existing contract; do not weaken masking to make examples look narrower. +2. Preserve effective rendering, public diff masking from both results, source + version evidence redaction, and effective secret-reference disclosure rules. + A positional historical query and an identity-following surviving query must + both mask all sensitive values they actually render. +3. Check the same safe-snapshot operation for whole-collection before/after + values, including removed records. Container queries retain physical order + and their original source evidence; per-record queries use paired positions. + +**Verification:** disposable canaries for sensitive moved fields, sensitive +removed records at collection level, dynamic sensitive tags, predicates, and +derived dependencies. Verify absence of canaries in safe inspection and failure +output. Include a dependency hidden by parent/child coalescing and a swap, so +masking cannot accidentally protect only the previous numeric position. Reuse +existing tests for ordinary dependency closure and credential non-retention. + +## Stage 12 — Verify the complete feature through the public facade + +**Scope:** representative integration and ownership regressions. + +1. Add a focused public fixture, for example + `facade_resolution_sequences_test.go`, using real `Load`/`LoadLayered`, public + `Definition.SequenceIdentities`, resolution records/helpers, metadata, and + explanation. Reuse engine coverage for the detailed algorithm matrix. +2. Cover the motivating ordinary-list field edit, a registered reorder plus + normalization, and removal/position reuse with distinct source evidence. + Include one dynamic registration and one ordinary/layered equivalence case. + Check codes and semantic paths for uncovered structural changes, rather than + snapshotting entire diagnostic messages. +3. Verify fixed callback entry queries versus final identity-aware queries; + correct final values, explicitness, origins, history, and safe explanation; + and a public `Diff` that still reports the containing list atomically. +4. Mutate returned configuration and query records after loading and show that + frozen explanation and other loads are unaffected. Reuse existing callback + lifetime and concurrency tests; add only correspondence-specific ownership + coverage where those tests would miss an alias. +5. Confirm whole-list-recording consumers still work, YAML-only identities do + not gain final matching semantics, and final registration checks can still + validate or reject defaults/CLI/resolution output at the established boundary. + +**Completion:** the feature is observable through the existing public API; +no exported type, selector, or callback lifecycle change is needed. + +## Stage 13 — Document the implemented contract and validate the sprint + +**Scope:** canonical documentation and final integration checks. + +1. Update `docs/config.md` resolution, layered metadata, historical explanation, + identity-registration, and private-comparison sections. Explain ordinary + positional comparison, eligible registered correspondence, atomic structural + coverage, final versus entry addresses, additions/removals, and reused-position + history. Keep public effective diff atomic. Include a short field-edit and + remove/compact example and link the ADR for rationale. +2. Update public comments in `lifecycle.go`, `metadata.go`, `inspection.go`, and + `sequence_identity.go` where their current descriptions need qualification. + Check the engine-facing counterparts as well. Preserve fixed `EntryInfo` and + `IsExplicit` semantics and avoid promising identity-addressed public paths. +3. Update only relevant architecture-policy provisions to recognize registered + identity correspondence during resolution inspection. Preserve composition, + final-only identity validation, public facade ownership, resource lifetimes, + and secret boundaries. Update `docs/internal/overview.md` for actual new + responsibilities and `docs/development.md` for the ADR and test owners. + Keep detailed current behavior in its canonical owner. +4. Update roadmap/plan completion status after reviewing the full target, without + deleting unrelated pending roadmaps. Do not prepare release notes or publish + a release as part of this sprint. +5. Format changed Go files and review whitespace. Run the checks below with the + installed compatible Go toolchain. These are development checks, not new + release gates; do not wait for CI. + +```sh +go test ./internal/engine -run 'Resolution|History|Explain|Sequence|Collection|Secret|Sensitivity|Diff|Independent|Provenance' +go test . -run 'Resolution|SequenceIdentity|FacadeInspection|Diff|PublicFacadeBoundary|DependencyImportBoundary|LocalDocumentationLinks' +go test ./examples/consumers +go test -race ./... +./scripts/check.sh +``` + +Use narrower applicable commands during earlier stages. On failure, resolve the +actual defect or deliberately changed contract rather than weakening unrelated +assertions. Do not add traversal-count, benchmark-threshold, or configuration +snapshot tests. No new network or credential requirement belongs in the suite. + +**Final acceptance review:** verify all roadmap examples, ordinary and registered +list rules, fallback without early rejection, complete field coverage, unchanged +helper/entry semantics, matched source/presence/history, added-record absence, +removed/reused-position evidence, safe moved/removed values, detached ownership, +existing history admission, and atomic public diff. A passing coverage audit +alone is insufficient. Review the final diff and commit the completed nonempty +work with a plain-English message, leaving the workspace clean and reporting +validation and any material limitations. diff --git a/docs/roadmap/resolution-sequences.md b/docs/roadmap/resolution-sequences.md index 6d1fe4d..6e4055e 100644 --- a/docs/roadmap/resolution-sequences.md +++ b/docs/roadmap/resolution-sequences.md @@ -2,7 +2,7 @@ Status: agreed direction; not implemented. This roadmap describes field-level resolution auditing and identity-aware inspection of registered collections. -A staged implementation plan will follow review of the integrated roadmap. +The [implementation plan](implementation.md) owns execution order and verification. ## Purpose