diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 4d259a8..18f1986 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -12,8 +12,10 @@ The native command boundary, actual ConfigKit inspection results, separate domai catalogs, independent selections, reporting, and cancellation architecture are substantially implemented. The initial pass identified findings 1–3. An integration-focused follow-up at `8be65fd` adds finding 4; the application source -is unchanged from the initial review. All four P2 findings remain open. -These reviews make no application changes. +is unchanged from the initial review. An algorithm-focused follow-up at +`65d24fb` adds findings 5–8: two P2 findings and two P3 optimization +opportunities. Findings 1–4 remain open. These reviews make no application +changes. ## 1. P2: Preserve campaign inheritance when materializing session documents @@ -175,6 +177,154 @@ Required remediation: bytes, and retained downstream success. Include changed roster contents as a counterexample that must invalidate affected work. +## 5. P2: Capture pipeline provenance once per catalog derivation + +Stages: 2 and 4; catalog consumers in Stages 5, 7, and 11. + +Locations: `internal/config/artifact_catalog.go:49–65,127–143`, +`internal/config/artifact_families.go:78`, and +`internal/config/pipeline_configkit.go:291–307`. + +Every authored concrete-artifact provenance lookup calls +`cfg.ConfigKitResult().Metadata()`. Family derivation calls the same accessor +once per family. In the selected ConfigKit v1.2.1, this accessor calls +`Inspection.Metadata`, which clones the complete effective-path and field-info +maps. It is a detached snapshot operation, rather than a constant-time handle +lookup. The immutable history model is shared; the effective metadata maps and +their nested records are copied. + +Let `D` be the number of authored artifact and family declarations, and `M` the +size of the metadata copied by one snapshot. Snapshot acquisition alone costs +`O(D * M)` work and cumulative allocation per catalog derivation, in addition to +the actual history queries. With bounded-size declarations and `M = Θ(D)`, this +is quadratic work. Family lookups also rebuild the complete source-path index +per family. The prior implementation reused its retained metadata snapshot; +the repeated full-snapshot acquisition is introduced by this migration. + +Refactoring opportunity: + +- Acquire one native metadata snapshot and one source-ID/path index inside a + catalog derivation. Share them across authored-artifact and family provenance + queries through a small local resolver or explicit helper parameters. +- This reduces snapshot work to `O(M)` plus the existing per-declaration native + queries and output processing. Build the source index once as well. The + existing `partyCharacterSourceLookup` already demonstrates an invocation-local + snapshot and lookup cache. +- Retain native history, inheritance, and source semantics, with no global + cache or replacement inspection result. Verify equivalent provenance for + concrete definitions, families, imports, and overlays; preserve the completed + ConfigKit values. + +This opportunity concerns how Narratio calls the public API. It does not require +changing ConfigKit's detached-snapshot contract or reviewing its selected +configuration algorithms. + +## 6. P3: Avoid cloning the complete family catalog for individual origins + +Stages: 5 and 11; artifact evidence in Stage 15. + +Locations: `internal/stage/analyze_resume.go:88–103`, +`internal/stage/analyze.go:179–185`, `internal/config/config.go:71–76`, and +`internal/config/artifact_families.go:395–418`. + +`exportAnalyzeResumeItems` calls `config.ArtifactFamilies(catalog)` inside its +per-item loop, then reads one member's family and character ID. That accessor +deep-copies every family and member, including dependency slices and input maps. +`analyzeArtifactOrigin` performs the same whole-catalog copy for individual +artifact checkpoint and diagnostic records. + +Let `L` be the number of origin lookups and `K` the total family/member catalog +payload. These lookups cost `O(L * K)` work and cumulative allocation. When a +plan reports a catalog-sized set of bounded-size members, `L` and `K` both grow +with member count, producing quadratic copy work for a linear-sized report. +This inefficiency predates the migration and remains in the rewired domain +consumers; it is an optimization opportunity, rather than a new regression. + +Refactoring opportunity: + +- Capture one detached family catalog outside a batch and reuse its member + index across all three resume-summary lists. Copy work becomes `O(K + L)`. +- For execution's individual lookups, read the already-derived member index + through a narrow accessor that returns the needed identity fields by value. + If a caller needs mutable dependency/input records, copy that member only. +- Preserve family identities, deterministic report ordering, and detached + records where callers need them. A new defensive-copy framework or global + cache is unnecessary. + +## 7. P3: Build publication-report lookup tables once + +Stage: 12. + +Locations: `internal/app/session_state_command.go:256–270`, +`internal/app/artifact_inspection.go:79–131`, and +`internal/artifacts/catalog.go:255–289`. + +For every publication rule, `inspectPublishedOutputs` calls +`helperPublishedOutputDest`, which rebuilds the configured-output map and +extraction-output set. Building each map first enumerates, sorts, and copies +the entire corresponding runtime catalog. The same loop calls +`committedPublishedOutput`, scanning all commit declarations to find one source. + +Let `R` be publication-rule count, `A` configured-artifact count, `X` +extraction-artifact count, and `K` commit-declaration count. With bounded-size +catalog records, the repeated local work is +`O(R * (A log A + X log X + K))`, plus copying any variable-sized entry payloads. +When `R`, `A`, and `K` grow together, this includes quadratic-logarithmic sorting +work and quadratic declaration scans. Required remote existence checks are +separate from this CPU/allocation cost. These helpers and repeated lookups +predate the migration and remain in the new structured-report collector. + +Refactoring opportunity: + +- Build configured/extraction lookup tables once per report and pass them to + destination resolution, or share the existing publication-action planner + where its selection and failure behavior fit inspection. +- Index published commit declarations once by source, preserving the existing + declaration-selection behavior. Resolve each report row through that index. +- Reusing the existing catalog enumeration APIs reduces this local lookup work + to `O(A log A + X log X + K + R)`, plus entry-payload copying. Keep configured + report ordering, partial error reporting, pointer-selected authority, exact + declared object keys, and the existing remote checks. + +There is no need to add remote concurrency, caching across invocations, or a new +publication protocol to obtain this improvement. + +## 8. P2: Use canonical publication sources in session reports + +Stage: 12; assembled correctness verification in Stage 21. + +Location: `internal/app/session_state_command.go:256–270`. + +Publication validation and execution accept a source with surrounding whitespace +and resolve it to its trimmed artifact identity. The new report collector uses +the authored `rule.Source` unchanged for lock, outcome, and committed-declaration +lookups, although its destination helper trims the same source. + +A focused diagnostic loads and validates `source: +" narratio.transcript.final_trimmed "` through the actual configuration loaders. +The shared publication planner produces `narratio.transcript.final_trimmed`. +Given a matching committed artifact and lock, inspection reports `missing`, +`unlocked`, an empty outcome, and no declared key. The otherwise identical +unspaced case reports `published`, `locked`, and `carried_locked`, with the +declared object identity. The previous collector normalized the source before +these lookups, making this a migration regression. + +Required remediation: + +- Use the same canonical source identity as publication for every report + lookup and the report's source field. Derive this identity locally or consume + the shared publication-action representation; keep completed ConfigKit values + unchanged. +- Preserve accepted configuration, rather than adding a whitespace prohibition. + Retain exact committed destination and origin-run reporting, including carried + outputs. +- Add a collector/native-report regression with an accepted spaced source, + committed output, lock, and outcome, alongside an unspaced control. + +Canonicalizing once per row is linear in the source's string length and adds no +catalog-wide traversal. This correctness fix is independent of the repeated +sorting/scanning optimization in finding 7. + ## Stage-by-stage coverage | Stage | Evidence reviewed and result | @@ -260,3 +410,49 @@ all passed on Go 1.27.4. A new native-invocation Go-overlay diagnostic reproduce finding 4 under `-race` and verified unchanged prepared input checksums. Its files remained under `/tmp`. Documentation checks and whitespace validation were repeated for this document update. + +## Algorithmic follow-up coverage and validation + +The algorithmic pass reviewed the completed stages in order, using the same +application source and the exact pinned dependency implementations. Findings 5–8 +add three scaling opportunities and one correctness regression. It preserves the +algorithms and consequential behavior expressly selected in the plan. + +| Stage | Algorithmic review | +| --- | --- | +| 1 | Fixed command declarations and native outcome adaptation; parser/reporting algorithms remain upstream-owned. | +| 2 | Native-result and source-metadata access; repeated catalog snapshots are covered by finding 5. | +| 3 | Presence-based defaults and fixed stable-input fields; no additional scaling finding. | +| 4 | Family expansion is output-sensitive in families × characters; maps support collision checks. Finding 5 covers avoidable metadata copies beyond that output work. | +| 5 | Concrete dependency/input consumers and family-origin access; finding 6 covers whole-catalog copies per lookup. | +| 6 | Safe materialization and semantic projections; retained checkpoint and publication algorithms were not reopened. | +| 7 | Invocation-local lazy acquisition reuses acquired products and keeps comparison sides separate. | +| 8 | Native scope dispatch and actual-result inspection; no replacement traversal algorithm is introduced. | +| 9 | Independent side acquisition and upstream comparison/override semantics; no additional scaling finding. | +| 10 | Artifact-name token processing uses a set and sorting; family selection expands the selected members, and stage selection operates on the fixed inventory. | +| 11 | Planning and catalog reporting use derived products; findings 5 and 6 apply before/during report construction. | +| 12 | Artifact/publication report collection; findings 7 and 8 cover repeated lookup construction and inconsistent source normalization. | +| 13 | Initialization and lock commands retain shared operations; no additional scaling finding. | +| 14 | Restore/cleanup action reports retain their existing planners and safety rules; no additional scaling finding in the conversion. | +| 15 | Durable summary collection traverses recorded outputs/checkpoints with per-stage sets and sorted keys; finding 6 also affects individual analysis records. | +| 16 | Cancellation propagation and worker settlement retain selected execution/checkpoint behavior. | +| 17 | Native diagnostic adaptation adds no growing history or extra durable progress collection. | +| 18 | Per-unit progress attaches to existing audio, speaker, and artifact completion/reuse events. | +| 19 | Static completion and legacy deletion add no campaign/artifact enumeration algorithm. | +| 20 | Documentation-only stage; no runtime algorithm to reassess. | +| 21 | Cross-checked assembled paths, selected dependency source, and focused correctness reproduction. | + +Complexity statements are derived from loops, call paths, and copied payloads; +they are not benchmark results or predictions of measured operator latency. +The pass did not reconsider upstream configuration algorithms, fixed pipeline +ordering/invalidation, immediate durable checkpoints, or the selected publication +and restore protocols. + +The focused Go-overlay diagnostic used real configuration loading/validation and +the shared publication planner with in-memory remote state. Its canonical-source +control passed; its spaced-source case failed the expected-state assertions as +described in finding 8. Diagnostic files remained under `/tmp`, with no repository +test or application changes. Earlier full application verification remains +applicable to the unchanged source. Documentation checks and `git diff --check` +were repeated for this update. No live services or performance measurements were +used.