Record algorithmic scaling opportunities
This commit is contained in:
@@ -10,14 +10,18 @@ Integration follow-up baseline: `fd134e85e54c6e0f74aab5c77df5bada7c1b1e6a`.
|
||||
The follow-up baseline adds only the initial review findings; application code
|
||||
is unchanged between these revisions.
|
||||
|
||||
Algorithmic follow-up baseline: `fdaf39162b46b528fddcd8a4cad18a09e1daf661`.
|
||||
Only this findings document changed after the application baseline above.
|
||||
|
||||
This review followed the 46 stages in [the implementation plan](implementation.md)
|
||||
against the [native migration target](promptkit.md), current owning code,
|
||||
focused tests, maintained examples, and canonical documentation. The migration
|
||||
is substantially complete. Findings 1–2 come from the initial review. Findings
|
||||
3–5 come from a second, stage-ordered pass through the interaction of the native
|
||||
migration with existing domain, validation, and diagnostic behavior. None calls
|
||||
for another compatibility layer or application execution abstraction. This
|
||||
document records findings, not implemented fixes.
|
||||
3–5 come from an integration pass. Findings 6–9 and the complexity note on
|
||||
Finding 1 come from a stage-ordered algorithmic pass. The latter are refactoring
|
||||
opportunities for consideration, not measured production bottlenecks or new
|
||||
performance guarantees. None calls for another compatibility layer or application
|
||||
execution abstraction. This document records findings, not implemented fixes.
|
||||
|
||||
## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection
|
||||
|
||||
@@ -69,6 +73,18 @@ application error taxonomy is needed.
|
||||
the existing native timeout fixture where useful. Run the pipeline tests and
|
||||
their concurrency/race checks.
|
||||
|
||||
### Algorithmic Follow-up: Select A Minimum Instead Of Sorting
|
||||
|
||||
With `E` collected errors, the current selector comparison-sorts all eligible
|
||||
records by `(stage, lane, chunk)` solely to return the first. It performs
|
||||
`O(E log E)` comparisons; stable sorting can also incur additional superlinear
|
||||
data movement. After eligibility classification, a minimum scan takes `O(E)`
|
||||
comparisons and constant auxiliary space; error-chain classification costs remain
|
||||
separate. Keep the first occurrence on equal keys, matching stable-sort behavior.
|
||||
This can accompany the correctness
|
||||
fix without changing error precedence or cancellation semantics. Its practical
|
||||
impact is smaller than the document/cache indexing findings below.
|
||||
|
||||
## Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee
|
||||
|
||||
Priority: P2. Relevant stages: 11, 38, 39, and 42.
|
||||
@@ -281,6 +297,160 @@ assessment/capture error types. Do not bridge native errors into these retired
|
||||
sentinels or add aliases for compatibility. Verify no current callers remain and
|
||||
run contracts, pipeline, CLI, and module integration tests after removal.
|
||||
|
||||
## Finding 6: Use Indexed Source Validation In The Chunk-Plan Hot Path
|
||||
|
||||
Priority: P2. Relevant stages: 16, 24, 28–29, and 37.
|
||||
|
||||
### Evidence And Complexity
|
||||
|
||||
The supporting [source validation code](../../internal/core/source/validation.go)
|
||||
predates this migration, but the new chunk-plan assessment/cache path repeatedly
|
||||
uses it. `ValidateDocument`, lines 95–101, validates each unit's self-reference
|
||||
with `ValidateRef`. That helper resolves both endpoints through the linear
|
||||
`UnitIndex` scan. For a valid document of `N` units, the second validation pass
|
||||
alone makes `2 * (1 + ... + N) = N(N+1)` unit-ID comparisons: `Theta(N^2)` work.
|
||||
|
||||
[`ValidateChunkPlan` and `MaterializeChunkPlan`](../../internal/core/source/chunk_plan.go)
|
||||
also repeatedly scan the source for each of `C` ranges. A materialization thus
|
||||
performs `O(N^2 + CN)` source-position work in addition to field checking,
|
||||
annotation processing, metadata ownership, and payload encoding.
|
||||
The pipeline's
|
||||
[`validateAndMaterializeChunkPlan`](../../internal/framework/pipeline/chunk_validation.go)
|
||||
validates once explicitly and then calls a materializer that validates again.
|
||||
The [new plan path](../../internal/framework/pipeline/runner_chunk_plan.go)
|
||||
invokes this during assessment, cache encode/decode, and final delivery. Cached
|
||||
plans therefore retain this deterministic cost even without inference.
|
||||
|
||||
### Potential Refactoring
|
||||
|
||||
Use the existing immutable `source.DocumentIndex` for bulk endpoint lookups.
|
||||
Build/reuse a unit-position map while validating the document, then check its
|
||||
self-references in linear expected time. Share resolved positions through a
|
||||
small private validation/materialization helper so materialization does not
|
||||
rescan endpoints or repeat the same validation unnecessarily. Keep standalone
|
||||
public validation/materialization entry points correctly validating their inputs.
|
||||
|
||||
The attainable position-lookup cost is `O(N + C)` expected work with `O(N)`
|
||||
index space. Field/payload inspection, annotation canonicalization, metadata
|
||||
ownership, and JSON encoding remain separate costs. This does not promise linear
|
||||
total output size: permitted overlapping generic ranges can themselves require
|
||||
superlinear copied output. Repeated materialization of accepted plans can be
|
||||
reduced where already-owned checked chunks are available, without retaining a
|
||||
whole-run ledger or removing necessary current-source checks.
|
||||
|
||||
### Preserve And Verify
|
||||
|
||||
Preserve document-order semantics for nonconsecutive/nonmonotonic unit IDs,
|
||||
duplicate/missing/wrong-source/reversed-reference rejection, annotation handling,
|
||||
deep ownership, and deterministic chunk IDs. Generic plans still permit their
|
||||
existing gaps/overlaps; D&D coverage rules remain with scene interpretation and
|
||||
configured validators. No stricter plan format or new size limit is needed.
|
||||
|
||||
Use the existing source validation/materialization, chunk-plan operation/cache,
|
||||
and chunk-map tests. Add only missing behavioral regressions; source inspection
|
||||
or optional synthetic operation counts can verify the scaling change without
|
||||
wall-clock thresholds or production data.
|
||||
|
||||
## Finding 7: Share A Lazy Source Index Across Artifact Cache Decoders
|
||||
|
||||
Priority: P2. Relevant stages: 11, 21–22, and 27–28.
|
||||
|
||||
### Evidence And Complexity
|
||||
|
||||
[`artifactProducerCache`](../../internal/framework/pipeline/runner_cache.go),
|
||||
line 17, eagerly creates a full `DocumentIndex` for each model-backed operation
|
||||
whenever a store collaborator exists. It does so before the producer helper
|
||||
checks bypass/refresh mode, incoming reuse eligibility, or whether a module takes
|
||||
a legitimate no-model branch. Cold misses also never use the resulting index.
|
||||
|
||||
For `J` such operations against `N` source units, this adds `Theta(JN)` index
|
||||
construction work. With a fixed number of lanes and chunks proportional to
|
||||
transcript length, that cache-specific term becomes quadratic in `N`.
|
||||
Concurrent operations can retain `O(min(J,W)N)` cache index entries while waiting
|
||||
for model work, where `W` is the active worker bound. This is separate from
|
||||
codec decoding and domain/validator input ownership costs.
|
||||
|
||||
### Potential Refactoring
|
||||
|
||||
Supply the cache decoders with one run-scoped, immutable source-position index.
|
||||
Acquire it lazily only when a decoded hit actually needs evidence checks, using
|
||||
ordinary concurrency-safe initialization. The cache-specific work becomes
|
||||
`O(N + R)` for all hits, where `R` is the total number of checked cached source
|
||||
references, with `O(N)` shared index space. Runs with no such checks need not
|
||||
construct the cache index at all.
|
||||
|
||||
This is a small in-memory collaborator, not another persistent cache or source
|
||||
identity framework. Keep the native message-content key, modes, hit validation,
|
||||
fallback eligibility, and no-semantic-reassessment policy. Finding 3's scene
|
||||
identity adaptation is a separate usability concern; shared indexing must not
|
||||
replace it. Other intentional domain/validator snapshots need not be weakened.
|
||||
|
||||
### Preserve And Verify
|
||||
|
||||
Retain wrong-source/missing/reversed citation rejection on hits, normal misses,
|
||||
refresh/bypass/ineligible/no-model behavior, and canceled-hit delivery. Exercise
|
||||
concurrent hits with the existing cache/runner tests and race checks. Do not add
|
||||
a public index-build-count contract or a timing-based release gate.
|
||||
|
||||
## Finding 8: Index Debug Sensitivity Paths Once
|
||||
|
||||
Priority: P3. Relevant stages: 30–31 and 33.
|
||||
|
||||
### Evidence And Complexity
|
||||
|
||||
[`newRunEvidenceRecorder`](../../internal/cli/run_capture.go), lines 64–91,
|
||||
collects all `H` historical configuration paths, then scans that full list for
|
||||
each of `P` profiles/backends across root and collection resources. Prefix
|
||||
matching therefore performs `O(PH)` work, plus path-string processing. With a
|
||||
roughly fixed number of fields per resource, `H` grows with `P`, yielding a
|
||||
quadratic resource-count term.
|
||||
|
||||
The caller correctly constructs this recorder only when capture is enabled.
|
||||
This finding concerns debug startup for large native configurations, not an
|
||||
unconditional cost on ordinary runs.
|
||||
|
||||
### Potential Refactoring
|
||||
|
||||
Walk metadata paths once and group relevant sensitive suffixes by their exact
|
||||
resource prefix. Then associate those groups with root/qualified resource IDs in
|
||||
one resource traversal. This takes `O(H + P)` path/resource visits, plus processing
|
||||
the path characters and retained masks, rather than rescanning every unrelated
|
||||
configuration path for every resource.
|
||||
|
||||
Retain native sensitivity classification, correct JSON-pointer segment unescaping,
|
||||
root/collection distinctions, nested maps/arrays, and nonmutation of native
|
||||
snapshots. Do not replace exact metadata with credential-like string searches or
|
||||
broaden omission policy. Use the existing configuration/native capture tests;
|
||||
add a mixed-resource case only if needed to verify equivalent mask routing.
|
||||
|
||||
## Finding 9: Traverse Dense Chunk Results Without Sorting Their Keys
|
||||
|
||||
Priority: P3. Relevant stages: 11, 25, and 37.
|
||||
|
||||
### Evidence And Complexity
|
||||
|
||||
[`finalizeLaneExtract`](../../internal/framework/pipeline/runner_concurrent.go),
|
||||
lines 333–349, copies result-map keys to a new slice and sorts them before
|
||||
assembling the lane. For `C` chunks this costs `O(C log C)` comparisons and
|
||||
`O(C)` temporary key storage per lane.
|
||||
|
||||
The runner's chunks are already assigned contiguous zero-based indexes by
|
||||
`MaterializeChunkPlan`; successful finalization occurs only after all of that
|
||||
lane's jobs return. The required order is therefore known, not arbitrary.
|
||||
|
||||
### Potential Refactoring And Verification
|
||||
|
||||
Traverse the known ordinal range directly for `O(C)` expected map-lookup work
|
||||
and no temporary key slice, or use an indexed result slice if it simplifies
|
||||
storage. Keep all artifact/diagnostic ordering, fallback counts, eligibility,
|
||||
failure handling, cancellation draining, and ready-job scheduling unchanged.
|
||||
No new chunker restriction is required: dense ordinals are already produced by
|
||||
the existing materializer, regardless of transcript unit-ID spacing.
|
||||
|
||||
The current runner concurrency and ready-job tests cover out-of-order completion
|
||||
and public result order. Reuse those checks; do not couple tests to a particular
|
||||
map/slice representation or add another scheduler.
|
||||
|
||||
## Review Coverage And Verification
|
||||
|
||||
Owning boundaries were inspected in implementation order:
|
||||
@@ -295,7 +465,7 @@ Owning boundaries were inspected in implementation order:
|
||||
| 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. |
|
||||
| 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. |
|
||||
|
||||
Checks completed during review:
|
||||
Checks completed during the initial review:
|
||||
|
||||
- `GOWORK=off GOPROXY=off go test ./...` passed.
|
||||
- `GOWORK=off GOPROXY=off go vet ./...` passed.
|
||||
@@ -308,7 +478,7 @@ Checks completed during review:
|
||||
- The migration diff contains no changes to model-facing JSON schemas or D&D
|
||||
intrinsic codec schema files. Historical ADR decision bodies are retained;
|
||||
supersession edits are confined to status descriptions.
|
||||
- Temporary Go-overlay probes reproduced both findings without modifying
|
||||
- Temporary Go-overlay probes reproduced Findings 1–2 without modifying
|
||||
repository code or tests. Their deliberately failing assertions are review
|
||||
evidence, separate from the passing default suite.
|
||||
|
||||
@@ -331,6 +501,40 @@ application code and dependencies were unchanged. Additional checks were:
|
||||
These probes live only in a temporary review directory. No production code,
|
||||
test, configuration, schema, or existing canonical documentation was changed.
|
||||
|
||||
The algorithmic pass followed the same stage order, inspecting actual loop/call
|
||||
costs rather than treating nested-loop graph metadata as proof of a bottleneck:
|
||||
|
||||
- Stages 1–7: native loading/delegation, bounded invocation-selection reuse,
|
||||
resource/input construction, and absence of a second application load.
|
||||
- Stages 8–11: candidate serialization/isolation, complete-chain traversal,
|
||||
native helper control flow, and runner/handoff state.
|
||||
- Stages 12–24: fixed extraction mappings, bounded reconciliation preparation,
|
||||
interval coalescing, indexed duplicate/group application, direct validators,
|
||||
deterministic paths, and the source/materialization costs in Finding 6.
|
||||
- Stages 25–29: constant-work ready-job dispatch, result/error ordering,
|
||||
filesystem byte traversal, source-index lifetime, and reuse/auxiliary assembly.
|
||||
- Stages 30–34: incremental capture, sensitivity-path routing, preflight,
|
||||
command composition, diagnostics, and publication.
|
||||
- Stages 35–46: retained/removal boundaries, example/assembled coverage, and
|
||||
documentation consistency with the algorithms and selected policies above.
|
||||
|
||||
Explicitly selected behavior was not reopened: complete validation on every
|
||||
candidate, fresh normalization application to fixed originals, native growing
|
||||
history, content-only cache identity, no semantic reassessment on hits, the ready-
|
||||
job scheduling policy, and self-contained debug records even when content repeats.
|
||||
The earlier expressly recommended per-handoff output index was also left alone.
|
||||
Sorting genuinely unordered citations/intervals and collision-safe indexed domain
|
||||
grouping are not findings merely because they contain loops or comparisons.
|
||||
|
||||
Complexity claims in Findings 6–9 follow from the current source and call sites;
|
||||
they are not throughput estimates. Position-map costs use ordinary expected
|
||||
hash-map complexity; retained field/payload processing is accounted for separately.
|
||||
The existing
|
||||
offline suites for `internal/core/source`, `internal/framework/pipeline`,
|
||||
`internal/framework/artifactcache`, `internal/framework/semanticreconcile`, and
|
||||
`internal/cli` passed again. No production workload or performance data was needed,
|
||||
and no new benchmark, timing assertion, or permanent gate was added.
|
||||
|
||||
No live provider inference was performed. This review does not establish
|
||||
production model quality or performance. Intentional changes such as the hard
|
||||
configuration/CLI cutover, native retry ownership, content-only cache identity,
|
||||
|
||||
Reference in New Issue
Block a user