diff --git a/docs/roadmap/audit/02-efficiency.md b/docs/roadmap/audit/02-efficiency.md new file mode 100644 index 0000000..7a94f4d --- /dev/null +++ b/docs/roadmap/audit/02-efficiency.md @@ -0,0 +1,223 @@ +# Algorithmic correctness and efficiency audit + +Reviewed the implementation at `0eca7b8`. Complexity estimates come from +source and call paths. Small synthetic probes checked match expansion, +ambiguous spans, timestamp ties, and strict parsing. The audit used no +production workloads or provider measurements. +The full offline suite passed with `go test -count=1 ./...` on Go 1.27.3. + +## Architecture and review scope + +Audita is a single-process Go CLI. CommandKit routes commands and streams, +ConfigKit resolves configuration, and the core packages parse, normalize, and +chunk transcripts and project glossary selections. The runner executes +correction modules in order over fixed chunks. Deterministic and LLM-backed +validators check proposals and each repair revision. PromptKit owns provider +admission, transport, and conversation coordination. The output session +publishes corrected transcripts, ordinary reports, and optional debug evidence. +[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) + +The review followed work from configuration and normalization through proposal +application and evidence assembly. It preserves the selected forward chunker, +stage barriers, replacement policies, PromptKit admission and replay, and +validator budgeting rules. Costs required by emitted payloads or full report +matrices are treated as output costs. The earlier correctness findings are +recorded separately. [Pipeline](../../internal/pipeline.md) +· [Validator budgeting](../../internal/validators.md#supplemental-validation-context) +· [Deferred traversal design](../future.md#independent-whole-pipeline-traversal-by-chunk) +· [Correctness audit](01-correctness.md) + +## Efficiency and scalability findings + +### 1. Normalization repeatedly copies and scans growing merge runs + +`shouldMergeWithReason` constructs the merged text and estimates its tokens +from scratch. On success, `mergeSegments` constructs the text again and +allocates a new category slice that copies the entire prefix. For a run of +`k` merged segments with fixed-size nonempty text and category lists, the +merge pass reads and copies `Θ(k²)` text bytes and category elements. +The exact bound is proportional to the sum of the merge-prefix sizes. +The token cap usually limits the run, but +nonempty whitespace-only fragments with short gaps can merge at zero estimated +tokens. The cost is therefore possible under the default token cap. +[Merge loop](../../../internal/core/normalization/normalize.go#L46) +· [Merge check and construction](../../../internal/core/normalization/normalize.go#L153) +· [Token estimator](../../../internal/core/normalization/tokens.go#L14) +· [Accepted segment text](../../integrations/transcript-glossary-files.md#transcript-json-contract) + +Keep the forward merge pass. Accumulate text in an amortized builder and append +categories to an owned buffer. Maintain unrounded word and punctuation counts +per segment and combine them with the joiner before testing the token cap. +The ellipsis joiner can start a word when the previous text ends in whitespace, +and punctuation must be rounded only after adding all components. Check time +limits before processing text. This makes the merge pass linear in text bytes +and category elements, in addition to the existing chronological sort. + +### 2. Application reporting searches the source chunk for each skipped edit + +Application builds a segment-ID index inside `ApplyProposals`. The subsequent +`attributeApplyFiltering` pass indexes the approved proposals and sections, +but searches the original section's segment slice again for each skipped +proposal. With `s` segments and `k` skips in a section, this is +`O(s × k)` segment comparisons. Two identical, approved corrections per +segment make the second skip after the first changes its span. With one skip +per segment, the scans visit exactly `s(s+1)/2` entries. These are ordinary +stale conflicts, so the work can produce zero filtering records. +[Indexed application](../../../internal/framework/proposals/apply.go#L42) +· [Skip attribution](../../../internal/framework/runner/runner.go#L877) + +Return immediately when there are no skips. For affected sections, index the +original segments once or carry original applicability forward from validation. +This reduces target lookup to expected `O(s + k)` work per affected section. +If defensive apply-time filtering adds actual defects, merge them once per +originating operation. The current singleton merge can rebuild the growing +deduplication map and sort the list after each addition. Preserve the +distinction between a response defect and a stale conflict. +[Filtering aggregation](../../../internal/framework/runner/runner.go#L837) +· [Recovery contract](../../integrations/output-files.md#error-and-recovery-records) + +### 3. Validator glossary context expands every occurrence into every contributor + +Context uses `ProtectedVocabulary.occurrences`, which emits one record per +matched occurrence per glossary identity. The context builder immediately +deduplicates those records into a set. For a spelling that occurs `m` times +and belongs to `g` selected entries, this costs `Θ(m × g)` records and +identity insertions after matching. A synthetic probe with 128 occurrences +of an alias shared by 96 entries produced 12,288 records to select 96 +definitions. The deterministic protection check needs occurrence multiplicity. +The context builder only needs membership. [Occurrence expansion](../../../internal/framework/validators/protected_terms.go#L76) +· [Context selection](../../../internal/framework/validators/llm_context.go#L40) + +Share the same scanner and match precedence. Let context deduplicate matched +spellings across the source, result, and neighbors before expanding their +identity lists. This case then takes expected `O(m + g)` membership work +before ordering the selected entries. Keep occurrence-level evidence for the +protection check and preserve glossary selection order. + +### 4. Validation measures optional context that cannot fit + +The production estimator correctly composes encoded fragment counts. Its +fitter nevertheless serializes all optional glossary entries and neighbors, +builds both prefix tables, and then removes entries from the end. It does +that even when the required fields alone exceed the budget. For `p` items +sharing `b` bytes of optional context across `g` entries, this can repeat +`Θ(p × b)` serialization and allocate `Θ(g)` prefix counters per fit even +when no optional entry is retained. +[Prefix measurement and fitting](../../../internal/framework/validators/llm_budget.go#L42) +· [Production batching caller](../../../internal/framework/validators/llm_batching.go#L17) + +Return the required-only result as soon as its cost exceeds the budget. +Otherwise measure glossary entries in order and stop at the first one that +makes the prefix exceed the budget. Measure neighbors only if the entire +glossary prefix fits. Costs are monotone, so this preserves the existing policy +of dropping neighbors before definitions and keeping the longest affordable +ordered prefixes. An incremental fit uses constant auxiliary space for +counters and meters only retained entries plus the first exclusion in each +eligible field. Reusable glossary entry costs can also be stored with the +immutable glossary view if repeated encodings remain material. The cost of +matching candidates to glossary entries is separate. +[Context priority](../../internal/validators.md#supplemental-validation-context) + +### 5. Ordinary runs retain debug-only decision history + +The runner only adds candidates to the returned candidate list when debug +capture is active. It still builds a per-module candidate slice on every run +and aggregates, sorts, and stores initial validator decisions and rejections +in every `ModuleResult`. Rejected records carry original and proposed text +references. With `d_i` decision and rejection records in module `i`, finished +modules retain `Θ(Σ d_i)` records through invocation finalization, and the +runner pays for ordering them even when their production consumer, the +correction ledger, is disabled. Ordinary reports read the work-unit states and +compact operation outcomes. The current section's decisions and rejections +still serve candidate validation and repair eligibility. +[Candidate collection](../../../internal/framework/runner/runner.go#L537) +· [History sorting](../../../internal/framework/runner/runner.go#L617) +· [Module retention](../../../internal/framework/runner/runner.go#L271) +· [Ordinary reporting](../../../internal/framework/processreport/report.go#L51) +· [Debug ledger wiring](../../../internal/cli/process_pipeline.go#L54) + +Separate detailed evidence retention from section-local decision state. In +the compact CLI path, skip copying candidates into aggregate slices and release +decision and rejection detail once the section finishes. Keep the operation +outcomes, counts, and failure origin used by ordinary reporting. Debug runs can +continue to retain and order the full evidence, including partially completed +work after a fatal return. + +### 6. Repeated pipeline selections duplicate setup work + +Module specifications select and copy the rich entries and project the +protected vocabulary independently for each pipeline entry, before the runner +starts. For `m` entries with the same selections of size `g`, setup time and +retained projections grow as `Θ(m × g)` with bounded-size entry fields. The +same data is used to prepare stage prompt variables and compile matchers again +on every repeated stage. Separately, readiness inspects a profile and its +backend for every role, even when they share IDs. PromptKit returns detached +inspection data, including copies of extra parameters. Selecting the same +resources `m` times with `b` extra parameter values can add `Θ(m × b)` +inspection work before processing. +[Module projection](../../../internal/framework/contracts/contracts.go#L107) +· [Entry copies](../../../internal/core/glossary/selection.go#L20) +· [Glossary preparation](../../../internal/framework/runner/runner.go#L169) +· [Role readiness](../../../internal/framework/preflight/preflight.go#L129) +· [Profile and backend inspection](../../../internal/cli/process_inputs.go#L125) + +Reuse read-only projections and prepared data for identical effective ordered +selections, including the glossary correction exemption in the key. +Alternatively, build the projection at the module boundary and release it +before the next stage to bound retained memory when selections differ. Cache +successful profile and backend inspections for the captured engine while +keeping role-by-role checks and first-failure attribution. Shared data keeps +identical views and inspections proportional to the number of distinct inputs +plus the number of stage entries. Request messages still carry the selected +context for each call. + +## Additional correctness findings + +### 7. The unique-span check accepts overlapping matches + +The grammar prompt requires an original span to appear exactly once. +`PreviewProposalForSegment` uses `strings.Count`, which counts nonoverlapping +matches. A probe proposed changing `ha ha` to `heh` in `ha ha ha`. The span +starts at byte offsets 0 and 3, but the preview returned applicable with +`heh ha` as the result. The same preview governs validation and application. +[Unique-span instruction](../../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml#L19) +· [Preview](../../../internal/framework/proposals/preview.go#L35) +· [Application](../../../internal/framework/proposals/apply.go#L59) + +For `require_unique`, locate the first occurrence and check for another +starting after its first byte, including an overlapping start. Stop as soon +as ambiguity is established. A successful check can use the recorded offset +to assemble the replacement. This retains a linear scan in the worst case for +a fixed literal and can exit early on ambiguity. Preserve the existing +nonoverlapping replacement behavior for `replace_all`. + +### 8. Normalization can reorder utterances with exactly tied timestamps + +`Normalize` compares only start times in an unstable sort. With 14 segments +whose starts alternate between 1 and 0 and whose ends are start plus one, a +normalization-only CLI run emitted the start-zero group in source-index order +`9,1,13,3,7,5,11`. Those segments have identical start and end times. With +distinct speakers, the ordering change occurs without any merge. The earlier +Python implementation at `0b1b670` ordered exact ties by source position, and +the Go rewrite recorded preservation of existing behavior as its baseline. +[Current sort](../../../internal/core/normalization/normalize.go#L58) + +Preserve the input order for exact time ties. A stable sort or an explicit +source-position tiebreaker retains `O(n log n)` comparisons. Specify the +timestamp tiebreaker so later refactoring preserves utterance order. + +### 9. The strict transcript parser panics on a missing ID + +`ParseTranscriptJSON` calls the source parser, which allows omitted and null +IDs, then dereferences every ID before checking that IDs are sequential. A +package probe with an otherwise valid segment and no ID reproduced the nil +pointer panic. This affects the strict package helper, which currently has +test callers. The processing command uses the tolerant source parser and +assigns canonical IDs separately. +[Strict parser](../../../internal/core/schema/transcript.go#L65) +· [Sequential ID validation](../../../internal/core/schema/transcript.go#L204) +· [CLI input acquisition](../../../internal/cli/process_inputs.go#L46) + +Check ID presence and return the normal field-validation error before +dereferencing. This adds a constant-time check per segment and preserves the +linear typed conversion pass.