Record algorithmic correctness and efficiency audit
This commit is contained in:
223
docs/roadmap/audit/02-efficiency.md
Normal file
223
docs/roadmap/audit/02-efficiency.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# Algorithmic correctness and efficiency audit
|
||||
|
||||
Reviewed the implementation at `0eca7b8`. Complexity estimates come from
|
||||
source and call paths. Small synthetic probes checked match expansion,
|
||||
ambiguous spans, timestamp ties, and strict parsing. The audit used no
|
||||
production workloads or provider measurements.
|
||||
The full offline suite passed with `go test -count=1 ./...` on Go 1.27.3.
|
||||
|
||||
## Architecture and review scope
|
||||
|
||||
Audita is a single-process Go CLI. CommandKit routes commands and streams,
|
||||
ConfigKit resolves configuration, and the core packages parse, normalize, and
|
||||
chunk transcripts and project glossary selections. The runner executes
|
||||
correction modules in order over fixed chunks. Deterministic and LLM-backed
|
||||
validators check proposals and each repair revision. PromptKit owns provider
|
||||
admission, transport, and conversation coordination. The output session
|
||||
publishes corrected transcripts, ordinary reports, and optional debug evidence.
|
||||
[Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md)
|
||||
|
||||
The review followed work from configuration and normalization through proposal
|
||||
application and evidence assembly. It preserves the selected forward chunker,
|
||||
stage barriers, replacement policies, PromptKit admission and replay, and
|
||||
validator budgeting rules. Costs required by emitted payloads or full report
|
||||
matrices are treated as output costs. The earlier correctness findings are
|
||||
recorded separately. [Pipeline](../../internal/pipeline.md)
|
||||
· [Validator budgeting](../../internal/validators.md#supplemental-validation-context)
|
||||
· [Deferred traversal design](../future.md#independent-whole-pipeline-traversal-by-chunk)
|
||||
· [Correctness audit](01-correctness.md)
|
||||
|
||||
## Efficiency and scalability findings
|
||||
|
||||
### 1. Normalization repeatedly copies and scans growing merge runs
|
||||
|
||||
`shouldMergeWithReason` constructs the merged text and estimates its tokens
|
||||
from scratch. On success, `mergeSegments` constructs the text again and
|
||||
allocates a new category slice that copies the entire prefix. For a run of
|
||||
`k` merged segments with fixed-size nonempty text and category lists, the
|
||||
merge pass reads and copies `Θ(k²)` text bytes and category elements.
|
||||
The exact bound is proportional to the sum of the merge-prefix sizes.
|
||||
The token cap usually limits the run, but
|
||||
nonempty whitespace-only fragments with short gaps can merge at zero estimated
|
||||
tokens. The cost is therefore possible under the default token cap.
|
||||
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
|
||||
· [Merge check and construction](../../../internal/core/normalization/normalize.go#L153)
|
||||
· [Token estimator](../../../internal/core/normalization/tokens.go#L14)
|
||||
· [Accepted segment text](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
||||
|
||||
Keep the forward merge pass. Accumulate text in an amortized builder and append
|
||||
categories to an owned buffer. Maintain unrounded word and punctuation counts
|
||||
per segment and combine them with the joiner before testing the token cap.
|
||||
The ellipsis joiner can start a word when the previous text ends in whitespace,
|
||||
and punctuation must be rounded only after adding all components. Check time
|
||||
limits before processing text. This makes the merge pass linear in text bytes
|
||||
and category elements, in addition to the existing chronological sort.
|
||||
|
||||
### 2. Application reporting searches the source chunk for each skipped edit
|
||||
|
||||
Application builds a segment-ID index inside `ApplyProposals`. The subsequent
|
||||
`attributeApplyFiltering` pass indexes the approved proposals and sections,
|
||||
but searches the original section's segment slice again for each skipped
|
||||
proposal. With `s` segments and `k` skips in a section, this is
|
||||
`O(s × k)` segment comparisons. Two identical, approved corrections per
|
||||
segment make the second skip after the first changes its span. With one skip
|
||||
per segment, the scans visit exactly `s(s+1)/2` entries. These are ordinary
|
||||
stale conflicts, so the work can produce zero filtering records.
|
||||
[Indexed application](../../../internal/framework/proposals/apply.go#L42)
|
||||
· [Skip attribution](../../../internal/framework/runner/runner.go#L877)
|
||||
|
||||
Return immediately when there are no skips. For affected sections, index the
|
||||
original segments once or carry original applicability forward from validation.
|
||||
This reduces target lookup to expected `O(s + k)` work per affected section.
|
||||
If defensive apply-time filtering adds actual defects, merge them once per
|
||||
originating operation. The current singleton merge can rebuild the growing
|
||||
deduplication map and sort the list after each addition. Preserve the
|
||||
distinction between a response defect and a stale conflict.
|
||||
[Filtering aggregation](../../../internal/framework/runner/runner.go#L837)
|
||||
· [Recovery contract](../../integrations/output-files.md#error-and-recovery-records)
|
||||
|
||||
### 3. Validator glossary context expands every occurrence into every contributor
|
||||
|
||||
Context uses `ProtectedVocabulary.occurrences`, which emits one record per
|
||||
matched occurrence per glossary identity. The context builder immediately
|
||||
deduplicates those records into a set. For a spelling that occurs `m` times
|
||||
and belongs to `g` selected entries, this costs `Θ(m × g)` records and
|
||||
identity insertions after matching. A synthetic probe with 128 occurrences
|
||||
of an alias shared by 96 entries produced 12,288 records to select 96
|
||||
definitions. The deterministic protection check needs occurrence multiplicity.
|
||||
The context builder only needs membership. [Occurrence expansion](../../../internal/framework/validators/protected_terms.go#L76)
|
||||
· [Context selection](../../../internal/framework/validators/llm_context.go#L40)
|
||||
|
||||
Share the same scanner and match precedence. Let context deduplicate matched
|
||||
spellings across the source, result, and neighbors before expanding their
|
||||
identity lists. This case then takes expected `O(m + g)` membership work
|
||||
before ordering the selected entries. Keep occurrence-level evidence for the
|
||||
protection check and preserve glossary selection order.
|
||||
|
||||
### 4. Validation measures optional context that cannot fit
|
||||
|
||||
The production estimator correctly composes encoded fragment counts. Its
|
||||
fitter nevertheless serializes all optional glossary entries and neighbors,
|
||||
builds both prefix tables, and then removes entries from the end. It does
|
||||
that even when the required fields alone exceed the budget. For `p` items
|
||||
sharing `b` bytes of optional context across `g` entries, this can repeat
|
||||
`Θ(p × b)` serialization and allocate `Θ(g)` prefix counters per fit even
|
||||
when no optional entry is retained.
|
||||
[Prefix measurement and fitting](../../../internal/framework/validators/llm_budget.go#L42)
|
||||
· [Production batching caller](../../../internal/framework/validators/llm_batching.go#L17)
|
||||
|
||||
Return the required-only result as soon as its cost exceeds the budget.
|
||||
Otherwise measure glossary entries in order and stop at the first one that
|
||||
makes the prefix exceed the budget. Measure neighbors only if the entire
|
||||
glossary prefix fits. Costs are monotone, so this preserves the existing policy
|
||||
of dropping neighbors before definitions and keeping the longest affordable
|
||||
ordered prefixes. An incremental fit uses constant auxiliary space for
|
||||
counters and meters only retained entries plus the first exclusion in each
|
||||
eligible field. Reusable glossary entry costs can also be stored with the
|
||||
immutable glossary view if repeated encodings remain material. The cost of
|
||||
matching candidates to glossary entries is separate.
|
||||
[Context priority](../../internal/validators.md#supplemental-validation-context)
|
||||
|
||||
### 5. Ordinary runs retain debug-only decision history
|
||||
|
||||
The runner only adds candidates to the returned candidate list when debug
|
||||
capture is active. It still builds a per-module candidate slice on every run
|
||||
and aggregates, sorts, and stores initial validator decisions and rejections
|
||||
in every `ModuleResult`. Rejected records carry original and proposed text
|
||||
references. With `d_i` decision and rejection records in module `i`, finished
|
||||
modules retain `Θ(Σ d_i)` records through invocation finalization, and the
|
||||
runner pays for ordering them even when their production consumer, the
|
||||
correction ledger, is disabled. Ordinary reports read the work-unit states and
|
||||
compact operation outcomes. The current section's decisions and rejections
|
||||
still serve candidate validation and repair eligibility.
|
||||
[Candidate collection](../../../internal/framework/runner/runner.go#L537)
|
||||
· [History sorting](../../../internal/framework/runner/runner.go#L617)
|
||||
· [Module retention](../../../internal/framework/runner/runner.go#L271)
|
||||
· [Ordinary reporting](../../../internal/framework/processreport/report.go#L51)
|
||||
· [Debug ledger wiring](../../../internal/cli/process_pipeline.go#L54)
|
||||
|
||||
Separate detailed evidence retention from section-local decision state. In
|
||||
the compact CLI path, skip copying candidates into aggregate slices and release
|
||||
decision and rejection detail once the section finishes. Keep the operation
|
||||
outcomes, counts, and failure origin used by ordinary reporting. Debug runs can
|
||||
continue to retain and order the full evidence, including partially completed
|
||||
work after a fatal return.
|
||||
|
||||
### 6. Repeated pipeline selections duplicate setup work
|
||||
|
||||
Module specifications select and copy the rich entries and project the
|
||||
protected vocabulary independently for each pipeline entry, before the runner
|
||||
starts. For `m` entries with the same selections of size `g`, setup time and
|
||||
retained projections grow as `Θ(m × g)` with bounded-size entry fields. The
|
||||
same data is used to prepare stage prompt variables and compile matchers again
|
||||
on every repeated stage. Separately, readiness inspects a profile and its
|
||||
backend for every role, even when they share IDs. PromptKit returns detached
|
||||
inspection data, including copies of extra parameters. Selecting the same
|
||||
resources `m` times with `b` extra parameter values can add `Θ(m × b)`
|
||||
inspection work before processing.
|
||||
[Module projection](../../../internal/framework/contracts/contracts.go#L107)
|
||||
· [Entry copies](../../../internal/core/glossary/selection.go#L20)
|
||||
· [Glossary preparation](../../../internal/framework/runner/runner.go#L169)
|
||||
· [Role readiness](../../../internal/framework/preflight/preflight.go#L129)
|
||||
· [Profile and backend inspection](../../../internal/cli/process_inputs.go#L125)
|
||||
|
||||
Reuse read-only projections and prepared data for identical effective ordered
|
||||
selections, including the glossary correction exemption in the key.
|
||||
Alternatively, build the projection at the module boundary and release it
|
||||
before the next stage to bound retained memory when selections differ. Cache
|
||||
successful profile and backend inspections for the captured engine while
|
||||
keeping role-by-role checks and first-failure attribution. Shared data keeps
|
||||
identical views and inspections proportional to the number of distinct inputs
|
||||
plus the number of stage entries. Request messages still carry the selected
|
||||
context for each call.
|
||||
|
||||
## Additional correctness findings
|
||||
|
||||
### 7. The unique-span check accepts overlapping matches
|
||||
|
||||
The grammar prompt requires an original span to appear exactly once.
|
||||
`PreviewProposalForSegment` uses `strings.Count`, which counts nonoverlapping
|
||||
matches. A probe proposed changing `ha ha` to `heh` in `ha ha ha`. The span
|
||||
starts at byte offsets 0 and 3, but the preview returned applicable with
|
||||
`heh ha` as the result. The same preview governs validation and application.
|
||||
[Unique-span instruction](../../../internal/prompts/assets/prompts/modules.grammar.proposal.yaml#L19)
|
||||
· [Preview](../../../internal/framework/proposals/preview.go#L35)
|
||||
· [Application](../../../internal/framework/proposals/apply.go#L59)
|
||||
|
||||
For `require_unique`, locate the first occurrence and check for another
|
||||
starting after its first byte, including an overlapping start. Stop as soon
|
||||
as ambiguity is established. A successful check can use the recorded offset
|
||||
to assemble the replacement. This retains a linear scan in the worst case for
|
||||
a fixed literal and can exit early on ambiguity. Preserve the existing
|
||||
nonoverlapping replacement behavior for `replace_all`.
|
||||
|
||||
### 8. Normalization can reorder utterances with exactly tied timestamps
|
||||
|
||||
`Normalize` compares only start times in an unstable sort. With 14 segments
|
||||
whose starts alternate between 1 and 0 and whose ends are start plus one, a
|
||||
normalization-only CLI run emitted the start-zero group in source-index order
|
||||
`9,1,13,3,7,5,11`. Those segments have identical start and end times. With
|
||||
distinct speakers, the ordering change occurs without any merge. The earlier
|
||||
Python implementation at `0b1b670` ordered exact ties by source position, and
|
||||
the Go rewrite recorded preservation of existing behavior as its baseline.
|
||||
[Current sort](../../../internal/core/normalization/normalize.go#L58)
|
||||
|
||||
Preserve the input order for exact time ties. A stable sort or an explicit
|
||||
source-position tiebreaker retains `O(n log n)` comparisons. Specify the
|
||||
timestamp tiebreaker so later refactoring preserves utterance order.
|
||||
|
||||
### 9. The strict transcript parser panics on a missing ID
|
||||
|
||||
`ParseTranscriptJSON` calls the source parser, which allows omitted and null
|
||||
IDs, then dereferences every ID before checking that IDs are sequential. A
|
||||
package probe with an otherwise valid segment and no ID reproduced the nil
|
||||
pointer panic. This affects the strict package helper, which currently has
|
||||
test callers. The processing command uses the tolerant source parser and
|
||||
assigns canonical IDs separately.
|
||||
[Strict parser](../../../internal/core/schema/transcript.go#L65)
|
||||
· [Sequential ID validation](../../../internal/core/schema/transcript.go#L204)
|
||||
· [CLI input acquisition](../../../internal/cli/process_inputs.go#L46)
|
||||
|
||||
Check ID presence and return the normal field-validation error before
|
||||
dereferencing. This adds a constant-time check per segment and preserves the
|
||||
linear typed conversion pass.
|
||||
Reference in New Issue
Block a user