19 KiB
Behavioral equivalence and shared abstraction audit
Reviewed the application at 7d064a6. Audita is a single-process Go CLI for
transcript polishing. The CLI loads configuration and owns invocation and
output coordination. Core packages handle transcript data, normalization,
chunking, and serialization. The framework runner coordinates proposal
generation, validation, repair, and application through explicit contracts.
The PromptKit adapter executes model requests, and reports are derived from
framework results. Package map ·
Pipeline ·
Dependency direction
The review traced direct comparisons and separate tests with common expected
behavior to their production implementations. Each finding states what the
assertions cover and where a shared implementation could remove duplicated
policy or mechanics. The full existing suite passed with go test -count=1 ./...
on Go 1.27.3. Testing policy
1. Share validation budget policy across measurement paths
Evidence. TestComposedValidationBudgetMatchesSerializedPayload runs the
same estimator through the optimized and generic paths on a 12-item fixture.
For every positive budget up to the full payload cost, it compares complete
batches and oversized decisions, including retained context, batch boundaries,
and token counts. Its separate payload assertion checks one-based wire
references and preserved run-level proposal IDs. The fixture includes Unicode,
JSON escapes, neighboring segments, and glossary entries. Budget comparison ·
Batching contract
Implementation. prepareValidationBatches picks the composed path for the
concrete simple estimator. It premeasures serialized items and context prefixes,
then uses additive costs. The generic estimator path serializes proposed
payloads when fitting and partitioning. Both implementations trim neighbor
entries from the end before trimming glossary entries, reject items whose
required payload still exceeds the budget, and form contiguous batches in input
order. Dispatch and partitioning ·
Composed fitting ·
Serialized fitting
Opportunity and tradeoffs. Give the context trim order one owner in
framework/validators. Share the greedy batch boundary rule if it can consume
measurements without obscuring the different cost models. Keep the composed
path additive and the generic path able to measure the actual serialized bytes
with any supplied estimator. The serialized measurement remains an independent
oracle for the optimized token calculation. Once policy is shared, an explicit
expected sequence of retained items and batch boundaries can catch policy
regressions that path-to-path equality alone would miss.
2. Share regular-file staging between transcript and report publication
Evidence. TestSubprocessOutputPermissionsHonorUmask checks the transcript
and all three reports against the same expected modes. It exercises creation
under two umasks and replacement of existing mode-0600 files under each.
TestFinalizationCollectsFailuresAndRetainsPublishedTranscript separately
checks that successfully publishing the transcript survives a later report
failure. Shared mode assertions ·
Partial finalization
Implementation. OutputSession.Prepare opens a payload in a private
temporary directory next to the transcript destination. It creates the payload
with mode 0644 subject to umask, then copies the existing file's permission
bits when replacing a transcript. writeTranscript later writes, closes, and
renames it. Reports use atomicReplace, which implements the same regular-file
staging within one call and has its own cleanup and error handling.
Transcript preparation ·
Transcript publication ·
Report replacement
Opportunity and tradeoffs. Factor temporary-file creation, permission handling, close, rename, and cleanup into a small private type in diagnostics. Let the transcript session hold it across processing and let each report replacement own it for one write. The caller still owns destination selection, the transcript's final type check, streaming, publication order, and reporting of partial failures. Keep subprocess assertions on the final files and the failure-order test at the session boundary. Output lifecycle · Related output ownership finding
3. Resolve model segment references in one place
Evidence. TestModuleProposalMapsLocalReferencesWithoutGuessing sends IDs
1 and 2 for segments with transcript IDs 71 and 903. It verifies that
returned positions map to canonical IDs, while returning 71 as a position
produces an invalid target. TestRepairRequestRefreshesPreviewAfterRetarget
uses canonical IDs 81 and 104 and checks that a repair response targeting
position 2 correctly supplies the second segment's text to the next attempt.
The separate retarget-and-approve test checks preservation of the proposal
index and an unrelated approval. Generation mapping ·
Repair mapping ·
Repair identity
Implementation. GenerateCandidates and the repair callback both check
that a correction's id is a positive in-range position and index the working
section at id - 1. Both leave the target ID at zero for an unresolved
position so proposal validation can handle it. Generation translation ·
Repair translation
Opportunity and tradeoffs. Put the conversion next to the correction
response type in proposal_generation, which repair already imports. A small
lookup can express this protocol without making the core transcript type aware
of model references. Preserve the raw reference for the conversation, the
existing invalid-target handling, and revalidation after retargeting. Repair's
proposal_ref and validation batch indices have their own namespaces. Keep
nonsequential canonical IDs in both caller tests so a positional identity bug
fails even after the callers share the lookup. Module proposal contract
4. Share structured failure facts across model operations
Evidence. The generation and validation tests establish that recoverable structured failures produce an operation with a stable reason while the caller can continue processing. The validation case checks approvals on either side of the failed batch, and the repair terminal-outcome table checks that a recoverable provider failure ends that branch as unavailable. Fatal setup fixtures check call facts at the generation boundary and previous decisions, attempted batches, and preservation of the underlying cause at the validation boundary. Recoverable generation · Recoverable validation · Repair outcomes · Fatal generation · Fatal validation
Implementation. structuredoutput.Failure already owns the typed class,
safe reason and message, and wrapped cause. Generation and validation each
unwrap the error, copy its reason or substitute internal_failure, and check
recoverability separately. Repair repeats the extraction with a different
edge case. A typed failure with an empty reason retains that empty reason in
repair, and repair checks recoverability by class alone. Failure type ·
Generation projection ·
Validation projection ·
Repair projection
Opportunity and tradeoffs. Put extraction of reason and class in a small
structuredoutput helper and keep each caller's decision about what to do with
the facts. Generation can return a failed response, validation can mark the
affected batch unavailable, and repair can end a single branch. Preserve the
wrapped cause, execution counts, and operation coordinates. Retain current
blank-reason handling until its desired semantics are decided explicitly. Keep
role tests with literal reason codes or constants and independently constructed
errors, including the tests for surviving work and fatal propagation.
5. Give optional interaction artifacts one writer contract
Evidence. Separate proposal, validation, and repair tests inject a write failure. They require a diagnostic warning and preserve the successful proposal operation, approved validation decision, or withdrawn repair branch. The proposal and validation schema-metadata tests also check the same metadata fields against each operation's registered response schema. Proposal write failure · Validation write failure · Repair write failure · Proposal schema metadata · Validation schema metadata
Implementation. Proposal generation and validators define equivalent
writer interfaces and four-path artifact values. Both convert the same fields
from llm.DiagnosticsWriter through adapters. Repair writes through that
concrete writer directly. File naming, serialization, and redaction already
live in the writer, while each caller separately converts writer failure to a
warning. Proposal contract ·
Validator contract ·
Proposal adapter ·
Validation adapter ·
Shared writer ·
Repair diagnostic write
Opportunity and tradeoffs. Define the writer interface and artifact path value once at the diagnostics boundary so the concrete writer can satisfy it without field translation. A common best-effort write helper could also report the warning and return paths according to one rule. Keep the role-specific payloads, coordinates, and credential lookups at their current boundaries. In particular, the validation adapter resolves secrets lazily, and repair can record the revision and revalidation decisions. Keep role-specific assertions for warning delivery and processing outcomes alongside the real writer's tests. Writer tests
6. Use one set of checks for inherited configuration fields
Evidence. TestResolvedSettingsValidation rejects an empty validation
profile at root and in a module entry. TestCorrectionRepairSettingsResolution
rejects a negative correction-repair budget at both locations and exercises
inheritance, explicit overrides, and clearing. Those invalid-case assertions
require a load error and do not inspect its diagnostic path. A separate test
checks that explicit null and the corresponding dependency survive in
resolution metadata. Profile validation ·
Repair settings ·
Resolution metadata
Implementation. config.validate repeats the same LLM checks for root
and per-module LLM values with different ck.Path prefixes. The duplicated
checks also include retry bounds, operation timeout, and optional repair
profiles. The validation prompt budget and description length checks repeat
at both levels as well. Root checks ·
Per-module checks
Opportunity and tradeoffs. Extract typed validation functions for the shared settings and pass the path of each value. This keeps changes to one field constraint from drifting between the two locations. Leave resolution, explicit null handling, provenance, and role-specific profile readiness with their respective owners. Preserve tests that load root and local documents independently. Path assertions would catch a helper that applies a correct rule at the wrong location. Configuration contract
7. Use the same simple token counter for text estimates
Evidence. TestSimpleTokenEstimator checks concrete counts for words,
punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization
merge-limit test independently expects that joining two single words exceeds a
one-token limit and records the corresponding skipped merge. This supports the
common word-count behavior. The normalization test does not independently fix
the punctuation weighting. Counter examples ·
Merge limit
Implementation. Normalization counts strings.Fields and then scans for
punctuation. Chunking counts word boundaries and punctuation in one pass. Both
count Unicode punctuation except ASCII apostrophes, hyphens, and underscores,
then add half the punctuation count using integer division. The chunking
counter also exposes those components for validation budget composition.
Normalization counter ·
Chunking counter ·
Validation cost calculation
Opportunity and tradeoffs. Reuse the single-pass counter and its component count, either directly or through a small core package if neither caller should own the dependency. Retain the consumer-specific budgets and tests for merge behavior, chunk behavior, and serialized validation payloads. Keep literal counts in the counter tests to detect a shared regression that would otherwise affect all consumers together. This is a shared heuristic, so altering its estimates has a wider effect than consolidating its implementation.
8. Centralize transcript copies without moving the copy boundaries
Evidence. TestRunnerDoesNotMutateInputTranscript checks the caller's
original text after a successful run. TestApplyProposalsNoMutationOfInputTranscript
compares the caller's transcript with a pre-edit snapshot after applying a
proposal. A separate application test asserts preservation of IDs, speakers,
timing, and categories on the edited output. The snapshot test uses the
production clone, and the runner test checks text only. Neither establishes
category slice independence or a convention for nil versus empty categories.
Runner ownership ·
Application ownership ·
Application metadata
Implementation. The runner copies the input for a run, and ApplyProposals
copies the input it received before any edit. Both independently rebuild every
segment and copy its categories. The proposal copy preserves an explicitly
empty category slice, while the runner's append to a nil slice collapses it.
Runner copy ·
Application copy
Opportunity and tradeoffs. Let the schema package own a transcript clone that copies segment values and then copies mutable member data. Keep both boundaries so callers retain the existing ownership guarantees. Choose and check the treatment of empty slices explicitly when moving the code. Use independent fixture values to verify metadata and ownership, since taking the expected snapshot with the shared implementation can hide a shallow-copy regression.
9. Reconcile the protection rule at validation and application
Evidence. The validator tests assert rejection for removing protected
terms and permit spoken-word repetition cleanup only when the identity
survives. TestApplicationChecksCurrentTextWithSelectedProtection demonstrates
the same preservation requirement at apply time. It applies one proposal and
skips the second, whose span is a fragment of a protected alias in the full
segment. TestFeedbackReportsProtectionAndBlankResult
also checks that a protection finding and a blank-result finding can both be
reported for one candidate. Protected-term rejection ·
Repetition policy ·
Protection at application ·
Independent feedback
Implementation. Matching and identity comparison already live in
ProtectedVocabulary. ValidateProtectedTerms starts with the proposed spans
and uses full-segment evidence for the glossary path and eligible spoken-word
repetition cleanup. The runner passes AllowsChange to application, which uses
the entire current segment and its replacement preview. Both assemble their
own decision about the before/after text and when to allow repetition. The
earlier correctness audit found that a span can hide the context needed to
recognize a protected name. Validator check ·
Application predicate ·
Runner wiring ·
Span-only correctness finding
Opportunity and tradeoffs. Use a single protection assessment that produces a reason from the applicable before-and-after segment text and the effective glossary view. Decide explicitly what evidence can support an independent finding when a proposal has no usable preview. Moving ordinary validation to segment scope is a behavior change for edits whose span would give a different answer. Keep application-time reevaluation against current text, and keep validation and application tests with their own expected outcomes and fixtures.