Files
audita/docs/roadmap/audit/10-equivalence.md

290 lines
19 KiB
Markdown

# Behavioral equivalence and shared abstraction audit
Reviewed the application at `7d064a6`. Audita is a single-process Go CLI for
transcript polishing. The CLI loads configuration and owns invocation and
output coordination. Core packages handle transcript data, normalization,
chunking, and serialization. The framework runner coordinates proposal
generation, validation, repair, and application through explicit contracts.
The PromptKit adapter executes model requests, and reports are derived from
framework results. [Package map](../../internal/overview.md) ·
[Pipeline](../../internal/pipeline.md) ·
[Dependency direction](../../policy/architecture.md#ownership-and-dependency-direction)
The review traced direct comparisons and separate tests with common expected
behavior to their production implementations. Each finding states what the
assertions cover and where a shared implementation could remove duplicated
policy or mechanics. The full existing suite passed with `go test -count=1 ./...`
on Go 1.27.3. [Testing policy](../../policy/testing.md)
## 1. Share validation budget policy across measurement paths
**Evidence.** `TestComposedValidationBudgetMatchesSerializedPayload` runs the
same estimator through the optimized and generic paths on a 12-item fixture.
For every positive budget up to the full payload cost, it compares complete
batches and oversized decisions, including retained context, batch boundaries,
and token counts. Its separate payload assertion checks one-based wire
references and preserved run-level proposal IDs. The fixture includes Unicode,
JSON escapes, neighboring segments, and glossary entries. [Budget comparison](../../../internal/framework/validators/llm_budget_test.go#L17) ·
[Batching contract](../../internal/validators.md#llm-validator-batching)
**Implementation.** `prepareValidationBatches` picks the composed path for the
concrete simple estimator. It premeasures serialized items and context prefixes,
then uses additive costs. The generic estimator path serializes proposed
payloads when fitting and partitioning. Both implementations trim neighbor
entries from the end before trimming glossary entries, reject items whose
required payload still exceeds the budget, and form contiguous batches in input
order. [Dispatch and partitioning](../../../internal/framework/validators/llm_batching.go#L17) ·
[Composed fitting](../../../internal/framework/validators/llm_budget.go#L59) ·
[Serialized fitting](../../../internal/framework/validators/llm_context.go#L74)
**Opportunity and tradeoffs.** Give the context trim order one owner in
`framework/validators`. Share the greedy batch boundary rule if it can consume
measurements without obscuring the different cost models. Keep the composed
path additive and the generic path able to measure the actual serialized bytes
with any supplied estimator. The serialized measurement remains an independent
oracle for the optimized token calculation. Once policy is shared, an explicit
expected sequence of retained items and batch boundaries can catch policy
regressions that path-to-path equality alone would miss.
## 2. Share regular-file staging between transcript and report publication
**Evidence.** `TestSubprocessOutputPermissionsHonorUmask` checks the transcript
and all three reports against the same expected modes. It exercises creation
under two umasks and replacement of existing mode-`0600` files under each.
`TestFinalizationCollectsFailuresAndRetainsPublishedTranscript` separately
checks that successfully publishing the transcript survives a later report
failure. [Shared mode assertions](../../../cmd/audita/main_integration_test.go#L1202) ·
[Partial finalization](../../../internal/core/diagnostics/output_session_test.go#L207)
**Implementation.** `OutputSession.Prepare` opens a payload in a private
temporary directory next to the transcript destination. It creates the payload
with mode `0644` subject to umask, then copies the existing file's permission
bits when replacing a transcript. `writeTranscript` later writes, closes, and
renames it. Reports use `atomicReplace`, which implements the same regular-file
staging within one call and has its own cleanup and error handling.
[Transcript preparation](../../../internal/core/diagnostics/output_session.go#L141) ·
[Transcript publication](../../../internal/core/diagnostics/output_session.go#L347) ·
[Report replacement](../../../internal/core/diagnostics/output_session.go#L410)
**Opportunity and tradeoffs.** Factor temporary-file creation, permission
handling, close, rename, and cleanup into a small private type in diagnostics.
Let the transcript session hold it across processing and let each report
replacement own it for one write. The caller still owns destination selection,
the transcript's final type check, streaming, publication order, and reporting
of partial failures. Keep subprocess assertions on the final files and the
failure-order test at the session boundary. [Output lifecycle](../../operations.md#outputs-and-lifecycle) ·
[Related output ownership finding](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership)
## 3. Resolve model segment references in one place
**Evidence.** `TestModuleProposalMapsLocalReferencesWithoutGuessing` sends IDs
`1` and `2` for segments with transcript IDs `71` and `903`. It verifies that
returned positions map to canonical IDs, while returning `71` as a position
produces an invalid target. `TestRepairRequestRefreshesPreviewAfterRetarget`
uses canonical IDs `81` and `104` and checks that a repair response targeting
position `2` correctly supplies the second segment's text to the next attempt.
The separate retarget-and-approve test checks preservation of the proposal
index and an unrelated approval. [Generation mapping](../../../internal/framework/proposal_generation/module_proposal_test.go#L102) ·
[Repair mapping](../../../internal/framework/runner/repair_test.go#L131) ·
[Repair identity](../../../internal/framework/runner/repair_test.go#L52)
**Implementation.** `GenerateCandidates` and the repair callback both check
that a correction's `id` is a positive in-range position and index the working
section at `id - 1`. Both leave the target ID at zero for an unresolved
position so proposal validation can handle it. [Generation translation](../../../internal/framework/proposal_generation/generate.go#L189) ·
[Repair translation](../../../internal/framework/runner/repair.go#L216)
**Opportunity and tradeoffs.** Put the conversion next to the correction
response type in `proposal_generation`, which repair already imports. A small
lookup can express this protocol without making the core transcript type aware
of model references. Preserve the raw reference for the conversation, the
existing invalid-target handling, and revalidation after retargeting. Repair's
`proposal_ref` and validation batch indices have their own namespaces. Keep
nonsequential canonical IDs in both caller tests so a positional identity bug
fails even after the callers share the lookup. [Module proposal contract](../../internal/modules.md#proposal-generation-ownership)
## 4. Share structured failure facts across model operations
**Evidence.** The generation and validation tests establish that recoverable
structured failures produce an operation with a stable reason while the caller
can continue processing. The validation case checks approvals on either side
of the failed batch, and the repair terminal-outcome table checks that a
recoverable provider failure ends that branch as unavailable. Fatal setup
fixtures check call facts at the generation boundary and previous decisions,
attempted batches, and preservation of the underlying cause at the validation
boundary. [Recoverable generation](../../../internal/framework/proposal_generation/generate_test.go#L201) ·
[Recoverable validation](../../../internal/framework/validators/llm_partial_test.go#L45) ·
[Repair outcomes](../../../internal/framework/runner/repair_test.go#L166) ·
[Fatal generation](../../../internal/framework/proposal_generation/generate_test.go#L218) ·
[Fatal validation](../../../internal/framework/validators/llm_partial_test.go#L137)
**Implementation.** `structuredoutput.Failure` already owns the typed class,
safe reason and message, and wrapped cause. Generation and validation each
unwrap the error, copy its reason or substitute `internal_failure`, and check
recoverability separately. Repair repeats the extraction with a different
edge case. A typed failure with an empty reason retains that empty reason in
repair, and repair checks recoverability by class alone. [Failure type](../../../internal/framework/structuredoutput/failure.go#L17) ·
[Generation projection](../../../internal/framework/proposal_generation/generate.go#L172) ·
[Validation projection](../../../internal/framework/validators/llm_validators.go#L176) ·
[Repair projection](../../../internal/framework/runner/repair.go#L187)
**Opportunity and tradeoffs.** Put extraction of reason and class in a small
`structuredoutput` helper and keep each caller's decision about what to do with
the facts. Generation can return a failed response, validation can mark the
affected batch unavailable, and repair can end a single branch. Preserve the
wrapped cause, execution counts, and operation coordinates. Retain current
blank-reason handling until its desired semantics are decided explicitly. Keep
role tests with literal reason codes or constants and independently constructed
errors, including the tests for surviving work and fatal propagation.
## 5. Give optional interaction artifacts one writer contract
**Evidence.** Separate proposal, validation, and repair tests inject a write
failure. They require a diagnostic warning and preserve the successful proposal
operation, approved validation decision, or withdrawn repair branch. The
proposal and validation schema-metadata tests also check the same metadata
fields against each operation's registered response schema.
[Proposal write failure](../../../internal/framework/proposal_generation/generate_test.go#L391) ·
[Validation write failure](../../../internal/framework/validators/llm_validators_test.go#L398) ·
[Repair write failure](../../../internal/framework/runner/repair_test.go#L324) ·
[Proposal schema metadata](../../../internal/framework/proposal_generation/generate_test.go#L143) ·
[Validation schema metadata](../../../internal/framework/validators/llm_validators_test.go#L335)
**Implementation.** Proposal generation and validators define equivalent
writer interfaces and four-path artifact values. Both convert the same fields
from `llm.DiagnosticsWriter` through adapters. Repair writes through that
concrete writer directly. File naming, serialization, and redaction already
live in the writer, while each caller separately converts writer failure to a
warning. [Proposal contract](../../../internal/framework/proposal_generation/generate.go#L23) ·
[Validator contract](../../../internal/framework/validators/models.go#L84) ·
[Proposal adapter](../../../internal/framework/proposal_generation/generate.go#L246) ·
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
[Shared writer](../../../internal/framework/llm/diagnostics.go#L17) ·
[Repair diagnostic write](../../../internal/framework/runner/repair.go#L178)
**Opportunity and tradeoffs.** Define the writer interface and artifact path
value once at the diagnostics boundary so the concrete writer can satisfy it
without field translation. A common best-effort write helper could also report
the warning and return paths according to one rule. Keep the role-specific
payloads, coordinates, and credential lookups at their current boundaries.
In particular, the validation adapter resolves secrets lazily, and repair can
record the revision and revalidation decisions. Keep role-specific assertions
for warning delivery and processing outcomes alongside the real writer's tests.
[Writer tests](../../../internal/framework/llm/diagnostics_test.go)
## 6. Use one set of checks for inherited configuration fields
**Evidence.** `TestResolvedSettingsValidation` rejects an empty validation
profile at root and in a module entry. `TestCorrectionRepairSettingsResolution`
rejects a negative correction-repair budget at both locations and exercises
inheritance, explicit overrides, and clearing. Those invalid-case assertions
require a load error and do not inspect its diagnostic path. A separate test
checks that explicit null and the corresponding dependency survive in
resolution metadata. [Profile validation](../../../internal/core/config/resolve_test.go#L97) ·
[Repair settings](../../../internal/core/config/resolve_test.go#L163) ·
[Resolution metadata](../../../internal/core/config/resolve_test.go#L119)
**Implementation.** `config.validate` repeats the same LLM checks for root
and per-module `LLM` values with different `ck.Path` prefixes. The duplicated
checks also include retry bounds, operation timeout, and optional repair
profiles. The validation prompt budget and description length checks repeat
at both levels as well. [Root checks](../../../internal/core/config/config.go#L161) ·
[Per-module checks](../../../internal/core/config/config.go#L175)
**Opportunity and tradeoffs.** Extract typed validation functions for the
shared settings and pass the path of each value. This keeps changes to one
field constraint from drifting between the two locations. Leave resolution,
explicit null handling, provenance, and role-specific profile readiness with
their respective owners. Preserve tests that load root and local documents
independently. Path assertions would catch a helper that applies a correct rule
at the wrong location. [Configuration contract](../../config.md)
## 7. Use the same simple token counter for text estimates
**Evidence.** `TestSimpleTokenEstimator` checks concrete counts for words,
punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization
merge-limit test independently expects that joining two single words exceeds a
one-token limit and records the corresponding skipped merge. This supports the
common word-count behavior. The normalization test does not independently fix
the punctuation weighting. [Counter examples](../../../internal/core/chunking/tokens_test.go#L7) ·
[Merge limit](../../../internal/core/normalization/normalize_test.go#L155)
**Implementation.** Normalization counts `strings.Fields` and then scans for
punctuation. Chunking counts word boundaries and punctuation in one pass. Both
count Unicode punctuation except ASCII apostrophes, hyphens, and underscores,
then add half the punctuation count using integer division. The chunking
counter also exposes those components for validation budget composition.
[Normalization counter](../../../internal/core/normalization/tokens.go#L14) ·
[Chunking counter](../../../internal/core/chunking/tokens.go#L25) ·
[Validation cost calculation](../../../internal/framework/validators/llm_budget.go#L27)
**Opportunity and tradeoffs.** Reuse the single-pass counter and its component
count, either directly or through a small core package if neither caller
should own the dependency. Retain the consumer-specific budgets and tests for
merge behavior, chunk behavior, and serialized validation payloads. Keep
literal counts in the counter tests to detect a shared regression that would
otherwise affect all consumers together. This is a shared heuristic, so
altering its estimates has a wider effect than consolidating its implementation.
## 8. Centralize transcript copies without moving the copy boundaries
**Evidence.** `TestRunnerDoesNotMutateInputTranscript` checks the caller's
original text after a successful run. `TestApplyProposalsNoMutationOfInputTranscript`
compares the caller's transcript with a pre-edit snapshot after applying a
proposal. A separate application test asserts preservation of IDs, speakers,
timing, and categories on the edited output. The snapshot test uses the
production clone, and the runner test checks text only. Neither establishes
category slice independence or a convention for nil versus empty categories.
[Runner ownership](../../../internal/framework/runner/runner_test.go#L708) ·
[Application ownership](../../../internal/framework/proposals/apply_test.go#L199) ·
[Application metadata](../../../internal/framework/proposals/apply_test.go#L221)
**Implementation.** The runner copies the input for a run, and `ApplyProposals`
copies the input it received before any edit. Both independently rebuild every
segment and copy its categories. The proposal copy preserves an explicitly
empty category slice, while the runner's append to a nil slice collapses it.
[Runner copy](../../../internal/framework/runner/runner.go#L972) ·
[Application copy](../../../internal/framework/proposals/apply.go#L107)
**Opportunity and tradeoffs.** Let the schema package own a transcript clone
that copies segment values and then copies mutable member data. Keep both
boundaries so callers retain the existing ownership guarantees. Choose and
check the treatment of empty slices explicitly when moving the code. Use
independent fixture values to verify metadata and ownership, since taking the
expected snapshot with the shared implementation can hide a shallow-copy
regression.
## 9. Reconcile the protection rule at validation and application
**Evidence.** The validator tests assert rejection for removing protected
terms and permit spoken-word repetition cleanup only when the identity
survives. `TestApplicationChecksCurrentTextWithSelectedProtection` demonstrates
the same preservation requirement at apply time. It applies one proposal and
skips the second, whose span is a fragment of a protected alias in the full
segment. `TestFeedbackReportsProtectionAndBlankResult`
also checks that a protection finding and a blank-result finding can both be
reported for one candidate. [Protected-term rejection](../../../internal/framework/validators/protected_terms_test.go#L31) ·
[Repetition policy](../../../internal/framework/validators/protected_terms_test.go#L84) ·
[Protection at application](../../../internal/framework/validators/vocabulary_test.go#L47) ·
[Independent feedback](../../../internal/framework/runner/feedback_test.go#L126)
**Implementation.** Matching and identity comparison already live in
`ProtectedVocabulary`. `ValidateProtectedTerms` starts with the proposed spans
and uses full-segment evidence for the glossary path and eligible spoken-word
repetition cleanup. The runner passes `AllowsChange` to application, which uses
the entire current segment and its replacement preview. Both assemble their
own decision about the before/after text and when to allow repetition. The
earlier correctness audit found that a span can hide the context needed to
recognize a protected name. [Validator check](../../../internal/framework/validators/deterministic.go#L100) ·
[Application predicate](../../../internal/framework/validators/protected_terms.go#L47) ·
[Runner wiring](../../../internal/framework/runner/runner.go#L328) ·
[Span-only correctness finding](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity)
**Opportunity and tradeoffs.** Use a single protection assessment that produces
a reason from the applicable before-and-after segment text and the effective
glossary view. Decide explicitly what evidence can support an independent
finding when a proposal has no usable preview. Moving ordinary validation to
segment scope is a behavior change for edits whose span would give a different
answer. Keep application-time reevaluation against current text, and keep
validation and application tests with their own expected outcomes and fixtures.