From 0f34231f46298947d49f65c92d824c96ecb8a1ba Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 15 Nov 2026 14:53:11 +0000 Subject: [PATCH] Record behavioral equivalence audit findings --- docs/roadmap/audit/10-equivalence.md | 289 +++++++++++++++++++++++++++ 1 file changed, 289 insertions(+) create mode 100644 docs/roadmap/audit/10-equivalence.md diff --git a/docs/roadmap/audit/10-equivalence.md b/docs/roadmap/audit/10-equivalence.md new file mode 100644 index 0000000..1aabef3 --- /dev/null +++ b/docs/roadmap/audit/10-equivalence.md @@ -0,0 +1,289 @@ +# Behavioral equivalence and shared abstraction audit + +Reviewed the application at `7d064a6`. Audita is a single-process Go CLI for +transcript polishing. The CLI loads configuration and owns invocation and +output coordination. Core packages handle transcript data, normalization, +chunking, and serialization. The framework runner coordinates proposal +generation, validation, repair, and application through explicit contracts. +The PromptKit adapter executes model requests, and reports are derived from +framework results. [Package map](../../internal/overview.md) · +[Pipeline](../../internal/pipeline.md) · +[Dependency direction](../../policy/architecture.md#ownership-and-dependency-direction) + +The review traced direct comparisons and separate tests with common expected +behavior to their production implementations. Each finding states what the +assertions cover and where a shared implementation could remove duplicated +policy or mechanics. The full existing suite passed with `go test -count=1 ./...` +on Go 1.27.3. [Testing policy](../../policy/testing.md) + +## 1. Share validation budget policy across measurement paths + +**Evidence.** `TestComposedValidationBudgetMatchesSerializedPayload` runs the +same estimator through the optimized and generic paths on a 12-item fixture. +For every positive budget up to the full payload cost, it compares complete +batches and oversized decisions, including retained context, batch boundaries, +and token counts. Its separate payload assertion checks one-based wire +references and preserved run-level proposal IDs. The fixture includes Unicode, +JSON escapes, neighboring segments, and glossary entries. [Budget comparison](../../../internal/framework/validators/llm_budget_test.go#L17) · +[Batching contract](../../internal/validators.md#llm-validator-batching) + +**Implementation.** `prepareValidationBatches` picks the composed path for the +concrete simple estimator. It premeasures serialized items and context prefixes, +then uses additive costs. The generic estimator path serializes proposed +payloads when fitting and partitioning. Both implementations trim neighbor +entries from the end before trimming glossary entries, reject items whose +required payload still exceeds the budget, and form contiguous batches in input +order. [Dispatch and partitioning](../../../internal/framework/validators/llm_batching.go#L17) · +[Composed fitting](../../../internal/framework/validators/llm_budget.go#L59) · +[Serialized fitting](../../../internal/framework/validators/llm_context.go#L74) + +**Opportunity and tradeoffs.** Give the context trim order one owner in +`framework/validators`. Share the greedy batch boundary rule if it can consume +measurements without obscuring the different cost models. Keep the composed +path additive and the generic path able to measure the actual serialized bytes +with any supplied estimator. The serialized measurement remains an independent +oracle for the optimized token calculation. Once policy is shared, an explicit +expected sequence of retained items and batch boundaries can catch policy +regressions that path-to-path equality alone would miss. + +## 2. Share regular-file staging between transcript and report publication + +**Evidence.** `TestSubprocessOutputPermissionsHonorUmask` checks the transcript +and all three reports against the same expected modes. It exercises creation +under two umasks and replacement of existing mode-`0600` files under each. +`TestFinalizationCollectsFailuresAndRetainsPublishedTranscript` separately +checks that successfully publishing the transcript survives a later report +failure. [Shared mode assertions](../../../cmd/audita/main_integration_test.go#L1202) · +[Partial finalization](../../../internal/core/diagnostics/output_session_test.go#L207) + +**Implementation.** `OutputSession.Prepare` opens a payload in a private +temporary directory next to the transcript destination. It creates the payload +with mode `0644` subject to umask, then copies the existing file's permission +bits when replacing a transcript. `writeTranscript` later writes, closes, and +renames it. Reports use `atomicReplace`, which implements the same regular-file +staging within one call and has its own cleanup and error handling. +[Transcript preparation](../../../internal/core/diagnostics/output_session.go#L141) · +[Transcript publication](../../../internal/core/diagnostics/output_session.go#L347) · +[Report replacement](../../../internal/core/diagnostics/output_session.go#L410) + +**Opportunity and tradeoffs.** Factor temporary-file creation, permission +handling, close, rename, and cleanup into a small private type in diagnostics. +Let the transcript session hold it across processing and let each report +replacement own it for one write. The caller still owns destination selection, +the transcript's final type check, streaming, publication order, and reporting +of partial failures. Keep subprocess assertions on the final files and the +failure-order test at the session boundary. [Output lifecycle](../../operations.md#outputs-and-lifecycle) · +[Related output ownership finding](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership) + +## 3. Resolve model segment references in one place + +**Evidence.** `TestModuleProposalMapsLocalReferencesWithoutGuessing` sends IDs +`1` and `2` for segments with transcript IDs `71` and `903`. It verifies that +returned positions map to canonical IDs, while returning `71` as a position +produces an invalid target. `TestRepairRequestRefreshesPreviewAfterRetarget` +uses canonical IDs `81` and `104` and checks that a repair response targeting +position `2` correctly supplies the second segment's text to the next attempt. +The separate retarget-and-approve test checks preservation of the proposal +index and an unrelated approval. [Generation mapping](../../../internal/framework/proposal_generation/module_proposal_test.go#L102) · +[Repair mapping](../../../internal/framework/runner/repair_test.go#L131) · +[Repair identity](../../../internal/framework/runner/repair_test.go#L52) + +**Implementation.** `GenerateCandidates` and the repair callback both check +that a correction's `id` is a positive in-range position and index the working +section at `id - 1`. Both leave the target ID at zero for an unresolved +position so proposal validation can handle it. [Generation translation](../../../internal/framework/proposal_generation/generate.go#L189) · +[Repair translation](../../../internal/framework/runner/repair.go#L216) + +**Opportunity and tradeoffs.** Put the conversion next to the correction +response type in `proposal_generation`, which repair already imports. A small +lookup can express this protocol without making the core transcript type aware +of model references. Preserve the raw reference for the conversation, the +existing invalid-target handling, and revalidation after retargeting. Repair's +`proposal_ref` and validation batch indices have their own namespaces. Keep +nonsequential canonical IDs in both caller tests so a positional identity bug +fails even after the callers share the lookup. [Module proposal contract](../../internal/modules.md#proposal-generation-ownership) + +## 4. Share structured failure facts across model operations + +**Evidence.** The generation and validation tests establish that recoverable +structured failures produce an operation with a stable reason while the caller +can continue processing. The validation case checks approvals on either side +of the failed batch, and the repair terminal-outcome table checks that a +recoverable provider failure ends that branch as unavailable. Fatal setup +fixtures check call facts at the generation boundary and previous decisions, +attempted batches, and preservation of the underlying cause at the validation +boundary. [Recoverable generation](../../../internal/framework/proposal_generation/generate_test.go#L201) · +[Recoverable validation](../../../internal/framework/validators/llm_partial_test.go#L45) · +[Repair outcomes](../../../internal/framework/runner/repair_test.go#L166) · +[Fatal generation](../../../internal/framework/proposal_generation/generate_test.go#L218) · +[Fatal validation](../../../internal/framework/validators/llm_partial_test.go#L137) + +**Implementation.** `structuredoutput.Failure` already owns the typed class, +safe reason and message, and wrapped cause. Generation and validation each +unwrap the error, copy its reason or substitute `internal_failure`, and check +recoverability separately. Repair repeats the extraction with a different +edge case. A typed failure with an empty reason retains that empty reason in +repair, and repair checks recoverability by class alone. [Failure type](../../../internal/framework/structuredoutput/failure.go#L17) · +[Generation projection](../../../internal/framework/proposal_generation/generate.go#L172) · +[Validation projection](../../../internal/framework/validators/llm_validators.go#L176) · +[Repair projection](../../../internal/framework/runner/repair.go#L187) + +**Opportunity and tradeoffs.** Put extraction of reason and class in a small +`structuredoutput` helper and keep each caller's decision about what to do with +the facts. Generation can return a failed response, validation can mark the +affected batch unavailable, and repair can end a single branch. Preserve the +wrapped cause, execution counts, and operation coordinates. Retain current +blank-reason handling until its desired semantics are decided explicitly. Keep +role tests with literal reason codes or constants and independently constructed +errors, including the tests for surviving work and fatal propagation. + +## 5. Give optional interaction artifacts one writer contract + +**Evidence.** Separate proposal, validation, and repair tests inject a write +failure. They require a diagnostic warning and preserve the successful proposal +operation, approved validation decision, or withdrawn repair branch. The +proposal and validation schema-metadata tests also check the same metadata +fields against each operation's registered response schema. +[Proposal write failure](../../../internal/framework/proposal_generation/generate_test.go#L391) · +[Validation write failure](../../../internal/framework/validators/llm_validators_test.go#L398) · +[Repair write failure](../../../internal/framework/runner/repair_test.go#L324) · +[Proposal schema metadata](../../../internal/framework/proposal_generation/generate_test.go#L143) · +[Validation schema metadata](../../../internal/framework/validators/llm_validators_test.go#L335) + +**Implementation.** Proposal generation and validators define equivalent +writer interfaces and four-path artifact values. Both convert the same fields +from `llm.DiagnosticsWriter` through adapters. Repair writes through that +concrete writer directly. File naming, serialization, and redaction already +live in the writer, while each caller separately converts writer failure to a +warning. [Proposal contract](../../../internal/framework/proposal_generation/generate.go#L23) · +[Validator contract](../../../internal/framework/validators/models.go#L84) · +[Proposal adapter](../../../internal/framework/proposal_generation/generate.go#L246) · +[Validation adapter](../../../internal/framework/runner/runner.go#L994) · +[Shared writer](../../../internal/framework/llm/diagnostics.go#L17) · +[Repair diagnostic write](../../../internal/framework/runner/repair.go#L178) + +**Opportunity and tradeoffs.** Define the writer interface and artifact path +value once at the diagnostics boundary so the concrete writer can satisfy it +without field translation. A common best-effort write helper could also report +the warning and return paths according to one rule. Keep the role-specific +payloads, coordinates, and credential lookups at their current boundaries. +In particular, the validation adapter resolves secrets lazily, and repair can +record the revision and revalidation decisions. Keep role-specific assertions +for warning delivery and processing outcomes alongside the real writer's tests. +[Writer tests](../../../internal/framework/llm/diagnostics_test.go) + +## 6. Use one set of checks for inherited configuration fields + +**Evidence.** `TestResolvedSettingsValidation` rejects an empty validation +profile at root and in a module entry. `TestCorrectionRepairSettingsResolution` +rejects a negative correction-repair budget at both locations and exercises +inheritance, explicit overrides, and clearing. Those invalid-case assertions +require a load error and do not inspect its diagnostic path. A separate test +checks that explicit null and the corresponding dependency survive in +resolution metadata. [Profile validation](../../../internal/core/config/resolve_test.go#L97) · +[Repair settings](../../../internal/core/config/resolve_test.go#L163) · +[Resolution metadata](../../../internal/core/config/resolve_test.go#L119) + +**Implementation.** `config.validate` repeats the same LLM checks for root +and per-module `LLM` values with different `ck.Path` prefixes. The duplicated +checks also include retry bounds, operation timeout, and optional repair +profiles. The validation prompt budget and description length checks repeat +at both levels as well. [Root checks](../../../internal/core/config/config.go#L161) · +[Per-module checks](../../../internal/core/config/config.go#L175) + +**Opportunity and tradeoffs.** Extract typed validation functions for the +shared settings and pass the path of each value. This keeps changes to one +field constraint from drifting between the two locations. Leave resolution, +explicit null handling, provenance, and role-specific profile readiness with +their respective owners. Preserve tests that load root and local documents +independently. Path assertions would catch a helper that applies a correct rule +at the wrong location. [Configuration contract](../../config.md) + +## 7. Use the same simple token counter for text estimates + +**Evidence.** `TestSimpleTokenEstimator` checks concrete counts for words, +punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization +merge-limit test independently expects that joining two single words exceeds a +one-token limit and records the corresponding skipped merge. This supports the +common word-count behavior. The normalization test does not independently fix +the punctuation weighting. [Counter examples](../../../internal/core/chunking/tokens_test.go#L7) · +[Merge limit](../../../internal/core/normalization/normalize_test.go#L155) + +**Implementation.** Normalization counts `strings.Fields` and then scans for +punctuation. Chunking counts word boundaries and punctuation in one pass. Both +count Unicode punctuation except ASCII apostrophes, hyphens, and underscores, +then add half the punctuation count using integer division. The chunking +counter also exposes those components for validation budget composition. +[Normalization counter](../../../internal/core/normalization/tokens.go#L14) · +[Chunking counter](../../../internal/core/chunking/tokens.go#L25) · +[Validation cost calculation](../../../internal/framework/validators/llm_budget.go#L27) + +**Opportunity and tradeoffs.** Reuse the single-pass counter and its component +count, either directly or through a small core package if neither caller +should own the dependency. Retain the consumer-specific budgets and tests for +merge behavior, chunk behavior, and serialized validation payloads. Keep +literal counts in the counter tests to detect a shared regression that would +otherwise affect all consumers together. This is a shared heuristic, so +altering its estimates has a wider effect than consolidating its implementation. + +## 8. Centralize transcript copies without moving the copy boundaries + +**Evidence.** `TestRunnerDoesNotMutateInputTranscript` checks the caller's +original text after a successful run. `TestApplyProposalsNoMutationOfInputTranscript` +compares the caller's transcript with a pre-edit snapshot after applying a +proposal. A separate application test asserts preservation of IDs, speakers, +timing, and categories on the edited output. The snapshot test uses the +production clone, and the runner test checks text only. Neither establishes +category slice independence or a convention for nil versus empty categories. +[Runner ownership](../../../internal/framework/runner/runner_test.go#L708) · +[Application ownership](../../../internal/framework/proposals/apply_test.go#L199) · +[Application metadata](../../../internal/framework/proposals/apply_test.go#L221) + +**Implementation.** The runner copies the input for a run, and `ApplyProposals` +copies the input it received before any edit. Both independently rebuild every +segment and copy its categories. The proposal copy preserves an explicitly +empty category slice, while the runner's append to a nil slice collapses it. +[Runner copy](../../../internal/framework/runner/runner.go#L972) · +[Application copy](../../../internal/framework/proposals/apply.go#L107) + +**Opportunity and tradeoffs.** Let the schema package own a transcript clone +that copies segment values and then copies mutable member data. Keep both +boundaries so callers retain the existing ownership guarantees. Choose and +check the treatment of empty slices explicitly when moving the code. Use +independent fixture values to verify metadata and ownership, since taking the +expected snapshot with the shared implementation can hide a shallow-copy +regression. + +## 9. Reconcile the protection rule at validation and application + +**Evidence.** The validator tests assert rejection for removing protected +terms and permit spoken-word repetition cleanup only when the identity +survives. `TestApplicationChecksCurrentTextWithSelectedProtection` demonstrates +the same preservation requirement at apply time. It applies one proposal and +skips the second, whose span is a fragment of a protected alias in the full +segment. `TestFeedbackReportsProtectionAndBlankResult` +also checks that a protection finding and a blank-result finding can both be +reported for one candidate. [Protected-term rejection](../../../internal/framework/validators/protected_terms_test.go#L31) · +[Repetition policy](../../../internal/framework/validators/protected_terms_test.go#L84) · +[Protection at application](../../../internal/framework/validators/vocabulary_test.go#L47) · +[Independent feedback](../../../internal/framework/runner/feedback_test.go#L126) + +**Implementation.** Matching and identity comparison already live in +`ProtectedVocabulary`. `ValidateProtectedTerms` starts with the proposed spans +and uses full-segment evidence for the glossary path and eligible spoken-word +repetition cleanup. The runner passes `AllowsChange` to application, which uses +the entire current segment and its replacement preview. Both assemble their +own decision about the before/after text and when to allow repetition. The +earlier correctness audit found that a span can hide the context needed to +recognize a protected name. [Validator check](../../../internal/framework/validators/deterministic.go#L100) · +[Application predicate](../../../internal/framework/validators/protected_terms.go#L47) · +[Runner wiring](../../../internal/framework/runner/runner.go#L328) · +[Span-only correctness finding](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity) + +**Opportunity and tradeoffs.** Use a single protection assessment that produces +a reason from the applicable before-and-after segment text and the effective +glossary view. Decide explicitly what evidence can support an independent +finding when a proposal has no usable preview. Moving ordinary validation to +segment scope is a behavior change for edits whose span would give a different +answer. Keep application-time reevaluation against current text, and keep +validation and application tests with their own expected outcomes and fixtures.