290 lines
19 KiB
Markdown
290 lines
19 KiB
Markdown
# Behavioral equivalence and shared abstraction audit
|
|
|
|
Reviewed the application at `7d064a6`. Audita is a single-process Go CLI for
|
|
transcript polishing. The CLI loads configuration and owns invocation and
|
|
output coordination. Core packages handle transcript data, normalization,
|
|
chunking, and serialization. The framework runner coordinates proposal
|
|
generation, validation, repair, and application through explicit contracts.
|
|
The PromptKit adapter executes model requests, and reports are derived from
|
|
framework results. [Package map](../../internal/overview.md) ·
|
|
[Pipeline](../../internal/pipeline.md) ·
|
|
[Dependency direction](../../policy/architecture.md#ownership-and-dependency-direction)
|
|
|
|
The review traced direct comparisons and separate tests with common expected
|
|
behavior to their production implementations. Each finding states what the
|
|
assertions cover and where a shared implementation could remove duplicated
|
|
policy or mechanics. The full existing suite passed with `go test -count=1 ./...`
|
|
on Go 1.27.3. [Testing policy](../../policy/testing.md)
|
|
|
|
## 1. Share validation budget policy across measurement paths
|
|
|
|
**Evidence.** `TestComposedValidationBudgetMatchesSerializedPayload` runs the
|
|
same estimator through the optimized and generic paths on a 12-item fixture.
|
|
For every positive budget up to the full payload cost, it compares complete
|
|
batches and oversized decisions, including retained context, batch boundaries,
|
|
and token counts. Its separate payload assertion checks one-based wire
|
|
references and preserved run-level proposal IDs. The fixture includes Unicode,
|
|
JSON escapes, neighboring segments, and glossary entries. [Budget comparison](../../../internal/framework/validators/llm_budget_test.go#L17) ·
|
|
[Batching contract](../../internal/validators.md#llm-validator-batching)
|
|
|
|
**Implementation.** `prepareValidationBatches` picks the composed path for the
|
|
concrete simple estimator. It premeasures serialized items and context prefixes,
|
|
then uses additive costs. The generic estimator path serializes proposed
|
|
payloads when fitting and partitioning. Both implementations trim neighbor
|
|
entries from the end before trimming glossary entries, reject items whose
|
|
required payload still exceeds the budget, and form contiguous batches in input
|
|
order. [Dispatch and partitioning](../../../internal/framework/validators/llm_batching.go#L17) ·
|
|
[Composed fitting](../../../internal/framework/validators/llm_budget.go#L59) ·
|
|
[Serialized fitting](../../../internal/framework/validators/llm_context.go#L74)
|
|
|
|
**Opportunity and tradeoffs.** Give the context trim order one owner in
|
|
`framework/validators`. Share the greedy batch boundary rule if it can consume
|
|
measurements without obscuring the different cost models. Keep the composed
|
|
path additive and the generic path able to measure the actual serialized bytes
|
|
with any supplied estimator. The serialized measurement remains an independent
|
|
oracle for the optimized token calculation. Once policy is shared, an explicit
|
|
expected sequence of retained items and batch boundaries can catch policy
|
|
regressions that path-to-path equality alone would miss.
|
|
|
|
## 2. Share regular-file staging between transcript and report publication
|
|
|
|
**Evidence.** `TestSubprocessOutputPermissionsHonorUmask` checks the transcript
|
|
and all three reports against the same expected modes. It exercises creation
|
|
under two umasks and replacement of existing mode-`0600` files under each.
|
|
`TestFinalizationCollectsFailuresAndRetainsPublishedTranscript` separately
|
|
checks that successfully publishing the transcript survives a later report
|
|
failure. [Shared mode assertions](../../../cmd/audita/main_integration_test.go#L1202) ·
|
|
[Partial finalization](../../../internal/core/diagnostics/output_session_test.go#L207)
|
|
|
|
**Implementation.** `OutputSession.Prepare` opens a payload in a private
|
|
temporary directory next to the transcript destination. It creates the payload
|
|
with mode `0644` subject to umask, then copies the existing file's permission
|
|
bits when replacing a transcript. `writeTranscript` later writes, closes, and
|
|
renames it. Reports use `atomicReplace`, which implements the same regular-file
|
|
staging within one call and has its own cleanup and error handling.
|
|
[Transcript preparation](../../../internal/core/diagnostics/output_session.go#L141) ·
|
|
[Transcript publication](../../../internal/core/diagnostics/output_session.go#L347) ·
|
|
[Report replacement](../../../internal/core/diagnostics/output_session.go#L410)
|
|
|
|
**Opportunity and tradeoffs.** Factor temporary-file creation, permission
|
|
handling, close, rename, and cleanup into a small private type in diagnostics.
|
|
Let the transcript session hold it across processing and let each report
|
|
replacement own it for one write. The caller still owns destination selection,
|
|
the transcript's final type check, streaming, publication order, and reporting
|
|
of partial failures. Keep subprocess assertions on the final files and the
|
|
failure-order test at the session boundary. [Output lifecycle](../../operations.md#outputs-and-lifecycle) ·
|
|
[Related output ownership finding](03-complexity.md#9-clarify-output-preparation-publication-and-failure-ownership)
|
|
|
|
## 3. Resolve model segment references in one place
|
|
|
|
**Evidence.** `TestModuleProposalMapsLocalReferencesWithoutGuessing` sends IDs
|
|
`1` and `2` for segments with transcript IDs `71` and `903`. It verifies that
|
|
returned positions map to canonical IDs, while returning `71` as a position
|
|
produces an invalid target. `TestRepairRequestRefreshesPreviewAfterRetarget`
|
|
uses canonical IDs `81` and `104` and checks that a repair response targeting
|
|
position `2` correctly supplies the second segment's text to the next attempt.
|
|
The separate retarget-and-approve test checks preservation of the proposal
|
|
index and an unrelated approval. [Generation mapping](../../../internal/framework/proposal_generation/module_proposal_test.go#L102) ·
|
|
[Repair mapping](../../../internal/framework/runner/repair_test.go#L131) ·
|
|
[Repair identity](../../../internal/framework/runner/repair_test.go#L52)
|
|
|
|
**Implementation.** `GenerateCandidates` and the repair callback both check
|
|
that a correction's `id` is a positive in-range position and index the working
|
|
section at `id - 1`. Both leave the target ID at zero for an unresolved
|
|
position so proposal validation can handle it. [Generation translation](../../../internal/framework/proposal_generation/generate.go#L189) ·
|
|
[Repair translation](../../../internal/framework/runner/repair.go#L216)
|
|
|
|
**Opportunity and tradeoffs.** Put the conversion next to the correction
|
|
response type in `proposal_generation`, which repair already imports. A small
|
|
lookup can express this protocol without making the core transcript type aware
|
|
of model references. Preserve the raw reference for the conversation, the
|
|
existing invalid-target handling, and revalidation after retargeting. Repair's
|
|
`proposal_ref` and validation batch indices have their own namespaces. Keep
|
|
nonsequential canonical IDs in both caller tests so a positional identity bug
|
|
fails even after the callers share the lookup. [Module proposal contract](../../internal/modules.md#proposal-generation-ownership)
|
|
|
|
## 4. Share structured failure facts across model operations
|
|
|
|
**Evidence.** The generation and validation tests establish that recoverable
|
|
structured failures produce an operation with a stable reason while the caller
|
|
can continue processing. The validation case checks approvals on either side
|
|
of the failed batch, and the repair terminal-outcome table checks that a
|
|
recoverable provider failure ends that branch as unavailable. Fatal setup
|
|
fixtures check call facts at the generation boundary and previous decisions,
|
|
attempted batches, and preservation of the underlying cause at the validation
|
|
boundary. [Recoverable generation](../../../internal/framework/proposal_generation/generate_test.go#L201) ·
|
|
[Recoverable validation](../../../internal/framework/validators/llm_partial_test.go#L45) ·
|
|
[Repair outcomes](../../../internal/framework/runner/repair_test.go#L166) ·
|
|
[Fatal generation](../../../internal/framework/proposal_generation/generate_test.go#L218) ·
|
|
[Fatal validation](../../../internal/framework/validators/llm_partial_test.go#L137)
|
|
|
|
**Implementation.** `structuredoutput.Failure` already owns the typed class,
|
|
safe reason and message, and wrapped cause. Generation and validation each
|
|
unwrap the error, copy its reason or substitute `internal_failure`, and check
|
|
recoverability separately. Repair repeats the extraction with a different
|
|
edge case. A typed failure with an empty reason retains that empty reason in
|
|
repair, and repair checks recoverability by class alone. [Failure type](../../../internal/framework/structuredoutput/failure.go#L17) ·
|
|
[Generation projection](../../../internal/framework/proposal_generation/generate.go#L172) ·
|
|
[Validation projection](../../../internal/framework/validators/llm_validators.go#L176) ·
|
|
[Repair projection](../../../internal/framework/runner/repair.go#L187)
|
|
|
|
**Opportunity and tradeoffs.** Put extraction of reason and class in a small
|
|
`structuredoutput` helper and keep each caller's decision about what to do with
|
|
the facts. Generation can return a failed response, validation can mark the
|
|
affected batch unavailable, and repair can end a single branch. Preserve the
|
|
wrapped cause, execution counts, and operation coordinates. Retain current
|
|
blank-reason handling until its desired semantics are decided explicitly. Keep
|
|
role tests with literal reason codes or constants and independently constructed
|
|
errors, including the tests for surviving work and fatal propagation.
|
|
|
|
## 5. Give optional interaction artifacts one writer contract
|
|
|
|
**Evidence.** Separate proposal, validation, and repair tests inject a write
|
|
failure. They require a diagnostic warning and preserve the successful proposal
|
|
operation, approved validation decision, or withdrawn repair branch. The
|
|
proposal and validation schema-metadata tests also check the same metadata
|
|
fields against each operation's registered response schema.
|
|
[Proposal write failure](../../../internal/framework/proposal_generation/generate_test.go#L391) ·
|
|
[Validation write failure](../../../internal/framework/validators/llm_validators_test.go#L398) ·
|
|
[Repair write failure](../../../internal/framework/runner/repair_test.go#L324) ·
|
|
[Proposal schema metadata](../../../internal/framework/proposal_generation/generate_test.go#L143) ·
|
|
[Validation schema metadata](../../../internal/framework/validators/llm_validators_test.go#L335)
|
|
|
|
**Implementation.** Proposal generation and validators define equivalent
|
|
writer interfaces and four-path artifact values. Both convert the same fields
|
|
from `llm.DiagnosticsWriter` through adapters. Repair writes through that
|
|
concrete writer directly. File naming, serialization, and redaction already
|
|
live in the writer, while each caller separately converts writer failure to a
|
|
warning. [Proposal contract](../../../internal/framework/proposal_generation/generate.go#L23) ·
|
|
[Validator contract](../../../internal/framework/validators/models.go#L84) ·
|
|
[Proposal adapter](../../../internal/framework/proposal_generation/generate.go#L246) ·
|
|
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
|
|
[Shared writer](../../../internal/framework/llm/diagnostics.go#L17) ·
|
|
[Repair diagnostic write](../../../internal/framework/runner/repair.go#L178)
|
|
|
|
**Opportunity and tradeoffs.** Define the writer interface and artifact path
|
|
value once at the diagnostics boundary so the concrete writer can satisfy it
|
|
without field translation. A common best-effort write helper could also report
|
|
the warning and return paths according to one rule. Keep the role-specific
|
|
payloads, coordinates, and credential lookups at their current boundaries.
|
|
In particular, the validation adapter resolves secrets lazily, and repair can
|
|
record the revision and revalidation decisions. Keep role-specific assertions
|
|
for warning delivery and processing outcomes alongside the real writer's tests.
|
|
[Writer tests](../../../internal/framework/llm/diagnostics_test.go)
|
|
|
|
## 6. Use one set of checks for inherited configuration fields
|
|
|
|
**Evidence.** `TestResolvedSettingsValidation` rejects an empty validation
|
|
profile at root and in a module entry. `TestCorrectionRepairSettingsResolution`
|
|
rejects a negative correction-repair budget at both locations and exercises
|
|
inheritance, explicit overrides, and clearing. Those invalid-case assertions
|
|
require a load error and do not inspect its diagnostic path. A separate test
|
|
checks that explicit null and the corresponding dependency survive in
|
|
resolution metadata. [Profile validation](../../../internal/core/config/resolve_test.go#L97) ·
|
|
[Repair settings](../../../internal/core/config/resolve_test.go#L163) ·
|
|
[Resolution metadata](../../../internal/core/config/resolve_test.go#L119)
|
|
|
|
**Implementation.** `config.validate` repeats the same LLM checks for root
|
|
and per-module `LLM` values with different `ck.Path` prefixes. The duplicated
|
|
checks also include retry bounds, operation timeout, and optional repair
|
|
profiles. The validation prompt budget and description length checks repeat
|
|
at both levels as well. [Root checks](../../../internal/core/config/config.go#L161) ·
|
|
[Per-module checks](../../../internal/core/config/config.go#L175)
|
|
|
|
**Opportunity and tradeoffs.** Extract typed validation functions for the
|
|
shared settings and pass the path of each value. This keeps changes to one
|
|
field constraint from drifting between the two locations. Leave resolution,
|
|
explicit null handling, provenance, and role-specific profile readiness with
|
|
their respective owners. Preserve tests that load root and local documents
|
|
independently. Path assertions would catch a helper that applies a correct rule
|
|
at the wrong location. [Configuration contract](../../config.md)
|
|
|
|
## 7. Use the same simple token counter for text estimates
|
|
|
|
**Evidence.** `TestSimpleTokenEstimator` checks concrete counts for words,
|
|
punctuation, apostrophes, hyphens, and Unicode text in chunking. The normalization
|
|
merge-limit test independently expects that joining two single words exceeds a
|
|
one-token limit and records the corresponding skipped merge. This supports the
|
|
common word-count behavior. The normalization test does not independently fix
|
|
the punctuation weighting. [Counter examples](../../../internal/core/chunking/tokens_test.go#L7) ·
|
|
[Merge limit](../../../internal/core/normalization/normalize_test.go#L155)
|
|
|
|
**Implementation.** Normalization counts `strings.Fields` and then scans for
|
|
punctuation. Chunking counts word boundaries and punctuation in one pass. Both
|
|
count Unicode punctuation except ASCII apostrophes, hyphens, and underscores,
|
|
then add half the punctuation count using integer division. The chunking
|
|
counter also exposes those components for validation budget composition.
|
|
[Normalization counter](../../../internal/core/normalization/tokens.go#L14) ·
|
|
[Chunking counter](../../../internal/core/chunking/tokens.go#L25) ·
|
|
[Validation cost calculation](../../../internal/framework/validators/llm_budget.go#L27)
|
|
|
|
**Opportunity and tradeoffs.** Reuse the single-pass counter and its component
|
|
count, either directly or through a small core package if neither caller
|
|
should own the dependency. Retain the consumer-specific budgets and tests for
|
|
merge behavior, chunk behavior, and serialized validation payloads. Keep
|
|
literal counts in the counter tests to detect a shared regression that would
|
|
otherwise affect all consumers together. This is a shared heuristic, so
|
|
altering its estimates has a wider effect than consolidating its implementation.
|
|
|
|
## 8. Centralize transcript copies without moving the copy boundaries
|
|
|
|
**Evidence.** `TestRunnerDoesNotMutateInputTranscript` checks the caller's
|
|
original text after a successful run. `TestApplyProposalsNoMutationOfInputTranscript`
|
|
compares the caller's transcript with a pre-edit snapshot after applying a
|
|
proposal. A separate application test asserts preservation of IDs, speakers,
|
|
timing, and categories on the edited output. The snapshot test uses the
|
|
production clone, and the runner test checks text only. Neither establishes
|
|
category slice independence or a convention for nil versus empty categories.
|
|
[Runner ownership](../../../internal/framework/runner/runner_test.go#L708) ·
|
|
[Application ownership](../../../internal/framework/proposals/apply_test.go#L199) ·
|
|
[Application metadata](../../../internal/framework/proposals/apply_test.go#L221)
|
|
|
|
**Implementation.** The runner copies the input for a run, and `ApplyProposals`
|
|
copies the input it received before any edit. Both independently rebuild every
|
|
segment and copy its categories. The proposal copy preserves an explicitly
|
|
empty category slice, while the runner's append to a nil slice collapses it.
|
|
[Runner copy](../../../internal/framework/runner/runner.go#L972) ·
|
|
[Application copy](../../../internal/framework/proposals/apply.go#L107)
|
|
|
|
**Opportunity and tradeoffs.** Let the schema package own a transcript clone
|
|
that copies segment values and then copies mutable member data. Keep both
|
|
boundaries so callers retain the existing ownership guarantees. Choose and
|
|
check the treatment of empty slices explicitly when moving the code. Use
|
|
independent fixture values to verify metadata and ownership, since taking the
|
|
expected snapshot with the shared implementation can hide a shallow-copy
|
|
regression.
|
|
|
|
## 9. Reconcile the protection rule at validation and application
|
|
|
|
**Evidence.** The validator tests assert rejection for removing protected
|
|
terms and permit spoken-word repetition cleanup only when the identity
|
|
survives. `TestApplicationChecksCurrentTextWithSelectedProtection` demonstrates
|
|
the same preservation requirement at apply time. It applies one proposal and
|
|
skips the second, whose span is a fragment of a protected alias in the full
|
|
segment. `TestFeedbackReportsProtectionAndBlankResult`
|
|
also checks that a protection finding and a blank-result finding can both be
|
|
reported for one candidate. [Protected-term rejection](../../../internal/framework/validators/protected_terms_test.go#L31) ·
|
|
[Repetition policy](../../../internal/framework/validators/protected_terms_test.go#L84) ·
|
|
[Protection at application](../../../internal/framework/validators/vocabulary_test.go#L47) ·
|
|
[Independent feedback](../../../internal/framework/runner/feedback_test.go#L126)
|
|
|
|
**Implementation.** Matching and identity comparison already live in
|
|
`ProtectedVocabulary`. `ValidateProtectedTerms` starts with the proposed spans
|
|
and uses full-segment evidence for the glossary path and eligible spoken-word
|
|
repetition cleanup. The runner passes `AllowsChange` to application, which uses
|
|
the entire current segment and its replacement preview. Both assemble their
|
|
own decision about the before/after text and when to allow repetition. The
|
|
earlier correctness audit found that a span can hide the context needed to
|
|
recognize a protected name. [Validator check](../../../internal/framework/validators/deterministic.go#L100) ·
|
|
[Application predicate](../../../internal/framework/validators/protected_terms.go#L47) ·
|
|
[Runner wiring](../../../internal/framework/runner/runner.go#L328) ·
|
|
[Span-only correctness finding](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity)
|
|
|
|
**Opportunity and tradeoffs.** Use a single protection assessment that produces
|
|
a reason from the applicable before-and-after segment text and the effective
|
|
glossary view. Decide explicitly what evidence can support an independent
|
|
finding when a proposal has no usable preview. Moving ordinary validation to
|
|
segment scope is a behavior change for edits whose span would give a different
|
|
answer. Keep application-time reevaluation against current text, and keep
|
|
validation and application tests with their own expected outcomes and fixtures.
|