262 lines
16 KiB
Markdown
262 lines
16 KiB
Markdown
# Functional correctness audit
|
|
|
|
Reviewed source at `c4ccd78`. All findings below were reproduced through the
|
|
built CLI with temporary files or a synthetic loopback provider. The existing
|
|
offline suite also passed with Go 1.27.3 using
|
|
`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`.
|
|
The provider fixtures exercised Audita's handling of scripted decisions.
|
|
Live-model editorial quality was outside the review.
|
|
|
|
## Architecture and review scope
|
|
|
|
Audita is a single-process Go CLI. CommandKit handles commands and streams,
|
|
ConfigKit composes configuration, and processing preflight checks role profiles,
|
|
glossary selections, and output destinations. Core packages parse and normalize
|
|
the transcript, derive glossary views, and partition the normalized segments into
|
|
fixed chunks. The framework runs configured correction modules in order while
|
|
sections can generate proposals concurrently. Deterministic checks and LLM
|
|
validators gate application. PromptKit handles the provider and continuation
|
|
history for bounded repair branches, and Audita revalidates revisions. The output
|
|
session publishes the transcript and reports, with opt-in diagnostics capturing
|
|
request, decision, and execution evidence. [Package map](../../internal/overview.md)
|
|
· [Processing sequence](../../internal/pipeline.md)
|
|
|
|
The review traced configuration and readiness into file handling, transcript
|
|
decoding and normalization, chunk feasibility, glossary matching, proposal
|
|
preview and application, validator batching and response reconciliation,
|
|
repair outcomes, cancellation, and ordinary and debug reporting. It checked
|
|
representative success paths alongside malformed inputs, boundaries, recoverable
|
|
failures, and interrupted work.
|
|
|
|
## 1. Output path resolution can replace a different file
|
|
|
|
**Expected.** A regular transcript should be published at the selected output
|
|
destination. Collision checks and publication should agree on which file that
|
|
path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
|
|
|
**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves
|
|
`logical/link/../out.json` to `physical/out.json`. With two distinct existing
|
|
files and reporting disabled, Audita exited 0, left `physical/out.json` untouched,
|
|
and overwrote `logical/out.json`. `Prepare` stats the supplied path, but
|
|
`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving
|
|
symlinks. The cleaned path is then used for publication. The same helper feeds
|
|
collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141)
|
|
· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377)
|
|
|
|
**Remediation.** Resolve directory components with filesystem traversal
|
|
semantics before collapsing `..`. Use the resulting destination for collision
|
|
checks, regular-file preparation, and publication. Derive the default report
|
|
directory from the same destination when file output is selected.
|
|
|
|
## 2. Glossary matching misses names with accented letters or punctuation at their edges
|
|
|
|
**Expected.** Selected protected names and aliases should participate in both
|
|
deterministic protection checks. Matching names should also be available to
|
|
validators when building context. The glossary accepts ordinary strings for
|
|
names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections)
|
|
· [Validator context](../../internal/validators.md#supplemental-validation-context)
|
|
|
|
**Observed.** With `Élodie` or `C++` in the selected context and protection
|
|
collection, scripted grammar proposals replaced the names in `Élodie arrived.`
|
|
and `Use C++ for this.`. Both edits reached model validation and were applied
|
|
after scripted approvals. The validator requests also omitted the corresponding
|
|
glossary entries. The shared matcher surrounds every alternative with Go
|
|
regular-expression `\b`, whose word character class is ASCII. At the start of
|
|
`Élodie` and after the final `+` in `C++`, the required boundary fails.
|
|
[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30)
|
|
· [Context matching](../../../internal/framework/validators/llm_context.go#L40)
|
|
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
|
|
|
|
**Remediation.** Match configured spellings with boundary checks against
|
|
surrounding runes, including spellings that begin or end with punctuation.
|
|
Preserve longest-match precedence and exemption behavior, and use consistent
|
|
case matching for both the scan and identity lookup.
|
|
|
|
## 3. Merging nested segments truncates the earlier timestamp range
|
|
|
|
**Expected.** The merged interval should cover both source intervals. The same
|
|
interval should determine whether the merge fits the configured duration limit
|
|
and the gap to the next segment. Overlapping source intervals pass input
|
|
validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
|
· [Processing sequence](../../internal/pipeline.md)
|
|
|
|
**Observed.** With the default normalization settings and an empty pipeline,
|
|
same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]`
|
|
containing the first two utterances and a separate `[20,21]` segment.
|
|
`mergeSegments` takes `next.End` even when it is earlier than the current end.
|
|
The next gap is then measured from 6, and duration checks likewise use
|
|
`next.End - current.Start`, which can let an over-limit merged interval pass.
|
|
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
|
|
· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153)
|
|
|
|
**Remediation.** Use the later of the two end times for both the prospective
|
|
duration and the resulting segment. Carry that end forward when calculating
|
|
subsequent gaps.
|
|
|
|
## 4. Balanced chunking rejects feasible explicit targets
|
|
|
|
**Expected.** A configured section count should succeed when the input can be
|
|
partitioned into that many contiguous sections within the token limit.
|
|
Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64)
|
|
|
|
**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers
|
|
to prevent normalization merges, fit in two chunks with a maximum of 6.
|
|
The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts
|
|
the target, but the builder stops the second chunk at 5 because it exceeds the
|
|
average target of 4. It leaves the last segment unassigned and the CLI exits 1
|
|
with “target_sections (2) is impossible under current constraints (got 2 sections).”
|
|
[Feasibility check](../../../internal/core/chunking/sections.go#L143)
|
|
· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249)
|
|
|
|
**Remediation.** Choose balanced cuts only when the remaining suffix can still
|
|
fit in the remaining count. A simpler fallback is to build the minimum number
|
|
of max-bounded chunks and split them deterministically to reach a feasible target.
|
|
|
|
## 5. Cancellation during FIFO delivery can hang and later report success
|
|
|
|
**Expected.** A FIFO can wait for a reader. A catchable cancellation while it
|
|
waits should end processing and attempt failed-status report finalization.
|
|
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
|
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
|
|
|
|
**Observed.** With an empty pipeline and a named pipe as output, the child
|
|
stayed blocked after SIGTERM and left reports at `in_progress`. When a reader
|
|
attached after the signal, it received the transcript and the child exited 0
|
|
with `completed` reports. The main signal handler cancels a context, and the
|
|
CLI checks it before calling `Finish`. The direct-output branch then blocks
|
|
in `os.OpenFile` without observing that context. Delivery and final report
|
|
generation have no check for cancellation during that wait.
|
|
[Signal handling](../../../cmd/audita/main.go#L16)
|
|
· [Output finalization](../../../internal/cli/process_output.go#L54)
|
|
· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318)
|
|
|
|
**Remediation.** Pass cancellation into delivery. For supported FIFO platforms,
|
|
use a context-aware wait for reader availability and for writable capacity,
|
|
handling cancellation through the same failed-report path. Define how a
|
|
cancellation that arrives after stream bytes have already been written is reported.
|
|
|
|
## 6. Transcript shape detection loses integer and unknown-field fidelity
|
|
|
|
**Expected.** Distinct supplied IDs in the supported integer range should
|
|
remain distinct through validation. Unknown fields should be ignored.
|
|
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
|
|
|
**Observed.** On Linux amd64, valid IDs `9007199254740992` and
|
|
`9007199254740993` collapsed to the former value and the CLI rejected them as
|
|
duplicates. An envelope with a valid `segments` array and an unknown field
|
|
`"ignored":1e400` also failed. The first decode targets `any`, so it converts
|
|
numbers throughout the payload to `float64` before the segment array is
|
|
marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35)
|
|
· [Array extraction](../../../internal/core/schema/transcript.go#L115)
|
|
|
|
**Remediation.** Select the top-level shape while retaining the raw segment
|
|
array, then decode it directly into the typed source segment fields. Let the
|
|
typed decode handle range and type validation.
|
|
|
|
## 7. An extra numeric validator reference can discard valid sibling decisions
|
|
|
|
**Expected.** If a validator returns one usable decision per requested
|
|
reference, unexpected reference numbers should be ignored and the successful
|
|
batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching)
|
|
|
|
**Observed.** A fixture returned two valid approvals and an extra decision
|
|
with index `34`. Both corrections applied. Keeping the approvals and changing
|
|
the extra index to `9223372036854775808` or `34.0` made both decisions
|
|
unavailable with `invalid_output`, so neither correction applied. The schema
|
|
accepts those nonnegative integer values. The adapter decodes all indices into
|
|
Go `int` before reconciliation, so one schema-valid but incompatible number
|
|
discards the batch before the out-of-range index can be ignored.
|
|
[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json)
|
|
· [Adapter decode](../../../internal/framework/llm/adapter.go#L191)
|
|
· [Decision index type](../../../internal/framework/validators/llm_models.go#L46)
|
|
· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299)
|
|
|
|
**Remediation.** Preserve validator references as JSON numbers until matching
|
|
them against the known local range. Normalize integral numeric spellings and
|
|
classify out-of-range values as extras before converting requested references
|
|
to Go indices.
|
|
|
|
## 8. Span-only protection can reject a valid edit or miss a repair opportunity
|
|
|
|
**Expected.** The protection validator should check whether a correction
|
|
changes protected occurrences in the target segment. A known protection
|
|
rejection can feed repair, and application should still recheck against the
|
|
text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution)
|
|
· [Processing sequence](../../internal/pipeline.md)
|
|
|
|
**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in
|
|
`The certAIN result follows.` was rejected because the isolated span looks
|
|
like a protected occurrence. The full segment has no occurrence of `AIN` as a
|
|
name. In the other direction, a grammar change from `Silver` to `Golden` in
|
|
`The Silver Shield is equipped.` passed deterministic protection with
|
|
`Silver Shield` protected. It received two scripted model approvals and was
|
|
skipped only at application. With a positive repair budget, the provider
|
|
received no repair request. The validator normally compares the spans alone,
|
|
while application compares the whole segment.
|
|
[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100)
|
|
· [Repair eligibility](../../../internal/framework/runner/repair.go#L40)
|
|
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
|
|
|
|
**Remediation.** Use the shared segment preview for applicable candidates in
|
|
all modules and preserve the spoken-word repetition exception. Keep the
|
|
application check against the current working text for conflicts introduced by
|
|
earlier edits.
|
|
|
|
## 9. Credential redaction can change the ordinary YAML schema
|
|
|
|
**Expected.** The ordinary reports should retain their fixed schema, including
|
|
the required `summary` key and documented status values, while sanitizing
|
|
credential content. [Report schema](../../integrations/output-files.md#corrected-transcript)
|
|
|
|
**Observed.** With the synthetic credential value `summary`, a successful
|
|
processing run produced YAML with `"[REDACTED]"` in place of its required
|
|
`summary` key. After serializing the typed report, redaction applies a
|
|
substitution to every string scalar, including field names and fixed status
|
|
values. The CLI exited 0 despite the schema change.
|
|
[Report serialization](../../../internal/core/reporting/report.go#L182)
|
|
· [YAML redaction](../../../internal/core/redaction/redaction.go#L104)
|
|
|
|
**Remediation.** Sanitize input-derived report values before serialization.
|
|
Keep Audita-owned field names, status enums, and reason codes under the
|
|
control of the typed report serializer.
|
|
|
|
## 10. A completed proposal can disappear from the interruption ledger
|
|
|
|
**Expected.** Debug capture should retain proposed candidates that reached a
|
|
usable response and were subsequently interrupted. Their ledger rows should
|
|
record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files)
|
|
· [Candidate capture](../../internal/diagnostics-reporting.md)
|
|
|
|
**Observed.** In a two-chunk run, the first chunk's proposal request was held
|
|
open while the second returned a correction. After the second chunk's debug
|
|
response artifact appeared, SIGTERM stopped the first request. The run
|
|
published no transcript and correctly reported one usable proposal response
|
|
and two interrupted work units. The correction ledger contained only the
|
|
module failure row. The runner appends candidates only as it drains the
|
|
ordered validation queue. A fatal return or cancellation can leave a complete
|
|
later response in that queue or skip queuing it after recording its operation.
|
|
[Ordered collection](../../../internal/framework/runner/runner.go#L491)
|
|
· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
|
|
|
|
**Remediation.** Retain successful section results independently of validation
|
|
scheduling. After joining workers on a fatal path, assign indices to
|
|
unprocessed successful responses in section and response order and record
|
|
their candidates as interrupted.
|
|
|
|
## 11. Required delivery failures omit the phase in run error records
|
|
|
|
**Expected.** A run-scoped error should identify its phase, including failed
|
|
required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records)
|
|
|
|
**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1
|
|
and wrote a failed `errors.yml` record with `reason: output_failure` and the
|
|
destination path. The record omitted `phase`. The output session distinguishes
|
|
transcript writes, report writes, and cleanup through `FileFailure.Kind`.
|
|
The CLI mapper fills `Phase` only for report initialization.
|
|
[Failure classification](../../../internal/core/diagnostics/output_session.go#L273)
|
|
· [Run error mapping](../../../internal/cli/process_output.go#L92)
|
|
|
|
**Remediation.** Map each required output failure kind to the corresponding
|
|
run error phase, preserving the specific write or cleanup operation through
|
|
finalization and any failed-report retry.
|