Files
audita/docs/roadmap/audit/01-correctness.md

262 lines
16 KiB
Markdown
Raw Normal View History

# Functional correctness audit
Reviewed source at `c4ccd78`. All findings below were reproduced through the
built CLI with temporary files or a synthetic loopback provider. The existing
offline suite also passed with Go 1.27.3 using
`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`.
The provider fixtures exercised Audita's handling of scripted decisions.
Live-model editorial quality was outside the review.
## Architecture and review scope
Audita is a single-process Go CLI. CommandKit handles commands and streams,
ConfigKit composes configuration, and processing preflight checks role profiles,
glossary selections, and output destinations. Core packages parse and normalize
the transcript, derive glossary views, and partition the normalized segments into
fixed chunks. The framework runs configured correction modules in order while
sections can generate proposals concurrently. Deterministic checks and LLM
validators gate application. PromptKit handles the provider and continuation
history for bounded repair branches, and Audita revalidates revisions. The output
session publishes the transcript and reports, with opt-in diagnostics capturing
request, decision, and execution evidence. [Package map](../../internal/overview.md)
· [Processing sequence](../../internal/pipeline.md)
The review traced configuration and readiness into file handling, transcript
decoding and normalization, chunk feasibility, glossary matching, proposal
preview and application, validator batching and response reconciliation,
repair outcomes, cancellation, and ordinary and debug reporting. It checked
representative success paths alongside malformed inputs, boundaries, recoverable
failures, and interrupted work.
## 1. Output path resolution can replace a different file
**Expected.** A regular transcript should be published at the selected output
destination. Collision checks and publication should agree on which file that
path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves
`logical/link/../out.json` to `physical/out.json`. With two distinct existing
files and reporting disabled, Audita exited 0, left `physical/out.json` untouched,
and overwrote `logical/out.json`. `Prepare` stats the supplied path, but
`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving
symlinks. The cleaned path is then used for publication. The same helper feeds
collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141)
· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377)
**Remediation.** Resolve directory components with filesystem traversal
semantics before collapsing `..`. Use the resulting destination for collision
checks, regular-file preparation, and publication. Derive the default report
directory from the same destination when file output is selected.
## 2. Glossary matching misses names with accented letters or punctuation at their edges
**Expected.** Selected protected names and aliases should participate in both
deterministic protection checks. Matching names should also be available to
validators when building context. The glossary accepts ordinary strings for
names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections)
· [Validator context](../../internal/validators.md#supplemental-validation-context)
**Observed.** With `Élodie` or `C++` in the selected context and protection
collection, scripted grammar proposals replaced the names in `Élodie arrived.`
and `Use C++ for this.`. Both edits reached model validation and were applied
after scripted approvals. The validator requests also omitted the corresponding
glossary entries. The shared matcher surrounds every alternative with Go
regular-expression `\b`, whose word character class is ASCII. At the start of
`Élodie` and after the final `+` in `C++`, the required boundary fails.
[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30)
· [Context matching](../../../internal/framework/validators/llm_context.go#L40)
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
**Remediation.** Match configured spellings with boundary checks against
surrounding runes, including spellings that begin or end with punctuation.
Preserve longest-match precedence and exemption behavior, and use consistent
case matching for both the scan and identity lookup.
## 3. Merging nested segments truncates the earlier timestamp range
**Expected.** The merged interval should cover both source intervals. The same
interval should determine whether the merge fits the configured duration limit
and the gap to the next segment. Overlapping source intervals pass input
validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
· [Processing sequence](../../internal/pipeline.md)
**Observed.** With the default normalization settings and an empty pipeline,
same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]`
containing the first two utterances and a separate `[20,21]` segment.
`mergeSegments` takes `next.End` even when it is earlier than the current end.
The next gap is then measured from 6, and duration checks likewise use
`next.End - current.Start`, which can let an over-limit merged interval pass.
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153)
**Remediation.** Use the later of the two end times for both the prospective
duration and the resulting segment. Carry that end forward when calculating
subsequent gaps.
## 4. Balanced chunking rejects feasible explicit targets
**Expected.** A configured section count should succeed when the input can be
partitioned into that many contiguous sections within the token limit.
Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64)
**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers
to prevent normalization merges, fit in two chunks with a maximum of 6.
The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts
the target, but the builder stops the second chunk at 5 because it exceeds the
average target of 4. It leaves the last segment unassigned and the CLI exits 1
with “target_sections (2) is impossible under current constraints (got 2 sections).”
[Feasibility check](../../../internal/core/chunking/sections.go#L143)
· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249)
**Remediation.** Choose balanced cuts only when the remaining suffix can still
fit in the remaining count. A simpler fallback is to build the minimum number
of max-bounded chunks and split them deterministically to reach a feasible target.
## 5. Cancellation during FIFO delivery can hang and later report success
**Expected.** A FIFO can wait for a reader. A catchable cancellation while it
waits should end processing and attempt failed-status report finalization.
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
**Observed.** With an empty pipeline and a named pipe as output, the child
stayed blocked after SIGTERM and left reports at `in_progress`. When a reader
attached after the signal, it received the transcript and the child exited 0
with `completed` reports. The main signal handler cancels a context, and the
CLI checks it before calling `Finish`. The direct-output branch then blocks
in `os.OpenFile` without observing that context. Delivery and final report
generation have no check for cancellation during that wait.
[Signal handling](../../../cmd/audita/main.go#L16)
· [Output finalization](../../../internal/cli/process_output.go#L54)
· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318)
**Remediation.** Pass cancellation into delivery. For supported FIFO platforms,
use a context-aware wait for reader availability and for writable capacity,
handling cancellation through the same failed-report path. Define how a
cancellation that arrives after stream bytes have already been written is reported.
## 6. Transcript shape detection loses integer and unknown-field fidelity
**Expected.** Distinct supplied IDs in the supported integer range should
remain distinct through validation. Unknown fields should be ignored.
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
**Observed.** On Linux amd64, valid IDs `9007199254740992` and
`9007199254740993` collapsed to the former value and the CLI rejected them as
duplicates. An envelope with a valid `segments` array and an unknown field
`"ignored":1e400` also failed. The first decode targets `any`, so it converts
numbers throughout the payload to `float64` before the segment array is
marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35)
· [Array extraction](../../../internal/core/schema/transcript.go#L115)
**Remediation.** Select the top-level shape while retaining the raw segment
array, then decode it directly into the typed source segment fields. Let the
typed decode handle range and type validation.
## 7. An extra numeric validator reference can discard valid sibling decisions
**Expected.** If a validator returns one usable decision per requested
reference, unexpected reference numbers should be ignored and the successful
batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching)
**Observed.** A fixture returned two valid approvals and an extra decision
with index `34`. Both corrections applied. Keeping the approvals and changing
the extra index to `9223372036854775808` or `34.0` made both decisions
unavailable with `invalid_output`, so neither correction applied. The schema
accepts those nonnegative integer values. The adapter decodes all indices into
Go `int` before reconciliation, so one schema-valid but incompatible number
discards the batch before the out-of-range index can be ignored.
[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json)
· [Adapter decode](../../../internal/framework/llm/adapter.go#L191)
· [Decision index type](../../../internal/framework/validators/llm_models.go#L46)
· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299)
**Remediation.** Preserve validator references as JSON numbers until matching
them against the known local range. Normalize integral numeric spellings and
classify out-of-range values as extras before converting requested references
to Go indices.
## 8. Span-only protection can reject a valid edit or miss a repair opportunity
**Expected.** The protection validator should check whether a correction
changes protected occurrences in the target segment. A known protection
rejection can feed repair, and application should still recheck against the
text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution)
· [Processing sequence](../../internal/pipeline.md)
**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in
`The certAIN result follows.` was rejected because the isolated span looks
like a protected occurrence. The full segment has no occurrence of `AIN` as a
name. In the other direction, a grammar change from `Silver` to `Golden` in
`The Silver Shield is equipped.` passed deterministic protection with
`Silver Shield` protected. It received two scripted model approvals and was
skipped only at application. With a positive repair budget, the provider
received no repair request. The validator normally compares the spans alone,
while application compares the whole segment.
[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100)
· [Repair eligibility](../../../internal/framework/runner/repair.go#L40)
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
**Remediation.** Use the shared segment preview for applicable candidates in
all modules and preserve the spoken-word repetition exception. Keep the
application check against the current working text for conflicts introduced by
earlier edits.
## 9. Credential redaction can change the ordinary YAML schema
**Expected.** The ordinary reports should retain their fixed schema, including
the required `summary` key and documented status values, while sanitizing
credential content. [Report schema](../../integrations/output-files.md#corrected-transcript)
**Observed.** With the synthetic credential value `summary`, a successful
processing run produced YAML with `"[REDACTED]"` in place of its required
`summary` key. After serializing the typed report, redaction applies a
substitution to every string scalar, including field names and fixed status
values. The CLI exited 0 despite the schema change.
[Report serialization](../../../internal/core/reporting/report.go#L182)
· [YAML redaction](../../../internal/core/redaction/redaction.go#L104)
**Remediation.** Sanitize input-derived report values before serialization.
Keep Audita-owned field names, status enums, and reason codes under the
control of the typed report serializer.
## 10. A completed proposal can disappear from the interruption ledger
**Expected.** Debug capture should retain proposed candidates that reached a
usable response and were subsequently interrupted. Their ledger rows should
record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files)
· [Candidate capture](../../internal/diagnostics-reporting.md)
**Observed.** In a two-chunk run, the first chunk's proposal request was held
open while the second returned a correction. After the second chunk's debug
response artifact appeared, SIGTERM stopped the first request. The run
published no transcript and correctly reported one usable proposal response
and two interrupted work units. The correction ledger contained only the
module failure row. The runner appends candidates only as it drains the
ordered validation queue. A fatal return or cancellation can leave a complete
later response in that queue or skip queuing it after recording its operation.
[Ordered collection](../../../internal/framework/runner/runner.go#L491)
· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
**Remediation.** Retain successful section results independently of validation
scheduling. After joining workers on a fatal path, assign indices to
unprocessed successful responses in section and response order and record
their candidates as interrupted.
## 11. Required delivery failures omit the phase in run error records
**Expected.** A run-scoped error should identify its phase, including failed
required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records)
**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1
and wrote a failed `errors.yml` record with `reason: output_failure` and the
destination path. The record omitted `phase`. The output session distinguishes
transcript writes, report writes, and cleanup through `FileFailure.Kind`.
The CLI mapper fills `Phase` only for report initialization.
[Failure classification](../../../internal/core/diagnostics/output_session.go#L273)
· [Run error mapping](../../../internal/cli/process_output.go#L92)
**Remediation.** Map each required output failure kind to the corresponding
run error phase, preserving the specific write or cleanup operation through
finalization and any failed-report retry.