Record functional correctness audit findings
This commit is contained in:
261
docs/roadmap/audit/01-correctness.md
Normal file
261
docs/roadmap/audit/01-correctness.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Functional correctness audit
|
||||
|
||||
Reviewed source at `c4ccd78`. All findings below were reproduced through the
|
||||
built CLI with temporary files or a synthetic loopback provider. The existing
|
||||
offline suite also passed with Go 1.27.3 using
|
||||
`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`.
|
||||
The provider fixtures exercised Audita's handling of scripted decisions.
|
||||
Live-model editorial quality was outside the review.
|
||||
|
||||
## Architecture and review scope
|
||||
|
||||
Audita is a single-process Go CLI. CommandKit handles commands and streams,
|
||||
ConfigKit composes configuration, and processing preflight checks role profiles,
|
||||
glossary selections, and output destinations. Core packages parse and normalize
|
||||
the transcript, derive glossary views, and partition the normalized segments into
|
||||
fixed chunks. The framework runs configured correction modules in order while
|
||||
sections can generate proposals concurrently. Deterministic checks and LLM
|
||||
validators gate application. PromptKit handles the provider and continuation
|
||||
history for bounded repair branches, and Audita revalidates revisions. The output
|
||||
session publishes the transcript and reports, with opt-in diagnostics capturing
|
||||
request, decision, and execution evidence. [Package map](../../internal/overview.md)
|
||||
· [Processing sequence](../../internal/pipeline.md)
|
||||
|
||||
The review traced configuration and readiness into file handling, transcript
|
||||
decoding and normalization, chunk feasibility, glossary matching, proposal
|
||||
preview and application, validator batching and response reconciliation,
|
||||
repair outcomes, cancellation, and ordinary and debug reporting. It checked
|
||||
representative success paths alongside malformed inputs, boundaries, recoverable
|
||||
failures, and interrupted work.
|
||||
|
||||
## 1. Output path resolution can replace a different file
|
||||
|
||||
**Expected.** A regular transcript should be published at the selected output
|
||||
destination. Collision checks and publication should agree on which file that
|
||||
path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
|
||||
**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves
|
||||
`logical/link/../out.json` to `physical/out.json`. With two distinct existing
|
||||
files and reporting disabled, Audita exited 0, left `physical/out.json` untouched,
|
||||
and overwrote `logical/out.json`. `Prepare` stats the supplied path, but
|
||||
`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving
|
||||
symlinks. The cleaned path is then used for publication. The same helper feeds
|
||||
collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141)
|
||||
· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377)
|
||||
|
||||
**Remediation.** Resolve directory components with filesystem traversal
|
||||
semantics before collapsing `..`. Use the resulting destination for collision
|
||||
checks, regular-file preparation, and publication. Derive the default report
|
||||
directory from the same destination when file output is selected.
|
||||
|
||||
## 2. Glossary matching misses names with accented letters or punctuation at their edges
|
||||
|
||||
**Expected.** Selected protected names and aliases should participate in both
|
||||
deterministic protection checks. Matching names should also be available to
|
||||
validators when building context. The glossary accepts ordinary strings for
|
||||
names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections)
|
||||
· [Validator context](../../internal/validators.md#supplemental-validation-context)
|
||||
|
||||
**Observed.** With `Élodie` or `C++` in the selected context and protection
|
||||
collection, scripted grammar proposals replaced the names in `Élodie arrived.`
|
||||
and `Use C++ for this.`. Both edits reached model validation and were applied
|
||||
after scripted approvals. The validator requests also omitted the corresponding
|
||||
glossary entries. The shared matcher surrounds every alternative with Go
|
||||
regular-expression `\b`, whose word character class is ASCII. At the start of
|
||||
`Élodie` and after the final `+` in `C++`, the required boundary fails.
|
||||
[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30)
|
||||
· [Context matching](../../../internal/framework/validators/llm_context.go#L40)
|
||||
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
|
||||
|
||||
**Remediation.** Match configured spellings with boundary checks against
|
||||
surrounding runes, including spellings that begin or end with punctuation.
|
||||
Preserve longest-match precedence and exemption behavior, and use consistent
|
||||
case matching for both the scan and identity lookup.
|
||||
|
||||
## 3. Merging nested segments truncates the earlier timestamp range
|
||||
|
||||
**Expected.** The merged interval should cover both source intervals. The same
|
||||
interval should determine whether the merge fits the configured duration limit
|
||||
and the gap to the next segment. Overlapping source intervals pass input
|
||||
validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
||||
· [Processing sequence](../../internal/pipeline.md)
|
||||
|
||||
**Observed.** With the default normalization settings and an empty pipeline,
|
||||
same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]`
|
||||
containing the first two utterances and a separate `[20,21]` segment.
|
||||
`mergeSegments` takes `next.End` even when it is earlier than the current end.
|
||||
The next gap is then measured from 6, and duration checks likewise use
|
||||
`next.End - current.Start`, which can let an over-limit merged interval pass.
|
||||
[Merge loop](../../../internal/core/normalization/normalize.go#L46)
|
||||
· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153)
|
||||
|
||||
**Remediation.** Use the later of the two end times for both the prospective
|
||||
duration and the resulting segment. Carry that end forward when calculating
|
||||
subsequent gaps.
|
||||
|
||||
## 4. Balanced chunking rejects feasible explicit targets
|
||||
|
||||
**Expected.** A configured section count should succeed when the input can be
|
||||
partitioned into that many contiguous sections within the token limit.
|
||||
Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64)
|
||||
|
||||
**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers
|
||||
to prevent normalization merges, fit in two chunks with a maximum of 6.
|
||||
The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts
|
||||
the target, but the builder stops the second chunk at 5 because it exceeds the
|
||||
average target of 4. It leaves the last segment unassigned and the CLI exits 1
|
||||
with “target_sections (2) is impossible under current constraints (got 2 sections).”
|
||||
[Feasibility check](../../../internal/core/chunking/sections.go#L143)
|
||||
· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249)
|
||||
|
||||
**Remediation.** Choose balanced cuts only when the remaining suffix can still
|
||||
fit in the remaining count. A simpler fallback is to build the minimum number
|
||||
of max-bounded chunks and split them deterministically to reach a feasible target.
|
||||
|
||||
## 5. Cancellation during FIFO delivery can hang and later report success
|
||||
|
||||
**Expected.** A FIFO can wait for a reader. A catchable cancellation while it
|
||||
waits should end processing and attempt failed-status report finalization.
|
||||
[Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
· [Subprocess cancellation](../../integrations/subprocess.md#result-handling)
|
||||
|
||||
**Observed.** With an empty pipeline and a named pipe as output, the child
|
||||
stayed blocked after SIGTERM and left reports at `in_progress`. When a reader
|
||||
attached after the signal, it received the transcript and the child exited 0
|
||||
with `completed` reports. The main signal handler cancels a context, and the
|
||||
CLI checks it before calling `Finish`. The direct-output branch then blocks
|
||||
in `os.OpenFile` without observing that context. Delivery and final report
|
||||
generation have no check for cancellation during that wait.
|
||||
[Signal handling](../../../cmd/audita/main.go#L16)
|
||||
· [Output finalization](../../../internal/cli/process_output.go#L54)
|
||||
· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318)
|
||||
|
||||
**Remediation.** Pass cancellation into delivery. For supported FIFO platforms,
|
||||
use a context-aware wait for reader availability and for writable capacity,
|
||||
handling cancellation through the same failed-report path. Define how a
|
||||
cancellation that arrives after stream bytes have already been written is reported.
|
||||
|
||||
## 6. Transcript shape detection loses integer and unknown-field fidelity
|
||||
|
||||
**Expected.** Distinct supplied IDs in the supported integer range should
|
||||
remain distinct through validation. Unknown fields should be ignored.
|
||||
[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract)
|
||||
|
||||
**Observed.** On Linux amd64, valid IDs `9007199254740992` and
|
||||
`9007199254740993` collapsed to the former value and the CLI rejected them as
|
||||
duplicates. An envelope with a valid `segments` array and an unknown field
|
||||
`"ignored":1e400` also failed. The first decode targets `any`, so it converts
|
||||
numbers throughout the payload to `float64` before the segment array is
|
||||
marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35)
|
||||
· [Array extraction](../../../internal/core/schema/transcript.go#L115)
|
||||
|
||||
**Remediation.** Select the top-level shape while retaining the raw segment
|
||||
array, then decode it directly into the typed source segment fields. Let the
|
||||
typed decode handle range and type validation.
|
||||
|
||||
## 7. An extra numeric validator reference can discard valid sibling decisions
|
||||
|
||||
**Expected.** If a validator returns one usable decision per requested
|
||||
reference, unexpected reference numbers should be ignored and the successful
|
||||
batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching)
|
||||
|
||||
**Observed.** A fixture returned two valid approvals and an extra decision
|
||||
with index `34`. Both corrections applied. Keeping the approvals and changing
|
||||
the extra index to `9223372036854775808` or `34.0` made both decisions
|
||||
unavailable with `invalid_output`, so neither correction applied. The schema
|
||||
accepts those nonnegative integer values. The adapter decodes all indices into
|
||||
Go `int` before reconciliation, so one schema-valid but incompatible number
|
||||
discards the batch before the out-of-range index can be ignored.
|
||||
[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json)
|
||||
· [Adapter decode](../../../internal/framework/llm/adapter.go#L191)
|
||||
· [Decision index type](../../../internal/framework/validators/llm_models.go#L46)
|
||||
· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299)
|
||||
|
||||
**Remediation.** Preserve validator references as JSON numbers until matching
|
||||
them against the known local range. Normalize integral numeric spellings and
|
||||
classify out-of-range values as extras before converting requested references
|
||||
to Go indices.
|
||||
|
||||
## 8. Span-only protection can reject a valid edit or miss a repair opportunity
|
||||
|
||||
**Expected.** The protection validator should check whether a correction
|
||||
changes protected occurrences in the target segment. A known protection
|
||||
rejection can feed repair, and application should still recheck against the
|
||||
text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution)
|
||||
· [Processing sequence](../../internal/pipeline.md)
|
||||
|
||||
**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in
|
||||
`The certAIN result follows.` was rejected because the isolated span looks
|
||||
like a protected occurrence. The full segment has no occurrence of `AIN` as a
|
||||
name. In the other direction, a grammar change from `Silver` to `Golden` in
|
||||
`The Silver Shield is equipped.` passed deterministic protection with
|
||||
`Silver Shield` protected. It received two scripted model approvals and was
|
||||
skipped only at application. With a positive repair budget, the provider
|
||||
received no repair request. The validator normally compares the spans alone,
|
||||
while application compares the whole segment.
|
||||
[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100)
|
||||
· [Repair eligibility](../../../internal/framework/runner/repair.go#L40)
|
||||
· [Apply-time protection](../../../internal/framework/runner/runner.go#L300)
|
||||
|
||||
**Remediation.** Use the shared segment preview for applicable candidates in
|
||||
all modules and preserve the spoken-word repetition exception. Keep the
|
||||
application check against the current working text for conflicts introduced by
|
||||
earlier edits.
|
||||
|
||||
## 9. Credential redaction can change the ordinary YAML schema
|
||||
|
||||
**Expected.** The ordinary reports should retain their fixed schema, including
|
||||
the required `summary` key and documented status values, while sanitizing
|
||||
credential content. [Report schema](../../integrations/output-files.md#corrected-transcript)
|
||||
|
||||
**Observed.** With the synthetic credential value `summary`, a successful
|
||||
processing run produced YAML with `"[REDACTED]"` in place of its required
|
||||
`summary` key. After serializing the typed report, redaction applies a
|
||||
substitution to every string scalar, including field names and fixed status
|
||||
values. The CLI exited 0 despite the schema change.
|
||||
[Report serialization](../../../internal/core/reporting/report.go#L182)
|
||||
· [YAML redaction](../../../internal/core/redaction/redaction.go#L104)
|
||||
|
||||
**Remediation.** Sanitize input-derived report values before serialization.
|
||||
Keep Audita-owned field names, status enums, and reason codes under the
|
||||
control of the typed report serializer.
|
||||
|
||||
## 10. A completed proposal can disappear from the interruption ledger
|
||||
|
||||
**Expected.** Debug capture should retain proposed candidates that reached a
|
||||
usable response and were subsequently interrupted. Their ledger rows should
|
||||
record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files)
|
||||
· [Candidate capture](../../internal/diagnostics-reporting.md)
|
||||
|
||||
**Observed.** In a two-chunk run, the first chunk's proposal request was held
|
||||
open while the second returned a correction. After the second chunk's debug
|
||||
response artifact appeared, SIGTERM stopped the first request. The run
|
||||
published no transcript and correctly reported one usable proposal response
|
||||
and two interrupted work units. The correction ledger contained only the
|
||||
module failure row. The runner appends candidates only as it drains the
|
||||
ordered validation queue. A fatal return or cancellation can leave a complete
|
||||
later response in that queue or skip queuing it after recording its operation.
|
||||
[Ordered collection](../../../internal/framework/runner/runner.go#L491)
|
||||
· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
|
||||
|
||||
**Remediation.** Retain successful section results independently of validation
|
||||
scheduling. After joining workers on a fatal path, assign indices to
|
||||
unprocessed successful responses in section and response order and record
|
||||
their candidates as interrupted.
|
||||
|
||||
## 11. Required delivery failures omit the phase in run error records
|
||||
|
||||
**Expected.** A run-scoped error should identify its phase, including failed
|
||||
required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records)
|
||||
|
||||
**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1
|
||||
and wrote a failed `errors.yml` record with `reason: output_failure` and the
|
||||
destination path. The record omitted `phase`. The output session distinguishes
|
||||
transcript writes, report writes, and cleanup through `FileFailure.Kind`.
|
||||
The CLI mapper fills `Phase` only for report initialization.
|
||||
[Failure classification](../../../internal/core/diagnostics/output_session.go#L273)
|
||||
· [Run error mapping](../../../internal/cli/process_output.go#L92)
|
||||
|
||||
**Remediation.** Map each required output failure kind to the corresponding
|
||||
run error phase, preserving the specific write or cleanup operation through
|
||||
finalization and any failed-report retry.
|
||||
Reference in New Issue
Block a user