diff --git a/docs/roadmap/audit/01-correctness.md b/docs/roadmap/audit/01-correctness.md new file mode 100644 index 0000000..c4b46bd --- /dev/null +++ b/docs/roadmap/audit/01-correctness.md @@ -0,0 +1,261 @@ +# Functional correctness audit + +Reviewed source at `c4ccd78`. All findings below were reproduced through the +built CLI with temporary files or a synthetic loopback provider. The existing +offline suite also passed with Go 1.27.3 using +`go test -count=1 -race -coverprofile=/tmp/audita-correctness-coverage.out ./...`. +The provider fixtures exercised Audita's handling of scripted decisions. +Live-model editorial quality was outside the review. + +## Architecture and review scope + +Audita is a single-process Go CLI. CommandKit handles commands and streams, +ConfigKit composes configuration, and processing preflight checks role profiles, +glossary selections, and output destinations. Core packages parse and normalize +the transcript, derive glossary views, and partition the normalized segments into +fixed chunks. The framework runs configured correction modules in order while +sections can generate proposals concurrently. Deterministic checks and LLM +validators gate application. PromptKit handles the provider and continuation +history for bounded repair branches, and Audita revalidates revisions. The output +session publishes the transcript and reports, with opt-in diagnostics capturing +request, decision, and execution evidence. [Package map](../../internal/overview.md) +· [Processing sequence](../../internal/pipeline.md) + +The review traced configuration and readiness into file handling, transcript +decoding and normalization, chunk feasibility, glossary matching, proposal +preview and application, validator batching and response reconciliation, +repair outcomes, cancellation, and ordinary and debug reporting. It checked +representative success paths alongside malformed inputs, boundaries, recoverable +failures, and interrupted work. + +## 1. Output path resolution can replace a different file + +**Expected.** A regular transcript should be published at the selected output +destination. Collision checks and publication should agree on which file that +path names. [Output lifecycle](../../operations.md#outputs-and-lifecycle) + +**Observed.** With `logical/link` pointing to `physical/inside`, Linux resolves +`logical/link/../out.json` to `physical/out.json`. With two distinct existing +files and reporting disabled, Audita exited 0, left `physical/out.json` untouched, +and overwrote `logical/out.json`. `Prepare` stats the supplied path, but +`canonicalDestination` uses `filepath.Abs` and cleans the path before resolving +symlinks. The cleaned path is then used for publication. The same helper feeds +collision checks and report replacement. [Output preparation](../../../internal/core/diagnostics/output_session.go#L141) +· [Destination resolution](../../../internal/core/diagnostics/output_session.go#L377) + +**Remediation.** Resolve directory components with filesystem traversal +semantics before collapsing `..`. Use the resulting destination for collision +checks, regular-file preparation, and publication. Derive the default report +directory from the same destination when file output is selected. + +## 2. Glossary matching misses names with accented letters or punctuation at their edges + +**Expected.** Selected protected names and aliases should participate in both +deterministic protection checks. Matching names should also be available to +validators when building context. The glossary accepts ordinary strings for +names and aliases. [Glossary selections](../../config.md#pipeline-and-glossary-selections) +· [Validator context](../../internal/validators.md#supplemental-validation-context) + +**Observed.** With `Élodie` or `C++` in the selected context and protection +collection, scripted grammar proposals replaced the names in `Élodie arrived.` +and `Use C++ for this.`. Both edits reached model validation and were applied +after scripted approvals. The validator requests also omitted the corresponding +glossary entries. The shared matcher surrounds every alternative with Go +regular-expression `\b`, whose word character class is ASCII. At the start of +`Élodie` and after the final `+` in `C++`, the required boundary fails. +[Protected vocabulary](../../../internal/framework/validators/protected_terms.go#L30) +· [Context matching](../../../internal/framework/validators/llm_context.go#L40) +· [Apply-time protection](../../../internal/framework/runner/runner.go#L300) + +**Remediation.** Match configured spellings with boundary checks against +surrounding runes, including spellings that begin or end with punctuation. +Preserve longest-match precedence and exemption behavior, and use consistent +case matching for both the scan and identity lookup. + +## 3. Merging nested segments truncates the earlier timestamp range + +**Expected.** The merged interval should cover both source intervals. The same +interval should determine whether the merge fits the configured duration limit +and the gap to the next segment. Overlapping source intervals pass input +validation. [Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) +· [Processing sequence](../../internal/pipeline.md) + +**Observed.** With the default normalization settings and an empty pipeline, +same-speaker intervals `[0,60]`, `[5,6]`, and `[20,21]` produced `[0,6]` +containing the first two utterances and a separate `[20,21]` segment. +`mergeSegments` takes `next.End` even when it is earlier than the current end. +The next gap is then measured from 6, and duration checks likewise use +`next.End - current.Start`, which can let an over-limit merged interval pass. +[Merge loop](../../../internal/core/normalization/normalize.go#L46) +· [Duration and merged segment](../../../internal/core/normalization/normalize.go#L153) + +**Remediation.** Use the later of the two end times for both the prospective +duration and the resulting segment. Carry that end forward when calculating +subsequent gaps. + +## 4. Balanced chunking rejects feasible explicit targets + +**Expected.** A configured section count should succeed when the input can be +partitioned into that many contiguous sections within the token limit. +Balancing is a preference. [Chunking contract](../../../internal/core/chunking/sections.go#L64) + +**Observed.** Three segments with token estimates `[2,5,1]`, alternating speakers +to prevent normalization merges, fit in two chunks with a maximum of 6. +The feasible partition is `[2]` and `[5,1]`. The feasibility check accepts +the target, but the builder stops the second chunk at 5 because it exceeds the +average target of 4. It leaves the last segment unassigned and the CLI exits 1 +with “target_sections (2) is impossible under current constraints (got 2 sections).” +[Feasibility check](../../../internal/core/chunking/sections.go#L143) +· [Explicit-target builder](../../../internal/core/chunking/sections.go#L249) + +**Remediation.** Choose balanced cuts only when the remaining suffix can still +fit in the remaining count. A simpler fallback is to build the minimum number +of max-bounded chunks and split them deterministically to reach a feasible target. + +## 5. Cancellation during FIFO delivery can hang and later report success + +**Expected.** A FIFO can wait for a reader. A catchable cancellation while it +waits should end processing and attempt failed-status report finalization. +[Output lifecycle](../../operations.md#outputs-and-lifecycle) +· [Subprocess cancellation](../../integrations/subprocess.md#result-handling) + +**Observed.** With an empty pipeline and a named pipe as output, the child +stayed blocked after SIGTERM and left reports at `in_progress`. When a reader +attached after the signal, it received the transcript and the child exited 0 +with `completed` reports. The main signal handler cancels a context, and the +CLI checks it before calling `Finish`. The direct-output branch then blocks +in `os.OpenFile` without observing that context. Delivery and final report +generation have no check for cancellation during that wait. +[Signal handling](../../../cmd/audita/main.go#L16) +· [Output finalization](../../../internal/cli/process_output.go#L54) +· [Direct stream delivery](../../../internal/core/diagnostics/output_session.go#L318) + +**Remediation.** Pass cancellation into delivery. For supported FIFO platforms, +use a context-aware wait for reader availability and for writable capacity, +handling cancellation through the same failed-report path. Define how a +cancellation that arrives after stream bytes have already been written is reported. + +## 6. Transcript shape detection loses integer and unknown-field fidelity + +**Expected.** Distinct supplied IDs in the supported integer range should +remain distinct through validation. Unknown fields should be ignored. +[Transcript input contract](../../integrations/transcript-glossary-files.md#transcript-json-contract) + +**Observed.** On Linux amd64, valid IDs `9007199254740992` and +`9007199254740993` collapsed to the former value and the CLI rejected them as +duplicates. An envelope with a valid `segments` array and an unknown field +`"ignored":1e400` also failed. The first decode targets `any`, so it converts +numbers throughout the payload to `float64` before the segment array is +marshaled and decoded again. [Initial decode](../../../internal/core/schema/transcript.go#L35) +· [Array extraction](../../../internal/core/schema/transcript.go#L115) + +**Remediation.** Select the top-level shape while retaining the raw segment +array, then decode it directly into the typed source segment fields. Let the +typed decode handle range and type validation. + +## 7. An extra numeric validator reference can discard valid sibling decisions + +**Expected.** If a validator returns one usable decision per requested +reference, unexpected reference numbers should be ignored and the successful +batch should record a filtering recovery. [Reconciliation contract](../../internal/validators.md#llm-validator-batching) + +**Observed.** A fixture returned two valid approvals and an extra decision +with index `34`. Both corrections applied. Keeping the approvals and changing +the extra index to `9223372036854775808` or `34.0` made both decisions +unavailable with `invalid_output`, so neither correction applied. The schema +accepts those nonnegative integer values. The adapter decodes all indices into +Go `int` before reconciliation, so one schema-valid but incompatible number +discards the batch before the out-of-range index can be ignored. +[Response schema](../../../internal/framework/responseschema/assets/validator_decision_set.v1.json) +· [Adapter decode](../../../internal/framework/llm/adapter.go#L191) +· [Decision index type](../../../internal/framework/validators/llm_models.go#L46) +· [Reconciliation](../../../internal/framework/validators/llm_validators.go#L299) + +**Remediation.** Preserve validator references as JSON numbers until matching +them against the known local range. Normalize integral numeric spellings and +classify out-of-range values as extras before converting requested references +to Go indices. + +## 8. Span-only protection can reject a valid edit or miss a repair opportunity + +**Expected.** The protection validator should check whether a correction +changes protected occurrences in the target segment. A known protection +rejection can feed repair, and application should still recheck against the +text produced by earlier corrections. [Validator execution](../../internal/validators.md#runtime-execution) +· [Processing sequence](../../internal/pipeline.md) + +**Observed.** With `AIN` protected, a grammar edit from `AIN` to `ain` in +`The certAIN result follows.` was rejected because the isolated span looks +like a protected occurrence. The full segment has no occurrence of `AIN` as a +name. In the other direction, a grammar change from `Silver` to `Golden` in +`The Silver Shield is equipped.` passed deterministic protection with +`Silver Shield` protected. It received two scripted model approvals and was +skipped only at application. With a positive repair budget, the provider +received no repair request. The validator normally compares the spans alone, +while application compares the whole segment. +[Deterministic protection](../../../internal/framework/validators/deterministic.go#L100) +· [Repair eligibility](../../../internal/framework/runner/repair.go#L40) +· [Apply-time protection](../../../internal/framework/runner/runner.go#L300) + +**Remediation.** Use the shared segment preview for applicable candidates in +all modules and preserve the spoken-word repetition exception. Keep the +application check against the current working text for conflicts introduced by +earlier edits. + +## 9. Credential redaction can change the ordinary YAML schema + +**Expected.** The ordinary reports should retain their fixed schema, including +the required `summary` key and documented status values, while sanitizing +credential content. [Report schema](../../integrations/output-files.md#corrected-transcript) + +**Observed.** With the synthetic credential value `summary`, a successful +processing run produced YAML with `"[REDACTED]"` in place of its required +`summary` key. After serializing the typed report, redaction applies a +substitution to every string scalar, including field names and fixed status +values. The CLI exited 0 despite the schema change. +[Report serialization](../../../internal/core/reporting/report.go#L182) +· [YAML redaction](../../../internal/core/redaction/redaction.go#L104) + +**Remediation.** Sanitize input-derived report values before serialization. +Keep Audita-owned field names, status enums, and reason codes under the +control of the typed report serializer. + +## 10. A completed proposal can disappear from the interruption ledger + +**Expected.** Debug capture should retain proposed candidates that reached a +usable response and were subsequently interrupted. Their ledger rows should +record that they never reached application. [Debug contract](../../integrations/output-files.md#debug-only-files) +· [Candidate capture](../../internal/diagnostics-reporting.md) + +**Observed.** In a two-chunk run, the first chunk's proposal request was held +open while the second returned a correction. After the second chunk's debug +response artifact appeared, SIGTERM stopped the first request. The run +published no transcript and correctly reported one usable proposal response +and two interrupted work units. The correction ledger contained only the +module failure row. The runner appends candidates only as it drains the +ordered validation queue. A fatal return or cancellation can leave a complete +later response in that queue or skip queuing it after recording its operation. +[Ordered collection](../../../internal/framework/runner/runner.go#L491) +· [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70) + +**Remediation.** Retain successful section results independently of validation +scheduling. After joining workers on a fatal path, assign indices to +unprocessed successful responses in section and response order and record +their candidates as interrupted. + +## 11. Required delivery failures omit the phase in run error records + +**Expected.** A run-scoped error should identify its phase, including failed +required output delivery. [Error record contract](../../integrations/output-files.md#error-and-recovery-records) + +**Observed.** Delivering a normalized transcript to Linux `/dev/full` exited 1 +and wrote a failed `errors.yml` record with `reason: output_failure` and the +destination path. The record omitted `phase`. The output session distinguishes +transcript writes, report writes, and cleanup through `FileFailure.Kind`. +The CLI mapper fills `Phase` only for report initialization. +[Failure classification](../../../internal/core/diagnostics/output_session.go#L273) +· [Run error mapping](../../../internal/cli/process_output.go#L92) + +**Remediation.** Map each required output failure kind to the corresponding +run error phase, preserving the specific write or cleanup operation through +finalization and any failed-report retry.