Record maintainability and complexity audit findings
This commit is contained in:
322
docs/roadmap/audit/03-complexity.md
Normal file
322
docs/roadmap/audit/03-complexity.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Maintainability and complexity audit
|
||||
|
||||
Reviewed source at `717d59c`. The codebase graph identified complex control
|
||||
flow and cross-package call paths. A Go AST scan counted branches and
|
||||
declaration spans for 353 functions and methods across 43 tracked Go
|
||||
directories, excluding test files. The score below starts at one and adds
|
||||
one per conditional, loop, non-default switch or select case, and short-circuit
|
||||
operator, including those inside callbacks. The ranking guided source review.
|
||||
Findings are based on static analysis.
|
||||
|
||||
## Architecture
|
||||
|
||||
Audita is a single-process Go CLI. CommandKit routes commands and streams.
|
||||
ConfigKit resolves application settings, and the core packages parse,
|
||||
normalize, and chunk transcripts and derive glossary selections. The
|
||||
runner executes modules in order over fixed chunks. Deterministic and
|
||||
LLM-backed validators gate proposals and revalidate repairs. PromptKit owns
|
||||
model execution, backend admission, and bounded conversation history. Audita
|
||||
applies approved edits and derives reports and debug artifacts from processing
|
||||
outcomes. [Package map](../../internal/overview.md)
|
||||
· [Pipeline](../../internal/pipeline.md)
|
||||
· [Ownership rules](../../policy/architecture.md#ownership-and-dependency-direction)
|
||||
|
||||
The runner and validator runtime had the largest non-test Go source totals,
|
||||
at 1,373 and 1,351 physical lines respectively. The highest-scoring functions
|
||||
concentrated in orchestration, model integration, and report projection.
|
||||
|
||||
| Function | Branch score | Declaration lines |
|
||||
| --- | ---: | ---: |
|
||||
| `runner.repairCandidates` | 46 | 225 |
|
||||
| `runner.runModulePipeline` | 36 | 212 |
|
||||
| `processreport.Build` | 36 | 140 |
|
||||
| `(*LLMBackedValidator).Validate` | 31 | 170 |
|
||||
| `(*Runner).Run` | 30 | 168 |
|
||||
| `runner.validateSectionCandidates` | 30 | 154 |
|
||||
| `(*RoleClient).prepareCall` | 28 | 153 |
|
||||
| `processreport.BuildCorrectionLedger` | 27 | 176 |
|
||||
| `proposal_generation.GenerateCandidates` | 25 | 148 |
|
||||
| `diagnostics.SelectOutput` | 21 | 66 |
|
||||
|
||||
## Refactoring opportunities
|
||||
|
||||
### 1. Give module execution one place to commit its result
|
||||
|
||||
`Run` owns the stage barrier, working transcript, inventory, module setup,
|
||||
application, and partial-return assembly. The pipeline-error, application-error,
|
||||
and success paths each construct a module result and copy work units into the
|
||||
invocation inventory. Repair approvals are marked interrupted in both `Run`
|
||||
and `runModulePipeline`. The latter also coordinates two worker lifetimes,
|
||||
ordered proposal indexing, cancellation, and evidence aggregation. Changing
|
||||
what an interrupted unit retains requires following both functions and their
|
||||
callbacks. The earlier missing-ledger-candidate finding follows from indexing
|
||||
and retention occurring only while scheduling validation. [Module commit and failure paths](../../../internal/framework/runner/runner.go#L187)
|
||||
· [Section orchestration](../../../internal/framework/runner/runner.go#L441)
|
||||
· [Interrupted evidence finding](01-correctness.md#10-a-completed-proposal-can-disappear-from-the-interruption-ledger)
|
||||
|
||||
Extract the section validation-and-repair worker from the collector. Let the
|
||||
collector own ordered indexing and the evidence returned by workers, with
|
||||
one path to register a fatal cause and cancel peers. Let one module-level
|
||||
completion path resolve applied, skipped, and interrupted repair outcomes
|
||||
and commit updated sections after successful application. Preserve validation
|
||||
starting as ordered proposals become available, deterministic proposal IDs,
|
||||
joined workers on fatal paths, and the stage barrier.
|
||||
|
||||
### 2. Separate one repair branch from the section's repair loop
|
||||
|
||||
`repairCandidates` selects candidates, builds each request, interprets
|
||||
delivered turns, revalidates revisions, attributes outcomes, and writes debug
|
||||
artifacts. Nested closures share mutable `current`, `localRef`, `findings`,
|
||||
attempt counts, and section-level slices. For example, a successful revision
|
||||
updates the branch, original proposal filtering, revalidation operations, and
|
||||
approval list inside the coordinator callback. Error, withdrawal, exhaustion,
|
||||
and accepted final-turn paths finish that work at different points.
|
||||
[Branch setup](../../../internal/framework/runner/repair.go#L56)
|
||||
· [Turn handling](../../../internal/framework/runner/repair.go#L146)
|
||||
· [Repair behavior](../../internal/modules.md#failure-and-outcome-behavior)
|
||||
|
||||
Keep the outer loop responsible for selecting and collecting branches. Have
|
||||
one branch-local operation return the terminal outcome, optional approved
|
||||
revision, and operation evidence. Separate construction of the next request,
|
||||
assessment of a delivered turn, and optional attempt recording. Define the
|
||||
branch's internal states once so application and reports can project their
|
||||
own stable labels. Keep PromptKit responsible for history and turn limits.
|
||||
Retain final-turn revalidation, target changes within the section, and
|
||||
independence of sibling approvals.
|
||||
|
||||
### 3. Put validator applicability and decision meaning at their owning boundary
|
||||
|
||||
`feedbackCandidates` switches on validator names and module keys to decide
|
||||
whether each validator has usable evidence. The validator implementations
|
||||
also check previews and module policy. The runner then rewrites an
|
||||
`original_text_presence` approval if its preview is ambiguous, so the same
|
||||
validator's returned verdict can mean something different outside the runner.
|
||||
`validateSectionCandidates` interleaves those checks with two eligibility
|
||||
modes, cardinality checks, filtering, and evidence recording. The framework
|
||||
also declares identical validator interfaces under `contracts` and
|
||||
`validators`. [Feedback selection](../../../internal/framework/runner/feedback.go#L13)
|
||||
· [Runner verdict handling](../../../internal/framework/runner/runner.go#L680)
|
||||
· [Original-text validator](../../../internal/framework/validators/deterministic.go#L37)
|
||||
· [Validator contract](../../../internal/framework/contracts/contracts.go#L29)
|
||||
· [Duplicate interface](../../../internal/framework/validators/models.go#L95)
|
||||
|
||||
Put each check's applicability and verdict together in its validator or its
|
||||
built-in registration. Use one validator interface and one internal decision
|
||||
disposition, deriving serialized approval from it. Keep the runner responsible
|
||||
for submitting the shortlist or collecting independent findings, checking
|
||||
returned identities, and applying only fully approved candidates. Preserve
|
||||
ordered chains, zero-budget shortlisting, unavailable decisions, and separate
|
||||
revalidation of revisions. The earlier audit records the span-only
|
||||
protection mismatch. [Protection correctness](01-correctness.md#8-span-only-protection-can-reject-a-valid-edit-or-miss-a-repair-opportunity)
|
||||
|
||||
### 4. Separate validation-batch assessment from the diagnostic writer
|
||||
|
||||
`LLMBackedValidator.Validate` interleaves request preparation, prompt selection,
|
||||
batch dispatch, reconciliation, partial-result assembly, and debug payload
|
||||
construction. Its nested writer consumes the original request, local
|
||||
reference mapping, raw response, reconciliation result, and failure. Proposal
|
||||
generation and validation declare writer interfaces with matching signatures
|
||||
and duplicate four-field artifact types. Their adapter wrappers copy each
|
||||
path from `llm.InteractionArtifacts`. Repair writes through the same concrete
|
||||
writer directly. A change to the common artifact envelope or safe error
|
||||
serialization therefore touches several packages.
|
||||
[Batch loop](../../../internal/framework/validators/llm_validators.go#L112)
|
||||
· [Proposal writer contract](../../../internal/framework/proposal_generation/generate.go#L23)
|
||||
· [Validation writer contract](../../../internal/framework/validators/models.go#L84)
|
||||
· [Writer adapter](../../../internal/framework/runner/runner.go#L994)
|
||||
· [Repair artifact capture](../../../internal/framework/runner/repair.go#L165)
|
||||
|
||||
Give one batch operation responsibility for dispatch and assessment. Return
|
||||
reconciled decisions, its compact outcome, and only the evidence needed by the
|
||||
diagnostic path. Reuse one internal artifact value type and writer seam for
|
||||
common metadata, redaction, and safe write failures. Keep the per-role
|
||||
payload construction with its domain owner. Preserve local-reference
|
||||
reconciliation, missing and duplicate decision handling, partial results on
|
||||
fatal return, and nonfatal debug-write warnings.
|
||||
|
||||
### 5. Make the structured client's call lifecycle explicit
|
||||
|
||||
`prepareCall` allocates a logical call, validates the output target, builds
|
||||
the native request, installs two observer paths, captures prepared evidence,
|
||||
and returns a closure for decoding and completion. That closure assigns
|
||||
captured `err` from a named result and executes a reverse-order stack of
|
||||
two finalizers before returning the mutated response. `Coordinate` also
|
||||
finalizes a pending call on its separate undelivered-result path. Following an
|
||||
error across both entry points requires keeping the prepared snapshot, native
|
||||
terminal event, raw output, typed result, and logical call status in mind.
|
||||
`Failure.Kind` and reason-based logical-call classification both describe
|
||||
malformed-output failures. The ordinary request also retains a continuation
|
||||
field whose current callers are adapter tests. [Call preparation and finalization](../../../internal/framework/llm/adapter.go#L54)
|
||||
· [Coordinated call](../../../internal/framework/llm/coordinate.go#L15)
|
||||
· [Request contract](../../../internal/framework/structuredoutput/call.go#L28)
|
||||
· [Failure contract](../../../internal/framework/structuredoutput/failure.go#L17)
|
||||
|
||||
Keep the two native entry points and make their shared finish operation
|
||||
explicit. Move native request construction and result decoding out of the
|
||||
bookkeeping closure. Have a concrete per-call record finalize observation,
|
||||
execution counts, and response evidence exactly once on both success and
|
||||
failure. Derive the existing diagnostic failure label from one classification
|
||||
and assess whether the test-only continuation path still needs to be exposed.
|
||||
Preserve terminal counts, preparation failures without a snapshot, and the
|
||||
rule that an undelivered response supplies diagnostic evidence only.
|
||||
[LLM runtime](../../internal/llm-runtime.md)
|
||||
|
||||
### 6. Use a single proposal handoff and wire repair capability explicitly
|
||||
|
||||
`GenerateCandidates` returns a correction slice, parallel segment references,
|
||||
and a second enriched proposal slice with its own indexing inputs.
|
||||
`ExecuteModuleProposal` forwards only the first two slices and the
|
||||
exchange. The runner later builds its own enriched candidates, which is where
|
||||
stage-wide index assignment takes place. Repair recovers the request-local
|
||||
segment reference by the proposal's slice position or falls back to looking
|
||||
up the resolved target. Separately, an enabled repair budget, proposal-exchange
|
||||
capture, the module's repair prompt method, and the repair client's coordinator
|
||||
capability are established at different boundaries.
|
||||
[Proposal result and mapping](../../../internal/framework/proposal_generation/generate.go#L72)
|
||||
· [Module handoff](../../../internal/framework/proposal_generation/module_proposal.go#L75)
|
||||
· [Ordered index assignment](../../../internal/framework/runner/runner.go#L521)
|
||||
· [Repair readiness at use](../../../internal/framework/runner/repair.go#L98)
|
||||
|
||||
Carry a single proposal record with the resolved target and original local
|
||||
reference, then assign the stage-wide index at the ordered runner boundary.
|
||||
Assess retirement of the generator's separate enriched return once its
|
||||
callers are migrated. Resolve repair availability with the concrete
|
||||
module and role client at module setup. Use that capability to determine
|
||||
exchange capture, and keep the disabled path simple. Preserve original
|
||||
response order, invalid reference evidence, required profile readiness,
|
||||
and zero-budget behavior.
|
||||
|
||||
### 7. Give reporting one source for operation classification and counts
|
||||
|
||||
`processreport.Build` has eleven work-unit and operation counters, nested
|
||||
aggregation and projection, and a separate pass for run failures.
|
||||
`SummaryOnly` bypasses selected detail paths inside that aggregation.
|
||||
The LLM adapter decides recoverability through `structuredoutput.Failure`,
|
||||
while `Build` uses a separate allowlist of recoverable reason codes after
|
||||
the class has been discarded from the operation. Adding a recoverable
|
||||
reason can therefore require coordinated edits to the adapter, domain
|
||||
operation, and report counting. Safe report messages are a presentation
|
||||
concern and can stay at that boundary.
|
||||
[Snapshot aggregation](../../../internal/framework/processreport/report.go#L36)
|
||||
· [Report-side classification](../../../internal/framework/processreport/report.go#L212)
|
||||
· [Processing operation](../../../internal/framework/outcomes/outcomes.go#L84)
|
||||
· [Adapter classification](../../../internal/framework/llm/adapter.go#L228)
|
||||
|
||||
Give operation outcomes a single source for success and recoverability.
|
||||
Accumulate work-unit facts and summary counts in one pass, then project
|
||||
rows and detail records when requested. Keep presentation ordering and safe
|
||||
messages explicit. Preserve unknown versus zero counts, a single recovery
|
||||
per successful operation, and agreement among YAML, Markdown, and CLI counts.
|
||||
[Ordinary report contract](../../integrations/output-files.md#summary-counts)
|
||||
|
||||
### 8. Join correction evidence before building ledger rows
|
||||
|
||||
`BuildCorrectionLedger` indexes decisions, repairs, and skips, then emits
|
||||
applied, skipped, rejected, repaired, interrupted, and module-failure rows
|
||||
from separate loops. Each candidate path repeats identity, replacement,
|
||||
and decision projections. Precedence is implicit in iteration order,
|
||||
the `seen` map, and checks for repair evidence. It is easy to add a field to
|
||||
one row shape and miss the other cases. The ledger also looks up validator
|
||||
class by name after execution, while the runner can use metadata supplied
|
||||
by the validator itself. [Ledger assembly](../../../internal/framework/processreport/correction_ledger.go#L70)
|
||||
· [Repair and interruption rows](../../../internal/framework/processreport/correction_ledger.go#L197)
|
||||
· [Decision grouping](../../../internal/framework/processreport/correction_ledger.go#L247)
|
||||
· [Runtime class selection](../../../internal/validators/metadata/metadata.go#L44)
|
||||
|
||||
Gather the existing evidence into one per-proposal record keyed by module
|
||||
instance and proposal index. Determine final disposition in one place, then
|
||||
project common fields once and add the application and repair fields that
|
||||
belong to that disposition. Consider recording the execution class used
|
||||
when the decision is made, preserving built-in metadata. Keep module-only
|
||||
failures distinct and retain the existing ordering, empty replacement fields,
|
||||
partial decisions, and repaired skip reasons.
|
||||
[Debug ledger contract](../../integrations/output-files.md#debug-only-files)
|
||||
|
||||
### 9. Clarify output preparation, publication, and failure ownership
|
||||
|
||||
The CLI builds initial and final report bundles and two failed-report
|
||||
renderers. Preflight wraps the initial renderer to cover setup cancellation.
|
||||
`OutputSession.Initialize` and `Finalize` each iterate required reports and
|
||||
run a second write pass for failed status. File failure purposes are strings
|
||||
interpreted again in the CLI. Destination selection also infers its role by
|
||||
comparing the path string to the transcript destination, and preparation and
|
||||
delivery each re-derive aspects of the path and file type.
|
||||
[CLI preparation](../../../internal/cli/process_inputs.go#L70)
|
||||
· [Preflight handoff](../../../internal/framework/preflight/preflight.go#L41)
|
||||
· [Output selection](../../../internal/core/diagnostics/output_session.go#L61)
|
||||
· [Publication and retry](../../../internal/core/diagnostics/output_session.go#L222)
|
||||
· [Failure projection](../../../internal/cli/process_output.go#L92)
|
||||
|
||||
Keep a checked destination with its explicit role and publication mode.
|
||||
Reuse a report-write helper for both passes and make write or cleanup phases
|
||||
typed facts of the output session. Keep report rendering and status in one CLI
|
||||
handoff that can distinguish a valid failed snapshot from an unavailable one.
|
||||
Preserve readiness before writes, intentional in-place transcript replacement,
|
||||
required-output ordering, cleanup, and the bounded best-effort failed-report
|
||||
pass. Related destination, cancellation, and phase defects are in the
|
||||
correctness audit. [Output lifecycle](../../operations.md#outputs-and-lifecycle)
|
||||
· [Correctness audit](01-correctness.md)
|
||||
|
||||
### 10. Share root and per-module configuration rules
|
||||
|
||||
`resolveSettings` repeats the same presence, null, inheritance, and fallback
|
||||
logic for the validation and repair profile fields. Each root field also
|
||||
has a separate fallback path after the stage loop. `validate` repeats the
|
||||
LLM value checks at the root and at every stage, with a separately built
|
||||
ConfigKit path for each. Repair and validation currently agree, but changing
|
||||
one profile rule requires reading multiple versions of it.
|
||||
[Profile resolution](../../../internal/core/config/resolve.go#L24)
|
||||
· [Configuration validation](../../../internal/core/config/config.go#L143)
|
||||
|
||||
Use a small typed helper for profile fallback that receives the correct
|
||||
path, value, and dependency. Capture explicitly supplied root values before
|
||||
recording root fallbacks, so a stage that selects a different proposal
|
||||
profile still resolves correctly. Validate the common LLM settings with one
|
||||
function taking a path prefix. Keep root-only constraints and readiness
|
||||
separate, and preserve recorded origins, explicit null, and zero values.
|
||||
[Configuration contract](../../config.md#settings)
|
||||
|
||||
## Additional correctness findings
|
||||
|
||||
### 11. A contradictory validator decision can advance an unavailable candidate
|
||||
|
||||
The internal `Decision` accepts `Approved: true` together with
|
||||
`DispositionUnavailable`. Cardinality checks validate only indices. With
|
||||
one such decision and no other rejection for that candidate, the runner
|
||||
treats it as approved in the shortlist path and leaves it unblocked in the
|
||||
independent-feedback path. This violates the invariant that an unavailable
|
||||
assessment cannot advance as approval. Bundled producers set matching
|
||||
values. The defect is at the internal validator result boundary. Make the
|
||||
disposition authoritative or reject contradictions before processing the
|
||||
decision. [Decision and cardinality](../../../internal/framework/validators/models.go#L56)
|
||||
· [Approval and blocking](../../../internal/framework/runner/runner.go#L769)
|
||||
· [Validator guarantees](../../internal/validators.md#llm-validator-batching)
|
||||
|
||||
### 12. The runner accepts an already canceled empty pipeline
|
||||
|
||||
With valid inputs and no module specifications, `Run` constructs the chunk
|
||||
inventory and returns success even when the parent context was canceled
|
||||
before entry. Its cancellation check sits inside the module loop. The
|
||||
processing CLI checks cancellation around the runner, so the gap is in the
|
||||
runner contract. Check the context at entry and at the run's successful
|
||||
completion boundary, including normalization-only invocations.
|
||||
[Runner entry](../../../internal/framework/runner/runner.go#L121)
|
||||
· [Runner success return](../../../internal/framework/runner/runner.go#L286)
|
||||
· [CLI cancellation check](../../../internal/cli/process.go#L65)
|
||||
· [Pipeline cancellation contract](../../internal/pipeline.md)
|
||||
|
||||
### 13. Missing failed-report rendering can erase initialized evidence
|
||||
|
||||
`preflight.Prepare` converts a nil `FailedReports` callback into a callable
|
||||
function returning a zero-value `ReportBundle`. If reporting is enabled and
|
||||
cancellation arrives after initialization, preflight passes that empty
|
||||
bundle to `Finish`. `Finalize` writes its three nil payloads as empty files,
|
||||
replacing the existing `in_progress` reports. The initialization retry can
|
||||
similarly replace reports that were written successfully in its first pass.
|
||||
The CLI supplies a failed renderer for its ordinary reporting path. At the
|
||||
internal preflight seam, carry the absence of a failed bundle through to
|
||||
publication or require a valid failed bundle before acquiring resources,
|
||||
so the existing evidence survives when a failed snapshot cannot be rendered.
|
||||
[Callback wrapping](../../../internal/framework/preflight/preflight.go#L68)
|
||||
· [Initialization write retry](../../../internal/core/diagnostics/output_session.go#L222)
|
||||
· [Final report write](../../../internal/core/diagnostics/output_session.go#L273)
|
||||
Reference in New Issue
Block a user