Files
audita/docs/roadmap/reliability-and-maintainability.md

30 KiB
Raw Blame History

Audita Reliability and Maintainability Roadmap

Date: 2026-11-15. Status: target state defined; implementation pending. Baseline: 730c1e4; application code is unchanged from the audited c4ccd78.

Purpose and scope

Audita should produce faithful transcripts and useful evidence, handle partial success and failures consistently, and use resources proportionately to the work and outputs selected. Its components should have clear owners, with less local machinery where standard Go facilities or the adopted libraries already supply the behavior. Operators requesting progress should be able to follow a run in detail through CommandKit.

This roadmap organizes the accepted audit remediation around that target state. The assessment owns the disposition of individual findings and the rationale for decisions D1–D7; the catalog links to their evidence. The coverage appendix below maps that evidence to this plan. Earlier source-audit remedies remain proposals where the assessment supersedes them. This document specifies outcomes and dependencies; a later implementation plan will choose reviewable changes, concrete APIs, and scheduling mechanics.

The current application already has native library integration, preflight, ordered module execution, partial-success validation, bounded correction repair, ordinary reports, and opt-in debug artifacts. This work improves those paths and adds processing progress. It does not redesign editorial policy or the public configuration schema. The D1 architecture-policy update is complete; its code cleanup and the other accepted remediation remain to be implemented.

Design boundaries

Follow the architecture policy and use the libraries' supported paths. ConfigKit owns composition, presence, provenance, and resolution before handoff. PromptKit owns credentials, model execution, admission, retries, and conversation coordination. CommandKit owns standard commands, reporting modes, and stream routing. ReleaseKit retains its release-validation ownership. Audita owns transcript meaning, role selection, domain validation and repair decisions, processing coordination, and presentation of its results.

The accepted decisions establish these boundaries for this remediation:

Area Selected scope
Diagnostic content Remove Audita's credential substitution and the discovery/plumbing supporting it; retain native library behavior and useful evidence.
Report destinations Keep existing placement; callers isolate concurrent jobs and retained run histories.
Admission Keep native waiting and recoverable rejection. Improve local scheduling without a second backend limiter, admission-retry loop, or eventual-admission promise.
Input and capture resources Remove waste; add no transcript-size cap, capture quota, automatic artifact deletion, or automatic stop-capture policy. Operators own retention and integrations own their upload/process budgets.
Correction repair Retain full conversation history and existing per-correction limits. Aggregate budgets and reduced context are not selected at this time.
Progress Supply relatively detailed processing activity through CommandKit when progress is requested, subject to its effective reporting mode.

Respect the three accepted Go limitations: standard HTTP redirect behavior, standard regexp word-boundary semantics, and the inability to interrupt an already-blocked ordinary file/stream call. Fix Audita's own choice of operation and its handling when control returns. Output correctness here does not promise hostile-filesystem race resistance, cross-file transactions, power-loss recovery, or atomic synchronization with cancellation. Keep ordinary symlink, in-place output, and stream support. Debug root locations and creation modes stay as they are for this cleanup.

The separate features in deferred work, including semantic recovery of unavailable validator decisions and independent whole-pipeline traversal by chunk, remain outside this roadmap. Internal refactoring should preserve current contracts except for the deliberate changes and correctness fixes described here; it need not preserve incidental implementation details or known defects.

1. Truthful output and useful diagnostic evidence

Target state

Required output writes use the destinations that were actually checked. Reports reflect known processing and delivery outcomes, retain available failure origin, and remain useful when finalization itself encounters an error. Optional debug capture has a separate lifecycle and does not determine processing success. Diagnostic representations preserve the information supplied to them through ordinary serialization.

The baseline resolves some output paths differently at checking and writing, can publish a completed report after learning of delivery failure, and can lose initialized evidence or staging-cleanup ownership. Credential substitution adds serialization passes and can alter schema keys, messages, and path announcements. The output contract and operations guide describe the existing output surfaces.

Work to reach the target

  • Give required-output preparation, staging, publication, and cleanup one clear owner. Resolve paths using filesystem semantics and carry checked destination information into publication. Share regular-file staging mechanics between transcript and reports, retaining private staging and intentional in-place replacement. Keep ownership of temporary paths until removal succeeds or absence is established; report cleanup failures alongside the primary error.
  • Keep transcript-before-report publication and nontransactional finalization. Select failed report status when a delivery failure is already known, carry report-rendering errors explicitly, and preserve initialized reports when a replacement failure bundle cannot be rendered. Retain bounded best-effort attempts to publish enabled reports; a later report failure does not erase an already-published transcript.
  • Check cancellation at natural publication boundaries and after stream delivery returns. A canceled run should report failure even if bytes have already escaped. Do not implement detached I/O workers to simulate interruption of standard blocking calls.
  • Build ordinary reports from processing-owned facts, preserving phase, available stage/chunk/operation origin, documented counts, and existing serialized identities. Distinguish the initiating failure, peer cancellation, and secondary output errors. Join correction evidence once per proposal before projecting ledger rows; carry the validator execution class actually used rather than reconstructing it from a name. Preserve disposition, repair lineage, empty replacement values, partial decisions, and module-only failures.
  • Complete the redaction-removal inventory: delete redaction-only credential enumeration, environment reads, name caches, role tracking, substitution utilities, and secret-only parameters/interfaces. Encode typed JSON/YAML directly. Save acquired source bytes without reconstructing or reopening the source; keep parsed and normalized views separately. Preserve ConfigKit rendering and native authentication/inspection.
  • Supply one interaction-writer contract from the adapter/composition boundary. Proposal, validation, and repair owners supply their own payloads and association coordinates. Remove lazy secret callbacks and cross-role secret lists. Retain self-contained interaction artifacts, malformed final response content as evidence, nonfatal write callbacks, and their synchronization. Consolidate equivalent write warnings with counts and affected-artifact information while continuing best-effort capture.
  • Keep Markdown escaping separate from substitution: input/output paths should render as literal text without changing their structured values. Document caller-owned report-directory isolation and artifact retention when updating the operator guide.

Completion evidence

Focused filesystem tests cover checked destinations, known delivery failure, failed report rendering, cleanup errors, and cancellation once I/O returns. Assertions inspect publication behavior as well as final files. Decoded reports retain their schema and types, and ledger evidence agrees with execution. Artifact tests establish source fidelity, role/attempt association, and nonfatal debug failures. Redaction-only code and assertions are removed without replacing them with no-op seams or losing native authentication coverage.

2. Bounded processing with clear operation ownership

Target state

Local preparation, dispatch, and completed-result handling avoid allocating a worker and prepared request for every chunk at once. Completed evidence is owned independently of ordered application. Each stage and model operation has a clear completion path, and typed outcomes reach reports without being reconstructed from display strings.

The baseline fans out work across all chunks, retains out-of-order results behind slow earlier work, and duplicates operation state across callbacks and projections. Backend admission already belongs to PromptKit; it does not bound all of this local preparation and retention. The pipeline remains ordered by stage, with fixed chunk membership and deterministic proposal/application identity.

Work to reach the target

  • Give stage execution a single result/commit boundary. Honor cancellation on entry, including an empty pipeline, and before committing new stage work. Cancel and join peers on fatal failure or parent cancellation, retaining earlier committed stage text and available completed evidence. Recoverable operation failures remain local under the existing partial-success rules.
  • Design bounded local preparation and dispatch together with concurrent draining of results. Preserve deterministic identities, validator requirements, ordered application, and the module barrier. Reduce avoidable within-stage waiting without introducing independent whole-pipeline chunk traversal. Retain completed proposal evidence on interruption even when an earlier section is unfinished. PromptKit's configured rejection remains valid; the old scheduling-dependent choice of rejected chunks is not a compatibility requirement.
  • Retain only the state needed by active processing and selected outputs. Keep live validation/repair state, compact summary counts, and required failure evidence. Avoid full debug histories or work-unit matrices in summary-only paths; preserve complete detail when that output is enabled. Remove duplicate operation arrays, unused module totals, and obsolete response/callback copies. This is not a constant-memory promise: selected outputs and useful conversation history can still grow with the workload.
  • Carry typed class, reason, and cause through proposal, validation, repair, and reporting. Preserve the original class/cause if a missing reason uses the internal-failure fallback. Distinguish parent cancellation, logical-operation timeout, and HTTP-attempt timeout using native classifications; an HTTP-only timeout uses the existing transport-failure reason.
  • Share ordinary-call and coordinator-turn completion, retaining native terminal counts and diagnostic evidence from undelivered responses. Remove the test-only direct-continuation branch after moving useful coverage onto production coordination. Keep one proposal handoff with raw local references; the runner owns global indexing. Review emitted metadata before deleting fields that only appear unused to a function-level reference search.
  • Give each repair branch an explicit execution/result boundary. Preserve native conversation coordination, full history, per-correction limits, sibling independence, withdrawal, and validation of the final permitted revision. Pass domain validator explanations directly instead of applying credential substitution; retain the distinction between editorial rejection and unavailable assessment. Measure aggregate work to guide efficiency work, without adding new stopping rules or ordinary-report fields for that analysis.

Completion evidence

Controlled overlap and cancellation tests exercise actual dispatch, admission, result draining, and stage commit paths. They establish native shared backend capacity, useful evidence retention, and deterministic identities under observed out-of-order completion. Distinct live timeout budgets exercise the correct failure owner. Representative measurements demonstrate reduced pending work and retained representations; they do not impose a new throughput target or require every chunk to obtain admission. Repair tests run through the production coordinator and preserve accepted feedback and revalidation behavior.

3. Faithful transcript and correction semantics

Target state

Transcript parsing preserves accepted values, normalization preserves time and ordering meaning, and chunking satisfies feasible configured constraints. Validation and application use consistent domain evidence. Efficient shared operations reduce repeated work without changing editorial acceptance rules.

The baseline's generic JSON shape conversion can round IDs; normalization and chunking have concrete interval, ordering, and feasibility counterexamples. Some protected-term checks use only edit spans, and validation response conversion can discard usable decisions because of irrelevant extra references.

Work to reach the target

  • Detect the supported transcript envelopes while keeping their segment array as raw JSON for typed decoding. Preserve optional IDs, accepted numeric ranges, unknown-field behavior, and both input shapes. Share live source-to-canonical conversion, retire obsolete strict/lenient wrappers, and release whole-input representations after their last use. Keep accepted formats and stream inputs.
  • Use the full merged time interval for merge eligibility, output, and subsequent gaps. Preserve source order for tied timestamps without introducing a new end-time sorting criterion. Build growing text/category state incrementally, preserving exact joins, category order, and the current token estimate.
  • Satisfy feasible explicit chunk count and maximum-size constraints before optimizing balance. Choose a proportionate deterministic algorithm and retain the configured budgets and current balancing objective. Keep accepted compatibility settings, including chunking.min_section_tokens, even where they are not active partition targets.
  • Give validator applicability and decision meaning one owner and one runtime contract. Reject contradictory internal outcomes before they advance a candidate; treat them as internal contract failures, not unavailable model assessments. Derive redundant approval representation where needed. Preserve decision cardinality, zero-budget shortlisting, independent repair feedback, and unavailable outcomes. Reconcile exact numeric response references before local conversion, accepting integral JSON number spellings without float rounding and ignoring irrelevant extras. Keep whole-response structural validation and existing partial-success behavior.
  • Share glossary-owned protection assessment using usable before/after segment evidence, and recheck current text at application. Preserve active-glossary exemptions, case/longest-match behavior, and spoken-word repetition rules. Missing or ambiguous previews cannot support a fabricated protection verdict; independently supported applicability findings remain valid. Empty-result previews can still supply protection evidence without authorizing whole-segment removal. Handle configured punctuation-edged spellings with standard primitives while accepting Go's existing word-boundary semantics.
  • For require-unique replacement, detect a second possible literal starting offset even when matches overlap. Preserve replace-all's existing nonoverlapping behavior. Share request-local segment lookup across initial and repaired proposals, retaining distinct reference namespaces, raw invalid references, and retargeted revalidation.
  • Reuse glossary identity membership for context selection rather than expanding every occurrence/contributor pair; keep occurrence evidence for protection. Avoid validation-context work once the usable budget is settled, sharing trimming/batching policy while preserving context priority, oversized-input handling, and an independent serialized measurement reference. Reuse section lookups and aggregate skipped-edit evidence once, distinguishing stale application conflicts from response filtering.

Completion evidence

Boundary regressions reproduce the audited numeric, nested-interval, tied-order, feasible-chunk, overlapping-match, and full-segment-protection cases. Validator tests pass actual structured responses through decoding and reconciliation, with usable siblings and irrelevant extra references. Shared optimized operations are checked against independently constructed expectations, including exact estimates and representative allocation/work measurements. Existing editorial acceptance, repair, and apply-time rules continue to govern results.

4. Smaller components with explicit ownership

Target state

Preflight and execution share resolved application decisions. Domain models own their conversions and copies; adapter boundaries own external effects; the composition boundary assembles built-ins. Modules and validators remain distinct components with documented concurrent-call and borrowed-data responsibilities.

The baseline repeats root/stage configuration rules, readiness selections, transcript projections/clones, token estimates, and shared prompt-variable work. Some interfaces and helpers survive only through tests or unused wrappers.

Work to reach the target

  • Consolidate Audita-owned root/stage validation and fallback rules after ConfigKit handoff, retaining null, explicit zero, inheritance, provenance, stage-dependent defaults, and diagnostic paths. Carry checked processing selections from preflight into execution. Reuse effective profile/glossary views within the captured runtime, with selection order and exemptions in reuse keys; retain first-failure attribution.
  • Share transcript cloning at the model boundary, preserving nil versus explicit empty mutable slices and the separate caller ownership boundaries. Give public encoders one segment projection while retaining both envelopes, schema/version behavior, categories, omission rules, and numeric fidelity. Use one implementation of the existing token estimate across consumers, retaining its exact results.
  • Assemble built-in modules at composition, preserving the runner factory seam and a dependency-light configuration catalog. Document concurrent-call and borrowed-input obligations at module/validator ports; permit synchronized stateful implementations and use owned copies where mutation is needed.
  • Merge shared prompt variables once per model call, preserving the different processing/inspection precedence, protected-term default, and operator prompt replacement support. Remove dead helpers, unused envelopes, redundant adapters, and abandoned test support after checking live registrations and serialization. Move useful tests to the surviving behavior before removing test-only paths.

These changes complement the writer, validator, operation, and repair ownership work above. Use small contracts and helpers where they serve an actual shared responsibility; package layout and private type names are implementation choices.

Completion evidence

Real consumers use the shared owners and obsolete paths are removed. Behavioral tests establish copy independence, both output envelopes, effective configuration semantics, native integration, and processing/inspection precedence. Refactoring does not depend on a new configuration loader, plugin framework, cross-invocation cache, or a generic replacement for the existing domain components.

5. Actionable errors and detailed processing progress

Target state

Operators can identify the failed phase and relevant setup requirement, and follow long jobs with meaningful progress. Standard inspection and processing remain integrated with CommandKit rather than acquiring parallel command or reporting systems.

The baseline hides some native setup detail, classifies some operator-correctable failures as runtime failures, and exposes progress without producing processing events. The current CLI contract supplies the stream and exit-status boundary that the fixes should follow.

Work to reach the target

  • Classify invalid processing selections and genuinely infeasible chunk requests according to the existing input-error contract; retain runtime classification for I/O and cancellation. Trace delegated inspection failures to Audita's loader or the upstream integration, then fix the responsible owner. An upstream defect should be addressed upstream rather than through replacement command routing.
  • Surface actionable native setup detail, including the configured missing credential requirement and available role/profile context, while preserving typed causes and failure classification. Remove CLI substitution of errors, warnings, and announced paths. Keep compact ordinary-report projections where they serve the report schema; deleting substitution is not a request to expand every report with raw provider causes.
  • Emit relatively detailed stage/chunk, proposal, validation, and repair activity through CommandKit's reporter, using actual lifecycle state and available upstream execution events. Include useful work identities and observed completion counts. Do not restrict --progress to coarse stage summaries just to minimize output. CommandKit determines effective reporting-mode precedence and routes output, including when flags are combined; stdout remains available for the transcript. Event granularity and wording are implementation choices.
  • Document the trusted configuration bundle, imports, and automatic companions for operators and subprocess integrations. Explain how caller-owned report destinations and artifact retention fit the workflow. Preserve native discovery and inspection; these clarifications do not add an isolation mode.

Completion evidence

CLI tests check the appropriate exit classifications, useful diagnostic context, native mode combinations, and unchanged transcript stdout. Progress tests exercise real processing activity and meaningful updates without fixing incidental wording or concurrent event order. Update user-facing documentation alongside the feature.

6. Verification and delivery that support change

Target state and work

Tests protect consequential behavior at its owning boundary and make valid refactoring and prompt iteration inexpensive. Apply the testing policy throughout the work, rather than adding a second set of implementation-shaped assertions at the end.

  • Exercise representative signal, cancellation, and exit behavior through the real executable or a shared production entry point. Keep smaller CLI fixtures for cases that do not need a process boundary. Use observed overlap and independently controlled deadlines for concurrency/timeout tests.
  • Verify registered identities, effective validator roles, embedded-asset preparability, schema identity, and protective copies through behavior. Replace incidental registry counts, duplicate normalization fixtures, and tests of test helpers with coverage of their actual consumer risks. Assert decoded artifact identities and association; keep a representative repair/filter/later-failure integration scenario and smaller independent-output cases.
  • Remove redaction-only tests and fixture modes while retaining authentication, serialization, evidence, and failure coverage. Avoid prompt-wording change detectors, broad incidental snapshots, and new checks that recreate withdrawn guarantees. The default suite remains offline and uses synthetic credentials.
  • Fix the CI bootstrap so failure to run its Git cleanliness check cannot be interpreted as a clean checkout. Test that command-status boundary and remove redundant tests of the release test's comparison helper. Keep ReleaseKit's selected checks and the existing release policy; this roadmap introduces no new release gates.
  • Update canonical CLI, configuration, operations, integration, and internal documentation with the corresponding changes. Remove stale redaction promises and recipes without adding a negative architecture assertion. Keep native configuration-rendering documentation and historical audit evidence. Use ADRs only where the documentation policy calls for a significant architectural record, not for every helper or audit entry.

Completion evidence

Run focused regression and boundary tests with each change, the full offline suite for substantial integration changes, and applicable existing candidate checks at release preparation. Use targeted race checks for changed shared-state paths and representative measurements for resource changes. Reuse successful checks for unchanged content. Link/example/whitespace checks accompany relevant documentation changes. Do not substitute coverage percentages or a collection of test counts for evidence of the selected outcomes.

Sequencing and implementation planning

Stage Reviewable outcome Dependencies and rationale
A. Simplify evidence and publication Direct serialization and credential-plumbing removal; clear required-output/debug ownership; destination, failed-report, and cleanup fixes. Removes obsolete constraints before building shared writers. Preserve useful tests while deleting their old mechanisms.
B. Establish domain correctness and shared owners Parsing, normalization, chunking, protection, decision reconciliation, and shared model/configuration helpers. Pure domain work can proceed alongside A. Establish authoritative decisions and data shapes before changing orchestration around them.
C. Consolidate operation and stage lifecycle One proposal handoff, typed failure facts, explicit stage completion and repair branches, checked-plan reuse, faithful report/ledger projection. Uses the shared domain and diagnostic boundaries; moves useful coverage off obsolete execution paths.
D. Bound local work and retain only needed state Bounded preparation/dispatch, active result draining, output-sensitive retention, and measured efficiency improvements. Build on C's explicit ownership so narrowing queues does not deadlock producers or discard evidence. Optimize independent domain scans as their owners are established.
E. Complete operator feedback and verification Detailed native progress, actionable setup/input errors, canonical docs, streamlined integration tests, and release-bootstrap correction. Progress consumes the real lifecycle from C/D. Error ownership investigation and the independent bootstrap fix can start earlier. Tests and docs accompany every stage; this stage closes integration gaps.

This is a dependency order, not a requirement for five large commits. The later implementation plan should split coherent fixes and supporting refactors into reviewable units. Significant refactoring is appropriate when it improves the selected ownership or resource behavior; an audit number is not a commit boundary.

Resolve these engineering details during planning/implementation:

  • Select and measure the local work/result window, including nested validation and repair interactions. No exact worker count or new operator setting is prescribed by this roadmap.
  • Identify the owner of delegated inspection classification and verify the available CommandKit reporting hooks. If an upstream change is required, record its concrete dependency and adopt the supporting release before that integration; independent work can continue.
  • Measure aggregate repair and debug costs using existing observations or focused instrumentation. Improve local representations while preserving full history, configured repair limits, and requested self-contained artifacts.

These are technical investigations within resolved scope. No significant product decision remains open. A concrete discovery requiring a substantive policy change should be brought back for discussion rather than silently expanding this roadmap.

Completion and audit coverage

The roadmap is complete when the target behaviors above are implemented and verified, the obsolete mechanisms are removed, and canonical documentation describes the resulting application. The coverage map accounts for every catalog entry, including closed and superseded remedies; it does not authorize those old remedies. Record implementation evidence against the target sections as work lands.

Target area or disposition Catalog entries
§1: checked output, truthful finalization, caller destinations 1, 6, 7, 8, 16, 42
§1: serialization, diagnostic ownership, and useful evidence 3, 9, 22, 26, 27, 33, 43, 63
§2: stage/operation lifecycle, scheduling, and retention 14, 15, 23, 30, 32, 34, 41, 44, 45, 46, 48, 64
§3: transcript fidelity, normalization, and chunking 13, 17, 20, 21, 31, 35, 51
§3: decision semantics, protection, references, and context 10, 11, 12, 18 (punctuation portion), 19, 36, 38, 39, 40, 47
§4: shared selections, models, configuration, and composition 37, 49, 50, 52, 53, 54, 55, 56
§5: operator guidance, errors, and progress 5, 24, 25, 57, 65
§6: verification, delivery, and dead machinery 28, 29, 58, 59, 60, 61, 62
Accepted Go limitations; no workaround 2; 6 (interrupting blocked I/O); 18 (Unicode word-boundary emulation)
Withdrawn private-debug-default remedy; current defaults retained 4

Cross-cutting test and documentation work belongs with each implementing change. The assessment remains the detailed boundary for mixed entries. All seven findings in the focused removal audit are covered by the evidence, repair, CLI, and verification sections above.