diff --git a/docs/roadmap/reliability-and-maintainability.md b/docs/roadmap/reliability-and-maintainability.md new file mode 100644 index 0000000..7d177ee --- /dev/null +++ b/docs/roadmap/reliability-and-maintainability.md @@ -0,0 +1,471 @@ +# Audita Reliability and Maintainability Roadmap + +**Date:** 2026-11-15. **Status:** target state defined; implementation pending. +**Baseline:** `730c1e4`; application code is unchanged from the audited `c4ccd78`. + +## Purpose and scope + +Audita should produce faithful transcripts and useful evidence, handle partial +success and failures consistently, and use resources proportionately to the work +and outputs selected. Its components should have clear owners, with less local +machinery where standard Go facilities or the adopted libraries already supply +the behavior. Operators requesting progress should be able to follow a run in +detail through CommandKit. + +This roadmap organizes the accepted audit remediation around that target state. +The [assessment](audit/assessment.md) owns the disposition of individual findings +and the rationale for decisions D1–D7; the [catalog](audit/catalog.md) links to +their evidence. The coverage appendix below maps that evidence to this plan. +Earlier source-audit remedies remain proposals where the assessment supersedes +them. This document specifies outcomes and dependencies; a later implementation +plan will choose reviewable changes, concrete APIs, and scheduling mechanics. + +The current application already has native library integration, preflight, +ordered module execution, partial-success validation, bounded correction repair, +ordinary reports, and opt-in debug artifacts. This work improves those paths and +adds processing progress. It does not redesign editorial policy or the public +configuration schema. The D1 architecture-policy update is complete; its code +cleanup and the other accepted remediation remain to be implemented. + +### Design boundaries + +Follow the [architecture policy](../policy/architecture.md) and use the libraries' +supported paths. ConfigKit owns composition, presence, provenance, and resolution +before handoff. PromptKit owns credentials, model execution, admission, retries, +and conversation coordination. CommandKit owns standard commands, reporting +modes, and stream routing. ReleaseKit retains its release-validation ownership. +Audita owns transcript meaning, role selection, domain validation and repair +decisions, processing coordination, and presentation of its results. + +The accepted decisions establish these boundaries for this remediation: + +| Area | Selected scope | +| --- | --- | +| Diagnostic content | Remove Audita's credential substitution and the discovery/plumbing supporting it; retain native library behavior and useful evidence. | +| Report destinations | Keep existing placement; callers isolate concurrent jobs and retained run histories. | +| Admission | Keep native waiting and recoverable rejection. Improve local scheduling without a second backend limiter, admission-retry loop, or eventual-admission promise. | +| Input and capture resources | Remove waste; add no transcript-size cap, capture quota, automatic artifact deletion, or automatic stop-capture policy. Operators own retention and integrations own their upload/process budgets. | +| Correction repair | Retain full conversation history and existing per-correction limits. Aggregate budgets and reduced context are not selected at this time. | +| Progress | Supply relatively detailed processing activity through CommandKit when progress is requested, subject to its effective reporting mode. | + +Respect the three accepted Go limitations: standard HTTP redirect behavior, +standard regexp word-boundary semantics, and the inability to interrupt an +already-blocked ordinary file/stream call. Fix Audita's own choice of operation +and its handling when control returns. Output correctness here does not promise +hostile-filesystem race resistance, cross-file transactions, power-loss recovery, +or atomic synchronization with cancellation. Keep ordinary symlink, in-place +output, and stream support. Debug root locations and creation modes stay as they +are for this cleanup. + +The separate features in [deferred work](future.md), including semantic recovery +of unavailable validator decisions and independent whole-pipeline traversal by +chunk, remain outside this roadmap. Internal refactoring should preserve current +contracts except for the deliberate changes and correctness fixes described here; +it need not preserve incidental implementation details or known defects. + +## 1. Truthful output and useful diagnostic evidence + +### Target state + +Required output writes use the destinations that were actually checked. Reports +reflect known processing and delivery outcomes, retain available failure origin, +and remain useful when finalization itself encounters an error. Optional debug +capture has a separate lifecycle and does not determine processing success. +Diagnostic representations preserve the information supplied to them through +ordinary serialization. + +The baseline resolves some output paths differently at checking and writing, +can publish a completed report after learning of delivery failure, and can lose +initialized evidence or staging-cleanup ownership. Credential substitution adds +serialization passes and can alter schema keys, messages, and path announcements. +The [output contract](../integrations/output-files.md) and +[operations guide](../operations.md) describe the existing output surfaces. + +### Work to reach the target + +- Give required-output preparation, staging, publication, and cleanup one clear + owner. Resolve paths using filesystem semantics and carry checked destination + information into publication. Share regular-file staging mechanics between + transcript and reports, retaining private staging and intentional in-place + replacement. Keep ownership of temporary paths until removal succeeds or + absence is established; report cleanup failures alongside the primary error. +- Keep transcript-before-report publication and nontransactional finalization. + Select failed report status when a delivery failure is already known, carry + report-rendering errors explicitly, and preserve initialized reports when a + replacement failure bundle cannot be rendered. Retain bounded best-effort + attempts to publish enabled reports; a later report failure does not erase an + already-published transcript. +- Check cancellation at natural publication boundaries and after stream delivery + returns. A canceled run should report failure even if bytes have already + escaped. Do not implement detached I/O workers to simulate interruption of + standard blocking calls. +- Build ordinary reports from processing-owned facts, preserving phase, available + stage/chunk/operation origin, documented counts, and existing serialized + identities. Distinguish the initiating failure, peer cancellation, and secondary + output errors. Join correction evidence once per proposal before projecting + ledger rows; carry the validator execution class actually used rather than + reconstructing it from a name. Preserve disposition, repair lineage, empty + replacement values, partial decisions, and module-only failures. +- Complete the [redaction-removal inventory](audit/14-redaction.md): delete + redaction-only credential enumeration, environment reads, name caches, role + tracking, substitution utilities, and secret-only parameters/interfaces. Encode typed + JSON/YAML directly. Save acquired source bytes without reconstructing or + reopening the source; keep parsed and normalized views separately. Preserve + ConfigKit rendering and native authentication/inspection. +- Supply one interaction-writer contract from the adapter/composition boundary. + Proposal, validation, and repair owners supply their own payloads and + association coordinates. Remove lazy secret callbacks and cross-role secret + lists. Retain self-contained interaction artifacts, malformed final response + content as evidence, nonfatal write callbacks, and their synchronization. + Consolidate equivalent write warnings with counts and affected-artifact + information while continuing best-effort capture. +- Keep Markdown escaping separate from substitution: input/output paths should + render as literal text without changing their structured values. Document + caller-owned report-directory isolation and artifact retention when updating + the operator guide. + +### Completion evidence + +Focused filesystem tests cover checked destinations, known delivery failure, +failed report rendering, cleanup errors, and cancellation once I/O returns. +Assertions inspect publication behavior as well as final files. Decoded reports +retain their schema and types, and ledger evidence agrees with execution. +Artifact tests establish source fidelity, role/attempt association, and nonfatal +debug failures. Redaction-only code and assertions are removed without replacing +them with no-op seams or losing native authentication coverage. + +## 2. Bounded processing with clear operation ownership + +### Target state + +Local preparation, dispatch, and completed-result handling avoid allocating a +worker and prepared request for every chunk at once. Completed evidence is owned +independently of ordered application. Each stage and model operation has a clear +completion path, and typed outcomes reach reports without being reconstructed +from display strings. + +The baseline fans out work across all chunks, retains out-of-order results behind +slow earlier work, and duplicates operation state across callbacks and projections. +Backend admission already belongs to PromptKit; it does not bound all of this +local preparation and retention. The [pipeline](../internal/pipeline.md) remains +ordered by stage, with fixed chunk membership and deterministic proposal/application +identity. + +### Work to reach the target + +- Give stage execution a single result/commit boundary. Honor cancellation on + entry, including an empty pipeline, and before committing new stage work. + Cancel and join peers on fatal failure or parent cancellation, retaining earlier + committed stage text and available completed evidence. Recoverable operation + failures remain local under the existing partial-success rules. +- Design bounded local preparation and dispatch together with concurrent draining + of results. Preserve deterministic identities, validator requirements, ordered + application, and the module barrier. Reduce avoidable within-stage waiting + without introducing independent whole-pipeline chunk traversal. Retain completed + proposal evidence on interruption even when an earlier section is unfinished. + PromptKit's configured rejection remains valid; the old scheduling-dependent + choice of rejected chunks is not a compatibility requirement. +- Retain only the state needed by active processing and selected outputs. Keep + live validation/repair state, compact summary counts, and required failure + evidence. Avoid full debug histories or work-unit matrices in summary-only + paths; preserve complete detail when that output is enabled. Remove duplicate + operation arrays, unused module totals, and obsolete response/callback copies. + This is not a constant-memory promise: selected outputs and useful conversation + history can still grow with the workload. +- Carry typed class, reason, and cause through proposal, validation, repair, and + reporting. Preserve the original class/cause if a missing reason uses the + internal-failure fallback. Distinguish parent cancellation, logical-operation + timeout, and HTTP-attempt timeout using native classifications; an HTTP-only + timeout uses the existing transport-failure reason. +- Share ordinary-call and coordinator-turn completion, retaining native terminal + counts and diagnostic evidence from undelivered responses. Remove the test-only + direct-continuation branch after moving useful coverage onto production + coordination. Keep one proposal handoff with raw local references; the runner + owns global indexing. Review emitted metadata before deleting fields that only + appear unused to a function-level reference search. +- Give each repair branch an explicit execution/result boundary. Preserve native + conversation coordination, full history, per-correction limits, sibling + independence, withdrawal, and validation of the final permitted revision. + Pass domain validator explanations directly instead of applying credential + substitution; retain the distinction between editorial rejection and unavailable + assessment. Measure aggregate work to guide efficiency work, without adding + new stopping rules or ordinary-report fields for that analysis. + +### Completion evidence + +Controlled overlap and cancellation tests exercise actual dispatch, admission, +result draining, and stage commit paths. They establish native shared backend +capacity, useful evidence retention, and deterministic identities under observed +out-of-order completion. Distinct live timeout budgets exercise the correct +failure owner. Representative measurements demonstrate reduced pending work and +retained representations; they do not impose a new throughput target or require +every chunk to obtain admission. Repair tests run through the production +coordinator and preserve accepted feedback and revalidation behavior. + +## 3. Faithful transcript and correction semantics + +### Target state + +Transcript parsing preserves accepted values, normalization preserves time and +ordering meaning, and chunking satisfies feasible configured constraints. +Validation and application use consistent domain evidence. Efficient shared +operations reduce repeated work without changing editorial acceptance rules. + +The baseline's generic JSON shape conversion can round IDs; normalization and +chunking have concrete interval, ordering, and feasibility counterexamples. +Some protected-term checks use only edit spans, and validation response conversion +can discard usable decisions because of irrelevant extra references. + +### Work to reach the target + +- Detect the supported transcript envelopes while keeping their segment array as + raw JSON for typed decoding. Preserve optional IDs, accepted numeric ranges, + unknown-field behavior, and both input shapes. Share live source-to-canonical + conversion, retire obsolete strict/lenient wrappers, and release whole-input + representations after their last use. Keep accepted formats and stream inputs. +- Use the full merged time interval for merge eligibility, output, and subsequent + gaps. Preserve source order for tied timestamps without introducing a new + end-time sorting criterion. Build growing text/category state incrementally, + preserving exact joins, category order, and the current token estimate. +- Satisfy feasible explicit chunk count and maximum-size constraints before + optimizing balance. Choose a proportionate deterministic algorithm and retain + the configured budgets and current balancing objective. Keep accepted + compatibility settings, including `chunking.min_section_tokens`, even where + they are not active partition targets. +- Give validator applicability and decision meaning one owner and one runtime + contract. Reject contradictory internal outcomes before they advance a + candidate; treat them as internal contract failures, not unavailable model + assessments. Derive redundant approval representation where needed. Preserve + decision cardinality, zero-budget shortlisting, independent repair feedback, + and unavailable outcomes. Reconcile exact numeric response references before + local conversion, accepting integral JSON number spellings without float + rounding and ignoring irrelevant extras. Keep whole-response structural + validation and existing partial-success behavior. +- Share glossary-owned protection assessment using usable before/after segment + evidence, and recheck current text at application. Preserve active-glossary + exemptions, case/longest-match behavior, and spoken-word repetition rules. + Missing or ambiguous previews cannot support a fabricated protection verdict; + independently supported applicability findings remain valid. Empty-result + previews can still supply protection evidence without authorizing whole-segment + removal. Handle configured punctuation-edged spellings with standard primitives + while accepting Go's existing word-boundary semantics. +- For require-unique replacement, detect a second possible literal starting + offset even when matches overlap. Preserve replace-all's existing nonoverlapping + behavior. Share request-local segment lookup across initial and repaired + proposals, retaining distinct reference namespaces, raw invalid references, + and retargeted revalidation. +- Reuse glossary identity membership for context selection rather than expanding + every occurrence/contributor pair; keep occurrence evidence for protection. + Avoid validation-context work once the usable budget is settled, sharing + trimming/batching policy while preserving context priority, oversized-input + handling, and an independent serialized measurement reference. Reuse section + lookups and aggregate skipped-edit evidence once, distinguishing stale + application conflicts from response filtering. + +### Completion evidence + +Boundary regressions reproduce the audited numeric, nested-interval, tied-order, +feasible-chunk, overlapping-match, and full-segment-protection cases. Validator +tests pass actual structured responses through decoding and reconciliation, with +usable siblings and irrelevant extra references. Shared optimized operations +are checked against independently constructed expectations, including exact +estimates and representative allocation/work measurements. Existing editorial +acceptance, repair, and apply-time rules continue to govern results. + +## 4. Smaller components with explicit ownership + +### Target state + +Preflight and execution share resolved application decisions. Domain models own +their conversions and copies; adapter boundaries own external effects; the +composition boundary assembles built-ins. Modules and validators remain distinct +components with documented concurrent-call and borrowed-data responsibilities. + +The baseline repeats root/stage configuration rules, readiness selections, +transcript projections/clones, token estimates, and shared prompt-variable work. +Some interfaces and helpers survive only through tests or unused wrappers. + +### Work to reach the target + +- Consolidate Audita-owned root/stage validation and fallback rules after + ConfigKit handoff, retaining null, explicit zero, inheritance, provenance, + stage-dependent defaults, and diagnostic paths. Carry checked processing + selections from preflight into execution. Reuse effective profile/glossary + views within the captured runtime, with selection order and exemptions in + reuse keys; retain first-failure attribution. +- Share transcript cloning at the model boundary, preserving nil versus explicit + empty mutable slices and the separate caller ownership boundaries. Give public + encoders one segment projection while retaining both envelopes, schema/version + behavior, categories, omission rules, and numeric fidelity. Use one implementation + of the existing token estimate across consumers, retaining its exact results. +- Assemble built-in modules at composition, preserving the runner factory seam + and a dependency-light configuration catalog. Document concurrent-call and + borrowed-input obligations at module/validator ports; permit synchronized + stateful implementations and use owned copies where mutation is needed. +- Merge shared prompt variables once per model call, preserving the different + processing/inspection precedence, protected-term default, and operator prompt + replacement support. Remove dead helpers, unused envelopes, redundant adapters, + and abandoned test support after checking live registrations and serialization. + Move useful tests to the surviving behavior before removing test-only paths. + +These changes complement the writer, validator, operation, and repair ownership +work above. Use small contracts and helpers where they serve an actual shared +responsibility; package layout and private type names are implementation choices. + +### Completion evidence + +Real consumers use the shared owners and obsolete paths are removed. Behavioral +tests establish copy independence, both output envelopes, effective configuration +semantics, native integration, and processing/inspection precedence. Refactoring +does not depend on a new configuration loader, plugin framework, cross-invocation +cache, or a generic replacement for the existing domain components. + +## 5. Actionable errors and detailed processing progress + +### Target state + +Operators can identify the failed phase and relevant setup requirement, and +follow long jobs with meaningful progress. Standard inspection and processing +remain integrated with CommandKit rather than acquiring parallel command or +reporting systems. + +The baseline hides some native setup detail, classifies some operator-correctable +failures as runtime failures, and exposes progress without producing processing +events. The current [CLI contract](../cli.md) supplies the stream and exit-status +boundary that the fixes should follow. + +### Work to reach the target + +- Classify invalid processing selections and genuinely infeasible chunk requests + according to the existing input-error contract; retain runtime classification + for I/O and cancellation. Trace delegated inspection failures to Audita's + loader or the upstream integration, then fix the responsible owner. An upstream + defect should be addressed upstream rather than through replacement command + routing. +- Surface actionable native setup detail, including the configured missing + credential requirement and available role/profile context, while preserving + typed causes and failure classification. Remove CLI substitution of errors, + warnings, and announced paths. Keep compact ordinary-report projections where + they serve the report schema; deleting substitution is not a request to expand + every report with raw provider causes. +- Emit relatively detailed stage/chunk, proposal, validation, and repair activity + through CommandKit's reporter, using actual lifecycle state and available + upstream execution events. Include useful work identities and observed + completion counts. Do not restrict `--progress` to coarse stage summaries just + to minimize output. CommandKit determines effective reporting-mode precedence + and routes output, including when flags are combined; stdout remains available + for the transcript. Event granularity and wording are implementation choices. +- Document the trusted configuration bundle, imports, and automatic companions + for operators and subprocess integrations. Explain how caller-owned report + destinations and artifact retention fit the workflow. Preserve native discovery + and inspection; these clarifications do not add an isolation mode. + +### Completion evidence + +CLI tests check the appropriate exit classifications, useful diagnostic context, +native mode combinations, and unchanged transcript stdout. Progress tests exercise +real processing activity and meaningful updates without fixing incidental wording +or concurrent event order. Update user-facing documentation alongside the feature. + +## 6. Verification and delivery that support change + +### Target state and work + +Tests protect consequential behavior at its owning boundary and make valid +refactoring and prompt iteration inexpensive. Apply the +[testing policy](../policy/testing.md) throughout the work, rather than adding a +second set of implementation-shaped assertions at the end. + +- Exercise representative signal, cancellation, and exit behavior through the + real executable or a shared production entry point. Keep smaller CLI fixtures + for cases that do not need a process boundary. Use observed overlap and + independently controlled deadlines for concurrency/timeout tests. +- Verify registered identities, effective validator roles, embedded-asset + preparability, schema identity, and protective copies through behavior. Replace + incidental registry counts, duplicate normalization fixtures, and tests of test + helpers with coverage of their actual consumer risks. Assert decoded artifact + identities and association; keep a representative repair/filter/later-failure + integration scenario and smaller independent-output cases. +- Remove redaction-only tests and fixture modes while retaining authentication, + serialization, evidence, and failure coverage. Avoid prompt-wording change + detectors, broad incidental snapshots, and new checks that recreate withdrawn + guarantees. The default suite remains offline and uses synthetic credentials. +- Fix the CI bootstrap so failure to run its Git cleanliness check cannot be + interpreted as a clean checkout. Test that command-status boundary and remove + redundant tests of the release test's comparison helper. Keep ReleaseKit's + selected checks and the existing [release policy](../policy/release.md); this + roadmap introduces no new release gates. +- Update canonical CLI, configuration, operations, integration, and internal + documentation with the corresponding changes. Remove stale redaction promises + and recipes without adding a negative architecture assertion. Keep native + configuration-rendering documentation and historical audit evidence. Use ADRs + only where the [documentation policy](../policy/documentation.md) calls for a + significant architectural record, not for every helper or audit entry. + +### Completion evidence + +Run focused regression and boundary tests with each change, the full offline +suite for substantial integration changes, and applicable existing candidate +checks at release preparation. Use targeted race checks for changed shared-state +paths and representative measurements for resource changes. Reuse successful +checks for unchanged content. Link/example/whitespace checks accompany relevant +documentation changes. Do not substitute coverage percentages or a collection +of test counts for evidence of the selected outcomes. + +## Sequencing and implementation planning + +| Stage | Reviewable outcome | Dependencies and rationale | +| --- | --- | --- | +| A. Simplify evidence and publication | Direct serialization and credential-plumbing removal; clear required-output/debug ownership; destination, failed-report, and cleanup fixes. | Removes obsolete constraints before building shared writers. Preserve useful tests while deleting their old mechanisms. | +| B. Establish domain correctness and shared owners | Parsing, normalization, chunking, protection, decision reconciliation, and shared model/configuration helpers. | Pure domain work can proceed alongside A. Establish authoritative decisions and data shapes before changing orchestration around them. | +| C. Consolidate operation and stage lifecycle | One proposal handoff, typed failure facts, explicit stage completion and repair branches, checked-plan reuse, faithful report/ledger projection. | Uses the shared domain and diagnostic boundaries; moves useful coverage off obsolete execution paths. | +| D. Bound local work and retain only needed state | Bounded preparation/dispatch, active result draining, output-sensitive retention, and measured efficiency improvements. | Build on C's explicit ownership so narrowing queues does not deadlock producers or discard evidence. Optimize independent domain scans as their owners are established. | +| E. Complete operator feedback and verification | Detailed native progress, actionable setup/input errors, canonical docs, streamlined integration tests, and release-bootstrap correction. | Progress consumes the real lifecycle from C/D. Error ownership investigation and the independent bootstrap fix can start earlier. Tests and docs accompany every stage; this stage closes integration gaps. | + +This is a dependency order, not a requirement for five large commits. The later +implementation plan should split coherent fixes and supporting refactors into +reviewable units. Significant refactoring is appropriate when it improves the +selected ownership or resource behavior; an audit number is not a commit boundary. + +Resolve these engineering details during planning/implementation: + +- Select and measure the local work/result window, including nested validation + and repair interactions. No exact worker count or new operator setting is + prescribed by this roadmap. +- Identify the owner of delegated inspection classification and verify the + available CommandKit reporting hooks. If an upstream change is required, record + its concrete dependency and adopt the supporting release before that integration; + independent work can continue. +- Measure aggregate repair and debug costs using existing observations or focused + instrumentation. Improve local representations while preserving full history, + configured repair limits, and requested self-contained artifacts. + +These are technical investigations within resolved scope. No significant product +decision remains open. A concrete discovery requiring a substantive policy change +should be brought back for discussion rather than silently expanding this roadmap. + +## Completion and audit coverage + +The roadmap is complete when the target behaviors above are implemented and +verified, the obsolete mechanisms are removed, and canonical documentation +describes the resulting application. The coverage map accounts for every catalog +entry, including closed and superseded remedies; it does not authorize those old +remedies. Record implementation evidence against the target sections as work lands. + +| Target area or disposition | Catalog entries | +| --- | --- | +| §1: checked output, truthful finalization, caller destinations | 1, 6, 7, 8, 16, 42 | +| §1: serialization, diagnostic ownership, and useful evidence | 3, 9, 22, 26, 27, 33, 43, 63 | +| §2: stage/operation lifecycle, scheduling, and retention | 14, 15, 23, 30, 32, 34, 41, 44, 45, 46, 48, 64 | +| §3: transcript fidelity, normalization, and chunking | 13, 17, 20, 21, 31, 35, 51 | +| §3: decision semantics, protection, references, and context | 10, 11, 12, 18 (punctuation portion), 19, 36, 38, 39, 40, 47 | +| §4: shared selections, models, configuration, and composition | 37, 49, 50, 52, 53, 54, 55, 56 | +| §5: operator guidance, errors, and progress | 5, 24, 25, 57, 65 | +| §6: verification, delivery, and dead machinery | 28, 29, 58, 59, 60, 61, 62 | +| Accepted Go limitations; no workaround | 2; 6 (interrupting blocked I/O); 18 (Unicode word-boundary emulation) | +| Withdrawn private-debug-default remedy; current defaults retained | 4 | + +Cross-cutting test and documentation work belongs with each implementing change. +The [assessment](audit/assessment.md) remains the detailed boundary for mixed +entries. All seven findings in the focused [removal audit](audit/14-redaction.md) +are covered by the evidence, repair, CLI, and verification sections above.