Files
audita/docs/roadmap/reliability-and-maintainability.md

445 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Audita Reliability and Maintainability Roadmap
**Date:** 2026-11-15. **Status:** [Implementation landed](implementation.md).
[Post-implementation findings and their status](remediation.md) track follow-up work.
**Planning baseline:** `730c1e4`; its application code matched the audited `c4ccd78`.
## Purpose and scope
Audita should produce faithful transcripts and useful evidence, handle partial
success and failures consistently, and use resources proportionately to the work
and outputs selected. Its components should have clear owners, with less local
machinery where standard Go facilities or the adopted libraries already supply
the behavior. Operators requesting progress should be able to follow a run in
detail through CommandKit.
This roadmap organizes the accepted audit remediation around that target state.
The [assessment](audit/assessment.md) owns the disposition of individual findings
and the rationale for decisions D1–D7; the [catalog](audit/catalog.md) links to
their evidence. The coverage appendix below maps that evidence to the target areas.
Earlier source-audit remedies remain proposals where the assessment supersedes
them. This document specifies the target state and accepted scope. The
[implementation plan](implementation.md) owns the sequence and verification.
The baseline already had native library integration, preflight,
ordered module execution, partial-success validation, bounded correction repair,
ordinary reports, and opt-in debug artifacts. The sprint improved those paths
and added processing progress. It preserved editorial policy and the public
configuration schema. The D1 architecture-policy update preceded the related
code cleanup.
### Design boundaries
Follow the [architecture policy](../policy/architecture.md) and use the libraries'
supported paths. ConfigKit owns composition, presence, provenance, and resolution
before handoff. PromptKit owns credentials, model execution, admission, retries,
and conversation coordination. CommandKit owns standard commands, reporting
modes, and stream routing. ReleaseKit retains its release-validation ownership.
Audita owns transcript meaning, role selection, domain validation and repair
decisions, processing coordination, and presentation of its results.
The accepted decisions establish these boundaries for this remediation:
| Area | Selected scope |
| --- | --- |
| Diagnostic content | Remove Audita's credential substitution and the discovery/plumbing supporting it; retain native library behavior and useful evidence. |
| Report destinations | Keep existing placement; callers isolate concurrent jobs and retained run histories. |
| Admission | Keep native waiting and recoverable rejection. Improve local scheduling without a second backend limiter, admission-retry loop, or eventual-admission promise. |
| Input and capture resources | Remove waste; add no transcript-size cap, capture quota, automatic artifact deletion, or automatic stop-capture policy. Operators own retention and integrations own their upload/process budgets. |
| Correction repair | Retain full conversation history and existing per-correction limits. Aggregate budgets and reduced context are not selected at this time. |
| Progress | Supply relatively detailed processing activity through CommandKit when progress is requested, subject to its effective reporting mode. |
Respect the three accepted Go limitations: standard HTTP redirect behavior,
standard regexp word-boundary semantics, and the inability to interrupt an
already-blocked ordinary file/stream call. Fix Audita's own choice of operation
and its handling when control returns. Output correctness here does not promise
hostile-filesystem race resistance, cross-file transactions, power-loss recovery,
or atomic synchronization with cancellation. Keep ordinary symlink, in-place
output, and stream support. Debug root locations and creation modes stay as they
are for this cleanup.
The separate features in [deferred work](future.md), including semantic recovery
of unavailable validator decisions and independent whole-pipeline traversal by
chunk, remain outside this roadmap. Internal refactoring should preserve current
contracts except for the deliberate changes and correctness fixes described here;
it need not preserve incidental implementation details or known defects.
## 1. Truthful output and useful diagnostic evidence
### Target state
Required output writes use the destinations that were actually checked. Reports
reflect known processing and delivery outcomes, retain available failure origin,
and remain useful when finalization itself encounters an error. Optional debug
capture has a separate lifecycle and does not determine processing success.
Diagnostic representations preserve the information supplied to them through
ordinary serialization.
The baseline resolves some output paths differently at checking and writing,
can publish a completed report after learning of delivery failure, and can lose
initialized evidence or staging-cleanup ownership. Credential substitution adds
serialization passes and can alter schema keys, messages, and path announcements.
The [output contract](../integrations/output-files.md) and
[operations guide](../operations.md) describe the existing output surfaces.
### Work to reach the target
- Give required-output preparation, staging, publication, and cleanup one clear
owner. Resolve paths using filesystem semantics and carry checked destination
information into publication. Share regular-file staging mechanics between
transcript and reports, retaining private staging and intentional in-place
replacement. Keep ownership of temporary paths until removal succeeds or
absence is established; report cleanup failures alongside the primary error.
- Keep transcript-before-report publication and nontransactional finalization.
Select failed report status when a delivery failure is already known, carry
report-rendering errors explicitly, and preserve initialized reports when a
replacement failure bundle cannot be rendered. Retain bounded best-effort
attempts to publish enabled reports; a later report failure does not erase an
already-published transcript.
- Check cancellation at natural publication boundaries and after stream delivery
returns. A canceled run should report failure even if bytes have already
escaped. Do not implement detached I/O workers to simulate interruption of
standard blocking calls.
- Build ordinary reports from processing-owned facts, preserving phase, available
stage/chunk/operation origin, documented counts, and existing serialized
identities. Distinguish the initiating failure, peer cancellation, and secondary
output errors. Join correction evidence once per proposal before projecting
ledger rows; carry the validator execution class actually used rather than
reconstructing it from a name. Preserve disposition, repair lineage, empty
replacement values, partial decisions, and module-only failures.
- Complete the [redaction-removal inventory](audit/14-redaction.md): delete
redaction-only credential enumeration, environment reads, name caches, role
tracking, substitution utilities, and secret-only parameters/interfaces. Encode typed
JSON/YAML directly. Save acquired source bytes without reconstructing or
reopening the source; keep parsed and normalized views separately. Preserve
ConfigKit rendering and native authentication/inspection.
- Supply one interaction-writer contract from the adapter/composition boundary.
Proposal, validation, and repair owners supply their own payloads and
association coordinates. Remove lazy secret callbacks and cross-role secret
lists. Retain self-contained interaction artifacts, malformed final response
content as evidence, nonfatal write callbacks, and their synchronization.
Consolidate equivalent write warnings with counts and affected-artifact
information while continuing best-effort capture.
- Keep Markdown escaping separate from substitution: input/output paths should
render as literal text without changing their structured values. Document
caller-owned report-directory isolation and artifact retention when updating
the operator guide.
### Completion evidence
Focused filesystem tests cover checked destinations, known delivery failure,
failed report rendering, cleanup errors, and cancellation once I/O returns.
Assertions inspect publication behavior as well as final files. Decoded reports
retain their schema and types, and ledger evidence agrees with execution.
Artifact tests establish source fidelity, role/attempt association, and nonfatal
debug failures. Redaction-only code and assertions are removed without replacing
them with no-op seams or losing native authentication coverage.
## 2. Bounded processing with clear operation ownership
### Target state
Local preparation, dispatch, and completed-result handling avoid allocating a
worker and prepared request for every chunk at once. Completed evidence is owned
independently of ordered application. Each stage and model operation has a clear
completion path, and typed outcomes reach reports without being reconstructed
from display strings.
The baseline fans out work across all chunks, retains out-of-order results behind
slow earlier work, and duplicates operation state across callbacks and projections.
Backend admission already belongs to PromptKit; it does not bound all of this
local preparation and retention. The [pipeline](../internal/pipeline.md) remains
ordered by stage, with fixed chunk membership and deterministic proposal/application
identity.
### Work to reach the target
- Give stage execution a single result/commit boundary. Honor cancellation on
entry, including an empty pipeline, and before committing new stage work.
Cancel and join peers on fatal failure or parent cancellation, retaining earlier
committed stage text and available completed evidence. Recoverable operation
failures remain local under the existing partial-success rules.
- Design bounded local preparation and dispatch together with concurrent draining
of results. Preserve deterministic identities, validator requirements, ordered
application, and the module barrier. Reduce avoidable within-stage waiting
without introducing independent whole-pipeline chunk traversal. Retain completed
proposal evidence on interruption even when an earlier section is unfinished.
Preserve section- and response-ordered serialized proposal indices. If
measured waiting warrants earlier validation, use local candidate references
during processing and assign serialized indices at ordered collection.
PromptKit's configured rejection remains valid; the old scheduling-dependent
choice of rejected chunks is not a compatibility requirement.
- Retain only the state needed by active processing and selected outputs. Keep
live validation/repair state, compact summary counts, and required failure
evidence. Avoid full debug histories or work-unit matrices in summary-only
paths; preserve complete detail when that output is enabled. Remove duplicate
operation arrays, unused module totals, and obsolete response/callback copies.
This is not a constant-memory promise: selected outputs and useful conversation
history can still grow with the workload.
- Carry typed class, reason, and cause through proposal, validation, repair, and
reporting. Preserve the original class/cause if a missing reason uses the
internal-failure fallback. Distinguish parent cancellation, logical-operation
timeout, and HTTP-attempt timeout using native classifications; an HTTP-only
timeout uses the existing transport-failure reason.
- Share ordinary-call and coordinator-turn completion, retaining native terminal
counts and diagnostic evidence from undelivered responses. Remove the test-only
direct-continuation branch after moving useful coverage onto production
coordination. Keep one proposal handoff with raw local references; the runner
owns global indexing. Review emitted metadata before deleting fields that only
appear unused to a function-level reference search.
- Give each repair branch an explicit execution/result boundary. Preserve native
conversation coordination, full history, per-correction limits, sibling
independence, withdrawal, and validation of the final permitted revision.
Pass domain validator explanations directly instead of applying credential
substitution; retain the distinction between editorial rejection and unavailable
assessment. Measure aggregate work to guide efficiency work, without adding
new stopping rules or ordinary-report fields for that analysis.
### Completion evidence
Controlled overlap and cancellation tests exercise actual dispatch, admission,
result draining, and stage commit paths. They establish native shared backend
capacity, useful evidence retention, and deterministic identities under observed
out-of-order completion. Distinct live timeout budgets exercise the correct
failure owner. Representative measurements demonstrate reduced pending work and
retained representations; they do not impose a new throughput target or require
every chunk to obtain admission. Repair tests run through the production
coordinator and preserve accepted feedback and revalidation behavior.
## 3. Faithful transcript and correction semantics
### Target state
Transcript parsing preserves accepted values, normalization preserves time and
ordering meaning, and chunking satisfies feasible configured constraints.
Validation and application use consistent domain evidence. Efficient shared
operations reduce repeated work without changing editorial acceptance rules.
The baseline's generic JSON shape conversion can round IDs; normalization and
chunking have concrete interval, ordering, and feasibility counterexamples.
Some protected-term checks use only edit spans, and validation response conversion
can discard usable decisions because of irrelevant extra references.
### Work to reach the target
- Detect the supported transcript envelopes while keeping their segment array as
raw JSON for typed decoding. Preserve optional IDs, accepted numeric ranges,
unknown-field behavior, and both input shapes. Share live source-to-canonical
conversion, retire obsolete strict/lenient wrappers, and release whole-input
representations after their last use. Keep accepted formats and stream inputs.
- Use the full merged time interval for merge eligibility, output, and subsequent
gaps. Preserve source order for tied timestamps without introducing a new
end-time sorting criterion. Build growing text/category state incrementally,
preserving exact joins, category order, and the current token estimate.
- Satisfy feasible explicit chunk count and maximum-size constraints before
optimizing balance. Preserve the single-segment exception above the token
maximum. Use a proportionate deterministic algorithm and retain the
configured budgets and current balancing objective. Keep accepted
compatibility settings, including `chunking.min_section_tokens`, even where
they are not active partition targets.
- Give validator applicability and decision meaning one owner and one runtime
contract. Reject contradictory internal outcomes before they advance a
candidate; treat them as internal contract failures, not unavailable model
assessments. Derive redundant approval representation where needed. Preserve
decision cardinality, zero-budget shortlisting, independent repair feedback,
and unavailable outcomes. Reconcile exact numeric response references before
local conversion, accepting integral JSON number spellings without float
rounding and ignoring irrelevant extras. Keep whole-response structural
validation and existing partial-success behavior.
- Share glossary-owned protection assessment using usable before/after segment
evidence, and recheck current text at application. Preserve active-glossary
exemptions, case/longest-match behavior, and spoken-word repetition rules.
Missing or ambiguous previews cannot support a fabricated protection verdict;
independently supported applicability findings remain valid. Empty-result
previews can still supply protection evidence without authorizing whole-segment
removal. Handle configured punctuation-edged spellings with standard primitives
while accepting Go's existing word-boundary semantics.
- For require-unique replacement, detect a second possible literal starting
offset even when matches overlap. Preserve replace-all's existing nonoverlapping
behavior. Share request-local segment lookup across initial and repaired
proposals, retaining distinct reference namespaces, raw invalid references,
and retargeted revalidation.
- Reuse glossary identity membership for context selection rather than expanding
every occurrence/contributor pair; keep occurrence evidence for protection.
Avoid validation-context work once the usable budget is settled, sharing
trimming/batching policy while preserving context priority, oversized-input
handling, and an independent serialized measurement reference. Reuse section
lookups and aggregate skipped-edit evidence once, distinguishing stale
application conflicts from response filtering.
### Completion evidence
Boundary regressions reproduce the audited numeric, nested-interval, tied-order,
feasible-chunk, overlapping-match, and full-segment-protection cases. Validator
tests pass actual structured responses through decoding and reconciliation, with
usable siblings and irrelevant extra references. Shared optimized operations
are checked against independently constructed expectations, including exact
estimates and representative allocation/work measurements. Existing editorial
acceptance, repair, and apply-time rules continue to govern results.
## 4. Smaller components with explicit ownership
### Target state
Preflight and execution share resolved application decisions. Domain models own
their conversions and copies; adapter boundaries own external effects; the
composition boundary assembles built-ins. Modules and validators remain distinct
components with documented concurrent-call and borrowed-data responsibilities.
The baseline repeats root/stage configuration rules, readiness selections,
transcript projections/clones, token estimates, and shared prompt-variable work.
Some interfaces and helpers survive only through tests or unused wrappers.
### Work to reach the target
- Consolidate Audita-owned root/stage validation and fallback rules after
ConfigKit handoff, retaining null, explicit zero, inheritance, provenance,
stage-dependent defaults, and diagnostic paths. Carry checked processing
selections from preflight into execution. Reuse effective profile/glossary
views within the captured runtime, with selection order and exemptions in
reuse keys; retain first-failure attribution.
- Share transcript cloning at the model boundary, preserving nil versus explicit
empty mutable slices and the separate caller ownership boundaries. Give public
encoders one segment projection while retaining both envelopes, schema/version
behavior, categories, omission rules, and numeric fidelity. Use one implementation
of the existing token estimate across consumers, retaining its exact results.
- Assemble built-in modules at composition, preserving the runner factory seam
and a dependency-light configuration catalog. Document concurrent-call and
borrowed-input obligations at module/validator ports; permit synchronized
stateful implementations and use owned copies where mutation is needed.
- Merge shared prompt variables once per model call, preserving the different
processing/inspection precedence, protected-term default, and operator prompt
replacement support. Remove dead helpers, unused envelopes, redundant adapters,
and abandoned test support after checking live registrations and serialization.
Move useful tests to the surviving behavior before removing test-only paths.
These changes complement the writer, validator, operation, and repair ownership
work above. Use small contracts and helpers where they serve an actual shared
responsibility; package layout and private type names are implementation choices.
### Completion evidence
Real consumers use the shared owners and obsolete paths are removed. Behavioral
tests establish copy independence, both output envelopes, effective configuration
semantics, native integration, and processing/inspection precedence. Refactoring
does not depend on a new configuration loader, plugin framework, cross-invocation
cache, or a generic replacement for the existing domain components.
## 5. Actionable errors and detailed processing progress
### Target state
Operators can identify the failed phase and relevant setup requirement, and
follow long jobs with meaningful progress. Standard inspection and processing
remain integrated with CommandKit rather than acquiring parallel command or
reporting systems.
The baseline hides some native setup detail, classifies some operator-correctable
failures as runtime failures, and exposes progress without producing processing
events. The current [CLI contract](../cli.md) supplies the stream and exit-status
boundary that the fixes should follow.
### Work to reach the target
- Classify invalid processing selections and genuinely infeasible chunk requests
according to the existing input-error contract; retain runtime classification
for I/O and cancellation. Trace delegated inspection failures to Audita's
loader or the upstream integration. The pinned ckprompt catalog handler
classifies malformed prompt configuration as an operational failure. Fix
that classification upstream while keeping native command routing.
Processing progress can use CommandKit's existing reporter independently.
- Surface actionable native setup detail, including the configured missing
credential requirement and available role/profile context, while preserving
typed causes and failure classification. Remove CLI substitution of errors,
warnings, and announced paths. Keep compact ordinary-report projections where
they serve the report schema; deleting substitution is not a request to expand
every report with raw provider causes.
- Emit relatively detailed stage/chunk, proposal, validation, and repair activity
through CommandKit's reporter, using actual lifecycle state and available
upstream execution events. Include useful work identities and observed
completion counts. Do not restrict `--progress` to coarse stage summaries just
to minimize output. CommandKit determines effective reporting-mode precedence
and routes output, including when flags are combined; stdout remains available
for the transcript. Event granularity and wording are implementation choices.
- Document the trusted configuration bundle, imports, and automatic companions
for operators and subprocess integrations. Explain how caller-owned report
destinations and artifact retention fit the workflow. Preserve native discovery
and inspection; these clarifications do not add an isolation mode.
### Completion evidence
CLI tests check the appropriate exit classifications, useful diagnostic context,
native mode combinations, and unchanged transcript stdout. Progress tests exercise
real processing activity and meaningful updates without fixing incidental wording
or concurrent event order. Update user-facing documentation alongside the feature.
## 6. Verification and delivery that support change
### Target state and work
Tests protect consequential behavior at its owning boundary and make valid
refactoring and prompt iteration inexpensive. Apply the
[testing policy](../policy/testing.md) throughout the work, rather than adding a
second set of implementation-shaped assertions at the end.
- Exercise representative signal, cancellation, and exit behavior through the
real executable or a shared production entry point. Keep smaller CLI fixtures
for cases that do not need a process boundary. Use observed overlap and
independently controlled deadlines for concurrency/timeout tests.
- Verify registered identities, effective validator roles, embedded-asset
preparability, schema identity, and protective copies through behavior. Replace
incidental registry counts, duplicate normalization fixtures, and tests of test
helpers with coverage of their actual consumer risks. Assert decoded artifact
identities and association; keep a representative repair/filter/later-failure
integration scenario and smaller independent-output cases.
- Remove redaction-only tests and fixture modes while retaining authentication,
serialization, evidence, and failure coverage. Avoid prompt-wording change
detectors, broad incidental snapshots, and new checks that recreate withdrawn
guarantees. The default suite remains offline and uses synthetic credentials.
- Fix the CI bootstrap so failure to run its Git cleanliness check cannot be
interpreted as a clean checkout. Test that command-status boundary and remove
redundant tests of the release test's comparison helper. Keep ReleaseKit's
selected checks and the existing [release policy](../policy/release.md); this
roadmap introduces no new release gates.
- Update canonical CLI, configuration, operations, integration, and internal
documentation with the corresponding changes. Remove stale redaction promises
and recipes without adding a negative architecture assertion. Keep native
configuration-rendering documentation and historical audit evidence. Use ADRs
only where the [documentation policy](../policy/documentation.md) calls for a
significant architectural record, not for every helper or audit entry.
### Completion evidence
Run focused regression and boundary tests with each change, the full offline
suite for substantial integration changes, and applicable existing candidate
checks at release preparation. Use targeted race checks for changed shared-state
paths and representative measurements for resource changes. Reuse successful
checks for unchanged content. Link/example/whitespace checks accompany relevant
documentation changes. Do not substitute coverage percentages or a collection
of test counts for evidence of the selected outcomes.
## Completion and audit coverage
The [implementation plan](implementation.md) records the landed changes and
verification. The coverage map accounts for every catalog entry, including
closed and superseded remedies. The [assessment](audit/assessment.md) records
which remedies were selected.
| Target area or disposition | Catalog entries |
| --- | --- |
| §1: checked output, truthful finalization, caller destinations | 1, 6, 7, 8, 16, 42 |
| §1: serialization, diagnostic ownership, and useful evidence | 3, 9, 22, 26, 27, 33, 43, 63 |
| §2: stage/operation lifecycle, scheduling, and retention | 14, 15, 23, 30, 32, 34, 41, 44, 45, 46, 48, 64 |
| §3: transcript fidelity, normalization, and chunking | 13, 17, 20, 21, 31, 35, 51 |
| §3: decision semantics, protection, references, and context | 10, 11, 12, 18 (punctuation portion), 19, 36, 38, 39, 40, 47 |
| §4: shared selections, models, configuration, and composition | 37, 49, 50, 52, 53, 54, 55, 56 |
| §5: operator guidance, errors, and progress | 5, 24, 25, 57, 65 |
| §6: verification, delivery, and dead machinery | 28, 29, 58, 59, 60, 61, 62 |
| Accepted Go limitations; no workaround | 2; 6 (interrupting blocked I/O); 18 (Unicode word-boundary emulation) |
| Withdrawn private-debug-default remedy; current defaults retained | 4 |
Cross-cutting test and documentation work belongs with each implementing change.
The [assessment](audit/assessment.md) remains the detailed boundary for mixed
entries. All seven findings in the focused [removal audit](audit/14-redaction.md)
are covered by the evidence, repair, CLI, and verification sections above.