Files
audita/docs/roadmap/audit/catalog.md

35 KiB
Raw Blame History

Consolidated audit catalog

Audita is a single-process Go CLI. CommandKit dispatches commands and ConfigKit resolves application settings. Core packages parse, normalize, and chunk transcripts. The runner coordinates proposals, validation, repair, and application through a PromptKit adapter. The CLI publishes the corrected transcript and ordinary reports, with separate opt-in debug artifacts. Package map · Pipeline

The 12 source audits contain 110 numbered findings. This catalog consolidates them into 65 entries. Priority reflects potential impact, exposure, and the strength of the reported evidence. A source can appear under several entries when it covers distinct issues. Each link leads to the original finding and its supporting evidence.

The assessment and decision register evaluates every entry using the maintainer-approved taxonomy and separates ready remediation from technical investigation and pending policy decisions. Consult it before planning changes; source-audit recommendations are evidence and proposals, not authorization. The three WONTFIX scopes recorded here remain controlling. The focused redaction-removal audit applies resolved D1: the earlier redaction remedies below are superseded, and #63–65 add distinct cleanup work. Entry numbers and existing anchors are retained for traceability.

Data protection and run integrity

1. Bind output writes to the checked destinations

Transcript path resolution can select the wrong target, and report writers can change referents after preflight. Use filesystem-consistent resolution and carry the checked destination into publication. Cover changed report referents as well as initially ambiguous paths. Correctness §1 · Concurrency §6 · Security §3 · Testing §3 · Complexity §9

2. Keep model requests within the configured transport boundary

Provider redirects can take transcript-bearing requests outside the selected origin or downgrade the transport.

WONTFIX. PromptKit inherits the standard net/http client's default redirect and credential-forwarding behavior. Audita accepts those defaults for the built-in transport. Security §2

3. Protect credentials inside nested debug responses

Superseded by D1. The original observation tested the withdrawn redaction requirement. Remove Audita's substitution package and use direct artifact serialization; retain useful responses, including malformed content. Retire masking-only tests and documentation while preserving evidence-fidelity and write-failure coverage. Credential discovery and repair/CLI consumers are covered by #63–65; no nested sanitizer or response-omission policy is selected. Security §4 · Testing §8 · Redaction removal §2 · Redaction removal §7

4. Make debug artifact permissions private by default

Prior remedy withdrawn under D1. Operators own artifact access and sharing; hostile local filesystem access is outside the selected threat model. The focused audit found ordinary creation and filename handling to retain, not a redundant private-root mechanism. Leave current locations and modes unchanged in this cleanup. Any future default change needs an independent justification. Security §1

5. Define the trust boundary for discovered configuration

The selected main file and its automatic companions can influence provider and credential selection. Document trusted configuration locations and keep upload-controlled paths separate from operator configuration. Security §5

6. Make blocking transcript input and delivery cancellation-aware

Input can block before preflight, and FIFO or stdout delivery can stay blocked after cancellation.

WONTFIX (interrupting already-blocked I/O). The standard file operations used here take no context, and io.Writer offers no general interruption mechanism. Audita accepts that those calls can remain blocked until I/O can proceed.

The application still owns the outcome when control returns. Check cancellation before publishing a regular file and after stream delivery returns so a canceled run exits nonzero and attempts to write failed-status reports. Correctness §5 · Concurrency §1 · Concurrency §2 · Resources §2 · Security §7 · Testing §3

7. Publish truthful failure reports and preserve initialized evidence

A delivery failure can briefly publish completed reports. The internal preflight seam can also replace initialized reports with empty files when failed-report rendering is unavailable. Verify the first published state and preserve valid existing evidence if a failed bundle cannot be rendered. Error handling §1 · Complexity §13 · Testing §3

8. Prevent concurrent runs from clobbering ordinary reports

Invocations sharing a report directory replace the same basenames and can leave a mixed set. Resolved D2: destination isolation remains the caller’s responsibility. Retain current report placement and document distinct report directories for concurrent jobs or retained run histories. Automatic per-run placement, locking, waiting, and competing-writer rejection are outside this remediation. Correct publication to checked destinations remains #1’s concern. Concurrency §7

9. Preserve the ordinary YAML schema during redaction

A credential value can match a required key or fixed enum and alter an otherwise successful report. Under resolved D1, remove the post-serialization redactor and serialize the typed report directly. Keep schema/scalar fidelity coverage; do not build a more selective sanitizer. Correctness §9 · Redaction removal §2

10. Reject contradictory validation decisions

The internal decision boundary can advance a candidate marked both approved and unavailable. Validate the disposition and approval invariant before filtering or repair. The bundled validators produce consistent decisions in the audited paths. Complexity §11

11. Preserve usable validator decisions when extra references cannot be decoded

A schema-valid unexpected numeric reference can fail whole-batch Go decoding before per-entry reconciliation. Preserve valid requested siblings and test the mixed response at the actual adapter boundary. Correctness §7 · Testing §5

12. Use full-segment evidence for protected-name checks

Span-only validation can reject a valid edit or miss a repair opportunity. Share a glossary-owned matching and protection assessment across validation and application, and recheck the current segment when applying an edit. Correctness §8 · Equivalence §9 · Boundaries §5

13. Preserve transcript input fidelity during JSON shape detection

Generic decoding can round large integer IDs and fail on values in fields the typed reader ignores. Select the segment array as raw JSON and test exact IDs across both supported input shapes. Correctness §6 · Testing §7 · Resources §4

14. Honor cancellation at the runner commit boundary

An already canceled empty pipeline can return success, and cancellation during module application can still allow the runner to commit edits. Check entry and final commit boundaries and give successful and interrupted results one assembly path. Complexity §1 · Complexity §12 · Concurrency §3

15. Retain completed proposal evidence behind slow sections

The ordered prefix gate can delay validation for ready sections and omit completed proposal evidence on interruption. Capture usable responses independently of validation dispatch and preserve deterministic identities. Correctness §10 · Concurrency §4 · Complexity §1

16. Keep ownership of failed staging cleanup

Removal errors can discard temporary paths or go unreported, preventing a retry. Keep paths until removal succeeds or the path is absent and verify cleanup failures and cancellation after output acquisition. Resources §1 · Testing §3

Correctness and failure reporting

17. Preserve the complete time interval when merging segments

A nested segment can shorten the merged end time and change later gap and duration decisions. Use the full interval for both eligibility and output, and cover nested intervals in normalization tests. Correctness §3 · Testing §7

18. Match protected names at Unicode and punctuation boundaries

The matcher misses some names with accented letters or punctuation at their edges.

WONTFIX (Unicode behavior of \b). Go's regexp defines \b as an ASCII word boundary. Audita accepts that definition for the standard regexp engine.

Configured aliases with punctuation at their edges still need a glossary matching rule for protection and context selection, even when their letters are ASCII. Correctness §2

19. Detect overlapping occurrences when requiring a unique edit span

Nonoverlapping counting can treat a repeated overlapping span as unique. Use one literal occurrence rule in preview, validation, and application. Efficiency §7

20. Satisfy feasible explicit chunk targets

A balancing preference can reject inputs that have a valid contiguous partition. Make the configured count and limits prerequisites, then optimize balance within feasible choices. Verify skewed inputs at the feasibility boundary. Correctness §4 · Testing §7

21. Keep input order for exactly tied timestamps

An unstable sort can reorder utterances with identical times. Use source position as a stable tie-breaker. Efficiency §8

22. Preserve failure phase and origin in run reports

Required output errors can omit their phase, and fatal model errors can lose chunk, role, validator, or batch coordinates. Carry typed failure facts from their source and assemble run records and counts in one place. Assert the serialized fields. Correctness §11 · Error handling §4 · Testing §3 · Boundaries §8 · Complexity §9

23. Classify provider attempt timeouts separately from operation deadlines

The adapter can report an HTTP attempt timeout as a logical operation deadline. Use the native classifications and test attempt, operation, and parent cancellation under distinct live budgets. Error handling §2 · Testing §4

24. Use input diagnostics for operator-correctable failures

Invalid processing selections and malformed inspection configuration can return the runtime exit status. A genuinely infeasible chunk target can also be reported as an internal defect. Preserve the distinction between readiness, chunking, and runtime failures and assert exact statuses in CLI tests. Preserve native inspection behavior; retired application-wide masking expectations are not an additional acceptance criterion. Error handling §5 · Error handling §6 · Testing §2

25. Identify missing required credentials in safe diagnostics

The process path reduces a known missing credential requirement to a generic setup failure. Surface actionable native setup detail and preserve the underlying typed cause and failure classification. D1 removes the blanket restriction on cause presentation; ordinary compact report projections remain useful. Error handling §3 · Redaction removal §6

26. Render input and output paths as plain Markdown text

Untrusted filenames can alter the rendered report or its navigation. Escape path text consistently with other untrusted labels and verify the rendered meaning. Security §6

27. Keep the validator execution class used by the runner

The ledger can infer a different class from an injected validator's name. Carry the resolved class in decision evidence and join the evidence once before emitting rows, preserving disposition and repair lineage. Complexity §8 · Boundaries §2

28. Verify signal and exit handling in the production executable

Subprocess fixtures recreate startup wiring and can miss regressions in the actual binary. Exercise the built executable or share the production entry point for representative startup and cancellation cases. Testing §1

29. Fail the release bootstrap when cleanliness cannot be established

A failed Git status command can be mistaken for a clean checkout. Check its status explicitly and exercise that boundary in place of testing the release test's own comparison helper. The delivery tool has a separate prepared-checkout check. Error handling §7 · Testing §11

Scalability and resource use

30. Bound pending proposal work before backend admission

One goroutine per chunk makes finite-capacity rejection depend on scheduling and can retain requests and results across the whole run. Bound dispatch and result retention without changing the module barrier or backend admission ownership. Resolved D3: preserve PromptKit's configured waiting and recoverable rejection. Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual- admission promise. Current scheduling-dependent rejection order is not a compatibility requirement. Verify memory use, actual contention, and cancellation while improving local scheduling; any future admission-policy gap belongs upstream. Concurrency §5 · Resources §3 · Testing §6

31. Reduce peak memory for large transcript input and output

Whole-file reads, a generic JSON tree, retained source bytes, and full output buffers amplify memory before and after model work. Reduce live copies and write the acquired source bytes directly to its debug snapshot instead of reconstructing them for redaction. Keep parsed/normalized views separate. Input-size limits remain a separate D4 decision. Redaction removal §3 · Resources §4 · Security §7

32. Retain only the run evidence required by enabled outputs

Ordinary runs retain debug-only candidates and decision history. Summary-only runs still allocate the full work-unit grid, and module results duplicate operation records and unused proposal totals. Preserve live validation and repair state, and collect compact counts when detailed evidence is disabled. Efficiency §5 · Resources §5 · Dead code §3

33. Limit debug retention costs within a run

Execution histories and repeated serialization increase peak memory, and retained interaction files and warnings can accumulate after write failures. Reduce duplicate representations and define retention and failure behavior appropriate to opt-in capture. Resources §6

34. Account for cumulative correction-repair history and work

Each rejected candidate can start its own bounded conversation and repeat the original prompt history, with additional revalidation. Measure total requests and bytes across branches before deciding on changes to context or run-level budgets. Resources §7

35. Avoid rebuilding growing normalization prefixes

Merge estimation and category accumulation repeatedly copy and scan the same growing text and category prefixes. Reuse incremental work while preserving exact text, category order, and the configured limit. Efficiency §1

36. Skip validation context work beyond the usable budget

Optional context is serialized even when required fields already exceed budget. Share trimming and batching policy, stop measuring context that cannot fit, and keep serialized measurement as an independent reference for the optimized path. Efficiency §4 · Equivalence §1

37. Reuse checked processing selections and matching views

Preflight and execution separately decide active roles, and repeated profile or glossary selections repeat inspection, copying, and preparation. Carry one checked processing plan and reuse immutable results for equivalent selections. Efficiency §6 · Boundaries §4

38. Select glossary context by identity membership

The validation context builder materializes each matched occurrence for every contributor and then deduplicates them. Use a membership path for context and preserve occurrence counts for protection. Efficiency §3

39. Avoid rescanning sections for skipped edits

Apply-time filtering searches the original chunk for every skipped proposal. Reuse a segment lookup and preserve stale-conflict and filtering distinctions. Efficiency §2

Ownership and maintainability

40. Use one validator contract and one owner for decision semantics

Runner-side applicability and verdict handling overlap with validator implementations. Consolidate the interface and acceptance rules, and remove the unused validation request envelope, segment ID field, and result name copy. Complexity §3 · Boundaries §1 · Dead code §4

41. Carry structured failure facts through model operations and reporting

Proposal, validation, and repair repeat class and reason extraction, and report counts reclassify failures from reason codes. Carry typed outcome facts forward while preserving role-specific recovery and fatal handling. Complexity §7 · Equivalence §4

42. Keep required output publication separate from optional debug retention

The output session combines unrelated lifecycles. Give required-output preparation and publication a clear owner, share regular-file staging mechanics for transcript and reports, and expose debug setup through its own API. Complexity §9 · Equivalence §2 · Boundaries §7

43. Inject one interaction diagnostics contract

Proposal, validation, and repair duplicate path types and writer wiring. Supply a diagnostic sink from the adapter boundary, with role-specific payloads and nonfatal write warnings. Remove secret-provider assertions, lazy discovery closures, combined-role secret lists, and preflight/writer credential arguments. Keep artifact association and optional write-error reporting. Complexity §4 · Equivalence §5 · Boundaries §6 · Redaction removal §4

44. Simplify structured-call completion and remove obsolete record fields

Share finalization across ordinary calls and coordinated turns, and remove obsolete error-kind state, response usage copies, and duplicate callback payloads. Keep typed causes and live execution and artifact evidence. Complexity §5 · Dead code §5

45. Retire the test-only direct continuation branch

The supported repair path uses the coordinator. Move direct-continuation assertions about sibling isolation and structural repair to coordinated calls, then remove the unused request branch. Complexity §5 · Dead code §1

46. Use a single proposal handoff and test real runner indexing

The generator builds enriched proposals the production wrapper discards. Give the runner index ownership, remove the parallel enrichment, and assert indices under observed out-of-order completion instead of in the test-only composition harness. Complexity §6 · Testing §6 · Testing §9 · Dead code §2

47. Centralize request-local segment reference resolution

Initial proposals and repair revisions duplicate the same local-position lookup. Preserve raw references, stable proposal identity, and revalidation for retargeted edits. Equivalence §3 · Complexity §6

48. Give one repair branch a clear execution and result boundary

Branch coordination, previews, revision validation, accounting, and diagnostics share state inside the section loop. Separate branch state and make repair prompt, exchange capture, and coordinator capability explicit. Complexity §2 · Complexity §6

49. Define concurrency and ownership in module and validator contracts

Workers share module and validator instances and borrow section and glossary data. Specify concurrent calls, read-only borrowed inputs, and cooperative cancellation at the interface boundary. Concurrency §8

50. Consolidate root and per-module configuration rules

Resolution and validation repeat role defaults, null and inheritance rules, and field checks. Share the rules while keeping path-specific diagnostics and configuration provenance. Complexity §10 · Equivalence §6

51. Unify transcript conversion and retire redundant parser wrappers

Source-to-canonical conversion is duplicated across the CLI, a lenient wrapper, and fixtures. The strict test-only wrapper panics on omitted IDs. Use one conversion and the supported decoder, and preserve independent fixture expectations. Efficiency §9 · Testing §12 · Boundaries §9 · Dead code §8

52. Give public transcript encoders one segment projection

The output registry implements the envelope and delegates the bare form to the schema package. Share the field projection at the output boundary and verify both wire shapes independently. Boundaries §9

53. Use one simple token estimate across text-processing packages

Normalization and chunking implement the same word and punctuation heuristic. Share the counter and preserve consumer-specific budgets and independent expected-count tests. Equivalence §7

54. Centralize transcript cloning

Runner and proposal application duplicate deep-copy code with different empty-category handling. Share the copy operation while keeping both caller ownership boundaries and verifying independence from fixture data. Equivalence §8

55. Assemble built-in modules at the composition boundary

The framework registry imports all concrete modules even though the runner consumes a factory interface. Keep config keys independent of implementations and place production construction with the CLI or built-ins. Boundaries §3

56. Merge shared prompt variables once per model call

Processing loads the embedded instruction values twice. Keep processing precedence and the missing protected-term default in one pass, and preserve caller precedence for inspection. Dead code §6

57. Clarify what the inherited progress flag does in processing

process advertises progress messages but supplies no producer. Define its behavior or clarify process help while preserving acceptance for existing scripts. Dead code §10

Test maintenance and dead code

58. Test registered validator roles and embedded assets through their behavior

Consolidate redundant execution-class checks and assert required prompt identities and preparability. Keep schema identity, diagnostic metadata, and protective-copy coverage. Testing §10

59. Consolidate overlapping normalization fixtures

Two different-speaker fixtures protect the same rejection. Keep the distinct domain cases and meaningful merge, text, and category assertions. Testing §12

60. Focus integration tests on decoded artifacts and independent output choices

Assert decoded IDs and evidence associations and use smaller fixtures for duplicate reporting and debug combinations. Keep a full scenario for attribution through repair and later failure. Remove masking expectations without discarding the independent artifact and outcome assertions mixed into those fixtures. Testing §13 · Redaction removal §7

61. Remove abandoned test support and unused fixture inputs

The orphan test-support package, unused CLI and runner helpers, and legacy glossary fixture input have no current callers. Preserve compile-time interface fixtures and active behavior assertions. Redaction removal §7 · Dead code §9 · Testing §13

62. Remove unreachable merge and file helpers

The uncalled merge predicate and standalone JSON and file-write helpers duplicate or bypass the live normalization, schema, and output paths. Dead code §7

Additional cleanup after resolved D1

63. Remove redaction-only credential discovery and tracking

Audita enumerates captured profiles/backends, reads credential environment values, caches names, and tracks prepared role names under a mutex solely for substitution. Remove that state and the secret-source methods after updating their consumers. Preserve PromptKit authentication, prepared evidence, execution synchronization, and native inspection behavior. Redaction removal §1

64. Preserve validator explanations in repair feedback

Repair rewrites validator explanations using both roles' credential values even with debug off. Remove this transformation and use domain feedback directly. Retain proposal association, unavailable-assessment semantics, and the existing bounded repair/revalidation behavior. Redaction removal §5

65. Remove credential substitution from CLI presentation

Preparation and completion paths scrub warnings, errors, and announced paths; completion discovers credentials even with reports/debug disabled. Use direct CommandKit presentation, remove obsolete arguments, and retain typed causes, reporting modes, exit status, and required-output failure handling. Coordinate with #25 for setup detail still hidden by the separate failure wrapper. Redaction removal §6