Files
audita/docs/roadmap/audit/14-redaction.md

17 KiB
Raw Blame History

Credential-redaction removal and simplification audit

Reviewed: 2026-11-15 at a68a951; application code remains at c4ccd78. Status: audit complete; implementation pending.

This review follows the maintainer's resolved D1 decision. It identifies machinery to remove or simplify after withdrawal of the application-wide credential-redaction requirement. Native library behavior, authentication, diagnostic evidence, and ordinary output correctness remain independently useful. The architecture policy has already been updated.

Scope and verification

The review traced credential discovery into CLI, reports, debug writers, proposal generation, validation, and repair. Knowledge-graph searches and caller traces were checked against source, including interface assertions and closures that the graph does not resolve completely. Tests were read to distinguish redaction-only expectations from authentication, serialization, artifact association, and failure behavior. Documentation and previous audit remedies were reviewed for assumptions that would reintroduce the withdrawn requirement.

This was a static review: no code changes, live provider calls, credential-value inspection, test execution, or performance measurements. Removed passes and allocations below are visible in the code; their production cost is unmeasured.

1. Remove credential discovery and tracking used only for substitution

Remove. Runtime.ConfiguredSecrets enumerates every captured profile and backend, caches environment-variable names with sync.Once, and reads their values whenever called. It includes raw and trimmed forms and deduplicates additional names. Discovery uses context.WithoutCancel. RoleClient separately collects prepared credential names under a mutex, solely to supply this method. None of this selects or supplies authentication to a provider. Discovery and reads · Runtime cache · Prepared-name tracking · Role accessor

Consumers are CLI report/debug preparation and finalization, the error redactor, proposal and validation writer setup, and combined repair/validation secret collection. Remove secrets.go, the runtime cache fields, role credential fields and locking, accessor methods, and secret-source assertions after updating those consumers. Keep OnPrepared request capture, terminal-event synchronization, call IDs, source-path tracking, and native engine execution; those have separate consumers. Do not leave empty secret-provider methods or no-op redactors. CLI consumers · Finalization · Repair collection

Delete TestConfiguredSecretsIncludesUnselectedBackends. In TestRoleClientUsesNativeTargetAndRedactsCredential, retain the assertion that PromptKit sends the selected, trimmed Authorization value; remove the independent redactor assertion. Keep native credential selection/optional-absence and provider-free inspection tests. Update the credential-tracking description in the runtime guide when removal lands. Catalog: #63; prerequisite consumers are covered by §§2, 4–6.

2. Replace sanitizing serialization with direct encoding

Remove / simplify. The entire core/redaction package serves substitution: raw/escaped secret expansion, longest-first matching, generic JSON decoding and tree reconstruction, key-collision rejection, and YAML node/comment traversal. Its JSON path marshals, decodes with UseNumber, walks, then marshals again; its YAML path parses and re-encodes already serialized data. These passes run even when the secret list is empty. Matching keys and fixed strings also causes the schema corruption already recorded in catalog #9. Redactor · Report YAML

Use direct JSON serialization in RunDirectory and DiagnosticsWriter, and direct typed YAML serialization for ordinary reports. Write already-produced YAML bytes without another parse/walk/encode pass. Remove secret parameters and stored redactors. In Markdown, remove substitution and the redactor parameter to writeMatrix, retaining the independent text/HTML/table escaping. Run artifacts · Interaction artifacts · Markdown

Retain serialization errors and contextual write errors, valid artifact envelopes, numeric and boolean types, optional fields, and raw final_content as a string even when its contents are malformed model JSON. No nested-content sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective configuration rendering; only Audita's subsequent YAML transformation goes away.

Move useful scalar/schema and failed-write assertions from redactor tests to the owning report/artifact tests before deleting the package. Update diagnostics/reporting internals. Catalog: #3 and #9; preserve #26's independent Markdown correction and coordinate with #33's redundant-serialization work without resolving D5.

3. Save the acquired source snapshot without reconstructing it

Simplify. WriteSourceTranscript converts the acquired bytes to a string, decodes a generic JSON tree, then sends it through the redactor's second serialization/decode/walk/serialization sequence. The comment explicitly ties this reconstruction to avoiding unredacted bytes. The CLI already has both the acquired source bytes and parsed source, and calls this writer only for debug. Snapshot writer · Caller

Write the acquired bytes directly to the source artifact, preserving the input that was actually read. Keep parsed-source and normalized artifacts as separately serialized typed views. Do not reopen the input file, which could have changed, or preserve a second parser as a validation gate. Broader input-buffer lifetime work remains catalog #31's concern.

Move source/normalized snapshot coverage out of the deleted redactor package. Check source-byte fidelity at this boundary and decoded fields for the typed views; preserve nonfatal debug-write behavior. Describe the source/parsed distinction in the diagnostics guide when implemented. Catalog: #31; depends on the direct writer changes in §2.

4. Remove secret plumbing from writer and preflight contracts

Simplify. Proposal generation discovers secrets through an optional client interface and constructs an adapter-specific writer. Validation uses a lazy func() []string closure and recreates a writer per interaction. Repair combines two roles' secret lists for its writer. These are extra dependencies on model clients for a filesystem concern. Proposal fallback · Validation setup · Validation adapter · Repair writer

Remove the discovery assertions, closure, combined-role helper, and secret constructor arguments. Complete the already-proposed common interaction writer contract without preserving these seams. Role owners still supply their payloads and stage/section/proposal/attempt/validator coordinates; the writer returns artifact paths and write errors. Retain the optional diagnostic-error callback and its synchronization, which report nonfatal write failures.

Also remove DebugInput.Secrets, the corresponding PrepareDebug and NewRunDirectory parameters, and secret captures in report-rendering closures. Preflight readiness, initial/failed report construction, output-session cleanup, and debug setup remain useful without any credential values. Preflight handoff · Debug preparation · Report closures

Adapt proposal, validation, repair, and preflight fixtures to the smaller contracts. Preserve role attribution, distinct repair artifact paths, and nonfatal write-failure coverage. Update the diagnostics guide with the resulting ownership. Catalog: #43, coordinated with #42 and #48; no retention or execution-policy change is implied.

5. Stop rewriting validator explanations sent to repair

Remove / simplify. Inside repairCandidates, safeFindings first constructs domain feedback, then replaces substrings matching credentials from both repair and validation clients. This runs for the initial rejection and subsequent revalidation findings, independently of debug capture. It changes the next model request, not just a saved diagnostic. A matching substring in otherwise useful validator text therefore changes the corrective instruction. Feedback transformation · Repair request

Use repairFindings directly at both points and remove the wrapper and its secret dependency. Retain per-proposal filtering, validator/reason labels, and the explicit unavailable-assessment representation: operational unavailability is not an actionable editorial explanation. Preserve bounded coordination, revalidation, withdrawal, and sibling independence. Domain feedback

Replace TestRepairFeedbackRedactsValidationCredentials with focused coverage of actual repair feedback if existing feedback/request tests do not already cover it. Assert explanation and proposal association through decoded request data, without introducing an environment-driven passthrough contract or prompt wording checks. Update the repair-flow description if needed. Catalog: #64; coordinate removal of repairSecrets with §4.

6. Simplify CLI presentation while retaining typed failure ownership

Remove / simplify. runProcess invokes Runtime.RedactError for preparation failures. finishProcess unconditionally discovers credentials and builds a redactor, even without reports or debug; it rewrites warnings, final errors, and announced report/debug paths. Removing these calls eliminates environment reads and permits paths to identify their actual destinations. Preparation failure · Final presentation

Present those existing errors and paths directly through CommandKit, preserving typed causes, reporting modes, completion behavior, and exit classification. Drop redaction-only parameters such as finishProcess's loaded argument once its only use disappears. Retain the required-output/nonfatal-debug distinction. Simplify the path-announcement subprocess fixture; retain useful destination, output-mode, and error assertions at their existing test owners.

There is also a distinct information-loss boundary: classifiedFailure reduces native setup errors to a generic message, and Failure.Error() returns only that message. Deleting substring replacement does not itself make missing credential diagnostics actionable. Address catalog #25 by carrying the native setup detail into the surfaced error, while retaining Class, Reason, cause unwrapping, cancellation precedence, and operation attribution. Classification · Failure wrapper

Do not indiscriminately unwrap every cause into every report. The compact reason/message projection in processreport serves the ordinary report schema, and structured failure types drive recoverability and peer cancellation. Retain those purposes; revise blanket comments/tests prohibiting cause presentation where they enforce only the withdrawn guarantee. Test setup diagnostics for actionable context and typed classification, not exact incidental wording. Report projection · Failure tests

Catalog: #65 and #25; coordinate with #22–24 and #41 without changing their classification or report-schema scope. Update operations and internal runtime documentation alongside the implementation.

7. Retire obsolete assertions and documentation without losing coverage

Remove / simplify. Redaction assertions are mixed into tests with independent value, and current documentation still describes the implemented sanitizer. Remove pure substitution tests and synthetic credential-echo fixture modes used only for them. In mixed tests, preserve native authentication, decoded request and response evidence, schema metadata, repair path uniqueness, scalar types, failure counts, and optional-artifact failure behavior.

Location Treatment when implementation lands
core/redaction/redaction_test.go, llm/secrets_test.go Delete removed-mechanism tests; relocate useful writer assertions as described above.
llm/diagnostics_test.go, proposal_generation/generate_test.go, runner/runner_test.go Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations.
runner/repair_test.go Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary.
core/reporting/report_test.go, llm/native_adapter_test.go Retain typed report encoding and actual native authentication; remove attached substitution assertions.
cmd/audita/main_integration_test.go Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements.
internal/testsupport/files.go Remove the already-unused secret scanners with catalog #61, rather than repurposing them.

Review operations, diagnostics/reporting, runtime, contributor recipes, component overview, and the migration guide when code lands. Remove broad redaction claims and obsolete implementation instructions. Preserve native configuration inspection/redaction documentation and the effective-config artifact name; neither requires a naming migration merely because local postprocessing goes away. Historical audit evidence stays intact, with revised remedies in the catalog and assessment. Do not add a negative redaction policy.

Catalog: #3, #60, and #61, and supporting work for all findings above. No new broad secret scanner or prompt-language change detector should replace the removed tests.

Retained responsibilities and boundaries

  • Native authentication and inspection: PromptKit still owns selected credentials and transport; ConfigKit/ckconfig/ckprompt own their normal rendering and inspection. Keep Audita's WriteEffective call and standard command wiring. Prepared snapshots contain messages and prompt metadata; removing substitution does not require adding headers or environment dumps. Configuration rendering · Command wiring · Prepared snapshot
  • Ordinary filesystem behavior: retain unique run-directory creation, run-ID validation, stage-name-to-filename conversion, required-output staging, destination checks, cleanup, and useful write errors. These prevent collisions or incorrect writes. No extra private-root or hostile-filesystem mechanism was found to remove in the debug writer. Catalog #4's previously proposed private defaults lack a requirement under D1; leave current location/modes unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain.
  • Domain and report behavior: retain transcript parsing, validation, feedback classification, typed operation failures, stable report identities, Markdown escaping, and nonfatal optional diagnostics. Removing credential substitution does not imply dropping these boundaries.

Suggested implementation order

  1. Replace serialization and source-snapshot reconstruction, removing writer and report secret arguments together with their callers (§§2–4).
  2. Remove repair-feedback and CLI substitution, then delete now-unreferenced discovery state and the redaction package (§§1, 5–6). These are a coordinated cleanup, not an interim design with empty secret lists or no-op methods.
  3. Complete common interaction-writer ownership and actionable setup diagnostics with catalog #43/#25. Update affected tests and canonical documentation in each implementing change (§7).

Use targeted boundary tests and then the full suite for the cross-package code change. Check decoded artifact fidelity, raw source preservation, native auth, repair feedback, report types, and failure behavior. No new product decision is needed for these D1 removals; D2–D7 remain separate and unresolved.