Files
audita/docs/roadmap/audit/14-redaction.md

281 lines
17 KiB
Markdown
Raw Normal View History

# Credential-redaction removal and simplification audit
**Reviewed:** 2026-11-15 at `a68a951`; application code remains at `c4ccd78`.
**Status:** audit complete; implementation pending.
This review follows the maintainer's [resolved D1 decision](assessment.md#resolved-d1-withdraw-application-wide-credential-redaction).
It identifies machinery to remove or simplify after withdrawal of the
application-wide credential-redaction requirement. Native library behavior,
authentication, diagnostic evidence, and ordinary output correctness remain
independently useful. The architecture policy has already been updated.
## Scope and verification
The review traced credential discovery into CLI, reports, debug writers,
proposal generation, validation, and repair. Knowledge-graph searches and caller
traces were checked against source, including interface assertions and closures
that the graph does not resolve completely. Tests were read to distinguish
redaction-only expectations from authentication, serialization, artifact
association, and failure behavior. Documentation and previous audit remedies
were reviewed for assumptions that would reintroduce the withdrawn requirement.
This was a static review: no code changes, live provider calls, credential-value
inspection, test execution, or performance measurements. Removed passes and
allocations below are visible in the code; their production cost is unmeasured.
## 1. Remove credential discovery and tracking used only for substitution
**Remove.** `Runtime.ConfiguredSecrets` enumerates every captured profile and
backend, caches environment-variable names with `sync.Once`, and reads their
values whenever called. It includes raw and trimmed forms and deduplicates
additional names. Discovery uses `context.WithoutCancel`. `RoleClient` separately
collects prepared credential names under a mutex, solely to supply this method.
None of this selects or supplies authentication to a provider.
[Discovery and reads](../../../internal/framework/llm/secrets.go#L11) ·
[Runtime cache](../../../internal/framework/llm/runtime.go#L22) ·
[Prepared-name tracking](../../../internal/framework/llm/adapter.go#L153) ·
[Role accessor](../../../internal/framework/llm/adapter.go#L314)
Consumers are CLI report/debug preparation and finalization, the error redactor,
proposal and validation writer setup, and combined repair/validation secret
collection. Remove `secrets.go`, the runtime cache fields, role credential fields
and locking, accessor methods, and secret-source assertions after updating those
consumers. Keep `OnPrepared` request capture, terminal-event synchronization,
call IDs, source-path tracking, and native engine execution; those have separate
consumers. Do not leave empty secret-provider methods or no-op redactors.
[CLI consumers](../../../internal/cli/process_inputs.go#L90) ·
[Finalization](../../../internal/cli/process_output.go#L17) ·
[Repair collection](../../../internal/framework/runner/repair.go#L282)
Delete `TestConfiguredSecretsIncludesUnselectedBackends`. In
`TestRoleClientUsesNativeTargetAndRedactsCredential`, retain the assertion that
PromptKit sends the selected, trimmed Authorization value; remove the independent
redactor assertion. Keep native credential selection/optional-absence and
provider-free inspection tests. Update the credential-tracking description in
the [runtime guide](../../internal/llm-runtime.md) when removal lands.
**Catalog:** #63; prerequisite consumers are covered by §§2, 4–6.
## 2. Replace sanitizing serialization with direct encoding
**Remove / simplify.** The entire `core/redaction` package serves substitution:
raw/escaped secret expansion, longest-first matching, generic JSON decoding and
tree reconstruction, key-collision rejection, and YAML node/comment traversal.
Its JSON path marshals, decodes with `UseNumber`, walks, then marshals again;
its YAML path parses and re-encodes already serialized data. These passes run
even when the secret list is empty. Matching keys and fixed strings also causes
the schema corruption already recorded in catalog #9.
[Redactor](../../../internal/core/redaction/redaction.go#L18) ·
[Report YAML](../../../internal/core/reporting/report.go#L181)
Use direct JSON serialization in `RunDirectory` and `DiagnosticsWriter`, and
direct typed YAML serialization for ordinary reports. Write already-produced
YAML bytes without another parse/walk/encode pass. Remove secret parameters and
stored redactors. In Markdown, remove substitution and the redactor parameter
to `writeMatrix`, retaining the independent text/HTML/table escaping.
[Run artifacts](../../../internal/core/diagnostics/run_dir.go#L70) ·
[Interaction artifacts](../../../internal/framework/llm/diagnostics.go#L84) ·
[Markdown](../../../internal/core/reporting/markdown.go#L27)
Retain serialization errors and contextual write errors, valid artifact
envelopes, numeric and boolean types, optional fields, and raw `final_content`
as a string even when its contents are malformed model JSON. No nested-content
sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective
configuration rendering; only Audita's subsequent YAML transformation goes away.
Move useful scalar/schema and failed-write assertions from redactor tests to
the owning report/artifact tests before deleting the package. Update
[diagnostics/reporting internals](../../internal/diagnostics-reporting.md).
**Catalog:** #3 and #9; preserve #26's independent Markdown correction and
coordinate with #33's redundant-serialization work without resolving D5.
## 3. Save the acquired source snapshot without reconstructing it
**Simplify.** `WriteSourceTranscript` converts the acquired bytes to a string,
decodes a generic JSON tree, then sends it through the redactor's second
serialization/decode/walk/serialization sequence. The comment explicitly ties
this reconstruction to avoiding unredacted bytes. The CLI already has both the
acquired source bytes and parsed source, and calls this writer only for debug.
[Snapshot writer](../../../internal/core/diagnostics/run_dir.go#L96) ·
[Caller](../../../internal/cli/process.go#L54)
Write the acquired bytes directly to the source artifact, preserving the input
that was actually read. Keep parsed-source and normalized artifacts as separately
serialized typed views. Do not reopen the input file, which could have changed,
or preserve a second parser as a validation gate. Broader input-buffer lifetime
work remains catalog #31's concern.
Move source/normalized snapshot coverage out of the deleted redactor package.
Check source-byte fidelity at this boundary and decoded fields for the typed
views; preserve nonfatal debug-write behavior. Describe the source/parsed
distinction in the diagnostics guide when implemented.
**Catalog:** #31; depends on the direct writer changes in §2.
## 4. Remove secret plumbing from writer and preflight contracts
**Simplify.** Proposal generation discovers secrets through an optional client
interface and constructs an adapter-specific writer. Validation uses a lazy
`func() []string` closure and recreates a writer per interaction. Repair combines
two roles' secret lists for its writer. These are extra dependencies on model
clients for a filesystem concern.
[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L128) ·
[Validation setup](../../../internal/framework/runner/runner.go#L703) ·
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
[Repair writer](../../../internal/framework/runner/repair.go#L178)
Remove the discovery assertions, closure, combined-role helper, and secret
constructor arguments. Complete the already-proposed common interaction writer
contract without preserving these seams. Role owners still supply their payloads
and stage/section/proposal/attempt/validator coordinates; the writer returns
artifact paths and write errors. Retain the optional diagnostic-error callback
and its synchronization, which report nonfatal write failures.
Also remove `DebugInput.Secrets`, the corresponding `PrepareDebug` and
`NewRunDirectory` parameters, and secret captures in report-rendering closures.
Preflight readiness, initial/failed report construction, output-session cleanup,
and debug setup remain useful without any credential values.
[Preflight handoff](../../../internal/framework/preflight/preflight.go#L26) ·
[Debug preparation](../../../internal/core/diagnostics/output_session.go#L253) ·
[Report closures](../../../internal/cli/process_inputs.go#L95)
Adapt proposal, validation, repair, and preflight fixtures to the smaller
contracts. Preserve role attribution, distinct repair artifact paths, and
nonfatal write-failure coverage. Update the diagnostics guide with the resulting
ownership. **Catalog:** #43, coordinated with #42 and #48; no retention or
execution-policy change is implied.
## 5. Stop rewriting validator explanations sent to repair
**Remove / simplify.** Inside `repairCandidates`, `safeFindings` first constructs
domain feedback, then replaces substrings matching credentials from both repair
and validation clients. This runs for the initial rejection and subsequent
revalidation findings, independently of debug capture. It changes the next model
request, not just a saved diagnostic. A matching substring in otherwise useful
validator text therefore changes the corrective instruction.
[Feedback transformation](../../../internal/framework/runner/repair.go#L111) ·
[Repair request](../../../internal/framework/runner/repair.go#L124)
Use `repairFindings` directly at both points and remove the wrapper and its
secret dependency. Retain per-proposal filtering, validator/reason labels, and
the explicit unavailable-assessment representation: operational unavailability
is not an actionable editorial explanation. Preserve bounded coordination,
revalidation, withdrawal, and sibling independence.
[Domain feedback](../../../internal/framework/runner/feedback.go#L53)
Replace `TestRepairFeedbackRedactsValidationCredentials` with focused coverage
of actual repair feedback if existing feedback/request tests do not already
cover it. Assert explanation and proposal association through decoded request
data, without introducing an environment-driven passthrough contract or prompt
wording checks. Update the repair-flow description if needed.
**Catalog:** #64; coordinate removal of `repairSecrets` with §4.
## 6. Simplify CLI presentation while retaining typed failure ownership
**Remove / simplify.** `runProcess` invokes `Runtime.RedactError` for preparation
failures. `finishProcess` unconditionally discovers credentials and builds a
redactor, even without reports or debug; it rewrites warnings, final errors, and
announced report/debug paths. Removing these calls eliminates environment reads
and permits paths to identify their actual destinations.
[Preparation failure](../../../internal/cli/process.go#L37) ·
[Final presentation](../../../internal/cli/process_output.go#L17)
Present those existing errors and paths directly through CommandKit, preserving
typed causes, reporting modes, completion behavior, and exit classification.
Drop redaction-only parameters such as `finishProcess`'s `loaded` argument once
its only use disappears. Retain the required-output/nonfatal-debug distinction.
Simplify the path-announcement subprocess fixture; retain useful destination,
output-mode, and error assertions at their existing test owners.
There is also a distinct information-loss boundary: `classifiedFailure` reduces
native setup errors to a generic message, and `Failure.Error()` returns only
that message. Deleting substring replacement does not itself make missing
credential diagnostics actionable. Address catalog #25 by carrying the native
setup detail into the surfaced error, while retaining `Class`, `Reason`, cause
unwrapping, cancellation precedence, and operation attribution.
[Classification](../../../internal/framework/llm/adapter.go#L228) ·
[Failure wrapper](../../../internal/framework/structuredoutput/failure.go#L18)
Do not indiscriminately unwrap every cause into every report. The compact
reason/message projection in `processreport` serves the ordinary report schema,
and structured failure types drive recoverability and peer cancellation. Retain
those purposes; revise blanket comments/tests prohibiting cause presentation
where they enforce only the withdrawn guarantee. Test setup diagnostics for
actionable context and typed classification, not exact incidental wording.
[Report projection](../../../internal/framework/processreport/report.go#L224) ·
[Failure tests](../../../internal/framework/structuredoutput/failure_test.go#L10)
**Catalog:** #65 and #25; coordinate with #22–24 and #41 without changing their
classification or report-schema scope. Update operations and internal runtime
documentation alongside the implementation.
## 7. Retire obsolete assertions and documentation without losing coverage
**Remove / simplify.** Redaction assertions are mixed into tests with independent
value, and current documentation still describes the implemented sanitizer.
Remove pure substitution tests and synthetic credential-echo fixture modes used
only for them. In mixed tests, preserve native authentication, decoded request
and response evidence, schema metadata, repair path uniqueness, scalar types,
failure counts, and optional-artifact failure behavior.
| Location | Treatment when implementation lands |
| --- | --- |
| `core/redaction/redaction_test.go`, `llm/secrets_test.go` | Delete removed-mechanism tests; relocate useful writer assertions as described above. |
| `llm/diagnostics_test.go`, `proposal_generation/generate_test.go`, `runner/runner_test.go` | Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations. |
| `runner/repair_test.go` | Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary. |
| `core/reporting/report_test.go`, `llm/native_adapter_test.go` | Retain typed report encoding and actual native authentication; remove attached substitution assertions. |
| `cmd/audita/main_integration_test.go` | Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements. |
| `internal/testsupport/files.go` | Remove the already-unused secret scanners with catalog #61, rather than repurposing them. |
Review [operations](../../operations.md),
[diagnostics/reporting](../../internal/diagnostics-reporting.md),
[runtime](../../internal/llm-runtime.md),
[contributor recipes](../../internal/cli-config.md),
[component overview](../../internal/overview.md), and the
[migration guide](../../migrations/v2.md) when code lands. Remove broad redaction
claims and obsolete implementation instructions. Preserve native configuration
inspection/redaction documentation and the effective-config artifact name;
neither requires a naming migration merely because local postprocessing goes
away. Historical audit evidence stays intact, with revised remedies in the
catalog and assessment. Do not add a negative redaction policy.
**Catalog:** #3, #60, and #61, and supporting work for all findings above.
No new broad secret scanner or prompt-language change detector should replace
the removed tests.
## Retained responsibilities and boundaries
- **Native authentication and inspection:** PromptKit still owns selected
credentials and transport; ConfigKit/ckconfig/ckprompt own their normal
rendering and inspection. Keep Audita's `WriteEffective` call and standard
command wiring. Prepared snapshots contain messages and prompt metadata;
removing substitution does not require adding headers or environment dumps.
[Configuration rendering](../../../internal/cli/process_inputs.go#L118) ·
[Command wiring](../../../internal/cli/app_wiring.go#L81) ·
[Prepared snapshot](../../../internal/framework/llm/adapter.go#L309)
- **Ordinary filesystem behavior:** retain unique run-directory creation,
run-ID validation, stage-name-to-filename conversion, required-output staging,
destination checks, cleanup, and useful write errors. These prevent collisions
or incorrect writes. No extra private-root or hostile-filesystem mechanism
was found to remove in the debug writer. Catalog #4's previously proposed
private defaults lack a requirement under D1; leave current location/modes
unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain.
- **Domain and report behavior:** retain transcript parsing, validation,
feedback classification, typed operation failures, stable report identities,
Markdown escaping, and nonfatal optional diagnostics. Removing credential
substitution does not imply dropping these boundaries.
## Suggested implementation order
1. Replace serialization and source-snapshot reconstruction, removing writer and
report secret arguments together with their callers (§§2–4).
2. Remove repair-feedback and CLI substitution, then delete now-unreferenced
discovery state and the redaction package (§§1, 5–6). These are a coordinated
cleanup, not an interim design with empty secret lists or no-op methods.
3. Complete common interaction-writer ownership and actionable setup diagnostics
with catalog #43/#25. Update affected tests and canonical documentation in
each implementing change (§7).
Use targeted boundary tests and then the full suite for the cross-package code
change. Check decoded artifact fidelity, raw source preservation, native auth,
repair feedback, report types, and failure behavior. No new product decision is
needed for these D1 removals; D2–D7 remain separate and unresolved.