281 lines
17 KiB
Markdown
281 lines
17 KiB
Markdown
# Credential-redaction removal and simplification audit
|
||
|
||
**Reviewed:** 2026-11-15 at `a68a951`; application code remains at `c4ccd78`.
|
||
**Status:** audit complete; implementation pending.
|
||
|
||
This review follows the maintainer's [resolved D1 decision](assessment.md#resolved-d1-withdraw-application-wide-credential-redaction).
|
||
It identifies machinery to remove or simplify after withdrawal of the
|
||
application-wide credential-redaction requirement. Native library behavior,
|
||
authentication, diagnostic evidence, and ordinary output correctness remain
|
||
independently useful. The architecture policy has already been updated.
|
||
|
||
## Scope and verification
|
||
|
||
The review traced credential discovery into CLI, reports, debug writers,
|
||
proposal generation, validation, and repair. Knowledge-graph searches and caller
|
||
traces were checked against source, including interface assertions and closures
|
||
that the graph does not resolve completely. Tests were read to distinguish
|
||
redaction-only expectations from authentication, serialization, artifact
|
||
association, and failure behavior. Documentation and previous audit remedies
|
||
were reviewed for assumptions that would reintroduce the withdrawn requirement.
|
||
|
||
This was a static review: no code changes, live provider calls, credential-value
|
||
inspection, test execution, or performance measurements. Removed passes and
|
||
allocations below are visible in the code; their production cost is unmeasured.
|
||
|
||
## 1. Remove credential discovery and tracking used only for substitution
|
||
|
||
**Remove.** `Runtime.ConfiguredSecrets` enumerates every captured profile and
|
||
backend, caches environment-variable names with `sync.Once`, and reads their
|
||
values whenever called. It includes raw and trimmed forms and deduplicates
|
||
additional names. Discovery uses `context.WithoutCancel`. `RoleClient` separately
|
||
collects prepared credential names under a mutex, solely to supply this method.
|
||
None of this selects or supplies authentication to a provider.
|
||
[Discovery and reads](../../../internal/framework/llm/secrets.go#L11) ·
|
||
[Runtime cache](../../../internal/framework/llm/runtime.go#L22) ·
|
||
[Prepared-name tracking](../../../internal/framework/llm/adapter.go#L153) ·
|
||
[Role accessor](../../../internal/framework/llm/adapter.go#L314)
|
||
|
||
Consumers are CLI report/debug preparation and finalization, the error redactor,
|
||
proposal and validation writer setup, and combined repair/validation secret
|
||
collection. Remove `secrets.go`, the runtime cache fields, role credential fields
|
||
and locking, accessor methods, and secret-source assertions after updating those
|
||
consumers. Keep `OnPrepared` request capture, terminal-event synchronization,
|
||
call IDs, source-path tracking, and native engine execution; those have separate
|
||
consumers. Do not leave empty secret-provider methods or no-op redactors.
|
||
[CLI consumers](../../../internal/cli/process_inputs.go#L90) ·
|
||
[Finalization](../../../internal/cli/process_output.go#L17) ·
|
||
[Repair collection](../../../internal/framework/runner/repair.go#L282)
|
||
|
||
Delete `TestConfiguredSecretsIncludesUnselectedBackends`. In
|
||
`TestRoleClientUsesNativeTargetAndRedactsCredential`, retain the assertion that
|
||
PromptKit sends the selected, trimmed Authorization value; remove the independent
|
||
redactor assertion. Keep native credential selection/optional-absence and
|
||
provider-free inspection tests. Update the credential-tracking description in
|
||
the [runtime guide](../../internal/llm-runtime.md) when removal lands.
|
||
**Catalog:** #63; prerequisite consumers are covered by §§2, 4–6.
|
||
|
||
## 2. Replace sanitizing serialization with direct encoding
|
||
|
||
**Remove / simplify.** The entire `core/redaction` package serves substitution:
|
||
raw/escaped secret expansion, longest-first matching, generic JSON decoding and
|
||
tree reconstruction, key-collision rejection, and YAML node/comment traversal.
|
||
Its JSON path marshals, decodes with `UseNumber`, walks, then marshals again;
|
||
its YAML path parses and re-encodes already serialized data. These passes run
|
||
even when the secret list is empty. Matching keys and fixed strings also causes
|
||
the schema corruption already recorded in catalog #9.
|
||
[Redactor](../../../internal/core/redaction/redaction.go#L18) ·
|
||
[Report YAML](../../../internal/core/reporting/report.go#L181)
|
||
|
||
Use direct JSON serialization in `RunDirectory` and `DiagnosticsWriter`, and
|
||
direct typed YAML serialization for ordinary reports. Write already-produced
|
||
YAML bytes without another parse/walk/encode pass. Remove secret parameters and
|
||
stored redactors. In Markdown, remove substitution and the redactor parameter
|
||
to `writeMatrix`, retaining the independent text/HTML/table escaping.
|
||
[Run artifacts](../../../internal/core/diagnostics/run_dir.go#L70) ·
|
||
[Interaction artifacts](../../../internal/framework/llm/diagnostics.go#L84) ·
|
||
[Markdown](../../../internal/core/reporting/markdown.go#L27)
|
||
|
||
Retain serialization errors and contextual write errors, valid artifact
|
||
envelopes, numeric and boolean types, optional fields, and raw `final_content`
|
||
as a string even when its contents are malformed model JSON. No nested-content
|
||
sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective
|
||
configuration rendering; only Audita's subsequent YAML transformation goes away.
|
||
|
||
Move useful scalar/schema and failed-write assertions from redactor tests to
|
||
the owning report/artifact tests before deleting the package. Update
|
||
[diagnostics/reporting internals](../../internal/diagnostics-reporting.md).
|
||
**Catalog:** #3 and #9; preserve #26's independent Markdown correction and
|
||
coordinate with #33's redundant-serialization work without resolving D5.
|
||
|
||
## 3. Save the acquired source snapshot without reconstructing it
|
||
|
||
**Simplify.** `WriteSourceTranscript` converts the acquired bytes to a string,
|
||
decodes a generic JSON tree, then sends it through the redactor's second
|
||
serialization/decode/walk/serialization sequence. The comment explicitly ties
|
||
this reconstruction to avoiding unredacted bytes. The CLI already has both the
|
||
acquired source bytes and parsed source, and calls this writer only for debug.
|
||
[Snapshot writer](../../../internal/core/diagnostics/run_dir.go#L96) ·
|
||
[Caller](../../../internal/cli/process.go#L54)
|
||
|
||
Write the acquired bytes directly to the source artifact, preserving the input
|
||
that was actually read. Keep parsed-source and normalized artifacts as separately
|
||
serialized typed views. Do not reopen the input file, which could have changed,
|
||
or preserve a second parser as a validation gate. Broader input-buffer lifetime
|
||
work remains catalog #31's concern.
|
||
|
||
Move source/normalized snapshot coverage out of the deleted redactor package.
|
||
Check source-byte fidelity at this boundary and decoded fields for the typed
|
||
views; preserve nonfatal debug-write behavior. Describe the source/parsed
|
||
distinction in the diagnostics guide when implemented.
|
||
**Catalog:** #31; depends on the direct writer changes in §2.
|
||
|
||
## 4. Remove secret plumbing from writer and preflight contracts
|
||
|
||
**Simplify.** Proposal generation discovers secrets through an optional client
|
||
interface and constructs an adapter-specific writer. Validation uses a lazy
|
||
`func() []string` closure and recreates a writer per interaction. Repair combines
|
||
two roles' secret lists for its writer. These are extra dependencies on model
|
||
clients for a filesystem concern.
|
||
[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L128) ·
|
||
[Validation setup](../../../internal/framework/runner/runner.go#L703) ·
|
||
[Validation adapter](../../../internal/framework/runner/runner.go#L994) ·
|
||
[Repair writer](../../../internal/framework/runner/repair.go#L178)
|
||
|
||
Remove the discovery assertions, closure, combined-role helper, and secret
|
||
constructor arguments. Complete the already-proposed common interaction writer
|
||
contract without preserving these seams. Role owners still supply their payloads
|
||
and stage/section/proposal/attempt/validator coordinates; the writer returns
|
||
artifact paths and write errors. Retain the optional diagnostic-error callback
|
||
and its synchronization, which report nonfatal write failures.
|
||
|
||
Also remove `DebugInput.Secrets`, the corresponding `PrepareDebug` and
|
||
`NewRunDirectory` parameters, and secret captures in report-rendering closures.
|
||
Preflight readiness, initial/failed report construction, output-session cleanup,
|
||
and debug setup remain useful without any credential values.
|
||
[Preflight handoff](../../../internal/framework/preflight/preflight.go#L26) ·
|
||
[Debug preparation](../../../internal/core/diagnostics/output_session.go#L253) ·
|
||
[Report closures](../../../internal/cli/process_inputs.go#L95)
|
||
|
||
Adapt proposal, validation, repair, and preflight fixtures to the smaller
|
||
contracts. Preserve role attribution, distinct repair artifact paths, and
|
||
nonfatal write-failure coverage. Update the diagnostics guide with the resulting
|
||
ownership. **Catalog:** #43, coordinated with #42 and #48; no retention or
|
||
execution-policy change is implied.
|
||
|
||
## 5. Stop rewriting validator explanations sent to repair
|
||
|
||
**Remove / simplify.** Inside `repairCandidates`, `safeFindings` first constructs
|
||
domain feedback, then replaces substrings matching credentials from both repair
|
||
and validation clients. This runs for the initial rejection and subsequent
|
||
revalidation findings, independently of debug capture. It changes the next model
|
||
request, not just a saved diagnostic. A matching substring in otherwise useful
|
||
validator text therefore changes the corrective instruction.
|
||
[Feedback transformation](../../../internal/framework/runner/repair.go#L111) ·
|
||
[Repair request](../../../internal/framework/runner/repair.go#L124)
|
||
|
||
Use `repairFindings` directly at both points and remove the wrapper and its
|
||
secret dependency. Retain per-proposal filtering, validator/reason labels, and
|
||
the explicit unavailable-assessment representation: operational unavailability
|
||
is not an actionable editorial explanation. Preserve bounded coordination,
|
||
revalidation, withdrawal, and sibling independence.
|
||
[Domain feedback](../../../internal/framework/runner/feedback.go#L53)
|
||
|
||
Replace `TestRepairFeedbackRedactsValidationCredentials` with focused coverage
|
||
of actual repair feedback if existing feedback/request tests do not already
|
||
cover it. Assert explanation and proposal association through decoded request
|
||
data, without introducing an environment-driven passthrough contract or prompt
|
||
wording checks. Update the repair-flow description if needed.
|
||
**Catalog:** #64; coordinate removal of `repairSecrets` with §4.
|
||
|
||
## 6. Simplify CLI presentation while retaining typed failure ownership
|
||
|
||
**Remove / simplify.** `runProcess` invokes `Runtime.RedactError` for preparation
|
||
failures. `finishProcess` unconditionally discovers credentials and builds a
|
||
redactor, even without reports or debug; it rewrites warnings, final errors, and
|
||
announced report/debug paths. Removing these calls eliminates environment reads
|
||
and permits paths to identify their actual destinations.
|
||
[Preparation failure](../../../internal/cli/process.go#L37) ·
|
||
[Final presentation](../../../internal/cli/process_output.go#L17)
|
||
|
||
Present those existing errors and paths directly through CommandKit, preserving
|
||
typed causes, reporting modes, completion behavior, and exit classification.
|
||
Drop redaction-only parameters such as `finishProcess`'s `loaded` argument once
|
||
its only use disappears. Retain the required-output/nonfatal-debug distinction.
|
||
Simplify the path-announcement subprocess fixture; retain useful destination,
|
||
output-mode, and error assertions at their existing test owners.
|
||
|
||
There is also a distinct information-loss boundary: `classifiedFailure` reduces
|
||
native setup errors to a generic message, and `Failure.Error()` returns only
|
||
that message. Deleting substring replacement does not itself make missing
|
||
credential diagnostics actionable. Address catalog #25 by carrying the native
|
||
setup detail into the surfaced error, while retaining `Class`, `Reason`, cause
|
||
unwrapping, cancellation precedence, and operation attribution.
|
||
[Classification](../../../internal/framework/llm/adapter.go#L228) ·
|
||
[Failure wrapper](../../../internal/framework/structuredoutput/failure.go#L18)
|
||
|
||
Do not indiscriminately unwrap every cause into every report. The compact
|
||
reason/message projection in `processreport` serves the ordinary report schema,
|
||
and structured failure types drive recoverability and peer cancellation. Retain
|
||
those purposes; revise blanket comments/tests prohibiting cause presentation
|
||
where they enforce only the withdrawn guarantee. Test setup diagnostics for
|
||
actionable context and typed classification, not exact incidental wording.
|
||
[Report projection](../../../internal/framework/processreport/report.go#L224) ·
|
||
[Failure tests](../../../internal/framework/structuredoutput/failure_test.go#L10)
|
||
|
||
**Catalog:** #65 and #25; coordinate with #22–24 and #41 without changing their
|
||
classification or report-schema scope. Update operations and internal runtime
|
||
documentation alongside the implementation.
|
||
|
||
## 7. Retire obsolete assertions and documentation without losing coverage
|
||
|
||
**Remove / simplify.** Redaction assertions are mixed into tests with independent
|
||
value, and current documentation still describes the implemented sanitizer.
|
||
Remove pure substitution tests and synthetic credential-echo fixture modes used
|
||
only for them. In mixed tests, preserve native authentication, decoded request
|
||
and response evidence, schema metadata, repair path uniqueness, scalar types,
|
||
failure counts, and optional-artifact failure behavior.
|
||
|
||
| Location | Treatment when implementation lands |
|
||
| --- | --- |
|
||
| `core/redaction/redaction_test.go`, `llm/secrets_test.go` | Delete removed-mechanism tests; relocate useful writer assertions as described above. |
|
||
| `llm/diagnostics_test.go`, `proposal_generation/generate_test.go`, `runner/runner_test.go` | Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations. |
|
||
| `runner/repair_test.go` | Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary. |
|
||
| `core/reporting/report_test.go`, `llm/native_adapter_test.go` | Retain typed report encoding and actual native authentication; remove attached substitution assertions. |
|
||
| `cmd/audita/main_integration_test.go` | Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements. |
|
||
| `internal/testsupport/files.go` | Remove the already-unused secret scanners with catalog #61, rather than repurposing them. |
|
||
|
||
Review [operations](../../operations.md),
|
||
[diagnostics/reporting](../../internal/diagnostics-reporting.md),
|
||
[runtime](../../internal/llm-runtime.md),
|
||
[contributor recipes](../../internal/cli-config.md),
|
||
[component overview](../../internal/overview.md), and the
|
||
[migration guide](../../migrations/v2.md) when code lands. Remove broad redaction
|
||
claims and obsolete implementation instructions. Preserve native configuration
|
||
inspection/redaction documentation and the effective-config artifact name;
|
||
neither requires a naming migration merely because local postprocessing goes
|
||
away. Historical audit evidence stays intact, with revised remedies in the
|
||
catalog and assessment. Do not add a negative redaction policy.
|
||
|
||
**Catalog:** #3, #60, and #61, and supporting work for all findings above.
|
||
No new broad secret scanner or prompt-language change detector should replace
|
||
the removed tests.
|
||
|
||
## Retained responsibilities and boundaries
|
||
|
||
- **Native authentication and inspection:** PromptKit still owns selected
|
||
credentials and transport; ConfigKit/ckconfig/ckprompt own their normal
|
||
rendering and inspection. Keep Audita's `WriteEffective` call and standard
|
||
command wiring. Prepared snapshots contain messages and prompt metadata;
|
||
removing substitution does not require adding headers or environment dumps.
|
||
[Configuration rendering](../../../internal/cli/process_inputs.go#L118) ·
|
||
[Command wiring](../../../internal/cli/app_wiring.go#L81) ·
|
||
[Prepared snapshot](../../../internal/framework/llm/adapter.go#L309)
|
||
- **Ordinary filesystem behavior:** retain unique run-directory creation,
|
||
run-ID validation, stage-name-to-filename conversion, required-output staging,
|
||
destination checks, cleanup, and useful write errors. These prevent collisions
|
||
or incorrect writes. No extra private-root or hostile-filesystem mechanism
|
||
was found to remove in the debug writer. Catalog #4's previously proposed
|
||
private defaults lack a requirement under D1; leave current location/modes
|
||
unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain.
|
||
- **Domain and report behavior:** retain transcript parsing, validation,
|
||
feedback classification, typed operation failures, stable report identities,
|
||
Markdown escaping, and nonfatal optional diagnostics. Removing credential
|
||
substitution does not imply dropping these boundaries.
|
||
|
||
## Suggested implementation order
|
||
|
||
1. Replace serialization and source-snapshot reconstruction, removing writer and
|
||
report secret arguments together with their callers (§§2–4).
|
||
2. Remove repair-feedback and CLI substitution, then delete now-unreferenced
|
||
discovery state and the redaction package (§§1, 5–6). These are a coordinated
|
||
cleanup, not an interim design with empty secret lists or no-op methods.
|
||
3. Complete common interaction-writer ownership and actionable setup diagnostics
|
||
with catalog #43/#25. Update affected tests and canonical documentation in
|
||
each implementing change (§7).
|
||
|
||
Use targeted boundary tests and then the full suite for the cross-package code
|
||
change. Check decoded artifact fidelity, raw source preservation, native auth,
|
||
repair feedback, report types, and failure behavior. No new product decision is
|
||
needed for these D1 removals; D2–D7 remain separate and unresolved.
|