diff --git a/docs/roadmap/audit/14-redaction.md b/docs/roadmap/audit/14-redaction.md new file mode 100644 index 0000000..172de0c --- /dev/null +++ b/docs/roadmap/audit/14-redaction.md @@ -0,0 +1,280 @@ +# Credential-redaction removal and simplification audit + +**Reviewed:** 2026-11-15 at `a68a951`; application code remains at `c4ccd78`. +**Status:** audit complete; implementation pending. + +This review follows the maintainer's [resolved D1 decision](assessment.md#resolved-d1-withdraw-application-wide-credential-redaction). +It identifies machinery to remove or simplify after withdrawal of the +application-wide credential-redaction requirement. Native library behavior, +authentication, diagnostic evidence, and ordinary output correctness remain +independently useful. The architecture policy has already been updated. + +## Scope and verification + +The review traced credential discovery into CLI, reports, debug writers, +proposal generation, validation, and repair. Knowledge-graph searches and caller +traces were checked against source, including interface assertions and closures +that the graph does not resolve completely. Tests were read to distinguish +redaction-only expectations from authentication, serialization, artifact +association, and failure behavior. Documentation and previous audit remedies +were reviewed for assumptions that would reintroduce the withdrawn requirement. + +This was a static review: no code changes, live provider calls, credential-value +inspection, test execution, or performance measurements. Removed passes and +allocations below are visible in the code; their production cost is unmeasured. + +## 1. Remove credential discovery and tracking used only for substitution + +**Remove.** `Runtime.ConfiguredSecrets` enumerates every captured profile and +backend, caches environment-variable names with `sync.Once`, and reads their +values whenever called. It includes raw and trimmed forms and deduplicates +additional names. Discovery uses `context.WithoutCancel`. `RoleClient` separately +collects prepared credential names under a mutex, solely to supply this method. +None of this selects or supplies authentication to a provider. +[Discovery and reads](../../../internal/framework/llm/secrets.go#L11) · +[Runtime cache](../../../internal/framework/llm/runtime.go#L22) · +[Prepared-name tracking](../../../internal/framework/llm/adapter.go#L153) · +[Role accessor](../../../internal/framework/llm/adapter.go#L314) + +Consumers are CLI report/debug preparation and finalization, the error redactor, +proposal and validation writer setup, and combined repair/validation secret +collection. Remove `secrets.go`, the runtime cache fields, role credential fields +and locking, accessor methods, and secret-source assertions after updating those +consumers. Keep `OnPrepared` request capture, terminal-event synchronization, +call IDs, source-path tracking, and native engine execution; those have separate +consumers. Do not leave empty secret-provider methods or no-op redactors. +[CLI consumers](../../../internal/cli/process_inputs.go#L90) · +[Finalization](../../../internal/cli/process_output.go#L17) · +[Repair collection](../../../internal/framework/runner/repair.go#L282) + +Delete `TestConfiguredSecretsIncludesUnselectedBackends`. In +`TestRoleClientUsesNativeTargetAndRedactsCredential`, retain the assertion that +PromptKit sends the selected, trimmed Authorization value; remove the independent +redactor assertion. Keep native credential selection/optional-absence and +provider-free inspection tests. Update the credential-tracking description in +the [runtime guide](../../internal/llm-runtime.md) when removal lands. +**Catalog:** #63; prerequisite consumers are covered by §§2, 4–6. + +## 2. Replace sanitizing serialization with direct encoding + +**Remove / simplify.** The entire `core/redaction` package serves substitution: +raw/escaped secret expansion, longest-first matching, generic JSON decoding and +tree reconstruction, key-collision rejection, and YAML node/comment traversal. +Its JSON path marshals, decodes with `UseNumber`, walks, then marshals again; +its YAML path parses and re-encodes already serialized data. These passes run +even when the secret list is empty. Matching keys and fixed strings also causes +the schema corruption already recorded in catalog #9. +[Redactor](../../../internal/core/redaction/redaction.go#L18) · +[Report YAML](../../../internal/core/reporting/report.go#L181) + +Use direct JSON serialization in `RunDirectory` and `DiagnosticsWriter`, and +direct typed YAML serialization for ordinary reports. Write already-produced +YAML bytes without another parse/walk/encode pass. Remove secret parameters and +stored redactors. In Markdown, remove substitution and the redactor parameter +to `writeMatrix`, retaining the independent text/HTML/table escaping. +[Run artifacts](../../../internal/core/diagnostics/run_dir.go#L70) · +[Interaction artifacts](../../../internal/framework/llm/diagnostics.go#L84) · +[Markdown](../../../internal/core/reporting/markdown.go#L27) + +Retain serialization errors and contextual write errors, valid artifact +envelopes, numeric and boolean types, optional fields, and raw `final_content` +as a string even when its contents are malformed model JSON. No nested-content +sanitizer or response-discard fallback is needed. Keep ConfigKit's own effective +configuration rendering; only Audita's subsequent YAML transformation goes away. + +Move useful scalar/schema and failed-write assertions from redactor tests to +the owning report/artifact tests before deleting the package. Update +[diagnostics/reporting internals](../../internal/diagnostics-reporting.md). +**Catalog:** #3 and #9; preserve #26's independent Markdown correction and +coordinate with #33's redundant-serialization work without resolving D5. + +## 3. Save the acquired source snapshot without reconstructing it + +**Simplify.** `WriteSourceTranscript` converts the acquired bytes to a string, +decodes a generic JSON tree, then sends it through the redactor's second +serialization/decode/walk/serialization sequence. The comment explicitly ties +this reconstruction to avoiding unredacted bytes. The CLI already has both the +acquired source bytes and parsed source, and calls this writer only for debug. +[Snapshot writer](../../../internal/core/diagnostics/run_dir.go#L96) · +[Caller](../../../internal/cli/process.go#L54) + +Write the acquired bytes directly to the source artifact, preserving the input +that was actually read. Keep parsed-source and normalized artifacts as separately +serialized typed views. Do not reopen the input file, which could have changed, +or preserve a second parser as a validation gate. Broader input-buffer lifetime +work remains catalog #31's concern. + +Move source/normalized snapshot coverage out of the deleted redactor package. +Check source-byte fidelity at this boundary and decoded fields for the typed +views; preserve nonfatal debug-write behavior. Describe the source/parsed +distinction in the diagnostics guide when implemented. +**Catalog:** #31; depends on the direct writer changes in §2. + +## 4. Remove secret plumbing from writer and preflight contracts + +**Simplify.** Proposal generation discovers secrets through an optional client +interface and constructs an adapter-specific writer. Validation uses a lazy +`func() []string` closure and recreates a writer per interaction. Repair combines +two roles' secret lists for its writer. These are extra dependencies on model +clients for a filesystem concern. +[Proposal fallback](../../../internal/framework/proposal_generation/generate.go#L128) · +[Validation setup](../../../internal/framework/runner/runner.go#L703) · +[Validation adapter](../../../internal/framework/runner/runner.go#L994) · +[Repair writer](../../../internal/framework/runner/repair.go#L178) + +Remove the discovery assertions, closure, combined-role helper, and secret +constructor arguments. Complete the already-proposed common interaction writer +contract without preserving these seams. Role owners still supply their payloads +and stage/section/proposal/attempt/validator coordinates; the writer returns +artifact paths and write errors. Retain the optional diagnostic-error callback +and its synchronization, which report nonfatal write failures. + +Also remove `DebugInput.Secrets`, the corresponding `PrepareDebug` and +`NewRunDirectory` parameters, and secret captures in report-rendering closures. +Preflight readiness, initial/failed report construction, output-session cleanup, +and debug setup remain useful without any credential values. +[Preflight handoff](../../../internal/framework/preflight/preflight.go#L26) · +[Debug preparation](../../../internal/core/diagnostics/output_session.go#L253) · +[Report closures](../../../internal/cli/process_inputs.go#L95) + +Adapt proposal, validation, repair, and preflight fixtures to the smaller +contracts. Preserve role attribution, distinct repair artifact paths, and +nonfatal write-failure coverage. Update the diagnostics guide with the resulting +ownership. **Catalog:** #43, coordinated with #42 and #48; no retention or +execution-policy change is implied. + +## 5. Stop rewriting validator explanations sent to repair + +**Remove / simplify.** Inside `repairCandidates`, `safeFindings` first constructs +domain feedback, then replaces substrings matching credentials from both repair +and validation clients. This runs for the initial rejection and subsequent +revalidation findings, independently of debug capture. It changes the next model +request, not just a saved diagnostic. A matching substring in otherwise useful +validator text therefore changes the corrective instruction. +[Feedback transformation](../../../internal/framework/runner/repair.go#L111) · +[Repair request](../../../internal/framework/runner/repair.go#L124) + +Use `repairFindings` directly at both points and remove the wrapper and its +secret dependency. Retain per-proposal filtering, validator/reason labels, and +the explicit unavailable-assessment representation: operational unavailability +is not an actionable editorial explanation. Preserve bounded coordination, +revalidation, withdrawal, and sibling independence. +[Domain feedback](../../../internal/framework/runner/feedback.go#L53) + +Replace `TestRepairFeedbackRedactsValidationCredentials` with focused coverage +of actual repair feedback if existing feedback/request tests do not already +cover it. Assert explanation and proposal association through decoded request +data, without introducing an environment-driven passthrough contract or prompt +wording checks. Update the repair-flow description if needed. +**Catalog:** #64; coordinate removal of `repairSecrets` with §4. + +## 6. Simplify CLI presentation while retaining typed failure ownership + +**Remove / simplify.** `runProcess` invokes `Runtime.RedactError` for preparation +failures. `finishProcess` unconditionally discovers credentials and builds a +redactor, even without reports or debug; it rewrites warnings, final errors, and +announced report/debug paths. Removing these calls eliminates environment reads +and permits paths to identify their actual destinations. +[Preparation failure](../../../internal/cli/process.go#L37) · +[Final presentation](../../../internal/cli/process_output.go#L17) + +Present those existing errors and paths directly through CommandKit, preserving +typed causes, reporting modes, completion behavior, and exit classification. +Drop redaction-only parameters such as `finishProcess`'s `loaded` argument once +its only use disappears. Retain the required-output/nonfatal-debug distinction. +Simplify the path-announcement subprocess fixture; retain useful destination, +output-mode, and error assertions at their existing test owners. + +There is also a distinct information-loss boundary: `classifiedFailure` reduces +native setup errors to a generic message, and `Failure.Error()` returns only +that message. Deleting substring replacement does not itself make missing +credential diagnostics actionable. Address catalog #25 by carrying the native +setup detail into the surfaced error, while retaining `Class`, `Reason`, cause +unwrapping, cancellation precedence, and operation attribution. +[Classification](../../../internal/framework/llm/adapter.go#L228) · +[Failure wrapper](../../../internal/framework/structuredoutput/failure.go#L18) + +Do not indiscriminately unwrap every cause into every report. The compact +reason/message projection in `processreport` serves the ordinary report schema, +and structured failure types drive recoverability and peer cancellation. Retain +those purposes; revise blanket comments/tests prohibiting cause presentation +where they enforce only the withdrawn guarantee. Test setup diagnostics for +actionable context and typed classification, not exact incidental wording. +[Report projection](../../../internal/framework/processreport/report.go#L224) · +[Failure tests](../../../internal/framework/structuredoutput/failure_test.go#L10) + +**Catalog:** #65 and #25; coordinate with #22–24 and #41 without changing their +classification or report-schema scope. Update operations and internal runtime +documentation alongside the implementation. + +## 7. Retire obsolete assertions and documentation without losing coverage + +**Remove / simplify.** Redaction assertions are mixed into tests with independent +value, and current documentation still describes the implemented sanitizer. +Remove pure substitution tests and synthetic credential-echo fixture modes used +only for them. In mixed tests, preserve native authentication, decoded request +and response evidence, schema metadata, repair path uniqueness, scalar types, +failure counts, and optional-artifact failure behavior. + +| Location | Treatment when implementation lands | +| --- | --- | +| `core/redaction/redaction_test.go`, `llm/secrets_test.go` | Delete removed-mechanism tests; relocate useful writer assertions as described above. | +| `llm/diagnostics_test.go`, `proposal_generation/generate_test.go`, `runner/runner_test.go` | Retain artifact contents/associations and diagnostic failure coverage; remove secret providers and masking expectations. | +| `runner/repair_test.go` | Keep unique branch/attempt artifacts and repair behavior; remove secret plumbing and rewrite the feedback test at its domain boundary. | +| `core/reporting/report_test.go`, `llm/native_adapter_test.go` | Retain typed report encoding and actual native authentication; remove attached substitution assertions. | +| `cmd/audita/main_integration_test.go` | Retire the padded-credential masking scenario and secret-based path fixture; preserve representative malformed-response evidence, outcomes, and destination announcements. | +| `internal/testsupport/files.go` | Remove the already-unused secret scanners with catalog #61, rather than repurposing them. | + +Review [operations](../../operations.md), +[diagnostics/reporting](../../internal/diagnostics-reporting.md), +[runtime](../../internal/llm-runtime.md), +[contributor recipes](../../internal/cli-config.md), +[component overview](../../internal/overview.md), and the +[migration guide](../../migrations/v2.md) when code lands. Remove broad redaction +claims and obsolete implementation instructions. Preserve native configuration +inspection/redaction documentation and the effective-config artifact name; +neither requires a naming migration merely because local postprocessing goes +away. Historical audit evidence stays intact, with revised remedies in the +catalog and assessment. Do not add a negative redaction policy. + +**Catalog:** #3, #60, and #61, and supporting work for all findings above. +No new broad secret scanner or prompt-language change detector should replace +the removed tests. + +## Retained responsibilities and boundaries + +- **Native authentication and inspection:** PromptKit still owns selected + credentials and transport; ConfigKit/ckconfig/ckprompt own their normal + rendering and inspection. Keep Audita's `WriteEffective` call and standard + command wiring. Prepared snapshots contain messages and prompt metadata; + removing substitution does not require adding headers or environment dumps. + [Configuration rendering](../../../internal/cli/process_inputs.go#L118) · + [Command wiring](../../../internal/cli/app_wiring.go#L81) · + [Prepared snapshot](../../../internal/framework/llm/adapter.go#L309) +- **Ordinary filesystem behavior:** retain unique run-directory creation, + run-ID validation, stage-name-to-filename conversion, required-output staging, + destination checks, cleanup, and useful write errors. These prevent collisions + or incorrect writes. No extra private-root or hostile-filesystem mechanism + was found to remove in the debug writer. Catalog #4's previously proposed + private defaults lack a requirement under D1; leave current location/modes + unchanged in this cleanup. Catalog #1's ordinary wrong-file defects remain. +- **Domain and report behavior:** retain transcript parsing, validation, + feedback classification, typed operation failures, stable report identities, + Markdown escaping, and nonfatal optional diagnostics. Removing credential + substitution does not imply dropping these boundaries. + +## Suggested implementation order + +1. Replace serialization and source-snapshot reconstruction, removing writer and + report secret arguments together with their callers (§§2–4). +2. Remove repair-feedback and CLI substitution, then delete now-unreferenced + discovery state and the redaction package (§§1, 5–6). These are a coordinated + cleanup, not an interim design with empty secret lists or no-op methods. +3. Complete common interaction-writer ownership and actionable setup diagnostics + with catalog #43/#25. Update affected tests and canonical documentation in + each implementing change (§7). + +Use targeted boundary tests and then the full suite for the cross-package code +change. Check decoded artifact fidelity, raw source preservation, native auth, +repair feedback, report types, and failure behavior. No new product decision is +needed for these D1 removals; D2–D7 remain separate and unresolved. diff --git a/docs/roadmap/audit/assessment.md b/docs/roadmap/audit/assessment.md index 776cbf4..b262d8c 100644 --- a/docs/roadmap/audit/assessment.md +++ b/docs/roadmap/audit/assessment.md @@ -2,14 +2,16 @@ **Reviewed baseline:** `faba8ab` (2026-11-15). **Status:** assessment complete; remediation not implemented. D1 was resolved on 2026-11-15 and the architecture -policy updated; its implementation cleanup awaits the focused audit below. +policy updated; the focused cleanup audit is complete and implementation remains +pending. This document owns classification, recommended scope, and pending decisions for the [catalog](catalog.md). The source audits retain their evidence; their suggested remedies are not approved requirements. -The catalog covers 103 source findings in 62 entries. Every catalog entry is +The catalog covers 110 source findings in 65 entries. Every catalog entry is assessed below. The application implementation remains at the audited baseline -`c4ccd78`; later documentation includes the D1 policy revision. This review used +`c4ccd78`; later documentation includes the D1 policy revision and the focused +[removal audit](14-redaction.md), reviewed at `a68a951`. This review used the source audits, current contracts and ADRs, and targeted implementation inspection. It did not independently rerun all audit probes, benchmarks, or the test suite. Reproductions reported by the audits remain attributed to them; @@ -68,8 +70,8 @@ behavior from proposed extensions. | --- | --- | --- | --- | | [1. Bind output writes to the checked destinations](catalog.md#1-bind-output-writes-to-the-checked-destinations) | C, A | Ready | The wrong-file reproduction and repeated report-path resolution substantiate application defects. Resolve paths with filesystem semantics, carry checked parent/destination information through publication, and share staging ownership. Preserve symlinks and intentional in-place replacement. No hostile-directory race-proof guarantee, cross-file transaction, or custom filesystem machinery is selected. | | [2. Keep model requests within the configured transport boundary](catalog.md#2-keep-model-requests-within-the-configured-transport-boundary) | — | Closed: WONTFIX | Accepted Go HTTP redirect and credential-forwarding behavior. The source audit's proposed redirect restrictions do not authorize implementation. | -| [3. Protect credentials inside nested debug responses](catalog.md#3-protect-credentials-inside-nested-debug-responses) | A, M, T | Superseded remedy; D1 cleanup audit | The encoded-secret observation tested the withdrawn redaction guarantee. Do not implement nested sanitization or discard malformed response bodies for that purpose. Audit and remove local credential discovery, sanitization, and their supporting abstractions and tests; retain useful diagnostic evidence and native upstream behavior. | -| [4. Make debug artifact permissions private by default](catalog.md#4-make-debug-artifact-permissions-private-by-default) | M; P | Reassess under D1 scope | The permission observation is factual, but hostile local filesystem access is outside the maintainer's threat model and operators own artifact access. The prior mandatory private-root/security remediation is withdrawn. The follow-up audit should distinguish ordinary file-creation mechanics from machinery justified only by local-adversary protection; any remaining default-location or permission proposal needs its own justification. | +| [3. Protect credentials inside nested debug responses](catalog.md#3-protect-credentials-inside-nested-debug-responses) | A, M, T | Ready under resolved D1; original remedy superseded | The encoded-secret observation tested the withdrawn guarantee. The focused audit confirms removal of the local substitution package in favor of direct serialization, plus retirement of masking-only tests/docs. Keep malformed response evidence and native behavior; no nested sanitizer or response-omission policy. Discovery and repair/CLI consumers are separately tracked in #63–65. | +| [4. Make debug artifact permissions private by default](catalog.md#4-make-debug-artifact-permissions-private-by-default) | M | No remediation under D1 scope | The permission observation is factual, but hostile local filesystem access is outside the selected threat model. The focused audit found ordinary directory creation and filename handling to retain, not a redundant private-root mechanism. Leave defaults unchanged in D1 cleanup; any future location or permission proposal needs an independent justification. This does not close #1’s wrong-file correctness work. | | [5. Define the trust boundary for discovered configuration](catalog.md#5-define-the-trust-boundary-for-discovered-configuration) | M; P | Ready for documentation; extension excluded | Automatic companions are intentional configuration composition, not a discovery bug. Explain that the searched locations, selected bundle, imports, and companions are trusted configuration, and keep uploaded transcripts separate. Disabling discovery or adding a new isolation mode is outside this remediation; no ConfigKit replacement loader. | | [6. Make blocking transcript input and delivery cancellation-aware](catalog.md#6-make-blocking-transcript-input-and-delivery-cancellation-aware) | C; — | Ready in part; WONTFIX in part | Honor cancellation before regular-file publication and when blocked delivery returns; produce failure status even if bytes have escaped. Do not promise interruption of already-blocked standard I/O. Do not ban FIFOs, add detached writer goroutines, or revive the source audits' context-aware stream machinery. | | [7. Publish truthful failure reports and preserve initialized evidence](catalog.md#7-publish-truthful-failure-reports-and-preserve-initialized-evidence) | C, A, T | Ready | Publishing completed reports after a known delivery failure contradicts their meaning. Select failed status before the first report write; preserve initialized evidence if rendering fails. Carry rendering errors explicitly and retain bounded best-effort finalization. Cross-file atomicity remains outside the contract. | @@ -90,13 +92,13 @@ behavior from proposed extensions. | [22. Preserve failure phase and origin in run reports](catalog.md#22-preserve-failure-phase-and-origin-in-run-reports) | C, A, T | Ready | Required phase and available operation origin are lost between owners. Carry structured facts and assemble run records once, keeping primary failure separate from canceled peers and secondary delivery failures. Preserve documented counts and serialized identities. | | [23. Classify provider attempt timeouts separately from operation deadlines](catalog.md#23-classify-provider-attempt-timeouts-separately-from-operation-deadlines) | C, T | Ready | The attempt-timeout probe and classifier order establish misclassification. Prefer native PromptKit classifications and parent cancellation precedence. Use the existing transport-failure reason for an HTTP-only timeout; a new public reason code is unnecessary for this fix. | | [24. Use input diagnostics for operator-correctable failures](catalog.md#24-use-input-diagnostics-for-operator-correctable-failures) | C, T | Ready; technical ownership check | Processing readiness and chunk infeasibility should follow the existing input-error contract while I/O and cancellation remain runtime failures. Delegated inspection has a demonstrated classification mismatch, but locate responsibility in Audita's loader versus ckprompt/CommandKit before editing. Fix or request a fix at that owner; do not create a local replacement command surface. | -| [25. Identify missing required credentials in safe diagnostics](catalog.md#25-identify-missing-required-credentials-in-safe-diagnostics) | C, M | Ready | The missing-credential cause is available but obscured. Surface a safe requirement and role/profile context from typed metadata, preserving the native cause. Keep PromptKit responsible for credential validation; do not move provider preparation or secret inspection into a new local subsystem. | +| [25. Identify missing required credentials in safe diagnostics](catalog.md#25-identify-missing-required-credentials-in-safe-diagnostics) | C, M | Ready | The missing-credential cause is available but obscured by generic setup presentation. Surface actionable native setup detail with available role/profile context, preserving typed causes and classification. D1 removes the blanket restriction on cause presentation; compact ordinary report projections remain useful. Keep PromptKit responsible for credential validation; do not introduce local secret inspection or a replacement preparation subsystem. | | [26. Render input and output paths as plain Markdown text](catalog.md#26-render-input-and-output-paths-as-plain-markdown-text) | C, T | Ready | Input paths are data; rendered Markdown currently gives some filenames unintended syntax. Escape them as literal text without altering structured path values. This restores report presentation rather than imposing filename restrictions. | | [27. Keep the validator execution class used by the runner](catalog.md#27-keep-the-validator-execution-class-used-by-the-runner) | C, A | Ready | Runtime and ledger can disagree for supported injected validator classes. Record the effective execution class with decisions and project it into the ledger. Preserve existing output fields and fallback behavior; no new validator registration policy. | | [28. Verify signal and exit handling in the production executable](catalog.md#28-verify-signal-and-exit-handling-in-the-production-executable) | T | Ready | The startup helper duplicates the production boundary, so coverage does not prove actual executable signal wiring. Exercise representative startup and cancellation through the real executable or shared production entry point; keep narrower tests for other CLI behavior. | | [29. Fail the release bootstrap when cleanliness cannot be established](catalog.md#29-fail-the-release-bootstrap-when-cleanliness-cannot-be-established) | C, T | Ready | The shell probe establishes that a failed status command can appear clean. Check its exit result explicitly and test that boundary. ReleaseKit already owns the independent delivery validation; do not add another local release validator or publication gate. | | [30. Bound pending proposal work before backend admission](catalog.md#30-bound-pending-proposal-work-before-backend-admission) | E, A, T; P | Investigate implementation; D3 for policy | Fan-out and retained results scale with chunks; finite native admission rejection is itself working as configured. Design bounded local preparation/dispatch and concurrent result draining without changing backend ownership, stable identities, cancellation joins, or configured rejection semantics. Measure representative memory/contention to size the implementation. Guaranteeing a turn despite capacity rejection or adding a new scheduling control requires D3. | -| [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A; P | Ready in part; D4 | Remove the generic JSON round trip and release whole-input representations after their last use. Share typed output projection; streaming into regular-file staging is an implementation option. Preserve encoding-failure and stream-delivery semantics. An application input-size limit or regular-file-only restriction is not an optimization; D4 covers size limits, and FIFO exclusion is not selected. | +| [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A; P | Ready in part; D4 | Remove generic JSON round trips and release whole-input representations after their last use. The D1 audit additionally identifies source-debug reconstruction: write the acquired bytes directly and retain parsed/normalized views separately, without reopening the input. Share typed output projection; preserve encoding-failure and stream-delivery semantics. An input-size limit remains D4, and FIFO exclusion is not selected. | | [32. Retain only the run evidence required by enabled outputs](catalog.md#32-retain-only-the-run-evidence-required-by-enabled-outputs) | E, A | Ready | The production consumers establish that much completed evidence is unused without its corresponding output. Retain only required live repair state, compact counts, and selected report/debug detail. Derive not-started counts without a full eager grid where possible; keep complete matrices when enabled and preserve fatal-origin evidence. | | [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A; P | Ready in part; D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected evidence. Those are efficiency improvements. Quotas, automatic deletion, stopping future capture, or replacing self-contained interactions with reference-based artifacts change retained evidence and need D5. | | [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E; P | Investigate costs; D6 | Repeated full context follows the deliberately selected repair architecture, so growth alone is not a defect. Measure cumulative requests, bytes, revalidation, and debug costs using existing observations or development instrumentation. Avoid duplicate local representations. New aggregate stopping budgets or reduced model context require D6; do not add ordinary-report fields merely to perform this analysis. | @@ -108,7 +110,7 @@ behavior from proposed extensions. | [40. Use one validator contract and one owner for decision semantics](catalog.md#40-use-one-validator-contract-and-one-owner-for-decision-semantics) | A, C, M | Ready | Duplicate ports and runner-specific verdict rewriting obscure one existing validation contract. Consolidate types, applicability, and decision meaning with their owner; remove unused envelope fields. Preserve zero-budget shortlisting, independent repair feedback, cardinality, and unavailable outcomes. No broad plugin framework or new validator policy is needed. | | [41. Carry structured failure facts through model operations and reporting](catalog.md#41-carry-structured-failure-facts-through-model-operations-and-reporting) | A, C | Ready | Failure class and reason already exist but are discarded and reconstructed. Carry typed facts through role-specific handling and report counts. Use a consistent internal-failure reason fallback for missing reasons while retaining the original failure class and cause. Avoid reclassifying an operation from its display reason. | | [42. Keep required output publication separate from optional debug retention](catalog.md#42-keep-required-output-publication-separate-from-optional-debug-retention) | A, C | Ready | Required publication and optional retained evidence have distinct lifecycles. Separate their APIs and share regular-file staging where behavior is common. Preserve transcript-before-report ordering, nontransactional finalization, private staging, and nonfatal debug errors. Coordinate with #1, #7, #16, and #22. | -| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready; simplify after D1 audit | Common interaction writing still has an ownership benefit, but the writer should not acquire redaction-only contracts, credential discovery, or combined-role secret plumbing. Audit those paths before choosing the smallest remaining writer interface. Preserve role-owned evidence, artifact association, and nonfatal write failures; D5 still governs retention changes. | +| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready under resolved D1 | The focused audit identifies removable secret-source assertions, lazy discovery closures, combined-role lists, and preflight/writer credential arguments. Complete common interaction writing without those dependencies. Preserve role-owned payloads, artifact association, and nonfatal write callbacks and synchronization; coordinate with #42/#48. D5 still governs retention changes. | | [44. Simplify structured-call completion and remove obsolete record fields](catalog.md#44-simplify-structured-call-completion-and-remove-obsolete-record-fields) | A, M, T | Ready | Ordinary calls and coordinator turns share completion responsibilities; obsolete payload copies have no production reader. Use explicit once-only completion and remove superseded state after moving meaningful assertions. Preserve native terminal counts, typed causes, and undelivered diagnostic evidence. | | [45. Retire the test-only direct continuation branch](catalog.md#45-retire-the-test-only-direct-continuation-branch) | M, T | Ready | Direct continuation is test-only; production uses coordination. Move sibling-isolation and structural-repair coverage onto the coordinator before removing the branch. Removing the redundant adapter surface does not remove PromptKit capability or production repair. | | [46. Use a single proposal handoff and test real runner indexing](catalog.md#46-use-a-single-proposal-handoff-and-test-real-runner-indexing) | A, E, M, T | Ready | Generator enrichment is discarded and real indices are assigned by the runner. Keep one proposal handoff with original local references, and assert actual deterministic runner indices under observed out-of-order completion. Review emitted diagnostic start_index separately before dropping it; move useful tests before removing their harness. | @@ -125,21 +127,24 @@ behavior from proposed extensions. | [57. Clarify what the inherited progress flag does in processing](catalog.md#57-clarify-what-the-inherited-progress-flag-does-in-processing) | M; P | Ready for clarification; D7 | The standard flag is exposed without processing events. Accurate help/documentation can proceed, preserving standard command acceptance. Adding a processing progress stream is a user-visible feature: D7 selects scope. Any standard help/capability gap belongs in CommandKit rather than local replacement routing. | | [58. Test registered validator roles and embedded assets through their behavior](catalog.md#58-test-registered-validator-roles-and-embedded-assets-through-their-behavior) | T, M | Ready | Registry tests duplicate representation assertions and prompt-count checks can miss substitutions. Check required identities, effective roles, preparability, schema identity, and protective copies at their actual owners. Do not replace them with prompt wording or private implementation assertions. | | [59. Consolidate overlapping normalization fixtures](catalog.md#59-consolidate-overlapping-normalization-fixtures) | T, M | Ready | Two fixtures protect the same nonmerge outcome. Consolidate their meaningful assertions, keeping distinct normalization behavior and independent input expectations. Coordinate with #51 so copied parser conversion is not retained in the tests. | -| [60. Focus integration tests on decoded artifacts and independent output choices](catalog.md#60-focus-integration-tests-on-decoded-artifacts-and-independent-output-choices) | T | Ready | Decoded artifact identities and evidence associations are the contract; escaping and repeated large fixtures are incidental. Keep one full repair/filter/later-failure attribution scenario and use smaller independent-output fixtures. Preserve real role selection, successive text, and output-choice coverage. | -| [61. Remove abandoned test support and unused fixture inputs](catalog.md#61-remove-abandoned-test-support-and-unused-fixture-inputs) | M, T | Ready | The reachability audit found abandoned helpers and unused fixture inputs after checking registrations and serialization. Delete them, retaining compile-time interface fixtures and active behavior assertions. Recheck callers when implementing if the code has advanced. | +| [60. Focus integration tests on decoded artifacts and independent output choices](catalog.md#60-focus-integration-tests-on-decoded-artifacts-and-independent-output-choices) | T | Ready | Decoded artifact identities and evidence associations are the contract; escaping and repeated large fixtures are incidental. Keep one full repair/filter/later-failure attribution scenario and use smaller independent-output fixtures. Remove masking assertions identified in the D1 audit without losing role selection, successive text, output-choice, and failure coverage. | +| [61. Remove abandoned test support and unused fixture inputs](catalog.md#61-remove-abandoned-test-support-and-unused-fixture-inputs) | M, T | Ready | The reachability audit found abandoned helpers and unused fixture inputs. Delete them, including unused secret scanners; the D1 audit also identifies credential-echo fixture machinery to retire when its masking-only consumers disappear. Retain compile-time interface fixtures and useful behavioral assertions, rechecking callers when implementing. | | [62. Remove unreachable merge and file helpers](catalog.md#62-remove-unreachable-merge-and-file-helpers) | M | Ready | Unreachable merge and file helpers duplicate or bypass live owners. Remove them after a current reference check, retaining live normalization, input diagnosis, and output-session behavior. | +| [63. Remove redaction-only credential discovery and tracking](catalog.md#63-remove-redaction-only-credential-discovery-and-tracking) | A, E, M, T | Ready under resolved D1 | Profile/backend enumeration, environment reads, name caches, and role tracking exist solely for substitution. Remove them after their consumers, preserving native authentication, inspection, prepared snapshots, call IDs, and terminal synchronization. | +| [64. Preserve validator explanations in repair feedback](catalog.md#64-preserve-validator-explanations-in-repair-feedback) | A, M, T | Ready under resolved D1 | Secret substitution affects live repair feedback even without debug. Use domain repair findings directly, retaining proposal association and unavailable-assessment semantics. Keep repair budgets, withdrawal, revalidation, and sibling independence unchanged. | +| [65. Remove credential substitution from CLI presentation](catalog.md#65-remove-credential-substitution-from-cli-presentation) | A, E, M, T | Ready under resolved D1 | Remove preparation/finalization sanitizers and obsolete arguments; present existing warnings, errors, and actual paths through CommandKit. Preserve typed causes, reporting modes, exit behavior, and output failure ownership. Coordinate with #25 for separately obscured setup detail. | ## Implementation planning implications -Group related work by owner rather than implementing catalog numbers as 62 +Group related work by owner rather than implementing catalog numbers as 65 independent tasks. Fixes and their supporting ownership changes belong together: - Output publication and diagnostic integrity: #1, #3–9, #16, #22, #26, - #42–43, incorporating the resolved D1 cleanup and keeping D2/D5 policy + #42–43, #63, #65, incorporating the resolved D1 cleanup and keeping D2/D5 policy portions separate. - Transcript and glossary correctness: #11–13, #17–21, #35–39, #51–54. - Runner and model-operation lifecycle: #10, #14–15, #23–25, #27, #30, - #32, #40–41, #44–49, #55–56. + #32, #40–41, #44–49, #55–56, #64. - Verification and obsolete machinery: #28–29, #58–62 and the tests attached to each other change. @@ -164,7 +169,8 @@ validator responses remain [separate deferred features](../future.md). ## Resolved D1: withdraw application-wide credential redaction -**Accepted:** 2026-11-15. **Policy update:** complete. **Implementation:** pending. +**Accepted:** 2026-11-15. **Policy update and focused audit:** complete. +**Implementation:** pending. The maintainer withdrew the application-wide credential-redaction requirement and approved removing machinery whose sole purpose is serving it, plus @@ -180,38 +186,22 @@ the premises of #3 and #4 and the remedies for #9 and #43. Ordinary publication correctness, authentication, and required output behavior still have independent purposes. The remaining six decisions retain their IDs. -### Focused follow-up audit +### Focused follow-up audit: complete -Perform a read-only audit before implementation, tracing both callers and data -flow. Its deliverable is an inventory of **remove**, **simplify**, and **retain** -items, with locations, remaining consumers, rationale, affected tests/docs, and -dependencies on catalog entries. It should cover: +The [removal and simplification audit](14-redaction.md) records seven findings, +traced consumers, retained responsibilities, tests/docs, and implementation +sequencing. It adds #63–65 and refines #3–4, #9, #25, #31, #43, and the related +test cleanup. It confirms that credential discovery, generic substitution, +serialization round trips, cross-role secret plumbing, repair-feedback rewriting, +and CLI substitution can be removed without replacing native library behavior. -1. Credential discovery performed solely for redaction, including profile/backend - enumeration, environment reads, caches, role tracking, and secret-provider - interfaces. Separate this from credentials needed by native authentication. -2. Generic redactors, encoded-secret handling, serialization round trips, - sanitizing copies, and fallback/error paths in report and artifact writers. -3. Redaction-related parameters, callbacks, adapters, and data structures across - proposal, validation, repair, CLI output, and execution reporting. Simplify - remaining responsibilities instead of replacing removed code with no-op seams. -4. Error presentation that hides actionable native information only to satisfy - the old guarantee. Preserve useful error classifications, attribution, and - stable output contracts while examining whether messages can be clearer. -5. Tests, fixtures, documentation, comments, and outstanding audit recommendations - that perpetuate the withdrawn requirement. Retain independent coverage of - serialization, evidence fidelity, output failures, and upstream integration. - -For each item, establish whether it serves an independent application requirement -or is supplied natively upstream. Preserve actual authentication and the standard -inspection surfaces. Evaluate implementation complexity and data loss, not merely -occurrences of the word "redaction." Do not execute live providers or print real -credential values during this audit. - -Use the inventory to plan coherent removal/refactoring units. Update implemented -behavior documentation alongside those code changes, so documentation does not -claim cleanup has already landed. The source audit files remain historical -evidence; this assessment controls the revised remediation scope. +Use that inventory to plan coherent removal/refactoring units. Preserve native +authentication and inspection, evidence fidelity, typed failure classification, +Markdown escaping, ordinary output correctness, and nonfatal debug failures. +No new product decision is required for these removals; D2–D7 remain unchanged. +The static audit did not run providers or tests or inspect real credentials. +Update implemented-behavior documentation alongside code changes. The source +audits remain historical evidence; this assessment controls remediation scope. ## Open Decisions diff --git a/docs/roadmap/audit/catalog.md b/docs/roadmap/audit/catalog.md index 4e22910..2e158e2 100644 --- a/docs/roadmap/audit/catalog.md +++ b/docs/roadmap/audit/catalog.md @@ -7,8 +7,8 @@ through a PromptKit adapter. The CLI publishes the corrected transcript and ordinary reports, with separate opt-in debug artifacts. [Package map](../../internal/overview.md) · [Pipeline](../../internal/pipeline.md) -The 11 source audits contain 103 numbered findings. This catalog consolidates them -into 62 entries. Priority reflects potential impact, exposure, and the strength of +The 12 source audits contain 110 numbered findings. This catalog consolidates them +into 65 entries. Priority reflects potential impact, exposure, and the strength of the reported evidence. A source can appear under several entries when it covers distinct issues. Each link leads to the original finding and its supporting evidence. @@ -18,6 +18,9 @@ the maintainer-approved taxonomy and separates ready remediation from technical investigation and pending policy decisions. Consult it before planning changes; source-audit recommendations are evidence and proposals, not authorization. The three WONTFIX scopes recorded here remain controlling. +The focused [redaction-removal audit](14-redaction.md) applies resolved D1: +the earlier redaction remedies below are superseded, and #63–65 add distinct +cleanup work. Entry numbers and existing anchors are retained for traceability. ## Data protection and run integrity @@ -45,17 +48,24 @@ built-in transport. ### 3. Protect credentials inside nested debug responses -Outer artifact redaction can leave credentials in a nested structured response. -Verify the decoded content of retained artifacts and define safe retention for -content that cannot be decoded. +**Superseded by D1.** The original observation tested the withdrawn redaction +requirement. Remove Audita's substitution package and use direct artifact +serialization; retain useful responses, including malformed content. Retire +masking-only tests and documentation while preserving evidence-fidelity and +write-failure coverage. Credential discovery and repair/CLI consumers are covered +by #63–65; no nested sanitizer or response-omission policy is selected. [Security §4](07-security.md#4-raw-structured-debug-responses-can-retain-an-encoded-credential) · -[Testing §8](08-testing.md#8-debug-redaction-assertions-mostly-inspect-the-outer-serialization) +[Testing §8](08-testing.md#8-debug-redaction-assertions-mostly-inspect-the-outer-serialization) · +[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding) · +[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) ### 4. Make debug artifact permissions private by default -Opt-in debug capture can expose transcript and model artifacts under a shared -parent. Use a per-user default and private creation modes, and document the trust -assumptions for explicitly shared destinations. +**Prior remedy withdrawn under D1.** Operators own artifact access and sharing; +hostile local filesystem access is outside the selected threat model. The focused +audit found ordinary creation and filename handling to retain, not a redundant +private-root mechanism. Leave current locations and modes unchanged in this +cleanup. Any future default change needs an independent justification. [Security §1](07-security.md#1-debug-capture-uses-a-shared-path-and-can-expose-private-artifacts) ### 5. Define the trust boundary for discovered configuration @@ -104,9 +114,11 @@ ownership if shared destinations are supported. ### 9. Preserve the ordinary YAML schema during redaction A credential value can match a required key or fixed enum and alter an otherwise -successful report. Redact data-bearing fields while preserving required keys and -values. -[Correctness §9](01-correctness.md#9-credential-redaction-can-change-the-ordinary-yaml-schema) +successful report. Under resolved D1, remove the post-serialization redactor and +serialize the typed report directly. Keep schema/scalar fidelity coverage; do +not build a more selective sanitizer. +[Correctness §9](01-correctness.md#9-credential-redaction-can-change-the-ordinary-yaml-schema) · +[Redaction removal §2](14-redaction.md#2-replace-sanitizing-serialization-with-direct-encoding) ### 10. Reject contradictory validation decisions @@ -234,8 +246,9 @@ cancellation under distinct live budgets. Invalid processing selections and malformed inspection configuration can return the runtime exit status. A genuinely infeasible chunk target can also be reported as an internal defect. Preserve the distinction between readiness, chunking, and -runtime failures and assert exact statuses in CLI tests. In the inspection tests, -use a fixture that actually exercises redaction if asserting credential masking. +runtime failures and assert exact statuses in CLI tests. Preserve native +inspection behavior; retired application-wide masking expectations are not an +additional acceptance criterion. [Error handling §5](06-error-handling.md#5-invalid-processing-selections-use-the-runtime-failure-exit-status) · [Error handling §6](06-error-handling.md#6-an-infeasible-chunk-target-is-recorded-as-an-internal-processing-failure) · [Testing §2](08-testing.md#2-inspection-failure-tests-accept-the-wrong-exit-classification) @@ -243,9 +256,11 @@ use a fixture that actually exercises redaction if asserting credential masking. ### 25. Identify missing required credentials in safe diagnostics The process path reduces a known missing credential requirement to a generic setup -failure. Surface the configured requirement safely and preserve the underlying -typed cause. -[Error handling §3](06-error-handling.md#3-missing-required-credentials-produce-a-generic-setup-message) +failure. Surface actionable native setup detail and preserve the underlying +typed cause and failure classification. D1 removes the blanket restriction on +cause presentation; ordinary compact report projections remain useful. +[Error handling §3](06-error-handling.md#3-missing-required-credentials-produce-a-generic-setup-message) · +[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership) ### 26. Render input and output paths as plain Markdown text @@ -292,7 +307,10 @@ Verify actual contention and cancellation in tests. Whole-file reads, a generic JSON tree, retained source bytes, and full output buffers amplify memory before and after model work. Reduce live copies and -consider explicit limits for integrations handling uploads. +write the acquired source bytes directly to its debug snapshot instead of +reconstructing them for redaction. Keep parsed/normalized views separate. +Input-size limits remain a separate D4 decision. +[Redaction removal §3](14-redaction.md#3-save-the-acquired-source-snapshot-without-reconstructing-it) · [Resources §4](05-resources.md#4-whole-transcript-copies-amplify-memory-at-input-and-publication) · [Security §7](07-security.md#7-untrusted-input-size-and-blocking-io-can-exhaust-subprocess-workers) @@ -388,11 +406,14 @@ and reports, and expose debug setup through its own API. ### 43. Inject one interaction diagnostics contract Proposal, validation, and repair duplicate path types and writer wiring. Supply a -diagnostic sink from the adapter boundary, with role-specific payloads, lazy -redaction inputs, and nonfatal write warnings. +diagnostic sink from the adapter boundary, with role-specific payloads and +nonfatal write warnings. Remove secret-provider assertions, lazy discovery +closures, combined-role secret lists, and preflight/writer credential arguments. +Keep artifact association and optional write-error reporting. [Complexity §4](03-complexity.md#4-separate-validation-batch-assessment-from-the-diagnostic-writer) · [Equivalence §5](10-equivalence.md#5-give-optional-interaction-artifacts-one-writer-contract) · -[Boundaries §6](11-boundaries.md#6-inject-interaction-diagnostics-at-the-adapter-boundary) +[Boundaries §6](11-boundaries.md#6-inject-interaction-diagnostics-at-the-adapter-boundary) · +[Redaction removal §4](14-redaction.md#4-remove-secret-plumbing-from-writer-and-preflight-contracts) ### 44. Simplify structured-call completion and remove obsolete record fields @@ -522,14 +543,17 @@ domain cases and meaningful merge, text, and category assertions. Assert decoded IDs and evidence associations and use smaller fixtures for duplicate reporting and debug combinations. Keep a full scenario for attribution -through repair and later failure. -[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) +through repair and later failure. Remove masking expectations without discarding +the independent artifact and outcome assertions mixed into those fixtures. +[Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) · +[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) ### 61. Remove abandoned test support and unused fixture inputs The orphan test-support package, unused CLI and runner helpers, and legacy glossary fixture input have no current callers. Preserve compile-time interface fixtures and active behavior assertions. +[Redaction removal §7](14-redaction.md#7-retire-obsolete-assertions-and-documentation-without-losing-coverage) · [Dead code §9](12-deadcode.md#9-clean-up-abandoned-test-support) · [Testing §13](08-testing.md#13-narrow-integration-assertions-to-their-distinct-contracts) @@ -538,3 +562,31 @@ fixtures and active behavior assertions. The uncalled merge predicate and standalone JSON and file-write helpers duplicate or bypass the live normalization, schema, and output paths. [Dead code §7](12-deadcode.md#7-remove-orphaned-normalization-and-file-helpers) + +## Additional cleanup after resolved D1 + +### 63. Remove redaction-only credential discovery and tracking + +Audita enumerates captured profiles/backends, reads credential environment values, +caches names, and tracks prepared role names under a mutex solely for substitution. +Remove that state and the secret-source methods after updating their consumers. +Preserve PromptKit authentication, prepared evidence, execution synchronization, +and native inspection behavior. +[Redaction removal §1](14-redaction.md#1-remove-credential-discovery-and-tracking-used-only-for-substitution) + +### 64. Preserve validator explanations in repair feedback + +Repair rewrites validator explanations using both roles' credential values even +with debug off. Remove this transformation and use domain feedback directly. +Retain proposal association, unavailable-assessment semantics, and the existing +bounded repair/revalidation behavior. +[Redaction removal §5](14-redaction.md#5-stop-rewriting-validator-explanations-sent-to-repair) + +### 65. Remove credential substitution from CLI presentation + +Preparation and completion paths scrub warnings, errors, and announced paths; +completion discovers credentials even with reports/debug disabled. Use direct +CommandKit presentation, remove obsolete arguments, and retain typed causes, +reporting modes, exit status, and required-output failure handling. Coordinate +with #25 for setup detail still hidden by the separate failure wrapper. +[Redaction removal §6](14-redaction.md#6-simplify-cli-presentation-while-retaining-typed-failure-ownership)