diff --git a/docs/roadmap/audit/assessment.md b/docs/roadmap/audit/assessment.md index b262d8c..ab85bf2 100644 --- a/docs/roadmap/audit/assessment.md +++ b/docs/roadmap/audit/assessment.md @@ -3,7 +3,8 @@ **Reviewed baseline:** `faba8ab` (2026-11-15). **Status:** assessment complete; remediation not implemented. D1 was resolved on 2026-11-15 and the architecture policy updated; the focused cleanup audit is complete and implementation remains -pending. +pending. D2 was resolved on 2026-11-15: callers own report destination isolation; +the operator-documentation update remains pending. This document owns classification, recommended scope, and pending decisions for the [catalog](catalog.md). The source audits retain their evidence; their suggested remedies are not approved requirements. @@ -37,8 +38,8 @@ when substantial restructuring is appropriate. **Ready** means suitable for implementation planning within the stated scope, not implemented or validated by this document. **Investigate** denotes technical -work the agent can undertake without a product decision. **D2–D7** identify -pending substantive choices; D1 is recorded below as resolved. **Closed** +work the agent can undertake without a product decision. **D3–D7** identify +pending substantive choices; D1 and D2 are recorded below as resolved. **Closed** denotes no remediation for the specified scope. Mixed entries deliberately separate ready work from decisions and exclusions. @@ -75,7 +76,7 @@ behavior from proposed extensions. | [5. Define the trust boundary for discovered configuration](catalog.md#5-define-the-trust-boundary-for-discovered-configuration) | M; P | Ready for documentation; extension excluded | Automatic companions are intentional configuration composition, not a discovery bug. Explain that the searched locations, selected bundle, imports, and companions are trusted configuration, and keep uploaded transcripts separate. Disabling discovery or adding a new isolation mode is outside this remediation; no ConfigKit replacement loader. | | [6. Make blocking transcript input and delivery cancellation-aware](catalog.md#6-make-blocking-transcript-input-and-delivery-cancellation-aware) | C; — | Ready in part; WONTFIX in part | Honor cancellation before regular-file publication and when blocked delivery returns; produce failure status even if bytes have escaped. Do not promise interruption of already-blocked standard I/O. Do not ban FIFOs, add detached writer goroutines, or revive the source audits' context-aware stream machinery. | | [7. Publish truthful failure reports and preserve initialized evidence](catalog.md#7-publish-truthful-failure-reports-and-preserve-initialized-evidence) | C, A, T | Ready | Publishing completed reports after a known delivery failure contradicts their meaning. Select failed status before the first report write; preserve initialized evidence if rendering fails. Carry rendering errors explicitly and retain bounded best-effort finalization. Cross-file atomicity remains outside the contract. | -| [8. Prevent concurrent runs from clobbering ordinary reports](catalog.md#8-prevent-concurrent-runs-from-clobbering-ordinary-reports) | M; P | Ready for documentation; D2 | The collision is demonstrated, but independent concurrent ownership of fixed report basenames is not promised. Document per-invocation directory selection now. Automatic run-specific placement, locking, or rejection of competing writers changes the workflow and belongs to D2. | +| [8. Prevent concurrent runs from clobbering ordinary reports](catalog.md#8-prevent-concurrent-runs-from-clobbering-ordinary-reports) | M | Ready for documentation under resolved D2 | The maintainer selected caller-owned destination isolation. Retain existing placement and document distinct report directories for concurrent jobs or retained run histories. Do not add automatic placement, locking, waiting, or competing-writer rejection. Ordinary checked-destination correctness remains #1’s independent scope. | | [9. Preserve the ordinary YAML schema during redaction](catalog.md#9-preserve-the-ordinary-yaml-schema-during-redaction) | C, A, M | Ready through D1 cleanup | Schema corruption is real, but preserving the redactor is no longer the remedy. Remove application credential substitution from ordinary report serialization and preserve the typed schema. Retain meaningful output-schema tests; retire tests whose sole purpose is satisfying the withdrawn redaction promise. | | [10. Reject contradictory validation decisions](catalog.md#10-reject-contradictory-validation-decisions) | C, A | Ready | Approved-plus-unavailable contradicts the existing acceptance invariant, although bundled producers are consistent. Establish an authoritative internal disposition and reject invalid internal results before use; derive redundant serialized approval as needed. This is an internal contract failure, distinct from an unavailable model assessment. | | [11. Preserve usable validator decisions when extra references cannot be decoded](catalog.md#11-preserve-usable-validator-decisions-when-extra-references-cannot-be-decoded) | C, T | Ready | The mixed-response probe violates existing sibling-preservation rules. Retain exact numeric references until range reconciliation, accepting integral JSON number spellings without float rounding. Ignore extras before conversion to local indices. Preserve whole-response structural validation; do not add semantic retries or general malformed-response salvage. | @@ -140,8 +141,8 @@ Group related work by owner rather than implementing catalog numbers as 65 independent tasks. Fixes and their supporting ownership changes belong together: - Output publication and diagnostic integrity: #1, #3–9, #16, #22, #26, - #42–43, #63, #65, incorporating the resolved D1 cleanup and keeping D2/D5 policy - portions separate. + #42–43, #63, #65, incorporating resolved D1 cleanup and D2 caller ownership while + keeping D5 policy portions separate. - Transcript and glossary correctness: #11–13, #17–21, #35–39, #51–54. - Runner and model-operation lifecycle: #10, #14–15, #23–25, #27, #30, #32, #40–41, #44–49, #55–56, #64. @@ -184,7 +185,7 @@ The maintainer's stated scope places artifact access and sharing with the operator and excludes hostile actors with local filesystem access. This changes the premises of #3 and #4 and the remedies for #9 and #43. Ordinary publication correctness, authentication, and required output behavior still have independent -purposes. The remaining six decisions retain their IDs. +purposes. Subsequent decisions retain their IDs. ### Focused follow-up audit: complete @@ -198,35 +199,34 @@ and CLI substitution can be removed without replacing native library behavior. Use that inventory to plan coherent removal/refactoring units. Preserve native authentication and inspection, evidence fidelity, typed failure classification, Markdown escaping, ordinary output correctness, and nonfatal debug failures. -No new product decision is required for these removals; D2–D7 remain unchanged. +No new product decision is required for these removals; subsequent decisions +are independent of D1. The static audit did not run providers or tests or inspect real credentials. Update implemented-behavior documentation alongside code changes. The source audits remain historical evidence; this assessment controls remediation scope. +## Resolved D2: callers own report destination isolation + +**Accepted:** 2026-11-15. **Applies to:** #8. +**Status:** decision recorded; operator-documentation update pending. + +Report destination isolation remains the caller’s responsibility. Keep the +current placement contract and fixed report basenames. Document in the operations +guide that concurrent jobs select distinct report directories, and that retaining +reports from successive runs likewise requires separate destinations or caller +archiving. Different transcript filenames in one directory do not isolate reports. + +Automatic per-run directories, destination locking, waiting, and competing-writer +rejection are not selected. They would add placement or contention semantics to +a workflow the caller already controls. The audit demonstrates overwriting when +callers share destinations; it does not establish an application promise of +per-run isolation. Correct writes to checked destinations in #1 remain independent. + ## Open Decisions -The six decisions D2–D7 below remain pending. Accepting or rejecting an optional +The five decisions D3–D7 below remain pending. Accepting or rejecting an optional extension must not delay the independent ready portions of its catalog entries. -### D2. Should Audita manage concurrent ownership of ordinary report destinations? - -**Applies to:** #8. - -**Recommendation:** retain the current placement contract and document that -concurrent callers select distinct report directories. This avoids new locking, -waiting, or placement semantics and works with the existing explicit destination -selection. The audit establishes clobbering under shared use, not a broken -promise of per-run isolation. - -**Alternative:** give runs automatically isolated report directories, with a -documented way to discover them. This makes concurrent jobs easier and preserves -each run's evidence, but changes default artifact locations and downstream -collection. Serializing or rejecting competing writers is another option, but -adds contention and recovery semantics merely to retain shared names. - -**Why approval:** automatic placement, waiting, and rejection change the operator -workflow. Correct publication to the checked destination in #1 is independent. - ### D3. Should local dispatch change what backend saturation means? **Applies to:** #30, with scheduling interactions in #15. diff --git a/docs/roadmap/audit/catalog.md b/docs/roadmap/audit/catalog.md index 2e158e2..b08dc7a 100644 --- a/docs/roadmap/audit/catalog.md +++ b/docs/roadmap/audit/catalog.md @@ -107,8 +107,11 @@ existing evidence if a failed bundle cannot be rendered. ### 8. Prevent concurrent runs from clobbering ordinary reports Invocations sharing a report directory replace the same basenames and can leave a -mixed set. Document destination isolation for callers and define publication -ownership if shared destinations are supported. +mixed set. **Resolved D2:** destination isolation remains the caller’s +responsibility. Retain current report placement and document distinct report +directories for concurrent jobs or retained run histories. Automatic per-run +placement, locking, waiting, and competing-writer rejection are outside this +remediation. Correct publication to checked destinations remains #1’s concern. [Concurrency §7](04-concurrency.md#7-concurrent-invocations-sharing-a-report-directory-replace-each-others-evidence) ### 9. Preserve the ordinary YAML schema during redaction