diff --git a/docs/roadmap/audit/assessment.md b/docs/roadmap/audit/assessment.md index aeb6df1..a0cd436 100644 --- a/docs/roadmap/audit/assessment.md +++ b/docs/roadmap/audit/assessment.md @@ -8,6 +8,8 @@ the operator-documentation update remains pending. D3 was resolved on the same date: native admission semantics remain in place; local scheduling improvements await implementation planning. D4 was also resolved on 2026-11-15: improve input memory use without introducing an application-level input-size limit. +D5 was resolved on the same date: retain best-effort debug capture and operator +ownership of retention and cleanup; efficiency improvements remain pending. This document owns classification, recommended scope, and pending decisions for the [catalog](catalog.md). The source audits retain their evidence; their suggested remedies are not approved requirements. @@ -41,8 +43,8 @@ when substantial restructuring is appropriate. **Ready** means suitable for implementation planning within the stated scope, not implemented or validated by this document. **Investigate** denotes technical -work the agent can undertake without a product decision. **D5–D7** identify -pending substantive choices; D1–D4 are recorded below as resolved. **Closed** +work the agent can undertake without a product decision. **D6–D7** identify +pending substantive choices; D1–D5 are recorded below as resolved. **Closed** denotes no remediation for the specified scope. Mixed entries deliberately separate ready work from decisions and exclusions. @@ -104,7 +106,7 @@ behavior from proposed extensions. | [30. Bound pending proposal work before backend admission](catalog.md#30-bound-pending-proposal-work-before-backend-admission) | E, A, T | Investigate implementation under resolved D3 | Bound local preparation/dispatch and drain results while preserving PromptKit admission ownership, configured waiting/rejection, stable identities, module barriers, and cancellation joins. Do not add an outer capacity-retry loop, duplicate backend limiter, or eventual-admission promise. The current scheduling-dependent selection of rejected chunks is not a compatibility requirement. Measure representative memory/contention to guide the implementation; any future admission-policy gap belongs upstream. | | [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A | Ready under resolved D4 | Remove generic JSON round trips and release whole-input representations after their last use. The D1 audit additionally identifies source-debug reconstruction: write the acquired bytes directly and retain parsed/normalized views separately, without reopening the input. Share typed output projection; preserve encoding-failure and stream-delivery semantics. Do not introduce an application-level input-size limit; calling integrations own their upload/process budgets. Preserve accepted input formats and stream support. | | [32. Retain only the run evidence required by enabled outputs](catalog.md#32-retain-only-the-run-evidence-required-by-enabled-outputs) | E, A | Ready | The production consumers establish that much completed evidence is unused without its corresponding output. Retain only required live repair state, compact counts, and selected report/debug detail. Derive not-started counts without a full eager grid where possible; keep complete matrices when enabled and preserve fatal-origin evidence. | -| [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A; P | Ready in part; D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected evidence. Those are efficiency improvements. Quotas, automatic deletion, stopping future capture, or replacing self-contained interactions with reference-based artifacts change retained evidence and need D5. | +| [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A | Ready under resolved D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected-artifact information. Preserve requested evidence, self-contained interactions, nonfatal debug failures, and continued best-effort capture. Add no quotas, automatic deletion, or automatic stopping after write failures; operators own retention and cleanup. | | [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E; P | Investigate costs; D6 | Repeated full context follows the deliberately selected repair architecture, so growth alone is not a defect. Measure cumulative requests, bytes, revalidation, and debug costs using existing observations or development instrumentation. Avoid duplicate local representations. New aggregate stopping budgets or reduced model context require D6; do not add ordinary-report fields merely to perform this analysis. | | [35. Avoid rebuilding growing normalization prefixes](catalog.md#35-avoid-rebuilding-growing-normalization-prefixes) | E | Ready | Growing merge prefixes are repeatedly scanned and copied. Use incremental text/category accumulation and exact estimator components while retaining joiner behavior, punctuation rounding, limits, and category order. Benchmarks should verify the improvement; large refactoring is not a reason to retain quadratic work. | | [36. Skip validation context work beyond the usable budget](catalog.md#36-skip-validation-context-work-beyond-the-usable-budget) | E, A, T | Ready | Unusable optional context is serialized needlessly. Stop measurement once required-only or ordered-prefix costs settle the fit. Share trimming/batching policy without losing the independent serialized estimator oracle or changing context priority and oversized-item handling. | @@ -114,7 +116,7 @@ behavior from proposed extensions. | [40. Use one validator contract and one owner for decision semantics](catalog.md#40-use-one-validator-contract-and-one-owner-for-decision-semantics) | A, C, M | Ready | Duplicate ports and runner-specific verdict rewriting obscure one existing validation contract. Consolidate types, applicability, and decision meaning with their owner; remove unused envelope fields. Preserve zero-budget shortlisting, independent repair feedback, cardinality, and unavailable outcomes. No broad plugin framework or new validator policy is needed. | | [41. Carry structured failure facts through model operations and reporting](catalog.md#41-carry-structured-failure-facts-through-model-operations-and-reporting) | A, C | Ready | Failure class and reason already exist but are discarded and reconstructed. Carry typed facts through role-specific handling and report counts. Use a consistent internal-failure reason fallback for missing reasons while retaining the original failure class and cause. Avoid reclassifying an operation from its display reason. | | [42. Keep required output publication separate from optional debug retention](catalog.md#42-keep-required-output-publication-separate-from-optional-debug-retention) | A, C | Ready | Required publication and optional retained evidence have distinct lifecycles. Separate their APIs and share regular-file staging where behavior is common. Preserve transcript-before-report ordering, nontransactional finalization, private staging, and nonfatal debug errors. Coordinate with #1, #7, #16, and #22. | -| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready under resolved D1 | The focused audit identifies removable secret-source assertions, lazy discovery closures, combined-role lists, and preflight/writer credential arguments. Complete common interaction writing without those dependencies. Preserve role-owned payloads, artifact association, and nonfatal write callbacks and synchronization; coordinate with #42/#48. D5 still governs retention changes. | +| [43. Inject one interaction diagnostics contract](catalog.md#43-inject-one-interaction-diagnostics-contract) | A, M | Ready under resolved D1 | The focused audit identifies removable secret-source assertions, lazy discovery closures, combined-role lists, and preflight/writer credential arguments. Complete common interaction writing without those dependencies. Preserve role-owned payloads, artifact association, and nonfatal write callbacks and synchronization; coordinate with #42/#48. Resolved D5 preserves best-effort capture and operator-owned retention. | | [44. Simplify structured-call completion and remove obsolete record fields](catalog.md#44-simplify-structured-call-completion-and-remove-obsolete-record-fields) | A, M, T | Ready | Ordinary calls and coordinator turns share completion responsibilities; obsolete payload copies have no production reader. Use explicit once-only completion and remove superseded state after moving meaningful assertions. Preserve native terminal counts, typed causes, and undelivered diagnostic evidence. | | [45. Retire the test-only direct continuation branch](catalog.md#45-retire-the-test-only-direct-continuation-branch) | M, T | Ready | Direct continuation is test-only; production uses coordination. Move sibling-isolation and structural-repair coverage onto the coordinator before removing the branch. Removing the redundant adapter surface does not remove PromptKit capability or production repair. | | [46. Use a single proposal handoff and test real runner indexing](catalog.md#46-use-a-single-proposal-handoff-and-test-real-runner-indexing) | A, E, M, T | Ready | Generator enrichment is discarded and real indices are assigned by the runner. Keep one proposal handoff with original local references, and assert actual deterministic runner indices under observed out-of-order completion. Review emitted diagnostic start_index separately before dropping it; move useful tests before removing their harness. | @@ -144,8 +146,8 @@ Group related work by owner rather than implementing catalog numbers as 65 independent tasks. Fixes and their supporting ownership changes belong together: - Output publication and diagnostic integrity: #1, #3–9, #16, #22, #26, - #42–43, #63, #65, incorporating resolved D1 cleanup and D2 caller ownership while - keeping D5 policy portions separate. + #42–43, #63, #65, incorporating resolved D1 cleanup, D2 caller ownership, and D5 + best-effort capture with operator-owned retention. - Transcript and glossary correctness: #11–13, #17–21, #35–39, #51–54. - Runner and model-operation lifecycle: #10, #14–15, #23–25, #27, #30, #32, #40–41, #44–49, #55–56, #64. @@ -267,31 +269,31 @@ that justifies rejecting otherwise valid CLI workloads. An optional input-byte budget could help integrations without their own enforcement; it is not selected because it adds configuration and rejection behavior without a demonstrated need. +## Resolved D5: preserve best-effort debug capture + +**Accepted:** 2026-11-15. **Applies to:** #33. +**Status:** decision recorded; efficiency improvements await implementation. + +Preserve requested diagnostic evidence and continue best-effort capture. Debug +write failures remain nonfatal. Remove redundant in-memory representations and +serialization, and consolidate equivalent warnings while retaining counts and +affected-artifact information. Keep interaction artifacts self-contained. + +Do not add capture quotas, automatic deletion, or automatic shutdown of later +capture after write failures in this remediation. An individual failure need +not prevent later writes from succeeding; continued capture can retain useful +evidence for an unsuccessful run. Operators own retention and cleanup. + +Optional budgets or persistent-failure stopping rules could reduce disk use and +repeated failed work, but deliberately omit evidence and add configuration and +reporting obligations. They are not selected. Internal efficiency improvements +should preserve the evidence operators requested. + ## Open Decisions -The three decisions D5–D7 below remain pending. Accepting or rejecting an optional +The two decisions D6–D7 below remain pending. Accepting or rejecting an optional extension must not delay the independent ready portions of its catalog entries. -### D5. Should debug capture acquire quotas, automatic retention rules, or an early-stop policy? - -**Applies to:** the policy portion of #33. - -**Recommendation:** preserve requested evidence and best-effort capture; optimize -its representation internally and coalesce equivalent write warnings with counts -and affected-artifact information. Do not add quotas, automatic deletion, or -disable later writes after the first failure in this remediation. - -**Alternative:** adopt explicit opt-in limits or a documented stop-capture policy -after persistent failures. This can protect disk and reduce repeated failed -work, but intentionally loses requested evidence and must expose what was -omitted. Automatic cleanup across runs requires separate retention ownership. -Replacing self-contained interaction artifacts with shared-history references -also needs a deliberate consumer contract. - -**Why approval:** these policies change what operators receive from debug mode. -Elimination of duplicate in-memory representations remains independent. -The permissions/default-root proposal in #4 is being reassessed under D1. - ### D6. Should correction repair gain aggregate budgets or reduced conversation context? **Applies to:** the optional behavior changes in #34. diff --git a/docs/roadmap/audit/catalog.md b/docs/roadmap/audit/catalog.md index 849169d..27522f3 100644 --- a/docs/roadmap/audit/catalog.md +++ b/docs/roadmap/audit/catalog.md @@ -337,8 +337,11 @@ collect compact counts when detailed evidence is disabled. Execution histories and repeated serialization increase peak memory, and retained interaction files and warnings can accumulate after write failures. Reduce -duplicate representations and define retention and failure behavior appropriate to -opt-in capture. +duplicate representations and serialization, and consolidate equivalent warnings +with counts and affected-artifact information. **Resolved D5:** preserve requested +evidence, self-contained interactions, nonfatal failures, and continued best-effort +capture. Add no quotas, automatic deletion, or automatic stopping after write +failures; operators own retention and cleanup. [Resources §6](05-resources.md#6-debug-capture-adds-memory-and-disk-pressure) ### 34. Account for cumulative correction-repair history and work