From a23ce1428592c7e7b779a8d8b32ecc3fb7198cd5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 16 Nov 2026 01:08:52 +0000 Subject: [PATCH] Record decision to retain repair context and limits --- docs/roadmap/audit/assessment.md | 52 +++++++++++++++++--------------- docs/roadmap/audit/catalog.md | 7 +++-- 2 files changed, 32 insertions(+), 27 deletions(-) diff --git a/docs/roadmap/audit/assessment.md b/docs/roadmap/audit/assessment.md index a0cd436..d0af8a9 100644 --- a/docs/roadmap/audit/assessment.md +++ b/docs/roadmap/audit/assessment.md @@ -10,6 +10,8 @@ await implementation planning. D4 was also resolved on 2026-11-15: improve input memory use without introducing an application-level input-size limit. D5 was resolved on the same date: retain best-effort debug capture and operator ownership of retention and cleanup; efficiency improvements remain pending. +D6 was also resolved on 2026-11-15: retain full repair conversation history and +per-correction limits, without aggregate budgets or context reduction at this time. This document owns classification, recommended scope, and pending decisions for the [catalog](catalog.md). The source audits retain their evidence; their suggested remedies are not approved requirements. @@ -43,8 +45,8 @@ when substantial restructuring is appropriate. **Ready** means suitable for implementation planning within the stated scope, not implemented or validated by this document. **Investigate** denotes technical -work the agent can undertake without a product decision. **D6–D7** identify -pending substantive choices; D1–D5 are recorded below as resolved. **Closed** +work the agent can undertake without a product decision. **D7** identifies +the remaining substantive choice; D1–D6 are recorded below as resolved. **Closed** denotes no remediation for the specified scope. Mixed entries deliberately separate ready work from decisions and exclusions. @@ -107,7 +109,7 @@ behavior from proposed extensions. | [31. Reduce peak memory for large transcript input and output](catalog.md#31-reduce-peak-memory-for-large-transcript-input-and-output) | E, A | Ready under resolved D4 | Remove generic JSON round trips and release whole-input representations after their last use. The D1 audit additionally identifies source-debug reconstruction: write the acquired bytes directly and retain parsed/normalized views separately, without reopening the input. Share typed output projection; preserve encoding-failure and stream-delivery semantics. Do not introduce an application-level input-size limit; calling integrations own their upload/process budgets. Preserve accepted input formats and stream support. | | [32. Retain only the run evidence required by enabled outputs](catalog.md#32-retain-only-the-run-evidence-required-by-enabled-outputs) | E, A | Ready | The production consumers establish that much completed evidence is unused without its corresponding output. Retain only required live repair state, compact counts, and selected report/debug detail. Derive not-started counts without a full eager grid where possible; keep complete matrices when enabled and preserve fatal-origin evidence. | | [33. Limit debug retention costs within a run](catalog.md#33-limit-debug-retention-costs-within-a-run) | E, A | Ready under resolved D5 | Reduce duplicate event representations and serialization; coalesce equivalent warnings while retaining failure counts and affected-artifact information. Preserve requested evidence, self-contained interactions, nonfatal debug failures, and continued best-effort capture. Add no quotas, automatic deletion, or automatic stopping after write failures; operators own retention and cleanup. | -| [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E; P | Investigate costs; D6 | Repeated full context follows the deliberately selected repair architecture, so growth alone is not a defect. Measure cumulative requests, bytes, revalidation, and debug costs using existing observations or development instrumentation. Avoid duplicate local representations. New aggregate stopping budgets or reduced model context require D6; do not add ordinary-report fields merely to perform this analysis. | +| [34. Account for cumulative correction-repair history and work](catalog.md#34-account-for-cumulative-correction-repair-history-and-work) | E | Investigate costs under resolved D6 | Retain full conversation history and existing per-correction limits. Measure cumulative requests, tokens/bytes, revalidation, and debug costs using existing observations or development instrumentation; remove unnecessary local duplication. Do not add aggregate stopping budgets or reduce model context at this time. Revisit only if production evidence demonstrates a need; no new ordinary-report fields are required for this analysis. | | [35. Avoid rebuilding growing normalization prefixes](catalog.md#35-avoid-rebuilding-growing-normalization-prefixes) | E | Ready | Growing merge prefixes are repeatedly scanned and copied. Use incremental text/category accumulation and exact estimator components while retaining joiner behavior, punctuation rounding, limits, and category order. Benchmarks should verify the improvement; large refactoring is not a reason to retain quadratic work. | | [36. Skip validation context work beyond the usable budget](catalog.md#36-skip-validation-context-work-beyond-the-usable-budget) | E, A, T | Ready | Unusable optional context is serialized needlessly. Stop measurement once required-only or ordered-prefix costs settle the fit. Share trimming/batching policy without losing the independent serialized estimator oracle or changing context priority and oversized-item handling. | | [37. Reuse checked processing selections and matching views](catalog.md#37-reuse-checked-processing-selections-and-matching-views) | E, A | Ready | Readiness and execution derive the same selections separately. Carry one checked processing plan and reuse immutable effective views and inspections per captured runtime. Preserve provenance and first-failure attribution; cache keys must include ordered selections and exemptions. Do not add cross-invocation caches or reload configuration. | @@ -289,30 +291,30 @@ repeated failed work, but deliberately omit evidence and add configuration and reporting obligations. They are not selected. Internal efficiency improvements should preserve the evidence operators requested. +## Resolved D6: retain repair context and per-correction limits + +**Accepted:** 2026-11-15. **Applies to:** #34. +**Status:** decision recorded; cost analysis and efficiency work remain pending. + +Retain full conversation history and the existing per-correction limits. Do not +add aggregate section/run repair budgets or reduce conversation context at this +time. Measure cumulative requests, tokens/bytes, revalidation, and debug costs, +and remove unnecessary local copies without changing the model's evidence or +which corrections receive repair attempts. Use existing observations or focused +development instrumentation; no new ordinary-report fields are needed merely +to perform the analysis. + +Resource growth partly reflects the deliberately selected repair behavior. +Aggregate budgets could leave later corrections unrepaired because earlier +ones consumed the allowance, while context reduction could reduce repair +quality. Neither change follows from the audit finding alone. Revisit these +options only if production evidence demonstrates a need, through a separate +policy decision. + ## Open Decisions -The two decisions D6–D7 below remain pending. Accepting or rejecting an optional -extension must not delay the independent ready portions of its catalog entries. - -### D6. Should correction repair gain aggregate budgets or reduced conversation context? - -**Applies to:** the optional behavior changes in #34. - -**Recommendation:** retain the selected full conversation history and -per-correction limits. Measure cumulative work and remove local duplication -before proposing any new stopping or context policy. The resource growth partly -reflects deliberate useful work rather than an implementation defect. - -**Alternative:** add an opt-in section/run budget covering repair and -revalidation, with explicit termination outcomes; or separately evaluate reduced -context. Budgets provide predictable resource envelopes but may leave later -corrections unrepaired. Context reduction can lower costs but can also reduce -editorial quality. Neither follows automatically from the audit's asymptotic -description. - -**Why approval:** these changes affect which corrections get another chance or -what evidence the model sees. Measurements using existing observations do not -require approval or new ordinary-report schema fields. +D7 is the only remaining open decision. Its resolution must not delay the +independent ready portions of the catalog. ### D7. Should this remediation add processing progress events? diff --git a/docs/roadmap/audit/catalog.md b/docs/roadmap/audit/catalog.md index 27522f3..7f9901e 100644 --- a/docs/roadmap/audit/catalog.md +++ b/docs/roadmap/audit/catalog.md @@ -347,8 +347,11 @@ failures; operators own retention and cleanup. ### 34. Account for cumulative correction-repair history and work Each rejected candidate can start its own bounded conversation and repeat the -original prompt history, with additional revalidation. Measure total requests and -bytes across branches before deciding on changes to context or run-level budgets. +original prompt history, with additional revalidation. **Resolved D6:** retain +full conversation history and existing per-correction limits; add no aggregate +budgets or context reduction at this time. Measure cumulative work and remove +unnecessary local duplication. Revisit behavior changes only if production +evidence demonstrates a need. [Resources §7](05-resources.md#7-correction-repair-replays-growing-history-under-per-correction-budgets) ### 35. Avoid rebuilding growing normalization prefixes