From d226bdd6972fff9e7d8d578575384a8034959a12 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 16 Nov 2026 03:24:06 +0000 Subject: [PATCH] Plan the reliability and maintainability sprint --- docs/roadmap/implementation.md | 557 ++++++++++++++++++ .../reliability-and-maintainability.md | 53 +- 2 files changed, 570 insertions(+), 40 deletions(-) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..3a919bb --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,557 @@ +# Reliability and Maintainability Implementation Plan + +Implement the [target roadmap](reliability-and-maintainability.md) in one +end-to-end sprint, starting from its application baseline `c4ccd78`. The +[assessment](audit/assessment.md) governs the accepted scope and decisions +D1–D7. Numbers under each stage refer to the [audit catalog](audit/catalog.md). +Use the original audits for reproductions and source references. Where an old +remedy differs from a later decision, follow the assessment and target roadmap. + +Work through the numbered stages in order. Each stage is sized for one Luna +implementation prompt and contributes directly to the final design. Intermediate +source may be incomplete or fail to build while adjacent changes are in flight. +Use the next stage to finish those connections. A stage needs neither a +standalone commit nor a passing full suite. Replace obsolete paths directly; +compatibility layers and temporary production adapters are unnecessary. + +Use the [development guide](../development.md) and applicable adopted policies, +especially [agents](../policy/agents.md), [architecture](../policy/architecture.md), +[testing](../policy/testing.md), and [documentation](../policy/documentation.md). +Add or adapt focused tests at the behavior owner and update canonical docs when +their implemented behavior changes. Run affected tests when the relevant path +builds, then close the sprint with Stage 20. Stage checks establish behavior +without fixing private names, scheduler sizes, event order, or prompt wording. +The target roadmap defines retained contracts, selected changes, and exclusions. + +## Stage 1 — Establish the delegated inspection dependency + +Audit #24, delegated part. + +1. Use a temporary CLI build to reproduce invalid operator configuration with + `promptkit list prompts` and `process`. Reuse the malformed companion fixture + in `internal/cli/runtime_errors_test.go` and include a separate cancellation + case. The baseline reported status 1 for the malformed companion via + inspection and status 2 via processing. Both commands must remain offline. +2. Trace `internal/cli/app_wiring.go` into the pinned `ckprompt` and `ckconfig` + commands. For the known catalog case, `ckprompt.runCatalogList` converts the + loader error to `definitionFailure` even when a typed input cause is + available. Check relevant configuration-scope acquisition paths separately. + Preserve typed causes in Audita's loader. Classify only input failures at + the responsible boundary, leaving cancellation and runtime failures distinct. +3. Check for a released upstream fix for each demonstrated delegated case. If a + suitable release exists, adopt the smallest compatible update using the + [OSS policy](../policy/oss.md) and verify classification at the Audita CLI. + Otherwise record the affected command, reproduction, owner, and needed + behavior for an upstream handoff. Continue the independent stages and use + Stage 20 to integrate and verify a supporting release if available. Keep the + native declarations and routing throughout. + +A fix is demonstrated by status 2 and actionable diagnostics for the invalid +input, and status 1 for cancellation. While the supporting release is pending, +record the mismatch explicitly. A test that asserts only nonzero status does +not verify this fix. + +## Stage 2 — Remove application credential substitution + +Audit #3, #9, #31 (source bytes), #63–65. Use the +[focused removal inventory](audit/14-redaction.md) for affected sites. + +1. Switch ordinary reports and debug artifacts to direct typed JSON/YAML + encoding. Keep encoded effective configuration from ConfigKit. In Markdown, + retain escaping and remove substitution. Keep malformed `final_content` as + the captured string. Write the acquired transcript bytes to the debug source + artifact and keep the parsed and normalized views separate. +2. Send domain `repairFindings` directly for initial feedback and subsequent + findings. Preserve the proposal association and typed unavailable assessment. + Present native errors, warnings, and actual paths through CommandKit. This + step removes substitution. Stage 12 exposes the native setup detail that is + currently hidden by generic failure presentation. +3. Remove writer/renderer secret arguments, `DebugInput.Secrets`, discovery + assertions and closures, repair role-list combination, cache and role + tracking, runtime secret discovery, and `internal/core/redaction`. Keep the + prepared and terminal callbacks, source-path tracking, native effective + rendering, and selected provider authentication. +4. Delete masking-only assertions, scanners, fixtures, and stale redaction + descriptions identified in the inventory. Move independent assertions to + writer or caller tests. Check decoded schema and scalar types, literal source + bytes, malformed response evidence, actual paths, and native auth using + synthetic data. Check feedback by decoding a production repair request. + +## Stage 3 — Unify optional interaction capture + +Audit #33 and #43. + +1. Define one small optional interaction sink at the shared execution port. + Compose the existing filesystem writer from the run directory in CLI wiring. + Pass the sink through proposal, validation, and repair. Each role supplies + its own payload and stage/chunk/operation coordinates, including proposal, + batch, validator, and repair turn where applicable. Keep returned paths and + write errors available for ledger association and warnings. +2. Reuse the write path and artifact envelope in + `internal/framework/llm/diagnostics.go`. Remove private writer adapters and + repeated construction. Keep artifacts self-contained, including revalidation + and a final unusable or undelivered response when available. Keep debug + directory placement, naming safety, and creation modes. +3. Collect optional write warnings safely across workers, grouping equivalent + failures by operation and cause with counts and affected artifact information. + Continue later capture after a write error. Keep the warning path outside + correction decisions and required file delivery. + +Verify role, chunk, batch, proposal, and attempt associations with decoded +artifacts. Exercise a failed optional write followed by later capture. Update +the [diagnostics internals](../internal/diagnostics-reporting.md) for the sink. + +## Stage 4 — Bind publication to checked destinations + +Audit #1, #6 (cooperative boundaries), #7, #8, #16, #26, #42. + +1. In `internal/core/diagnostics/output_session.go` and its CLI callers, resolve + checked transcript and report destinations with filesystem semantics. + Carry those destination and parent choices through staging and publication. + Keep protected-source checks, intentional in-place replacement, ordinary + symlink and stream behavior, and disabled reporting's inactive destination. + Add no directory creation or locking for required destinations. +2. Give the required-output owner a shared regular-file staging primitive for + transcript and reports. Keep optional debug setup separate from that owner. + Keep temporary paths until removal succeeds or absence is established. + Surface cleanup errors alongside the initiating error, including after + partial preparation, publication failure, or cancellation. +3. Make report rendering return an explicit error through initialization and + finalization. Publish the transcript first when processing succeeded. + Select a failed bundle before the first final report write when a required + delivery or cleanup failure is already known. If a report write first + establishes failure, switch the remaining writes to failed status and make + one best-effort pass over reports that still need failed evidence. If the + failed bundle cannot be rendered, keep initialized or published evidence + and return the rendering error. +4. Check context when transcript acquisition returns, at natural preparation + and publication boundaries, and after stream delivery returns. A cancellation + observed after bytes escape still fails the invocation. Join cancellation + and I/O/cleanup errors where available and keep the initiating failure + distinguishable. Use ordinary blocking I/O and retain completed transcript + publication if a later report write fails. +5. Escape input/output paths as literal Markdown text in `core/reporting`, + preserving raw values in structured artifacts. Update + [operations](../operations.md) for caller-owned destination isolation, + retention, and the resulting publication lifecycle. + +Exercise symlink/path resolution, in-place and stream output, cancellation +after I/O, cleanup errors, initialized evidence when rendering fails, and +write behavior after known failure. Observe publication attempts as well as +final files. Leave processing fact and ledger projection to Stage 17. + +## Stage 5 — Consolidate typed transcript data + +Audit #13, #31, #51, #52, #54, #62 (file helper). + +1. In `core/schema`, detect the two accepted root shapes while keeping the + segment array as raw JSON. Decode typed segments directly. Preserve source + ID presence, native integer range, duplicate checks, ignored unknown fields, + and existing format and error behavior. Keep one live source-to-canonical + conversion and retire obsolete parser wrappers after moving meaningful tests. +2. Centralize transcript cloning in the model. Use it at the runner and other + copy boundary, preserving independence and nil versus explicit empty + mutable slices. Share one typed segment projection across both encoders in + `core/schema` and `core/outputschema`. Preserve the envelopes, version, + numeric fidelity, category and omission rules, and encoding error behavior. +3. Release raw input after its final parse or source-snapshot write. Drop the + parsed source after normalization and its debug consumers. Avoid retaining + it via CLI finalization. Remove the audited unused file helper after checking + callers. Keep both file and stream inputs and current output delivery + semantics. Use a representative large fixture to check that the removed + representations no longer overlap needlessly. A fully streaming encoder is + optional if supported by the measurements and existing error semantics. + +Test both envelopes, adjacent accepted IDs beyond exact float precision where +supported by native `int`, clone independence with independent expected data, +and the distinction between raw and typed debug snapshots. Update the +[input contract](../integrations/transcript-glossary-files.md) and +[encoder internals](../internal/output-schemas.md) where affected. + +## Stage 6 — Correct and simplify normalization + +Audit #17, #21, #35, #53, #59, #62 (merge helper). + +1. In `core/normalization`, sort stably by the existing time key and retain + input order for ties. Compute the prospective merge end as the later end + and use that interval for the duration limit, merged segment, and next gap. + Keep overlap acceptance and the existing speaker, gap, and token rules. +2. Use one implementation of the existing token estimate across normalization, + chunking, and its consumers. Maintain the components needed for that exact + estimate as a merge grows. Accumulate text and category membership/order + incrementally with the existing joiners. Avoid rescanning or copying a + growing prefix on each append. Retire the unreachable merge helper. +3. Consolidate overlapping nonmerge tests. Keep independent fixtures for + nested intervals, ties, joiners, category ordering, and exact estimates. + Compare allocations or growth on a representative run of same-speaker merges. + +## Stage 7 — Make feasible explicit section targets succeed + +Audit #20 and the infeasibility portion of #24. + +1. In `core/chunking/sections.go`, use indivisible, ordered segments as the + partition units. Preserve the empty-transcript behavior and the singleton + exception for segments individually above the token maximum. Keep + `min_section_tokens` validation and compatibility without adding a new + partition constraint. +2. For an explicit target, compute the farthest legal end for each start + with a rolling token sum, honoring the oversized singleton exception. Work + backward to compute each suffix's minimum section count as one plus the + count at its farthest end. In the forward builder, reserve enough segments + for the remaining nonempty sections and use the suffix count to reject cuts + that leave too little capacity. Prefer the current average-token target + among the feasible cuts. Count sections by segment presence, including + zero-token segments. Preserve implicit-target behavior. +3. Return a typed infeasibility error for impossible requested targets. Carry + it through the runner and CLI so the existing input status applies while + internal and cancellation failures retain their classes. + +Verify the audited weights `[2,5,1]` with target 2 and maximum 6, impossible +targets, and the oversized singleton case. Check contiguity, order, coverage, +and feasibility independently of the chosen cut. Keep a single chunking pass. + +## Stage 8 — Share edit location and protection evidence + +Audit #12, #18 (punctuation portion), #19, #38, #47. + +1. In `framework/proposals`, search for a second literal starting offset for + require-unique replacements, including overlaps. Use the same applicability + rule in previews and apply. Preserve replace-all's nonoverlapping behavior + and the existing blank-result check. +2. Share the positional segment lookup for initial proposals and repair + revisions. Carry raw invalid model references to evidence, keep segment + positions distinct from candidate and validator response indices, and + revalidate a repair that retargets to another segment. +3. Move protected-name comparison to the glossary owner. Compare usable + full-segment before/after text and repeat the check against the current + segment at apply time. A missing or ambiguous preview cannot support a + protection verdict. An available empty preview may support the verdict + independently of blank-result rejection. Preserve active-glossary + exemptions, case and longest-match rules, and spoken-word repetition cleanup. + Use standard literal-edge checks for punctuation-edged spellings while + retaining Go regexp word-boundary semantics. +4. Reuse the glossary scan for context identity membership without expanding + each occurrence/contributor pair. Keep occurrence evidence for protection + and preserve the order of selected context. + +Test overlapping occurrences, full-segment protection, punctuation edges, +ambiguous previews, spoken-word cleanup, and stale apply-time changes at the +owning boundaries. Check the initial and repair lookup against independent +expected references, and compare work on a repeated-glossary fixture. + +## Stage 9 — Consolidate validator decisions and reconciliation + +Audit #10, #11, #27 (runtime class), #40, #49 (validator port). + +1. Use one shared owner for the validator interface, applicability, decisions, + and cardinality helpers. Remove duplicate ports and unused validation-only + envelopes after tracing live consumers. Keep concrete validator chains, + built-in prompt/rule ownership, and reusable LLM batching. +2. Resolve a single authoritative disposition before a candidate advances. + Reject contradictory internal results as fatal contract failures, and derive + redundant serialized approval from the validated disposition. Preserve full + normal cardinality and checked partial results on fatal returns. Preserve + zero-budget shortlisting, independent repair feedback, unavailable + assessments, and deterministic validator ordering. +3. Record the effective execution class selected by the runner on both initial + and revalidation decisions, including supported instance overrides. Keep + the existing default for validators without an explicit class. Document + concurrent-call and borrowed-input behavior at the port and allow + synchronized stateful implementations. +4. Preserve exact numeric references through structured-response decoding + using `json.Number` or raw numeric tokens. Reconcile with the requested + batch-local range before converting to local `int`. Recognize equivalent + integral decimal and exponent forms exactly and ignore extras that cannot + name a requested candidate, including values beyond native integer range. + Keep supplied numeric references in diagnostic evidence. Continue whole + response schema validation and structural repair upstream. One usable + decision keeps its outcome. Omission or duplication affects only its + requested candidate, and ignored extras on a successful batch retain their + existing recovery classification. + +Test with actual response decoding on a mixed batch and one legitimate decision +beside an out-of-range reference. Cover equivalent numeric spellings, +contradictory internal results, and class override from execution through the +record supplied to Stage 17. Update [validator internals](../internal/validators.md). + +## Stage 10 — Reduce validation context work + +Audit #36. + +1. In `framework/validators`, consolidate budget and optional-context policy + across the live validation batch builder and trimmer. Measure the required + payload first and stop testing optional material beyond the usable + ordered prefix. Reuse computations where the exact serialized estimate + permits it. Keep context priority, numbering, effective budget, oversized + single-input behavior, and independently measured wire size. +2. Retain a test oracle built from serialized payloads rather than the shared + estimator. Use a representative long optional-context fixture to demonstrate + reduced measurements or allocation. Retain request-shape and available + sibling expectations under over-budget input. + +## Stage 11 — Hand checked selections to execution + +Audit #24 (readiness), #37, #49 (module port), #50, #55, #56. + +1. Consolidate Audita-owned root and stage validation/fallback helpers in + `core/config` after ConfigKit composition. Preserve presence/null, + explicit zero, inherited values, diagnostic paths and provenance, and + stage/role-dependent defaults. ConfigKit retains its load and override paths. +2. Replace duplicate derivation in `framework/preflight` and + `internal/cli/process_pipeline.go` with one checked processing plan. Store + resolved specs, validated role choices, and effective glossary/protection + views. Cache repeated profile/backend inspections and immutable glossary + views only within the captured runtime. Include ordered selection and + exemptions in reuse keys. Preserve readiness check order, first failure + attribution, protected source paths, no model calls during preflight, and + preflight-before-publication. Return typed status-2 failures for invalid + selections; keep runtime/canceled outcomes distinct. +3. Move built-in construction to composition beside the concrete modules or + in CLI wiring. Keep the injected runner factory and dependency-light key + catalog. Document concurrent calls and borrowed data at the module port. + Copy owned inputs where mutation is required. +4. In processing and inspection request construction, merge shared variables + once per call while preserving each path's precedence, the protected-term + default, and operator replacements for bundled prompts. + +Check representative effective selections and provenance, first-failure +attribution, provider-free inspection, constructible catalog entries, and both +request precedence paths. Update [CLI/config internals](../internal/cli-config.md), +[module internals](../internal/modules.md), and configuration docs where affected. + +## Stage 12 — Unify structured-call completion and failure facts + +Audit #23, #25, #41, #44. + +1. In `framework/llm` and `framework/structuredoutput`, share once-only + completion between ordinary execution and coordinated turns. Keep + prepared snapshots, call IDs, terminal synchronization, native group and + call counts, coverage and unknown usage, and diagnostic-only content from + undelivered responses. Remove obsolete response/usage copies and callback + payloads after checking report and debug consumers. +2. Carry typed class, reason and cause from adapter through operation records. + Use the internal-failure reason when a reason is absent, retaining the + original class and cause. Apply parent cancellation precedence, identify + native logical-operation deadline expiry, then native transport/attempt + timeout before treating a generic deadline as logical timeout. Use the + existing transport-failure reason for HTTP-only timeouts. +3. Make setup diagnostics actionable at the CLI. Surface the native missing + credential requirement and available role/profile context with the typed + cause intact. Keep ordinary reports on their compact typed projections. + +Exercise distinct parent, operation, and attempt timeouts with independently +controlled budgets. Check setup class/cause and useful native information, +plus ordinary/coordinator terminal and undelivered-evidence paths. Update +[LLM runtime internals](../internal/llm-runtime.md). + +## Stage 13 — Use one proposal handoff + +Audit #46. + +1. In `framework/proposal_generation` and `framework/runner`, return one + set of decoded proposal results with raw local references, filtering + evidence, and optional native exchange. Remove the generator's enriched + projection discarded by the live wrapper. Keep runner ownership of + sequential proposal indices within a stage, ordered by chunk and response. +2. Check emitted interaction metadata such as `start_index` separately from + private references. Preserve its meaningful serialized association and + retire fields confirmed to have no surviving purpose. +3. Move proposal-order assertions from the test-only composition harness to + real runner results. Observe out-of-order completion in the fixture and + verify indices and original references. Keep model response and reference + coverage in the generator's owning tests. + +## Stage 14 — Own each correction repair branch explicitly + +Audit #34, #45, #48, #64 (branch integration). + +1. In `framework/runner/repair.go`, give one proposal branch a clear + execution input and result. Make repair capability explicit at setup. Keep + branch state and terminal outcome together. Merge outcomes into the section + after each branch, without mutating unrelated approvals or decisions. +2. Run repairs through PromptKit coordination with full exchange/history and + configured per-correction limits. Preserve initial and subsequent direct + domain feedback, sibling independence, withdrawal, unavailable assessment, + retargeting, and revalidation of the final allowed revision. Move useful + structural repair and isolation tests onto this path before deleting the + direct-continuation-only test branch. +3. Use controlled observations or temporary instrumentation to measure + cumulative turns, available token/byte costs, revalidation calls, and debug + work on a representative multi-correction fixture. Remove unnecessary local + copies. Keep those measurements out of the ordinary report contract. + +Check independent approved siblings, attempts/turn counts, actual branch +history and feedback, final-turn revalidation, and associated artifacts in +production coordination. + +## Stage 15 — Give stage execution one commit boundary + +Audit #14, #15 (evidence), #22 and #41 (failure facts). + +1. In `framework/runner`, check context at entry, including empty pipelines, + and before committing a stage. Use one stage completion path to record + outcomes and evidence. Only a successful active stage advances the working + transcript. Preserve previously committed text for partial results. +2. Carry initiating cause, phase and available stage/chunk/operation reference + from the owner. Cancel and join peers on fatal failure or parent cancellation; + keep peer interruption and secondary errors attributable without promoting + them over the initiating failure. Continue recoverable work under existing + partial-success semantics. +3. Collect completed proposal evidence independently of dispatch to validators. + After workers join, include usable out-of-order completions in interrupted + diagnostics with deterministic indices and associations. Keep raw evidence + from undelivered responses separate from usable proposals. Retain the ordered + indexing/validation boundary until Stage 16 evaluates it. + +Test canceled entry and precommit, a later completed proposal behind a slow +earlier chunk, and a fatal stage after earlier committed text. Assert causal +origin and preserved evidence as well as returned failure. Leave report/ledger +projection to Stage 17. + +## Stage 16 — Bound local preparation and actively drain results + +Audit #15 (dispatch) and #30, under D3. + +1. Replace the per-chunk goroutines and chunk-count-sized pending channels in + `collectSectionProposals` and `runModulePipeline` with a private rolling + stage window. Admit chunks to the window in input order. Charge a slot for + each chunk still in preparation, model work, or awaiting downstream work. + Release it when the chunk is finished and its final facts have been drained. + Start with a small fixed local window and keep it private. Drain outcomes + while workers run and on cancellation, then join and finish. Keep compact + or requested completed evidence apart from remaining work. Collect results + while the workers can still send them. +2. Measure staged overlap and peak pending work with many sections, a + controlled slow first proposal, and shared PromptKit capacity. Where active + draining and ordered indexing yield adequate progress, keep that simpler + boundary. If ordered indexing still leaves repeatable idle capacity with + validation-ready chunks, use a chunk index and response position to identify + candidates during work. At ordered collection, assign the existing stage + indices and translate every projected proposal reference, including failure + indices, repair lineage, and interaction-to-ledger mappings. Keep the + identity-sensitive artifact data in the charged window until the index is + assigned; use the Stage 3 writer for the final artifacts. Keep batch-local + model references and raw response evidence intact. Check the same mappings + after a cancellation joins the workers. +3. Keep the module barrier and deterministic application. PromptKit retains + admission, queueing, limits, retries and recoverable rejection. Include + nested validation/repair in the local resource and cancellation checks + without duplicating backend admission or holding its permits across a + nested call. Tune the window using the measured workload and contention; + no public window, throughput minimum, or eventual-admission guarantee follows. + +Test actual observed overlap, finite native contention, cancellation with work +pending, and identical index ordering for the same usable results under +different completion orders. Report representative pending-work measurements +without pinning a timing or worker-count threshold in the default suite. + +## Stage 17 — Project typed report and ledger facts + +Audit #22, #27 (ledger), #39, #41 (projection). + +1. In `framework/processreport`, build the snapshot from the Stage 12 and 15 + facts and Stage 4 delivery outcomes. Preserve phase and available origin for + setup, processing and publication errors. Keep recovery and work-unit counts + and known versus unknown values. Distinguish the primary run error, + interrupted peers, and secondary output failures without counting a fatal + run failure again as a recoverable unit failure. +2. Join proposals, validator decisions, repairs, application and skip evidence + once by stable identities in `correction_ledger.go`. Use the recorded + execution class from Stage 9. Project one candidate row with correct + disposition, lineage and literal empty replacement values. Retain + module-only failures. Preserve serialized fields and logical ordering. +3. In runner application attribution, reuse section/segment lookups and + aggregate skipped evidence once. Preserve the distinction between stale + apply-time conflicts and response filtering. Keep exact correction, + filtered-proposal and validation-decision identities across projection. + +Verify decoded YAML, Markdown counts, and ledger facts against an independently +specified mixed run, including partial decisions, repair/filter interaction, +later failure, and an overridden validator class. Update the +[output contract](../integrations/output-files.md) and diagnostics internals. + +## Stage 18 — Retain evidence according to selected outputs + +Audit #32 and supporting work for #30, #33, and #44. + +1. Derive evidence needs once from enabled ordinary reports and effective debug + capture. Keep compact CLI totals and available failure facts in summary-only + mode. Allocate work-unit detail only when a consumer needs it, and avoid + eager whole-run grids and debug-only candidate/decision histories in + summary-only paths. Keep complete enabled report matrices and debug evidence. +2. Discard active validation, exchange and repair state after their last + consumer, including data copied solely for obsolete runner projections. + Remove duplicate operation lists and unused module totals after checking + their projections. Keep native observations and requested self-contained + artifacts when capture is selected. +3. Compare live/retained state for representative long runs with summary only, + ordinary reports, debug, and their existing combinations. Check shared + totals and logical results across modes, with detail present where selected. + Resource measurements support reduction of waste without setting a + constant-memory or disk-use requirement. + +## Stage 19 — Emit native processing progress and update operator guidance + +Audit #5, #57, and the operator-facing parts of #8, #24 and #25. + +1. Use a small event callback from the live lifecycle owners. Supply stage and + chunk activity, proposal/validation/repair phases and useful completions + with available identities and observed counts. Reuse suitable PromptKit + events where they convey useful activity. Keep event delivery within the + invocation and finish workers before the reporter closes. +2. Wire that callback in CLI to `Invocation.Reporter().Progress` in pinned + CommandKit. Let CommandKit determine visibility, routing, and presentation + failure status, independently of transcript/report processing outcomes. + Preserve stdout transcript data. Respect ordinary mode, explicit progress, + debug defaults, explicit `--progress=false`, and quiet suppression. +3. Update the [CLI reference](../cli.md), [operations](../operations.md) and + [subprocess contract](../integrations/subprocess.md) at their owned + boundaries. Cover meaningful progress, caller-owned report destinations and + artifact retention, and the trusted configuration bundle, including + discovery, imports and automatic companions. Keep the distinction between + native configuration inspection and processing readiness. + +Use representative command tests for live activity and combined modes, +asserting useful identities and stream separation without exact wording or +concurrent ordering. Keep a progress-specific check separate from delegated +inspection availability. + +## Stage 20 — Verify the integrated sprint and release bootstrap + +Audit #28, #29, #58, #60–62 and sprint closure. + +1. In `.woodpecker/release.yml`, capture `git status --porcelain --untracked-files=all` + in an assignment that checks the command's exit status separately from its + content. Then require empty output. Preserve ReleaseKit's independent + prepared-checkout validation and tag delivery. In `scripts/release_test.go`, + cover clean, dirty and failed-status-with-empty-output behavior at the + command boundary. Remove redundant tests of the comparison helper while + keeping real release target and wiring checks. +2. Verify signal, cancellation and exit behavior through the real executable + or a shared production entry point. Keep narrower checks where a process + adds no evidence. Verify effective validator roles, required prompt + identities, embedded asset preparability, schema identity, and protective + copies through use. Retain a representative repair/filter/later-failure + integration scenario and independent output-choice cases. Remove the + audited abandoned fixture inputs and helpers after checking callers. +3. Recheck Stage 1. Adopt and verify a released upstream fix that became + available using the same scope. Otherwise report the exact outstanding + upstream-dependent behavior and its disposition. Treat that item as + outstanding, while verifying the independent Audita work. Keep the + invalid-input and cancellation cases distinct. +4. Account for the [catalog](audit/catalog.md) against the landed changes and + stage evidence. Entries #2 and #4 need no implementation, and only the + accepted portions of #6 and #18 were selected. Check canonical docs, + examples, and affected links and remove stale implemented-behavior claims. + Keep historical audit observations intact. +5. From the integrated tree, build the CLI and run the full offline + `go test ./...` with the supported Go toolchain. Run targeted race checks + for changed shared-state paths and review the representative measurements + from Stages 6, 8, 10, 14, 16 and 18. Verify formatting and whitespace, + and inspect the final diff. Reuse successful checks for unchanged content. + Existing [release validation](../policy/release.md#candidate-validation) + applies when a release is prepared. + +The completed sprint should leave the target architecture reachable through +`audita process` and the standard delegated commands. Report checks and +measurements actually performed, including any outstanding external dependency. diff --git a/docs/roadmap/reliability-and-maintainability.md b/docs/roadmap/reliability-and-maintainability.md index 7d177ee..f26e8b1 100644 --- a/docs/roadmap/reliability-and-maintainability.md +++ b/docs/roadmap/reliability-and-maintainability.md @@ -15,10 +15,10 @@ detail through CommandKit. This roadmap organizes the accepted audit remediation around that target state. The [assessment](audit/assessment.md) owns the disposition of individual findings and the rationale for decisions D1–D7; the [catalog](audit/catalog.md) links to -their evidence. The coverage appendix below maps that evidence to this plan. +their evidence. The coverage appendix below maps that evidence to the target areas. Earlier source-audit remedies remain proposals where the assessment supersedes -them. This document specifies outcomes and dependencies; a later implementation -plan will choose reviewable changes, concrete APIs, and scheduling mechanics. +them. This document specifies the target state and accepted scope. The +[implementation plan](implementation.md) owns the sequence and verification. The current application already has native library integration, preflight, ordered module execution, partial-success validation, bounded correction repair, @@ -163,6 +163,9 @@ identity. application, and the module barrier. Reduce avoidable within-stage waiting without introducing independent whole-pipeline chunk traversal. Retain completed proposal evidence on interruption even when an earlier section is unfinished. + Preserve section- and response-ordered serialized proposal indices. If + measured waiting warrants earlier validation, use local candidate references + during processing and assign serialized indices at ordered collection. PromptKit's configured rejection remains valid; the old scheduling-dependent choice of rejected chunks is not a compatibility requirement. - Retain only the state needed by active processing and selected outputs. Keep @@ -228,8 +231,9 @@ can discard usable decisions because of irrelevant extra references. end-time sorting criterion. Build growing text/category state incrementally, preserving exact joins, category order, and the current token estimate. - Satisfy feasible explicit chunk count and maximum-size constraints before - optimizing balance. Choose a proportionate deterministic algorithm and retain - the configured budgets and current balancing objective. Keep accepted + optimizing balance. Preserve the single-segment exception above the token + maximum. Use a proportionate deterministic algorithm and retain the + configured budgets and current balancing objective. Keep accepted compatibility settings, including `chunking.min_section_tokens`, even where they are not active partition targets. - Give validator applicability and decision meaning one owner and one runtime @@ -339,9 +343,10 @@ boundary that the fixes should follow. - Classify invalid processing selections and genuinely infeasible chunk requests according to the existing input-error contract; retain runtime classification for I/O and cancellation. Trace delegated inspection failures to Audita's - loader or the upstream integration, then fix the responsible owner. An upstream - defect should be addressed upstream rather than through replacement command - routing. + loader or the upstream integration. The pinned ckprompt catalog handler + classifies malformed prompt configuration as an operational failure. Fix + that classification upstream while keeping native command routing. + Processing progress can use CommandKit's existing reporter independently. - Surface actionable native setup detail, including the configured missing credential requirement and available role/profile context, while preserving typed causes and failure classification. Remove CLI substitution of errors, @@ -412,38 +417,6 @@ checks for unchanged content. Link/example/whitespace checks accompany relevant documentation changes. Do not substitute coverage percentages or a collection of test counts for evidence of the selected outcomes. -## Sequencing and implementation planning - -| Stage | Reviewable outcome | Dependencies and rationale | -| --- | --- | --- | -| A. Simplify evidence and publication | Direct serialization and credential-plumbing removal; clear required-output/debug ownership; destination, failed-report, and cleanup fixes. | Removes obsolete constraints before building shared writers. Preserve useful tests while deleting their old mechanisms. | -| B. Establish domain correctness and shared owners | Parsing, normalization, chunking, protection, decision reconciliation, and shared model/configuration helpers. | Pure domain work can proceed alongside A. Establish authoritative decisions and data shapes before changing orchestration around them. | -| C. Consolidate operation and stage lifecycle | One proposal handoff, typed failure facts, explicit stage completion and repair branches, checked-plan reuse, faithful report/ledger projection. | Uses the shared domain and diagnostic boundaries; moves useful coverage off obsolete execution paths. | -| D. Bound local work and retain only needed state | Bounded preparation/dispatch, active result draining, output-sensitive retention, and measured efficiency improvements. | Build on C's explicit ownership so narrowing queues does not deadlock producers or discard evidence. Optimize independent domain scans as their owners are established. | -| E. Complete operator feedback and verification | Detailed native progress, actionable setup/input errors, canonical docs, streamlined integration tests, and release-bootstrap correction. | Progress consumes the real lifecycle from C/D. Error ownership investigation and the independent bootstrap fix can start earlier. Tests and docs accompany every stage; this stage closes integration gaps. | - -This is a dependency order, not a requirement for five large commits. The later -implementation plan should split coherent fixes and supporting refactors into -reviewable units. Significant refactoring is appropriate when it improves the -selected ownership or resource behavior; an audit number is not a commit boundary. - -Resolve these engineering details during planning/implementation: - -- Select and measure the local work/result window, including nested validation - and repair interactions. No exact worker count or new operator setting is - prescribed by this roadmap. -- Identify the owner of delegated inspection classification and verify the - available CommandKit reporting hooks. If an upstream change is required, record - its concrete dependency and adopt the supporting release before that integration; - independent work can continue. -- Measure aggregate repair and debug costs using existing observations or focused - instrumentation. Improve local representations while preserving full history, - configured repair limits, and requested self-contained artifacts. - -These are technical investigations within resolved scope. No significant product -decision remains open. A concrete discovery requiring a substantive policy change -should be brought back for discussion rather than silently expanding this roadmap. - ## Completion and audit coverage The roadmap is complete when the target behaviors above are implemented and