639 lines
38 KiB
Markdown
639 lines
38 KiB
Markdown
# Reliability and Maintainability Implementation Plan
|
||
|
||
Implement the [target roadmap](reliability-and-maintainability.md) in one
|
||
end-to-end sprint, starting from its application baseline `c4ccd78`. The
|
||
[assessment](audit/assessment.md) governs the accepted scope and decisions
|
||
D1–D7. Numbers under each stage refer to the [audit catalog](audit/catalog.md).
|
||
Use the original audits for reproductions and source references. Where an old
|
||
remedy differs from a later decision, follow the assessment and target roadmap.
|
||
|
||
Work through the numbered stages in order. Each stage is sized for one Luna
|
||
implementation prompt and contributes directly to the final design. Intermediate
|
||
source may be incomplete or fail to build while adjacent changes are in flight.
|
||
Use the next stage to finish those connections. A stage needs neither a
|
||
standalone commit nor a passing full suite. Replace obsolete paths directly;
|
||
compatibility layers and temporary production adapters are unnecessary.
|
||
|
||
Use the [development guide](../development.md) and applicable adopted policies,
|
||
especially [agents](../policy/agents.md), [architecture](../policy/architecture.md),
|
||
[testing](../policy/testing.md), and [documentation](../policy/documentation.md).
|
||
Add or adapt focused tests at the behavior owner and update canonical docs when
|
||
their implemented behavior changes. Run affected tests when the relevant path
|
||
builds, then close the sprint with Stage 20. Stage checks establish behavior
|
||
without fixing private names, scheduler sizes, event order, or prompt wording.
|
||
The target roadmap defines retained contracts, selected changes, and exclusions.
|
||
|
||
## Stage 1 — Establish the delegated inspection dependency
|
||
|
||
Audit #24, delegated part.
|
||
|
||
1. Use a temporary CLI build to reproduce invalid operator configuration with
|
||
`promptkit list prompts` and `process`. Reuse the malformed companion fixture
|
||
in `internal/cli/runtime_errors_test.go` and include a separate cancellation
|
||
case. The baseline reported status 1 for the malformed companion via
|
||
inspection and status 2 via processing. Both commands must remain offline.
|
||
2. Trace `internal/cli/app_wiring.go` into the pinned `ckprompt` and `ckconfig`
|
||
commands. For the known catalog case, `ckprompt.runCatalogList` converts the
|
||
loader error to `definitionFailure` even when a typed input cause is
|
||
available. Check relevant configuration-scope acquisition paths separately.
|
||
Preserve typed causes in Audita's loader. Classify only input failures at
|
||
the responsible boundary, leaving cancellation and runtime failures distinct.
|
||
3. Check for a released upstream fix for each demonstrated delegated case. If a
|
||
suitable release exists, adopt the smallest compatible update using the
|
||
[OSS policy](../policy/oss.md) and verify classification at the Audita CLI.
|
||
Otherwise record the affected command, reproduction, owner, and needed
|
||
behavior for an upstream handoff. Continue the independent stages and use
|
||
Stage 20 to integrate and verify a supporting release if available. Keep the
|
||
native declarations and routing throughout.
|
||
|
||
A fix is demonstrated by status 2 and actionable diagnostics for the invalid
|
||
input, and status 1 for cancellation. While the supporting release is pending,
|
||
record the mismatch explicitly. A test that asserts only nonzero status does
|
||
not verify this fix.
|
||
|
||
**Complete.** The upstream handoff is recorded below. The exit-status fix
|
||
remains outstanding for the final verification pass.
|
||
|
||
Build a temporary executable with `go build -o /tmp/audita-inspection ./cmd/audita`.
|
||
In a scratch directory, put `version: 3` in `config.yml`, `profiles: [` in the
|
||
adjacent `prompts.yml`, and `[]` in `transcript.json`. End each file with a newline.
|
||
|
||
| Invocation with `/tmp/audita-inspection` | Status | Diagnostic |
|
||
| --- | --- | --- |
|
||
| `promptkit list prompts --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` |
|
||
| `process transcript.json --config config.yml` | 2 | `yaml.invalid` in `prompts.yml` |
|
||
| `config show --scope promptkit --config config.yml` | 1 | `yaml.invalid` in `prompts.yml` |
|
||
|
||
`config sources`, `config validate`, `config explain --list`, and
|
||
`config diff --against-config config.yml` also returned 1 for the `promptkit`
|
||
scope with the same diagnostic. All five `app` scope operations succeeded
|
||
with the malformed companion and returned 1 with `yaml.invalid` for a
|
||
separately selected main file containing `version: [`. These failures occur
|
||
during configuration loading, before provider work. The cancellation test
|
||
passes an already canceled context through the application entry point and
|
||
verifies status 1, `context.Canceled`, and empty stdout for catalog listing,
|
||
configuration show, and processing.
|
||
|
||
Both native adapters receive the typed ConfigKit load cause from Audita's
|
||
existing loaders. In `ckprompt`, `runCatalogList` assigns the operational
|
||
outcome through `definitionFailure`. In `ckconfig`, the scope handlers
|
||
assign it through `configFailure`, with `show` calling it through
|
||
`configAcquisitionFailure`. The upstream owners are the respective CommandKit
|
||
integrations. They need to assign status 2 for typed operator-input failures
|
||
at those boundaries, preserving diagnostics, available inspection evidence,
|
||
and the cause. Cancellation and other operational failures need status 1.
|
||
The upstream tag list contained releases through `v0.7.0` for each
|
||
integration, which are already pinned. Recheck both in the final stage.
|
||
|
||
## Stage 2 — Remove application credential substitution
|
||
|
||
Audit #3, #9, #31 (source bytes), #63–65. Use the
|
||
[focused removal inventory](audit/14-redaction.md) for affected sites.
|
||
|
||
1. Switch ordinary reports and debug artifacts to direct typed JSON/YAML
|
||
encoding. Keep encoded effective configuration from ConfigKit. In Markdown,
|
||
retain escaping and remove substitution. Keep malformed `final_content` as
|
||
the captured string. Write the acquired transcript bytes to the debug source
|
||
artifact and keep the parsed and normalized views separate.
|
||
2. Send domain `repairFindings` directly for initial feedback and subsequent
|
||
findings. Preserve the proposal association and typed unavailable assessment.
|
||
Present native errors, warnings, and actual paths through CommandKit. This
|
||
step removes substitution. Stage 12 exposes the native setup detail that is
|
||
currently hidden by generic failure presentation.
|
||
3. Remove writer/renderer secret arguments, `DebugInput.Secrets`, discovery
|
||
assertions and closures, repair role-list combination, cache and role
|
||
tracking, runtime secret discovery, and `internal/core/redaction`. Keep the
|
||
prepared and terminal callbacks, source-path tracking, native effective
|
||
rendering, and selected provider authentication.
|
||
4. Delete masking-only assertions, scanners, fixtures, and stale redaction
|
||
descriptions identified in the inventory. Move independent assertions to
|
||
writer or caller tests. Check decoded schema and scalar types, literal source
|
||
bytes, malformed response evidence, actual paths, and native auth using
|
||
synthetic data. Check feedback by decoding a production repair request.
|
||
|
||
**Complete.** Reports and debug artifacts encode typed data directly, source
|
||
snapshots preserve the acquired bytes, and repair requests carry the original
|
||
domain findings. Removed application credential discovery and substitution.
|
||
The CLI build and full offline suite passed with Go 1.27.3, including checks
|
||
for native authentication, malformed response capture, and destination paths.
|
||
|
||
## Stage 3 — Unify optional interaction capture
|
||
|
||
Audit #33 and #43.
|
||
|
||
1. Define one small optional interaction sink at the shared execution port.
|
||
Compose the existing filesystem writer from the run directory in CLI wiring.
|
||
Pass the sink through proposal, validation, and repair. Each role supplies
|
||
its own payload and stage/chunk/operation coordinates, including proposal,
|
||
batch, validator, and repair turn where applicable. Keep returned paths and
|
||
write errors available for ledger association and warnings.
|
||
2. Reuse the write path and artifact envelope in
|
||
`internal/framework/llm/diagnostics.go`. Remove private writer adapters and
|
||
repeated construction. Keep artifacts self-contained, including revalidation
|
||
and a final unusable or undelivered response when available. Keep debug
|
||
directory placement, naming safety, and creation modes.
|
||
3. Collect optional write warnings safely across workers, grouping equivalent
|
||
failures by operation and cause with counts and affected artifact information.
|
||
Continue later capture after a write error. Keep the warning path outside
|
||
correction decisions and required file delivery.
|
||
|
||
Verify role, chunk, batch, proposal, and attempt associations with decoded
|
||
artifacts. Exercise a failed optional write followed by later capture. Update
|
||
the [diagnostics internals](../internal/diagnostics-reporting.md) for the sink.
|
||
|
||
**Complete.** The CLI supplies one optional sink for proposal, validation,
|
||
repair, and revalidation artifacts. The writer records successful paths even
|
||
after a partial write and groups failures by role, artifact operation, and
|
||
cause. Decoded-artifact and recovery checks passed, along with the full Go
|
||
suite and targeted race checks.
|
||
|
||
## Stage 4 — Bind publication to checked destinations
|
||
|
||
Audit #1, #6 (cooperative boundaries), #7, #8, #16, #26, #42.
|
||
|
||
1. In `internal/core/diagnostics/output_session.go` and its CLI callers, resolve
|
||
checked transcript and report destinations with filesystem semantics.
|
||
Carry those destination and parent choices through staging and publication.
|
||
Keep protected-source checks, intentional in-place replacement, ordinary
|
||
symlink and stream behavior, and disabled reporting's inactive destination.
|
||
Add no directory creation or locking for required destinations.
|
||
2. Give the required-output owner a shared regular-file staging primitive for
|
||
transcript and reports. Keep optional debug setup separate from that owner.
|
||
Keep temporary paths until removal succeeds or absence is established.
|
||
Surface cleanup errors alongside the initiating error, including after
|
||
partial preparation, publication failure, or cancellation.
|
||
3. Make report rendering return an explicit error through initialization and
|
||
finalization. Publish the transcript first when processing succeeded.
|
||
Select a failed bundle before the first final report write when a required
|
||
delivery or cleanup failure is already known. If a report write first
|
||
establishes failure, switch the remaining writes to failed status and make
|
||
one best-effort pass over reports that still need failed evidence. If the
|
||
failed bundle cannot be rendered, keep initialized or published evidence
|
||
and return the rendering error.
|
||
4. Check context when transcript acquisition returns, at natural preparation
|
||
and publication boundaries, and after stream delivery returns. A cancellation
|
||
observed after bytes escape still fails the invocation. Join cancellation
|
||
and I/O/cleanup errors where available and keep the initiating failure
|
||
distinguishable. Use ordinary blocking I/O and retain completed transcript
|
||
publication if a later report write fails.
|
||
5. Escape input/output paths as literal Markdown text in `core/reporting`,
|
||
preserving raw values in structured artifacts. Update
|
||
[operations](../operations.md) for caller-owned destination isolation,
|
||
retention, and the resulting publication lifecycle.
|
||
|
||
Exercise symlink/path resolution, in-place and stream output, cancellation
|
||
after I/O, cleanup errors, initialized evidence when rendering fails, and
|
||
write behavior after known failure. Observe publication attempts as well as
|
||
final files. Leave processing fact and ledger projection to Stage 17.
|
||
|
||
**Complete.** Required outputs use the destinations resolved during selection
|
||
and share regular-file staging. Publication checks cancellation, retains and
|
||
reports cleanup failures, and uses failed-status reports when required delivery
|
||
cannot complete. Tests verified publication attempts, retained evidence when
|
||
rendering failed, symlink destinations, stream cancellation, and Markdown path
|
||
escaping. The CLI build and full offline Go suite passed.
|
||
|
||
## Stage 5 — Consolidate typed transcript data
|
||
|
||
Audit #13, #31, #51, #52, #54, #62 (file helper).
|
||
|
||
1. In `core/schema`, detect the two accepted root shapes while keeping the
|
||
segment array as raw JSON. Decode typed segments directly. Preserve source
|
||
ID presence, native integer range, duplicate checks, ignored unknown fields,
|
||
and existing format and error behavior. Keep one live source-to-canonical
|
||
conversion and retire obsolete parser wrappers after moving meaningful tests.
|
||
2. Centralize transcript cloning in the model. Use it at the runner and other
|
||
copy boundary, preserving independence and nil versus explicit empty
|
||
mutable slices. Share one typed segment projection across both encoders in
|
||
`core/schema` and `core/outputschema`. Preserve the envelopes, version,
|
||
numeric fidelity, category and omission rules, and encoding error behavior.
|
||
3. Release raw input after its final parse or source-snapshot write. Drop the
|
||
parsed source after normalization and its debug consumers. Avoid retaining
|
||
it via CLI finalization. Remove the audited unused file helper after checking
|
||
callers. Keep both file and stream inputs and current output delivery
|
||
semantics. Use a representative large fixture to check that the removed
|
||
representations no longer overlap needlessly. A fully streaming encoder is
|
||
optional if supported by the measurements and existing error semantics.
|
||
|
||
Test both envelopes, adjacent accepted IDs beyond exact float precision where
|
||
supported by native `int`, clone independence with independent expected data,
|
||
and the distinction between raw and typed debug snapshots. Update the
|
||
[input contract](../integrations/transcript-glossary-files.md) and
|
||
[encoder internals](../internal/output-schemas.md) where affected.
|
||
|
||
**Complete.** Source segments decode directly from both accepted JSON shapes.
|
||
Exact IDs, clone independence, encoder values, and debug snapshot fidelity were
|
||
verified. Processing releases raw bytes after parsing or source capture and
|
||
releases the parsed view before normalization. On a 100,000-segment object
|
||
fixture (Go 1.27.3, five iterations), parser allocations fell from 240.6 MB to
|
||
84.1 MB per operation. The CLI build and full offline suite passed.
|
||
|
||
## Stage 6 — Correct and simplify normalization
|
||
|
||
Audit #17, #21, #35, #53, #59, #62 (merge helper).
|
||
|
||
1. In `core/normalization`, sort stably by the existing time key and retain
|
||
input order for ties. Compute the prospective merge end as the later end
|
||
and use that interval for the duration limit, merged segment, and next gap.
|
||
Keep overlap acceptance and the existing speaker, gap, and token rules.
|
||
2. Use one implementation of the existing token estimate across normalization,
|
||
chunking, and its consumers. Maintain the components needed for that exact
|
||
estimate as a merge grows. Accumulate text and category membership/order
|
||
incrementally with the existing joiners. Avoid rescanning or copying a
|
||
growing prefix on each append. Retire the unreachable merge helper.
|
||
3. Consolidate overlapping nonmerge tests. Keep independent fixtures for
|
||
nested intervals, ties, joiners, category ordering, and exact estimates.
|
||
Compare allocations or growth on a representative run of same-speaker merges.
|
||
|
||
**Complete.** Verified stable start-time ties, nested intervals, exact joiner
|
||
and token behavior, and category order. The merge uses incremental text,
|
||
categories, and the shared token estimator. On synthetic same-speaker runs
|
||
(Go 1.27.3, three iterations each), allocations fell from 238.1 MB to 0.42 MB
|
||
per operation for 2,000 segments and from 960.1 MB to 0.85 MB for 4,000. The
|
||
CLI build and full offline suite passed.
|
||
|
||
## Stage 7 — Make feasible explicit section targets succeed
|
||
|
||
Audit #20 and the infeasibility portion of #24.
|
||
|
||
1. In `core/chunking/sections.go`, use indivisible, ordered segments as the
|
||
partition units. Preserve the empty-transcript behavior and the singleton
|
||
exception for segments individually above the token maximum. Keep
|
||
`min_section_tokens` validation and compatibility without adding a new
|
||
partition constraint.
|
||
2. For an explicit target, compute the farthest legal end for each start
|
||
with a rolling token sum, honoring the oversized singleton exception. Work
|
||
backward to compute each suffix's minimum section count as one plus the
|
||
count at its farthest end. In the forward builder, reserve enough segments
|
||
for the remaining nonempty sections and use the suffix count to reject cuts
|
||
that leave too little capacity. Prefer the current average-token target
|
||
among the feasible cuts. Count sections by segment presence, including
|
||
zero-token segments. Preserve implicit-target behavior.
|
||
3. Return a typed infeasibility error for impossible requested targets. Carry
|
||
it through the runner and CLI so the existing input status applies while
|
||
internal and cancellation failures retain their classes.
|
||
|
||
Verify the audited weights `[2,5,1]` with target 2 and maximum 6, impossible
|
||
targets, and the oversized singleton case. Check contiguity, order, coverage,
|
||
and feasibility independently of the chosen cut. Keep a single chunking pass.
|
||
|
||
**Complete.** Explicit targets use rolling legal ends and suffix feasibility to
|
||
admit exactly the requested count when possible. An independent enumeration
|
||
checked 1,024 weight sequences for every possible count, including zero-token
|
||
and oversized segments. The real CLI returned status 2 for infeasibility
|
||
without dispatching model work. The CLI build and full offline suite passed.
|
||
|
||
## Stage 8 — Share edit location and protection evidence
|
||
|
||
Audit #12, #18 (punctuation portion), #19, #38, #47.
|
||
|
||
1. In `framework/proposals`, search for a second literal starting offset for
|
||
require-unique replacements, including overlaps. Use the same applicability
|
||
rule in previews and apply. Preserve replace-all's nonoverlapping behavior
|
||
and the existing blank-result check.
|
||
2. Share the positional segment lookup for initial proposals and repair
|
||
revisions. Carry raw invalid model references to evidence, keep segment
|
||
positions distinct from candidate and validator response indices, and
|
||
revalidate a repair that retargets to another segment.
|
||
3. Move protected-name comparison to the glossary owner. Compare usable
|
||
full-segment before/after text and repeat the check against the current
|
||
segment at apply time. A missing or ambiguous preview cannot support a
|
||
protection verdict. An available empty preview may support the verdict
|
||
independently of blank-result rejection. Preserve active-glossary
|
||
exemptions, case and longest-match rules, and spoken-word repetition cleanup.
|
||
Use standard literal-edge checks for punctuation-edged spellings while
|
||
retaining Go regexp word-boundary semantics.
|
||
4. Reuse the glossary scan for context identity membership without expanding
|
||
each occurrence/contributor pair. Keep occurrence evidence for protection
|
||
and preserve the order of selected context.
|
||
|
||
Test overlapping occurrences, full-segment protection, punctuation edges,
|
||
ambiguous previews, spoken-word cleanup, and stale apply-time changes at the
|
||
owning boundaries. Check the initial and repair lookup against independent
|
||
expected references, and compare work on a repeated-glossary fixture.
|
||
|
||
**Complete.** Proposal preview and application agree on overlapping spans.
|
||
Initial and repair responses share positional lookup and retain original model
|
||
references in diagnostic evidence. Protection runs on usable full-segment
|
||
previews and checks the current text at application. The 2,000-mention
|
||
fixture with 128 shared contributors measured 81.0 MB to 0.27 MB allocated
|
||
and 77.8 ms to 2.1 ms per operation (Go 1.27.3, ten iterations). The CLI build
|
||
and full offline suite passed.
|
||
|
||
## Stage 9 — Consolidate validator decisions and reconciliation
|
||
|
||
Audit #10, #11, #27 (runtime class), #40, #49 (validator port).
|
||
|
||
1. Use one shared owner for the validator interface, applicability, decisions,
|
||
and cardinality helpers. Remove duplicate ports and unused validation-only
|
||
envelopes after tracing live consumers. Keep concrete validator chains,
|
||
built-in prompt/rule ownership, and reusable LLM batching.
|
||
2. Resolve a single authoritative disposition before a candidate advances.
|
||
Reject contradictory internal results as fatal contract failures, and derive
|
||
redundant serialized approval from the validated disposition. Preserve full
|
||
normal cardinality and checked partial results on fatal returns. Preserve
|
||
zero-budget shortlisting, independent repair feedback, unavailable
|
||
assessments, and deterministic validator ordering.
|
||
3. Record the effective execution class selected by the runner on both initial
|
||
and revalidation decisions, including supported instance overrides. Keep
|
||
the existing default for validators without an explicit class. Document
|
||
concurrent-call and borrowed-input behavior at the port and allow
|
||
synchronized stateful implementations.
|
||
4. Preserve exact numeric references through structured-response decoding
|
||
using `json.Number` or raw numeric tokens. Reconcile with the requested
|
||
batch-local range before converting to local `int`. Recognize equivalent
|
||
integral decimal and exponent forms exactly and ignore extras that cannot
|
||
name a requested candidate, including values beyond native integer range.
|
||
Keep supplied numeric references in diagnostic evidence. Continue whole
|
||
response schema validation and structural repair upstream. One usable
|
||
decision keeps its outcome. Omission or duplication affects only its
|
||
requested candidate, and ignored extras on a successful batch retain their
|
||
existing recovery classification.
|
||
|
||
Test with actual response decoding on a mixed batch and one legitimate decision
|
||
beside an out-of-range reference. Cover equivalent numeric spellings,
|
||
contradictory internal results, and class override from execution through the
|
||
record supplied to Stage 17. Update [validator internals](../internal/validators.md).
|
||
|
||
## Stage 10 — Reduce validation context work
|
||
|
||
Audit #36.
|
||
|
||
1. In `framework/validators`, consolidate budget and optional-context policy
|
||
across the live validation batch builder and trimmer. Measure the required
|
||
payload first and stop testing optional material beyond the usable
|
||
ordered prefix. Reuse computations where the exact serialized estimate
|
||
permits it. Keep context priority, numbering, effective budget, oversized
|
||
single-input behavior, and independently measured wire size.
|
||
2. Retain a test oracle built from serialized payloads rather than the shared
|
||
estimator. Use a representative long optional-context fixture to demonstrate
|
||
reduced measurements or allocation. Retain request-shape and available
|
||
sibling expectations under over-budget input.
|
||
|
||
## Stage 11 — Hand checked selections to execution
|
||
|
||
Audit #24 (readiness), #37, #49 (module port), #50, #55, #56.
|
||
|
||
1. Consolidate Audita-owned root and stage validation/fallback helpers in
|
||
`core/config` after ConfigKit composition. Preserve presence/null,
|
||
explicit zero, inherited values, diagnostic paths and provenance, and
|
||
stage/role-dependent defaults. ConfigKit retains its load and override paths.
|
||
2. Replace duplicate derivation in `framework/preflight` and
|
||
`internal/cli/process_pipeline.go` with one checked processing plan. Store
|
||
resolved specs, validated role choices, and effective glossary/protection
|
||
views. Cache repeated profile/backend inspections and immutable glossary
|
||
views only within the captured runtime. Include ordered selection and
|
||
exemptions in reuse keys. Preserve readiness check order, first failure
|
||
attribution, protected source paths, no model calls during preflight, and
|
||
preflight-before-publication. Return typed status-2 failures for invalid
|
||
selections; keep runtime/canceled outcomes distinct.
|
||
3. Move built-in construction to composition beside the concrete modules or
|
||
in CLI wiring. Keep the injected runner factory and dependency-light key
|
||
catalog. Document concurrent calls and borrowed data at the module port.
|
||
Copy owned inputs where mutation is required.
|
||
4. In processing and inspection request construction, merge shared variables
|
||
once per call while preserving each path's precedence, the protected-term
|
||
default, and operator replacements for bundled prompts.
|
||
|
||
Check representative effective selections and provenance, first-failure
|
||
attribution, provider-free inspection, constructible catalog entries, and both
|
||
request precedence paths. Update [CLI/config internals](../internal/cli-config.md),
|
||
[module internals](../internal/modules.md), and configuration docs where affected.
|
||
|
||
## Stage 12 — Unify structured-call completion and failure facts
|
||
|
||
Audit #23, #25, #41, #44.
|
||
|
||
1. In `framework/llm` and `framework/structuredoutput`, share once-only
|
||
completion between ordinary execution and coordinated turns. Keep
|
||
prepared snapshots, call IDs, terminal synchronization, native group and
|
||
call counts, coverage and unknown usage, and diagnostic-only content from
|
||
undelivered responses. Remove obsolete response/usage copies and callback
|
||
payloads after checking report and debug consumers.
|
||
2. Carry typed class, reason and cause from adapter through operation records.
|
||
Use the internal-failure reason when a reason is absent, retaining the
|
||
original class and cause. Apply parent cancellation precedence, identify
|
||
native logical-operation deadline expiry, then native transport/attempt
|
||
timeout before treating a generic deadline as logical timeout. Use the
|
||
existing transport-failure reason for HTTP-only timeouts.
|
||
3. Make setup diagnostics actionable at the CLI. Surface the native missing
|
||
credential requirement and available role/profile context with the typed
|
||
cause intact. Keep ordinary reports on their compact typed projections.
|
||
|
||
Exercise distinct parent, operation, and attempt timeouts with independently
|
||
controlled budgets. Check setup class/cause and useful native information,
|
||
plus ordinary/coordinator terminal and undelivered-evidence paths. Update
|
||
[LLM runtime internals](../internal/llm-runtime.md).
|
||
|
||
## Stage 13 — Use one proposal handoff
|
||
|
||
Audit #46.
|
||
|
||
1. In `framework/proposal_generation` and `framework/runner`, return one
|
||
set of decoded proposal results with raw local references, filtering
|
||
evidence, and optional native exchange. Remove the generator's enriched
|
||
projection discarded by the live wrapper. Keep runner ownership of
|
||
sequential proposal indices within a stage, ordered by chunk and response.
|
||
2. Check emitted interaction metadata such as `start_index` separately from
|
||
private references. Preserve its meaningful serialized association and
|
||
retire fields confirmed to have no surviving purpose.
|
||
3. Move proposal-order assertions from the test-only composition harness to
|
||
real runner results. Observe out-of-order completion in the fixture and
|
||
verify indices and original references. Keep model response and reference
|
||
coverage in the generator's owning tests.
|
||
|
||
## Stage 14 — Own each correction repair branch explicitly
|
||
|
||
Audit #34, #45, #48, #64 (branch integration).
|
||
|
||
1. In `framework/runner/repair.go`, give one proposal branch a clear
|
||
execution input and result. Make repair capability explicit at setup. Keep
|
||
branch state and terminal outcome together. Merge outcomes into the section
|
||
after each branch, without mutating unrelated approvals or decisions.
|
||
2. Run repairs through PromptKit coordination with full exchange/history and
|
||
configured per-correction limits. Preserve initial and subsequent direct
|
||
domain feedback, sibling independence, withdrawal, unavailable assessment,
|
||
retargeting, and revalidation of the final allowed revision. Move useful
|
||
structural repair and isolation tests onto this path before deleting the
|
||
direct-continuation-only test branch.
|
||
3. Use controlled observations or temporary instrumentation to measure
|
||
cumulative turns, available token/byte costs, revalidation calls, and debug
|
||
work on a representative multi-correction fixture. Remove unnecessary local
|
||
copies. Keep those measurements out of the ordinary report contract.
|
||
|
||
Check independent approved siblings, attempts/turn counts, actual branch
|
||
history and feedback, final-turn revalidation, and associated artifacts in
|
||
production coordination.
|
||
|
||
## Stage 15 — Give stage execution one commit boundary
|
||
|
||
Audit #14, #15 (evidence), #22 and #41 (failure facts).
|
||
|
||
1. In `framework/runner`, check context at entry, including empty pipelines,
|
||
and before committing a stage. Use one stage completion path to record
|
||
outcomes and evidence. Only a successful active stage advances the working
|
||
transcript. Preserve previously committed text for partial results.
|
||
2. Carry initiating cause, phase and available stage/chunk/operation reference
|
||
from the owner. Cancel and join peers on fatal failure or parent cancellation;
|
||
keep peer interruption and secondary errors attributable without promoting
|
||
them over the initiating failure. Continue recoverable work under existing
|
||
partial-success semantics.
|
||
3. Collect completed proposal evidence independently of dispatch to validators.
|
||
After workers join, include usable out-of-order completions in interrupted
|
||
diagnostics with deterministic indices and associations. Keep raw evidence
|
||
from undelivered responses separate from usable proposals. Retain the ordered
|
||
indexing/validation boundary until Stage 16 evaluates it.
|
||
|
||
Test canceled entry and precommit, a later completed proposal behind a slow
|
||
earlier chunk, and a fatal stage after earlier committed text. Assert causal
|
||
origin and preserved evidence as well as returned failure. Leave report/ledger
|
||
projection to Stage 17.
|
||
|
||
## Stage 16 — Bound local preparation and actively drain results
|
||
|
||
Audit #15 (dispatch) and #30, under D3.
|
||
|
||
1. Replace the per-chunk goroutines and chunk-count-sized pending channels in
|
||
`collectSectionProposals` and `runModulePipeline` with a private rolling
|
||
stage window. Admit chunks to the window in input order. Charge a slot for
|
||
each chunk still in preparation, model work, or awaiting downstream work.
|
||
Release it when the chunk is finished and its final facts have been drained.
|
||
Start with a small fixed local window and keep it private. Drain outcomes
|
||
while workers run and on cancellation, then join and finish. Keep compact
|
||
or requested completed evidence apart from remaining work. Collect results
|
||
while the workers can still send them.
|
||
2. Measure staged overlap and peak pending work with many sections, a
|
||
controlled slow first proposal, and shared PromptKit capacity. Where active
|
||
draining and ordered indexing yield adequate progress, keep that simpler
|
||
boundary. If ordered indexing still leaves repeatable idle capacity with
|
||
validation-ready chunks, use a chunk index and response position to identify
|
||
candidates during work. At ordered collection, assign the existing stage
|
||
indices and translate every projected proposal reference, including failure
|
||
indices, repair lineage, and interaction-to-ledger mappings. Keep the
|
||
identity-sensitive artifact data in the charged window until the index is
|
||
assigned; use the Stage 3 writer for the final artifacts. Keep batch-local
|
||
model references and raw response evidence intact. Check the same mappings
|
||
after a cancellation joins the workers.
|
||
3. Keep the module barrier and deterministic application. PromptKit retains
|
||
admission, queueing, limits, retries and recoverable rejection. Include
|
||
nested validation/repair in the local resource and cancellation checks
|
||
without duplicating backend admission or holding its permits across a
|
||
nested call. Tune the window using the measured workload and contention;
|
||
no public window, throughput minimum, or eventual-admission guarantee follows.
|
||
|
||
Test actual observed overlap, finite native contention, cancellation with work
|
||
pending, and identical index ordering for the same usable results under
|
||
different completion orders. Report representative pending-work measurements
|
||
without pinning a timing or worker-count threshold in the default suite.
|
||
|
||
## Stage 17 — Project typed report and ledger facts
|
||
|
||
Audit #22, #27 (ledger), #39, #41 (projection).
|
||
|
||
1. In `framework/processreport`, build the snapshot from the Stage 12 and 15
|
||
facts and Stage 4 delivery outcomes. Preserve phase and available origin for
|
||
setup, processing and publication errors. Keep recovery and work-unit counts
|
||
and known versus unknown values. Distinguish the primary run error,
|
||
interrupted peers, and secondary output failures without counting a fatal
|
||
run failure again as a recoverable unit failure.
|
||
2. Join proposals, validator decisions, repairs, application and skip evidence
|
||
once by stable identities in `correction_ledger.go`. Use the recorded
|
||
execution class from Stage 9. Project one candidate row with correct
|
||
disposition, lineage and literal empty replacement values. Retain
|
||
module-only failures. Preserve serialized fields and logical ordering.
|
||
3. In runner application attribution, reuse section/segment lookups and
|
||
aggregate skipped evidence once. Preserve the distinction between stale
|
||
apply-time conflicts and response filtering. Keep exact correction,
|
||
filtered-proposal and validation-decision identities across projection.
|
||
|
||
Verify decoded YAML, Markdown counts, and ledger facts against an independently
|
||
specified mixed run, including partial decisions, repair/filter interaction,
|
||
later failure, and an overridden validator class. Update the
|
||
[output contract](../integrations/output-files.md) and diagnostics internals.
|
||
|
||
## Stage 18 — Retain evidence according to selected outputs
|
||
|
||
Audit #32 and supporting work for #30, #33, and #44.
|
||
|
||
1. Derive evidence needs once from enabled ordinary reports and effective debug
|
||
capture. Keep compact CLI totals and available failure facts in summary-only
|
||
mode. Allocate work-unit detail only when a consumer needs it, and avoid
|
||
eager whole-run grids and debug-only candidate/decision histories in
|
||
summary-only paths. Keep complete enabled report matrices and debug evidence.
|
||
2. Discard active validation, exchange and repair state after their last
|
||
consumer, including data copied solely for obsolete runner projections.
|
||
Remove duplicate operation lists and unused module totals after checking
|
||
their projections. Keep native observations and requested self-contained
|
||
artifacts when capture is selected.
|
||
3. Compare live/retained state for representative long runs with summary only,
|
||
ordinary reports, debug, and their existing combinations. Check shared
|
||
totals and logical results across modes, with detail present where selected.
|
||
Resource measurements support reduction of waste without setting a
|
||
constant-memory or disk-use requirement.
|
||
|
||
## Stage 19 — Emit native processing progress and update operator guidance
|
||
|
||
Audit #5, #57, and the operator-facing parts of #8, #24 and #25.
|
||
|
||
1. Use a small event callback from the live lifecycle owners. Supply stage and
|
||
chunk activity, proposal/validation/repair phases and useful completions
|
||
with available identities and observed counts. Reuse suitable PromptKit
|
||
events where they convey useful activity. Keep event delivery within the
|
||
invocation and finish workers before the reporter closes.
|
||
2. Wire that callback in CLI to `Invocation.Reporter().Progress` in pinned
|
||
CommandKit. Let CommandKit determine visibility, routing, and presentation
|
||
failure status, independently of transcript/report processing outcomes.
|
||
Preserve stdout transcript data. Respect ordinary mode, explicit progress,
|
||
debug defaults, explicit `--progress=false`, and quiet suppression.
|
||
3. Update the [CLI reference](../cli.md), [operations](../operations.md) and
|
||
[subprocess contract](../integrations/subprocess.md) at their owned
|
||
boundaries. Cover meaningful progress, caller-owned report destinations and
|
||
artifact retention, and the trusted configuration bundle, including
|
||
discovery, imports and automatic companions. Keep the distinction between
|
||
native configuration inspection and processing readiness.
|
||
|
||
Use representative command tests for live activity and combined modes,
|
||
asserting useful identities and stream separation without exact wording or
|
||
concurrent ordering. Keep a progress-specific check separate from delegated
|
||
inspection availability.
|
||
|
||
## Stage 20 — Verify the integrated sprint and release bootstrap
|
||
|
||
Audit #28, #29, #58, #60–62 and sprint closure.
|
||
|
||
1. In `.woodpecker/release.yml`, capture `git status --porcelain --untracked-files=all`
|
||
in an assignment that checks the command's exit status separately from its
|
||
content. Then require empty output. Preserve ReleaseKit's independent
|
||
prepared-checkout validation and tag delivery. In `scripts/release_test.go`,
|
||
cover clean, dirty and failed-status-with-empty-output behavior at the
|
||
command boundary. Remove redundant tests of the comparison helper while
|
||
keeping real release target and wiring checks.
|
||
2. Verify signal, cancellation and exit behavior through the real executable
|
||
or a shared production entry point. Keep narrower checks where a process
|
||
adds no evidence. Verify effective validator roles, required prompt
|
||
identities, embedded asset preparability, schema identity, and protective
|
||
copies through use. Retain a representative repair/filter/later-failure
|
||
integration scenario and independent output-choice cases. Remove the
|
||
audited abandoned fixture inputs and helpers after checking callers.
|
||
3. Recheck Stage 1. Adopt and verify a released upstream fix that became
|
||
available using the same scope. Otherwise report the exact outstanding
|
||
upstream-dependent behavior and its disposition. Treat that item as
|
||
outstanding, while verifying the independent Audita work. Keep the
|
||
invalid-input and cancellation cases distinct.
|
||
4. Account for the [catalog](audit/catalog.md) against the landed changes and
|
||
stage evidence. Entries #2 and #4 need no implementation, and only the
|
||
accepted portions of #6 and #18 were selected. Check canonical docs,
|
||
examples, and affected links and remove stale implemented-behavior claims.
|
||
Keep historical audit observations intact.
|
||
5. From the integrated tree, build the CLI and run the full offline
|
||
`go test ./...` with the supported Go toolchain. Run targeted race checks
|
||
for changed shared-state paths and review the representative measurements
|
||
from Stages 6, 8, 10, 14, 16 and 18. Verify formatting and whitespace,
|
||
and inspect the final diff. Reuse successful checks for unchanged content.
|
||
Existing [release validation](../policy/release.md#candidate-validation)
|
||
applies when a release is prepared.
|
||
|
||
The completed sprint should leave the target architecture reachable through
|
||
`audita process` and the standard delegated commands. Report checks and
|
||
measurements actually performed, including any outstanding external dependency.
|