2026-11-17 01:51:41 +00:00
|
|
|
|
# Reliability and Maintainability Review Remediation
|
|
|
|
|
|
|
|
|
|
|
|
**Reviewed revision:** `e8c0708`
|
|
|
|
|
|
|
2026-11-17 03:49:35 +00:00
|
|
|
|
**Status:** Fourteen open findings from review of the 20
|
2026-11-17 01:51:41 +00:00
|
|
|
|
[implementation stages](implementation.md) against the
|
2026-11-17 03:49:35 +00:00
|
|
|
|
[target roadmap](reliability-and-maintainability.md), including follow-up
|
|
|
|
|
|
audits of component interactions and algorithmic costs.
|
2026-11-17 01:51:41 +00:00
|
|
|
|
|
|
|
|
|
|
## Review Result
|
|
|
|
|
|
|
|
|
|
|
|
The sprint established the target architecture. ConfigKit and native
|
|
|
|
|
|
CommandKit integrations handle configuration and inspection, and preflight
|
|
|
|
|
|
hands checked processing selections to the runner. Audita decodes and
|
|
|
|
|
|
normalizes transcripts directly and partitions them once. A bounded section
|
|
|
|
|
|
window allows proposal, validation, and repair to overlap within a stage.
|
|
|
|
|
|
The runner assigns final proposal indices in section order and commits edits
|
|
|
|
|
|
at the stage boundary. PromptKit owns model execution, backend admission,
|
|
|
|
|
|
retries, and repair history. Required publication uses checked destinations.
|
|
|
|
|
|
Ordinary reports use typed processing facts, while optional debug capture
|
|
|
|
|
|
retains interactions and decision history. CommandKit renders live progress.
|
|
|
|
|
|
|
|
|
|
|
|
The offline CLI build and full Go suite passed, as did targeted race checks
|
|
|
|
|
|
for the runner, LLM adapter, validators, report projection, diagnostics,
|
|
|
|
|
|
glossary, and CLI. The ordered repair artifact test also passed 150 repetitions
|
|
|
|
|
|
under the race detector. A built CLI returned status 2 with source diagnostics
|
|
|
|
|
|
for malformed PromptKit and application configuration across the delegated
|
|
|
|
|
|
commands and processing cases in the plan. Focused review probes reproduced
|
|
|
|
|
|
R1, R3, and R4. R2 and R5 follow from the versioned output contract and current
|
|
|
|
|
|
documentation.
|
|
|
|
|
|
|
2026-11-17 02:00:31 +00:00
|
|
|
|
The follow-up audit traced the new paths through CLI dispatch, the built-in
|
|
|
|
|
|
validator chains, model execution, publication, and report and debug readers.
|
|
|
|
|
|
Additional probes reproduced the case-variant input behavior in R4 and the
|
|
|
|
|
|
cross-component failures in R6–R8.
|
|
|
|
|
|
|
2026-11-17 03:49:35 +00:00
|
|
|
|
The algorithmic audit kept the selected stage window, full repair history,
|
|
|
|
|
|
and chunk-partitioning algorithm. The normalization merge builds text and
|
|
|
|
|
|
categories incrementally after sorting, and the explicit partitioner advances
|
|
|
|
|
|
through the weights in linear passes. The remaining opportunities are
|
|
|
|
|
|
primarily in repeated validation and glossary preparation, queued debug
|
|
|
|
|
|
snapshots, and evidence retained after it has served its purpose. R9 was
|
|
|
|
|
|
reproduced through the PromptKit role adapter with a synthetic provider.
|
|
|
|
|
|
R10–R14 follow from source analysis. Focused offline tests passed for
|
|
|
|
|
|
normalization, chunking, preflight, validators, the runner, and reports.
|
|
|
|
|
|
|
2026-11-17 01:51:41 +00:00
|
|
|
|
## Findings
|
|
|
|
|
|
|
|
|
|
|
|
### R1 — Include Later Delivery Failures in Writable Reports (P2)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stage 4, item 3, and Stage 17, item 1.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Report publication](../../internal/core/diagnostics/output_session.go#L311)
|
|
|
|
|
|
renders a failed bundle using the failures accumulated at that point. Later
|
|
|
|
|
|
writes append to the returned error list, but the remaining reports and the
|
|
|
|
|
|
best-effort pass reuse the same bytes. The
|
|
|
|
|
|
[report projection](../../internal/framework/processreport/failures.go#L41)
|
|
|
|
|
|
can include secondary file failures only if publication supplies them.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** Start finalization with a processing failure. Make
|
|
|
|
|
|
`errors.yml` and `recoveries.yml` unwritable and let `report.md` succeed.
|
|
|
|
|
|
The returned error contains the processing failure and both delivery failures.
|
|
|
|
|
|
The renderer receives only the processing failure, so the writable report
|
|
|
|
|
|
omits the two already-observed failures. The same stale-bundle problem can
|
|
|
|
|
|
follow a report error that first switches a successful run to failed status.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Use the freshest observed failures at the report
|
|
|
|
|
|
publication boundary and make a bounded best-effort update of already-written
|
|
|
|
|
|
reports when their failure evidence becomes stale. Retain existing evidence
|
|
|
|
|
|
if rendering the updated failed bundle fails. Keep transcript-first,
|
|
|
|
|
|
nontransactional publication.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Extend the output-session regression to inspect the facts
|
|
|
|
|
|
rendered into a surviving report when an earlier failure is followed by a
|
|
|
|
|
|
second write failure. Check failure attribution and counts as well as status.
|
|
|
|
|
|
|
|
|
|
|
|
### R2 — Preserve the Version 1 Error Record Discriminator (P2)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stage 17, item 1.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Report projection](../../internal/framework/processreport/report.go#L108)
|
|
|
|
|
|
adds a top-level `scope: interrupted` record to the
|
|
|
|
|
|
[version 1 error document](../integrations/output-files.md#error-and-recovery-records).
|
|
|
|
|
|
The baseline contract described `work_unit` and `run` records. An exhaustive
|
|
|
|
|
|
reader of those variants encounters a third variant under the same version.
|
|
|
|
|
|
The target asks to retain peer attribution and existing counts and schema.
|
|
|
|
|
|
Audita's [release policy](../policy/release.md#versioning-and-compatibility)
|
|
|
|
|
|
also preserves the documented output contract for minor and patch releases.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Keep the existing top-level variants. Carry available
|
|
|
|
|
|
peer interruption details as optional evidence on an existing record, for
|
|
|
|
|
|
example an optional sequence on the initiating run failure. Keep the primary
|
|
|
|
|
|
origin and peer origins separately identifiable and preserve the work-unit and
|
|
|
|
|
|
run-error counts. Treat a new top-level variant as an explicit contract change
|
|
|
|
|
|
if that representation is preferred.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Exercise a fatal operation and interrupted peer in one run.
|
|
|
|
|
|
Check that a consumer of the existing record variants can still read the
|
|
|
|
|
|
report, and that the peer's available phase, reason, and origin remain present.
|
|
|
|
|
|
|
|
|
|
|
|
### R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stage 12, item 2, and Stage 17, item 1.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Operation projection](../../internal/framework/structuredoutput/failure.go#L42)
|
|
|
|
|
|
uses `internal_failure` when a typed model failure has no reason and retains
|
|
|
|
|
|
its original class and cause. [Run-failure projection](../../internal/framework/processreport/failures.go#L16)
|
|
|
|
|
|
recognizes a typed failure only when its reason is nonempty. It can then derive
|
|
|
|
|
|
a different reason from a wrapped cause or the processing phase.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** A typed fatal failure with an empty reason and a
|
|
|
|
|
|
wrapped `context.DeadlineExceeded` is recorded on the operation as
|
|
|
|
|
|
`fatal` / `internal_failure` and serialized in the run report as
|
|
|
|
|
|
`canceled`. The cause is available, but it does not make this a parent
|
|
|
|
|
|
cancellation.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Share the typed reason projection and its empty-reason
|
|
|
|
|
|
fallback. Derive a reason from the raw cause only when a typed failure is absent.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Verify matching fallbacks for the operation and the run record
|
|
|
|
|
|
and retain the phase, origin, and original cause. Keep genuine parent
|
|
|
|
|
|
cancellation distinct.
|
|
|
|
|
|
|
|
|
|
|
|
### R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stage 5, item 1.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The baseline selected the final value of an exact duplicate JSON
|
|
|
|
|
|
member when it decoded the segment into a generic object, before applying
|
|
|
|
|
|
field types. [Typed transcript decoding](../../internal/core/schema/transcript.go#L39)
|
|
|
|
|
|
now applies the field type to every occurrence. An earlier type error survives
|
|
|
|
|
|
even when the final value is valid. The refactor was meant to retain accepted
|
|
|
|
|
|
inputs while improving numeric fidelity.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** The baseline at `d226bdd` accepts
|
|
|
|
|
|
`[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}]` and reads
|
|
|
|
|
|
speaker `A`. The sprint revision rejects the same input because of the
|
|
|
|
|
|
superseded numeric value.
|
|
|
|
|
|
|
2026-11-17 02:00:31 +00:00
|
|
|
|
The follow-up audit also found a silent change when known member names differ
|
|
|
|
|
|
only by case. For a segment containing `"text":"expected utterance"` followed
|
|
|
|
|
|
by `"TEXT":"other metadata"`, the baseline passes `expected utterance` to
|
|
|
|
|
|
normalization. The new decoder passes `other metadata`. Both accept the input.
|
|
|
|
|
|
The old map round trip ordered the member names before typed decoding. Direct
|
|
|
|
|
|
decoding applies them in input order.
|
|
|
|
|
|
|
2026-11-17 01:51:41 +00:00
|
|
|
|
**Required change:** Preserve the effective-member behavior for existing
|
|
|
|
|
|
inputs without reviving whole-transcript generic decoding or float conversion
|
|
|
|
|
|
for IDs. Keep the compatibility handling focused on this changed boundary.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Cover an earlier superseded type mismatch and a type mismatch
|
2026-11-17 02:00:31 +00:00
|
|
|
|
in the effective member, plus the case-variant example in both member orders.
|
|
|
|
|
|
Keep the existing exact-ID and unknown-field tests for both input envelopes.
|
2026-11-17 01:51:41 +00:00
|
|
|
|
|
|
|
|
|
|
### R5 — Align Roadmap Status and Runtime Guidance (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 15, 19, and 20.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [feature roadmap](reliability-and-maintainability.md) still
|
|
|
|
|
|
describes the implementation as pending despite the completed sprint.
|
|
|
|
|
|
The [pipeline guide](../internal/pipeline.md) says work-unit
|
|
|
|
|
|
evidence preserves each peer's interruption or error. The runner preserves
|
|
|
|
|
|
available outcomes and projects detailed work units according to output
|
|
|
|
|
|
selection. The [CLI guide](../cli.md#inspection-and-outcomes) contains a
|
|
|
|
|
|
literal `\n` between “selected” and “profiles.”
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Mark the roadmap with the landed sprint and this review,
|
|
|
|
|
|
while retaining its historical baseline context. Describe interruption detail
|
|
|
|
|
|
in terms of outcomes actually available and selected for retention. Fix the
|
|
|
|
|
|
CLI sentence and keep public guidance focused on the operator-visible behavior.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** The roadmap and implementation status agree, the pipeline guide
|
|
|
|
|
|
reflects the evidence modes, and the CLI paragraph renders normally.
|
2026-11-17 02:00:31 +00:00
|
|
|
|
|
|
|
|
|
|
### R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 12 and 19.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Pipeline wiring](../../internal/cli/process_pipeline.go#L30)
|
|
|
|
|
|
calls [progress rendering](../../internal/cli/process_progress.go#L34)
|
|
|
|
|
|
synchronously from the [adapter's native observer](../../internal/framework/llm/adapter.go#L139).
|
|
|
|
|
|
PromptKit v0.23.1 starts the operation deadline before emitting the start
|
|
|
|
|
|
event, so a blocked or slow write to CommandKit's diagnostic stream spends
|
|
|
|
|
|
that operation's budget before the provider request starts.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** With a 500 ms model-operation timeout and an immediate
|
|
|
|
|
|
loopback provider, the ordinary run dispatched three requests and changed
|
|
|
|
|
|
`hello world` to `Hello world`. With `--progress` and a diagnostic writer that
|
|
|
|
|
|
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
|
|
|
|
|
|
reported `operation_timeout`, and returned the uncorrected transcript with
|
|
|
|
|
|
status 0 under the existing recoverable-failure policy. The writer eventually
|
|
|
|
|
|
succeeded in delivering the message.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Deliver progress outside PromptKit's timed model-call
|
|
|
|
|
|
observation path. Keep native observations for execution facts and use
|
|
|
|
|
|
application lifecycle boundaries for synchronous progress delivery, or another
|
|
|
|
|
|
delivery mechanism that keeps stderr waits out of the per-operation budget.
|
|
|
|
|
|
Preserve CommandKit's routing and presentation status, and keep the parent
|
|
|
|
|
|
context authoritative for cancellation of the invocation.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Vary only the delay on the progress stream of a controlled run.
|
|
|
|
|
|
Verify that it still dispatches and applies the same correction within the
|
|
|
|
|
|
model's own budget, and retain live progress and presentation-failure coverage.
|
|
|
|
|
|
|
|
|
|
|
|
### R7 — Classify Configuration Acquisition Errors Consistently (P2)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 1, 11, and 20.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The adopted `ckconfig` v0.8.0 and `ckprompt` v0.9.0 paths use
|
|
|
|
|
|
configuration diagnostics to distinguish invalid content from operational
|
|
|
|
|
|
errors. The existing [process entry point](../../internal/cli/process.go#L27)
|
|
|
|
|
|
uses `ckconfig.AdaptDiagnostics` for every `ConfigKit.LoadError`, then
|
|
|
|
|
|
unconditionally returns a usage outcome for those errors.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** A readable main file beside an unreadable automatic
|
|
|
|
|
|
`prompts.yml` produces `source.permission` in all three commands.
|
|
|
|
|
|
`promptkit list prompts` and `config show --scope promptkit` return status 1.
|
|
|
|
|
|
`process` returns status 2 and prints usage guidance for the same loading
|
|
|
|
|
|
failure.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Use the shared `ckconfig.ClassifyConfigurationError`
|
|
|
|
|
|
policy at the processing configuration boundary. Preserve typed causes and
|
|
|
|
|
|
source diagnostics for both outcome classes.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Compare `process` and representative delegated commands for an
|
|
|
|
|
|
operational acquisition failure and malformed YAML. Check status 1 and status
|
|
|
|
|
|
2 respectively, and preserve cancellation precedence and the diagnostic source.
|
|
|
|
|
|
|
|
|
|
|
|
### R8 — Use the Same Span Applicability With and Without Repair (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 8 and 9, with ledger projection in Stage 17.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [original-text validator](../../internal/framework/validators/deterministic.go#L38)
|
|
|
|
|
|
rejects an ambiguous require-unique span only when independent repair evidence
|
|
|
|
|
|
is enabled. Without repair, the existing shortlist forwards the candidate to
|
|
|
|
|
|
the [protected-term validator](../../internal/framework/validators/deterministic.go#L106),
|
|
|
|
|
|
which records approval even when it cannot obtain a usable preview. The
|
|
|
|
|
|
[LLM validator's request builder](../../internal/framework/validators/llm_validators.go#L251)
|
|
|
|
|
|
later rejects that same local ambiguity before dispatching a provider call.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** In the built-in grammar stage, offer to replace
|
|
|
|
|
|
`Mage Armor` with `ward` in `Mage Armor and Mage Armor`, with `Mage Armor`
|
|
|
|
|
|
protected. With a zero repair budget, the recorded decisions show protection
|
|
|
|
|
|
approval and `editorial_review` rejection for `ambiguous_original_text`.
|
|
|
|
|
|
With repair enabled, `original_text_presence` records the ambiguity and
|
|
|
|
|
|
protection emits no decision. Neither run applies the ambiguous proposal.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Give the local applicability check ownership of
|
|
|
|
|
|
require-unique ambiguity in both paths. Keep independent evidence and repair
|
|
|
|
|
|
attempts conditional on the configured budget. Retain the ability to assess
|
|
|
|
|
|
protection when a located edit produces an empty preview.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Use the built-in chain with zero and positive repair budgets.
|
|
|
|
|
|
Verify consistent local attribution, no unsupported protection approval,
|
|
|
|
|
|
and no model validation call for the inapplicable candidate. Keep repair
|
|
|
|
|
|
available in the positive-budget case.
|
2026-11-17 03:49:35 +00:00
|
|
|
|
|
|
|
|
|
|
### R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 8, 12–14. The mismatch predates the sprint. Stage 9
|
|
|
|
|
|
already handles the corresponding validator references.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** The [proposal schema](../../internal/framework/responseschema/assets/correction_set.v1.json#L20)
|
|
|
|
|
|
and [repair schema](../../internal/framework/responseschema/assets/correction_repair.v1.json#L9)
|
|
|
|
|
|
specify integer references. The [proposal response type](../../internal/framework/proposal_generation/generate.go#L21)
|
|
|
|
|
|
and [repair response decoder](../../internal/framework/proposal_generation/repair.go#L24)
|
|
|
|
|
|
use Go `int` fields. The [role adapter](../../internal/framework/llm/adapter.go#L184)
|
|
|
|
|
|
classifies a conversion error after schema validation as recoverable invalid
|
|
|
|
|
|
output. The [proposal generator](../../internal/framework/proposal_generation/generate.go#L123)
|
|
|
|
|
|
then returns an operation failure with no candidates.
|
|
|
|
|
|
|
|
|
|
|
|
**Reproduced behavior:** A synthetic provider returned two proposals with
|
|
|
|
|
|
references `1` and `2.0`. PromptKit accepted the response in one request with
|
|
|
|
|
|
zero structural repairs. Audita returned `invalid_output` at typed decode.
|
|
|
|
|
|
The control case with `1` and `2` decoded both proposals. The repair decoder
|
|
|
|
|
|
has the same conversion for `proposal_ref` and the revision's `id`.
|
|
|
|
|
|
|
|
|
|
|
|
**Required change:** Decode wire references as exact numeric tokens and
|
|
|
|
|
|
reuse the [existing integer conversion](../../internal/core/jsonnumber/integer.go#L9)
|
|
|
|
|
|
before local position or correspondence checks. Preserve the distinction
|
|
|
|
|
|
between segment and proposal references, existing out-of-range handling, and
|
|
|
|
|
|
the original response in debug capture.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Exercise the actual structured paths with an integer and an
|
|
|
|
|
|
equivalent decimal or exponent form. A valid sibling survives proposal
|
|
|
|
|
|
decoding, and the repair revision reaches the same revalidation as its integer
|
|
|
|
|
|
form. Retain existing handling for a mismatched repair reference.
|
|
|
|
|
|
|
|
|
|
|
|
### R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 8 and 10.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Item construction](../../internal/framework/validators/llm_validators.go#L251)
|
|
|
|
|
|
builds optional context for every previewable candidate before
|
|
|
|
|
|
[fitting the payload](../../internal/framework/validators/llm_budget.go#L50).
|
|
|
|
|
|
[Context construction](../../internal/framework/validators/llm_context.go#L39)
|
|
|
|
|
|
rescans the original and adjacent segments, finds matches in the proposed
|
|
|
|
|
|
text, sorts matched entry positions, and builds the entry slice. The fitter
|
|
|
|
|
|
can discard all optional context while retaining the original slice's
|
|
|
|
|
|
backing array on a fitted item.
|
|
|
|
|
|
|
|
|
|
|
|
**Cost:** For `P` candidates at one position, a fixed vocabulary, and `B`
|
|
|
|
|
|
bytes of shared source and neighboring text, the unchanged text is scanned
|
|
|
|
|
|
`P` times, for `Θ(P × B)` input bytes. Up to `G` matched entries per candidate
|
|
|
|
|
|
add `O(P × G log G)` sorting work and `O(P × G)` entry storage. This work
|
|
|
|
|
|
happens even when required fields consume the budget. Distinct proposed texts
|
|
|
|
|
|
still need their own match evidence when optional context is used.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Fit the required fields before collecting optional
|
|
|
|
|
|
context for a candidate. Reuse identity membership for unchanged text within
|
|
|
|
|
|
the section and combine it with the corrected-text matches only when needed.
|
|
|
|
|
|
Keep existing glossary and neighbor order, exact serialized budget accounting,
|
|
|
|
|
|
and oversized-input behavior. Drop unused optional storage before dispatch.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Compare the emitted payload to the existing serialized-budget
|
|
|
|
|
|
oracle for usable and oversized candidates. A controlled no-room case should
|
|
|
|
|
|
skip optional matching. Sibling candidates should share unchanged-text lookup
|
|
|
|
|
|
while still receiving definitions specific to their revisions.
|
|
|
|
|
|
|
|
|
|
|
|
### R11 — Reuse Glossary Selections Before Building Their Views (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stage 11, item 2.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Spec resolution](../../internal/framework/contracts/contracts.go#L115)
|
|
|
|
|
|
clones selected glossary entries and aliases and derives protection terms for
|
|
|
|
|
|
every stage. The [preflight cache](../../internal/framework/preflight/preflight.go#L152)
|
|
|
|
|
|
can reuse prompt variables and validator views only after all those copies
|
|
|
|
|
|
have been built. The retained specs still hold separate entry and term slices
|
|
|
|
|
|
for identical selections.
|
|
|
|
|
|
|
|
|
|
|
|
**Cost:** With `S` stages sharing a selection of `G` entries and spelling
|
|
|
|
|
|
contributions, materializing and retaining those selection data costs
|
|
|
|
|
|
`Θ(S × G)`. Sharing immutable selection data brings this portion to
|
|
|
|
|
|
`Θ(G + S)` for repeated fixed-length selection keys.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Resolve and prepare the shared glossary data on the
|
|
|
|
|
|
first occurrence of an ordered context/protection selection and exemption.
|
|
|
|
|
|
Reuse the complete read-only selection in later specs. Keep stage identity,
|
|
|
|
|
|
first-failure attribution, and copy-on-mutation at consumer boundaries.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Exercise repeated selections, a reordered selection, and a
|
|
|
|
|
|
glossary-stage exemption. Reused selections should supply the same data to
|
|
|
|
|
|
proposal, validation, and application as the existing independent views.
|
|
|
|
|
|
|
|
|
|
|
|
### R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 3, 14, 16, and 18.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Prepared capture](../../internal/framework/llm/adapter.go#L151)
|
|
|
|
|
|
converts every turn's full rendered message list to a new slice. Each
|
|
|
|
|
|
[repair builder](../../internal/framework/runner/repair.go#L194) captures that
|
|
|
|
|
|
snapshot. The [section sink](../../internal/framework/runner/section_evidence.go#L22)
|
|
|
|
|
|
holds the builders until ordered proposal indices are assigned, including
|
|
|
|
|
|
during earlier sections' model work.
|
|
|
|
|
|
|
|
|
|
|
|
**Cost:** A branch with `H` starting messages and `T` successful turns that
|
|
|
|
|
|
add a constant number of messages per turn can retain `Θ(T × H + T²)` message
|
|
|
|
|
|
descriptors at once. Content strings may share backing storage. The window
|
|
|
|
|
|
bounds the number of sections held, and a single section can still contain
|
|
|
|
|
|
many turns and branches. Full-history requests and self-contained debug files
|
|
|
|
|
|
require repeated serialization by design. They can be serialized one at a
|
|
|
|
|
|
time without keeping every prepared list live.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Keep compact evidence pending ordered identity
|
|
|
|
|
|
assignment. For example, stage the request payload under section-local
|
|
|
|
|
|
coordinates while it is live and use its final name when indices are assigned.
|
|
|
|
|
|
An in-memory approach can share immutable history prefixes and expand each
|
|
|
|
|
|
snapshot at write time if it can reproduce the exact rendered messages.
|
|
|
|
|
|
Retain full model context, nonfatal capture, and self-contained final files.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Exercise a multi-turn repair behind a delayed earlier section.
|
|
|
|
|
|
Compare its decoded request bodies, final artifact names, and ledger paths
|
|
|
|
|
|
with the current representation, including a capture failure followed by a
|
|
|
|
|
|
successful write.
|
|
|
|
|
|
|
|
|
|
|
|
### R13 — Retain Only Operations Used by Ordinary Reports (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 12, 17, and 18.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Ordered collection](../../internal/framework/runner/runner.go#L632)
|
|
|
|
|
|
counts each work unit and retains all its operations when reports are enabled.
|
|
|
|
|
|
It also [stores every candidate index](../../internal/framework/runner/runner.go#L548)
|
|
|
|
|
|
on the successful proposal operation. [Report projection](../../internal/framework/processreport/report.go#L79)
|
|
|
|
|
|
uses only recoverable failures and successful recoveries. Failure projection
|
|
|
|
|
|
uses affected indices and typed facts. Recovery projection uses counts and
|
|
|
|
|
|
filtered evidence. Neither uses the cause stored on each operation, which is
|
|
|
|
|
|
needed earlier for [failure attribution](../../internal/framework/runner/failure.go#L22).
|
|
|
|
|
|
|
|
|
|
|
|
**Cost:** The retained operation records, unused successful proposal index
|
|
|
|
|
|
lists, and causes account for `Θ(M + P + B)` storage, for `M` operation
|
|
|
|
|
|
records, `P` successful proposal indices, and `B` bytes of retained causes.
|
|
|
|
|
|
PromptKit's non-success HTTP error can include a bounded
|
|
|
|
|
|
provider diagnostic in each cause. Even an all-success run with no recoveries
|
|
|
|
|
|
keeps and later sorts operation rows that produce no error or recovery entry.
|
|
|
|
|
|
The report matrix itself still needs a row per work unit.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** After counting and resolving application filtering and
|
|
|
|
|
|
failure attribution, retain compact facts only for operations that feed the
|
|
|
|
|
|
reports. Keep typed initiating and peer causes where they belong on execution
|
|
|
|
|
|
failures. Debug execution observations and correction-ledger inputs have
|
|
|
|
|
|
their own consumers. Avoid retaining proposal index lists on operations that
|
|
|
|
|
|
never project those lists.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Compare decoded errors, recoveries, work-unit and CLI totals,
|
|
|
|
|
|
and debug association across existing evidence modes. Include clean successes,
|
|
|
|
|
|
both initial and repair filtering, and an interrupted run with available origin.
|
|
|
|
|
|
|
|
|
|
|
|
### R14 — Reuse Original Applicability When Attributing Application Skips (P3)
|
|
|
|
|
|
|
|
|
|
|
|
**Affected plan:** Stages 8 and 17, item 3.
|
|
|
|
|
|
|
|
|
|
|
|
**Evidence:** [Application](../../internal/framework/proposals/apply.go#L59)
|
|
|
|
|
|
checks each proposal against evolving segment text. On a defective-looking
|
|
|
|
|
|
skip, the [runner attribution pass](../../internal/framework/runner/runner.go#L953)
|
|
|
|
|
|
previews the proposal again against the original section to distinguish a
|
|
|
|
|
|
stale conflict from a defective response. The built-in validation path has
|
|
|
|
|
|
already determined the latter preview's applicability. Its result is discarded
|
|
|
|
|
|
before the attribution pass.
|
|
|
|
|
|
|
|
|
|
|
|
**Cost:** For `K` conflicting proposals against one original segment of `B`
|
|
|
|
|
|
bytes, the attribution pass can scan and construct `Θ(K × B)` bytes of
|
|
|
|
|
|
original-text previews, even if application quickly rejected those proposals
|
|
|
|
|
|
against the changed text. This is separate from the checks required against
|
|
|
|
|
|
current text. When a validator chain can approve originally inapplicable
|
|
|
|
|
|
revisions, [repair filtering attribution](../../internal/framework/runner/work_totals.go#L46)
|
|
|
|
|
|
can also scan `O(F × M)` operations for `F` such filtered revisions and `M`
|
|
|
|
|
|
operations in the same unit. Built-in validators normally reject those revisions
|
|
|
|
|
|
before application.
|
|
|
|
|
|
|
|
|
|
|
|
**Recommended change:** Carry a compact original-applicability fact for
|
|
|
|
|
|
approved candidates when the validator already computed it. Recompute only
|
|
|
|
|
|
when no such fact is available. If the repair attribution branch needs changes,
|
|
|
|
|
|
index or group originating repair operations once per affected unit. Preserve
|
|
|
|
|
|
ordered application, current-text protection checks, and the distinction
|
|
|
|
|
|
between response filtering and staleness.
|
|
|
|
|
|
|
|
|
|
|
|
**Acceptance:** Keep both a stale-conflict case with initially valid proposals
|
|
|
|
|
|
and an originally inapplicable approval from a controlled validator. Verify
|
|
|
|
|
|
the skip, correction counts, and filtering or recovery attribution.
|