25 KiB
Reliability and Maintainability Review Remediation
Reviewed revision: e8c0708
Status: Findings and remediation progress from review of the 20 implementation stages against the target roadmap, including follow-up audits of component interactions and algorithmic costs.
Review Result
The sprint established the target architecture. ConfigKit and native CommandKit integrations handle configuration and inspection, and preflight hands checked processing selections to the runner. Audita decodes and normalizes transcripts directly and partitions them once. A bounded section window allows proposal, validation, and repair to overlap within a stage. The runner assigns final proposal indices in section order and commits edits at the stage boundary. PromptKit owns model execution, backend admission, retries, and repair history. Required publication uses checked destinations. Ordinary reports use typed processing facts, while optional debug capture retains interactions and decision history. CommandKit renders live progress.
The offline CLI build and full Go suite passed, as did targeted race checks for the runner, LLM adapter, validators, report projection, diagnostics, glossary, and CLI. The ordered repair artifact test also passed 150 repetitions under the race detector. A built CLI returned status 2 with source diagnostics for malformed PromptKit and application configuration across the delegated commands and processing cases in the plan. Focused review probes reproduced R1, R3, and R4. R2 and R5 follow from the versioned output contract and current documentation.
The follow-up audit traced the new paths through CLI dispatch, the built-in validator chains, model execution, publication, and report and debug readers. Additional probes reproduced the case-variant input behavior in R4 and the cross-component failures in R6–R8.
The algorithmic audit kept the selected stage window, full repair history, and chunk-partitioning algorithm. The normalization merge builds text and categories incrementally after sorting, and the explicit partitioner advances through the weights in linear passes. The remaining opportunities are primarily in repeated validation and glossary preparation, queued debug snapshots, and evidence retained after it has served its purpose. R9 was reproduced through the PromptKit role adapter with a synthetic provider. R10–R14 follow from source analysis. Focused offline tests passed for normalization, chunking, preflight, validators, the runner, and reports.
Findings
R1 — Include Later Delivery Failures in Writable Reports (P2)
Complete. Later report writes include newly observed delivery failures. One best-effort pass updates earlier reports with stale evidence. Diagnostics tests cover both initially failed runs and failures that arise during publication of a completed run.
Affected plan: Stage 4, item 3, and Stage 17, item 1.
Evidence: Report publication renders a failed bundle using the failures accumulated at that point. Later writes append to the returned error list, but the remaining reports and the best-effort pass reuse the same bytes. The report projection can include secondary file failures only if publication supplies them.
Reproduced behavior: Start finalization with a processing failure. Make
errors.yml and recoveries.yml unwritable and let report.md succeed.
The returned error contains the processing failure and both delivery failures.
The renderer receives only the processing failure, so the writable report
omits the two already-observed failures. The same stale-bundle problem can
follow a report error that first switches a successful run to failed status.
Required change: Use the freshest observed failures at the report publication boundary and make a bounded best-effort update of already-written reports when their failure evidence becomes stale. Retain existing evidence if rendering the updated failed bundle fails. Keep transcript-first, nontransactional publication.
Acceptance: Extend the output-session regression to inspect the facts rendered into a surviving report when an earlier failure is followed by a second write failure. Check failure attribution and counts as well as status.
R2 — Preserve the Version 1 Error Record Discriminator (P2)
Complete. Version 1 retains the work_unit and run scopes. The
initiating run record carries available peer details in an optional
interruptions sequence. The mixed-run YAML and Markdown test verifies
peer attribution and unchanged counts.
Affected plan: Stage 17, item 1.
Evidence: Report projection
adds a top-level scope: interrupted record to the
version 1 error document.
The baseline contract described work_unit and run records. An exhaustive
reader of those variants encounters a third variant under the same version.
The target asks to retain peer attribution and existing counts and schema.
Audita's release policy
also preserves the documented output contract for minor and patch releases.
Required change: Keep the existing top-level variants. Carry available peer interruption details as optional evidence on an existing record, for example an optional sequence on the initiating run failure. Keep the primary origin and peer origins separately identifiable and preserve the work-unit and run-error counts. Treat a new top-level variant as an explicit contract change if that representation is preferred.
Acceptance: Exercise a fatal operation and interrupted peer in one run. Check that a consumer of the existing record variants can still read the report, and that the peer's available phase, reason, and origin remain present.
R3 — Use the Same Missing-Reason Fallback in Run Reports (P3)
Complete. The operation and run report use the same typed reason fallback. The regression covers a missing reason with a wrapped deadline, retained operation origin and cause, and distinct parent cancellation.
Affected plan: Stage 12, item 2, and Stage 17, item 1.
Evidence: Operation projection
uses internal_failure when a typed model failure has no reason and retains
its original class and cause. Run-failure projection
recognizes a typed failure only when its reason is nonempty. It can then derive
a different reason from a wrapped cause or the processing phase.
Reproduced behavior: A typed fatal failure with an empty reason and a
wrapped context.DeadlineExceeded is recorded on the operation as
fatal / internal_failure and serialized in the run report as
canceled. The cause is available, but it does not make this a parent
cancellation.
Required change: Share the typed reason projection and its empty-reason fallback. Derive a reason from the raw cause only when a typed failure is absent.
Acceptance: Verify matching fallbacks for the operation and the run record and retain the phase, origin, and original cause. Keep genuine parent cancellation distinct.
R4 — Preserve Effective Transcript Members During Typed Decoding (P3)
Affected plan: Stage 5, item 1.
Evidence: The baseline selected the final value of an exact duplicate JSON member when it decoded the segment into a generic object, before applying field types. Typed transcript decoding now applies the field type to every occurrence. An earlier type error survives even when the final value is valid. The refactor was meant to retain accepted inputs while improving numeric fidelity.
Reproduced behavior: The baseline at d226bdd accepts
[{"speaker":17,"speaker":"A","text":"hello","start":0,"end":1}] and reads
speaker A. The sprint revision rejects the same input because of the
superseded numeric value.
The follow-up audit also found a silent change when known member names differ
only by case. For a segment containing "text":"expected utterance" followed
by "TEXT":"other metadata", the baseline passes expected utterance to
normalization. The new decoder passes other metadata. Both accept the input.
The old map round trip ordered the member names before typed decoding. Direct
decoding applies them in input order.
Required change: Preserve the effective-member behavior for existing inputs without reviving whole-transcript generic decoding or float conversion for IDs. Keep the compatibility handling focused on this changed boundary.
Acceptance: Cover an earlier superseded type mismatch and a type mismatch in the effective member, plus the case-variant example in both member orders. Keep the existing exact-ID and unknown-field tests for both input envelopes.
R5 — Align Roadmap Status and Runtime Guidance (P3)
Affected plan: Stages 15, 19, and 20.
Evidence: The feature roadmap still
describes the implementation as pending despite the completed sprint.
The pipeline guide says work-unit
evidence preserves each peer's interruption or error. The runner preserves
available outcomes and projects detailed work units according to output
selection. The CLI guide contains a
literal \n between “selected” and “profiles.”
Required change: Mark the roadmap with the landed sprint and this review, while retaining its historical baseline context. Describe interruption detail in terms of outcomes actually available and selected for retention. Fix the CLI sentence and keep public guidance focused on the operator-visible behavior.
Acceptance: The roadmap and implementation status agree, the pipeline guide reflects the evidence modes, and the CLI paragraph renders normally.
R6 — Keep Progress Writes Outside Model Operation Deadlines (P2)
Affected plan: Stages 12 and 19.
Evidence: Pipeline wiring calls progress rendering synchronously from the adapter's native observer. PromptKit v0.23.1 starts the operation deadline before emitting the start event, so a blocked or slow write to CommandKit's diagnostic stream spends that operation's budget before the provider request starts.
Reproduced behavior: With a 500 ms model-operation timeout and an immediate
loopback provider, the ordinary run dispatched three requests and changed
hello world to Hello world. With --progress and a diagnostic writer that
delayed the proposal-start message by 750 ms, Audita dispatched zero requests,
reported operation_timeout, and returned the uncorrected transcript with
status 0 under the existing recoverable-failure policy. The writer eventually
succeeded in delivering the message.
Required change: Deliver progress outside PromptKit's timed model-call observation path. Keep native observations for execution facts and use application lifecycle boundaries for synchronous progress delivery, or another delivery mechanism that keeps stderr waits out of the per-operation budget. Preserve CommandKit's routing and presentation status, and keep the parent context authoritative for cancellation of the invocation.
Acceptance: Vary only the delay on the progress stream of a controlled run. Verify that it still dispatches and applies the same correction within the model's own budget, and retain live progress and presentation-failure coverage.
R7 — Classify Configuration Acquisition Errors Consistently (P2)
Affected plan: Stages 1, 11, and 20.
Evidence: The adopted ckconfig v0.8.0 and ckprompt v0.9.0 paths use
configuration diagnostics to distinguish invalid content from operational
errors. The existing process entry point
uses ckconfig.AdaptDiagnostics for every ConfigKit.LoadError, then
unconditionally returns a usage outcome for those errors.
Reproduced behavior: A readable main file beside an unreadable automatic
prompts.yml produces source.permission in all three commands.
promptkit list prompts and config show --scope promptkit return status 1.
process returns status 2 and prints usage guidance for the same loading
failure.
Required change: Use the shared ckconfig.ClassifyConfigurationError
policy at the processing configuration boundary. Preserve typed causes and
source diagnostics for both outcome classes.
Acceptance: Compare process and representative delegated commands for an
operational acquisition failure and malformed YAML. Check status 1 and status
2 respectively, and preserve cancellation precedence and the diagnostic source.
R8 — Use the Same Span Applicability With and Without Repair (P3)
Affected plan: Stages 8 and 9, with ledger projection in Stage 17.
Evidence: The original-text validator rejects an ambiguous require-unique span only when independent repair evidence is enabled. Without repair, the existing shortlist forwards the candidate to the protected-term validator, which records approval even when it cannot obtain a usable preview. The LLM validator's request builder later rejects that same local ambiguity before dispatching a provider call.
Reproduced behavior: In the built-in grammar stage, offer to replace
Mage Armor with ward in Mage Armor and Mage Armor, with Mage Armor
protected. With a zero repair budget, the recorded decisions show protection
approval and editorial_review rejection for ambiguous_original_text.
With repair enabled, original_text_presence records the ambiguity and
protection emits no decision. Neither run applies the ambiguous proposal.
Required change: Give the local applicability check ownership of require-unique ambiguity in both paths. Keep independent evidence and repair attempts conditional on the configured budget. Retain the ability to assess protection when a located edit produces an empty preview.
Acceptance: Use the built-in chain with zero and positive repair budgets. Verify consistent local attribution, no unsupported protection approval, and no model validation call for the inapplicable candidate. Keep repair available in the positive-budget case.
R9 — Accept Schema-Valid Integer Spellings in Proposal and Repair References (P2)
Affected plan: Stages 8, 12–14. The mismatch predates the sprint. Stage 9 already handles the corresponding validator references.
Evidence: The proposal schema
and repair schema
specify integer references. The proposal response type
and repair response decoder
use Go int fields. The role adapter
classifies a conversion error after schema validation as recoverable invalid
output. The proposal generator
then returns an operation failure with no candidates.
Reproduced behavior: A synthetic provider returned two proposals with
references 1 and 2.0. PromptKit accepted the response in one request with
zero structural repairs. Audita returned invalid_output at typed decode.
The control case with 1 and 2 decoded both proposals. The repair decoder
has the same conversion for proposal_ref and the revision's id.
Required change: Decode wire references as exact numeric tokens and reuse the existing integer conversion before local position or correspondence checks. Preserve the distinction between segment and proposal references, existing out-of-range handling, and the original response in debug capture.
Acceptance: Exercise the actual structured paths with an integer and an equivalent decimal or exponent form. A valid sibling survives proposal decoding, and the repair revision reaches the same revalidation as its integer form. Retain existing handling for a mismatched repair reference.
R10 — Fit Required Validation Data Before Collecting Optional Context (P3)
Affected plan: Stages 8 and 10.
Evidence: Item construction builds optional context for every previewable candidate before fitting the payload. Context construction rescans the original and adjacent segments, finds matches in the proposed text, sorts matched entry positions, and builds the entry slice. The fitter can discard all optional context while retaining the original slice's backing array on a fitted item.
Cost: For P candidates at one position, a fixed vocabulary, and B
bytes of shared source and neighboring text, the unchanged text is scanned
P times, for Θ(P × B) input bytes. Up to G matched entries per candidate
add O(P × G log G) sorting work and O(P × G) entry storage. This work
happens even when required fields consume the budget. Distinct proposed texts
still need their own match evidence when optional context is used.
Recommended change: Fit the required fields before collecting optional context for a candidate. Reuse identity membership for unchanged text within the section and combine it with the corrected-text matches only when needed. Keep existing glossary and neighbor order, exact serialized budget accounting, and oversized-input behavior. Drop unused optional storage before dispatch.
Acceptance: Compare the emitted payload to the existing serialized-budget oracle for usable and oversized candidates. A controlled no-room case should skip optional matching. Sibling candidates should share unchanged-text lookup while still receiving definitions specific to their revisions.
R11 — Reuse Glossary Selections Before Building Their Views (P3)
Affected plan: Stage 11, item 2.
Evidence: Spec resolution clones selected glossary entries and aliases and derives protection terms for every stage. The preflight cache can reuse prompt variables and validator views only after all those copies have been built. The retained specs still hold separate entry and term slices for identical selections.
Cost: With S stages sharing a selection of G entries and spelling
contributions, materializing and retaining those selection data costs
Θ(S × G). Sharing immutable selection data brings this portion to
Θ(G + S) for repeated fixed-length selection keys.
Recommended change: Resolve and prepare the shared glossary data on the first occurrence of an ordered context/protection selection and exemption. Reuse the complete read-only selection in later specs. Keep stage identity, first-failure attribution, and copy-on-mutation at consumer boundaries.
Acceptance: Exercise repeated selections, a reordered selection, and a glossary-stage exemption. Reused selections should supply the same data to proposal, validation, and application as the existing independent views.
R12 — Avoid Queuing Full Repair Histories in Debug Capture (P3)
Affected plan: Stages 3, 14, 16, and 18.
Evidence: Prepared capture converts every turn's full rendered message list to a new slice. Each repair builder captures that snapshot. The section sink holds the builders until ordered proposal indices are assigned, including during earlier sections' model work.
Cost: A branch with H starting messages and T successful turns that
add a constant number of messages per turn can retain Θ(T × H + T²) message
descriptors at once. Content strings may share backing storage. The window
bounds the number of sections held, and a single section can still contain
many turns and branches. Full-history requests and self-contained debug files
require repeated serialization by design. They can be serialized one at a
time without keeping every prepared list live.
Recommended change: Keep compact evidence pending ordered identity assignment. For example, stage the request payload under section-local coordinates while it is live and use its final name when indices are assigned. An in-memory approach can share immutable history prefixes and expand each snapshot at write time if it can reproduce the exact rendered messages. Retain full model context, nonfatal capture, and self-contained final files.
Acceptance: Exercise a multi-turn repair behind a delayed earlier section. Compare its decoded request bodies, final artifact names, and ledger paths with the current representation, including a capture failure followed by a successful write.
R13 — Retain Only Operations Used by Ordinary Reports (P3)
Affected plan: Stages 12, 17, and 18.
Evidence: Ordered collection counts each work unit and retains all its operations when reports are enabled. It also stores every candidate index on the successful proposal operation. Report projection uses only recoverable failures and successful recoveries. Failure projection uses affected indices and typed facts. Recovery projection uses counts and filtered evidence. Neither uses the cause stored on each operation, which is needed earlier for failure attribution.
Cost: The retained operation records, unused successful proposal index
lists, and causes account for Θ(M + P + B) storage, for M operation
records, P successful proposal indices, and B bytes of retained causes.
PromptKit's non-success HTTP error can include a bounded
provider diagnostic in each cause. Even an all-success run with no recoveries
keeps and later sorts operation rows that produce no error or recovery entry.
The report matrix itself still needs a row per work unit.
Recommended change: After counting and resolving application filtering and failure attribution, retain compact facts only for operations that feed the reports. Keep typed initiating and peer causes where they belong on execution failures. Debug execution observations and correction-ledger inputs have their own consumers. Avoid retaining proposal index lists on operations that never project those lists.
Acceptance: Compare decoded errors, recoveries, work-unit and CLI totals, and debug association across existing evidence modes. Include clean successes, both initial and repair filtering, and an interrupted run with available origin.
R14 — Reuse Original Applicability When Attributing Application Skips (P3)
Affected plan: Stages 8 and 17, item 3.
Evidence: Application checks each proposal against evolving segment text. On a defective-looking skip, the runner attribution pass previews the proposal again against the original section to distinguish a stale conflict from a defective response. The built-in validation path has already determined the latter preview's applicability. Its result is discarded before the attribution pass.
Cost: For K conflicting proposals against one original segment of B
bytes, the attribution pass can scan and construct Θ(K × B) bytes of
original-text previews, even if application quickly rejected those proposals
against the changed text. This is separate from the checks required against
current text. When a validator chain can approve originally inapplicable
revisions, repair filtering attribution
can also scan O(F × M) operations for F such filtered revisions and M
operations in the same unit. Built-in validators normally reject those revisions
before application.
Recommended change: Carry a compact original-applicability fact for approved candidates when the validator already computed it. Recompute only when no such fact is available. If the repair attribution branch needs changes, index or group originating repair operations once per affected unit. Preserve ordered application, current-text protection checks, and the distinction between response filtering and staleness.
Acceptance: Keep both a stale-conflict case with initially valid proposals and an originally inapplicable approval from a controlled validator. Verify the skip, correction counts, and filtering or recovery attribution.