promptkit/docs/roadmap/audit-remediation.md

Audit Remediation Roadmap

Purpose And Status

Prepare PromptKit's audited implementation for a focused round of correctness, configuration simplification, resource-use improvements, and internal cleanup. This document defines the target and work boundaries for a subsequent staged implementation plan; it is not that plan or a description of completed changes.

Baseline: repository commit bd3dfa9, with the audit catalog covering 41 source findings consolidated into 34 remediation entries. All six catalog decisions have been accepted; their implementation remains pending. The catalog and supporting audits own finding evidence and decision rationale. This roadmap owns remediation organization, dependencies, completion evidence, and subsequent workstream status. Earlier remediation suggestions in the source audits do not override accepted decisions.

All workstreams below are pending. Catalog numbers are traceability identifiers, not a required implementation order or 34 separate delivery stages. The scope contains confirmed defects, documentation/test gaps, and maintainability work; those categories do not have equal urgency or automatically become release gates.

Targeted End State

PromptKit remains an application-neutral Go library with the existing public facade and ConfigKit composition boundary. The intended changes are:

• Programmatic backend/profile sources hand supported values to ConfigKit directly, removing PromptKit's recursive compatibility conversion.
• Schemas supplied for validation and native structured output are self-contained documents, while in-document references remain supported.
• Artifact-reader ownership, public errors, and smoke-test outcomes accurately express the owning operation and collected evidence.
• Authentication response consumption is bounded by the selected internal allowance, without treating overflow as rejected credentials.
• Identified unnecessary retention, repeated traversal, and temporary allocation are reduced with proportionate changes that preserve established behavior.
• Production execution and its tests use the real generation coordinator; obsolete adapters, state, helpers, and duplicate test expectations are removed.
• Current documentation describes actual behavior, and deliberate compatibility changes are explained where consumers encounter them.

The resolved decisions are authoritative for the selected changes. They do not authorize stronger numerical precision, provider compatibility, scheduling, hash stability, or performance guarantees.

Scope Boundaries

Keep the work within the architecture policy and configuration format boundary. ConfigKit owns source composition and its raw-source diagnostics; PromptKit validates its final effective values.

This round does not include a schema bundler, a new local-only schema mode, an upstream ConfigKit extension, a numeric conversion subsystem, new inference limits or application deadlines, or a general reflection/validation framework. Schema bundling remains a future idea. Runtime provider overrides retain their distinct supported-value and copying behavior.

Refactoring does not authorize expanding recognized template expressions, introducing additional retries, adding pagination or runtime inventory caches, or redesigning the capacity manager. Tests should protect established semantics, not turn incidental implementation choices into new requirements. Existing privacy and ownership boundaries remain applicable throughout.

Workstreams

A. Maintainer Toolchain Baseline

Catalog: 1. Decision: 5.

Current gap: The audited local toolchain predates the networking fixes identified by the security audit. Toolchain selection and consumer compatibility must remain separate concerns.

Work: Use the latest locally available Go installation for ordinary builds, tests, and release validation. Update the local installation as needed for the identified fixes and repeat the focused assessment with the selected version. Make the ordinary-development instruction explicit in contributor guidance. Follow the release procedure and consumer-minimum policy: normally declare the initial stable release of the selected series, advance the series in minor releases, and retain the existing documented exceptions. Merely selecting a newer patch for validation does not raise the consumer minimum.

Completion evidence: Record the toolchain actually used and the focused reassessment outcome, distinguishing resolved findings from any remaining applicability. This does not establish a new recurring scan gate. Advising consumers about generally applicable Go vulnerabilities is outside PromptKit's responsibility; no such advisory work belongs in this roadmap.

B. Configuration And Schema Simplification

Catalog: 3, 5, 6, 15, 21. Decisions: 2 and 4.

Current gap: Convenience-source conversion broadens ConfigKit's accepted input forms by recursively rebuilding them, causing silent numeric changes and potentially exponential expansion. Schema validation loads external resources locally but sends only the root document to the provider. Compilation is repeated across direct/prepared and host-file/fs.FS paths.

Work:

1. Remove the provider-value compatibility adapter from ProfileSource and BackendSource, retaining ordinary field projection. Pass provider maps directly to ConfigKit Mapping for snapshotting and deferred diagnostics. Callers supply supported built-in scalars, nil, map[string]any, and []any recursively. Do not replace the removed adapter with a precision checker, pre-validation pass, or memoized converter.
2. Require self-contained schema documents. Continue loading the chosen root schema from its configured source; reject references requiring another document with an actionable preparation error. Preserve supported references within the document, including recursion where otherwise supported. Apply the selected resource policy consistently across direct and prepared schema compilation without changing malformed-output short-circuiting into eager I/O.
3. Simplify and share the remaining compilation lifecycle behind the needed source adapters. Express the runner's preparation dependency in its internal interface rather than discovering it through a runtime assertion. Preserve source containment, cancellation, frozen plans, and existing validation semantics outside the selected external-resource removal.
4. Cover WithSchemaFile through public construction and preparation/validation, including option precedence and relevant file errors. Coordinate removal of its unused private helper result with workstream F.

Completion evidence: Focused source-helper cases demonstrate supported-value handoff, immediate source capture, and ConfigKit's deferred unsupported/cyclic value diagnostics. Include formerly converted input forms to demonstrate the migration boundary; do not duplicate ConfigKit's entire limit suite. Schema cases establish external-reference rejection before provider dispatch, retained in-document references, and an outbound self-contained constraint usable without other local files. Retain independent source-path and direct/prepared coverage. These checks establish packaging completeness, not universal provider support.

Compatibility: Explain caller conversion of json.Number, named scalars, typed containers, arrays, and nested pointers into ConfigKit-supported forms. Explain removal of external schema loading. No arbitrary-precision promise or change to runtime request overrides accompanies this simplification.

C. Ownership, Errors, And Diagnostic Evidence

Catalog: 2, 4, 7, 8, 9, 11, 26, 27. Decisions: 1 and 3.

Current gap: The documented artifact buffer lifetime permits races; nested errors can obscure their owning operation; response read failures can be misclassified; some pre-dispatch smoke failures produce a completed report; and successful authentication responses are read without a byte bound.

Work:

1. Align artifact GoDoc, adapters, and tests with successful Read transferring the artifact and mutable body storage to PromptKit. Readers stop mutating that storage, including retained aliases. Keep the interface signature and existing defensive copy initially. Readers may reuse private buffers when they return an independent copy; no release protocol is introduced.
2. Preserve the owning public error category independently of nested causes, retaining appropriate errors.Is/errors.As access. Do not require category exclusivity: retained causes can make more than one category match. Preserve meaningful direct-not-found versus load-failure distinctions.
3. Distinguish successful-response body I/O failures from malformed JSON at both initial decoding and end verification. Preserve the redacted transport boundary, cancellation precedence, and genuine malformed, truncated, or oversized-content behavior without adding retries.
4. Return local inference-diagnostic construction failures as ErrInvalidRequest using explicit preparation-stage information. Preserve completed diagnostics for missing credentials, capacity outcomes, and unsupported/inapplicable operations; absence of dispatch alone does not imply an invalid request.
5. Define the authentication response allowance as a named internal constant of 1 MiB (1,048,576 bytes). Enforce against actual reads regardless of declared length, detect overflow, and close without unbounded draining. Report an explicit limit diagnostic with credential acceptance unknown and retain available HTTP status/timing. Choose its concrete report representation during implementation design; overflow is not malformed JSON or credential rejection. No public configuration knob is added now.
6. Move OpenRouter-specific authentication request/response interpretation into the outbound adapter. Leave selection, applicability, completion, and neutral reporting in the use case. Preserve anonymous probing and credential/body privacy. Narrow smoke copying to actual consumers, removing unused request cloning and replacing full-report cloning used only to detach a target.
7. Exercise RunRequest's privacy formatting and JSON omission boundary with synthetic markers and the relevant value/pointer forms.

Completion evidence: Focused concurrent artifact tests follow the selected ownership contract; they do not promise safety for readers that violate it. Offline regressions cover composed public causes, both response read-failure locations, construction failures with no dispatch, and authentication boundary and overflow cases including missing/inaccurate lengths. Assert diagnostic facts and privacy rather than complete incidental serialized shapes. Preserve active backend copying, facade projection, and provider-map error behavior.

Dependencies: The authentication bound can land before protocol relocation; cleanup must not delay the defect remedy. Correct error/stage behavior before restructuring generation accounting in E.

D. Resource Use And Inspection Efficiency

Catalog: 10, 16, 17, 18, 19, 20.

Current gap: The audited paths repeatedly copy requirement evidence, retain unneeded backing storage or engine ownership, serialize unbounded diagnostic text before truncation, reclone overlay descendants, and scan complete catalogs for namespace-filtered listings.

Work and completion evidence:

Area	Intended remedy	Evidence and limits
Requirement evidence (10)	Accumulate or merge evidence without repeatedly copying growing sets.	Exercise representative left-heavy conditions; preserve classification and conservative coverage without new expression limits.
Prepared handles (16)	Prefer a small standalone ownership identity that does not retain the runner graph.	Preserve foreign-owner checks, one-time claim, namespace-view behavior, details, and concurrent claim/discard behavior. Identity cannot rely on distinct zero-sized allocations comparing unequal.
Repair feedback (17)	Encode bounded feedback incrementally rather than fully serializing then truncating.	Preserve output semantics, stored diagnostics, and previous candidate; measure large and heavily escaped diagnostics. No exact peak-heap guarantee or new input limit.
Session IDs (18)	Detach the accepted normalized string from a larger caller allocation.	Preserve trimming, content/code-point validation, wire values, and hashes; no raw-input restriction.
Provider overlays (19)	Avoid repeated descendant cloning while preserving owned results.	Compare representative nested overlays and preserve recursive-map, null, array, and isolation behavior.
Namespace listings (20)	Capture buckets or ranges to query the relevant inventory subset.	Evaluate sparse/absent matches, ordering, exact identity, cancellation, caller-owned results, and construction/retained-memory cost.

Measurements guide proportionate remedies, particularly for 17, 19, and 20; they are not new performance contracts, benchmark thresholds, or recurring release gates. If a proposed optimization's benefit does not justify its complexity, record the evidence and an explicit disposition rather than silently marking it implemented. Do not broaden the remediation merely to improve a synthetic benchmark. Coordinate evidence accumulation with E's traversal work and overlays with E's compatible-copying work.

E. Internal Coordination And Test Assembly

Catalog: 22, 23, 24, 25, 34. Decision: 6.

Current gap: Large functions combine distinct responsibilities, JSON copying is duplicated across boundaries with different policies, and a test-only legacy capacity wrapper remains a production constructor.

Work:

• Extract focused configuration validation/materialization helpers with visible inputs and ownership. Preserve final-handoff validation, presence/null rules, origins, and meaningful error precedence without pinning incidental ordering of independent failures.
• Separate requirement recognition/evaluation from reachability and evidence bookkeeping, sharing if/with orchestration where appropriate. Preserve scope differences, inactive computed-name behavior, and independent comparison with the actual renderer. This does not expand inspection into an interpreter.
• Consolidate structural copying only where policies genuinely agree. Keep facade projection and runtime validation distinct; preserve existing types, nils, supported numbers, schema keys, and ownership. Infallible projection must not acquire new errors or silent fallbacks. Retain small separate walkers if sharing would require a configurable reflection framework. B's removed source compatibility adapter must not migrate into this shared machinery.
• Separate generation attempt results and accounting from dispatch/retry control flow with small internal helpers. Preserve built-in/injected differences, permit release before callbacks or backoff, frozen requests, existing retry classification, private evidence, and known/unknown usage distinctions.
• Move the initial/repair runner scheduling test to the production generation coordinator, then remove the legacy capacity client wrapper. Preserve useful FIFO, cancellation, independent-pool, and panic-release scenarios with their owning components. Keep any useful one-shot scaffolding test-only; do not wrap the coordinator in another permit-acquisition layer.

Completion evidence: Existing meaningful behavioral tests and focused cases for changed seams continue to exercise real internal collaborators. Tests no longer require the retired production wrapper. Review demonstrates simpler ownership/control flow; lowering complexity metrics alone is not an acceptance criterion. No capacity-manager redesign or public DTO unification is needed.

F. Documentation, Representation, And Test Maintenance

Catalog: 12, 13, 14, 28, 29, 30, 31, 32, 33.

Current gap: Several comments contradict implemented behavior, expectations are duplicated or over-specified, and internal fields/helpers have no consumer.

Work:

• Correct credential-inspection GoDoc and its example: explicit profile credential environment selection can coexist with required credentials; only the backend-derived default is suppressed in that case. Correct provider-map wire documentation to distinguish configured recursive overlays from runtime replacement. Remove stale trimmed-ID and prompt-version comments.
• Give complete bundled-catalog expectations one independently specified test owner. Retain smaller tests for distinct loading, wire, and smoke behavior, using controlled fixtures where bundled content is irrelevant.
• Replace literal artifact-hash algorithm assertions with documented opaque equality/ownership and byte/size behavior; keep benchmark setup outside timed work. This does not require changing the artifact hash algorithm.
• Remove unused backend presence records and QueueCapacitySet, preserving effective queue defaults, explicit zero, provenance, and needed profile/input presence tracking.
• Remove duplicated captured prompt output-format and unused message-file fields. Preserve actual content_file loading, origins, and override-default behavior. Account for observable definition-hash changes without promising permanent encoding or bit-identical hashes across this change.
• Remove unused private reachability endpoint state and the unused schema-helper result, preserving actual report evidence and public single-file behavior. Remove orphan filecatalog.RelativePath, resourceNamespaceFromID, and the unused runner-test stringPtr helper with their obsolete self-tests.

Completion evidence: Documentation agrees with the owning GoDoc/format/wire contracts. Caller checks precede deletion; meaningful discovery, identity, namespace, schema, privacy, and bundled behavior remain covered. Tests do not merely reproduce the loader's output as their expected answer.

Sequencing For The Implementation Plan

Use reviewable units with tests and owning documentation included. Workstream letters group outcomes; they are not mandatory serial phases.

1. Establish A's toolchain baseline for implementation validation. Independent documentation work can proceed without waiting for that environment work.
2. Prioritize B's simplifications and C's confirmed defects. They can be split into independent units; avoid bundling unrelated public behavior changes.
3. Sequence B's schema policy with its compilation simplification, and remove source conversion before E's copying consolidation. Coordinate B's schema coverage with F's helper-result removal.
4. Fix C's error/stage behavior before E's generation accounting refactor. Bound authentication reads before or alongside relocating the protocol adapter, with the behavioral remedy independently reviewable.
5. Coordinate D's evidence algorithm with E's traversal cleanup and D's overlays with E's copying work. Preserve each remedy's independent evidence rather than assuming a general refactor also fixes its associated defect.
6. Migrate the scheduling test before deleting the wrapper. Adjust over-specified hash tests before or alongside captured-definition cleanup. Fold other small F changes into their owning units where useful, without obscuring fixes.

The later plan should assign concrete implementation units, validation, and status to these outcomes. No significant product decisions remain open now. Report representation, helper extraction boundaries, and index layout remain ordinary implementation choices within the selected behavior. Escalate only new evidence requiring a contract or scope change.

Compatibility And Documentation Delivery

The selected ownership, schema, and convenience-source changes are deliberate pre-1.0 compatibility changes. Deliver them in a minor release under the existing release policy, with migration notes explaining independent artifact storage, self-contained schemas, and ConfigKit-supported input types. Mention changed definition hashes so consumers can account for comparisons or caches; do not establish a stable encoding. Release selection and publication are not part of writing this roadmap.

Update exported GoDoc, consumer guidance, formats, wire integration, and relevant internal documentation when their owning behavior lands. Clarify ordinary development toolchain selection in development guidance, linking to existing release policy. Correct standalone documentation defects without waiting for unrelated code.

Validation And Completion

Follow the testing policy and existing maintainer validation workflow. Use focused behavioral regressions for demonstrated defects, race testing for changed concurrent ownership/scheduling paths, and offline wire tests where local success does not prove the emitted request is correct. Reuse successful checks for unchanged content; release validation remains the existing workflow. No live inference, provider credentials, generalized benchmark suite, or new recurring release gate is introduced.

The workstream catalog lists partition all 34 entries. During implementation, record for each workstream its completed units, evidence, and any explicitly justified deferral. A measured deferral is not an implemented fix; optional cleanup does not hold unrelated fixes hostage. Full implementation completion requires the selected changes, their relevant validation, migration guidance, and removal of obsolete documentation/tests. Any remaining work must be named and retained in an appropriate roadmap rather than lost during consolidation.

When this roadmap's work is complete or otherwise explicitly disposed of, move still-useful contract information to its canonical owners and retire temporary planning documents under the documentation lifecycle. Keep audit evidence and decision provenance discoverable while they remain useful; do not maintain competing current-behavior references.