7.8 KiB
PromptKit Adoption Review Findings
Review baseline: Scriptorium c5e54edaa5f6621d504cf510b4b5b5c1e4d1cfd4,
with the approved PromptKit v0.23.3 pin and ConfigKit v1.2.1. Stage numbers
refer to the implementation plan.
R1: Bound the HTTP timeout regression tests (Stage 4)
Complete. The tests use independent parent deadlines and capture the target invocation's error without sharing a channel. All three passed twice under the race detector.
In handler_test.go,
TestHandlerDeadlineDuringBuiltInGeneration runs the handler synchronously
while the fake provider waits on a channel that is released only after the
handler returns. If the operation timeout stops reaching PromptKit, the test
can wait for the unrelated provider/transport timeout or the suite watchdog
instead of promptly failing. The retry- and capacity-wait tests likewise rely
on the behavior being tested to return control. The added
observeRunnerFailures helper also sends all failures to a shared,
one-slot channel before allowing each runner invocation to return. An
unexpected failure of the first request in the capacity test can fill that
slot and block the second request from returning the failure the test needs
to inspect.
Use a test-only fallback deadline or bounded synchronization independent of
the operation budget, and capture the error of the specific observed
invocation without a blocking send shared by concurrent requests. Keep the
real engine and loopback provider, the assertion on
ErrOperationTimeout, and the existing status, observation, and
capacity-release checks. This is a test liveness fix, not a new production
timeout or release gate.
R2: Describe source-relative message paths without a colocation rule (Stage 7)
Complete. The migration guide describes source-relative resolution and links to PromptKit's path rules. A render with a message in a subdirectory confirmed that no colocation is required.
The upgrade guide instructs users to
keep message files beside their owning definitions. PromptKit resolves
content_file against the field's declaring source within its rooted
filesystem; it does not require the file to be in the same directory. A
freshly built Scriptorium binary rendered a valid content_file: ./messages/request.md without moving the file beside its framework YAML.
The current wording can lead operators to rearrange valid layouts during
migration.
Describe source-relative resolution and link to the pinned PromptKit format contract for containment rules. A colocated file can remain an example, not a requirement. No loader or path-policy change is needed.
R3: Link the Go Text patent grant notice (Stage 7)
Complete. The attribution entry links PATENTS at the selected Go Text
commit. The notice exists in the pinned module, and the local BSD text and
upstream LICENSE link are unchanged.
The attribution entry for
golang.org/x/text v0.44.0 links LICENSE but omits PATENTS. Build
dependency inspection confirms Go Text is part of the executable. Its
selected module contains the separate patent grant at upstream commit
6070ac32abcdd1670599eeb3ee4d513c8184a513; PromptKit's attribution for
that same revision links it. The OSS policy
calls for applicable companion texts to be directly reachable.
Add a PATENTS link at the already selected Go Text commit. Retain the
verified local BSD text and existing LICENSE link. No separate local
patent notice or change to the project license is needed under the
adopted attribution format.
R4: Distinguish discovery peers from ordered framework layers (Stages 6–7)
Complete. The migration guide and configuration reference distinguish peer conflicts from ordered replacement. The same two definitions were verified to conflict as peers and replace as separate selectors.
The upgrade guide says a later source
selecting the same ID replaces the whole record. That holds for a later
framework underlay, not for two checkout entries selected by the same
discovery file. Scriptorium turns each resolved promptkit_sources selector
into one underlay. The entries discovered by the imports of that selector
are composed as peers; their filename, glob, or checkout order is not
override precedence.
The built binary rejected two discovered checkout entries defining
shared/fast with compose.conflict. Selecting the very same entry files
as two ordered promptkit_sources items instead succeeded: the later
record's model was selected and max_tokens, absent from that record, did
not carry forward from the earlier one. This is the intended distinction, but
the migration wording can lead operators to put an intended override in
a discovery peer, which prevents the engine from loading.
Qualify whole-record replacement in the upgrade guide and explain in the checkout discovery reference that same-ID conflicts within one discovery underlay fail. For an intentional override, select a separate, later framework source using the existing application selector list. No scanner, de-duplication, new namespace restriction, or change to composition is needed.
R5: Replay framework explanation history once (Stage 3)
Framework configuration inspection
calls view.Explain(path) to check existence, discards the complete
explanation, then calls view.WriteExplanation. The pinned native writer
calls Explain itself. A successful request therefore replays the same
captured history and builds the same redacted value snapshots twice. If
H is the work to construct that explanation and W the work to render
it, the command does roughly 2H + W work instead of H + W. This is a
constant-factor cost, not a new quadratic algorithm, but it repeats a
history- and value-sized operation and its allocations. The legacy wrapper
already used two calls; the native inspection path need not retain them.
Delegate once to view.WriteExplanation, as app configuration inspection
already does with its native writer. A direct probe with the selected
PromptKit and ConfigKit releases confirmed that this writes a known path,
returns an error without output for an unknown path, and preserves writer
errors. Keep JSON Pointer parsing, native redaction, and the existing exit
and stream behavior; no new formatter or replay algorithm is needed.
R6: Prevent artifact read-limit overflow (Stage 4)
The restricted HTTP artifact reader
allows the same nonnegative int64 limit as application configuration
and the CLI. It computes maxBytes + 1 for io.LimitReader to detect
oversized files. At the valid setting 9223372036854775807
(math.MaxInt64), that addition wraps to a negative number. The limited
reader then immediately returns EOF, so a nonempty file is accepted as an
empty artifact. A loopback check confirmed HTTP 200 at this setting while
the outgoing prompt silently lost the file content. The same synthetic
request included the file at zero and at the ordinary positive limit.
The defect predates the migration and remains in the reader retained
by Stage 4.
Use overflow-safe byte-limit arithmetic: for example, retain the capped
limit + 1 read for smaller positive limits and skip the increment at
math.MaxInt64, or limit the read first and probe separately for an extra
byte. Accepted files should still require linear time and buffer space in
the bytes read; normal limits should continue to reject oversized files
without a full read. Add a boundary test using a small nonempty file.
Do not reduce the accepted configuration range or change containment
or the zero-limit semantics.