Retire the legacy CLI and document native operator workflows

This commit is contained in:
2026-11-20 05:08:40 +00:00
parent 10387d4aea
commit bd7136ceeb
52 changed files with 846 additions and 7067 deletions

View File

@@ -5,6 +5,9 @@ an HTTP service. It prepares prompt requests, runs them against
OpenAI-compatible model endpoints, and returns generated output with validation
metadata.
CommandKit supplies native configuration, resource discovery, preview, reporting,
and completion interfaces alongside application generation and HTTP startup.
The application uses
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/)
for prompt, profile, schema, preparation, generation, and validation behavior.
@@ -16,11 +19,10 @@ behavior, output presentation, and HTTP artifact-containment policy.
From the repository root:
```bash
go run ./cmd/scriptorium render \
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--input-file transcript=./examples/fixtures/transcript.md \
--input-file glossary=./examples/fixtures/glossary.yml \
--format json
```
@@ -38,6 +40,7 @@ a model. For complete invocation and output behavior, see the
- [Operations guide](docs/operations.md)
- [Consumer integration overview](docs/consumers/api.md)
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
- [Native CLI migration](docs/consumers/upgrading-cli.md)
- [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md)
- [Subprocess integration](docs/integrations/subprocess.md)
- [Architecture policy](docs/policy/architecture.md)

View File

@@ -1,377 +1,186 @@
# CLI Reference
This is the canonical contract for invoking Scriptorium. Configuration discovery,
precedence, application source locations, and server settings are defined in the
[configuration reference](config.md). The [HTTP API reference](api.md) owns
service request and response behavior.
## Shortest Useful Command
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
`render` prepares a request without calling an LLM.
Scriptorium uses CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
This document owns application invocation and delivery behavior. The pinned
[upstream CLI contract][native-cli] owns standard family grammar and reports.
[Configuration](config.md) owns discovery, settings, and source composition;
the [HTTP API](api.md) owns the service contract.
Existing callers should read the [CLI migration guide](consumers/upgrading-cli.md).
## Commands
- `scriptorium run`: prepare a prompt, call the configured LLM, and write the
generated artifact.
- `scriptorium render`: prepare a prompt and write prepared-run output.
- `scriptorium serve`: start the HTTP server.
- `scriptorium inspect prompt`: inspect one prompt definition without model execution.
- `scriptorium inspect profile`: inspect one resolved profile without model execution.
- `scriptorium inspect configuration`: validate or inspect Promptkit's captured framework configuration.
- `scriptorium config validate`: validate application configuration without building an engine.
- `scriptorium config show`: show the resolved application configuration.
- `scriptorium config sources`: show the ordered configuration sources.
- `scriptorium config explain`: explain one effective or historical configuration path.
All command families accept `--config <path>` and `--config-overlay <name>`;
positional arguments are rejected. `--config-overlay` selects one application
configuration overlay; it is separate from `--profile`, which selects a
Promptkit execution profile. See [configuration composition](config.md#explicit-imports-and-selected-overlays)
for name, root, and absent-main behavior. `run`, `render`, `serve`, `inspect prompt`,
`inspect profile`, and `inspect configuration` accept repeated
`--promptkit-source <path>` flags and `--schema-dir <dir>` where applicable.
They replace the configured source sequence in flag order. `--prompt-dir` and
`--profile-dir` are no longer supported; use `promptkit_sources` in application
configuration or the repeated source flags. Application `config` commands
remain independent of framework source loading. Explicit application flags,
including empty values, override file settings. Configuration load diagnostics
go to stderr and exit with status 1; successful load warnings are printed once
before command work.
## `scriptorium run`
```text
scriptorium run [flags]
```
Required flags:
| Flag | Meaning |
| --- | --- |
| `--prompt <id>` | Prompt ID to execute. |
Optional flags:
| Flag | Meaning |
| --- | --- |
| `--config <path>` | Application configuration file. |
| `--config-overlay <name>` | Selected application configuration overlay. |
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
| `--schema-dir <dir>` | Schema base-directory override. |
| `--profile <id>` | Execution-profile override. |
| `--session-id <id>` | Optional direct session identifier. |
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
| `--out <path>` | Write generated content to this file instead of stdout. |
| `--execution-report <path>` | Write an opt-in terminal execution report as newline-terminated JSON to a separate file. |
| `--llm-base-url <url>` | Runtime endpoint override. |
| `--model <name>` | Runtime model override. |
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
| `--temperature <float>` | Runtime temperature override. |
| `--max-tokens <int>` | Runtime maximum-token override. |
| `--top-p <float>` | Runtime top-p override. |
| `--reasoning-effort <value>` | Runtime reasoning-effort override. |
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
| `--run-timeout <duration>` | Optional whole-run engine deadline using Go duration syntax; zero or omission adds no deadline, negative values are invalid. |
| `--transient-retries <integer>` | Nonnegative additional transient attempt count for this request; `0` disables additional attempts. |
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
`--profile`.
Omitted numeric runtime flags preserve the selected effective value; explicit
zero values override it. `--timeout 0s` disables the per-attempt generation deadline
only; the caller context and configured transport cap remain active. CLI
durations are converted to whole seconds by truncation toward zero, so any
duration whose absolute value is below one second becomes an explicit
zero-second override. The timeout layers are defined in the
[Promptkit outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
With the built-in client, each transient retry gets a fresh generation timeout;
this flag does not limit the complete run.
`--run-timeout` keeps subsecond precision and starts immediately before engine
execution. It bounds preparation, capacity waits, retry waits, generation, and
repairs where they honor context. Parsing, engine construction, and output
writes are outside that interval. A failed run identified by PromptKit as an
operation timeout exits 1 with a safe deadline diagnostic; a completed invalid
output still exits 2. Synchronous work may not stop immediately.
Omitting `--transient-retries` inherits the application setting or Promptkit's
current default of two additional attempts. The flag overrides only this
request; it does not alter the configured engine. Structural repair generations
have separate transient retry budgets.
There is no raw API-key flag. Use `--api-key-env`.
Prompt and profile IDs are passed unchanged, including collection-qualified
IDs. The removed `--prompt-version` flag is rejected. `--input` is optional
at the CLI boundary: PromptKit decides whether the selected definition requires
declared inputs or template-referenced values.
`--session-id` supplies a direct, non-secret session identifier. A nonblank
value replaces a definition-rendered session ID; an omitted or blank value lets
the definition supply one. Promptkit trims direct values and limits them to 256
Unicode code points. Use stable, non-sensitive identifiers because effective
session IDs may appear in prepared output, run metadata, and provider-facing
requests.
`--reasoning-effort` is presence-aware: omitting it inherits the selected
profile value, a nonblank value replaces that value, and
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
nonblank values as provider-specific opaque strings.
### Execution report file
`--execution-report` requires a nonempty path. It is available only for `run`;
`-` is an ordinary filename. Parent directories are not created. An existing
file is replaced through ordinary file writing, without rollback or an atomic
replacement guarantee. Generated content still goes to stdout or `--out`.
Equal output/report paths, including existing paths that refer to the same
file, are rejected before execution.
The file is written after engine execution when Promptkit supplies a terminal
event, including for a failed run or invalid generated output. A failure before
invocation leaves the report path untouched. If no terminal event is supplied,
Scriptorium leaves the path untouched and emits a diagnostic; an existing file
may therefore contain an older report. Artifact output is attempted first on a
successful run, and report writing is attempted even if artifact writing fails.
Any output or report write failure exits 1; already-written output is not
undone. Without a write failure, the actual run keeps its usual exit status.
The [HTTP API reference](api.md#execution-report) defines the shared report
fields and usage-presence semantics. The file contains that JSON object followed
by a newline.
The report excludes content, credentials, URLs, headers, and raw errors, but
its identifiers can still be sensitive. The caller owns file retention and
access.
## `scriptorium render`
```text
scriptorium render [flags]
```
`--prompt <id>` is required. The following optional flags are supported:
`--config`, `--config-overlay`, repeatable `--promptkit-source`, `--schema-dir`, `--profile`,
`--input`, `--var`, `--out`, `--llm-base-url`,
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
`--reasoning-effort`, `--session-id`, `--timeout`, `--transient-retries`, and `--format text|json`. Their meanings match the corresponding
`run` flags; `--format` selects prepared-run output and otherwise uses
`defaults.render_format`.
The same deprecated aliases and numeric/timeout behavior as `run` apply.
The same session and reasoning inheritance, replacement, and clearing behavior
also applies.
`render` validates the retry override but does not generate model output or
include a retry field in prepared output.
`render` resolves profiles and schemas as part of preparation but
does not call an LLM.
To render from independently maintained collection checkouts:
```bash
go run ./cmd/scriptorium render --config ./examples/config.discovery.yml \
--prompt support/answer --profile support/fast \
--input transcript=./examples/fixtures/transcript.md
```
The [discovery example](../examples/discovery/framework.yml) also provides
`review/answer`; see [configuration discovery](config.md#collection-checkout-discovery).
## `scriptorium serve`
```text
scriptorium run PROMPT_ID [flags]
scriptorium serve [flags]
scriptorium config scopes
scriptorium config validate|show|sources|explain|diff [flags]
scriptorium promptkit list prompts|profiles|backends|namespaces|collections [flags]
scriptorium promptkit show prompt|profile|backend ID [flags]
scriptorium promptkit requirements show|validate PROMPT_ID [flags]
scriptorium promptkit show prepared PROMPT_ID [flags]
scriptorium promptkit test KIND [flags]
scriptorium help [COMMAND ...]
scriptorium version
scriptorium completion bash|zsh|fish
```
Optional flags:
Native help is also available with `--help`. Help, version, scope discovery,
and static completion do not load configuration or contact a provider.
Version/revision metadata is supplied by the build; ordinary builds identify
themselves as `dev` with no revision. Release build mechanics belong to the
[release procedure](release.md).
Commands that consume configuration accept `--config PATH` and `--overlay NAME`.
An explicitly empty config value remains explicit; overlay names follow
[ConfigKit selection](config.md#explicit-imports-and-selected-overlays).
There are no source/schema convenience flags or legacy command aliases.
Native shared `--format`, `--quiet`, `--verbose`, `--debug`, and progress
controls retain their upstream meanings. Diagnostics use stderr, not stdout.
Verbose exposes emitted Info and Warning messages without changing result bytes.
## Generation
`run` requires exactly one positional prompt ID. It has no duplicate prompt
flag or direct backend-selection flag; choose a profile for backend routing.
It uses these native bindings, with the preferred aliases below. Canonical
spellings prepend `llm-` (for example, `--llm-profile`).
| Preferred flags | Meaning |
| --- | --- |
| `--profile ID`, `--session-id ID` | Profile selection and optional non-secret session context. |
| `--var KEY=VALUE` | Repeatable caller-variable mappings. |
| `--input-text NAME=TEXT`, `--input NAME=TEXT` | Repeatable literal input mappings; `--llm-input` is also a literal-text alias. |
| `--input-file NAME=PATH` | Repeatable file references. |
| `--endpoint URL`, `--model NAME` | Native endpoint/model overrides. |
| `--temperature N`, `--max-tokens N`, `--top-p N` | Presence-aware numeric overrides, including zero. |
| `--timeout-seconds N` | Integer per-attempt generation timeout; zero disables that deadline, not caller cancellation or the transport cap. |
| `--reasoning-effort VALUE`, `--service-tier VALUE` | Native provider controls; an explicit empty reasoning value is preserved. |
| `--api-key-env NAME` | Credential environment-variable name, never raw key bytes. |
| `--extra-params JSON` | Native JSON-object provider parameters. |
| `--transient-retries N` | Request retry override; omission leaves the engine default. |
| `--prompt-namespace NAME`, `--profile-namespace NAME`, `--backend-namespace NAME` | Independent native namespace views. Qualified resource IDs remain exact; stored references are not rewritten. |
Mappings split at the first `=`; commas and subsequent equals signs are literal.
Repeat flags for multiple mappings. The last text/file occurrence for the same
input wins in command-line order. PromptKit validates selected definitions,
inputs, namespaces, provider settings, and credential requirements; Scriptorium
does not add an execution-readiness preflight or configurable default namespace.
Native string/pointer override semantics remain upstream-owned.
Application-owned generation controls:
| Flag | Meaning |
| --- | --- |
| `--config <path>` | Application configuration file. |
| `--config-overlay <name>` | Selected application configuration overlay. |
| `--addr <listen-address>` | HTTP listen-address override. |
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
| `--schema-dir <dir>` | Schema base-directory override. |
| `--artifact-root <dir>` | Root for HTTP `file` input references. |
| `--max-request-bytes <n>` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
| `--max-artifact-bytes <n>` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
| `--max-response-bytes <n>` | Maximum encoded HTTP response bytes; `0` disables the limit. |
| `--run-timeout DURATION` | Go duration passed unchanged to PromptKit's operation budget. Positive values cover engine preparation/execution, including cooperative waits and repairs. Zero, omission, and negative values add no budget. Parsing, configuration/engine setup, and delivery are outside it. |
| `--report PATH` | Additionally write the returned native result to a separate file. |
| `--report-format text\|yaml\|json` | Explicit separate-report format; omission leaves native automatic rendering selected. |
`serve` accepts no runtime model override flags. HTTP request fields, response
schemas, and error codes are defined in the [HTTP API reference](api.md).
The operation budget is independent of the integer per-attempt timeout.
Earlier parent cancellation/deadlines still apply. Cancellation is cooperative;
synchronous work that ignores context cannot be forcibly interrupted.
## `scriptorium inspect prompt`
### Results And Separate Reports
```text
scriptorium inspect prompt --prompt ID
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
```
The main report goes through the invocation reporter to stdout. Automatic/text
output is exactly `Artifact.Body`, with no added newline, header, escaping, or
summary. JSON/YAML output is the full native PromptKit `RunResult`, without a
Scriptorium envelope. In native structured serialization, artifact bytes use
the upstream representation (JSON base64), not an application text projection.
`--prompt` is required. Inspection uses normal configuration discovery and the
selected Promptkit sources, defaults to text regardless of
`defaults.render_format`, and writes to stdout unless `--out` is supplied.
Save main output with shell redirection or the subprocess stdout destination.
There is no `--output` or `--out` flag. Omit `--quiet` when capturing the main
result: native quiet suppresses it even when stdout is redirected.
Both definition-inspection commands first construct the same engine used for
execution from the complete selected source sequence. Construction resolves
and validates all effective definitions, including unused ones, and captures
referenced message-file contents and static template syntax. Invalid unused
definitions or unreadable message resources can therefore prevent inspection.
No prompt collection is mandatory, but selected prompts are not ignored.
`--report` renders the same result through CommandKit without the artifact-only
text template. Its format is independent of stdout's `--format` and the filename
extension. Omission passes an unset format to the upstream renderer; it currently
renders structured values as YAML in automatic/text mode. Choose an explicit
format when required rather than relying on that upstream default.
Inspection does not prepare an execution request, load request-time schemas or
artifacts, resolve credential values, reserve backend capacity, or call a model.
Prompt inspection does not select an execution profile or render request-specific
templates.
Prompt output includes ID, hash, optional default profile and input metadata,
and the output contract. Absent optional values are omitted; explicitly empty
or zero values are retained. Captured message text is not projected.
The requested file is written even in quiet mode. Omission creates no separate
report. An explicitly empty path is a usage error before generation; `-` is an
ordinary filename. New files use mode 0644 subject to umask, existing files use
normal OS replacement behavior, and parent directories are not created.
Render, write, and close failures are reported.
## `scriptorium inspect profile`
Only a returned result is eligible for delivery. A resultless failure leaves an
existing report untouched; it does not synthesize an invocation-error result.
Main delivery is attempted first, then the requested report even if main
delivery fails. Writes are nontransactional; bytes already written remain.
Callers own redirected paths, retention, and keeping main and report destinations
separate. A shell may truncate its redirected file before the process starts.
```text
scriptorium inspect profile --profile ID
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
```
## Application Configuration Inspection
`--profile` is required. After the [full-source engine construction described
above](#scriptorium-inspect-prompt), inspection selects one resolved profile
with inheritance. `profile_model_params` contains only fields present on that
profile; it does not fill backend defaults. `api_key_required` is a separate
optional field. Explicit zero and empty values remain visible. Use `render`
for request-completed model settings.
The native `config` family defaults to `app`; select `--scope promptkit` for
captured framework configuration or use native scope discovery/all-scope modes.
`app` uses the actual completed ConfigKit result and does not construct an engine.
## Application configuration inspection
Native `--set POINTER=JSON` is enabled only for `app`. Records go directly to
ConfigKit: duplicate and ancestor/descendant-overlapping paths are rejected,
not resolved by flag order. No application-specific configuration flags or
custom `--against-*` counterparts are installed.
```text
scriptorium config validate [flags]
scriptorium config show [flags]
scriptorium config sources [flags]
scriptorium config explain --path JSON_POINTER [flags]
```
Native diff uses independent `--against-config`, `--against-overlay`, and
supported `--against-set` selections. Omitted right-side values do not inherit
the left side. See the [upstream contract][native-cli] for focused paths,
metadata discovery, source IDs, findings, explanations, and detailed comparisons.
Each accepts `--config`, `--config-overlay`, `--schema-dir`, `--addr`,
`--artifact-root`, `--max-request-bytes`,
`--max-artifact-bytes`, and `--max-response-bytes`. These are application
configuration selectors and overrides; request, credential, backend-editing,
output-format, and output-file flags are not accepted. Positional arguments
are rejected.
## Framework Configuration Inspection
`validate` writes no success output; status 0 means the application configuration
loaded and passed its own validation. It does not establish that selected
Promptkit sources can construct an engine, credentials are present, or a
provider is available. `show` writes ConfigKit's resolved YAML snapshot. `sources`
writes one newline-terminated JSON array of source records in load order. Each
record has `id`, `label`, and `layer`, plus `path` when available. Application
main-file and import records use the `main` layer; explicit CLI
overrides appear as synthetic sources. Defaults-only input without overrides
produces `[]` for `sources`.
The `promptkit` config scope loads application settings selected by `--config`
and `--overlay`, constructs an engine, and exposes its captured native view.
It does not accept temporary overrides; native capability checks also apply
when an all-scope operation requests them.
`explain` requires an explicitly supplied `--path` in JSON Pointer syntax.
The empty string selects the root; `/server/addr` selects a field; `~1` and
`~0` escape `/` and `~` inside mapping keys. Numeric tokens are interpreted
according to the containing mapping or sequence. Dotted paths, file paths,
and URL fragments are not accepted. ConfigKit writes a human-readable account
of the effective value and its chronological default, file/import, overlay,
CLI, normalization, and removed-value history. This is not a JSON schema or a
byte-for-byte stable prose format. Unknown and malformed paths exit 1 without
successful explanation output.
Framework effective/explanation views retain native classified-field redaction,
including history. Environment-reference names and ordinary values remain
visible. This is not arbitrary secret detection and does not redact all prompt,
prepared, application, or generated content. Protect reports accordingly.
Diagnostics and warnings go to stderr, never into successful stdout. Parse,
load, output, and diagnostic-write failures exit 1. Ordinary output writes may
leave partial data on failure. Application inspection can contain paths and
configured values; protect its output accordingly. Raw API-key values are not
accepted by the application configuration schema.
## Discovery, Preview, And Explicit Smoke Testing
## Framework configuration inspection
Inventories retain native defaults and `--namespace`/`--all-namespaces` filters.
Named inspections expose native prompt messages and profile/backend values;
prepared preview exposes rendered content, not a summary projection.
```text
scriptorium inspect configuration validate [flags]
scriptorium inspect configuration show [flags]
scriptorium inspect configuration sources [flags]
scriptorium inspect configuration explain --path JSON_POINTER [flags]
```
Requirements discovery works before input artifacts exist and without schema
loading, credential resolution, message rendering, or provider access. It still
loads configuration and captured message resources. Reports preserve exact names,
metadata, variable presence (including empty values), completeness, and issue
codes. Partial discovery may succeed with status zero; completeness is not
binding satisfaction or execution readiness. Concrete validation and preview
keep their native preparation-time prerequisites and make no generation call.
Each operation accepts `--config`, `--config-overlay`, repeatable
`--promptkit-source`, and `--schema-dir`. These flags select the application
settings and framework sources; positional arguments, `--format`, and `--out`
are not accepted. Unlike `config validate`, framework `validate` constructs
the Promptkit engine and validates all effective definitions, including
unused ones. It writes nothing on success and does not prepare a request,
resolve a credential, or contact a provider.
Smoke-test kinds are `reachability`, `authentication`, `text_inference`,
`structured_output`, or `all`, with native aliases as documented upstream.
These explicit commands may contact providers and incur cost. Their presence
does not add a startup/provider probe. Read the native findings: unsuccessful or
unsupported checks are not automatically invocation failures.
`show` writes Promptkit's native effective YAML, an inspection snapshot rather
than a promised reloadable file. `sources` writes a
newline-terminated JSON array in captured source order. Each source record has
`id`, `label`, `layer`, `kind`, and `underlay_index`, plus `path` when available
and `imported_from` for imported sources. That nested object has `source_id`,
`line`, and `column`. Values and output structure are a projection of the
captured Promptkit view, not Scriptorium's application-source records.
## Server And Statuses
`explain` requires an explicitly supplied `--path` in ConfigKit JSON Pointer
syntax; `--path ''` selects the root. For example,
`/collections/support/profiles/0/model` addresses the first support profile's
model in the captured list. Escape `/` as `~1` and `~` as `~0`; dots in a key
remain literal. Malformed and unknown pointers exit 1 without a successful
report. Explanation prose is native and is not a stable machine-readable
format.
`serve` accepts configuration selection and presence-aware HTTP controls:
`--addr`, `--artifact-root`, `--max-request-bytes`, `--max-artifact-bytes`,
and `--max-response-bytes`. It has no generation request controls. Values and
defaults belong to [configuration](config.md).
Framework `show` and `explain` use Promptkit's native redaction for its
classified credential-shaped provider fields, including historical values.
Environment-variable reference names and ordinary provider parameters remain
visible. Arbitrary prompt or provider content is not classified as secret by
name inspection, so treat these reports as sensitive. This redaction policy is
limited to framework configuration reports; it does not add redaction to
`inspect profile` or prepared-run output. Credentials are not resolved by any
of these inspection operations. Promptkit owns detailed field and source
semantics; see its tagged [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
Warnings and errors go to stderr. Successful `validate` writes nothing to
stdout; successful `show`, `sources`, and `explain` write their reports there.
Parse, load, path, and output failures exit 1. A native writer error may leave
partial output.
## Input And Variable Syntax
`--input name=path` maps an input name to a local file; `--var name=value`
maps a template variable to a string. Both flags can be repeated or contain
comma-separated mappings. Values may contain `=` after the first separator.
Empty names and values are rejected.
CLI inputs are file references. HTTP inline inputs are defined by the
[HTTP API reference](api.md).
## Output And Exit Behavior
- `run` writes generated content to stdout, or to `--out` when supplied, and
writes a concise summary to stderr. The summary includes `backend=<id>` when
Promptkit selected a backend; endpoint-only profiles omit it.
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
without a success summary. Text output includes `selected_backend_id` after
`selected_profile_id` when Promptkit selected one; endpoint-only profiles
omit it.
- `serve` writes startup and server errors to stderr.
Exit statuses:
Interrupt/SIGTERM cancels the invocation context. The server closes its listener
and active connections; canceled shutdown returns operational status, with no
graceful-drain deadline or guarantee that context-ignoring work has finished.
| Status | Meaning |
| --- | --- |
| `0` | Success. |
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. |
| `2` | `run` generated and wrote output, but validation failed. |
| 0 | Native success, subject to successful presentation/delivery. |
| 1 | Operational failure, including lookup/preparation/generation/capacity, cancellation/timeouts, unclassified load errors, and delivery failure for an otherwise successful run. |
| 2 | Native usage/declaration-input errors, or configuration errors classified as invalid input by native diagnostic evidence. |
| 3 | Native `config diff --check` found differences. |
| 4 | Completed generation returned `ValidationFailed`; artifact and native result remain deliverable. |
A backend admission rejection is a runtime error and prints `run error: model
backend capacity is exhausted`. The HTTP capacity response is defined in the
[HTTP API reference](api.md).
Native finalization handles presentation errors: success/usage become operational;
an already nonzero domain outcome retains its classification/status. A report-file
failure preserves completed-validation status 4 while adding its explanation and
cause. An operational failure to perform validation is status 1, not status 4.
JSON selection does not turn failures into a separate JSON error envelope.
## Workflows And Examples
The [maintained render script](../examples/render-markdown-summary.sh) is a
copyable render workflow. The [HTTP request example](../examples/http-run.json)
is for a running `serve` process.
[native-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md

View File

@@ -1,87 +1,126 @@
# Configuration Reference
This is the canonical reference for Scriptorium application settings. Prompt,
profile, backend, schema, execution-setting, built-in profile, and framework
credential semantics are defined by the
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and
outcomes, see the [HTTP API reference](api.md). For migration examples, see
[Upgrading Promptkit sources and definitions](consumers/upgrading-promptkit.md).
This document owns Scriptorium's application settings, discovery, and source
selection. [CLI syntax](cli.md) and the [HTTP API](api.md) have separate owners.
Framework definitions and execution semantics belong to the pinned
[PromptKit v0.23.3 formats][formats]; ConfigKit v1.2.1 owns
[native composition and inspection][configkit].
## Discovery And Precedence
ConfigKit assembles the selected main source, imports, and overlay, resolves
inheritance, and decodes the result with Scriptorium's typed defaults. It then
applies explicitly supplied application CLI values, records application
normalization, and validates the result. Source, composition, inheritance, and
decoding failures happen before CLI assignment and cannot be repaired by it;
an override can correct a decoded value before application validation.
An explicit `--config` selects that file, including an explicitly empty value
(which fails rather than falling back). With no selection, ConfigKit searches
these native presets in order, selecting one main file:
When `--config` is omitted, Scriptorium searches
`/usr/local/etc/scriptorium/config.yml` and then `/etc/scriptorium/config.yml`.
ConfigKit performs this ordered search for each load. If neither exists, it
uses built-in defaults; an error in a selected candidate does not cause a
fallback. An explicit `--config` path must exist and decode successfully,
including when an empty path is explicitly supplied.
| Preset | Directories in order | Filename groups |
| --- | --- | --- |
| WorkingDirectory | Current working directory | Named only |
| UserConfigDirectory | Go user-config directory / `scriptorium`, then the user-config directory | Named then Generic in the application directory; Named only in the base |
| HomeDirectory | Go home directory / `.scriptorium`, then the home directory | Named then Generic in the application directory; Named only in the base |
| UnixSystemDirectories | `/usr/local/etc/scriptorium`, `/etc/scriptorium`, then `/etc` | Named then Generic in application directories; Named only in `/etc` |
The maintained [minimal configuration](../examples/config.yml),
[complete configuration](../examples/config.full.yml),
[composed configuration](../examples/config.composed.yml),
[collection configuration](../examples/config.collections.yml), and
[framework inheritance configuration](../examples/config.inheritance.yml) are
copyable examples.
Named means `scriptorium.yml`, then `scriptorium.yaml`. Generic means
`config.yml`, `config.yaml`, `main.yml`, then `main.yaml`.
ConfigKit uses native Go directory resolution (including its user-config/home
environment behavior) and checks every filename in one directory before the next.
Candidates are not configuration layers. Optional exhaustion uses application
defaults; a selected-file acquisition/decoding error is terminal.
## Application Configuration File
ConfigKit assembles the main, enabled imports, and selected overlay, applies its
inheritance/composition rules, creates typed defaults and strictly decodes, applies
explicit override records, then runs application normalization and validation.
An override can correct a decoded value before validation, but cannot repair
source/composition/decoding errors. The completed result is authoritative:
runtime settings conversion does not merge, default, or normalize again.
Application configuration is decoded by ConfigKit v1.2.1 as strict YAML.
Unknown fields, empty files, multiple YAML documents, and YAML aliases/anchors
are rejected.
An explicit empty string is a value and may clear a file-provided optional
string. A scalar `null` is not an empty string and is rejected for non-nullable
fields; nullable fields such as `transient_retries` retain their documented
meaning. Raw API-key fields and unknown application fields are rejected.
## Application Settings
Only explicitly supplied application CLI flags override decoded settings. Paths
in the application document remain relative to the process working directory, not to the file's
directory. Load diagnostics are printed safely on stderr; invalid application
configuration exits with status 1.
### Promptkit source selectors
`promptkit_sources` is an ordered sequence of explicit Promptkit source
selectors. Each item must set exactly one of `file` or `directory`:
Complete copyable configurations are maintained under
[examples](../examples/config.yml), including [full settings](../examples/config.full.yml),
[application imports](../examples/config.composed.yml),
[framework fragment composition](../examples/config.collections.yml),
[collection checkout discovery](../examples/config.discovery.yml), and
[framework inheritance](../examples/config.inheritance.yml).
```yaml
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/promptkit.yml
schema_dir: ./examples/schemas
```
An empty or null final resolved `promptkit_sources` list selects no
application-supplied framework sources; PromptKit's bundled OpenRouter underlay
still applies. A later sequence item overlays an earlier
one. Files within a selected directory are composed together, so conflicting
definitions in that group fail instead of using filename order as precedence.
An explicit file may use ConfigKit imports. The complete framework format and
source rules belong to the [PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
The `promptkit` section contains acquisition settings, not embedded framework
definitions. Profiles, prompts, backends, and collections belong in the separate
framework document. The retired `promptkit_sources`, top-level `schema_dir`,
and `defaults.render_format` fields have no compatibility conversion and are
rejected by strict decoding, as are unknown/raw credential fields.
Selector paths are relative to the process working directory, not to the
application configuration file. Application validation checks the selector
shape but does not open or validate Promptkit sources. `config validate`
therefore does not prove that selected sources can build an engine; use
`inspect configuration validate` for that separate check. Repeated
`--promptkit-source` flags replace the entire configured selector sequence
and preserve flag order. Without an authored collection policy, a selected
overlay replaces `promptkit_sources` as a whole; a declared sequence policy
may instead combine the lists.
| Field | Default | Meaning |
| --- | --- | --- |
| `promptkit.configuration` | empty | One external framework file; omission/empty selects no external main. |
| `promptkit.schema_dir` | `.` | Schema base directory passed to PromptKit. |
| `transient_retries` | upstream default | Optional nonnegative engine retry count. Omitted/null delegates upstream; zero disables additional transient attempts. |
| `server.addr` | `:8080` | HTTP listen address. |
| `server.artifact_root` | empty | Root enabling HTTP file-artifact references. |
| `server.max_request_bytes` | `16777216` | Encoded HTTP request limit; zero is unlimited. |
| `server.max_artifact_bytes` | `16777216` | HTTP file-artifact limit; zero is unlimited. |
| `server.max_response_bytes` | `16777216` | Encoded HTTP response limit; zero is unlimited. |
### Collection checkout discovery
Size fields are nonnegative signed 64-bit values. [HTTP behavior](api.md) defines
their enforcement; [operations](operations.md) covers safe deployment choices.
Select a framework discovery file using the existing file selector, as in
[config.discovery.yml](../examples/config.discovery.yml). Its native imports
select both supported collection-entry spellings:
ConfigKit strictly decodes application YAML: empty files, unknown fields,
multiple documents, aliases/anchors, and incompatible/null values fail under its
schema rules. Explicit empty strings and numeric zeros remain values. The
nullable retry field preserves null. Application paths are trimmed/cleaned and
the server address is trimmed within ConfigKit's resolver, with native
normalization evidence. Paths remain relative to process cwd, not the application
file's directory. No general environment-variable override layer is installed.
## Explicit Imports And Selected Overlays
Application imports use native root `config.imports` and are enabled during
loading. ConfigKit owns paths/directories/globs, requiredness, destination/item
placement, inheritance, sequence policy, and source-root confinement; consult its
[pinned contract][configkit]. Scriptorium does not widen the selected file's
canonical source root or implement another traversal/merge policy.
`--overlay NAME` passes the name unchanged to ConfigKit. At the selected release,
empty and exact lowercase `default` select none; other names use letters,
digits, underscores, or hyphens and select `overlays/NAME.yml` beneath the main
source root. A missing selected overlay fails. It is not a filesystem-path flag,
inherited stack, or fallback-root mechanism. An active overlay after absent-main
search exhaustion has no independent filesystem root in this application.
Only `serve` has named application override flags, for its HTTP settings.
Native application-scope `--set` records are passed directly to ConfigKit.
Nonconflicting records are supported; duplicate and ancestor/descendant paths
are rejected rather than resolved by order. Configuration commands have no
application-defined source/schema/HTTP selectors or comparison counterparts.
Native comparison selections and supported overrides are independent on each side.
## Framework Source Mapping
A nonempty resolved `promptkit.configuration` becomes an import-enabled
`configkit.File` main input to PromptKit. An empty value leaves its main absent.
The native bundled OpenRouter underlay remains enabled; no application source
list, directory expansion, or local precedence interpreter is installed.
Imports in that external document compose framework definitions using PromptKit's
schema and ConfigKit's native rules. The [maintained entry file](../examples/promptkit.yml)
imports the application's maintained prompt/profile fragments and backend
definitions. Peer conflicts are errors, not filename-order precedence.
Native whole-record replacement and inheritance remain framework-owned.
Application settings do not embed or re-decode framework definitions.
PromptKit captures message-file resources with their defining source context
when constructing the engine. Schema/artifact reads, credential resolution, and
provider work retain native preparation/runtime lifecycles. The server keeps one
engine until restarted; later CLI invocations perform fresh loads.
### Collection Checkout Discovery
The [discovery configuration](../examples/config.discovery.yml) selects
[one framework entry](../examples/discovery/framework.yml) with two optional imports:
```yaml
config:
@@ -90,195 +129,30 @@ config:
- glob: collections/*/collection.yaml
```
Both globs are optional, so either spelling may have no matches. Paths are
relative to the framework file; this example's `*` selects one checkout
level and is not a general layout restriction. Each entry declares its own
collection namespace and imports only its intended resource fragments. The
checkout directory name does not determine the namespace. New checkouts are
available on the next invocation without editing this file; a running server
does not reload them. See the [maintained collection examples](../examples/discovery/collections/).
Each independently maintained checkout declares its namespace and explicitly
imports its own definitions relative to its entry file. Checkout names do not
determine resource IDs. The [support example](../examples/discovery/collections/support-repo/collection.yml)
imports multiple profiles/prompts and a source-relative message; the
[review example](../examples/discovery/collections/review-repo/collection.yaml)
uses the other spelling. Arbitrary checkout YAML is not loaded implicitly.
Entries imported by one discovery file are composed together as peers.
Conflicting definitions of the same qualified resource ID fail to load;
filename, checkout, and glob order do not set precedence. To override a
discovered record, select a separate framework file later in
`promptkit_sources` and supply the complete replacement record.
Either glob may have no matches. New checkouts become visible on a later
invocation, not through server hot reload. Entries in one discovery document are
peers: conflicting qualified IDs fail instead of relying on glob order.
### Explicit imports and selected overlays
## Inspection And Credentials
Application files declare imports under root `config.imports`. Native `path`,
`dir`, and `glob` selectors, filters, recursion, destinations, and item
placement are available as described in the [pinned ConfigKit contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md).
Imports are optional by default; set `required: true` when absence must fail.
The old `optional` field is not supported. Directory selection has no default
filename filter; author a pattern when only YAML files are intended. ConfigKit
composes peer sources strictly, rather than using filename order to resolve
conflicts. Imports are resolved relative to their declaring file within the
native source root. Scriptorium does not widen that root.
The native `app` scope exposes the actual application result without acquiring
framework definitions or credentials. The `promptkit` scope exposes the captured
framework result selected by the application config/overlay. They are not merged
into one synthetic document. See [CLI inspection](cli.md#application-configuration-inspection)
for operations and independent comparison.
Root `config.sequences` can declare an identity key and merge mode for a
collection. Scriptorium does not register or infer an identity for
`promptkit_sources`: absent an authored policy, unkeyed lists replace across
precedence boundaries. The [collection example](../examples/config.collections.yml)
declares `file` identity with `upsert`, combines imported file selectors, and
lets an overlay add another. CLI list overrides still replace the whole list.
PromptKit owns profiles, namespaces, provider controls, repairs, retries, and
credential resolution. Store credential environment-variable names, not raw
values; supply the selected environment to the process. Native framework
redaction is limited to classified fields and history, not arbitrary sensitive
content. Protect application, prompt, prepared, and generated reports as data.
At application precedence boundaries, omission preserves lower-precedence
selectors. Under ordinary list replacement, `[]` clears the selection and
`null` resolves to no selectors. Under authored `append` or `upsert`, `[]`
contributes no records and `null` is invalid. To clear a collection with an
inherited policy, declare `mode: replace` at
`destination: [promptkit_sources]` in the higher-precedence source and supply
`promptkit_sources: []`. See
[ConfigKit's collection policy contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#yaml-declarations-and-policy-lifetime)
for policy lifetime and combination rules.
These application root `config` controls do not change the separate
PromptKit v0.23.3 framework format: its selected framework files use the
[Promptkit contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
and their own native ConfigKit `config` controls.
`--config-overlay NAME` selects one file at `overlays/NAME.yml` below that
root. The name must use letters, digits, `_`, or `-`; it is not a path. Empty
and exact lowercase `default` select no overlay. There is no fallback,
case folding, inherited stack, or automatic selection; a missing selected
overlay fails. Mappings merge; sequence contributions and nulls at collection
destinations follow the effective collection policy described above. Other
explicit nulls replace lower values subject to the application schema.
Defaults-only loading can enable imports, but an absent main has no import
declarations. This CLI supplies no independent overlay root, so an active
overlay cannot be located without a selected main context. That is not a general
ConfigKit restriction.
The [composed example](../examples/config.composed.yml) imports the application
[source-selection fragment](../examples/config-fragments/sources.yml) and has
a selectable [local overlay](../examples/overlays/local.yml). That fragment
selects the [framework backend file](../examples/framework/backends.yml) through
`promptkit_sources`; backend definitions are not imported into the application
document. From the repository root:
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.composed.yml --config-overlay local \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
Inspect the collection overlay and render without contacting a provider:
```bash
go run ./cmd/scriptorium config show --config ./examples/config.collections.yml --config-overlay collection
go run ./cmd/scriptorium render --config ./examples/config.collections.yml --config-overlay collection \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
Import and overlay file locations are constrained by ConfigKit; application
path values such as `promptkit_sources`, `schema_dir`, and `server.artifact_root`
remain relative to the process working directory even when authored in a
fragment or overlay. These source constraints are separate from Promptkit
message-resource paths and HTTP artifact containment.
| Field | Default | Meaning |
| --- | --- | --- |
| `promptkit_sources` | empty sequence | Ordered file/directory sources for Promptkit backends, profiles, and prompts. |
| `schema_dir` | `.` | Promptkit schema source directory for relative schema paths. |
| `server.addr` | `:8080` | Address used by `serve`. |
| `server.artifact_root` | unset | Root that enables HTTP `file` input references. |
| `server.max_request_bytes` | `16777216` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
| `transient_retries` | Promptkit default | Optional nonnegative count of additional built-in transient generation attempts. Omitted or `null` delegates to Promptkit; `0` disables additional attempts. |
The size fields must be zero or greater. The [HTTP API](api.md) defines how
each limit is enforced and reported. `server.artifact_root` configures an HTTP
deployment boundary; see [operations](operations.md) for deployment handling.
The `config validate`, `config show`, `config sources`, and `config explain`
commands inspect this application configuration without creating a Promptkit
engine. `show` is a resolved inspection snapshot, not necessarily a reloadable
file. Application validation does not check framework sources, backend
connectivity, or credential availability. Use `inspect configuration` for
Promptkit's separately captured framework configuration; its four operations
are documented in the [CLI reference](cli.md#framework-configuration-inspection).
`config explain --path /server/addr` uses JSON Pointer syntax to show the
selected field's current value and history from defaults, files, overlays,
explicit CLI overrides, and normalization. `--path ''` selects the root.
History includes replaced and removed values; `show` presents the final
resolved snapshot. The application schema rejects raw credential fields, but
do not treat inspection as a general-purpose filter for sensitive strings.
## Framework Source Mapping
Scriptorium assembles selected Promptkit source files and directories in the
order described above, then constructs the Promptkit engine. Framework
definitions and custom backends belong in those selected source files, not in
the Scriptorium application configuration. Message `content_file` paths use
the defining source's resource context and are captured when the engine is
constructed; schema and request-artifact loading remain preparation-time
operations. Scriptorium does not redefine or independently parse framework
file formats.
- Prompt selection by ID, message templates, inputs, output contracts, and
session IDs are Promptkit contracts.
- Profile fields, numeric ranges, execution defaults, overlay precedence,
built-in profiles, and credential rules are Promptkit contracts.
- Schema path behavior and generated-content validation are Promptkit
contracts.
See the
[tagged Promptkit format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
for all of those definitions. The files under
[`examples/prompts`](../examples/prompts/),
[`examples/profiles`](../examples/profiles/), and
[`examples/schemas`](../examples/schemas/) are maintained Scriptorium
application inputs using that tagged format.
PromptKit supplies a default-on OpenRouter bundle as the lowest-priority
underlay. Choose an actual bundled profile such as `auto`, or a complete
application-owned profile; old catalog IDs are not aliases. Caller definitions
can replace bundled records by normal composition. Bundle availability is not
a readiness check or a promise of a fixed provider model.
## Credentials And Outbound Behavior
Scriptorium maps `--api-key-env` and HTTP `model.api_key_env` into Promptkit
request overrides. Keep secret values in environment variables and store only
their names in configuration or framework source files. Do not place raw keys
in configuration, prompts, profiles, CLI arguments, examples, or HTTP
payloads.
An optional `api_key_env` whose environment value is absent or empty can result
in an unauthenticated provider request. A profile that declares credentials
required still fails as an invalid request when no credential source is
selected, and fails with `ErrAPIKeyEnvMissing` when its selected environment
source is absent or empty. Scriptorium never reads or emits the environment
value itself.
Unset optional provider controls are omitted from compatible provider requests.
A positive Promptkit `repair_attempts` budget can add provider calls, latency,
token use, and cost; see the tagged format reference for its permitted values
and validation-mode requirements.
Promptkit's built-in client makes up to two additional attempts for eligible
transient HTTP statuses by default. The application `transient_retries` setting
changes that budget for its constructed engine; an explicit `run` request
override takes precedence. This can increase latency, request count, and cost.
Each initial or repair generation has its own transient budget; an injected
client retains its own retry behavior. The tagged integration contract owns
eligibility and timing.
Promptkit's
[OpenAI-compatible integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
defines outbound authentication, provider request mapping, transport limits,
and timeout layering.
## Related References
- [CLI reference](cli.md)
- [HTTP API reference](api.md)
- [Operations guide](operations.md)
- [Promptkit framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md
[configkit]: https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md

View File

@@ -11,6 +11,7 @@ and the HTTP service according to the boundary your application needs.
- CLI subprocess: [subprocess integration](../integrations/subprocess.md)
- HTTP service: [HTTP API reference](../api.md)
- Application configuration: [configuration reference](../config.md)
- Existing CLI callers: [native CLI migration](upgrading-cli.md)
Go applications that need an in-process prompt framework should import
`go.maximumdirect.net/promptkit` directly. The tagged
@@ -27,7 +28,7 @@ Consumers are responsible for:
- supplying required inputs and template variables;
- supplying credentials through the chosen interface;
- protecting rendered prompts and generated artifacts as potentially
sensitive;
sensitive, including native inspection and full result reports;
- deciding whether validation-failed output is usable; and
- retrying only when another model call is acceptable.

View File

@@ -0,0 +1,95 @@
# Migrate To The Native CLI
This guide covers Scriptorium's breaking CommandKit-native CLI cutover. It does
not select a release version or change historical release notes. The
[CLI reference](../cli.md) owns current syntax, streams, and statuses;
[configuration](../config.md) owns source discovery and settings.
The earlier [PromptKit upgrade](upgrading-promptkit.md) covers framework formats,
not a compatibility layer for this interface.
## Replace Commands And Selections
| Former interface | Current interface |
| --- | --- |
| `run --prompt ID` / `--prompt-id` | `run ID`; exactly one positional prompt ID. |
| `render --prompt ID` | `promptkit show prepared ID`. |
| `inspect prompt --prompt ID` | `promptkit show prompt ID`. |
| `inspect profile --profile ID` | `promptkit show profile ID`. |
| `inspect configuration ...` | Native `config ... --scope promptkit`. |
| Application `config ...` | Native `config ... --scope app` (the default scope), including diff. |
| `--config-overlay` | `--overlay NAME`; a native overlay name, not an arbitrary path. |
| `--input NAME=PATH` | `--input-file NAME=PATH`; `--input` now means literal text. |
| `--llm-base-url` | `--endpoint` (canonical `--llm-endpoint`). |
| Duration-valued `--timeout` | Integer `--timeout-seconds` for each generation attempt. |
| `--out` / output-file flags | Caller stdout redirection. |
| `--execution-report` | `--report PATH` with native result rendering. |
Prefer native short aliases such as `--profile`, `--var`, `--input-text`, and
`--input-file`; canonical `--llm-*` spellings remain available.
There is no duplicate prompt flag, direct backend-ID override, or retained
`--profile-id` alias on generation.
Repeat mappings rather than comma-separating them. The first equals sign splits
each mapping; commas and later equals signs are literal. Input text/file
occurrences share native last-occurrence precedence.
## Migrate Application Configuration
Replace the source-selector list and top-level schema directory with
`promptkit.configuration` and `promptkit.schema_dir`.
The [maintained example](../../examples/config.yml) selects a separate framework
entry file that imports prompt/profile/backend definitions through native
ConfigKit composition. No source/schema CLI flags or compatibility conversion
remain. Do not embed framework definitions inside the application section.
Discovery now uses native cwd, user-config, home, then Unix-system presets and
their filename conventions. It is no longer limited to the two former
`config.yml` paths. Set an explicit config path when migrating unattended jobs.
Overlays remain native name selections. Application-scope native `--set` records
retain conflict rejection; order does not make duplicate paths authoritative.
There are no custom source/schema/HTTP selectors under `config`.
## Capture Artifacts And Reports
For an artifact file plus full native report, from the repository root:
```sh
scriptorium run generic.markdown_summary --config examples/config.yml \
--input-file transcript=examples/fixtures/transcript.md \
--input-file glossary=examples/fixtures/glossary.yml \
--report run-result.json --report-format json > summary.md
```
This generation example contacts the configured provider. Use native prepared
preview instead when only checking inputs/settings.
Main automatic/text output is the artifact alone, with no added newline.
Main JSON/YAML is the native result; artifact bytes use native structured
serialization. Separate reports contain that full native result, not the old
observer-derived metadata shape. `--report-format` independently chooses
text/YAML/JSON; omission leaves the format unset for native automatic rendering
(currently YAML for structured values). Extensions do not select formats.
Omit quiet to capture main output, including redirected stdout. Quiet still
allows an explicitly requested report file. Reports are available only when a
result is returned; resultless failures do not replace an existing report.
Partial delivery can leave one or both destinations written. Keep paths separate
and check status/diagnostics before treating files as fresh.
## Adjust Status And Sensitivity Handling
Completed generated-content validation failure is now domain status 4, not the
old status 2. Native usage/invalid-configuration input is 2, diff-check
differences are 3, and operational failures are 1. Presentation/file failures
can change success into failure while preserving an already nonzero domain
classification. The [CLI status table](../cli.md#server-and-statuses) owns details.
Native prompt inspection includes messages; prepared reports include rendered
content; full run reports include generated content and metadata. Framework
classified-field redaction does not make arbitrary report data safe to expose.
Requirements discovery is separate from validation/preparation: it can run
before artifacts exist, and a partial report may succeed. Read its native
completeness/issue fields. Smoke tests are explicit provider operations, not
startup checks. See [subprocess guidance](../integrations/subprocess.md) for
stream capture, cancellation, and caller responsibilities.

View File

@@ -1,33 +1,23 @@
# Upgrade PromptKit Sources And Definitions
This guide covers the CLI and framework configuration cutover to
This guide covers the earlier framework configuration cutover to
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md)
and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
for the full upstream contract.
for the full upstream contract. The subsequent [native CLI migration](upgrading-cli.md)
owns command, reporting, application-discovery, and source-setting changes.
Removing version selectors and changing framework definition shapes are
intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility).
## Application configuration
Existing application fields that already select the desired sources retain
their meaning. For example, this application configuration works both before
and after the cutover:
```yaml
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
```
The selected framework files must still be rewritten as described below.
Scriptorium uses one native application schema: discovery, imports, selected
overlays, source order, explicit overrides, and working-directory-relative
selector paths remain application configuration behavior. It loads framework
sources separately with PromptKit's schema. No second application schema or
legacy definition reader is provided. See the [configuration reference](../config.md).
Framework definition migration and application acquisition are separate changes.
Rewrite older framework files as below, then select their entry file using the
current [application configuration](../config.md#application-settings).
The old source-selector list and top-level schema root are not current aliases;
follow the [CLI migration guide](upgrading-cli.md#migrate-application-configuration).
Scriptorium does not provide a legacy definition reader or embedded-definition
extraction/reload path.
## Rewrite framework definitions
@@ -82,9 +72,9 @@ Convert framework source controls from `configkit:` to `config:` and add
header, if present, is `version: 2`. Use sibling inheritance such as
`extends: {id: base}`.
Across ordered `promptkit_sources` entries, a later framework underlay replaces
the whole record for the same ID. Supply the complete intended replacement,
or create a distinct inherited record. Checkouts imported by the same
The single external framework entry uses native ConfigKit import composition;
there is no application-managed source-list precedence. Native record replacement
and inheritance remain upstream-owned. Checkouts imported by the same
[discovery file](../config.md#collection-checkout-discovery) are peers:
conflicting definitions of the same ID fail instead of overriding one another.
@@ -106,5 +96,5 @@ records can replace it through ordinary composition.
For multiple independently maintained repositories, see the
[collection-discovery example](../../examples/config.discovery.yml). Its
entry files declare their own namespaces; checkout names do not become IDs.
The [CLI reference](../cli.md) demonstrates qualified rendering and the
The [CLI reference](../cli.md) defines qualified selection and prepared preview; the
[operations guide](../operations.md) describes when updates take effect.

View File

@@ -5,6 +5,8 @@ application that consumes the public
[PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md);
framework implementation work belongs in Promptkit. Application configuration
uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local.
Operator interfaces use CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0;
standard command families and presentation remain upstream-owned.
PromptKit uses the same ConfigKit release to load framework sources under its
own separate schema and format contract.
@@ -46,8 +48,8 @@ Before recommending, planning, or implementing work:
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) |
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests |
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` |
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#promptkit-source-selectors), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and the [native CLI contract](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md) |
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#framework-source-mapping), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
| Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) |
| OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) |
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |

View File

@@ -1,64 +1,67 @@
# Subprocess Integration
This document covers process-boundary behavior for callers that invoke
Scriptorium as a child process. Command syntax, flags, output, and exit codes
are defined by the [CLI reference](../cli.md). Interface selection belongs in
the [consumer integration overview](../consumers/api.md).
Use the executable when a local process boundary is appropriate.
The [CLI contract](../cli.md) owns exact command/flag/status syntax; the
[consumer overview](../consumers/api.md) compares subprocess and HTTP use.
## Process Contract
## Artifact-First Execution And Inspection
Use `scriptorium render` when the caller needs prepared output without a model
call, and `scriptorium run` for generation. Pass an explicit `--config` or
make the configuration search paths available to the child process; configuration
discovery, fields, profile selection, and credential mechanisms are defined in
the [configuration reference](../config.md).
Select a stored prompt and capture its generated artifact from stdout. Use native
requirements discovery before generated input files exist, concrete validation
once bindings exist, and prepared preview to render without a model call.
The [CLI migration guide](../consumers/upgrading-cli.md) explains replacements
for older command and file-output conventions.
Pass required API-key environment variables through the child environment. Do
not place raw API keys in arguments. Keep the environment limited to the values
needed for the selected profile.
Pass an explicit application configuration or deliberately arrange the child's
[configuration discovery](../config.md). Supply only the environment needed for
selected credentials; never put raw keys in arguments.
For application-only preflight, invoke `scriptorium config validate`; it exits
0 with empty stdout when the application configuration loads, but does not
prove that Promptkit sources or credentials are usable. `config show` emits
resolved YAML and `config sources` emits newline-terminated JSON provenance.
All three use the same source selection and explicit overrides as execution
commands and require separate stderr capture for diagnostics. Their output can
include source paths, identifiers, credential reference names, and ordinary
backend parameters; treat it as potentially sensitive.
Discovery completeness is distinct from binding satisfaction and execution
readiness. Partial reports can succeed. Consumers own their binding graph and
policy for incomplete findings; Scriptorium supplies no workflow state.
## Streams And Output Ownership
## Streams, Reports, And Availability
Capture stdout and stderr separately. Stdout contains the requested artifact or
prepared output unless the caller selects an output file; stderr contains
summaries, diagnostics, and server messages. The exact destinations and status
meanings are part of the [CLI reference](../cli.md), not a stable stderr data
protocol.
Capture stdout and stderr separately. Diagnostics are not a machine-readable
error protocol, even when structured result output is selected. Prefer exit
classification and native report fields; do not parse diagnostic prose.
When using `--out`, the caller owns the output path, its permissions, and
cleanup. Treat rendered prompts, generated artifacts, stdout, and stderr as
potentially sensitive.
The default generated result is artifact content. Native structured results and
optional separate reports carry the full returned result, including content and
metadata. They are not the former metadata-only CLI observation report. Native
prompt/prepared inspection also carries content. Treat all captured streams and
reports as potentially sensitive.
For `run`, an optional [execution report file](../cli.md) contains terminal
metadata separate from generated stdout or `--out`. The caller owns that
path's permissions and retention. Output and report files are not a
transaction: one may be written when the other fails, and a preexisting report
may remain when the invocation has no terminal event. Check the exit status
and diagnostics rather than assuming a report path contains this run's data.
To save main output, supply a file as the subprocess stdout destination or use
shell redirection. Omit quiet when that main result is needed: quiet suppresses
it even for redirected stdout. A separately requested report remains an
application file effect independent of quiet and main format selection.
A completed content-validation failure can still deliver an artifact/result;
consumers decide whether it is usable. An operational failure with no result
produces diagnostics and a nonzero status, not a synthesized result/error
envelope. An existing separate report may then describe an older invocation.
## Partial Delivery And Caller Responsibility
Main output is attempted before a requested report, and report delivery is
attempted even after main presentation failure. Either destination may contain
partial data after an error. File delivery is not transactional and has no
rollback or atomic-replacement promise.
Callers own redirected file creation/truncation, permissions, retention, and
keeping destinations separate. A shell may truncate its file before the process
starts, unlike a separate report that is opened only after a result exists.
Check the finalized exit status and diagnostic stream; do not infer success or
freshness from a file's existence.
## Cancellation And Recovery
A CLI invocation performs one synchronous request and creates no durable run
state. A supervising process that needs cancellation must terminate the child
process according to its own process-management policy. A later invocation is a
new request and can make another model call; there is no resume or checkpoint
protocol.
Interrupt/SIGTERM cancels the child's invocation context. Engine cancellation is
cooperative. The [whole-operation budget](../cli.md#generation) is separate from
per-attempt provider timeouts and excludes process/configuration setup and
delivery. Supervisors still own forced process termination when required.
For `run`, callers may set the [whole-run deadline](../cli.md) to bound the
engine invocation. It is separate from the per-attempt generation timeout and
does not include child startup or output writing. A typed operation deadline
failure exits 1.
Supervisors still own process termination when they need to stop the child
outside that engine interval.
For deployment, filesystem permissions, and sensitive-artifact handling, see
the [operations guide](../operations.md).
A new invocation is a new request that may incur another provider call/cost.
There is no checkpoint, archive, conversation, or resume protocol. Deployment and
sensitive-data handling belong to [operations](../operations.md).

View File

@@ -1,157 +1,89 @@
# Adapter Internals
## Purpose
Adapters map executable inputs and effects through public upstream values.
The [CLI](../cli.md), [HTTP API](../api.md), and [architecture policy](../policy/architecture.md)
own their external contracts and boundaries.
Scriptorium adapters translate executable inputs into Promptkit public requests
and translate Promptkit results or errors back to CLI or HTTP behavior. They
own IO and presentation mechanics, not framework decisions.
## CLI Composition And Acquisition
External contracts are canonical in the [CLI reference](../cli.md) and
[HTTP API reference](../api.md). Promptkit's public engine contract is
described by its tagged
[Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md).
`app.go` composes application run/serve declarations with native configuration,
PromptKit, completion, help, and version support. CommandKit owns parsing,
declaration-input checks, reporting selection, presentation, and finalization.
## Components And Collaborators
`selection.go` maps explicit native config/overlay choices and server-only HTTP
bindings into `internal/config.Load`. It preserves supplied empty/zero values.
Native application `--set` records remain upstream records with native conflict
handling. No application selectors are installed under configuration commands.
- `cmd/scriptorium` passes process arguments and streams to
`internal/adapter/cli`.
- `internal/adapter/cli` resolves settings through `internal/config`,
constructs `promptkit.Engine`, maps CLI values to `promptkit.RunRequest`,
exposes application and captured framework configuration inspection, and
owns output files, summaries, and exit codes.
- `internal/config` delegates application YAML decoding, default resolution,
overrides, and validation ordering to ConfigKit v1.2.1. The CLI supplies only
explicitly visited application flags as overrides and renders safe structured
load diagnostics on stderr. PromptKit uses the same ConfigKit release to
load framework sources under its own definition and format contract. Both
domains use ConfigKit's native diagnostic writer.
- `internal/adapter/http` strictly decodes request DTOs, maps them to Promptkit
public values, calls its adapter-owned `Runner` interface, and maps results
and errors to HTTP DTOs.
- `internal/format` renders `promptkit.PreparedRun` values as deterministic text
or JSON.
- `internal/report` maps Promptkit's terminal execution event to a small
application-owned value without invoking the engine or writing output.
`native_config.go` adapts the actual application result through
`ckconfig.FromConfigKit` and framework inspection through `ckprompt.ConfigScope`.
Native views own effective values, paths, metadata, sources, findings, explanations,
redaction, and comparison. Application inspection does not construct an engine.
Comparison loads each side from that side's selection; there is no config/engine
cache or effective-document reconstruction.
## Wiring Flows
`acquisitionState` contains application diagnostic evidence only, never loaded
configuration or engines. Callback decorators forward it through native
`Acquired` channels. An application LoadError is transported separately while
the framework adapter runs, then its original typed cause is restored before
native outcome classification. This prevents app findings from becoming
framework findings. Native framework evidence remains unchanged; comparison
application messages identify left/right context. Callback marks prevent replay
when operations share invocation state.
### CLI
`native_prompt.go` installs all twelve ckprompt callbacks, nil request builders,
and empty-default native namespace views. Inventories, named resources,
requirements, validation, preparation, and smoke testing retain their upstream
lifecycles and reports.
`run` calls `promptkit.Engine.Run`; `render` calls
`promptkit.Engine.Prepare`. Both share request mapping for exact prompt ID
and profile selection, optional file inputs, variables, and presence-aware
execution overrides. Omitted framework settings remain zero values so Promptkit
resolves its own defaults and definition-required inputs.
The application retry setting enters shared engine construction, while a
run/render flag enters only the mapped request. The server uses the same engine
setting and retains one backend admission state. Render prepares without
generation.
## Generation And Delivery
`inspect prompt` maps the selected ID to
`promptkit.Engine.InspectPrompt`, then formats Scriptorium-owned inspection
data. `inspect profile` maps an explicit profile ID to `Engine.InspectProfile`
and formats Scriptorium-owned resolved profile data without backend completion.
`native_run.go` binds the required operand and reusable resource/input/execution
flags into a native RunRequest. Ordinary typed namespace flags feed
`Engine.WithNamespaces`; stored references are not rewritten. One `Engine.Run`
call receives the invocation context and unchanged operation duration. PromptKit
owns preparation, credentials, admission, retries, repairs, and content validation.
Both paths construct the same full-source engine as execution before inspecting
the chosen definition; selected prompts are not filtered out. They do not
perform request preparation or generation. The [CLI definition-inspection
contract](../cli.md#scriptorium-inspect-prompt) owns the loading and runtime
boundaries, including the distinction between an optional prompt collection
and validating any prompts that were selected.
`outcomes.go` maps typed failures and native configuration evidence to native
outcomes, preserving causes while keeping provider payloads out of terminal
explanations. A returned completed validation failure remains a domain outcome;
an unexpected nil successful result is an integration failure.
`inspect configuration` builds the same engine from the selected native
Promptkit source sequence, reads its captured configuration view, and delegates
effective output and explanation to Promptkit's native writers. The adapter
projects captured source records into its small JSON DTO. Its validation
operation constructs the engine without preparing a request. Framework source
semantics and report redaction are owned by Promptkit; the [CLI
contract](../cli.md#framework-configuration-inspection) documents the exposed
operations and their deliberately limited redaction scope.
`delivery.go` submits the actual RunResult to the invocation reporter with a
reusable artifact-byte text template. The optional file renders that same value
without the template through CommandKit's writer API, outside reporter callbacks
to avoid reentrancy. File failures add explanations/causes while reporter
presentation failures remain native. See [delivery semantics](../cli.md#results-and-separate-reports).
The `config` family loads the same application ConfigKit result once and uses
its validation, effective-output, source, and explanation views directly. It
does not construct a Promptkit engine. The CLI maps source records into a small
JSON DTO for `sources`; `show` and `explain` use ConfigKit's own writers.
## Server And HTTP
`serve` constructs Scriptorium's restricted HTTP artifact reader, injects it
with `promptkit.WithArtifactReader`, passes the engine through the HTTP
adapter's consumer-owned `Runner` interface, and starts the server.
`server.go` loads application settings, injects the restricted artifact reader
into one engine, installs the existing HTTP handler/limits, and opens a listener.
It retains the header timeout and engine retry defaults. Context cancellation
calls Server.Close; callback coordination finishes connection/listener closing
before return, without waiting for arbitrary context-ignoring application work.
The command paths assemble the engine from application-selected framework
source files and directories plus the schema directory and retry setting.
Individual files become import-enabled ConfigKit inputs; directory selections
are expanded through Promptkit's directory-input helper. Scriptorium supplies
those ordered underlays through Promptkit's public configuration API rather
than parsing or mapping framework definitions itself. The server constructs
one engine, which retains the framework's backend registry and associated
capacity state across requests.
The HTTP handler strictly decodes DTOs and applies transport policy before
mapping to PromptKit requests. Its Runner interface is a consumer-side external
boundary using native values. Request controls do not mutate shared engine
defaults; shared backend capacity accounting remains in that engine.
### HTTP
The handler enforces transport limits and strict JSON decoding before mapping
DTOs into `promptkit.RunRequest`, `promptkit.ArtifactRef`, and
`promptkit.ExecutionTargetOverride`. On success it reads Promptkit artifact,
validation, model, usage, and metadata values directly.
The optional HTTP retry count maps to `RunRequest.TransientRetries` without
mutating the shared engine. Negative values are rejected before invocation;
Promptkit controls retries and backend admission for concurrent requests.
CLI and HTTP operation budgets map to PromptKit's `OperationTimeout`; HTTP
passes the caller context through unchanged. The public sentinels determine
timeout classification.
Failure mapping uses `errors.Is` against Promptkit's public sentinels and the
HTTP reader's Scriptorium-owned containment and size errors. Wrapped reader
errors preserve their identity through Promptkit's artifact-load boundary.
The report collector is per invocation. It ignores nonterminal events and
copies terminal aggregates once, including usage presence and completeness.
It retains no history or shared state; adapters decide whether to attach it and
how to present its snapshot.
CLI `run` attaches one only when a report path is selected, writes generated
output first on success, and then writes the terminal snapshot when present.
Its report path remains untouched for pre-invocation failures or absent
terminal observations; report and artifact writes are independent.
HTTP `POST /v1/runs` attaches a separate collector to each invocation only when
`include_execution_report` is true. Its optional top-level `execution` field
appears in success or error envelopes only when Promptkit emits a terminal
event. Pre-invocation adapter errors and absent terminal events leave it out.
The complete serialized success or report-bearing error response is subject to
the configured response-size limit; overflow returns the compact 413 error
without a report. Field meanings are owned by the [API contract](../api.md#execution-report).
## Package-Local Guarantees
- Adapters contain no copied framework types or orchestration.
- Configuration is resolved before Promptkit engine construction.
- Explicit numeric overrides preserve presence, including zero.
- HTTP DTO and error mapping remains stable and transport-owned.
- Resolved secrets are not serialized or printed.
- No adapter creates durable run state.
HTTP optional reports use one per-request `internal/report.Collector`, attached
only for opted-in requests. It copies terminal aggregates, not event history,
and is never the CLI result report. Success/error attachment and encoded-size
handling retain the [HTTP contract](../api.md#execution-report). Reader sentinel
identities survive PromptKit wrapping for status mapping.
## Verification
Inspect:
Native composition/configuration/prompt/run/delivery/lifecycle tests protect the
application's integration choices using real libraries and controlled local
providers. Checkout-discovery tests cover fresh invocations and entry-file
spellings. Entrypoint tests protect identity/status mapping; HTTP reader/DTO/
report/capacity regressions and the architecture guard remain their own owners.
Do not duplicate upstream parsers, analyzers, serializers, or exhaustive format
matrices to change an adapter.
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/handler_test.go`
- `internal/adapter/http/artifact_reader_test.go`
- `internal/format/prepared_run_test.go`
- `internal/adapter/dependency_test.go`
The adapter tests protect parsing, configuration mapping, output, status
mapping, restricted artifacts, and representative real Promptkit-engine
workflows. The dependency test protects the repository boundary.
## Change Recipes
For a CLI or HTTP change:
1. identify the Scriptorium-owned external contract;
2. map through Promptkit public values without copying framework semantics;
3. add or update the narrow application-owned test;
4. update the canonical Scriptorium contract; and
5. coordinate and tag Promptkit first if a required public capability is
genuinely absent.
Update [source internals](sources.md) when application source locations or HTTP
artifact containment changes.
When changing behavior, update its canonical external contract and focused
integration protection. Required shared capabilities belong upstream before
tagged adoption, not in a resurrected application framework facade.

View File

@@ -1,19 +1,18 @@
# Internal Component Overview
This is the complete inventory of Scriptorium's implemented Go components.
The [architecture policy](../policy/architecture.md) owns normative boundaries;
This inventory describes implemented Go components. The
[architecture policy](../policy/architecture.md) owns normative boundaries;
public behavior belongs in the linked contracts.
| Component | Implemented responsibility | References |
| --- | --- | --- |
| `cmd/scriptorium` | Process entrypoint that delegates arguments and streams to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
| `internal/adapter/cli` | Parses execution, application-configuration, and captured framework-configuration inspection commands; resolves settings; constructs Promptkit engines for execution, definition inspection, and framework inspection; maps requests; and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md), [source internals](sources.md) |
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) |
| `internal/config` | Selects application sources through ConfigKit v1.2.1, supplies the application definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md), [source internals](sources.md) |
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) |
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output; framework-configuration output uses Promptkit's native captured view and writers. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
| `internal/report` | Projects one terminal Promptkit observation into Scriptorium-owned execution metadata. | [adapter internals](adapters.md) |
| `cmd/scriptorium` | Build identity, signal-aware context, native status-to-process exit. | [CLI](../cli.md), [adapters](adapters.md) |
| `internal/adapter/cli` | CommandKit composition, ckconfig/ckprompt acquisition, application loading, native request mapping, run/report effects, and HTTP process lifecycle. | [CLI](../cli.md), [adapters](adapters.md), [sources](sources.md) |
| `internal/adapter/http` | Routes, strict DTO decoding, transport limits, public request/result/error mapping, and restricted artifact reader. | [HTTP API](../api.md), [adapters](adapters.md), [sources](sources.md) |
| `internal/config` | ConfigKit application schema, discovery/selection, defaults/resolution/validation callbacks, and completed-value conversion. | [Configuration](../config.md), [sources](sources.md) |
| `internal/defaults` | Application and HTTP default constants. | [Configuration](../config.md) |
| `internal/report` | HTTP terminal-observation metadata projection. It is not used for CLI native result reports. | [HTTP execution report](../api.md#execution-report), [adapters](adapters.md) |
Framework implementation packages are provided by
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
and are not part of this repository.
CommandKit, ckconfig, ckprompt, ConfigKit, and PromptKit are released upstream
dependencies, not local framework packages. There is no custom CLI formatter
package or compatibility dispatch surface.

View File

@@ -1,117 +1,55 @@
# Source Internals
## Purpose
This document covers application acquisition and the restricted HTTP artifact
reader. [Configuration](../config.md) owns field/discovery/composition contracts;
the [PromptKit formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
own framework resources and execution semantics.
This document covers Scriptorium-owned source locations and the restricted HTTP
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
owned by the tagged
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
## Application Boundary And Engine Construction
## Application Source Locations
`internal/config.Load` selects an explicit File or the ordered native SearchConfig
presets and performs one ConfigKit load with imports, overlay selection, and
explicit overrides. It returns the actual Result with native inspection evidence.
Definition callbacks supply application defaults, record actual path/address
normalizations through ConfigKit, and validate retry/HTTP-size domains. Settings
conversion consumes completed values without another merge or normalization pass.
`internal/config` resolves the ordered `promptkit_sources` sequence and
`schema_dir` from application configuration and explicit CLI overrides. It
owns selector-shape validation, not framework file parsing. During engine
construction, `internal/adapter/cli` turns file selections into import-enabled
ConfigKit inputs and expands directory selections with
`promptkit.DirectoryInputs`, then passes the ordered groups as Promptkit
configuration underlays, one per selector. Directory inputs retain the
helper's strict-composition grouping; no selected source becomes an implicit
framework main. The shared Promptkit engine receives the selected schema
directory and retry setting, and the default OpenRouter bundle remains enabled.
The typed application PromptKit section selects acquisition only. In
`engine.go`, a nonempty resolved configuration path becomes an import-enabled
main Input; an empty path leaves Main nil. The schema base and retry default are
passed directly, retaining the bundled underlay. There is no source-list parser,
directory classification/expansion, or one-underlay-per-selector algorithm.
Promptkit uses ConfigKit to load and compose those framework inputs according
to Promptkit's own schema and format. Scriptorium does not parse or validate
Promptkit definitions. Promptkit owns definition selection and validation,
profile built-ins and overlays, schema resolution, ordinary file artifacts,
captured framework configuration inspection, and related error identities.
Framework imports resolve within the native source context. Message resources
are captured from their defining contexts at engine construction. Schema and
request-artifact acquisition retain their preparation-time behavior.
Independent collection checkouts are discovered by native file import globs,
not application traversal. New invocations reload normally; a server retains
its single engine until restart.
The [configuration reference](../config.md) owns Scriptorium's source-location
fields and precedence. Maintained files under `examples/` are application
inputs that use Promptkit's tagged formats.
## Application Configuration Boundary
`internal/config` contains the application-owned ConfigKit-native `Document`
definition. Its fresh defaults come from Scriptorium's existing default
constants; its resolver records only actual leaf-level trimming, path cleaning, and
render-format normalization. The validator reports selector-shape, retry,
size-limit, and format errors without mutating the document or exposing
rejected values. Application path values keep working-directory-relative
meaning. ConfigKit v1.2.1 owns strict application YAML decoding and presence
semantics. Framework configuration is separately loaded by PromptKit using
the same ConfigKit release under PromptKit's own schema and source contract.
The `Load` boundary selects an explicit file or constructs a native ConfigKit
search over the ordered implicit paths. Discovery exhaustion uses defaults;
a selected-file error stops loading. One ConfigKit load enables imports,
applies explicitly supplied application overrides after decoding and before
resolution and validation, and returns the document and its frozen inspection
records. `Settings` copies selectors in their resolved order into runtime
settings; it does not read files or apply another merge. ConfigKit uses the selected file's
canonical directory as the source root; the application does not widen it.
Application path values retain process-working-directory-relative meaning.
Runtime conversion consumes the resolved application values through the
existing `Settings` path. Promptkit receives selected framework sources only
after application configuration has loaded and validated; its source merge,
definition validation, and captured inspection view remain framework-owned.
Every configuration-loading command uses this boundary. The CLI maps only
explicitly visited application flags to ConfigKit overrides and prints
structured load diagnostics before engine construction. Application and
framework load errors and warnings use their respective native writers. The old application
YAML decoder and merge path have been removed.
Application `config` inspection reuses the result's frozen effective,
provenance, and history views without constructing an engine or reloading
sources. It does not resolve credentials. `inspect configuration` separately
constructs an engine and exposes Promptkit's captured framework view; only its
framework `show` and `explain` reports receive Promptkit's native,
credential-shaped redaction. Neither inspection family is a general-purpose
filter for arbitrary sensitive values.
Application-only inspection uses its completed result without framework
acquisition. Framework inspection exposes the native captured view separately.
Diagnostic decoration in the [CLI adapter](adapters.md) carries application
messages without combining evidence trees or effective documents. There is no
per-invocation engine/config cache or cross-invocation snapshot guarantee.
## Restricted HTTP Artifact Reader
`internal/adapter/http` implements `promptkit.ArtifactReader` for HTTP
requests. The `serve` path injects it with
`promptkit.WithArtifactReader`, replacing Promptkit's ordinary reader for
inbound HTTP inputs.
The HTTP adapter implements `promptkit.ArtifactReader`; server construction
injects it through the public option. Ordinary CLI artifact loading remains
upstream-owned.
The reader:
The reader accepts inline content without a root. File references need a root;
relative paths resolve below it and absolute paths must be lexically contained.
Traversal outside is rejected and the configured file byte limit applies, with
zero unlimited. Reads preserve native artifact metadata and honor cancellation.
- accepts inline references without an artifact root;
- denies file references when no root is configured;
- resolves relative paths below the configured root;
- accepts absolute paths only when they are lexically within that root;
- rejects lexical traversal outside the root;
- applies the configured file byte limit, with zero meaning unlimited;
- preserves content type, body, size, hash, name, and URI metadata; and
- honors context cancellation.
Containment does not resolve symlinks. The OS may follow a symlink inside the
root to an outside target; this is not a filesystem sandbox or a defense against
privileged concurrent filesystem modification. Reader errors retain identity
through upstream artifact-load wrapping, enabling the existing HTTP mapping.
Containment is lexical and does not resolve symlinks. The operating system
follows symlinks after the check. The [HTTP API](../api.md) owns observable
request outcomes, and [operations](../operations.md) owns safe deployment
permissions and root selection.
Reader errors remain identifiable after Promptkit wraps them as artifact-load
failures, allowing the HTTP adapter to preserve Scriptorium status and error
codes.
## Verification And Change Recipe
Inspect:
- `internal/config/document_test.go`
- `internal/config/load_test.go`
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/artifact_reader_test.go`
- `internal/adapter/http/handler_test.go`
When changing an application source location or HTTP artifact policy:
1. preserve strict configuration precedence and the Promptkit public boundary;
2. keep containment and size policy in Scriptorium;
3. update focused configuration, reader, and handler tests;
4. update the [configuration](../config.md), [HTTP](../api.md), and
[operations](../operations.md) contracts as applicable; and
5. do not duplicate Promptkit loaders, formats, or ordinary artifact behavior.
The [HTTP API](../api.md) owns outcomes and [operations](../operations.md) owns
safe root/permission selection. Configuration, CLI acquisition/engine, and HTTP
reader/handler tests own their respective boundaries; upstream owns exhaustive
source traversal, format parsing, ordinary artifacts, and inspection semantics.

View File

@@ -1,243 +1,120 @@
# Operations Guide
## Scope And References
This runbook covers deployment, normal operation, capacity, sensitive data, and
recovery. Exact [CLI syntax/statuses](cli.md), [configuration](config.md), and
[HTTP wire behavior](api.md) remain in their canonical references.
See the [CLI migration guide](consumers/upgrading-cli.md) for changed jobs and
the [framework upgrade guide](consumers/upgrading-promptkit.md) for definitions.
This runbook covers deployment, normal operation, capacity planning, and safe
recovery for Scriptorium. It does not redefine invocation syntax, configuration
fields, or HTTP wire behavior.
## Deploy Sources And Process
- [CLI reference](cli.md): commands, output destinations, and exit codes.
- [Configuration reference](config.md): application settings, source
locations, defaults, and credential mapping.
- [Promptkit source upgrade guide](consumers/upgrading-promptkit.md): concise
source-selector and definition migration examples.
- [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md):
prompt, profile, schema, execution-setting, and framework credential
contracts.
- [HTTP API reference](api.md): route, request/response schema, status codes,
limits, and HTTP artifact access.
- [Consumer integration overview](consumers/api.md): caller responsibilities.
Provide readable application configuration, its selected framework entry file,
and explicitly imported definitions/resources. Set cwd deliberately: application
path values are cwd-relative, while framework imports and message resources keep
their native source contexts. ConfigKit confines source imports/overlays to the
selected main's canonical root; Scriptorium does not widen it.
## Operational Model And State
Engine construction validates selected effective definitions and captures message
resources/template syntax, including definitions not used by a particular request.
Schema/artifact reads, credential resolution, and model work retain their native
preparation/runtime lifecycles. A server constructs one engine at startup and
does not hot-reload changed files; restart to capture updates. Separate CLI
invocations load afresh rather than sharing an immutable snapshot.
Scriptorium handles one prompt request for each CLI invocation or HTTP request.
It has no durable run store, archive, checkpoint, cache, or resume mechanism.
A failed or interrupted request is recovered by correcting its inputs,
configuration, or environment and submitting a new request.
For independent collections, install checkouts beneath the layout selected by
the [discovery entry](../examples/discovery/framework.yml). Each checkout declares
its namespace and explicit fragment imports; arbitrary YAML is not a framework
source. The maintained [discovery example](../examples/config.discovery.yml)
supports both entry-file spellings and multiple definitions. New checkouts or
updates become visible to later invocations or a restarted server.
Generated artifacts, rendered prompts, model output, and run metadata are
caller-owned data. Retention, encryption, backup, and deletion are deployment
responsibilities.
Optional [CLI execution reports](cli.md) also belong to the caller. They can
contain sensitive identifiers even though they omit prompt and response bodies.
Keep report paths separate from generated output and apply the same access and
retention controls. Output and report writes are not transactional; an older
report may remain when no terminal observation was produced.
HTTP callers may also request a [terminal execution report](api.md#execution-report).
It shares the same identifier sensitivity and should receive response access
and retention controls. An opted-in report increases the encoded response size;
an oversized success or report-bearing error becomes the compact 413 response
without a report.
Supply only needed credential environment variables through a supervisor or
secret-management mechanism. Confirm presence without printing their values.
Raw keys do not belong in arguments or checked-in configuration.
## Deploy The Filesystem And Process
## Normal Operator Workflow
Provide the process with readable selected Promptkit source files/directories
and schema sources that follow the tagged framework formats. At engine
construction, PromptKit v0.23.3 validates every selected effective definition,
including unused definitions, and captures referenced message-file contents
and static template syntax. `serve` constructs one engine at startup; it does
not reload changed configuration or message content. Restart the process to
validate and capture a new configuration. Schema and request-artifact reads,
credential resolution, and provider work remain request-preparation or runtime
concerns. For an HTTP deployment that accepts file artifacts, use a dedicated,
narrow artifact directory rather than a general-purpose or sensitive filesystem
tree.
1. Validate application configuration through the native application scope.
This does not acquire framework sources or prove provider readiness.
2. Inspect framework configuration/resources and discover requirements for the
selected prompt/profile/namespaces. Discovery can precede generated input
artifacts; completeness does not establish satisfied bindings.
3. Once artifacts exist, validate the concrete request or inspect prepared
content. These are provider-free preparation checks with native prerequisites.
4. Invoke a smoke test only when its provider effects are intended. Findings
can be unsupported or unsuccessful while the command itself succeeds.
Smoke tests are not an automatic startup or release prerequisite.
5. Generate with the same selections and bindings, then inspect the finalized
status and any returned validation result before accepting the output.
If application configuration uses explicit imports or a selected overlay,
deploy those files beneath the selected configuration file's canonical
directory. ConfigKit keeps imported and overlay files within that root;
Scriptorium does not widen it. Application path values inside those files still
resolve from the process working directory, so set that directory deliberately
in the service manager. This configuration-source boundary is distinct from
the HTTP artifact root and does not constrain prompt or schema directories.
Use the [maintained preview script](../examples/render-markdown-summary.sh)
for a provider-free example, and the [CLI contract](cli.md) for commands.
A complete discovery report can still list required inputs that are not bound;
a partial report can still be useful. Consumers own their policy for incomplete
findings and changed resources between invocations.
For independently maintained collections, place checkouts beneath the
directory selected by your framework discovery file. The maintained
[discovery example](../examples/config.discovery.yml) selects optional
`collection.yml` and `collection.yaml` entry files; each checkout owns its
namespace and explicit fragment imports. Update a checkout independently and
invoke the CLI again, or restart a running server to capture the update. Do
not treat arbitrary YAML in a checkout as a framework source. When upgrading
definitions, review exact profile IDs and replace retired catalog references
as described in the [migration guide](consumers/upgrading-promptkit.md).
## Sensitive Data And Retention
Run Scriptorium under an identity that can:
Artifacts, prompt messages, prepared content, native run reports, metadata,
diagnostics, and redirected streams are caller-owned potentially sensitive data.
The separate CLI report contains the full returned result, not metadata alone.
Control access, retention, encryption, backup, and deletion accordingly.
Framework effective/explanation views apply native classified-field redaction,
including history, but do not detect arbitrary secrets. Other content-bearing
reports are not automatically made safe for publication.
- read only the prompt, profile, schema, and allowed input-artifact paths it
needs;
- read the required credential environment variables without writing them to
files or logs; and
- write only caller-selected output locations when CLI output files are used.
Keep main output and report destinations separate. Delivery is nontransactional:
one destination or partial bytes can remain after failure. An existing report
can remain on a resultless error; check status and report identity, not just file
existence. A shell can truncate a redirected destination before the child starts.
Omit quiet if main stdout must be captured; the separate report effect remains
independent. Exact delivery behavior belongs to [CLI results](cli.md#results-and-separate-reports).
Do not make the HTTP artifact directory writable by untrusted users. The HTTP
artifact containment behavior is lexical and the operating system follows
symlinks; account for that when choosing ownership and mount boundaries. See
the [HTTP API reference](api.md) for the externally observable behavior.
HTTP's optional [execution report](api.md#execution-report) remains a distinct
terminal-observation projection. Protect its identifiers and any response data.
It increases encoded response size and remains subject to the configured limit.
## Supply Credentials And Protect Runtime Data
## HTTP Deployment And Shutdown
Set secret values in the process environment and configure only their
environment-variable names. Do not put raw keys in configuration, prompt or
profile files, process arguments, HTTP payloads, captured command lines, or
debug dumps.
The service has no built-in authentication/authorization. Deploy on a trusted
network or behind appropriate authenticated proxy/gateway controls. Use a narrow,
dedicated artifact root with controlled ownership and permissions when accepting
file inputs; this is lexical containment, not a symlink sandbox.
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP
responses as potentially sensitive. Send service logs to a controlled collector
and apply the same retention and access rules as for model input and output.
Use a supervisor for cwd, environment, restart policy, and stream capture.
Startup information uses native Info reporting, not stdout banners.
Interrupt/SIGTERM cancels the invocation and closes the listener and active
connections. Canceled shutdown is nonzero operational termination. There is no
graceful-drain deadline or promise that context-ignoring work has finished;
supervisors own forced termination when necessary.
## Run A Normal Workflow
## Capacity, Limits, And Recovery
Before changing production inputs, profiles, or schemas:
One server engine shares backend admission/queue state across concurrent requests.
That state is transient operational coordination, not a durable workflow or
queue of resumable runs. HTTP capacity exhaustion retains its
[API outcome](api.md); callers decide whether another model call is safe.
1. use `config validate`, `config show`, or `config sources` to check
Scriptorium's application configuration; these do not open Promptkit sources
or construct an engine;
2. use `inspect configuration validate` to construct the engine and validate
all selected framework definitions; this does not prepare a request, check
credentials, load request-time schemas/artifacts, or contact a provider;
3. confirm the deployed configuration selects the intended sources and
environment-variable names;
4. use [`render`](cli.md) with the same request inputs and variables to confirm
preparation without a model call;
5. use [`run`](cli.md) for generation; and
6. retain or discard validation-failed output according to the caller's
policy.
Plan for input/output sizes, schema complexity, provider latency, retries, and
repair costs. Native transient attempts and repair generations can add calls and
waiting; engine defaults and per-request overrides belong to
[configuration](config.md) and the [CLI](cli.md)/[HTTP](api.md) contracts.
Whole-operation budgets are distinct from per-attempt generation timeouts.
They remain cooperative and do not include configuration/setup or delivery.
Application configuration inspection can include paths and configured values.
Framework `inspect configuration show` and `explain` use Promptkit's native
redaction for classified credential-shaped provider fields, including
historical values, while leaving environment-reference names and ordinary
provider parameters visible. Neither report identifies arbitrary sensitive
content by value or name. Protect inspection output accordingly; Scriptorium
stores environment-variable names, not the resolved credential bytes.
Before increasing HTTP byte limits, measure representative requests/artifacts/
responses, confirm memory/network/provider capacity, retain authentication and
upstream request controls, and test a representative workload. Do not widen the
artifact root just to make an arbitrary host path accessible.
The [maintained render script](../examples/render-markdown-summary.sh) is a
copyable preflight example. The CLI reference owns its complete invocation and
exit semantics.
For failures, capture diagnostics without publishing sensitive data. Verify
selected configuration, resource identity, readable inputs, profile/credentials,
and schema deployment. Use native prepared preview to isolate pre-generation
problems. A completed content-validation failure is distinct from an operational
failure to validate; consumers choose whether failed content is usable.
Correct the cause, then issue a new request only when another provider call and
its cost are acceptable.
## Expose The HTTP Service
The HTTP service has no built-in authentication or authorization. Place it on a
trusted network or behind an authenticated reverse proxy, API gateway, or
equivalent access control. Restrict who can reach it and who can read the
artifact root.
Use a service manager or supervisor appropriate to the deployment to manage
process lifetime, restart policy, log capture, and environment injection. The
[HTTP API reference](api.md) owns client request shapes, status behavior, and
artifact-access outcomes.
## Plan Capacity And Limits
Capacity is primarily determined by concurrent model calls, input and output
sizes, schema complexity, provider latency, and network behavior. Size limits
protect request bodies, HTTP file artifacts, and encoded responses; configure
them through the [configuration reference](config.md) and rely on the
[HTTP API reference](api.md) for their response effects.
Configured backend concurrency and queue capacity are enforced per constructed
Promptkit engine. A `serve` process constructs one engine for its handler, so
concurrent HTTP requests share that transient admission state. Scriptorium does
not retain workflow state: capacity is neither durable nor a queue of resumable
runs. When admission is exhausted, HTTP returns `503 capacity_exceeded` without
retry timing; callers choose any retry policy that is safe for another model
call.
Promptkit's built-in client now allows two additional attempts for eligible
transient provider statuses by default. Account for added calls, latency, and
cost when planning capacity. Each repair generation has its own transient
budget, separate from the structural repair count. Configure the engine budget
with [`transient_retries`](config.md); `run` and `render` accept a request
override described in the [CLI contract](cli.md), and HTTP callers can select
one through the [API contract](api.md). Omission inherits the engine
setting and then Promptkit's default, while explicit zero disables additional
attempts. Existing generation timeouts apply
per attempt, while the caller context covers the complete operation. Provider
`Retry-After` timing can exceed Promptkit's ordinary backoff cap; consult the
[outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
for details.
CLI `run` can set a [whole-run deadline](cli.md) when total execution time must
be bounded. The existing generation timeout applies afresh to each attempt and
does not cap retry waits or the complete invocation. The whole-run interval
covers the engine call, including capacity waiting and repairs as supported by
context cancellation; it excludes parsing, engine setup, and output writes.
HTTP callers can use [`run_timeout_seconds`](api.md) to set the engine's
operation budget for the same interval. The API distinguishes a typed expired
execution deadline from caller cancellation and from a provider-attempt
timeout. A canceled capacity waiter releases its Promptkit admission state.
Before increasing a limit:
1. measure representative input, generated-output, and optional raw-output
sizes;
2. confirm memory, network, and upstream-provider capacity;
3. retain an upstream request-size and authentication boundary; and
4. test the intended workload in a non-production environment.
For large local inputs, prefer a controlled file-artifact directory over
placing arbitrary paths on the service host. Avoid disabling a limit unless an
equivalent trusted control exists elsewhere.
## Diagnose And Recover
### Preparation Or Configuration Failure
Capture the CLI diagnostic or HTTP error response, then verify the selected
configuration, prompt ID, profile selection, source readability, and input
mapping. Use `render` with the same request when it is unclear whether failure
occurs before model execution. Consult the [CLI reference](cli.md), the
[configuration reference](config.md), and the [HTTP API reference](api.md) for
the exact interface contract.
### Credential Or Provider Failure
Confirm that the process environment contains the configured credential name
without printing the secret. Check endpoint reachability and provider health
from the process network. If preparation succeeds but generation fails, inspect
the selected model settings in prepared output and the service's controlled
logs. Correct the deployment or provider issue, then submit a new request.
### Artifact Or Permission Failure
Verify that the process can read the intended local input. For HTTP file
artifacts, verify the deployment's artifact root, ownership, path layout, and
file size. Do not widen filesystem permissions or the allowed root merely to
make an arbitrary path work; move or copy the required artifact into the
controlled location instead.
### Validation Failure
A generated-content validation failure is distinct from a runtime failure.
CLI `run` reports the validation result and error count in its success summary;
it does not print the individual validation messages. For HTTP, inspect the
validation object in the response according to the [HTTP API reference](api.md).
Use rendered input and generated output to determine whether prompt instructions,
the selected model, or the schema needs correction. If schema loading or
compilation itself fails, correct the source deployment or schema document
before rerunning.
### HTTP Limit Or Request Failure
Compare the request, artifact, or expected response size with the deployed
configuration, and validate the request against the [HTTP API reference](api.md).
Reduce the payload, use an appropriate controlled artifact source, omit
unneeded raw output, or adjust the deployment limit after capacity review.
## Cleanup And Reruns
Because no run state is retained, cleanup concerns caller-owned output files,
logs, and artifacts only. Remove or rotate them using the deployment's normal
retention policy. After a correction, rerun the request from the beginning;
there is no safe resume point.
There is no durable run store, archive, checkpoint, or resume mechanism.
Cleanup concerns caller-owned files/logs only. Rotate/delete them under the
deployment's normal retention policy; recovery starts from a new request.

View File

@@ -39,8 +39,8 @@ Do not promote incidental implementation choices into architectural invariants.
## Scriptorium's Purpose And System Shape
Scriptorium is an executable application with CLI `run`, `render`,
`inspect`, and `config` command families and an HTTP service started by
Scriptorium is an executable application with CLI `run`, native `promptkit`
and `config` families, and an HTTP service started by
`serve`. It offers simple, request-independent execution of predefined
prompts, not a reusable Scriptorium root Go API or a workflow framework.
The [internal overview](../internal/overview.md) owns the concrete package
@@ -71,9 +71,12 @@ externally observable application behavior; PromptKit owns framework formats.
As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to
neutrally expose upstream features and semantics. The current CLI still uses
custom presentation; the [adoption roadmap](../roadmap/commandkit-adoption.md)
tracks its replacement. This decision complements ADRs 0004 and 0005.
neutrally expose upstream features and semantics. CommandKit owns shared command,
parsing, presentation, and outcome mechanics; application commands and effects
remain local. Standard configuration inspection uses native adapters without
custom effective documents or evidence reconciliation. This decision complements
ADRs 0004 and 0005; the [adoption roadmap](../roadmap/commandkit-adoption.md)
records the implementation work.
The CLI exposes Promptkit's captured framework configuration inspection. Its
`show` and `explain` reports use Promptkit's native redaction for classified

View File

@@ -2,7 +2,7 @@
**Base Policy Version:** 1.3
**Last Updated**: 2026-11-17
**Last Updated**: 2026-11-19
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md
@@ -178,12 +178,15 @@ Give each layer its distinct responsibility and keep expectation updates localiz
diagnostics. ConfigKit owns exhaustive YAML grammar, source traversal,
import and overlay composition, provenance, and inspection renderers.
- CLI tests own argument mapping, streams, output, exit behavior, and
representative run, render, inspect, serve, and config workflows.
representative native run, PromptKit resource/requirements/preview/smoke,
serve, and configuration comparison workflows.
- HTTP tests own DTO decoding, transport limits, status and error mapping,
restricted artifact reading, and representative server workflows.
- Formatting tests own prepared-run and definition-inspection text and JSON
presentation. Architecture guards own dependency direction and removed
framework-package boundaries.
- CommandKit and its integrations own standard parsing and presentation.
Scriptorium's integration tests protect native adoption, byte-preserving main
output, separate report effects, outcomes, and acquisition boundaries without
reproducing the upstream formatting matrix. Architecture guards own dependency
direction and removed framework-package boundaries.
- Promptkit owns framework parsing, orchestration, validation, profiles, and
model-client behavior. Scriptorium's definition and execution fixtures
protect adoption through its public adapter boundary without duplicating

View File

@@ -643,6 +643,25 @@ version output. Test-local fallback deadlines prevent regressions from hanging.
## Stage 8: Retire Superseded Machinery And Update Canonical Documentation
**Status: Complete.** Removed legacy parsing/dispatch, custom inspection/source
DTOs and writers, retired CLI tests, and the unused formatter package. Retained
architecture boundaries, native integration protection, checkout-discovery
coverage, and the unchanged HTTP/report implementation and regression suites.
Maintained examples now select external framework entries with native imports;
both collection-entry spellings and multi-definition discovery remain present.
Canonical CLI/configuration/process/operations/internal/development/policy owners
and navigation reflect the cutover, with a focused CLI migration guide distinct
from the earlier framework-format guide. Accepted ADRs and release-note history
are unchanged; deferred HTTP redesign and dynamic completion remain deferred.
Verified on Go 1.27.4: full tests and race suite, vet, ordinary build, module
tidiness, formatting/diff checks, all maintained application configs in both
native scopes, application overlays, maintained preview scripts, a structured
schema preview, and local Markdown links/anchors (42 files). Added maintained
example loading/preparation protection and adapted fresh-checkout discovery to
native preview. Entrypoint package tests now run and pass. Linked identity,
release-target builds, and the final assembled acceptance audit remain Stage 9.
**Primary files:** old CLI parser/dispatch/inspection files and tests,
`internal/format`, documentation and maintained examples listed below.

View File

@@ -1,8 +1,9 @@
# Prompt input requirements for Narratio
Status: accepted direction, pending implementation through the CommandKit
migration. This proposal records Narratio's consumer needs and their mapping
to the native interface; it does not describe shipped Scriptorium behavior.
Status: the native interface and focused consumer scenarios are implemented.
The [implementation plan](implementation.md) tracks final assembled verification.
This proposal records Narratio's consumer needs and their mapping to the native
interface; the [CLI contract](../cli.md) owns current behavior.
The [CommandKit notepad](commandkit-notepad.md) records the migration decisions.
## Consumer need
@@ -27,8 +28,8 @@ Default and explicit text output use the native YAML presentation.
The [upstream CLI contract][requirements-cli] owns syntax and report details.
Use the same application configuration and framework source selection as
execution, including `--config`, `--overlay`, and applicable application
selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
execution, including `--config` and `--overlay`. There are no application-defined
source/schema selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
prompt/profile namespace flags supply the relevant discovery context.
Preserve upstream selection semantics, including the prompt's default profile
when an explicit profile is omitted. Qualified IDs identify installed resources;

View File

@@ -0,0 +1,6 @@
config:
imports:
- path: collections.yml
required: true
- path: profiles/local-gpu.yml
required: true

5
examples/collections.yml Normal file
View File

@@ -0,0 +1,5 @@
config:
imports:
- dir: config-fragments/collections
pattern: '*.yml'
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources:
- file: ./examples/framework/backends.yml
config:
imports:
- path: ../../framework/backends.yml
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources:
- file: ./examples/profiles/local-fast.yaml
config:
imports:
- path: ../../profiles/local-fast.yaml
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources:
- file: ./examples/prompts/generic.markdown_summary.yaml
config:
imports:
- path: ../../prompts/generic.markdown_summary.yaml
required: true

View File

@@ -1,5 +1,3 @@
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/promptkit.yml
schema_dir: ./examples/schemas

View File

@@ -1,10 +1,3 @@
config:
sequences:
- destination: [promptkit_sources]
key: file
mode: upsert
imports:
- dir: config-fragments/collections
pattern: '*.yml'
required: true
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/collections.yml
schema_dir: ./examples/schemas

View File

@@ -1,2 +1,2 @@
promptkit_sources:
- file: ./examples/discovery/framework.yml
promptkit:
configuration: ./examples/discovery/framework.yml

View File

@@ -1,8 +1,6 @@
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/promptkit.yml
schema_dir: ./examples/schemas
server:
addr: 127.0.0.1:8080
@@ -10,6 +8,3 @@ server:
max_request_bytes: 16777216
max_artifact_bytes: 16777216
max_response_bytes: 16777216
defaults:
render_format: text

View File

@@ -1,6 +1,4 @@
# Application paths are relative to the repository invocation directory.
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/inheritance.yml
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/inheritance.yml
schema_dir: ./examples/schemas

View File

@@ -1,12 +1,7 @@
promptkit_sources:
- directory: ./examples/prompts
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
promptkit:
configuration: ./examples/promptkit.yml
schema_dir: ./examples/schemas
server:
addr: :8080
artifact_root: .
defaults:
render_format: text

8
examples/definitions.yml Normal file
View File

@@ -0,0 +1,8 @@
config:
imports:
- dir: prompts
pattern: '*.yaml'
required: true
- dir: profiles
pattern: '*.y*ml'
required: true

6
examples/inheritance.yml Normal file
View File

@@ -0,0 +1,6 @@
config:
imports:
- path: definitions.yml
required: true
- path: framework/inheritance.yml
required: true

View File

@@ -1,2 +1,2 @@
promptkit_sources:
- file: ./examples/profiles/local-gpu.yml
promptkit:
configuration: ./examples/collection-gpu.yml

View File

@@ -1,4 +1,2 @@
defaults:
render_format: json
server:
addr: 127.0.0.1:8080

6
examples/promptkit.yml Normal file
View File

@@ -0,0 +1,6 @@
config:
imports:
- path: definitions.yml
required: true
- path: framework/backends.yml
required: true

View File

@@ -5,9 +5,8 @@ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"
go run ./cmd/scriptorium render \
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--input-file transcript=./examples/fixtures/transcript.md \
--input-file glossary=./examples/fixtures/glossary.yml \
--format text

View File

@@ -4,21 +4,18 @@ set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"
go run ./cmd/scriptorium render \
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.full.yml \
--prompt generic.markdown_summary \
--profile local-gpu \
--session-id example-session \
--reasoning-effort= \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
--input-file transcript=./examples/fixtures/transcript.md \
--input-file glossary=./examples/fixtures/glossary.yml
go run ./cmd/scriptorium inspect prompt \
go run ./cmd/scriptorium promptkit show prompt generic.markdown_summary \
--config ./examples/config.full.yml \
--prompt generic.markdown_summary \
--format json
go run ./cmd/scriptorium inspect profile \
go run ./cmd/scriptorium promptkit show profile local-gpu \
--config ./examples/config.full.yml \
--profile local-gpu \
--format json

2
go.mod
View File

@@ -8,6 +8,7 @@ require (
go.maximumdirect.net/commandkit v0.7.0
go.maximumdirect.net/configkit v1.2.1
go.maximumdirect.net/promptkit v0.23.3
go.yaml.in/yaml/v4 v4.0.0
)
require (
@@ -15,6 +16,5 @@ require (
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect
github.com/spf13/cobra v1.10.2 // indirect
github.com/spf13/pflag v1.1.0 // indirect
go.yaml.in/yaml/v4 v4.0.0 // indirect
golang.org/x/text v0.44.0 // indirect
)

View File

@@ -1,39 +0,0 @@
package cli
import (
"errors"
"fmt"
"io"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"go.maximumdirect.net/configkit"
)
func loadApplicationSettings(configPath, overlay string, explicit bool, overrides []configkit.Override) (appconfig.AppSettings, []configkit.Diagnostic, error) {
result, err := appconfig.Load(appconfig.LoadOptions{
Path: configPath, Overlay: overlay, Explicit: explicit, Overrides: overrides,
})
if err != nil {
return appconfig.AppSettings{}, nil, err
}
return appconfig.Settings(result.Config), result.Diagnostics(), nil
}
func writeApplicationError(output io.Writer, label string, err error) {
var loadError *configkit.LoadError
if errors.As(err, &loadError) {
_, _ = fmt.Fprintf(output, "%s error:\n", label)
if writeErr := configkit.WriteDiagnostics(output, loadError.Diagnostics()); writeErr != nil {
_, _ = fmt.Fprintf(output, "%s diagnostic write error: %v\n", label, writeErr)
}
return
}
_, _ = fmt.Fprintf(output, "%s error: %v\n", label, err)
}
func writeApplicationWarnings(output io.Writer, diagnostics []configkit.Diagnostic) error {
if len(diagnostics) == 0 {
return nil
}
return configkit.WriteDiagnostics(output, diagnostics)
}

View File

@@ -1,101 +0,0 @@
package cli
import (
"encoding/json"
"flag"
"fmt"
"io"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"go.maximumdirect.net/configkit"
)
type configurationSource struct {
ID string `json:"id"`
Label string `json:"label"`
Layer configkit.Layer `json:"layer"`
Path string `json:"path,omitempty"`
}
func configCommand(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "config parse error: configuration mode is required")
return ExitRuntimeError
}
mode := args[0]
if mode != "validate" && mode != "show" && mode != "sources" && mode != "explain" {
fmt.Fprintln(stderr, "config parse error: unknown configuration mode")
return ExitRuntimeError
}
fs := flag.NewFlagSet("config "+mode, flag.ContinueOnError)
fs.SetOutput(io.Discard)
var path, overlay, pointer string
registerConfigSelectionFlags(fs, &path, &overlay)
registerConfigurationOverrides(fs)
if mode == "explain" {
fs.StringVar(&pointer, "path", "", "configuration path as a JSON Pointer")
}
if err := fs.Parse(args[1:]); err != nil {
writeApplicationError(stderr, "config parse", err)
return ExitRuntimeError
}
if fs.NArg() != 0 {
fmt.Fprintln(stderr, "config parse error: positional arguments are not accepted")
return ExitRuntimeError
}
var explanationPath configkit.Path
if mode == "explain" {
if !flagWasSet(fs, "path") {
fmt.Fprintln(stderr, "config parse error: --path is required")
return ExitRuntimeError
}
var err error
explanationPath, err = configkit.ParsePath(pointer)
if err != nil {
fmt.Fprintln(stderr, "config parse error: --path must be a JSON Pointer")
return ExitRuntimeError
}
}
result, err := appconfig.Load(appconfig.LoadOptions{
Path: path, Explicit: flagWasSet(fs, "config"), Overlay: overlay,
Overrides: applicationOverrides(fs),
})
if err != nil {
writeApplicationError(stderr, "config", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, result.Diagnostics()); err != nil {
return ExitRuntimeError
}
switch mode {
case "validate":
return ExitOK
case "show":
err = result.WriteEffective(stdout)
case "sources":
sources := make([]configurationSource, 0, len(result.Sources()))
for _, source := range result.Sources() {
sources = append(sources, configurationSource{
ID: source.ID, Label: source.Label, Layer: source.Layer, Path: source.Path,
})
}
err = json.NewEncoder(stdout).Encode(sources)
case "explain":
err = result.WriteExplanation(stdout, explanationPath)
}
if err != nil {
fmt.Fprintf(stderr, "config output error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func registerConfigurationOverrides(fs *flag.FlagSet) {
fs.String("addr", "", "HTTP listen address override")
fs.String("artifact-root", "", "HTTP artifact root override")
fs.Int64("max-request-bytes", 0, "HTTP request byte limit override")
fs.Int64("max-artifact-bytes", 0, "HTTP artifact byte limit override")
fs.Int64("max-response-bytes", 0, "HTTP response byte limit override")
}

View File

@@ -1,226 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"io"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
)
func TestConfigCommandsInspectApplicationWithoutEngine(t *testing.T) {
path := writeAppConfigFile(t, "{}\n")
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
if code != ExitOK || stderr != "" {
t.Fatalf("%s failed: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
}
switch mode {
case "validate":
if stdout != "" {
t.Fatalf("validation wrote stdout: %q", stdout)
}
case "show":
if !strings.Contains(stdout, "render_format: text") || !strings.Contains(stdout, "max_request_bytes: 16777216") {
t.Fatalf("effective defaults missing: %q", stdout)
}
case "sources":
var sources []configurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "main" {
t.Fatalf("unexpected source records: %q (%v)", stdout, err)
}
}
}
}
func TestCollectionExampleLoadsAndRenders(t *testing.T) {
t.Chdir(filepath.Join("..", "..", ".."))
want := []appconfig.SourceSelection{
{File: "examples/framework/backends.yml"},
{File: "examples/profiles/local-fast.yaml"},
{File: "examples/prompts/generic.markdown_summary.yaml"},
}
for _, overlay := range []string{"", "collection"} {
result, err := appconfig.Load(appconfig.LoadOptions{Path: "examples/config.collections.yml", Explicit: true, Overlay: overlay})
if err != nil {
t.Fatal(err)
}
expected := append([]appconfig.SourceSelection{}, want...)
if overlay != "" {
expected = append(expected, appconfig.SourceSelection{File: "examples/profiles/local-gpu.yml"})
}
if !reflect.DeepEqual(result.Config.PromptKitSources, expected) {
t.Fatalf("overlay %q selectors = %+v, want %+v", overlay, result.Config.PromptKitSources, expected)
}
args := []string{"--config", "examples/config.collections.yml", "--prompt", "generic.markdown_summary", "--input", "transcript=examples/fixtures/transcript.md", "--input", "glossary=examples/fixtures/glossary.yml"}
if overlay != "" {
args = append(args, "--config-overlay", overlay)
}
code, stdout, stderr := runCLICommand(t, renderCommand, args)
if code != ExitOK || !strings.Contains(stdout, "generic.markdown_summary") || !strings.Contains(stdout, "local-fast") {
t.Fatalf("collection render failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
}
}
func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
t.Fatal(err)
}
base := filepath.Join(dir, "config.yml")
for path, body := range map[string]string{
base: "config:\n imports:\n - path: fragment.yml\n required: true\npromptkit_sources:\n - directory: ' ./before/../prompts '\nschema_dir: ./schemas\n",
filepath.Join(dir, "fragment.yml"): "server:\n addr: :8081\n",
filepath.Join(dir, "overlays", "local.yml"): "promptkit_sources:\n - directory: ./overlay-prompts\nserver:\n addr: :9090\n",
} {
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
args := []string{"--config", base, "--config-overlay", "local", "--schema-dir=", "--addr", ":7070"}
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"show"}, args...))
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "directory: overlay-prompts") || !strings.Contains(stdout, "schema_dir: ''") || !strings.Contains(stdout, "addr: :7070") || strings.Contains(stdout, "before/../prompts") {
t.Fatalf("resolved effective output changed: code=%d out=%q err=%q", code, stdout, stderr)
}
code, stdout, stderr = runCLICommand(t, configCommand, append([]string{"sources"}, args...))
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
t.Fatalf("source command failed: code=%d out=%q err=%q", code, stdout, stderr)
}
var sources []configurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 5 {
t.Fatalf("unexpected composed source records: %+v (%v)", sources, err)
}
for i, layer := range []string{"main", "main", "overlay", "cli", "cli"} {
if string(sources[i].Layer) != layer {
t.Fatalf("source %d layer = %q, want %q", i, sources[i].Layer, layer)
}
}
}
func TestApplicationInspectionDoesNotAcquireFrameworkSources(t *testing.T) {
dir := t.TempDir()
frameworkPath := filepath.Join(dir, "invalid-framework.yml")
if err := os.WriteFile(frameworkPath, []byte("not: [valid\n"), 0o600); err != nil {
t.Fatal(err)
}
for _, frameworkSource := range []struct {
name string
path string
}{
{name: "invalid definition", path: frameworkPath},
{name: "missing path", path: filepath.Join(dir, "missing-framework.yml")},
} {
t.Run(frameworkSource.name, func(t *testing.T) {
path := writeAppConfigFile(t, "promptkit_sources:\n - file: "+frameworkSource.path+"\n")
for _, args := range [][]string{
{"validate"},
{"show"},
{"sources"},
{"explain", "--path", "/promptkit_sources/0/file"},
} {
code, stdout, stderr := runCLICommand(t, configCommand, append(args, "--config", path))
if code != ExitOK || stderr != "" {
t.Fatalf("application inspection acquired configured framework source: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
}
}
})
}
}
func TestConfigCommandsRejectRetiredFrameworkFields(t *testing.T) {
for _, field := range []string{"prompt_dir", "profile_dir", "backends"} {
t.Run(field, func(t *testing.T) {
path := writeAppConfigFile(t, field+": {}\n")
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", path})
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("retired application field %q was accepted: code=%d out=%q err=%q", field, code, stdout, stderr)
}
})
}
}
type errorOutputWriter struct{}
func (errorOutputWriter) Write([]byte) (int, error) { return 0, errors.New("output unavailable") }
func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) {
path := writeAppConfigFile(t, "server:\n max_request_bytes: -1\n")
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "validate.error") {
t.Fatalf("%s misclassified invalid config: code=%d out=%q err=%q", mode, code, stdout, stderr)
}
}
path = writeAppConfigFile(t, "{}\n")
var stderr bytes.Buffer
if code := configCommand([]string{"show", "--config", path}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
t.Fatalf("writer error not reported: code=%d stderr=%q", code, stderr.String())
}
if code := configCommand([]string{"sources", "--config", path}, errorOutputWriter{}, io.Discard); code != ExitRuntimeError {
t.Fatalf("source writer error returned %d", code)
}
}
func TestConfigExplainUsesResolvedHistory(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
t.Fatal(err)
}
base := filepath.Join(dir, "config.yml")
for path, body := range map[string]string{
base: `promptkit_sources:
- directory: " ./prompts/../prompts "
server:
addr: :8080
`,
filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n",
} {
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
args := []string{"--config", base, "--config-overlay", "local", "--addr", ":9090"}
for _, tc := range []struct {
pointer string
want []string
}{
{pointer: "/server/addr", want: []string{":8080", ":9090", "--addr"}},
{pointer: "/promptkit_sources/0/directory", want: []string{"normalized application path", "prompts"}},
{pointer: "", want: []string{"promptkit_sources", "server", "addr"}},
} {
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", tc.pointer}, args...))
if code != ExitOK || stderr != "" {
t.Fatalf("explain %q failed: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
}
for _, want := range tc.want {
if !strings.Contains(stdout, want) {
t.Fatalf("explain %q lacks %q: %q", tc.pointer, want, stdout)
}
}
}
}
func TestConfigExplainRejectsUnknownOrMalformedPaths(t *testing.T) {
path := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"explain", "--config", path},
{"explain", "--config", path, "--path", "not-a-pointer"},
{"explain", "--config", path, "--path", "/missing"},
} {
code, stdout, stderr := runCLICommand(t, configCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid explanation path accepted: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
}
}
var stderr bytes.Buffer
if code := configCommand([]string{"explain", "--config", path, "--path", ""}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
t.Fatalf("explanation writer failure not reported: code=%d err=%q", code, stderr.String())
}
}

View File

@@ -1,7 +1,9 @@
package cli
import (
"encoding/json"
"fmt"
"go.maximumdirect.net/promptkit"
"os"
"path/filepath"
"strings"
@@ -41,37 +43,46 @@ func TestCollectionCheckoutDiscoveryAcrossInvocations(t *testing.T) {
t.Fatal(err)
}
}
config := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkPath))
config := filepath.Join(root, "app.yml")
writeNativeFile(t, config, fmt.Sprintf("promptkit:\n configuration: %q\n", frameworkPath))
input := filepath.Join(root, "transcript.md")
if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil {
t.Fatal(err)
}
args := func(name string) []string {
if name == "support-repo" {
return []string{"--config", config, "--prompt", "support/answer", "--profile", "support/fast", "--input", "transcript=" + input}
return []string{"support/answer", "--config", config, "--profile", "support/fast", "--input-file", "transcript=" + input}
}
return []string{"--config", config, "--prompt", "review/answer", "--profile", "review/base"}
return []string{"review/answer", "--config", config, "--profile", "review/base"}
}
check := func(name string) {
t.Helper()
code, out, stderr := runCLICommand(t, renderCommand, args(name))
if code != ExitOK {
t.Fatalf("render %s: code=%d stderr=%s", name, code, stderr)
command := append([]string{"promptkit", "show", "prepared"}, args(name)...)
outcome, out, stderr := executeNative(t, append(command, "--format=json")...)
if outcome.Status != 0 {
t.Fatalf("prepare %s: outcome=%+v stderr=%s", name, outcome, stderr)
}
var prepared promptkit.PreparedRun
if err := json.Unmarshal([]byte(out), &prepared); err != nil {
t.Fatal(err)
}
if name == "support-repo" {
for _, want := range []string{"prompt: support/answer", "selected_profile_id: support/fast", "model: support-example", "max_tokens: 256", "Answer the support request", "synthetic discovery transcript"} {
for _, want := range []string{"support/answer", "support/fast", "support-example", "Answer the support request", "synthetic discovery transcript"} {
if !strings.Contains(out, want) {
t.Fatalf("missing %q: %s", want, out)
}
}
} else if !strings.Contains(out, "prompt: review/answer") || !strings.Contains(out, "model: review-example") || !strings.Contains(out, "Review this material") {
if prepared.EffectiveModelParams.MaxTokens != 256 {
t.Fatalf("inherited profile setting missing: %+v", prepared)
}
} else if !strings.Contains(out, "review/answer") || !strings.Contains(out, "review-example") || !strings.Contains(out, "Review this material") {
t.Fatalf("unexpected review render: %s", out)
}
}
install(tc.first)
check(tc.first)
code, _, _ := runCLICommand(t, renderCommand, args(tc.second))
if code != ExitRuntimeError {
outcome, _, _ := executeNative(t, append([]string{"promptkit", "show", "prepared"}, args(tc.second)...)...)
if outcome.Status != 1 {
t.Fatal("uninstalled collection unexpectedly selected")
}
install(tc.second)

View File

@@ -0,0 +1,42 @@
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestMaintainedApplicationExamplesUseNativeLoading(t *testing.T) {
root, err := filepath.Abs(filepath.Join("..", "..", ".."))
if err != nil {
t.Fatal(err)
}
t.Chdir(root)
entries, err := os.ReadDir("examples")
if err != nil {
t.Fatal(err)
}
for _, entry := range entries {
if entry.IsDir() || !strings.HasPrefix(entry.Name(), "config") || !strings.HasSuffix(entry.Name(), ".yml") {
continue
}
path := filepath.Join("examples", entry.Name())
for _, scope := range []string{"app", "promptkit"} {
outcome, _, diagnostics := executeNative(t, "config", "validate", "--config="+path, "--scope="+scope)
if outcome.Status != 0 {
t.Fatalf("%s/%s: %+v %q", path, scope, outcome, diagnostics)
}
}
}
for _, test := range []struct{ config, overlay string }{{"examples/config.yml", ""}, {"examples/config.composed.yml", "local"}, {"examples/config.collections.yml", "collection"}, {"examples/config.inheritance.yml", ""}} {
args := []string{"promptkit", "show", "prepared", "generic.markdown_summary", "--config=" + test.config, "--profile=local-gpu", "--input-file=transcript=examples/fixtures/transcript.md", "--input-file=glossary=examples/fixtures/glossary.yml", "--format=json"}
if test.overlay != "" {
args = append(args, "--overlay="+test.overlay)
}
outcome, output, diagnostics := executeNative(t, args...)
if outcome.Status != 0 || !strings.Contains(output, "generic.markdown_summary") || !strings.Contains(output, "Session Transcript") {
t.Fatalf("%+v: %+v %q %q", test, outcome, output, diagnostics)
}
}
}

View File

@@ -1,159 +0,0 @@
package cli
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"go.maximumdirect.net/configkit"
)
type frameworkConfigurationConfig struct {
mode, configPath, configOverlay, schemaDir string
path string
configExplicit bool
pathExplicit bool
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
type frameworkConfigurationSource struct {
ID string `json:"id"`
Label string `json:"label"`
Layer configkit.Layer `json:"layer"`
Kind configkit.SourceKind `json:"kind"`
UnderlayIndex int `json:"underlay_index"`
Path string `json:"path,omitempty"`
ImportedFrom *frameworkConfigurationLocation `json:"imported_from,omitempty"`
}
type frameworkConfigurationLocation struct {
SourceID string `json:"source_id"`
Line int `json:"line"`
Column int `json:"column"`
}
func inspectFrameworkConfigurationCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseFrameworkConfigurationArgs(args)
if err != nil {
writeApplicationError(stderr, "inspect configuration parse", err)
return ExitRuntimeError
}
var explanationPath configkit.Path
if cfg.mode == "explain" {
explanationPath, err = configkit.ParsePath(cfg.path)
if err != nil {
writeApplicationError(stderr, "inspect configuration path", err)
return ExitRuntimeError
}
}
settings, err := resolveAppSettingsForFrameworkConfiguration(cfg)
if err != nil {
writeApplicationError(stderr, "inspect configuration", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
view, err := engine.InspectConfiguration()
if err != nil {
writeApplicationError(stderr, "inspect configuration", err)
return ExitRuntimeError
}
if err := configkit.WriteDiagnostics(stderr, view.Diagnostics()); err != nil {
return ExitRuntimeError
}
if cfg.mode == "validate" {
return ExitOK
}
if cfg.mode == "explain" {
if err := view.WriteExplanation(stdout, explanationPath); err != nil {
fmt.Fprintf(stderr, "inspect configuration explain %q: %v\n", cfg.path, err)
return ExitRuntimeError
}
return ExitOK
}
switch cfg.mode {
case "show":
err = view.WriteEffective(stdout)
case "sources":
sources := frameworkConfigurationSources(view.Sources())
err = json.NewEncoder(stdout).Encode(sources)
}
if err != nil {
fmt.Fprintf(stderr, "inspect configuration output error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseFrameworkConfigurationArgs(args []string) (*frameworkConfigurationConfig, error) {
if len(args) == 0 {
return nil, errors.New("configuration operation is required")
}
cfg := &frameworkConfigurationConfig{mode: args[0]}
switch cfg.mode {
case "validate", "show", "sources", "explain":
default:
return nil, fmt.Errorf("unknown configuration operation %q", cfg.mode)
}
fs := flag.NewFlagSet("inspect configuration "+cfg.mode, flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
fs.StringVar(&cfg.path, "path", "", "JSON Pointer to explain (required for explain)")
if err := fs.Parse(args[1:]); err != nil {
return nil, err
}
if fs.NArg() != 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.pathExplicit = flagWasSet(fs, "path")
if cfg.mode == "explain" && !cfg.pathExplicit {
return nil, errors.New("--path is required for explain")
}
cfg.overrides = applicationOverrides(fs)
return cfg, nil
}
func resolveAppSettingsForFrameworkConfiguration(cfg *frameworkConfigurationConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{
promptkitSources: settings.PromptKitSources,
schemaDir: settings.SchemaDir,
transientRetries: settings.TransientRetries,
}, nil
}
func frameworkConfigurationSources(sources []configkit.SourceInfo) []frameworkConfigurationSource {
result := make([]frameworkConfigurationSource, 0, len(sources))
for _, source := range sources {
item := frameworkConfigurationSource{
ID: source.ID, Label: source.Label, Layer: source.Layer,
Kind: source.Kind, UnderlayIndex: source.UnderlayIndex, Path: source.Path,
}
if source.ImportedFrom != nil {
item.ImportedFrom = &frameworkConfigurationLocation{
SourceID: source.ImportedFrom.SourceID,
Line: source.ImportedFrom.Line,
Column: source.ImportedFrom.Column,
}
}
result = append(result, item)
}
return result
}

View File

@@ -1,391 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
configkit "go.maximumdirect.net/configkit"
)
func TestFrameworkConfigurationCommandsNeedNoPromptOrCredential(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
"configuration", "validate", "--config", configPath,
})
if code != ExitOK || stdout != "" || stderr != "" {
t.Fatalf("source-free validation failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
t.Setenv("SCRIPTORIUM_FRAMEWORK_KEY", "")
root := t.TempDir()
frameworkFile := filepath.Join(root, "framework.yml")
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_FRAMEWORK_KEY
profiles:
- id: protected
backend: local
model: offline-model
api_key_required: true
`)
configPath = writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr = runCLICommand(t, inspectCommand, []string{
"configuration", mode, "--config", configPath,
})
if code != ExitOK || stderr != "" {
t.Fatalf("%s required a prompt or credential: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
}
if mode == "validate" && stdout != "" {
t.Fatalf("validation wrote success output: %q", stdout)
}
if mode == "show" && (!strings.Contains(stdout, "profiles:") || !strings.Contains(stdout, "offline-model")) {
t.Fatalf("effective framework YAML is missing: %q", stdout)
}
}
}
func TestFrameworkConfigurationSourcesProjectsOrderedNativeRecords(t *testing.T) {
root := t.TempDir()
importDir := filepath.Join(root, "catalog")
if err := os.MkdirAll(importDir, 0o755); err != nil {
t.Fatal(err)
}
mainPath := writeFrameworkSourceFile(t, root, "main.yml", `config:
imports:
- path: catalog/fragment.yml
required: true
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
`)
fragmentPath := writeFrameworkSourceFile(t, importDir, "fragment.yml", `profiles:
- id: imported
model: imported-model
`)
directory := t.TempDir()
directoryPath := writeFrameworkSourceFile(t, directory, "directory.yml", `profiles:
- id: directory
model: directory-model
`)
configuredMissing := filepath.Join(root, "configured-but-replaced.yml")
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", configuredMissing))
args := []string{
"configuration", "sources", "--config", configPath,
"--promptkit-source", mainPath,
"--promptkit-source", directory,
}
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
t.Fatalf("framework sources command failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
var sources []frameworkConfigurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil {
t.Fatalf("decode framework source DTOs: %v\n%s", err, stdout)
}
var rawSources []map[string]any
if err := json.Unmarshal([]byte(stdout), &rawSources); err != nil {
t.Fatalf("decode framework source fields: %v", err)
}
mainIndex := findFrameworkSource(t, sources, mainPath)
fragmentIndex := findFrameworkSource(t, sources, fragmentPath)
directoryIndex := findFrameworkSource(t, sources, directoryPath)
main := sources[mainIndex]
fragment := sources[fragmentIndex]
directorySource := sources[directoryIndex]
if !(mainIndex < fragmentIndex && fragmentIndex < directoryIndex) {
t.Fatalf("captured source order changed: main=%d fragment=%d directory=%d", mainIndex, fragmentIndex, directoryIndex)
}
if main.ID == "" || main.Label == "" || main.Layer != configkit.LayerUnderlay || main.Kind != configkit.SourceKindFile || main.UnderlayIndex < 1 {
t.Fatalf("explicit file source fields are incomplete: %+v", main)
}
for _, key := range []string{"id", "label", "layer", "kind", "underlay_index", "path"} {
if _, exists := rawSources[mainIndex][key]; !exists {
t.Fatalf("source DTO omitted %q: %#v", key, rawSources[mainIndex])
}
}
for _, key := range []string{"ID", "Label", "Layer", "Kind", "UnderlayIndex", "filesystem"} {
if _, exists := rawSources[mainIndex][key]; exists {
t.Fatalf("source DTO exposed implementation field %q: %#v", key, rawSources[mainIndex])
}
}
if fragment.Layer != configkit.LayerUnderlay || fragment.Kind != configkit.SourceKindFile || fragment.ImportedFrom == nil ||
fragment.ImportedFrom.SourceID != main.ID || fragment.ImportedFrom.Line < 1 || fragment.ImportedFrom.Column < 1 {
t.Fatalf("import source fields are incomplete: %+v", fragment)
}
if importedFrom, ok := rawSources[fragmentIndex]["imported_from"].(map[string]any); !ok || importedFrom["source_id"] != main.ID || importedFrom["line"] == nil || importedFrom["column"] == nil {
t.Fatalf("import location DTO fields are incomplete: %#v", rawSources[fragmentIndex])
}
if directorySource.Layer != configkit.LayerUnderlay || directorySource.Kind != configkit.SourceKindFile || directorySource.UnderlayIndex <= main.UnderlayIndex {
t.Fatalf("directory-selected source fields/order are incomplete: %+v (main %+v)", directorySource, main)
}
}
func TestFrameworkConfigurationShowFailsBeforeOutputForInvalidFinalDefinition(t *testing.T) {
root := t.TempDir()
frameworkFile := writeFrameworkSourceFile(t, root, "invalid.yml", `prompts:
- id: unused
messages:
- role: user
content: "{{"
output:
format: text
validation_mode: none
`)
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", configPath})
if code != ExitOK || stdout != "" || stderr != "" {
t.Fatalf("application validation loaded framework definitions: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
for _, args := range [][]string{
{"configuration", "validate", "--config", configPath},
{"configuration", "show", "--config", configPath},
{"configuration", "sources", "--config", configPath},
{"configuration", "explain", "--config", configPath, "--path", ""},
} {
code, stdout, stderr = runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid final framework definition was reported as success: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
}
func TestFrameworkConfigurationCommandsRejectExtraArgumentsAndReportWriterErrors(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"configuration"},
{"configuration", "explain"},
{"configuration", "show", "unexpected"},
{"configuration", "sources", "--format", "json"},
} {
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid inspection arguments were accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
writerError := errors.New("synthetic framework output failure")
var stderr bytes.Buffer
code := inspectCommand([]string{"configuration", "show", "--config", configPath}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
t.Fatalf("effective-output writer failure was not reported: code=%d stderr=%q", code, stderr.String())
}
stderr.Reset()
code = inspectCommand([]string{"configuration", "explain", "--config", configPath, "--path", ""}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
t.Fatalf("explanation writer failure was not reported: code=%d stderr=%q", code, stderr.String())
}
}
func TestFrameworkConfigurationExplainUsesLiteralJSONPointersAndNativeHistory(t *testing.T) {
root := t.TempDir()
lower := writeFrameworkSourceFile(t, root, "lower.yml", `collections:
support:
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: backend-token-displaced-canary
password: backend-password-canary
visible_option: backend-visible-value
provider.option: dotted-visible-value
profiles:
- id: summary
backend: support/local
model: lower-model
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: profile-token-displaced-canary
password: profile-password-displaced-canary
visible_option: profile-visible-value
provider.option: dotted-visible-value
prompts:
- id: explainable
default_profile: support/summary
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
upperRoot := t.TempDir()
upper := writeFrameworkSourceFile(t, upperRoot, "upper.yml", `collections:
support:
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: backend-token-effective-canary
password: backend-password-effective-canary
visible_option: backend-visible-value
provider.option: dotted-visible-value
profiles:
- id: summary
backend: support/local
model: effective-model
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: profile-token-effective-canary
password: profile-password-effective-canary
visible_option: profile-visible-value
provider.option: dotted-visible-value
`)
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n - file: %q\n", lower, upper))
const secret = "environment-value-must-never-be-emitted"
t.Setenv("SCRIPTORIUM_EXPLAIN_REFERENCE", secret)
for _, tc := range []struct {
path string
want []string
unwant []string
}{
{
path: "/collections/support/backends/0/extra_params/token",
unwant: []string{"backend-token-displaced-canary", "backend-token-effective-canary"},
},
{
path: "/collections/support/backends/0/extra_params/password",
unwant: []string{"backend-password-displaced-canary", "backend-password-effective-canary"},
},
{
path: "/collections/support/profiles/0/model",
want: []string{"effective-model", "lower-model"},
unwant: []string{},
},
{
path: "/collections/support/profiles/0/extra_params/token",
unwant: []string{"profile-token-displaced-canary", "profile-token-effective-canary"},
},
{
path: "/collections/support/profiles/0/extra_params/password",
unwant: []string{"profile-password-displaced-canary", "profile-password-effective-canary"},
},
{
path: "/collections/support/profiles/0/api_key_env",
want: []string{"SCRIPTORIUM_EXPLAIN_REFERENCE"},
unwant: []string{secret},
},
{
path: "/collections/support/profiles/0/extra_params/provider.option",
want: []string{"dotted-visible-value"},
unwant: []string{"provider/option"},
},
{
path: "",
want: []string{"profiles:", "effective-model"},
unwant: []string{"profile-token-displaced-canary", "backend-password-effective-canary"},
},
} {
args := []string{"configuration", "explain", "--config", configPath, "--path", tc.path}
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitOK || stderr != "" {
t.Fatalf("explain %q failed: code=%d stdout=%q stderr=%q", tc.path, code, stdout, stderr)
}
if stdout == "" {
t.Fatalf("explain %q produced no output", tc.path)
}
for _, value := range tc.want {
if !strings.Contains(stdout, value) {
t.Errorf("explain %q omitted %q: %s", tc.path, value, stdout)
}
}
for _, value := range tc.unwant {
if strings.Contains(stdout, value) {
t.Errorf("explain %q disclosed %q: %s", tc.path, value, stdout)
}
}
if strings.Contains(stdout, secret) {
t.Errorf("explain %q disclosed the resolved environment value: %s", tc.path, stdout)
}
}
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{"configuration", "show", "--config", configPath})
if code != ExitOK || stderr != "" {
t.Fatalf("framework show failed: code=%d stderr=%q", code, stderr)
}
for _, value := range []string{"SCRIPTORIUM_EXPLAIN_REFERENCE", "backend-visible-value", "profile-visible-value", "dotted-visible-value"} {
if !strings.Contains(stdout, value) {
t.Errorf("effective report omitted visible configuration %q: %s", value, stdout)
}
}
for _, value := range []string{
"backend-token-displaced-canary", "backend-token-effective-canary",
"backend-password-displaced-canary", "backend-password-effective-canary",
"profile-token-displaced-canary", "profile-token-effective-canary",
"profile-password-displaced-canary", "profile-password-effective-canary", secret,
} {
if strings.Contains(stdout, value) {
t.Errorf("effective report disclosed classified or resolved value %q: %s", value, stdout)
}
}
profileCode, profileOutput, profileError := runCLICommand(t, inspectCommand, []string{
"profile", "--config", configPath, "--profile", "support/summary", "--format", "json",
})
if profileCode != ExitOK || profileError != "" {
t.Fatalf("existing profile inspection failed: code=%d stdout=%q stderr=%q", profileCode, profileOutput, profileError)
}
var profile renderformat.ProfileInspection
if err := json.Unmarshal([]byte(profileOutput), &profile); err != nil {
t.Fatalf("decode profile inspection: %v\n%s", err, profileOutput)
}
if pointerValue(profile.ProfileModelParams.Model) != "effective-model" || pointerValue(profile.ProfileModelParams.ExtraParams)["visible_option"] != "profile-visible-value" {
t.Fatalf("existing profile inspection changed configured provider data: %+v", profile.ProfileModelParams)
}
if strings.Contains(profileOutput, secret) {
t.Fatalf("profile inspection emitted resolved environment data: %s", profileOutput)
}
code, rendered, stderr := runCLICommand(t, renderCommand, []string{"--config", configPath, "--prompt", "support/explainable"})
if code != ExitOK || stderr != "" {
t.Fatalf("prepared output failed: code=%d stdout=%q stderr=%q", code, rendered, stderr)
}
if !strings.Contains(rendered, "profile-visible-value") || strings.Contains(rendered, secret) {
t.Fatalf("prepared output did not preserve ordinary provider data or disclosed environment data: %s", rendered)
}
}
func TestFrameworkConfigurationExplainRejectsMissingMalformedAndUnknownPaths(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"configuration", "explain"},
{"configuration", "explain", "--path", "not-a-pointer"},
{"configuration", "explain", "--path", "/profiles/team~2summary/model"},
{"configuration", "explain", "--config", configPath, "--path", "/profiles/missing/model"},
} {
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid explanation path was accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
}
func findFrameworkSource(t *testing.T, sources []frameworkConfigurationSource, path string) int {
t.Helper()
for index, source := range sources {
if filepath.Clean(source.Path) == filepath.Clean(path) {
return index
}
}
t.Fatalf("selected source path %q was not present: %+v", path, sources)
return -1
}
type frameworkConfigurationFailingWriter struct {
err error
}
func (writer frameworkConfigurationFailingWriter) Write([]byte) (int, error) {
return 0, writer.err
}

View File

@@ -1,17 +0,0 @@
package cli
import (
"fmt"
"io"
"go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
func writeFrameworkDiagnostics(output io.Writer, engine *promptkit.Engine) error {
view, err := engine.InspectConfiguration()
if err != nil {
return fmt.Errorf("inspect captured PromptKit diagnostics: %w", err)
}
return configkit.WriteDiagnostics(output, view.Diagnostics())
}

View File

@@ -1,386 +0,0 @@
package cli
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"testing"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
configkit "go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
func TestFrameworkSourceGroupsComposeAndValidateEffectiveDefinitions(t *testing.T) {
conflicting := t.TempDir()
writeFrameworkSourceFile(t, conflicting, "first.yml", `profiles:
- id: selected
model: first
`)
writeFrameworkSourceFile(t, conflicting, "second.yml", `profiles:
- id: selected
model: second
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{Directory: conflicting}}}); err == nil {
t.Fatal("conflicting files in one source group unexpectedly constructed an engine")
}
lower := t.TempDir()
upper := t.TempDir()
empty := t.TempDir()
writeFrameworkSourceFile(t, lower, "lower.yml", `profiles:
- id: selected
endpoint: not-a-url
model: lower-model
temperature: 10
`)
writeFrameworkSourceFile(t, upper, "upper.yml", `profiles:
- id: selected
endpoint: http://127.0.0.1:1/v1
model: upper-model
temperature: 0.5
`)
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{
{Directory: lower}, {Directory: empty}, {Directory: upper},
}, "")
inspection, err := engine.InspectProfile(context.Background(), "selected")
if err != nil {
t.Fatalf("inspect profile after later source replacement: %v", err)
}
if pointerValue(inspection.Endpoint) != "http://127.0.0.1:1/v1" || pointerValue(inspection.Model) != "upper-model" || pointerValue(inspection.Temperature) != 0.5 {
t.Fatalf("later source did not replace lower values before validation: %+v", inspection)
}
}
func TestFrameworkSourceConstructionUsesCombinedDocumentAndRejectsInvalidSources(t *testing.T) {
root := t.TempDir()
combined := filepath.Join(root, "combined.yml")
writeFrameworkSourceFile(t, root, "combined.yml", `backends:
- id: local
endpoint: http://127.0.0.1:1/v1
profiles:
- id: selected
backend: local
model: combined-model
prompts:
- id: combined.prompt
default_profile: selected
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
client := &frameworkSourceCaptureClient{}
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: combined}}, "", promptkit.WithLLMClient(client))
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "combined.prompt"}); err != nil {
t.Fatalf("run definitions from one combined document: %v", err)
}
if len(client.requests) != 1 || client.requests[0].Target.BackendID != "local" || client.requests[0].Target.Model != "combined-model" {
t.Fatalf("combined backend/profile/prompt values did not reach generation: %+v", client.requests)
}
malformed := filepath.Join(root, "malformed.yml")
writeFrameworkSourceFile(t, root, "malformed.yml", "not: [valid\n")
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: malformed}}}); err == nil {
t.Fatal("malformed framework source unexpectedly constructed an engine")
}
invalidUnused := filepath.Join(root, "invalid-unused.yml")
writeFrameworkSourceFile(t, root, "invalid-unused.yml", `prompts:
- id: unused.prompt
messages:
- role: user
content: "{{"
output:
format: text
validation_mode: none
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: invalidUnused}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("invalid unused definition did not fail as framework configuration: %v", err)
}
}
func TestFrameworkRejectsObsoleteVersionKeyedPrompt(t *testing.T) {
root := t.TempDir()
path := writeFrameworkSourceFile(t, root, "obsolete.yml", `prompts:
old-prompt:
"1":
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: path}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("obsolete version-keyed definition was not rejected by PromptKit: %v", err)
}
}
func TestFrameworkImportsRetainResourceOriginsAndCaptureAtConstruction(t *testing.T) {
root := t.TempDir()
catalogDir := filepath.Join(root, "catalog")
if err := os.MkdirAll(catalogDir, 0o755); err != nil {
t.Fatal(err)
}
schemaDir := filepath.Join(root, "schemas")
if err := os.MkdirAll(schemaDir, 0o755); err != nil {
t.Fatal(err)
}
mainFile := filepath.Join(root, "main.yml")
writeFrameworkSourceFile(t, root, "main.yml", `config:
imports:
- path: catalog/base.yml
required: true
`)
writeFrameworkSourceFile(t, catalogDir, "base.yml", `profiles:
- id: profile-primary
endpoint: http://127.0.0.1:1/v1
model: imported-model
prompts:
- id: team-prompt
default_profile: profile-primary
inputs:
- name: source
required: true
messages:
- role: user
content_file: ./body.md
- role: user
content: 'Artifact: {{input "source"}}'
output:
format: json
validation_mode: json_schema
schema_path: response.schema.json
`)
writeFrameworkSourceFile(t, catalogDir, "body.md", "Captured message for {{.audience}}")
artifactPath := filepath.Join(root, "request-artifact.txt")
writeFrameworkSourceFile(t, root, "request-artifact.txt", "initial artifact")
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"initial schema"}`)
selections := []appconfig.SourceSelection{{File: mainFile}}
engine := newFrameworkSourceTestEngine(t, selections, schemaDir)
writeFrameworkSourceFile(t, catalogDir, "body.md", "edited message for {{.audience}}")
writeFrameworkSourceFile(t, root, "request-artifact.txt", "prepared artifact")
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"prepared schema"}`)
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "team-prompt",
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
Vars: map[string]string{"audience": "reader"},
})
if err != nil {
t.Fatalf("prepare inherited imported prompt: %v", err)
}
if len(prepared.Messages) != 2 || prepared.Messages[0].Content != "Captured message for reader" || prepared.Messages[1].Content != "Artifact: prepared artifact" {
t.Fatalf("resource origin or capture/preparation behavior changed: %+v", prepared.Messages)
}
if prepared.SelectedProfileID != "profile-primary" {
t.Fatalf("profile ID was not preserved: %+v", prepared)
}
if prepared.StructuredOutput == nil || prepared.StructuredOutput.JSONSchema == nil || prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)["title"] != "prepared schema" {
t.Fatalf("schema was not loaded during preparation: %+v", prepared.StructuredOutput)
}
newEngine := newFrameworkSourceTestEngine(t, selections, schemaDir)
newPrepared, err := newEngine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "team-prompt",
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
Vars: map[string]string{"audience": "reader"},
})
if err != nil {
t.Fatalf("prepare with newly captured message: %v", err)
}
if newPrepared.Messages[0].Content != "edited message for reader" {
t.Fatalf("new engine did not observe the message edit: %+v", newPrepared.Messages[0])
}
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\nschema_dir: %q\n", mainFile, schemaDir))
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--config", configPath,
"--prompt", "team-prompt",
"--input", "source=" + artifactPath,
"--var", "audience=reader",
"--format", "json",
})
if code != ExitOK {
t.Fatalf("render exact literal IDs through CLI: code=%d stderr=%q", code, stderr)
}
var rendered promptkit.PreparedRun
if err := json.Unmarshal([]byte(stdout), &rendered); err != nil {
t.Fatalf("decode rendered literal-ID fixture: %v", err)
}
if rendered.PromptID != "team-prompt" || rendered.SelectedProfileID != "profile-primary" {
t.Fatalf("CLI changed exact prompt/profile IDs: %+v", rendered)
}
}
func TestNativeProfileInheritanceAndPresenceReachTheClientBoundary(t *testing.T) {
root := t.TempDir()
frameworkFile := filepath.Join(root, "framework.yml")
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
- id: first
endpoint: https://first.example/v1
api_key_env: FIRST_BACKEND_KEY
extra_params:
source: first
nested:
first_backend: true
- id: second
endpoint: https://second.example/v1
api_key_env: SECOND_BACKEND_KEY
extra_params:
source: second
nested:
second_backend: true
profiles:
- id: parent
backend: first
model: parent-model
temperature: 0.7
top_p: 0.6
max_tokens: 45
timeout_seconds: 30
service_tier: flex
extra_params:
nested:
parent_profile: true
retained: "yes"
- id: backend-child
extends: {id: parent}
backend: second
model: child-model
- id: presence-child
extends: {id: parent}
backend: second
api_key_env: null
temperature: null
max_tokens: 0
timeout_seconds: 0
extra_params:
nested:
explicit_false: false
- id: map-null
backend: second
model: map-model
extra_params: null
- id: map-empty
backend: second
model: map-model
extra_params: {}
prompts:
- id: execution.prompt
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
client := &frameworkSourceCaptureClient{}
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: frameworkFile}}, "", promptkit.WithLLMClient(client))
for _, profileID := range []string{"backend-child", "presence-child", "map-null", "map-empty"} {
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "execution.prompt", ProfileID: profileID}); err != nil {
t.Fatalf("run profile %q: %v", profileID, err)
}
}
if len(client.requests) != 4 {
t.Fatalf("expected four provider-bound requests, got %d", len(client.requests))
}
backendChild := client.requests[0].Target
wantBackendParams := map[string]any{"source": "second", "nested": map[string]any{
"second_backend": true, "parent_profile": true, "retained": "yes",
}}
if backendChild.BackendID != "second" || backendChild.Endpoint != "https://second.example/v1" || backendChild.APIKeyEnv != "SECOND_BACKEND_KEY" || backendChild.Model != "child-model" ||
backendChild.Temperature != 0.7 || backendChild.TopP != 0.6 || backendChild.MaxTokens != 45 || backendChild.TimeoutSeconds != 30 || backendChild.ServiceTier != "flex" || !reflect.DeepEqual(backendChild.ExtraParams, wantBackendParams) {
t.Fatalf("backend change lost backend-derived or explicit inherited values: %+v", backendChild)
}
presenceRequest := client.requests[1]
presenceTarget := presenceRequest.Target
if presenceTarget.APIKeyEnv != "SECOND_BACKEND_KEY" || presenceTarget.Temperature != 0 || presenceTarget.MaxTokens != 0 || presenceTarget.TimeoutSeconds != 0 ||
!presenceRequest.TargetPresence.MaxTokens || !presenceRequest.TargetPresence.TimeoutSeconds || presenceRequest.TargetPresence.Temperature || !presenceRequest.TargetPresence.TopP {
t.Fatalf("explicit null/zero presence did not reach the injected client: target=%+v presence=%+v", presenceTarget, presenceRequest.TargetPresence)
}
wantPresenceParams := map[string]any{"source": "second", "nested": map[string]any{
"second_backend": true, "parent_profile": true, "retained": "yes", "explicit_false": false,
}}
if !reflect.DeepEqual(presenceTarget.ExtraParams, wantPresenceParams) {
t.Fatalf("recursive provider-map retention changed: %#v", presenceTarget.ExtraParams)
}
if !reflect.DeepEqual(client.requests[2].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
t.Fatalf("whole-map null did not fall back to backend defaults: %#v", client.requests[2].Target.ExtraParams)
}
if !reflect.DeepEqual(client.requests[3].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
t.Fatalf("empty provider map unexpectedly cleared backend defaults: %#v", client.requests[3].Target.ExtraParams)
}
}
func TestCLICompletesPartialFrameworkProfileWithRequestOverrides(t *testing.T) {
root := t.TempDir()
frameworkFile := writeFrameworkSourceFile(t, root, "partial.yml", `profiles:
- id: partial
prompts:
- id: partial.prompt
default_profile: partial
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
provider := newTestLLMServer("completed partial profile", nil)
defer provider.Close()
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--config", configPath,
"--prompt", "partial.prompt",
"--llm-base-url", provider.URL + "/v1",
"--model", "request-model",
})
if code != ExitOK || stdout != "completed partial profile" {
t.Fatalf("request overrides did not complete the partial profile: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
}
func newFrameworkSourceTestEngine(t *testing.T, selections []appconfig.SourceSelection, schemaDir string, options ...promptkit.Option) *promptkit.Engine {
t.Helper()
engine, err := newEngine(engineSettings{promptkitSources: selections, schemaDir: schemaDir}, options...)
if err != nil {
var loadError *configkit.LoadError
if errors.As(err, &loadError) {
t.Fatalf("construct engine from selected framework sources: %v (%+v)", err, loadError.Diagnostics())
}
t.Fatalf("construct engine from selected framework sources: %v", err)
}
return engine
}
func writeFrameworkSourceFile(t *testing.T, directory, name, contents string) string {
t.Helper()
if err := os.MkdirAll(directory, 0o755); err != nil {
t.Fatalf("create framework source directory: %v", err)
}
path := filepath.Join(directory, name)
if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
t.Fatalf("write framework source %q: %v", path, err)
}
return path
}
type frameworkSourceCaptureClient struct {
requests []promptkit.GenerateRequest
}
func (client *frameworkSourceCaptureClient) Generate(_ context.Context, request promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
client.requests = append(client.requests, request)
return &promptkit.GenerateResponse{Content: "ok"}, nil
}

View File

@@ -1,212 +0,0 @@
package cli
import (
"context"
"errors"
"flag"
"fmt"
"io"
"path/filepath"
"strings"
appformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
"go.maximumdirect.net/configkit"
)
type promptInspectionConfig struct {
configPath, promptID, schemaDir, outputPath string
configOverlay string
configExplicit bool
outputFormat appformat.OutputFormat
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
type profileInspectionConfig struct {
configPath, profileID, schemaDir, outputPath string
configOverlay string
configExplicit bool
outputFormat appformat.OutputFormat
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
func inspectCommand(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "inspect parse error: inspection mode is required")
return ExitRuntimeError
}
if args[0] == "profile" {
return inspectProfileCommand(args[1:], stdout, stderr)
}
if args[0] == "configuration" {
return inspectFrameworkConfigurationCommand(args[1:], stdout, stderr)
}
if args[0] != "prompt" {
fmt.Fprintln(stderr, "inspect parse error: unknown inspection mode")
return ExitRuntimeError
}
cfg, err := parsePromptInspectionArgs(args[1:])
if err != nil {
writeApplicationError(stderr, "inspect parse", err)
return ExitRuntimeError
}
settings, err := resolveAppSettingsForPromptInspection(cfg)
if err != nil {
writeApplicationError(stderr, "inspect", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
data, err := appformat.FormatPromptInspection(inspection, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func inspectProfileCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseProfileInspectionArgs(args)
if err != nil {
writeApplicationError(stderr, "inspect parse", err)
return ExitRuntimeError
}
settings, err := resolveAppSettingsForProfileInspection(cfg)
if err != nil {
writeApplicationError(stderr, "inspect", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
inspection, err := engine.InspectProfile(context.Background(), cfg.profileID)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
data, err := appformat.FormatProfileInspection(inspection, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseProfileInspectionArgs(args []string) (*profileInspectionConfig, error) {
cfg := &profileInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
fs := flag.NewFlagSet("inspect profile", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
var promptkitSources promptKitSourceFlagValue
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
fs.StringVar(&cfg.profileID, "profile", "", "profile ID to inspect")
rawFormat := ""
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.overrides = applicationOverrides(fs)
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
if strings.TrimSpace(cfg.profileID) == "" {
return nil, errors.New("--profile is required")
}
format, err := appformat.ParseOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
return cfg, nil
}
func resolveAppSettingsForProfileInspection(cfg *profileInspectionConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
}
func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
cfg := &promptInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
fs := flag.NewFlagSet("inspect prompt", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
var promptkitSources promptKitSourceFlagValue
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to inspect")
rawFormat := ""
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.overrides = applicationOverrides(fs)
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
if strings.TrimSpace(cfg.promptID) == "" {
return nil, errors.New("--prompt is required")
}
format, err := appformat.ParseOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
return cfg, nil
}
func resolveAppSettingsForPromptInspection(cfg *promptInspectionConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
}

View File

@@ -1,636 +0,0 @@
package cli
import (
"context"
"errors"
"flag"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"time"
httpadapter "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
"go.maximumdirect.net/commandkit"
"go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
const (
ExitOK = 0
ExitRuntimeError = 1
ExitValidationFailed = 2
)
type runConfig struct {
configPath string
promptkitSources []appconfig.SourceSelection
promptID string
profileID string
sessionID string
inputRaw listFlag
varRaw listFlag
outputPath string
executionReportPath string
llmBaseURL string
apiKeyEnv string
model string
temperature float64
maxTokens int
topP float64
reasoningEffort string
schemaDir string
timeout time.Duration
runTimeout time.Duration
transientRetries int
configuredTransientRetries *int
diagnostics []configkit.Diagnostic
configOverlay string
defaultRenderFormat renderformat.PreparedRunOutputFormat
llmBaseURLSet bool
apiKeyEnvSet bool
modelSet bool
temperatureSet bool
maxTokensSet bool
topPSet bool
reasoningEffortSet bool
timeoutSet bool
transientRetriesSet bool
}
type renderConfig struct {
runConfig
outputFormat renderformat.PreparedRunOutputFormat
}
type serveConfig struct {
configPath string
addr string
promptkitSources []appconfig.SourceSelection
schemaDir string
artifactRoot string
maxRequestBytes int64
maxArtifactBytes int64
maxResponseBytes int64
transientRetries *int
diagnostics []configkit.Diagnostic
configOverlay string
}
type commonCommandSettings struct {
promptkitSources []appconfig.SourceSelection
schemaDir string
serverAddr string
artifactRoot string
maxRequestBytes int64
maxArtifactBytes int64
maxResponseBytes int64
defaultRenderFormat renderformat.PreparedRunOutputFormat
transientRetries *int
diagnostics []configkit.Diagnostic
}
type listFlag []string
func (l *listFlag) String() string {
return strings.Join(*l, ",")
}
func (l *listFlag) Set(v string) error {
*l = append(*l, v)
return nil
}
func Run(args []string, stdout, stderr io.Writer) int {
return Execute(context.Background(), commandkit.Identity{Name: "scriptorium", Version: "dev"}, args, stdout, stderr).Status
}
func runCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseRunArgs(args)
if err != nil {
writeApplicationError(stderr, "run parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
req, err := buildRunRequestFromConfig(cfg)
if err != nil {
fmt.Fprintf(stderr, "run parse error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.engineSettings())
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
return executeRun(cfg, req, engine.Run, stdout, stderr)
}
func renderCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseRenderArgs(args)
if err != nil {
writeApplicationError(stderr, "render parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
req, err := buildRunRequestFromConfig(&cfg.runConfig)
if err != nil {
fmt.Fprintf(stderr, "render parse error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.runConfig.engineSettings())
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
prepared, prepErr := engine.Prepare(context.Background(), req)
if prepErr != nil {
fmt.Fprintf(stderr, "render error: %v\n", prepErr)
return ExitRuntimeError
}
out, err := renderformat.FormatPreparedRun(prepared, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "render format error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, out); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func serveCommand(args []string, stderr io.Writer) int {
cfg, err := parseServeArgs(args)
if err != nil {
writeApplicationError(stderr, "serve parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
artifactReader, err := httpadapter.NewRestrictedArtifactReader(cfg.artifactRoot, cfg.maxArtifactBytes)
if err != nil {
fmt.Fprintf(stderr, "artifact root error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.engineSettings(), promptkit.WithArtifactReader(artifactReader))
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
h := httpadapter.NewHandlerWithOptions(engine, httpadapter.HandlerOptions{
MaxRequestBytes: cfg.maxRequestBytes,
MaxResponseBytes: cfg.maxResponseBytes,
})
srv := &http.Server{
Addr: cfg.addr,
Handler: h,
ReadHeaderTimeout: defaults.HTTPReadHeaderTimeoutDefault,
}
fmt.Fprintf(stderr, "serving on %s\n", cfg.addr)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
fmt.Fprintf(stderr, "server error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseRunArgs(args []string) (*runConfig, error) {
cfg := &runConfig{}
fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
registerExecutionRequestFlags(fs, cfg)
fs.DurationVar(&cfg.runTimeout, "run-timeout", 0, "whole-run execution deadline")
fs.StringVar(&cfg.executionReportPath, "execution-report", "", "JSON execution report file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
if err := finalizeExecutionRequestConfig(fs, cfg); err != nil {
return nil, err
}
if cfg.runTimeout < 0 {
return nil, errors.New("--run-timeout must be greater than or equal to 0")
}
if flagWasSet(fs, "execution-report") {
if cfg.executionReportPath == "" {
return nil, errors.New("--execution-report requires a nonempty path")
}
cfg.executionReportPath = filepath.Clean(cfg.executionReportPath)
if cfg.outputPath != "" {
collision, err := outputPathsCollide(cfg.outputPath, cfg.executionReportPath)
if err != nil {
return nil, err
}
if collision {
return nil, errors.New("--out and --execution-report select the same file")
}
}
}
return cfg, nil
}
func outputPathsCollide(first, second string) (bool, error) {
firstAbs, err := filepath.Abs(first)
if err != nil {
return false, err
}
secondAbs, err := filepath.Abs(second)
if err != nil {
return false, err
}
if firstAbs == secondAbs {
return true, nil
}
firstInfo, firstErr := os.Stat(firstAbs)
secondInfo, secondErr := os.Stat(secondAbs)
return firstErr == nil && secondErr == nil && os.SameFile(firstInfo, secondInfo), nil
}
func parseRenderArgs(args []string) (*renderConfig, error) {
cfg := &renderConfig{
outputFormat: renderformat.DefaultPreparedRunOutputFormat,
}
fs := flag.NewFlagSet("render", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
registerExecutionRequestFlags(fs, &cfg.runConfig)
var rawFormat string
fs.StringVar(&rawFormat, "format", "", "render output format (text|json)")
if err := fs.Parse(args); err != nil {
return nil, err
}
if err := finalizeExecutionRequestConfig(fs, &cfg.runConfig); err != nil {
return nil, err
}
if flagWasSet(fs, "format") {
format, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
} else {
cfg.outputFormat = cfg.defaultRenderFormat
}
return cfg, nil
}
func parseServeArgs(args []string) (*serveConfig, error) {
cfg := &serveConfig{}
fs := flag.NewFlagSet("serve", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
fs.StringVar(&cfg.addr, "addr", "", "HTTP listen address")
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
fs.Int64Var(&cfg.maxArtifactBytes, "max-artifact-bytes", 0, "maximum HTTP file artifact bytes; 0 disables the limit")
fs.Int64Var(&cfg.maxResponseBytes, "max-response-bytes", 0, "maximum HTTP response body bytes; 0 disables the limit")
if err := fs.Parse(args); err != nil {
return nil, err
}
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
if err != nil {
return nil, err
}
cfg.promptkitSources = settings.promptkitSources
cfg.schemaDir = settings.schemaDir
cfg.addr = settings.serverAddr
cfg.artifactRoot = settings.artifactRoot
cfg.maxRequestBytes = settings.maxRequestBytes
cfg.maxArtifactBytes = settings.maxArtifactBytes
cfg.maxResponseBytes = settings.maxResponseBytes
cfg.transientRetries = settings.transientRetries
cfg.diagnostics = settings.diagnostics
return cfg, nil
}
func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
fs.StringVar(&cfg.sessionID, "session-id", "", "optional session ID")
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1")
fs.StringVar(&cfg.apiKeyEnv, "api-key-env", "", "environment variable name containing API key")
fs.StringVar(&cfg.model, "model", "", "model name")
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
fs.Float64Var(&cfg.topP, "top-p", 0, "optional top_p override")
fs.StringVar(&cfg.reasoningEffort, "reasoning-effort", "", "optional reasoning effort override")
fs.DurationVar(&cfg.timeout, "timeout", 0, "LLM request timeout")
fs.IntVar(&cfg.transientRetries, "transient-retries", 0, "additional transient generation attempts")
fs.StringVar(&cfg.promptID, "prompt-id", "", "deprecated alias for --prompt")
fs.StringVar(&cfg.profileID, "profile-id", "", "deprecated alias for --profile")
}
func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
if fs.NArg() > 0 {
return fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
if err != nil {
return err
}
cfg.promptkitSources = settings.promptkitSources
cfg.schemaDir = settings.schemaDir
cfg.defaultRenderFormat = settings.defaultRenderFormat
cfg.configuredTransientRetries = settings.transientRetries
cfg.diagnostics = settings.diagnostics
if strings.TrimSpace(cfg.promptID) == "" {
return errors.New("--prompt is required")
}
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
cfg.llmBaseURLSet = flagWasSet(fs, "llm-base-url")
cfg.apiKeyEnvSet = flagWasSet(fs, "api-key-env")
cfg.modelSet = flagWasSet(fs, "model")
cfg.temperatureSet = flagWasSet(fs, "temperature")
cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
cfg.topPSet = flagWasSet(fs, "top-p")
cfg.reasoningEffortSet = flagWasSet(fs, "reasoning-effort")
cfg.timeoutSet = flagWasSet(fs, "timeout")
cfg.transientRetriesSet = flagWasSet(fs, "transient-retries")
if cfg.transientRetriesSet && cfg.transientRetries < 0 {
return errors.New("--transient-retries must be greater than or equal to 0")
}
return nil
}
func registerConfigSelectionFlags(fs *flag.FlagSet, target, overlay *string) {
fs.StringVar(
target,
"config",
"",
fmt.Sprintf(
"application config path (default search: %s, then %s)",
appconfig.DefaultConfigPathLocal,
appconfig.DefaultConfigPath,
),
)
fs.StringVar(overlay, "config-overlay", "", "selected application configuration overlay")
}
func resolveCommonSettings(fs *flag.FlagSet, configPath, overlay string) (commonCommandSettings, error) {
settings, diagnostics, err := loadApplicationSettings(configPath, overlay, flagWasSet(fs, "config"), applicationOverrides(fs))
if err != nil {
return commonCommandSettings{}, err
}
return commonCommandSettings{
promptkitSources: settings.PromptKitSources,
schemaDir: settings.SchemaDir,
serverAddr: settings.ServerAddr,
artifactRoot: settings.ArtifactRoot,
maxRequestBytes: settings.MaxRequestBytes,
maxArtifactBytes: settings.MaxArtifactBytes,
maxResponseBytes: settings.MaxResponseBytes,
defaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
transientRetries: settings.TransientRetries,
diagnostics: diagnostics,
}, nil
}
type engineSettings struct {
promptkitSources []appconfig.SourceSelection
schemaDir string
transientRetries *int
}
func (c runConfig) engineSettings() engineSettings {
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.configuredTransientRetries}
}
func (c serveConfig) engineSettings() engineSettings {
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.transientRetries}
}
func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
var inputMappings map[string]string
var err error
if len(cfg.inputRaw) > 0 {
inputMappings, err = parseMappings(cfg.inputRaw, false)
if err != nil {
return promptkit.RunRequest{}, fmt.Errorf("input parse error: %w", err)
}
}
varMappings := map[string]string{}
if len(cfg.varRaw) > 0 {
varMappings, err = parseMappings(cfg.varRaw, false)
if err != nil {
return promptkit.RunRequest{}, fmt.Errorf("var parse error: %w", err)
}
}
var inputs map[string]promptkit.ArtifactRef
if len(inputMappings) > 0 {
inputs = make(map[string]promptkit.ArtifactRef, len(inputMappings))
for name, path := range inputMappings {
inputs[name] = promptkit.File(path)
}
}
var modelOverride *promptkit.ExecutionTargetOverride
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.reasoningEffortSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
modelOverride = &promptkit.ExecutionTargetOverride{
Endpoint: cfg.llmBaseURL,
Model: cfg.model,
APIKeyEnv: cfg.apiKeyEnv,
}
if cfg.temperatureSet {
modelOverride.Temperature = &cfg.temperature
}
if cfg.maxTokensSet {
modelOverride.MaxTokens = &cfg.maxTokens
}
if cfg.topPSet {
modelOverride.TopP = &cfg.topP
}
if cfg.reasoningEffortSet {
modelOverride.ReasoningEffort = &cfg.reasoningEffort
}
if cfg.timeoutSet {
timeoutSeconds := int(cfg.timeout.Seconds())
modelOverride.TimeoutSeconds = &timeoutSeconds
}
}
var transientRetries *int
if cfg.transientRetriesSet {
transientRetries = &cfg.transientRetries
}
return promptkit.RunRequest{
PromptID: cfg.promptID,
ProfileID: cfg.profileID,
SessionID: cfg.sessionID,
Inputs: inputs,
Vars: varMappings,
Execution: modelOverride,
TransientRetries: transientRetries,
}, nil
}
func parseMappings(raw []string, allowEmptyValue bool) (map[string]string, error) {
out := make(map[string]string)
for _, entry := range raw {
for _, piece := range strings.Split(entry, ",") {
piece = strings.TrimSpace(piece)
if piece == "" {
continue
}
key, value, err := parseMapping(piece)
if err != nil {
return nil, err
}
if !allowEmptyValue && strings.TrimSpace(value) == "" {
return nil, fmt.Errorf("mapping %q has empty value", piece)
}
out[key] = value
}
}
if len(out) == 0 {
return nil, errors.New("no valid mappings provided")
}
return out, nil
}
func parseMapping(value string) (string, string, error) {
idx := strings.IndexRune(value, '=')
if idx <= 0 {
return "", "", fmt.Errorf("invalid mapping %q, expected name=value", value)
}
key := strings.TrimSpace(value[:idx])
val := strings.TrimSpace(value[idx+1:])
if key == "" {
return "", "", fmt.Errorf("invalid mapping %q, empty name", value)
}
return key, val, nil
}
func flagWasSet(fs *flag.FlagSet, name string) bool {
set := false
fs.Visit(func(f *flag.Flag) {
if f.Name == name {
set = true
}
})
return set
}
func writeOutput(stdout io.Writer, outputPath string, body []byte) error {
if outputPath == "" {
_, err := stdout.Write(body)
return err
}
return os.WriteFile(outputPath, body, 0644)
}
func determineExitCode(runErr error, result *promptkit.RunResult) int {
if runErr != nil {
return ExitRuntimeError
}
if result != nil && result.Validation.Status == promptkit.ValidationFailed {
return ExitValidationFailed
}
return ExitOK
}
func printSummary(stderr io.Writer, res *promptkit.RunResult) {
if res == nil {
return
}
fmt.Fprintf(stderr, "prompt=%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
res.PromptID,
res.SelectedProfileID,
res.ModelName,
res.Validation.Status,
res.Validation.Mode,
len(res.Validation.Errors),
res.RenderedPromptHash,
len(res.InputHashes),
res.Usage.PromptTokens,
res.Usage.CompletionTokens,
res.Usage.TotalTokens,
)
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
}
if res.SelectedBackendID != "" {
fmt.Fprintf(stderr, " backend=%s", res.SelectedBackendID)
}
fmt.Fprintln(stderr)
}
func runErrorMessage(err error) string {
if errors.Is(err, promptkit.ErrCapacityExceeded) {
return "run error: model backend capacity is exhausted"
}
return fmt.Sprintf("run error: %v", err)
}
func printUsage(w io.Writer) {
fmt.Fprintln(w, "usage: scriptorium <run|render|serve|inspect|config> ...")
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]")
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]")
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--addr %s] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
fmt.Fprintln(w, " inspect configuration: scriptorium inspect configuration <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--path JSON_POINTER (required for explain)]")
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [application overrides] [--path /pointer for explain]")
}

File diff suppressed because it is too large Load Diff

View File

@@ -153,7 +153,7 @@ import (
_ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/config"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/format"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/report"
)
`
if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {

View File

@@ -1,179 +0,0 @@
package format
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"strconv"
"go.maximumdirect.net/promptkit"
)
type PromptInspection struct {
PromptID string `json:"prompt_id"`
PromptHash string `json:"prompt_hash"`
DefaultProfileID *string `json:"default_profile_id,omitempty"`
Inputs *[]PromptInspectionInput `json:"inputs,omitempty"`
OutputContract OutputContract `json:"output_contract"`
}
type PromptInspectionInput struct {
Name string `json:"name"`
Required *bool `json:"required,omitempty"`
ContentType *string `json:"content_type,omitempty"`
Description *string `json:"description,omitempty"`
}
type OutputContract struct {
Format string `json:"format"`
ValidationMode string `json:"validation_mode"`
SchemaPath *string `json:"schema_path,omitempty"`
RepairAttempts *int `json:"repair_attempts,omitempty"`
}
type ProfileInspection struct {
ProfileID string `json:"profile_id"`
ProfileModelParams ProfileModelParams `json:"profile_model_params"`
APIKeyRequired *bool `json:"api_key_required,omitempty"`
}
type ProfileModelParams struct {
BackendID *string `json:"backend_id,omitempty"`
Endpoint *string `json:"endpoint,omitempty"`
Model *string `json:"model,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
ServiceTier *string `json:"service_tier,omitempty"`
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
APIKeyEnv *string `json:"api_key_env,omitempty"`
ExtraParams *map[string]any `json:"extra_params,omitempty"`
}
func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat OutputFormat) ([]byte, error) {
if value == nil {
return nil, errors.New("prompt inspection is nil")
}
dto := PromptInspection{
PromptID: value.PromptID, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
OutputContract: OutputContract{Format: string(value.OutputContract.Format), ValidationMode: string(value.OutputContract.ValidationMode), SchemaPath: value.OutputContract.SchemaPath, RepairAttempts: value.OutputContract.RepairAttempts},
}
if value.Inputs != nil {
inputs := make([]PromptInspectionInput, len(*value.Inputs))
for i, input := range *value.Inputs {
inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
}
dto.Inputs = &inputs
}
switch outputFormat {
case OutputFormatJSON:
data, err := json.MarshalIndent(dto, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
case OutputFormatText:
var b bytes.Buffer
fmt.Fprintf(&b, "prompt_id: %s\nprompt_hash: %s\n", dto.PromptID, dto.PromptHash)
if dto.DefaultProfileID != nil {
fmt.Fprintf(&b, "default_profile_id: %s\n", *dto.DefaultProfileID)
}
if dto.Inputs != nil {
if len(*dto.Inputs) == 0 {
fmt.Fprintln(&b, "inputs: []")
} else {
fmt.Fprintln(&b, "inputs:")
for _, input := range *dto.Inputs {
fmt.Fprintf(&b, " - name: %s\n", input.Name)
if input.Required != nil {
fmt.Fprintf(&b, " required: %t\n", *input.Required)
}
if input.ContentType != nil {
fmt.Fprintf(&b, " content_type: %s\n", *input.ContentType)
}
if input.Description != nil {
fmt.Fprintf(&b, " description: %s\n", strconv.Quote(*input.Description))
}
}
}
}
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode)
if dto.OutputContract.SchemaPath != nil {
fmt.Fprintf(&b, " schema_path: %s\n", *dto.OutputContract.SchemaPath)
}
if dto.OutputContract.RepairAttempts != nil {
fmt.Fprintf(&b, " repair_attempts: %d\n", *dto.OutputContract.RepairAttempts)
}
return b.Bytes(), nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}
func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat OutputFormat) ([]byte, error) {
if value == nil {
return nil, errors.New("profile inspection is nil")
}
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, ProfileModelParams: ProfileModelParams{
BackendID: value.Backend, Endpoint: value.Endpoint, Model: value.Model, Temperature: value.Temperature,
MaxTokens: value.MaxTokens, TopP: value.TopP, TimeoutSeconds: value.TimeoutSeconds, ServiceTier: value.ServiceTier,
ReasoningEffort: value.ReasoningEffort, APIKeyEnv: value.APIKeyEnv, ExtraParams: value.ExtraParams,
}}
switch outputFormat {
case OutputFormatJSON:
data, err := json.MarshalIndent(dto, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
case OutputFormatText:
var b bytes.Buffer
fmt.Fprintf(&b, "profile_id: %s\nprofile_model_params:\n", dto.ProfileID)
p := dto.ProfileModelParams
if p.BackendID != nil {
fmt.Fprintf(&b, " backend_id: %s\n", *p.BackendID)
}
if p.Endpoint != nil {
fmt.Fprintf(&b, " endpoint: %s\n", *p.Endpoint)
}
if p.Model != nil {
fmt.Fprintf(&b, " model: %s\n", *p.Model)
}
if p.Temperature != nil {
fmt.Fprintf(&b, " temperature: %g\n", *p.Temperature)
}
if p.MaxTokens != nil {
fmt.Fprintf(&b, " max_tokens: %d\n", *p.MaxTokens)
}
if p.TopP != nil {
fmt.Fprintf(&b, " top_p: %g\n", *p.TopP)
}
if p.TimeoutSeconds != nil {
fmt.Fprintf(&b, " timeout_seconds: %d\n", *p.TimeoutSeconds)
}
if p.ServiceTier != nil {
fmt.Fprintf(&b, " service_tier: %s\n", *p.ServiceTier)
}
if p.ReasoningEffort != nil {
fmt.Fprintf(&b, " reasoning_effort: %s\n", *p.ReasoningEffort)
}
if p.APIKeyEnv != nil {
fmt.Fprintf(&b, " api_key_env: %s\n", *p.APIKeyEnv)
}
if p.ExtraParams != nil {
paramsJSON, err := json.Marshal(p.ExtraParams)
if err != nil {
return nil, err
}
fmt.Fprintf(&b, " extra_params: %s\n", paramsJSON)
}
if dto.APIKeyRequired != nil {
fmt.Fprintf(&b, "api_key_required: %t\n", *dto.APIKeyRequired)
}
return b.Bytes(), nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}

View File

@@ -1,95 +0,0 @@
package format
import (
"encoding/json"
"strings"
"testing"
"go.maximumdirect.net/promptkit"
)
func ptr[T any](value T) *T { return &value }
func TestPromptInspectionOptionalFieldsAndDisclosure(t *testing.T) {
inputs := []promptkit.PromptInputInspection{{Name: "first", Required: ptr(false), ContentType: ptr(""), Description: ptr("")}, {Name: "second"}}
value := &promptkit.PromptInspection{PromptID: "support/p", PromptHash: "hash", DefaultProfileID: ptr(""), Inputs: &inputs,
Messages: []promptkit.PromptMessageInspection{{Role: "user", Content: "private message"}}, Description: ptr("private description"), SessionID: ptr("private session"),
OutputContract: promptkit.PromptOutputInspection{Format: "text", ValidationMode: promptkit.ValidationNone, SchemaPath: ptr(""), RepairAttempts: ptr(0)}}
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatPromptInspection(value, format)
if err != nil {
t.Fatal(err)
}
s := string(output)
for _, forbidden := range []string{"private message", "private description", "private session", "prompt_version"} {
if strings.Contains(s, forbidden) {
t.Fatalf("inspection exposed %q: %s", forbidden, s)
}
}
if strings.Index(s, "first") > strings.Index(s, "second") {
t.Fatalf("input order changed: %s", s)
}
if !strings.Contains(s, "default_profile_id") || !strings.Contains(s, "repair_attempts") || !strings.Contains(s, "required") {
t.Fatalf("present empty values missing: %s", s)
}
}
value.DefaultProfileID, value.Inputs = nil, nil
value.OutputContract.SchemaPath, value.OutputContract.RepairAttempts = nil, nil
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatPromptInspection(value, format)
if err != nil {
t.Fatal(err)
}
for _, absent := range []string{"default_profile_id", "inputs", "schema_path", "repair_attempts"} {
if strings.Contains(string(output), absent) {
t.Fatalf("absent %s rendered: %s", absent, output)
}
}
}
empty := []promptkit.PromptInputInspection{}
value.Inputs = &empty
output, err := FormatPromptInspection(value, OutputFormatJSON)
if err != nil || !strings.Contains(string(output), `"inputs": []`) {
t.Fatalf("empty inputs lost: %s %v", output, err)
}
if _, err := FormatPromptInspection(nil, OutputFormatText); err == nil {
t.Fatal("nil prompt accepted")
}
}
func TestProfileInspectionPreservesOnlyPresentValues(t *testing.T) {
params := map[string]any{}
value := &promptkit.ProfileInspection{ProfileID: "support/fast", Backend: ptr("backend"), Model: ptr(""), Temperature: ptr(0.0), MaxTokens: ptr(0), APIKeyRequired: ptr(false), ExtraParams: &params}
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatProfileInspection(value, format)
if err != nil {
t.Fatal(err)
}
s := string(output)
for _, key := range []string{"profile_model_params", "backend_id", "model", "temperature", "max_tokens", "api_key_required", "extra_params"} {
if !strings.Contains(s, key) {
t.Fatalf("present %s missing: %s", key, s)
}
}
for _, key := range []string{"effective_model_params", "endpoint", "api_key_env", "top_p"} {
if strings.Contains(s, key) {
t.Fatalf("absent %s appeared: %s", key, s)
}
}
}
output, err := FormatProfileInspection(value, OutputFormatJSON)
if err != nil {
t.Fatal(err)
}
var decoded map[string]any
if err := json.Unmarshal(output, &decoded); err != nil {
t.Fatal(err)
}
p := decoded["profile_model_params"].(map[string]any)
if p["backend_id"] != "backend" || p["model"] != "" || p["temperature"] != float64(0) || decoded["api_key_required"] != false || len(p["extra_params"].(map[string]any)) != 0 {
t.Fatalf("wrong values: %#v", decoded)
}
if _, err := FormatProfileInspection(nil, OutputFormatText); err == nil {
t.Fatal("nil profile accepted")
}
}

View File

@@ -1,210 +0,0 @@
// Package format formats already-prepared public data for adapters.
package format
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"go.maximumdirect.net/promptkit"
)
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
// OutputFormat selects an application-owned textual or JSON representation.
type OutputFormat string
const (
OutputFormatText OutputFormat = "text"
OutputFormatJSON OutputFormat = "json"
DefaultOutputFormat OutputFormat = OutputFormatText
)
// PreparedRunOutputFormat is the output format for prepared render data.
type PreparedRunOutputFormat = OutputFormat
const (
PreparedRunFormatText = OutputFormatText
PreparedRunFormatJSON = OutputFormatJSON
DefaultPreparedRunOutputFormat = DefaultOutputFormat
)
// PreparedRunFormatter serializes a prepared run without performing use case work.
type PreparedRunFormatter interface {
Format(prepared *promptkit.PreparedRun) ([]byte, error)
}
// ParsePreparedRunOutputFormat parses a format name.
func ParsePreparedRunOutputFormat(raw string) (PreparedRunOutputFormat, error) {
return ParseOutputFormat(raw)
}
// ParseOutputFormat parses the shared inspection and prepared-run format names.
func ParseOutputFormat(raw string) (OutputFormat, error) {
switch OutputFormat(strings.ToLower(strings.TrimSpace(raw))) {
case "":
return DefaultOutputFormat, nil
case OutputFormatText:
return OutputFormatText, nil
case OutputFormatJSON:
return OutputFormatJSON, nil
default:
return "", fmt.Errorf("%w: %q (supported: %s, %s)", ErrUnknownPreparedRunFormat, raw, OutputFormatText, OutputFormatJSON)
}
}
// FormatterForPreparedRun returns a formatter strategy for the chosen format.
func FormatterForPreparedRun(outputFormat PreparedRunOutputFormat) (PreparedRunFormatter, error) {
switch outputFormat {
case PreparedRunFormatText:
return textPreparedRunFormatter{}, nil
case PreparedRunFormatJSON:
return jsonPreparedRunFormatter{}, nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}
// FormatPreparedRun formats a prepared run using the selected format.
func FormatPreparedRun(prepared *promptkit.PreparedRun, outputFormat PreparedRunOutputFormat) ([]byte, error) {
formatter, err := FormatterForPreparedRun(outputFormat)
if err != nil {
return nil, err
}
return formatter.Format(prepared)
}
// FormatPreparedRunByName parses a format name and formats a prepared run.
func FormatPreparedRunByName(prepared *promptkit.PreparedRun, rawFormat string) ([]byte, error) {
outputFormat, err := ParsePreparedRunOutputFormat(rawFormat)
if err != nil {
return nil, err
}
return FormatPreparedRun(prepared, outputFormat)
}
type jsonPreparedRunFormatter struct{}
func (jsonPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
if prepared == nil {
return nil, errors.New("prepared run is nil")
}
return json.MarshalIndent(prepared, "", " ")
}
type textPreparedRunFormatter struct{}
func (textPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
if prepared == nil {
return nil, errors.New("prepared run is nil")
}
var b bytes.Buffer
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
if prepared.SelectedBackendID != "" {
fmt.Fprintf(&b, "selected_backend_id: %s\n", prepared.SelectedBackendID)
}
if prepared.PromptHash != "" {
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
}
if prepared.SessionID != "" {
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
}
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
target := prepared.EffectiveModelParams
fmt.Fprintln(&b, "effective_model_params:")
fmt.Fprintf(&b, " endpoint: %s\n", target.Endpoint)
fmt.Fprintf(&b, " model: %s\n", target.Model)
fmt.Fprintf(&b, " temperature: %g\n", target.Temperature)
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
if target.ServiceTier != "" {
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
}
if target.ReasoningEffort != "" {
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
}
if target.APIKeyEnv != "" {
fmt.Fprintf(&b, " api_key_env: %s\n", target.APIKeyEnv)
}
if len(target.ExtraParams) > 0 {
fmt.Fprintln(&b, " extra_params:")
keys := make([]string, 0, len(target.ExtraParams))
for k := range target.ExtraParams {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
if err != nil {
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
}
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
}
}
fmt.Fprintln(&b, "input_hashes:")
inputKeys := make([]string, 0, len(prepared.InputHashes))
for k := range prepared.InputHashes {
inputKeys = append(inputKeys, k)
}
sort.Strings(inputKeys)
for _, k := range inputKeys {
fmt.Fprintf(&b, " %s: %s\n", k, prepared.InputHashes[k])
}
fmt.Fprintln(&b, "messages:")
roleOrder := make([]string, 0)
byRole := make(map[string][]promptkit.RenderedMessage)
for _, msg := range prepared.Messages {
if _, exists := byRole[msg.Role]; !exists {
roleOrder = append(roleOrder, msg.Role)
}
byRole[msg.Role] = append(byRole[msg.Role], msg)
}
for _, role := range roleOrder {
fmt.Fprintf(&b, " %s:\n", role)
messages := byRole[role]
for i, msg := range messages {
fmt.Fprintf(&b, " - message: %d\n", i+1)
if msg.CacheControl != nil {
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
if msg.CacheControl.TTL != "" {
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
}
fmt.Fprintln(&b)
}
fmt.Fprintln(&b, " content: |")
content := msg.Content
if content == "" {
fmt.Fprintln(&b, " ")
continue
}
for _, line := range strings.Split(content, "\n") {
fmt.Fprintf(&b, " %s\n", line)
}
}
}
return b.Bytes(), nil
}
func formatExtraParamTextValue(value any) (string, error) {
if s, ok := value.(string); ok {
return s, nil
}
b, err := json.Marshal(value)
if err != nil {
return "", err
}
return string(b), nil
}

View File

@@ -1,375 +0,0 @@
package format
import (
"encoding/json"
"errors"
"strings"
"testing"
"go.maximumdirect.net/promptkit"
)
func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if strings.Contains(s, "prompt_version") {
t.Fatalf("unexpected prompt version in text output: %s", s)
}
for _, want := range []string{
"prompt: prompt.id",
"selected_profile_id: local-fast",
"selected_backend_id: local",
"endpoint: http://llm/v1",
"model: gpt-test",
"temperature: 0.4",
"max_tokens: 256",
"top_p: 0.8",
"timeout_seconds: 45",
"service_tier: priority",
"reasoning_effort: medium",
"api_key_env: SCRIPTORIUM_API_KEY",
"prompt_hash: prompt-hash",
"rendered_prompt_hash: rendered-hash",
"glossary: hash-glossary",
"transcript: hash-transcript",
"messages:",
" system:",
" user:",
"System guidance.",
"Summarize the transcript.",
"Include key entities.",
"Second user message.",
} {
if !strings.Contains(s, want) {
t.Fatalf("expected text output to include %q, got:\n%s", want, s)
}
}
}
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
prepared := samplePreparedRun()
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"z_string": "enabled",
"b_number": 42,
"a_object": map[string]any{
"nested": "value",
"count": 2,
},
"c_array": []any{"first", 3, false},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
want := strings.Join([]string{
" extra_params:",
" a_object: {\"count\":2,\"nested\":\"value\"}",
" b_number: 42",
" c_array: [\"first\",3,false]",
" z_string: enabled",
}, "\n")
if !strings.Contains(s, want) {
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
}
}
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), secret) {
t.Fatalf("text output should not include resolved api key value: %s", out)
}
}
func TestTextFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
const directKey = "direct-format-key"
// PreparedRun intentionally has no field for direct API keys.
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), directKey) {
t.Fatalf("text output should not include direct api key value: %s", out)
}
}
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
t.Fatalf("expected system message cache control before content, got:\n%s", s)
}
if strings.Count(s, "cache_control:") != 1 {
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
}
}
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if !strings.Contains(string(out), "session_id: session-123\n") {
t.Fatalf("expected session_id in text output, got:\n%s", out)
}
}
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
},
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " cache_control: ephemeral\n") {
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
}
if strings.Contains(s, "ttl=") {
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
}
}
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"number": 42,
"nested": map[string]any{
"enabled": true,
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded map[string]any
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if decoded["prompt_id"] != "prompt.id" {
t.Fatalf("expected prompt_id in json output, got %#v", decoded["prompt_id"])
}
if _, ok := decoded["prompt_version"]; ok {
t.Fatalf("unexpected prompt_version in json output: %#v", decoded)
}
if decoded["selected_profile_id"] != "local-fast" {
t.Fatalf("expected selected_profile_id in json output, got %#v", decoded["selected_profile_id"])
}
if decoded["rendered_prompt_hash"] != "rendered-hash" {
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
}
if decoded["session_id"] != "session-123" {
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
}
modelParams, ok := decoded["effective_model_params"].(map[string]any)
if !ok {
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
}
extraParams, ok := modelParams["extra_params"].(map[string]any)
if !ok {
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
}
if extraParams["number"] != float64(42) {
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
}
nested, ok := extraParams["nested"].(map[string]any)
if !ok || nested["enabled"] != true {
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
}
if _, ok := decoded["input_hashes"]; !ok {
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
}
if _, ok := decoded["messages"]; !ok {
t.Fatalf("expected messages in json output, got %#v", decoded)
}
}
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(promptkit.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), secret) {
t.Fatalf("json output should not include resolved api key value: %s", out)
}
}
func TestJSONFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
const directKey = "direct-format-key"
// PreparedRun intentionally has no field for direct API keys.
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), directKey) {
t.Fatalf("json output should not include direct api key value: %s", out)
}
}
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
tests := []struct {
name string
input string
want PreparedRunOutputFormat
}{
{name: "default empty", input: "", want: DefaultPreparedRunOutputFormat},
{name: "text", input: "text", want: PreparedRunFormatText},
{name: "json", input: "json", want: PreparedRunFormatJSON},
{name: "trim and case", input: " JSON ", want: PreparedRunFormatJSON},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, err := ParsePreparedRunOutputFormat(tc.input)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if got != tc.want {
t.Fatalf("expected %q, got %q", tc.want, got)
}
})
}
}
func TestParsePreparedRunOutputFormatUnknownFails(t *testing.T) {
_, err := ParsePreparedRunOutputFormat("yaml")
if err == nil {
t.Fatal("expected error for unknown format")
}
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
}
}
func TestFormatPreparedRunByNameUnknownFailsClearly(t *testing.T) {
_, err := FormatPreparedRunByName(samplePreparedRun(), "yaml")
if err == nil {
t.Fatal("expected unknown format error")
}
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
}
}
func samplePreparedRun() *promptkit.PreparedRun {
return &promptkit.PreparedRun{
PromptID: "prompt.id",
PromptHash: "prompt-hash",
SelectedProfileID: "local-fast",
SelectedBackendID: "local",
EffectiveModelParams: promptkit.ExecutionTarget{
BackendID: "local",
Endpoint: "http://llm/v1",
Model: "gpt-test",
Temperature: 0.4,
MaxTokens: 256,
TopP: 0.8,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
},
InputHashes: map[string]string{
"transcript": "hash-transcript",
"glossary": "hash-glossary",
},
RenderedPromptHash: "rendered-hash",
Messages: []promptkit.RenderedMessage{
{Role: "system", Content: "System guidance."},
{Role: "user", Content: "Summarize the transcript.\nInclude key entities."},
{Role: "user", Content: "Second user message."},
},
}
}