Retire the legacy CLI and document native operator workflows

This commit is contained in:
2026-11-20 05:08:40 +00:00
parent 10387d4aea
commit bd7136ceeb
52 changed files with 846 additions and 7067 deletions

View File

@@ -5,6 +5,9 @@ an HTTP service. It prepares prompt requests, runs them against
OpenAI-compatible model endpoints, and returns generated output with validation OpenAI-compatible model endpoints, and returns generated output with validation
metadata. metadata.
CommandKit supplies native configuration, resource discovery, preview, reporting,
and completion interfaces alongside application generation and HTTP startup.
The application uses The application uses
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/) [PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/)
for prompt, profile, schema, preparation, generation, and validation behavior. for prompt, profile, schema, preparation, generation, and validation behavior.
@@ -16,11 +19,10 @@ behavior, output presentation, and HTTP artifact-containment policy.
From the repository root: From the repository root:
```bash ```bash
go run ./cmd/scriptorium render \ go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.yml \ --config ./examples/config.yml \
--prompt generic.markdown_summary \ --input-file transcript=./examples/fixtures/transcript.md \
--input transcript=./examples/fixtures/transcript.md \ --input-file glossary=./examples/fixtures/glossary.yml \
--input glossary=./examples/fixtures/glossary.yml \
--format json --format json
``` ```
@@ -38,6 +40,7 @@ a model. For complete invocation and output behavior, see the
- [Operations guide](docs/operations.md) - [Operations guide](docs/operations.md)
- [Consumer integration overview](docs/consumers/api.md) - [Consumer integration overview](docs/consumers/api.md)
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md) - [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
- [Native CLI migration](docs/consumers/upgrading-cli.md)
- [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md) - [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md)
- [Subprocess integration](docs/integrations/subprocess.md) - [Subprocess integration](docs/integrations/subprocess.md)
- [Architecture policy](docs/policy/architecture.md) - [Architecture policy](docs/policy/architecture.md)

View File

@@ -1,377 +1,186 @@
# CLI Reference # CLI Reference
This is the canonical contract for invoking Scriptorium. Configuration discovery, Scriptorium uses CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
precedence, application source locations, and server settings are defined in the This document owns application invocation and delivery behavior. The pinned
[configuration reference](config.md). The [HTTP API reference](api.md) owns [upstream CLI contract][native-cli] owns standard family grammar and reports.
service request and response behavior. [Configuration](config.md) owns discovery, settings, and source composition;
the [HTTP API](api.md) owns the service contract.
## Shortest Useful Command Existing callers should read the [CLI migration guide](consumers/upgrading-cli.md).
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
`render` prepares a request without calling an LLM.
## Commands ## Commands
- `scriptorium run`: prepare a prompt, call the configured LLM, and write the
generated artifact.
- `scriptorium render`: prepare a prompt and write prepared-run output.
- `scriptorium serve`: start the HTTP server.
- `scriptorium inspect prompt`: inspect one prompt definition without model execution.
- `scriptorium inspect profile`: inspect one resolved profile without model execution.
- `scriptorium inspect configuration`: validate or inspect Promptkit's captured framework configuration.
- `scriptorium config validate`: validate application configuration without building an engine.
- `scriptorium config show`: show the resolved application configuration.
- `scriptorium config sources`: show the ordered configuration sources.
- `scriptorium config explain`: explain one effective or historical configuration path.
All command families accept `--config <path>` and `--config-overlay <name>`;
positional arguments are rejected. `--config-overlay` selects one application
configuration overlay; it is separate from `--profile`, which selects a
Promptkit execution profile. See [configuration composition](config.md#explicit-imports-and-selected-overlays)
for name, root, and absent-main behavior. `run`, `render`, `serve`, `inspect prompt`,
`inspect profile`, and `inspect configuration` accept repeated
`--promptkit-source <path>` flags and `--schema-dir <dir>` where applicable.
They replace the configured source sequence in flag order. `--prompt-dir` and
`--profile-dir` are no longer supported; use `promptkit_sources` in application
configuration or the repeated source flags. Application `config` commands
remain independent of framework source loading. Explicit application flags,
including empty values, override file settings. Configuration load diagnostics
go to stderr and exit with status 1; successful load warnings are printed once
before command work.
## `scriptorium run`
```text
scriptorium run [flags]
```
Required flags:
| Flag | Meaning |
| --- | --- |
| `--prompt <id>` | Prompt ID to execute. |
Optional flags:
| Flag | Meaning |
| --- | --- |
| `--config <path>` | Application configuration file. |
| `--config-overlay <name>` | Selected application configuration overlay. |
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
| `--schema-dir <dir>` | Schema base-directory override. |
| `--profile <id>` | Execution-profile override. |
| `--session-id <id>` | Optional direct session identifier. |
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
| `--out <path>` | Write generated content to this file instead of stdout. |
| `--execution-report <path>` | Write an opt-in terminal execution report as newline-terminated JSON to a separate file. |
| `--llm-base-url <url>` | Runtime endpoint override. |
| `--model <name>` | Runtime model override. |
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
| `--temperature <float>` | Runtime temperature override. |
| `--max-tokens <int>` | Runtime maximum-token override. |
| `--top-p <float>` | Runtime top-p override. |
| `--reasoning-effort <value>` | Runtime reasoning-effort override. |
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
| `--run-timeout <duration>` | Optional whole-run engine deadline using Go duration syntax; zero or omission adds no deadline, negative values are invalid. |
| `--transient-retries <integer>` | Nonnegative additional transient attempt count for this request; `0` disables additional attempts. |
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
`--profile`.
Omitted numeric runtime flags preserve the selected effective value; explicit
zero values override it. `--timeout 0s` disables the per-attempt generation deadline
only; the caller context and configured transport cap remain active. CLI
durations are converted to whole seconds by truncation toward zero, so any
duration whose absolute value is below one second becomes an explicit
zero-second override. The timeout layers are defined in the
[Promptkit outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
With the built-in client, each transient retry gets a fresh generation timeout;
this flag does not limit the complete run.
`--run-timeout` keeps subsecond precision and starts immediately before engine
execution. It bounds preparation, capacity waits, retry waits, generation, and
repairs where they honor context. Parsing, engine construction, and output
writes are outside that interval. A failed run identified by PromptKit as an
operation timeout exits 1 with a safe deadline diagnostic; a completed invalid
output still exits 2. Synchronous work may not stop immediately.
Omitting `--transient-retries` inherits the application setting or Promptkit's
current default of two additional attempts. The flag overrides only this
request; it does not alter the configured engine. Structural repair generations
have separate transient retry budgets.
There is no raw API-key flag. Use `--api-key-env`.
Prompt and profile IDs are passed unchanged, including collection-qualified
IDs. The removed `--prompt-version` flag is rejected. `--input` is optional
at the CLI boundary: PromptKit decides whether the selected definition requires
declared inputs or template-referenced values.
`--session-id` supplies a direct, non-secret session identifier. A nonblank
value replaces a definition-rendered session ID; an omitted or blank value lets
the definition supply one. Promptkit trims direct values and limits them to 256
Unicode code points. Use stable, non-sensitive identifiers because effective
session IDs may appear in prepared output, run metadata, and provider-facing
requests.
`--reasoning-effort` is presence-aware: omitting it inherits the selected
profile value, a nonblank value replaces that value, and
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
nonblank values as provider-specific opaque strings.
### Execution report file
`--execution-report` requires a nonempty path. It is available only for `run`;
`-` is an ordinary filename. Parent directories are not created. An existing
file is replaced through ordinary file writing, without rollback or an atomic
replacement guarantee. Generated content still goes to stdout or `--out`.
Equal output/report paths, including existing paths that refer to the same
file, are rejected before execution.
The file is written after engine execution when Promptkit supplies a terminal
event, including for a failed run or invalid generated output. A failure before
invocation leaves the report path untouched. If no terminal event is supplied,
Scriptorium leaves the path untouched and emits a diagnostic; an existing file
may therefore contain an older report. Artifact output is attempted first on a
successful run, and report writing is attempted even if artifact writing fails.
Any output or report write failure exits 1; already-written output is not
undone. Without a write failure, the actual run keeps its usual exit status.
The [HTTP API reference](api.md#execution-report) defines the shared report
fields and usage-presence semantics. The file contains that JSON object followed
by a newline.
The report excludes content, credentials, URLs, headers, and raw errors, but
its identifiers can still be sensitive. The caller owns file retention and
access.
## `scriptorium render`
```text
scriptorium render [flags]
```
`--prompt <id>` is required. The following optional flags are supported:
`--config`, `--config-overlay`, repeatable `--promptkit-source`, `--schema-dir`, `--profile`,
`--input`, `--var`, `--out`, `--llm-base-url`,
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
`--reasoning-effort`, `--session-id`, `--timeout`, `--transient-retries`, and `--format text|json`. Their meanings match the corresponding
`run` flags; `--format` selects prepared-run output and otherwise uses
`defaults.render_format`.
The same deprecated aliases and numeric/timeout behavior as `run` apply.
The same session and reasoning inheritance, replacement, and clearing behavior
also applies.
`render` validates the retry override but does not generate model output or
include a retry field in prepared output.
`render` resolves profiles and schemas as part of preparation but
does not call an LLM.
To render from independently maintained collection checkouts:
```bash
go run ./cmd/scriptorium render --config ./examples/config.discovery.yml \
--prompt support/answer --profile support/fast \
--input transcript=./examples/fixtures/transcript.md
```
The [discovery example](../examples/discovery/framework.yml) also provides
`review/answer`; see [configuration discovery](config.md#collection-checkout-discovery).
## `scriptorium serve`
```text ```text
scriptorium run PROMPT_ID [flags]
scriptorium serve [flags] scriptorium serve [flags]
scriptorium config scopes
scriptorium config validate|show|sources|explain|diff [flags]
scriptorium promptkit list prompts|profiles|backends|namespaces|collections [flags]
scriptorium promptkit show prompt|profile|backend ID [flags]
scriptorium promptkit requirements show|validate PROMPT_ID [flags]
scriptorium promptkit show prepared PROMPT_ID [flags]
scriptorium promptkit test KIND [flags]
scriptorium help [COMMAND ...]
scriptorium version
scriptorium completion bash|zsh|fish
``` ```
Optional flags: Native help is also available with `--help`. Help, version, scope discovery,
and static completion do not load configuration or contact a provider.
Version/revision metadata is supplied by the build; ordinary builds identify
themselves as `dev` with no revision. Release build mechanics belong to the
[release procedure](release.md).
Commands that consume configuration accept `--config PATH` and `--overlay NAME`.
An explicitly empty config value remains explicit; overlay names follow
[ConfigKit selection](config.md#explicit-imports-and-selected-overlays).
There are no source/schema convenience flags or legacy command aliases.
Native shared `--format`, `--quiet`, `--verbose`, `--debug`, and progress
controls retain their upstream meanings. Diagnostics use stderr, not stdout.
Verbose exposes emitted Info and Warning messages without changing result bytes.
## Generation
`run` requires exactly one positional prompt ID. It has no duplicate prompt
flag or direct backend-selection flag; choose a profile for backend routing.
It uses these native bindings, with the preferred aliases below. Canonical
spellings prepend `llm-` (for example, `--llm-profile`).
| Preferred flags | Meaning |
| --- | --- |
| `--profile ID`, `--session-id ID` | Profile selection and optional non-secret session context. |
| `--var KEY=VALUE` | Repeatable caller-variable mappings. |
| `--input-text NAME=TEXT`, `--input NAME=TEXT` | Repeatable literal input mappings; `--llm-input` is also a literal-text alias. |
| `--input-file NAME=PATH` | Repeatable file references. |
| `--endpoint URL`, `--model NAME` | Native endpoint/model overrides. |
| `--temperature N`, `--max-tokens N`, `--top-p N` | Presence-aware numeric overrides, including zero. |
| `--timeout-seconds N` | Integer per-attempt generation timeout; zero disables that deadline, not caller cancellation or the transport cap. |
| `--reasoning-effort VALUE`, `--service-tier VALUE` | Native provider controls; an explicit empty reasoning value is preserved. |
| `--api-key-env NAME` | Credential environment-variable name, never raw key bytes. |
| `--extra-params JSON` | Native JSON-object provider parameters. |
| `--transient-retries N` | Request retry override; omission leaves the engine default. |
| `--prompt-namespace NAME`, `--profile-namespace NAME`, `--backend-namespace NAME` | Independent native namespace views. Qualified resource IDs remain exact; stored references are not rewritten. |
Mappings split at the first `=`; commas and subsequent equals signs are literal.
Repeat flags for multiple mappings. The last text/file occurrence for the same
input wins in command-line order. PromptKit validates selected definitions,
inputs, namespaces, provider settings, and credential requirements; Scriptorium
does not add an execution-readiness preflight or configurable default namespace.
Native string/pointer override semantics remain upstream-owned.
Application-owned generation controls:
| Flag | Meaning | | Flag | Meaning |
| --- | --- | | --- | --- |
| `--config <path>` | Application configuration file. | | `--run-timeout DURATION` | Go duration passed unchanged to PromptKit's operation budget. Positive values cover engine preparation/execution, including cooperative waits and repairs. Zero, omission, and negative values add no budget. Parsing, configuration/engine setup, and delivery are outside it. |
| `--config-overlay <name>` | Selected application configuration overlay. | | `--report PATH` | Additionally write the returned native result to a separate file. |
| `--addr <listen-address>` | HTTP listen-address override. | | `--report-format text\|yaml\|json` | Explicit separate-report format; omission leaves native automatic rendering selected. |
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
| `--schema-dir <dir>` | Schema base-directory override. |
| `--artifact-root <dir>` | Root for HTTP `file` input references. |
| `--max-request-bytes <n>` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
| `--max-artifact-bytes <n>` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
| `--max-response-bytes <n>` | Maximum encoded HTTP response bytes; `0` disables the limit. |
`serve` accepts no runtime model override flags. HTTP request fields, response The operation budget is independent of the integer per-attempt timeout.
schemas, and error codes are defined in the [HTTP API reference](api.md). Earlier parent cancellation/deadlines still apply. Cancellation is cooperative;
synchronous work that ignores context cannot be forcibly interrupted.
## `scriptorium inspect prompt` ### Results And Separate Reports
```text The main report goes through the invocation reporter to stdout. Automatic/text
scriptorium inspect prompt --prompt ID output is exactly `Artifact.Body`, with no added newline, header, escaping, or
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH] summary. JSON/YAML output is the full native PromptKit `RunResult`, without a
``` Scriptorium envelope. In native structured serialization, artifact bytes use
the upstream representation (JSON base64), not an application text projection.
`--prompt` is required. Inspection uses normal configuration discovery and the Save main output with shell redirection or the subprocess stdout destination.
selected Promptkit sources, defaults to text regardless of There is no `--output` or `--out` flag. Omit `--quiet` when capturing the main
`defaults.render_format`, and writes to stdout unless `--out` is supplied. result: native quiet suppresses it even when stdout is redirected.
Both definition-inspection commands first construct the same engine used for `--report` renders the same result through CommandKit without the artifact-only
execution from the complete selected source sequence. Construction resolves text template. Its format is independent of stdout's `--format` and the filename
and validates all effective definitions, including unused ones, and captures extension. Omission passes an unset format to the upstream renderer; it currently
referenced message-file contents and static template syntax. Invalid unused renders structured values as YAML in automatic/text mode. Choose an explicit
definitions or unreadable message resources can therefore prevent inspection. format when required rather than relying on that upstream default.
No prompt collection is mandatory, but selected prompts are not ignored.
Inspection does not prepare an execution request, load request-time schemas or The requested file is written even in quiet mode. Omission creates no separate
artifacts, resolve credential values, reserve backend capacity, or call a model. report. An explicitly empty path is a usage error before generation; `-` is an
Prompt inspection does not select an execution profile or render request-specific ordinary filename. New files use mode 0644 subject to umask, existing files use
templates. normal OS replacement behavior, and parent directories are not created.
Prompt output includes ID, hash, optional default profile and input metadata, Render, write, and close failures are reported.
and the output contract. Absent optional values are omitted; explicitly empty
or zero values are retained. Captured message text is not projected.
## `scriptorium inspect profile` Only a returned result is eligible for delivery. A resultless failure leaves an
existing report untouched; it does not synthesize an invocation-error result.
Main delivery is attempted first, then the requested report even if main
delivery fails. Writes are nontransactional; bytes already written remain.
Callers own redirected paths, retention, and keeping main and report destinations
separate. A shell may truncate its redirected file before the process starts.
```text ## Application Configuration Inspection
scriptorium inspect profile --profile ID
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
```
`--profile` is required. After the [full-source engine construction described The native `config` family defaults to `app`; select `--scope promptkit` for
above](#scriptorium-inspect-prompt), inspection selects one resolved profile captured framework configuration or use native scope discovery/all-scope modes.
with inheritance. `profile_model_params` contains only fields present on that `app` uses the actual completed ConfigKit result and does not construct an engine.
profile; it does not fill backend defaults. `api_key_required` is a separate
optional field. Explicit zero and empty values remain visible. Use `render`
for request-completed model settings.
## Application configuration inspection Native `--set POINTER=JSON` is enabled only for `app`. Records go directly to
ConfigKit: duplicate and ancestor/descendant-overlapping paths are rejected,
not resolved by flag order. No application-specific configuration flags or
custom `--against-*` counterparts are installed.
```text Native diff uses independent `--against-config`, `--against-overlay`, and
scriptorium config validate [flags] supported `--against-set` selections. Omitted right-side values do not inherit
scriptorium config show [flags] the left side. See the [upstream contract][native-cli] for focused paths,
scriptorium config sources [flags] metadata discovery, source IDs, findings, explanations, and detailed comparisons.
scriptorium config explain --path JSON_POINTER [flags]
```
Each accepts `--config`, `--config-overlay`, `--schema-dir`, `--addr`, ## Framework Configuration Inspection
`--artifact-root`, `--max-request-bytes`,
`--max-artifact-bytes`, and `--max-response-bytes`. These are application
configuration selectors and overrides; request, credential, backend-editing,
output-format, and output-file flags are not accepted. Positional arguments
are rejected.
`validate` writes no success output; status 0 means the application configuration The `promptkit` config scope loads application settings selected by `--config`
loaded and passed its own validation. It does not establish that selected and `--overlay`, constructs an engine, and exposes its captured native view.
Promptkit sources can construct an engine, credentials are present, or a It does not accept temporary overrides; native capability checks also apply
provider is available. `show` writes ConfigKit's resolved YAML snapshot. `sources` when an all-scope operation requests them.
writes one newline-terminated JSON array of source records in load order. Each
record has `id`, `label`, and `layer`, plus `path` when available. Application
main-file and import records use the `main` layer; explicit CLI
overrides appear as synthetic sources. Defaults-only input without overrides
produces `[]` for `sources`.
`explain` requires an explicitly supplied `--path` in JSON Pointer syntax. Framework effective/explanation views retain native classified-field redaction,
The empty string selects the root; `/server/addr` selects a field; `~1` and including history. Environment-reference names and ordinary values remain
`~0` escape `/` and `~` inside mapping keys. Numeric tokens are interpreted visible. This is not arbitrary secret detection and does not redact all prompt,
according to the containing mapping or sequence. Dotted paths, file paths, prepared, application, or generated content. Protect reports accordingly.
and URL fragments are not accepted. ConfigKit writes a human-readable account
of the effective value and its chronological default, file/import, overlay,
CLI, normalization, and removed-value history. This is not a JSON schema or a
byte-for-byte stable prose format. Unknown and malformed paths exit 1 without
successful explanation output.
Diagnostics and warnings go to stderr, never into successful stdout. Parse, ## Discovery, Preview, And Explicit Smoke Testing
load, output, and diagnostic-write failures exit 1. Ordinary output writes may
leave partial data on failure. Application inspection can contain paths and
configured values; protect its output accordingly. Raw API-key values are not
accepted by the application configuration schema.
## Framework configuration inspection Inventories retain native defaults and `--namespace`/`--all-namespaces` filters.
Named inspections expose native prompt messages and profile/backend values;
prepared preview exposes rendered content, not a summary projection.
```text Requirements discovery works before input artifacts exist and without schema
scriptorium inspect configuration validate [flags] loading, credential resolution, message rendering, or provider access. It still
scriptorium inspect configuration show [flags] loads configuration and captured message resources. Reports preserve exact names,
scriptorium inspect configuration sources [flags] metadata, variable presence (including empty values), completeness, and issue
scriptorium inspect configuration explain --path JSON_POINTER [flags] codes. Partial discovery may succeed with status zero; completeness is not
``` binding satisfaction or execution readiness. Concrete validation and preview
keep their native preparation-time prerequisites and make no generation call.
Each operation accepts `--config`, `--config-overlay`, repeatable Smoke-test kinds are `reachability`, `authentication`, `text_inference`,
`--promptkit-source`, and `--schema-dir`. These flags select the application `structured_output`, or `all`, with native aliases as documented upstream.
settings and framework sources; positional arguments, `--format`, and `--out` These explicit commands may contact providers and incur cost. Their presence
are not accepted. Unlike `config validate`, framework `validate` constructs does not add a startup/provider probe. Read the native findings: unsuccessful or
the Promptkit engine and validates all effective definitions, including unsupported checks are not automatically invocation failures.
unused ones. It writes nothing on success and does not prepare a request,
resolve a credential, or contact a provider.
`show` writes Promptkit's native effective YAML, an inspection snapshot rather ## Server And Statuses
than a promised reloadable file. `sources` writes a
newline-terminated JSON array in captured source order. Each source record has
`id`, `label`, `layer`, `kind`, and `underlay_index`, plus `path` when available
and `imported_from` for imported sources. That nested object has `source_id`,
`line`, and `column`. Values and output structure are a projection of the
captured Promptkit view, not Scriptorium's application-source records.
`explain` requires an explicitly supplied `--path` in ConfigKit JSON Pointer `serve` accepts configuration selection and presence-aware HTTP controls:
syntax; `--path ''` selects the root. For example, `--addr`, `--artifact-root`, `--max-request-bytes`, `--max-artifact-bytes`,
`/collections/support/profiles/0/model` addresses the first support profile's and `--max-response-bytes`. It has no generation request controls. Values and
model in the captured list. Escape `/` as `~1` and `~` as `~0`; dots in a key defaults belong to [configuration](config.md).
remain literal. Malformed and unknown pointers exit 1 without a successful
report. Explanation prose is native and is not a stable machine-readable
format.
Framework `show` and `explain` use Promptkit's native redaction for its Interrupt/SIGTERM cancels the invocation context. The server closes its listener
classified credential-shaped provider fields, including historical values. and active connections; canceled shutdown returns operational status, with no
Environment-variable reference names and ordinary provider parameters remain graceful-drain deadline or guarantee that context-ignoring work has finished.
visible. Arbitrary prompt or provider content is not classified as secret by
name inspection, so treat these reports as sensitive. This redaction policy is
limited to framework configuration reports; it does not add redaction to
`inspect profile` or prepared-run output. Credentials are not resolved by any
of these inspection operations. Promptkit owns detailed field and source
semantics; see its tagged [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
Warnings and errors go to stderr. Successful `validate` writes nothing to
stdout; successful `show`, `sources`, and `explain` write their reports there.
Parse, load, path, and output failures exit 1. A native writer error may leave
partial output.
## Input And Variable Syntax
`--input name=path` maps an input name to a local file; `--var name=value`
maps a template variable to a string. Both flags can be repeated or contain
comma-separated mappings. Values may contain `=` after the first separator.
Empty names and values are rejected.
CLI inputs are file references. HTTP inline inputs are defined by the
[HTTP API reference](api.md).
## Output And Exit Behavior
- `run` writes generated content to stdout, or to `--out` when supplied, and
writes a concise summary to stderr. The summary includes `backend=<id>` when
Promptkit selected a backend; endpoint-only profiles omit it.
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
without a success summary. Text output includes `selected_backend_id` after
`selected_profile_id` when Promptkit selected one; endpoint-only profiles
omit it.
- `serve` writes startup and server errors to stderr.
Exit statuses:
| Status | Meaning | | Status | Meaning |
| --- | --- | | --- | --- |
| `0` | Success. | | 0 | Native success, subject to successful presentation/delivery. |
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. | | 1 | Operational failure, including lookup/preparation/generation/capacity, cancellation/timeouts, unclassified load errors, and delivery failure for an otherwise successful run. |
| `2` | `run` generated and wrote output, but validation failed. | | 2 | Native usage/declaration-input errors, or configuration errors classified as invalid input by native diagnostic evidence. |
| 3 | Native `config diff --check` found differences. |
| 4 | Completed generation returned `ValidationFailed`; artifact and native result remain deliverable. |
A backend admission rejection is a runtime error and prints `run error: model Native finalization handles presentation errors: success/usage become operational;
backend capacity is exhausted`. The HTTP capacity response is defined in the an already nonzero domain outcome retains its classification/status. A report-file
[HTTP API reference](api.md). failure preserves completed-validation status 4 while adding its explanation and
cause. An operational failure to perform validation is status 1, not status 4.
JSON selection does not turn failures into a separate JSON error envelope.
## Workflows And Examples [native-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md
The [maintained render script](../examples/render-markdown-summary.sh) is a
copyable render workflow. The [HTTP request example](../examples/http-run.json)
is for a running `serve` process.

View File

@@ -1,87 +1,126 @@
# Configuration Reference # Configuration Reference
This is the canonical reference for Scriptorium application settings. Prompt, This document owns Scriptorium's application settings, discovery, and source
profile, backend, schema, execution-setting, built-in profile, and framework selection. [CLI syntax](cli.md) and the [HTTP API](api.md) have separate owners.
credential semantics are defined by the Framework definitions and execution semantics belong to the pinned
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md). [PromptKit v0.23.3 formats][formats]; ConfigKit v1.2.1 owns
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and [native composition and inspection][configkit].
outcomes, see the [HTTP API reference](api.md). For migration examples, see
[Upgrading Promptkit sources and definitions](consumers/upgrading-promptkit.md).
## Discovery And Precedence ## Discovery And Precedence
ConfigKit assembles the selected main source, imports, and overlay, resolves An explicit `--config` selects that file, including an explicitly empty value
inheritance, and decodes the result with Scriptorium's typed defaults. It then (which fails rather than falling back). With no selection, ConfigKit searches
applies explicitly supplied application CLI values, records application these native presets in order, selecting one main file:
normalization, and validates the result. Source, composition, inheritance, and
decoding failures happen before CLI assignment and cannot be repaired by it;
an override can correct a decoded value before application validation.
When `--config` is omitted, Scriptorium searches | Preset | Directories in order | Filename groups |
`/usr/local/etc/scriptorium/config.yml` and then `/etc/scriptorium/config.yml`. | --- | --- | --- |
ConfigKit performs this ordered search for each load. If neither exists, it | WorkingDirectory | Current working directory | Named only |
uses built-in defaults; an error in a selected candidate does not cause a | UserConfigDirectory | Go user-config directory / `scriptorium`, then the user-config directory | Named then Generic in the application directory; Named only in the base |
fallback. An explicit `--config` path must exist and decode successfully, | HomeDirectory | Go home directory / `.scriptorium`, then the home directory | Named then Generic in the application directory; Named only in the base |
including when an empty path is explicitly supplied. | UnixSystemDirectories | `/usr/local/etc/scriptorium`, `/etc/scriptorium`, then `/etc` | Named then Generic in application directories; Named only in `/etc` |
The maintained [minimal configuration](../examples/config.yml), Named means `scriptorium.yml`, then `scriptorium.yaml`. Generic means
[complete configuration](../examples/config.full.yml), `config.yml`, `config.yaml`, `main.yml`, then `main.yaml`.
[composed configuration](../examples/config.composed.yml), ConfigKit uses native Go directory resolution (including its user-config/home
[collection configuration](../examples/config.collections.yml), and environment behavior) and checks every filename in one directory before the next.
[framework inheritance configuration](../examples/config.inheritance.yml) are Candidates are not configuration layers. Optional exhaustion uses application
copyable examples. defaults; a selected-file acquisition/decoding error is terminal.
## Application Configuration File ConfigKit assembles the main, enabled imports, and selected overlay, applies its
inheritance/composition rules, creates typed defaults and strictly decodes, applies
explicit override records, then runs application normalization and validation.
An override can correct a decoded value before validation, but cannot repair
source/composition/decoding errors. The completed result is authoritative:
runtime settings conversion does not merge, default, or normalize again.
Application configuration is decoded by ConfigKit v1.2.1 as strict YAML. ## Application Settings
Unknown fields, empty files, multiple YAML documents, and YAML aliases/anchors
are rejected.
An explicit empty string is a value and may clear a file-provided optional
string. A scalar `null` is not an empty string and is rejected for non-nullable
fields; nullable fields such as `transient_retries` retain their documented
meaning. Raw API-key fields and unknown application fields are rejected.
Only explicitly supplied application CLI flags override decoded settings. Paths Complete copyable configurations are maintained under
in the application document remain relative to the process working directory, not to the file's [examples](../examples/config.yml), including [full settings](../examples/config.full.yml),
directory. Load diagnostics are printed safely on stderr; invalid application [application imports](../examples/config.composed.yml),
configuration exits with status 1. [framework fragment composition](../examples/config.collections.yml),
[collection checkout discovery](../examples/config.discovery.yml), and
### Promptkit source selectors [framework inheritance](../examples/config.inheritance.yml).
`promptkit_sources` is an ordered sequence of explicit Promptkit source
selectors. Each item must set exactly one of `file` or `directory`:
```yaml ```yaml
promptkit_sources: promptkit:
- directory: ./examples/prompts configuration: ./examples/promptkit.yml
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas schema_dir: ./examples/schemas
``` ```
An empty or null final resolved `promptkit_sources` list selects no The `promptkit` section contains acquisition settings, not embedded framework
application-supplied framework sources; PromptKit's bundled OpenRouter underlay definitions. Profiles, prompts, backends, and collections belong in the separate
still applies. A later sequence item overlays an earlier framework document. The retired `promptkit_sources`, top-level `schema_dir`,
one. Files within a selected directory are composed together, so conflicting and `defaults.render_format` fields have no compatibility conversion and are
definitions in that group fail instead of using filename order as precedence. rejected by strict decoding, as are unknown/raw credential fields.
An explicit file may use ConfigKit imports. The complete framework format and
source rules belong to the [PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
Selector paths are relative to the process working directory, not to the | Field | Default | Meaning |
application configuration file. Application validation checks the selector | --- | --- | --- |
shape but does not open or validate Promptkit sources. `config validate` | `promptkit.configuration` | empty | One external framework file; omission/empty selects no external main. |
therefore does not prove that selected sources can build an engine; use | `promptkit.schema_dir` | `.` | Schema base directory passed to PromptKit. |
`inspect configuration validate` for that separate check. Repeated | `transient_retries` | upstream default | Optional nonnegative engine retry count. Omitted/null delegates upstream; zero disables additional transient attempts. |
`--promptkit-source` flags replace the entire configured selector sequence | `server.addr` | `:8080` | HTTP listen address. |
and preserve flag order. Without an authored collection policy, a selected | `server.artifact_root` | empty | Root enabling HTTP file-artifact references. |
overlay replaces `promptkit_sources` as a whole; a declared sequence policy | `server.max_request_bytes` | `16777216` | Encoded HTTP request limit; zero is unlimited. |
may instead combine the lists. | `server.max_artifact_bytes` | `16777216` | HTTP file-artifact limit; zero is unlimited. |
| `server.max_response_bytes` | `16777216` | Encoded HTTP response limit; zero is unlimited. |
### Collection checkout discovery Size fields are nonnegative signed 64-bit values. [HTTP behavior](api.md) defines
their enforcement; [operations](operations.md) covers safe deployment choices.
Select a framework discovery file using the existing file selector, as in ConfigKit strictly decodes application YAML: empty files, unknown fields,
[config.discovery.yml](../examples/config.discovery.yml). Its native imports multiple documents, aliases/anchors, and incompatible/null values fail under its
select both supported collection-entry spellings: schema rules. Explicit empty strings and numeric zeros remain values. The
nullable retry field preserves null. Application paths are trimmed/cleaned and
the server address is trimmed within ConfigKit's resolver, with native
normalization evidence. Paths remain relative to process cwd, not the application
file's directory. No general environment-variable override layer is installed.
## Explicit Imports And Selected Overlays
Application imports use native root `config.imports` and are enabled during
loading. ConfigKit owns paths/directories/globs, requiredness, destination/item
placement, inheritance, sequence policy, and source-root confinement; consult its
[pinned contract][configkit]. Scriptorium does not widen the selected file's
canonical source root or implement another traversal/merge policy.
`--overlay NAME` passes the name unchanged to ConfigKit. At the selected release,
empty and exact lowercase `default` select none; other names use letters,
digits, underscores, or hyphens and select `overlays/NAME.yml` beneath the main
source root. A missing selected overlay fails. It is not a filesystem-path flag,
inherited stack, or fallback-root mechanism. An active overlay after absent-main
search exhaustion has no independent filesystem root in this application.
Only `serve` has named application override flags, for its HTTP settings.
Native application-scope `--set` records are passed directly to ConfigKit.
Nonconflicting records are supported; duplicate and ancestor/descendant paths
are rejected rather than resolved by order. Configuration commands have no
application-defined source/schema/HTTP selectors or comparison counterparts.
Native comparison selections and supported overrides are independent on each side.
## Framework Source Mapping
A nonempty resolved `promptkit.configuration` becomes an import-enabled
`configkit.File` main input to PromptKit. An empty value leaves its main absent.
The native bundled OpenRouter underlay remains enabled; no application source
list, directory expansion, or local precedence interpreter is installed.
Imports in that external document compose framework definitions using PromptKit's
schema and ConfigKit's native rules. The [maintained entry file](../examples/promptkit.yml)
imports the application's maintained prompt/profile fragments and backend
definitions. Peer conflicts are errors, not filename-order precedence.
Native whole-record replacement and inheritance remain framework-owned.
Application settings do not embed or re-decode framework definitions.
PromptKit captures message-file resources with their defining source context
when constructing the engine. Schema/artifact reads, credential resolution, and
provider work retain native preparation/runtime lifecycles. The server keeps one
engine until restarted; later CLI invocations perform fresh loads.
### Collection Checkout Discovery
The [discovery configuration](../examples/config.discovery.yml) selects
[one framework entry](../examples/discovery/framework.yml) with two optional imports:
```yaml ```yaml
config: config:
@@ -90,195 +129,30 @@ config:
- glob: collections/*/collection.yaml - glob: collections/*/collection.yaml
``` ```
Both globs are optional, so either spelling may have no matches. Paths are Each independently maintained checkout declares its namespace and explicitly
relative to the framework file; this example's `*` selects one checkout imports its own definitions relative to its entry file. Checkout names do not
level and is not a general layout restriction. Each entry declares its own determine resource IDs. The [support example](../examples/discovery/collections/support-repo/collection.yml)
collection namespace and imports only its intended resource fragments. The imports multiple profiles/prompts and a source-relative message; the
checkout directory name does not determine the namespace. New checkouts are [review example](../examples/discovery/collections/review-repo/collection.yaml)
available on the next invocation without editing this file; a running server uses the other spelling. Arbitrary checkout YAML is not loaded implicitly.
does not reload them. See the [maintained collection examples](../examples/discovery/collections/).
Entries imported by one discovery file are composed together as peers. Either glob may have no matches. New checkouts become visible on a later
Conflicting definitions of the same qualified resource ID fail to load; invocation, not through server hot reload. Entries in one discovery document are
filename, checkout, and glob order do not set precedence. To override a peers: conflicting qualified IDs fail instead of relying on glob order.
discovered record, select a separate framework file later in
`promptkit_sources` and supply the complete replacement record.
### Explicit imports and selected overlays ## Inspection And Credentials
Application files declare imports under root `config.imports`. Native `path`, The native `app` scope exposes the actual application result without acquiring
`dir`, and `glob` selectors, filters, recursion, destinations, and item framework definitions or credentials. The `promptkit` scope exposes the captured
placement are available as described in the [pinned ConfigKit contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md). framework result selected by the application config/overlay. They are not merged
Imports are optional by default; set `required: true` when absence must fail. into one synthetic document. See [CLI inspection](cli.md#application-configuration-inspection)
The old `optional` field is not supported. Directory selection has no default for operations and independent comparison.
filename filter; author a pattern when only YAML files are intended. ConfigKit
composes peer sources strictly, rather than using filename order to resolve
conflicts. Imports are resolved relative to their declaring file within the
native source root. Scriptorium does not widen that root.
Root `config.sequences` can declare an identity key and merge mode for a PromptKit owns profiles, namespaces, provider controls, repairs, retries, and
collection. Scriptorium does not register or infer an identity for credential resolution. Store credential environment-variable names, not raw
`promptkit_sources`: absent an authored policy, unkeyed lists replace across values; supply the selected environment to the process. Native framework
precedence boundaries. The [collection example](../examples/config.collections.yml) redaction is limited to classified fields and history, not arbitrary sensitive
declares `file` identity with `upsert`, combines imported file selectors, and content. Protect application, prompt, prepared, and generated reports as data.
lets an overlay add another. CLI list overrides still replace the whole list.
At application precedence boundaries, omission preserves lower-precedence [formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md
selectors. Under ordinary list replacement, `[]` clears the selection and [configkit]: https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md
`null` resolves to no selectors. Under authored `append` or `upsert`, `[]`
contributes no records and `null` is invalid. To clear a collection with an
inherited policy, declare `mode: replace` at
`destination: [promptkit_sources]` in the higher-precedence source and supply
`promptkit_sources: []`. See
[ConfigKit's collection policy contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#yaml-declarations-and-policy-lifetime)
for policy lifetime and combination rules.
These application root `config` controls do not change the separate
PromptKit v0.23.3 framework format: its selected framework files use the
[Promptkit contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
and their own native ConfigKit `config` controls.
`--config-overlay NAME` selects one file at `overlays/NAME.yml` below that
root. The name must use letters, digits, `_`, or `-`; it is not a path. Empty
and exact lowercase `default` select no overlay. There is no fallback,
case folding, inherited stack, or automatic selection; a missing selected
overlay fails. Mappings merge; sequence contributions and nulls at collection
destinations follow the effective collection policy described above. Other
explicit nulls replace lower values subject to the application schema.
Defaults-only loading can enable imports, but an absent main has no import
declarations. This CLI supplies no independent overlay root, so an active
overlay cannot be located without a selected main context. That is not a general
ConfigKit restriction.
The [composed example](../examples/config.composed.yml) imports the application
[source-selection fragment](../examples/config-fragments/sources.yml) and has
a selectable [local overlay](../examples/overlays/local.yml). That fragment
selects the [framework backend file](../examples/framework/backends.yml) through
`promptkit_sources`; backend definitions are not imported into the application
document. From the repository root:
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.composed.yml --config-overlay local \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
Inspect the collection overlay and render without contacting a provider:
```bash
go run ./cmd/scriptorium config show --config ./examples/config.collections.yml --config-overlay collection
go run ./cmd/scriptorium render --config ./examples/config.collections.yml --config-overlay collection \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml
```
Import and overlay file locations are constrained by ConfigKit; application
path values such as `promptkit_sources`, `schema_dir`, and `server.artifact_root`
remain relative to the process working directory even when authored in a
fragment or overlay. These source constraints are separate from Promptkit
message-resource paths and HTTP artifact containment.
| Field | Default | Meaning |
| --- | --- | --- |
| `promptkit_sources` | empty sequence | Ordered file/directory sources for Promptkit backends, profiles, and prompts. |
| `schema_dir` | `.` | Promptkit schema source directory for relative schema paths. |
| `server.addr` | `:8080` | Address used by `serve`. |
| `server.artifact_root` | unset | Root that enables HTTP `file` input references. |
| `server.max_request_bytes` | `16777216` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
| `transient_retries` | Promptkit default | Optional nonnegative count of additional built-in transient generation attempts. Omitted or `null` delegates to Promptkit; `0` disables additional attempts. |
The size fields must be zero or greater. The [HTTP API](api.md) defines how
each limit is enforced and reported. `server.artifact_root` configures an HTTP
deployment boundary; see [operations](operations.md) for deployment handling.
The `config validate`, `config show`, `config sources`, and `config explain`
commands inspect this application configuration without creating a Promptkit
engine. `show` is a resolved inspection snapshot, not necessarily a reloadable
file. Application validation does not check framework sources, backend
connectivity, or credential availability. Use `inspect configuration` for
Promptkit's separately captured framework configuration; its four operations
are documented in the [CLI reference](cli.md#framework-configuration-inspection).
`config explain --path /server/addr` uses JSON Pointer syntax to show the
selected field's current value and history from defaults, files, overlays,
explicit CLI overrides, and normalization. `--path ''` selects the root.
History includes replaced and removed values; `show` presents the final
resolved snapshot. The application schema rejects raw credential fields, but
do not treat inspection as a general-purpose filter for sensitive strings.
## Framework Source Mapping
Scriptorium assembles selected Promptkit source files and directories in the
order described above, then constructs the Promptkit engine. Framework
definitions and custom backends belong in those selected source files, not in
the Scriptorium application configuration. Message `content_file` paths use
the defining source's resource context and are captured when the engine is
constructed; schema and request-artifact loading remain preparation-time
operations. Scriptorium does not redefine or independently parse framework
file formats.
- Prompt selection by ID, message templates, inputs, output contracts, and
session IDs are Promptkit contracts.
- Profile fields, numeric ranges, execution defaults, overlay precedence,
built-in profiles, and credential rules are Promptkit contracts.
- Schema path behavior and generated-content validation are Promptkit
contracts.
See the
[tagged Promptkit format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
for all of those definitions. The files under
[`examples/prompts`](../examples/prompts/),
[`examples/profiles`](../examples/profiles/), and
[`examples/schemas`](../examples/schemas/) are maintained Scriptorium
application inputs using that tagged format.
PromptKit supplies a default-on OpenRouter bundle as the lowest-priority
underlay. Choose an actual bundled profile such as `auto`, or a complete
application-owned profile; old catalog IDs are not aliases. Caller definitions
can replace bundled records by normal composition. Bundle availability is not
a readiness check or a promise of a fixed provider model.
## Credentials And Outbound Behavior
Scriptorium maps `--api-key-env` and HTTP `model.api_key_env` into Promptkit
request overrides. Keep secret values in environment variables and store only
their names in configuration or framework source files. Do not place raw keys
in configuration, prompts, profiles, CLI arguments, examples, or HTTP
payloads.
An optional `api_key_env` whose environment value is absent or empty can result
in an unauthenticated provider request. A profile that declares credentials
required still fails as an invalid request when no credential source is
selected, and fails with `ErrAPIKeyEnvMissing` when its selected environment
source is absent or empty. Scriptorium never reads or emits the environment
value itself.
Unset optional provider controls are omitted from compatible provider requests.
A positive Promptkit `repair_attempts` budget can add provider calls, latency,
token use, and cost; see the tagged format reference for its permitted values
and validation-mode requirements.
Promptkit's built-in client makes up to two additional attempts for eligible
transient HTTP statuses by default. The application `transient_retries` setting
changes that budget for its constructed engine; an explicit `run` request
override takes precedence. This can increase latency, request count, and cost.
Each initial or repair generation has its own transient budget; an injected
client retains its own retry behavior. The tagged integration contract owns
eligibility and timing.
Promptkit's
[OpenAI-compatible integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
defines outbound authentication, provider request mapping, transport limits,
and timeout layering.
## Related References
- [CLI reference](cli.md)
- [HTTP API reference](api.md)
- [Operations guide](operations.md)
- [Promptkit framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)

View File

@@ -11,6 +11,7 @@ and the HTTP service according to the boundary your application needs.
- CLI subprocess: [subprocess integration](../integrations/subprocess.md) - CLI subprocess: [subprocess integration](../integrations/subprocess.md)
- HTTP service: [HTTP API reference](../api.md) - HTTP service: [HTTP API reference](../api.md)
- Application configuration: [configuration reference](../config.md) - Application configuration: [configuration reference](../config.md)
- Existing CLI callers: [native CLI migration](upgrading-cli.md)
Go applications that need an in-process prompt framework should import Go applications that need an in-process prompt framework should import
`go.maximumdirect.net/promptkit` directly. The tagged `go.maximumdirect.net/promptkit` directly. The tagged
@@ -27,7 +28,7 @@ Consumers are responsible for:
- supplying required inputs and template variables; - supplying required inputs and template variables;
- supplying credentials through the chosen interface; - supplying credentials through the chosen interface;
- protecting rendered prompts and generated artifacts as potentially - protecting rendered prompts and generated artifacts as potentially
sensitive; sensitive, including native inspection and full result reports;
- deciding whether validation-failed output is usable; and - deciding whether validation-failed output is usable; and
- retrying only when another model call is acceptable. - retrying only when another model call is acceptable.

View File

@@ -0,0 +1,95 @@
# Migrate To The Native CLI
This guide covers Scriptorium's breaking CommandKit-native CLI cutover. It does
not select a release version or change historical release notes. The
[CLI reference](../cli.md) owns current syntax, streams, and statuses;
[configuration](../config.md) owns source discovery and settings.
The earlier [PromptKit upgrade](upgrading-promptkit.md) covers framework formats,
not a compatibility layer for this interface.
## Replace Commands And Selections
| Former interface | Current interface |
| --- | --- |
| `run --prompt ID` / `--prompt-id` | `run ID`; exactly one positional prompt ID. |
| `render --prompt ID` | `promptkit show prepared ID`. |
| `inspect prompt --prompt ID` | `promptkit show prompt ID`. |
| `inspect profile --profile ID` | `promptkit show profile ID`. |
| `inspect configuration ...` | Native `config ... --scope promptkit`. |
| Application `config ...` | Native `config ... --scope app` (the default scope), including diff. |
| `--config-overlay` | `--overlay NAME`; a native overlay name, not an arbitrary path. |
| `--input NAME=PATH` | `--input-file NAME=PATH`; `--input` now means literal text. |
| `--llm-base-url` | `--endpoint` (canonical `--llm-endpoint`). |
| Duration-valued `--timeout` | Integer `--timeout-seconds` for each generation attempt. |
| `--out` / output-file flags | Caller stdout redirection. |
| `--execution-report` | `--report PATH` with native result rendering. |
Prefer native short aliases such as `--profile`, `--var`, `--input-text`, and
`--input-file`; canonical `--llm-*` spellings remain available.
There is no duplicate prompt flag, direct backend-ID override, or retained
`--profile-id` alias on generation.
Repeat mappings rather than comma-separating them. The first equals sign splits
each mapping; commas and later equals signs are literal. Input text/file
occurrences share native last-occurrence precedence.
## Migrate Application Configuration
Replace the source-selector list and top-level schema directory with
`promptkit.configuration` and `promptkit.schema_dir`.
The [maintained example](../../examples/config.yml) selects a separate framework
entry file that imports prompt/profile/backend definitions through native
ConfigKit composition. No source/schema CLI flags or compatibility conversion
remain. Do not embed framework definitions inside the application section.
Discovery now uses native cwd, user-config, home, then Unix-system presets and
their filename conventions. It is no longer limited to the two former
`config.yml` paths. Set an explicit config path when migrating unattended jobs.
Overlays remain native name selections. Application-scope native `--set` records
retain conflict rejection; order does not make duplicate paths authoritative.
There are no custom source/schema/HTTP selectors under `config`.
## Capture Artifacts And Reports
For an artifact file plus full native report, from the repository root:
```sh
scriptorium run generic.markdown_summary --config examples/config.yml \
--input-file transcript=examples/fixtures/transcript.md \
--input-file glossary=examples/fixtures/glossary.yml \
--report run-result.json --report-format json > summary.md
```
This generation example contacts the configured provider. Use native prepared
preview instead when only checking inputs/settings.
Main automatic/text output is the artifact alone, with no added newline.
Main JSON/YAML is the native result; artifact bytes use native structured
serialization. Separate reports contain that full native result, not the old
observer-derived metadata shape. `--report-format` independently chooses
text/YAML/JSON; omission leaves the format unset for native automatic rendering
(currently YAML for structured values). Extensions do not select formats.
Omit quiet to capture main output, including redirected stdout. Quiet still
allows an explicitly requested report file. Reports are available only when a
result is returned; resultless failures do not replace an existing report.
Partial delivery can leave one or both destinations written. Keep paths separate
and check status/diagnostics before treating files as fresh.
## Adjust Status And Sensitivity Handling
Completed generated-content validation failure is now domain status 4, not the
old status 2. Native usage/invalid-configuration input is 2, diff-check
differences are 3, and operational failures are 1. Presentation/file failures
can change success into failure while preserving an already nonzero domain
classification. The [CLI status table](../cli.md#server-and-statuses) owns details.
Native prompt inspection includes messages; prepared reports include rendered
content; full run reports include generated content and metadata. Framework
classified-field redaction does not make arbitrary report data safe to expose.
Requirements discovery is separate from validation/preparation: it can run
before artifacts exist, and a partial report may succeed. Read its native
completeness/issue fields. Smoke tests are explicit provider operations, not
startup checks. See [subprocess guidance](../integrations/subprocess.md) for
stream capture, cancellation, and caller responsibilities.

View File

@@ -1,33 +1,23 @@
# Upgrade PromptKit Sources And Definitions # Upgrade PromptKit Sources And Definitions
This guide covers the CLI and framework configuration cutover to This guide covers the earlier framework configuration cutover to
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md) [PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md)
and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
for the full upstream contract. for the full upstream contract. The subsequent [native CLI migration](upgrading-cli.md)
owns command, reporting, application-discovery, and source-setting changes.
Removing version selectors and changing framework definition shapes are Removing version selectors and changing framework definition shapes are
intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility). intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility).
## Application configuration ## Application configuration
Existing application fields that already select the desired sources retain Framework definition migration and application acquisition are separate changes.
their meaning. For example, this application configuration works both before Rewrite older framework files as below, then select their entry file using the
and after the cutover: current [application configuration](../config.md#application-settings).
The old source-selector list and top-level schema root are not current aliases;
```yaml follow the [CLI migration guide](upgrading-cli.md#migrate-application-configuration).
promptkit_sources: Scriptorium does not provide a legacy definition reader or embedded-definition
- directory: ./examples/prompts extraction/reload path.
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas
```
The selected framework files must still be rewritten as described below.
Scriptorium uses one native application schema: discovery, imports, selected
overlays, source order, explicit overrides, and working-directory-relative
selector paths remain application configuration behavior. It loads framework
sources separately with PromptKit's schema. No second application schema or
legacy definition reader is provided. See the [configuration reference](../config.md).
## Rewrite framework definitions ## Rewrite framework definitions
@@ -82,9 +72,9 @@ Convert framework source controls from `configkit:` to `config:` and add
header, if present, is `version: 2`. Use sibling inheritance such as header, if present, is `version: 2`. Use sibling inheritance such as
`extends: {id: base}`. `extends: {id: base}`.
Across ordered `promptkit_sources` entries, a later framework underlay replaces The single external framework entry uses native ConfigKit import composition;
the whole record for the same ID. Supply the complete intended replacement, there is no application-managed source-list precedence. Native record replacement
or create a distinct inherited record. Checkouts imported by the same and inheritance remain upstream-owned. Checkouts imported by the same
[discovery file](../config.md#collection-checkout-discovery) are peers: [discovery file](../config.md#collection-checkout-discovery) are peers:
conflicting definitions of the same ID fail instead of overriding one another. conflicting definitions of the same ID fail instead of overriding one another.
@@ -106,5 +96,5 @@ records can replace it through ordinary composition.
For multiple independently maintained repositories, see the For multiple independently maintained repositories, see the
[collection-discovery example](../../examples/config.discovery.yml). Its [collection-discovery example](../../examples/config.discovery.yml). Its
entry files declare their own namespaces; checkout names do not become IDs. entry files declare their own namespaces; checkout names do not become IDs.
The [CLI reference](../cli.md) demonstrates qualified rendering and the The [CLI reference](../cli.md) defines qualified selection and prepared preview; the
[operations guide](../operations.md) describes when updates take effect. [operations guide](../operations.md) describes when updates take effect.

View File

@@ -5,6 +5,8 @@ application that consumes the public
[PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md); [PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md);
framework implementation work belongs in Promptkit. Application configuration framework implementation work belongs in Promptkit. Application configuration
uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local. uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local.
Operator interfaces use CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0;
standard command families and presentation remain upstream-owned.
PromptKit uses the same ConfigKit release to load framework sources under its PromptKit uses the same ConfigKit release to load framework sources under its
own separate schema and format contract. own separate schema and format contract.
@@ -46,8 +48,8 @@ Before recommending, planning, or implementing work:
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) | | CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) |
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) | | HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests | | Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests |
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` | | Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and the [native CLI contract](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md) |
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#promptkit-source-selectors), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) | | Promptkit source selection or captured framework inspection | [Configuration contract](config.md#framework-source-mapping), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
| Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) | | Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) |
| OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) | | OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) |
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) | | Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |

View File

@@ -1,64 +1,67 @@
# Subprocess Integration # Subprocess Integration
This document covers process-boundary behavior for callers that invoke Use the executable when a local process boundary is appropriate.
Scriptorium as a child process. Command syntax, flags, output, and exit codes The [CLI contract](../cli.md) owns exact command/flag/status syntax; the
are defined by the [CLI reference](../cli.md). Interface selection belongs in [consumer overview](../consumers/api.md) compares subprocess and HTTP use.
the [consumer integration overview](../consumers/api.md).
## Process Contract ## Artifact-First Execution And Inspection
Use `scriptorium render` when the caller needs prepared output without a model Select a stored prompt and capture its generated artifact from stdout. Use native
call, and `scriptorium run` for generation. Pass an explicit `--config` or requirements discovery before generated input files exist, concrete validation
make the configuration search paths available to the child process; configuration once bindings exist, and prepared preview to render without a model call.
discovery, fields, profile selection, and credential mechanisms are defined in The [CLI migration guide](../consumers/upgrading-cli.md) explains replacements
the [configuration reference](../config.md). for older command and file-output conventions.
Pass required API-key environment variables through the child environment. Do Pass an explicit application configuration or deliberately arrange the child's
not place raw API keys in arguments. Keep the environment limited to the values [configuration discovery](../config.md). Supply only the environment needed for
needed for the selected profile. selected credentials; never put raw keys in arguments.
For application-only preflight, invoke `scriptorium config validate`; it exits Discovery completeness is distinct from binding satisfaction and execution
0 with empty stdout when the application configuration loads, but does not readiness. Partial reports can succeed. Consumers own their binding graph and
prove that Promptkit sources or credentials are usable. `config show` emits policy for incomplete findings; Scriptorium supplies no workflow state.
resolved YAML and `config sources` emits newline-terminated JSON provenance.
All three use the same source selection and explicit overrides as execution
commands and require separate stderr capture for diagnostics. Their output can
include source paths, identifiers, credential reference names, and ordinary
backend parameters; treat it as potentially sensitive.
## Streams And Output Ownership ## Streams, Reports, And Availability
Capture stdout and stderr separately. Stdout contains the requested artifact or Capture stdout and stderr separately. Diagnostics are not a machine-readable
prepared output unless the caller selects an output file; stderr contains error protocol, even when structured result output is selected. Prefer exit
summaries, diagnostics, and server messages. The exact destinations and status classification and native report fields; do not parse diagnostic prose.
meanings are part of the [CLI reference](../cli.md), not a stable stderr data
protocol.
When using `--out`, the caller owns the output path, its permissions, and The default generated result is artifact content. Native structured results and
cleanup. Treat rendered prompts, generated artifacts, stdout, and stderr as optional separate reports carry the full returned result, including content and
potentially sensitive. metadata. They are not the former metadata-only CLI observation report. Native
prompt/prepared inspection also carries content. Treat all captured streams and
reports as potentially sensitive.
For `run`, an optional [execution report file](../cli.md) contains terminal To save main output, supply a file as the subprocess stdout destination or use
metadata separate from generated stdout or `--out`. The caller owns that shell redirection. Omit quiet when that main result is needed: quiet suppresses
path's permissions and retention. Output and report files are not a it even for redirected stdout. A separately requested report remains an
transaction: one may be written when the other fails, and a preexisting report application file effect independent of quiet and main format selection.
may remain when the invocation has no terminal event. Check the exit status
and diagnostics rather than assuming a report path contains this run's data. A completed content-validation failure can still deliver an artifact/result;
consumers decide whether it is usable. An operational failure with no result
produces diagnostics and a nonzero status, not a synthesized result/error
envelope. An existing separate report may then describe an older invocation.
## Partial Delivery And Caller Responsibility
Main output is attempted before a requested report, and report delivery is
attempted even after main presentation failure. Either destination may contain
partial data after an error. File delivery is not transactional and has no
rollback or atomic-replacement promise.
Callers own redirected file creation/truncation, permissions, retention, and
keeping destinations separate. A shell may truncate its file before the process
starts, unlike a separate report that is opened only after a result exists.
Check the finalized exit status and diagnostic stream; do not infer success or
freshness from a file's existence.
## Cancellation And Recovery ## Cancellation And Recovery
A CLI invocation performs one synchronous request and creates no durable run Interrupt/SIGTERM cancels the child's invocation context. Engine cancellation is
state. A supervising process that needs cancellation must terminate the child cooperative. The [whole-operation budget](../cli.md#generation) is separate from
process according to its own process-management policy. A later invocation is a per-attempt provider timeouts and excludes process/configuration setup and
new request and can make another model call; there is no resume or checkpoint delivery. Supervisors still own forced process termination when required.
protocol.
For `run`, callers may set the [whole-run deadline](../cli.md) to bound the A new invocation is a new request that may incur another provider call/cost.
engine invocation. It is separate from the per-attempt generation timeout and There is no checkpoint, archive, conversation, or resume protocol. Deployment and
does not include child startup or output writing. A typed operation deadline sensitive-data handling belong to [operations](../operations.md).
failure exits 1.
Supervisors still own process termination when they need to stop the child
outside that engine interval.
For deployment, filesystem permissions, and sensitive-artifact handling, see
the [operations guide](../operations.md).

View File

@@ -1,157 +1,89 @@
# Adapter Internals # Adapter Internals
## Purpose Adapters map executable inputs and effects through public upstream values.
The [CLI](../cli.md), [HTTP API](../api.md), and [architecture policy](../policy/architecture.md)
own their external contracts and boundaries.
Scriptorium adapters translate executable inputs into Promptkit public requests ## CLI Composition And Acquisition
and translate Promptkit results or errors back to CLI or HTTP behavior. They
own IO and presentation mechanics, not framework decisions.
External contracts are canonical in the [CLI reference](../cli.md) and `app.go` composes application run/serve declarations with native configuration,
[HTTP API reference](../api.md). Promptkit's public engine contract is PromptKit, completion, help, and version support. CommandKit owns parsing,
described by its tagged declaration-input checks, reporting selection, presentation, and finalization.
[Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md).
## Components And Collaborators `selection.go` maps explicit native config/overlay choices and server-only HTTP
bindings into `internal/config.Load`. It preserves supplied empty/zero values.
Native application `--set` records remain upstream records with native conflict
handling. No application selectors are installed under configuration commands.
- `cmd/scriptorium` passes process arguments and streams to `native_config.go` adapts the actual application result through
`internal/adapter/cli`. `ckconfig.FromConfigKit` and framework inspection through `ckprompt.ConfigScope`.
- `internal/adapter/cli` resolves settings through `internal/config`, Native views own effective values, paths, metadata, sources, findings, explanations,
constructs `promptkit.Engine`, maps CLI values to `promptkit.RunRequest`, redaction, and comparison. Application inspection does not construct an engine.
exposes application and captured framework configuration inspection, and Comparison loads each side from that side's selection; there is no config/engine
owns output files, summaries, and exit codes. cache or effective-document reconstruction.
- `internal/config` delegates application YAML decoding, default resolution,
overrides, and validation ordering to ConfigKit v1.2.1. The CLI supplies only
explicitly visited application flags as overrides and renders safe structured
load diagnostics on stderr. PromptKit uses the same ConfigKit release to
load framework sources under its own definition and format contract. Both
domains use ConfigKit's native diagnostic writer.
- `internal/adapter/http` strictly decodes request DTOs, maps them to Promptkit
public values, calls its adapter-owned `Runner` interface, and maps results
and errors to HTTP DTOs.
- `internal/format` renders `promptkit.PreparedRun` values as deterministic text
or JSON.
- `internal/report` maps Promptkit's terminal execution event to a small
application-owned value without invoking the engine or writing output.
## Wiring Flows `acquisitionState` contains application diagnostic evidence only, never loaded
configuration or engines. Callback decorators forward it through native
`Acquired` channels. An application LoadError is transported separately while
the framework adapter runs, then its original typed cause is restored before
native outcome classification. This prevents app findings from becoming
framework findings. Native framework evidence remains unchanged; comparison
application messages identify left/right context. Callback marks prevent replay
when operations share invocation state.
### CLI `native_prompt.go` installs all twelve ckprompt callbacks, nil request builders,
and empty-default native namespace views. Inventories, named resources,
requirements, validation, preparation, and smoke testing retain their upstream
lifecycles and reports.
`run` calls `promptkit.Engine.Run`; `render` calls ## Generation And Delivery
`promptkit.Engine.Prepare`. Both share request mapping for exact prompt ID
and profile selection, optional file inputs, variables, and presence-aware
execution overrides. Omitted framework settings remain zero values so Promptkit
resolves its own defaults and definition-required inputs.
The application retry setting enters shared engine construction, while a
run/render flag enters only the mapped request. The server uses the same engine
setting and retains one backend admission state. Render prepares without
generation.
`inspect prompt` maps the selected ID to `native_run.go` binds the required operand and reusable resource/input/execution
`promptkit.Engine.InspectPrompt`, then formats Scriptorium-owned inspection flags into a native RunRequest. Ordinary typed namespace flags feed
data. `inspect profile` maps an explicit profile ID to `Engine.InspectProfile` `Engine.WithNamespaces`; stored references are not rewritten. One `Engine.Run`
and formats Scriptorium-owned resolved profile data without backend completion. call receives the invocation context and unchanged operation duration. PromptKit
owns preparation, credentials, admission, retries, repairs, and content validation.
Both paths construct the same full-source engine as execution before inspecting `outcomes.go` maps typed failures and native configuration evidence to native
the chosen definition; selected prompts are not filtered out. They do not outcomes, preserving causes while keeping provider payloads out of terminal
perform request preparation or generation. The [CLI definition-inspection explanations. A returned completed validation failure remains a domain outcome;
contract](../cli.md#scriptorium-inspect-prompt) owns the loading and runtime an unexpected nil successful result is an integration failure.
boundaries, including the distinction between an optional prompt collection
and validating any prompts that were selected.
`inspect configuration` builds the same engine from the selected native `delivery.go` submits the actual RunResult to the invocation reporter with a
Promptkit source sequence, reads its captured configuration view, and delegates reusable artifact-byte text template. The optional file renders that same value
effective output and explanation to Promptkit's native writers. The adapter without the template through CommandKit's writer API, outside reporter callbacks
projects captured source records into its small JSON DTO. Its validation to avoid reentrancy. File failures add explanations/causes while reporter
operation constructs the engine without preparing a request. Framework source presentation failures remain native. See [delivery semantics](../cli.md#results-and-separate-reports).
semantics and report redaction are owned by Promptkit; the [CLI
contract](../cli.md#framework-configuration-inspection) documents the exposed
operations and their deliberately limited redaction scope.
The `config` family loads the same application ConfigKit result once and uses ## Server And HTTP
its validation, effective-output, source, and explanation views directly. It
does not construct a Promptkit engine. The CLI maps source records into a small
JSON DTO for `sources`; `show` and `explain` use ConfigKit's own writers.
`serve` constructs Scriptorium's restricted HTTP artifact reader, injects it `server.go` loads application settings, injects the restricted artifact reader
with `promptkit.WithArtifactReader`, passes the engine through the HTTP into one engine, installs the existing HTTP handler/limits, and opens a listener.
adapter's consumer-owned `Runner` interface, and starts the server. It retains the header timeout and engine retry defaults. Context cancellation
calls Server.Close; callback coordination finishes connection/listener closing
before return, without waiting for arbitrary context-ignoring application work.
The command paths assemble the engine from application-selected framework The HTTP handler strictly decodes DTOs and applies transport policy before
source files and directories plus the schema directory and retry setting. mapping to PromptKit requests. Its Runner interface is a consumer-side external
Individual files become import-enabled ConfigKit inputs; directory selections boundary using native values. Request controls do not mutate shared engine
are expanded through Promptkit's directory-input helper. Scriptorium supplies defaults; shared backend capacity accounting remains in that engine.
those ordered underlays through Promptkit's public configuration API rather
than parsing or mapping framework definitions itself. The server constructs
one engine, which retains the framework's backend registry and associated
capacity state across requests.
### HTTP HTTP optional reports use one per-request `internal/report.Collector`, attached
only for opted-in requests. It copies terminal aggregates, not event history,
The handler enforces transport limits and strict JSON decoding before mapping and is never the CLI result report. Success/error attachment and encoded-size
DTOs into `promptkit.RunRequest`, `promptkit.ArtifactRef`, and handling retain the [HTTP contract](../api.md#execution-report). Reader sentinel
`promptkit.ExecutionTargetOverride`. On success it reads Promptkit artifact, identities survive PromptKit wrapping for status mapping.
validation, model, usage, and metadata values directly.
The optional HTTP retry count maps to `RunRequest.TransientRetries` without
mutating the shared engine. Negative values are rejected before invocation;
Promptkit controls retries and backend admission for concurrent requests.
CLI and HTTP operation budgets map to PromptKit's `OperationTimeout`; HTTP
passes the caller context through unchanged. The public sentinels determine
timeout classification.
Failure mapping uses `errors.Is` against Promptkit's public sentinels and the
HTTP reader's Scriptorium-owned containment and size errors. Wrapped reader
errors preserve their identity through Promptkit's artifact-load boundary.
The report collector is per invocation. It ignores nonterminal events and
copies terminal aggregates once, including usage presence and completeness.
It retains no history or shared state; adapters decide whether to attach it and
how to present its snapshot.
CLI `run` attaches one only when a report path is selected, writes generated
output first on success, and then writes the terminal snapshot when present.
Its report path remains untouched for pre-invocation failures or absent
terminal observations; report and artifact writes are independent.
HTTP `POST /v1/runs` attaches a separate collector to each invocation only when
`include_execution_report` is true. Its optional top-level `execution` field
appears in success or error envelopes only when Promptkit emits a terminal
event. Pre-invocation adapter errors and absent terminal events leave it out.
The complete serialized success or report-bearing error response is subject to
the configured response-size limit; overflow returns the compact 413 error
without a report. Field meanings are owned by the [API contract](../api.md#execution-report).
## Package-Local Guarantees
- Adapters contain no copied framework types or orchestration.
- Configuration is resolved before Promptkit engine construction.
- Explicit numeric overrides preserve presence, including zero.
- HTTP DTO and error mapping remains stable and transport-owned.
- Resolved secrets are not serialized or printed.
- No adapter creates durable run state.
## Verification ## Verification
Inspect: Native composition/configuration/prompt/run/delivery/lifecycle tests protect the
application's integration choices using real libraries and controlled local
providers. Checkout-discovery tests cover fresh invocations and entry-file
spellings. Entrypoint tests protect identity/status mapping; HTTP reader/DTO/
report/capacity regressions and the architecture guard remain their own owners.
Do not duplicate upstream parsers, analyzers, serializers, or exhaustive format
matrices to change an adapter.
- `internal/adapter/cli/run_test.go` When changing behavior, update its canonical external contract and focused
- `internal/adapter/http/handler_test.go` integration protection. Required shared capabilities belong upstream before
- `internal/adapter/http/artifact_reader_test.go` tagged adoption, not in a resurrected application framework facade.
- `internal/format/prepared_run_test.go`
- `internal/adapter/dependency_test.go`
The adapter tests protect parsing, configuration mapping, output, status
mapping, restricted artifacts, and representative real Promptkit-engine
workflows. The dependency test protects the repository boundary.
## Change Recipes
For a CLI or HTTP change:
1. identify the Scriptorium-owned external contract;
2. map through Promptkit public values without copying framework semantics;
3. add or update the narrow application-owned test;
4. update the canonical Scriptorium contract; and
5. coordinate and tag Promptkit first if a required public capability is
genuinely absent.
Update [source internals](sources.md) when application source locations or HTTP
artifact containment changes.

View File

@@ -1,19 +1,18 @@
# Internal Component Overview # Internal Component Overview
This is the complete inventory of Scriptorium's implemented Go components. This inventory describes implemented Go components. The
The [architecture policy](../policy/architecture.md) owns normative boundaries; [architecture policy](../policy/architecture.md) owns normative boundaries;
public behavior belongs in the linked contracts. public behavior belongs in the linked contracts.
| Component | Implemented responsibility | References | | Component | Implemented responsibility | References |
| --- | --- | --- | | --- | --- | --- |
| `cmd/scriptorium` | Process entrypoint that delegates arguments and streams to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) | | `cmd/scriptorium` | Build identity, signal-aware context, native status-to-process exit. | [CLI](../cli.md), [adapters](adapters.md) |
| `internal/adapter/cli` | Parses execution, application-configuration, and captured framework-configuration inspection commands; resolves settings; constructs Promptkit engines for execution, definition inspection, and framework inspection; maps requests; and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md), [source internals](sources.md) | | `internal/adapter/cli` | CommandKit composition, ckconfig/ckprompt acquisition, application loading, native request mapping, run/report effects, and HTTP process lifecycle. | [CLI](../cli.md), [adapters](adapters.md), [sources](sources.md) |
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) | | `internal/adapter/http` | Routes, strict DTO decoding, transport limits, public request/result/error mapping, and restricted artifact reader. | [HTTP API](../api.md), [adapters](adapters.md), [sources](sources.md) |
| `internal/config` | Selects application sources through ConfigKit v1.2.1, supplies the application definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md), [source internals](sources.md) | | `internal/config` | ConfigKit application schema, discovery/selection, defaults/resolution/validation callbacks, and completed-value conversion. | [Configuration](../config.md), [sources](sources.md) |
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) | | `internal/defaults` | Application and HTTP default constants. | [Configuration](../config.md) |
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output; framework-configuration output uses Promptkit's native captured view and writers. | [CLI contract](../cli.md), [adapter internals](adapters.md) | | `internal/report` | HTTP terminal-observation metadata projection. It is not used for CLI native result reports. | [HTTP execution report](../api.md#execution-report), [adapters](adapters.md) |
| `internal/report` | Projects one terminal Promptkit observation into Scriptorium-owned execution metadata. | [adapter internals](adapters.md) |
Framework implementation packages are provided by CommandKit, ckconfig, ckprompt, ConfigKit, and PromptKit are released upstream
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) dependencies, not local framework packages. There is no custom CLI formatter
and are not part of this repository. package or compatibility dispatch surface.

View File

@@ -1,117 +1,55 @@
# Source Internals # Source Internals
## Purpose This document covers application acquisition and the restricted HTTP artifact
reader. [Configuration](../config.md) owns field/discovery/composition contracts;
the [PromptKit formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
own framework resources and execution semantics.
This document covers Scriptorium-owned source locations and the restricted HTTP ## Application Boundary And Engine Construction
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
owned by the tagged
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
## Application Source Locations `internal/config.Load` selects an explicit File or the ordered native SearchConfig
presets and performs one ConfigKit load with imports, overlay selection, and
explicit overrides. It returns the actual Result with native inspection evidence.
Definition callbacks supply application defaults, record actual path/address
normalizations through ConfigKit, and validate retry/HTTP-size domains. Settings
conversion consumes completed values without another merge or normalization pass.
`internal/config` resolves the ordered `promptkit_sources` sequence and The typed application PromptKit section selects acquisition only. In
`schema_dir` from application configuration and explicit CLI overrides. It `engine.go`, a nonempty resolved configuration path becomes an import-enabled
owns selector-shape validation, not framework file parsing. During engine main Input; an empty path leaves Main nil. The schema base and retry default are
construction, `internal/adapter/cli` turns file selections into import-enabled passed directly, retaining the bundled underlay. There is no source-list parser,
ConfigKit inputs and expands directory selections with directory classification/expansion, or one-underlay-per-selector algorithm.
`promptkit.DirectoryInputs`, then passes the ordered groups as Promptkit
configuration underlays, one per selector. Directory inputs retain the
helper's strict-composition grouping; no selected source becomes an implicit
framework main. The shared Promptkit engine receives the selected schema
directory and retry setting, and the default OpenRouter bundle remains enabled.
Promptkit uses ConfigKit to load and compose those framework inputs according Framework imports resolve within the native source context. Message resources
to Promptkit's own schema and format. Scriptorium does not parse or validate are captured from their defining contexts at engine construction. Schema and
Promptkit definitions. Promptkit owns definition selection and validation, request-artifact acquisition retain their preparation-time behavior.
profile built-ins and overlays, schema resolution, ordinary file artifacts, Independent collection checkouts are discovered by native file import globs,
captured framework configuration inspection, and related error identities. not application traversal. New invocations reload normally; a server retains
its single engine until restart.
The [configuration reference](../config.md) owns Scriptorium's source-location Application-only inspection uses its completed result without framework
fields and precedence. Maintained files under `examples/` are application acquisition. Framework inspection exposes the native captured view separately.
inputs that use Promptkit's tagged formats. Diagnostic decoration in the [CLI adapter](adapters.md) carries application
messages without combining evidence trees or effective documents. There is no
## Application Configuration Boundary per-invocation engine/config cache or cross-invocation snapshot guarantee.
`internal/config` contains the application-owned ConfigKit-native `Document`
definition. Its fresh defaults come from Scriptorium's existing default
constants; its resolver records only actual leaf-level trimming, path cleaning, and
render-format normalization. The validator reports selector-shape, retry,
size-limit, and format errors without mutating the document or exposing
rejected values. Application path values keep working-directory-relative
meaning. ConfigKit v1.2.1 owns strict application YAML decoding and presence
semantics. Framework configuration is separately loaded by PromptKit using
the same ConfigKit release under PromptKit's own schema and source contract.
The `Load` boundary selects an explicit file or constructs a native ConfigKit
search over the ordered implicit paths. Discovery exhaustion uses defaults;
a selected-file error stops loading. One ConfigKit load enables imports,
applies explicitly supplied application overrides after decoding and before
resolution and validation, and returns the document and its frozen inspection
records. `Settings` copies selectors in their resolved order into runtime
settings; it does not read files or apply another merge. ConfigKit uses the selected file's
canonical directory as the source root; the application does not widen it.
Application path values retain process-working-directory-relative meaning.
Runtime conversion consumes the resolved application values through the
existing `Settings` path. Promptkit receives selected framework sources only
after application configuration has loaded and validated; its source merge,
definition validation, and captured inspection view remain framework-owned.
Every configuration-loading command uses this boundary. The CLI maps only
explicitly visited application flags to ConfigKit overrides and prints
structured load diagnostics before engine construction. Application and
framework load errors and warnings use their respective native writers. The old application
YAML decoder and merge path have been removed.
Application `config` inspection reuses the result's frozen effective,
provenance, and history views without constructing an engine or reloading
sources. It does not resolve credentials. `inspect configuration` separately
constructs an engine and exposes Promptkit's captured framework view; only its
framework `show` and `explain` reports receive Promptkit's native,
credential-shaped redaction. Neither inspection family is a general-purpose
filter for arbitrary sensitive values.
## Restricted HTTP Artifact Reader ## Restricted HTTP Artifact Reader
`internal/adapter/http` implements `promptkit.ArtifactReader` for HTTP The HTTP adapter implements `promptkit.ArtifactReader`; server construction
requests. The `serve` path injects it with injects it through the public option. Ordinary CLI artifact loading remains
`promptkit.WithArtifactReader`, replacing Promptkit's ordinary reader for upstream-owned.
inbound HTTP inputs.
The reader: The reader accepts inline content without a root. File references need a root;
relative paths resolve below it and absolute paths must be lexically contained.
Traversal outside is rejected and the configured file byte limit applies, with
zero unlimited. Reads preserve native artifact metadata and honor cancellation.
- accepts inline references without an artifact root; Containment does not resolve symlinks. The OS may follow a symlink inside the
- denies file references when no root is configured; root to an outside target; this is not a filesystem sandbox or a defense against
- resolves relative paths below the configured root; privileged concurrent filesystem modification. Reader errors retain identity
- accepts absolute paths only when they are lexically within that root; through upstream artifact-load wrapping, enabling the existing HTTP mapping.
- rejects lexical traversal outside the root;
- applies the configured file byte limit, with zero meaning unlimited;
- preserves content type, body, size, hash, name, and URI metadata; and
- honors context cancellation.
Containment is lexical and does not resolve symlinks. The operating system The [HTTP API](../api.md) owns outcomes and [operations](../operations.md) owns
follows symlinks after the check. The [HTTP API](../api.md) owns observable safe root/permission selection. Configuration, CLI acquisition/engine, and HTTP
request outcomes, and [operations](../operations.md) owns safe deployment reader/handler tests own their respective boundaries; upstream owns exhaustive
permissions and root selection. source traversal, format parsing, ordinary artifacts, and inspection semantics.
Reader errors remain identifiable after Promptkit wraps them as artifact-load
failures, allowing the HTTP adapter to preserve Scriptorium status and error
codes.
## Verification And Change Recipe
Inspect:
- `internal/config/document_test.go`
- `internal/config/load_test.go`
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/artifact_reader_test.go`
- `internal/adapter/http/handler_test.go`
When changing an application source location or HTTP artifact policy:
1. preserve strict configuration precedence and the Promptkit public boundary;
2. keep containment and size policy in Scriptorium;
3. update focused configuration, reader, and handler tests;
4. update the [configuration](../config.md), [HTTP](../api.md), and
[operations](../operations.md) contracts as applicable; and
5. do not duplicate Promptkit loaders, formats, or ordinary artifact behavior.

View File

@@ -1,243 +1,120 @@
# Operations Guide # Operations Guide
## Scope And References This runbook covers deployment, normal operation, capacity, sensitive data, and
recovery. Exact [CLI syntax/statuses](cli.md), [configuration](config.md), and
[HTTP wire behavior](api.md) remain in their canonical references.
See the [CLI migration guide](consumers/upgrading-cli.md) for changed jobs and
the [framework upgrade guide](consumers/upgrading-promptkit.md) for definitions.
This runbook covers deployment, normal operation, capacity planning, and safe ## Deploy Sources And Process
recovery for Scriptorium. It does not redefine invocation syntax, configuration
fields, or HTTP wire behavior.
- [CLI reference](cli.md): commands, output destinations, and exit codes. Provide readable application configuration, its selected framework entry file,
- [Configuration reference](config.md): application settings, source and explicitly imported definitions/resources. Set cwd deliberately: application
locations, defaults, and credential mapping. path values are cwd-relative, while framework imports and message resources keep
- [Promptkit source upgrade guide](consumers/upgrading-promptkit.md): concise their native source contexts. ConfigKit confines source imports/overlays to the
source-selector and definition migration examples. selected main's canonical root; Scriptorium does not widen it.
- [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md):
prompt, profile, schema, execution-setting, and framework credential
contracts.
- [HTTP API reference](api.md): route, request/response schema, status codes,
limits, and HTTP artifact access.
- [Consumer integration overview](consumers/api.md): caller responsibilities.
## Operational Model And State Engine construction validates selected effective definitions and captures message
resources/template syntax, including definitions not used by a particular request.
Schema/artifact reads, credential resolution, and model work retain their native
preparation/runtime lifecycles. A server constructs one engine at startup and
does not hot-reload changed files; restart to capture updates. Separate CLI
invocations load afresh rather than sharing an immutable snapshot.
Scriptorium handles one prompt request for each CLI invocation or HTTP request. For independent collections, install checkouts beneath the layout selected by
It has no durable run store, archive, checkpoint, cache, or resume mechanism. the [discovery entry](../examples/discovery/framework.yml). Each checkout declares
A failed or interrupted request is recovered by correcting its inputs, its namespace and explicit fragment imports; arbitrary YAML is not a framework
configuration, or environment and submitting a new request. source. The maintained [discovery example](../examples/config.discovery.yml)
supports both entry-file spellings and multiple definitions. New checkouts or
updates become visible to later invocations or a restarted server.
Generated artifacts, rendered prompts, model output, and run metadata are Supply only needed credential environment variables through a supervisor or
caller-owned data. Retention, encryption, backup, and deletion are deployment secret-management mechanism. Confirm presence without printing their values.
responsibilities. Raw keys do not belong in arguments or checked-in configuration.
Optional [CLI execution reports](cli.md) also belong to the caller. They can
contain sensitive identifiers even though they omit prompt and response bodies.
Keep report paths separate from generated output and apply the same access and
retention controls. Output and report writes are not transactional; an older
report may remain when no terminal observation was produced.
HTTP callers may also request a [terminal execution report](api.md#execution-report).
It shares the same identifier sensitivity and should receive response access
and retention controls. An opted-in report increases the encoded response size;
an oversized success or report-bearing error becomes the compact 413 response
without a report.
## Deploy The Filesystem And Process ## Normal Operator Workflow
Provide the process with readable selected Promptkit source files/directories 1. Validate application configuration through the native application scope.
and schema sources that follow the tagged framework formats. At engine This does not acquire framework sources or prove provider readiness.
construction, PromptKit v0.23.3 validates every selected effective definition, 2. Inspect framework configuration/resources and discover requirements for the
including unused definitions, and captures referenced message-file contents selected prompt/profile/namespaces. Discovery can precede generated input
and static template syntax. `serve` constructs one engine at startup; it does artifacts; completeness does not establish satisfied bindings.
not reload changed configuration or message content. Restart the process to 3. Once artifacts exist, validate the concrete request or inspect prepared
validate and capture a new configuration. Schema and request-artifact reads, content. These are provider-free preparation checks with native prerequisites.
credential resolution, and provider work remain request-preparation or runtime 4. Invoke a smoke test only when its provider effects are intended. Findings
concerns. For an HTTP deployment that accepts file artifacts, use a dedicated, can be unsupported or unsuccessful while the command itself succeeds.
narrow artifact directory rather than a general-purpose or sensitive filesystem Smoke tests are not an automatic startup or release prerequisite.
tree. 5. Generate with the same selections and bindings, then inspect the finalized
status and any returned validation result before accepting the output.
If application configuration uses explicit imports or a selected overlay, Use the [maintained preview script](../examples/render-markdown-summary.sh)
deploy those files beneath the selected configuration file's canonical for a provider-free example, and the [CLI contract](cli.md) for commands.
directory. ConfigKit keeps imported and overlay files within that root; A complete discovery report can still list required inputs that are not bound;
Scriptorium does not widen it. Application path values inside those files still a partial report can still be useful. Consumers own their policy for incomplete
resolve from the process working directory, so set that directory deliberately findings and changed resources between invocations.
in the service manager. This configuration-source boundary is distinct from
the HTTP artifact root and does not constrain prompt or schema directories.
For independently maintained collections, place checkouts beneath the ## Sensitive Data And Retention
directory selected by your framework discovery file. The maintained
[discovery example](../examples/config.discovery.yml) selects optional
`collection.yml` and `collection.yaml` entry files; each checkout owns its
namespace and explicit fragment imports. Update a checkout independently and
invoke the CLI again, or restart a running server to capture the update. Do
not treat arbitrary YAML in a checkout as a framework source. When upgrading
definitions, review exact profile IDs and replace retired catalog references
as described in the [migration guide](consumers/upgrading-promptkit.md).
Run Scriptorium under an identity that can: Artifacts, prompt messages, prepared content, native run reports, metadata,
diagnostics, and redirected streams are caller-owned potentially sensitive data.
The separate CLI report contains the full returned result, not metadata alone.
Control access, retention, encryption, backup, and deletion accordingly.
Framework effective/explanation views apply native classified-field redaction,
including history, but do not detect arbitrary secrets. Other content-bearing
reports are not automatically made safe for publication.
- read only the prompt, profile, schema, and allowed input-artifact paths it Keep main output and report destinations separate. Delivery is nontransactional:
needs; one destination or partial bytes can remain after failure. An existing report
- read the required credential environment variables without writing them to can remain on a resultless error; check status and report identity, not just file
files or logs; and existence. A shell can truncate a redirected destination before the child starts.
- write only caller-selected output locations when CLI output files are used. Omit quiet if main stdout must be captured; the separate report effect remains
independent. Exact delivery behavior belongs to [CLI results](cli.md#results-and-separate-reports).
Do not make the HTTP artifact directory writable by untrusted users. The HTTP HTTP's optional [execution report](api.md#execution-report) remains a distinct
artifact containment behavior is lexical and the operating system follows terminal-observation projection. Protect its identifiers and any response data.
symlinks; account for that when choosing ownership and mount boundaries. See It increases encoded response size and remains subject to the configured limit.
the [HTTP API reference](api.md) for the externally observable behavior.
## Supply Credentials And Protect Runtime Data ## HTTP Deployment And Shutdown
Set secret values in the process environment and configure only their The service has no built-in authentication/authorization. Deploy on a trusted
environment-variable names. Do not put raw keys in configuration, prompt or network or behind appropriate authenticated proxy/gateway controls. Use a narrow,
profile files, process arguments, HTTP payloads, captured command lines, or dedicated artifact root with controlled ownership and permissions when accepting
debug dumps. file inputs; this is lexical containment, not a symlink sandbox.
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP Use a supervisor for cwd, environment, restart policy, and stream capture.
responses as potentially sensitive. Send service logs to a controlled collector Startup information uses native Info reporting, not stdout banners.
and apply the same retention and access rules as for model input and output. Interrupt/SIGTERM cancels the invocation and closes the listener and active
connections. Canceled shutdown is nonzero operational termination. There is no
graceful-drain deadline or promise that context-ignoring work has finished;
supervisors own forced termination when necessary.
## Run A Normal Workflow ## Capacity, Limits, And Recovery
Before changing production inputs, profiles, or schemas: One server engine shares backend admission/queue state across concurrent requests.
That state is transient operational coordination, not a durable workflow or
queue of resumable runs. HTTP capacity exhaustion retains its
[API outcome](api.md); callers decide whether another model call is safe.
1. use `config validate`, `config show`, or `config sources` to check Plan for input/output sizes, schema complexity, provider latency, retries, and
Scriptorium's application configuration; these do not open Promptkit sources repair costs. Native transient attempts and repair generations can add calls and
or construct an engine; waiting; engine defaults and per-request overrides belong to
2. use `inspect configuration validate` to construct the engine and validate [configuration](config.md) and the [CLI](cli.md)/[HTTP](api.md) contracts.
all selected framework definitions; this does not prepare a request, check Whole-operation budgets are distinct from per-attempt generation timeouts.
credentials, load request-time schemas/artifacts, or contact a provider; They remain cooperative and do not include configuration/setup or delivery.
3. confirm the deployed configuration selects the intended sources and
environment-variable names;
4. use [`render`](cli.md) with the same request inputs and variables to confirm
preparation without a model call;
5. use [`run`](cli.md) for generation; and
6. retain or discard validation-failed output according to the caller's
policy.
Application configuration inspection can include paths and configured values. Before increasing HTTP byte limits, measure representative requests/artifacts/
Framework `inspect configuration show` and `explain` use Promptkit's native responses, confirm memory/network/provider capacity, retain authentication and
redaction for classified credential-shaped provider fields, including upstream request controls, and test a representative workload. Do not widen the
historical values, while leaving environment-reference names and ordinary artifact root just to make an arbitrary host path accessible.
provider parameters visible. Neither report identifies arbitrary sensitive
content by value or name. Protect inspection output accordingly; Scriptorium
stores environment-variable names, not the resolved credential bytes.
The [maintained render script](../examples/render-markdown-summary.sh) is a For failures, capture diagnostics without publishing sensitive data. Verify
copyable preflight example. The CLI reference owns its complete invocation and selected configuration, resource identity, readable inputs, profile/credentials,
exit semantics. and schema deployment. Use native prepared preview to isolate pre-generation
problems. A completed content-validation failure is distinct from an operational
failure to validate; consumers choose whether failed content is usable.
Correct the cause, then issue a new request only when another provider call and
its cost are acceptable.
## Expose The HTTP Service There is no durable run store, archive, checkpoint, or resume mechanism.
Cleanup concerns caller-owned files/logs only. Rotate/delete them under the
The HTTP service has no built-in authentication or authorization. Place it on a deployment's normal retention policy; recovery starts from a new request.
trusted network or behind an authenticated reverse proxy, API gateway, or
equivalent access control. Restrict who can reach it and who can read the
artifact root.
Use a service manager or supervisor appropriate to the deployment to manage
process lifetime, restart policy, log capture, and environment injection. The
[HTTP API reference](api.md) owns client request shapes, status behavior, and
artifact-access outcomes.
## Plan Capacity And Limits
Capacity is primarily determined by concurrent model calls, input and output
sizes, schema complexity, provider latency, and network behavior. Size limits
protect request bodies, HTTP file artifacts, and encoded responses; configure
them through the [configuration reference](config.md) and rely on the
[HTTP API reference](api.md) for their response effects.
Configured backend concurrency and queue capacity are enforced per constructed
Promptkit engine. A `serve` process constructs one engine for its handler, so
concurrent HTTP requests share that transient admission state. Scriptorium does
not retain workflow state: capacity is neither durable nor a queue of resumable
runs. When admission is exhausted, HTTP returns `503 capacity_exceeded` without
retry timing; callers choose any retry policy that is safe for another model
call.
Promptkit's built-in client now allows two additional attempts for eligible
transient provider statuses by default. Account for added calls, latency, and
cost when planning capacity. Each repair generation has its own transient
budget, separate from the structural repair count. Configure the engine budget
with [`transient_retries`](config.md); `run` and `render` accept a request
override described in the [CLI contract](cli.md), and HTTP callers can select
one through the [API contract](api.md). Omission inherits the engine
setting and then Promptkit's default, while explicit zero disables additional
attempts. Existing generation timeouts apply
per attempt, while the caller context covers the complete operation. Provider
`Retry-After` timing can exceed Promptkit's ordinary backoff cap; consult the
[outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
for details.
CLI `run` can set a [whole-run deadline](cli.md) when total execution time must
be bounded. The existing generation timeout applies afresh to each attempt and
does not cap retry waits or the complete invocation. The whole-run interval
covers the engine call, including capacity waiting and repairs as supported by
context cancellation; it excludes parsing, engine setup, and output writes.
HTTP callers can use [`run_timeout_seconds`](api.md) to set the engine's
operation budget for the same interval. The API distinguishes a typed expired
execution deadline from caller cancellation and from a provider-attempt
timeout. A canceled capacity waiter releases its Promptkit admission state.
Before increasing a limit:
1. measure representative input, generated-output, and optional raw-output
sizes;
2. confirm memory, network, and upstream-provider capacity;
3. retain an upstream request-size and authentication boundary; and
4. test the intended workload in a non-production environment.
For large local inputs, prefer a controlled file-artifact directory over
placing arbitrary paths on the service host. Avoid disabling a limit unless an
equivalent trusted control exists elsewhere.
## Diagnose And Recover
### Preparation Or Configuration Failure
Capture the CLI diagnostic or HTTP error response, then verify the selected
configuration, prompt ID, profile selection, source readability, and input
mapping. Use `render` with the same request when it is unclear whether failure
occurs before model execution. Consult the [CLI reference](cli.md), the
[configuration reference](config.md), and the [HTTP API reference](api.md) for
the exact interface contract.
### Credential Or Provider Failure
Confirm that the process environment contains the configured credential name
without printing the secret. Check endpoint reachability and provider health
from the process network. If preparation succeeds but generation fails, inspect
the selected model settings in prepared output and the service's controlled
logs. Correct the deployment or provider issue, then submit a new request.
### Artifact Or Permission Failure
Verify that the process can read the intended local input. For HTTP file
artifacts, verify the deployment's artifact root, ownership, path layout, and
file size. Do not widen filesystem permissions or the allowed root merely to
make an arbitrary path work; move or copy the required artifact into the
controlled location instead.
### Validation Failure
A generated-content validation failure is distinct from a runtime failure.
CLI `run` reports the validation result and error count in its success summary;
it does not print the individual validation messages. For HTTP, inspect the
validation object in the response according to the [HTTP API reference](api.md).
Use rendered input and generated output to determine whether prompt instructions,
the selected model, or the schema needs correction. If schema loading or
compilation itself fails, correct the source deployment or schema document
before rerunning.
### HTTP Limit Or Request Failure
Compare the request, artifact, or expected response size with the deployed
configuration, and validate the request against the [HTTP API reference](api.md).
Reduce the payload, use an appropriate controlled artifact source, omit
unneeded raw output, or adjust the deployment limit after capacity review.
## Cleanup And Reruns
Because no run state is retained, cleanup concerns caller-owned output files,
logs, and artifacts only. Remove or rotate them using the deployment's normal
retention policy. After a correction, rerun the request from the beginning;
there is no safe resume point.

View File

@@ -39,8 +39,8 @@ Do not promote incidental implementation choices into architectural invariants.
## Scriptorium's Purpose And System Shape ## Scriptorium's Purpose And System Shape
Scriptorium is an executable application with CLI `run`, `render`, Scriptorium is an executable application with CLI `run`, native `promptkit`
`inspect`, and `config` command families and an HTTP service started by and `config` families, and an HTTP service started by
`serve`. It offers simple, request-independent execution of predefined `serve`. It offers simple, request-independent execution of predefined
prompts, not a reusable Scriptorium root Go API or a workflow framework. prompts, not a reusable Scriptorium root Go API or a workflow framework.
The [internal overview](../internal/overview.md) owns the concrete package The [internal overview](../internal/overview.md) owns the concrete package
@@ -71,9 +71,12 @@ externally observable application behavior; PromptKit owns framework formats.
As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md), As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to
neutrally expose upstream features and semantics. The current CLI still uses neutrally expose upstream features and semantics. CommandKit owns shared command,
custom presentation; the [adoption roadmap](../roadmap/commandkit-adoption.md) parsing, presentation, and outcome mechanics; application commands and effects
tracks its replacement. This decision complements ADRs 0004 and 0005. remain local. Standard configuration inspection uses native adapters without
custom effective documents or evidence reconciliation. This decision complements
ADRs 0004 and 0005; the [adoption roadmap](../roadmap/commandkit-adoption.md)
records the implementation work.
The CLI exposes Promptkit's captured framework configuration inspection. Its The CLI exposes Promptkit's captured framework configuration inspection. Its
`show` and `explain` reports use Promptkit's native redaction for classified `show` and `explain` reports use Promptkit's native redaction for classified

View File

@@ -2,7 +2,7 @@
**Base Policy Version:** 1.3 **Base Policy Version:** 1.3
**Last Updated**: 2026-11-17 **Last Updated**: 2026-11-19
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md **Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md
@@ -178,12 +178,15 @@ Give each layer its distinct responsibility and keep expectation updates localiz
diagnostics. ConfigKit owns exhaustive YAML grammar, source traversal, diagnostics. ConfigKit owns exhaustive YAML grammar, source traversal,
import and overlay composition, provenance, and inspection renderers. import and overlay composition, provenance, and inspection renderers.
- CLI tests own argument mapping, streams, output, exit behavior, and - CLI tests own argument mapping, streams, output, exit behavior, and
representative run, render, inspect, serve, and config workflows. representative native run, PromptKit resource/requirements/preview/smoke,
serve, and configuration comparison workflows.
- HTTP tests own DTO decoding, transport limits, status and error mapping, - HTTP tests own DTO decoding, transport limits, status and error mapping,
restricted artifact reading, and representative server workflows. restricted artifact reading, and representative server workflows.
- Formatting tests own prepared-run and definition-inspection text and JSON - CommandKit and its integrations own standard parsing and presentation.
presentation. Architecture guards own dependency direction and removed Scriptorium's integration tests protect native adoption, byte-preserving main
framework-package boundaries. output, separate report effects, outcomes, and acquisition boundaries without
reproducing the upstream formatting matrix. Architecture guards own dependency
direction and removed framework-package boundaries.
- Promptkit owns framework parsing, orchestration, validation, profiles, and - Promptkit owns framework parsing, orchestration, validation, profiles, and
model-client behavior. Scriptorium's definition and execution fixtures model-client behavior. Scriptorium's definition and execution fixtures
protect adoption through its public adapter boundary without duplicating protect adoption through its public adapter boundary without duplicating

View File

@@ -643,6 +643,25 @@ version output. Test-local fallback deadlines prevent regressions from hanging.
## Stage 8: Retire Superseded Machinery And Update Canonical Documentation ## Stage 8: Retire Superseded Machinery And Update Canonical Documentation
**Status: Complete.** Removed legacy parsing/dispatch, custom inspection/source
DTOs and writers, retired CLI tests, and the unused formatter package. Retained
architecture boundaries, native integration protection, checkout-discovery
coverage, and the unchanged HTTP/report implementation and regression suites.
Maintained examples now select external framework entries with native imports;
both collection-entry spellings and multi-definition discovery remain present.
Canonical CLI/configuration/process/operations/internal/development/policy owners
and navigation reflect the cutover, with a focused CLI migration guide distinct
from the earlier framework-format guide. Accepted ADRs and release-note history
are unchanged; deferred HTTP redesign and dynamic completion remain deferred.
Verified on Go 1.27.4: full tests and race suite, vet, ordinary build, module
tidiness, formatting/diff checks, all maintained application configs in both
native scopes, application overlays, maintained preview scripts, a structured
schema preview, and local Markdown links/anchors (42 files). Added maintained
example loading/preparation protection and adapted fresh-checkout discovery to
native preview. Entrypoint package tests now run and pass. Linked identity,
release-target builds, and the final assembled acceptance audit remain Stage 9.
**Primary files:** old CLI parser/dispatch/inspection files and tests, **Primary files:** old CLI parser/dispatch/inspection files and tests,
`internal/format`, documentation and maintained examples listed below. `internal/format`, documentation and maintained examples listed below.

View File

@@ -1,8 +1,9 @@
# Prompt input requirements for Narratio # Prompt input requirements for Narratio
Status: accepted direction, pending implementation through the CommandKit Status: the native interface and focused consumer scenarios are implemented.
migration. This proposal records Narratio's consumer needs and their mapping The [implementation plan](implementation.md) tracks final assembled verification.
to the native interface; it does not describe shipped Scriptorium behavior. This proposal records Narratio's consumer needs and their mapping to the native
interface; the [CLI contract](../cli.md) owns current behavior.
The [CommandKit notepad](commandkit-notepad.md) records the migration decisions. The [CommandKit notepad](commandkit-notepad.md) records the migration decisions.
## Consumer need ## Consumer need
@@ -27,8 +28,8 @@ Default and explicit text output use the native YAML presentation.
The [upstream CLI contract][requirements-cli] owns syntax and report details. The [upstream CLI contract][requirements-cli] owns syntax and report details.
Use the same application configuration and framework source selection as Use the same application configuration and framework source selection as
execution, including `--config`, `--overlay`, and applicable application execution, including `--config` and `--overlay`. There are no application-defined
selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and source/schema selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
prompt/profile namespace flags supply the relevant discovery context. prompt/profile namespace flags supply the relevant discovery context.
Preserve upstream selection semantics, including the prompt's default profile Preserve upstream selection semantics, including the prompt's default profile
when an explicit profile is omitted. Qualified IDs identify installed resources; when an explicit profile is omitted. Qualified IDs identify installed resources;

View File

@@ -0,0 +1,6 @@
config:
imports:
- path: collections.yml
required: true
- path: profiles/local-gpu.yml
required: true

5
examples/collections.yml Normal file
View File

@@ -0,0 +1,5 @@
config:
imports:
- dir: config-fragments/collections
pattern: '*.yml'
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources: config:
- file: ./examples/framework/backends.yml imports:
- path: ../../framework/backends.yml
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources: config:
- file: ./examples/profiles/local-fast.yaml imports:
- path: ../../profiles/local-fast.yaml
required: true

View File

@@ -1,2 +1,4 @@
promptkit_sources: config:
- file: ./examples/prompts/generic.markdown_summary.yaml imports:
- path: ../../prompts/generic.markdown_summary.yaml
required: true

View File

@@ -1,5 +1,3 @@
promptkit_sources: promptkit:
- directory: ./examples/prompts configuration: ./examples/promptkit.yml
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas schema_dir: ./examples/schemas

View File

@@ -1,10 +1,3 @@
config: promptkit:
sequences: configuration: ./examples/collections.yml
- destination: [promptkit_sources]
key: file
mode: upsert
imports:
- dir: config-fragments/collections
pattern: '*.yml'
required: true
schema_dir: ./examples/schemas schema_dir: ./examples/schemas

View File

@@ -1,2 +1,2 @@
promptkit_sources: promptkit:
- file: ./examples/discovery/framework.yml configuration: ./examples/discovery/framework.yml

View File

@@ -1,7 +1,5 @@
promptkit_sources: promptkit:
- directory: ./examples/prompts configuration: ./examples/promptkit.yml
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas schema_dir: ./examples/schemas
server: server:
@@ -10,6 +8,3 @@ server:
max_request_bytes: 16777216 max_request_bytes: 16777216
max_artifact_bytes: 16777216 max_artifact_bytes: 16777216
max_response_bytes: 16777216 max_response_bytes: 16777216
defaults:
render_format: text

View File

@@ -1,6 +1,4 @@
# Application paths are relative to the repository invocation directory. # Application paths are relative to the repository invocation directory.
promptkit_sources: promptkit:
- directory: ./examples/prompts configuration: ./examples/inheritance.yml
- directory: ./examples/profiles
- file: ./examples/framework/inheritance.yml
schema_dir: ./examples/schemas schema_dir: ./examples/schemas

View File

@@ -1,12 +1,7 @@
promptkit_sources: promptkit:
- directory: ./examples/prompts configuration: ./examples/promptkit.yml
- directory: ./examples/profiles
- file: ./examples/framework/backends.yml
schema_dir: ./examples/schemas schema_dir: ./examples/schemas
server: server:
addr: :8080 addr: :8080
artifact_root: . artifact_root: .
defaults:
render_format: text

8
examples/definitions.yml Normal file
View File

@@ -0,0 +1,8 @@
config:
imports:
- dir: prompts
pattern: '*.yaml'
required: true
- dir: profiles
pattern: '*.y*ml'
required: true

6
examples/inheritance.yml Normal file
View File

@@ -0,0 +1,6 @@
config:
imports:
- path: definitions.yml
required: true
- path: framework/inheritance.yml
required: true

View File

@@ -1,2 +1,2 @@
promptkit_sources: promptkit:
- file: ./examples/profiles/local-gpu.yml configuration: ./examples/collection-gpu.yml

View File

@@ -1,4 +1,2 @@
defaults:
render_format: json
server: server:
addr: 127.0.0.1:8080 addr: 127.0.0.1:8080

6
examples/promptkit.yml Normal file
View File

@@ -0,0 +1,6 @@
config:
imports:
- path: definitions.yml
required: true
- path: framework/backends.yml
required: true

View File

@@ -5,9 +5,8 @@ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root" cd "$repo_root"
go run ./cmd/scriptorium render \ go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.yml \ --config ./examples/config.yml \
--prompt generic.markdown_summary \ --input-file transcript=./examples/fixtures/transcript.md \
--input transcript=./examples/fixtures/transcript.md \ --input-file glossary=./examples/fixtures/glossary.yml \
--input glossary=./examples/fixtures/glossary.yml \
--format text --format text

View File

@@ -4,21 +4,18 @@ set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root" cd "$repo_root"
go run ./cmd/scriptorium render \ go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
--config ./examples/config.full.yml \ --config ./examples/config.full.yml \
--prompt generic.markdown_summary \
--profile local-gpu \ --profile local-gpu \
--session-id example-session \ --session-id example-session \
--reasoning-effort= \ --reasoning-effort= \
--input transcript=./examples/fixtures/transcript.md \ --input-file transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml --input-file glossary=./examples/fixtures/glossary.yml
go run ./cmd/scriptorium inspect prompt \ go run ./cmd/scriptorium promptkit show prompt generic.markdown_summary \
--config ./examples/config.full.yml \ --config ./examples/config.full.yml \
--prompt generic.markdown_summary \
--format json --format json
go run ./cmd/scriptorium inspect profile \ go run ./cmd/scriptorium promptkit show profile local-gpu \
--config ./examples/config.full.yml \ --config ./examples/config.full.yml \
--profile local-gpu \
--format json --format json

2
go.mod
View File

@@ -8,6 +8,7 @@ require (
go.maximumdirect.net/commandkit v0.7.0 go.maximumdirect.net/commandkit v0.7.0
go.maximumdirect.net/configkit v1.2.1 go.maximumdirect.net/configkit v1.2.1
go.maximumdirect.net/promptkit v0.23.3 go.maximumdirect.net/promptkit v0.23.3
go.yaml.in/yaml/v4 v4.0.0
) )
require ( require (
@@ -15,6 +16,5 @@ require (
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect
github.com/spf13/cobra v1.10.2 // indirect github.com/spf13/cobra v1.10.2 // indirect
github.com/spf13/pflag v1.1.0 // indirect github.com/spf13/pflag v1.1.0 // indirect
go.yaml.in/yaml/v4 v4.0.0 // indirect
golang.org/x/text v0.44.0 // indirect golang.org/x/text v0.44.0 // indirect
) )

View File

@@ -1,39 +0,0 @@
package cli
import (
"errors"
"fmt"
"io"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"go.maximumdirect.net/configkit"
)
func loadApplicationSettings(configPath, overlay string, explicit bool, overrides []configkit.Override) (appconfig.AppSettings, []configkit.Diagnostic, error) {
result, err := appconfig.Load(appconfig.LoadOptions{
Path: configPath, Overlay: overlay, Explicit: explicit, Overrides: overrides,
})
if err != nil {
return appconfig.AppSettings{}, nil, err
}
return appconfig.Settings(result.Config), result.Diagnostics(), nil
}
func writeApplicationError(output io.Writer, label string, err error) {
var loadError *configkit.LoadError
if errors.As(err, &loadError) {
_, _ = fmt.Fprintf(output, "%s error:\n", label)
if writeErr := configkit.WriteDiagnostics(output, loadError.Diagnostics()); writeErr != nil {
_, _ = fmt.Fprintf(output, "%s diagnostic write error: %v\n", label, writeErr)
}
return
}
_, _ = fmt.Fprintf(output, "%s error: %v\n", label, err)
}
func writeApplicationWarnings(output io.Writer, diagnostics []configkit.Diagnostic) error {
if len(diagnostics) == 0 {
return nil
}
return configkit.WriteDiagnostics(output, diagnostics)
}

View File

@@ -1,101 +0,0 @@
package cli
import (
"encoding/json"
"flag"
"fmt"
"io"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"go.maximumdirect.net/configkit"
)
type configurationSource struct {
ID string `json:"id"`
Label string `json:"label"`
Layer configkit.Layer `json:"layer"`
Path string `json:"path,omitempty"`
}
func configCommand(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "config parse error: configuration mode is required")
return ExitRuntimeError
}
mode := args[0]
if mode != "validate" && mode != "show" && mode != "sources" && mode != "explain" {
fmt.Fprintln(stderr, "config parse error: unknown configuration mode")
return ExitRuntimeError
}
fs := flag.NewFlagSet("config "+mode, flag.ContinueOnError)
fs.SetOutput(io.Discard)
var path, overlay, pointer string
registerConfigSelectionFlags(fs, &path, &overlay)
registerConfigurationOverrides(fs)
if mode == "explain" {
fs.StringVar(&pointer, "path", "", "configuration path as a JSON Pointer")
}
if err := fs.Parse(args[1:]); err != nil {
writeApplicationError(stderr, "config parse", err)
return ExitRuntimeError
}
if fs.NArg() != 0 {
fmt.Fprintln(stderr, "config parse error: positional arguments are not accepted")
return ExitRuntimeError
}
var explanationPath configkit.Path
if mode == "explain" {
if !flagWasSet(fs, "path") {
fmt.Fprintln(stderr, "config parse error: --path is required")
return ExitRuntimeError
}
var err error
explanationPath, err = configkit.ParsePath(pointer)
if err != nil {
fmt.Fprintln(stderr, "config parse error: --path must be a JSON Pointer")
return ExitRuntimeError
}
}
result, err := appconfig.Load(appconfig.LoadOptions{
Path: path, Explicit: flagWasSet(fs, "config"), Overlay: overlay,
Overrides: applicationOverrides(fs),
})
if err != nil {
writeApplicationError(stderr, "config", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, result.Diagnostics()); err != nil {
return ExitRuntimeError
}
switch mode {
case "validate":
return ExitOK
case "show":
err = result.WriteEffective(stdout)
case "sources":
sources := make([]configurationSource, 0, len(result.Sources()))
for _, source := range result.Sources() {
sources = append(sources, configurationSource{
ID: source.ID, Label: source.Label, Layer: source.Layer, Path: source.Path,
})
}
err = json.NewEncoder(stdout).Encode(sources)
case "explain":
err = result.WriteExplanation(stdout, explanationPath)
}
if err != nil {
fmt.Fprintf(stderr, "config output error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func registerConfigurationOverrides(fs *flag.FlagSet) {
fs.String("addr", "", "HTTP listen address override")
fs.String("artifact-root", "", "HTTP artifact root override")
fs.Int64("max-request-bytes", 0, "HTTP request byte limit override")
fs.Int64("max-artifact-bytes", 0, "HTTP artifact byte limit override")
fs.Int64("max-response-bytes", 0, "HTTP response byte limit override")
}

View File

@@ -1,226 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"io"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
)
func TestConfigCommandsInspectApplicationWithoutEngine(t *testing.T) {
path := writeAppConfigFile(t, "{}\n")
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
if code != ExitOK || stderr != "" {
t.Fatalf("%s failed: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
}
switch mode {
case "validate":
if stdout != "" {
t.Fatalf("validation wrote stdout: %q", stdout)
}
case "show":
if !strings.Contains(stdout, "render_format: text") || !strings.Contains(stdout, "max_request_bytes: 16777216") {
t.Fatalf("effective defaults missing: %q", stdout)
}
case "sources":
var sources []configurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "main" {
t.Fatalf("unexpected source records: %q (%v)", stdout, err)
}
}
}
}
func TestCollectionExampleLoadsAndRenders(t *testing.T) {
t.Chdir(filepath.Join("..", "..", ".."))
want := []appconfig.SourceSelection{
{File: "examples/framework/backends.yml"},
{File: "examples/profiles/local-fast.yaml"},
{File: "examples/prompts/generic.markdown_summary.yaml"},
}
for _, overlay := range []string{"", "collection"} {
result, err := appconfig.Load(appconfig.LoadOptions{Path: "examples/config.collections.yml", Explicit: true, Overlay: overlay})
if err != nil {
t.Fatal(err)
}
expected := append([]appconfig.SourceSelection{}, want...)
if overlay != "" {
expected = append(expected, appconfig.SourceSelection{File: "examples/profiles/local-gpu.yml"})
}
if !reflect.DeepEqual(result.Config.PromptKitSources, expected) {
t.Fatalf("overlay %q selectors = %+v, want %+v", overlay, result.Config.PromptKitSources, expected)
}
args := []string{"--config", "examples/config.collections.yml", "--prompt", "generic.markdown_summary", "--input", "transcript=examples/fixtures/transcript.md", "--input", "glossary=examples/fixtures/glossary.yml"}
if overlay != "" {
args = append(args, "--config-overlay", overlay)
}
code, stdout, stderr := runCLICommand(t, renderCommand, args)
if code != ExitOK || !strings.Contains(stdout, "generic.markdown_summary") || !strings.Contains(stdout, "local-fast") {
t.Fatalf("collection render failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
}
}
func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
t.Fatal(err)
}
base := filepath.Join(dir, "config.yml")
for path, body := range map[string]string{
base: "config:\n imports:\n - path: fragment.yml\n required: true\npromptkit_sources:\n - directory: ' ./before/../prompts '\nschema_dir: ./schemas\n",
filepath.Join(dir, "fragment.yml"): "server:\n addr: :8081\n",
filepath.Join(dir, "overlays", "local.yml"): "promptkit_sources:\n - directory: ./overlay-prompts\nserver:\n addr: :9090\n",
} {
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
args := []string{"--config", base, "--config-overlay", "local", "--schema-dir=", "--addr", ":7070"}
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"show"}, args...))
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "directory: overlay-prompts") || !strings.Contains(stdout, "schema_dir: ''") || !strings.Contains(stdout, "addr: :7070") || strings.Contains(stdout, "before/../prompts") {
t.Fatalf("resolved effective output changed: code=%d out=%q err=%q", code, stdout, stderr)
}
code, stdout, stderr = runCLICommand(t, configCommand, append([]string{"sources"}, args...))
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
t.Fatalf("source command failed: code=%d out=%q err=%q", code, stdout, stderr)
}
var sources []configurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 5 {
t.Fatalf("unexpected composed source records: %+v (%v)", sources, err)
}
for i, layer := range []string{"main", "main", "overlay", "cli", "cli"} {
if string(sources[i].Layer) != layer {
t.Fatalf("source %d layer = %q, want %q", i, sources[i].Layer, layer)
}
}
}
func TestApplicationInspectionDoesNotAcquireFrameworkSources(t *testing.T) {
dir := t.TempDir()
frameworkPath := filepath.Join(dir, "invalid-framework.yml")
if err := os.WriteFile(frameworkPath, []byte("not: [valid\n"), 0o600); err != nil {
t.Fatal(err)
}
for _, frameworkSource := range []struct {
name string
path string
}{
{name: "invalid definition", path: frameworkPath},
{name: "missing path", path: filepath.Join(dir, "missing-framework.yml")},
} {
t.Run(frameworkSource.name, func(t *testing.T) {
path := writeAppConfigFile(t, "promptkit_sources:\n - file: "+frameworkSource.path+"\n")
for _, args := range [][]string{
{"validate"},
{"show"},
{"sources"},
{"explain", "--path", "/promptkit_sources/0/file"},
} {
code, stdout, stderr := runCLICommand(t, configCommand, append(args, "--config", path))
if code != ExitOK || stderr != "" {
t.Fatalf("application inspection acquired configured framework source: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
}
}
})
}
}
func TestConfigCommandsRejectRetiredFrameworkFields(t *testing.T) {
for _, field := range []string{"prompt_dir", "profile_dir", "backends"} {
t.Run(field, func(t *testing.T) {
path := writeAppConfigFile(t, field+": {}\n")
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", path})
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("retired application field %q was accepted: code=%d out=%q err=%q", field, code, stdout, stderr)
}
})
}
}
type errorOutputWriter struct{}
func (errorOutputWriter) Write([]byte) (int, error) { return 0, errors.New("output unavailable") }
func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) {
path := writeAppConfigFile(t, "server:\n max_request_bytes: -1\n")
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "validate.error") {
t.Fatalf("%s misclassified invalid config: code=%d out=%q err=%q", mode, code, stdout, stderr)
}
}
path = writeAppConfigFile(t, "{}\n")
var stderr bytes.Buffer
if code := configCommand([]string{"show", "--config", path}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
t.Fatalf("writer error not reported: code=%d stderr=%q", code, stderr.String())
}
if code := configCommand([]string{"sources", "--config", path}, errorOutputWriter{}, io.Discard); code != ExitRuntimeError {
t.Fatalf("source writer error returned %d", code)
}
}
func TestConfigExplainUsesResolvedHistory(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
t.Fatal(err)
}
base := filepath.Join(dir, "config.yml")
for path, body := range map[string]string{
base: `promptkit_sources:
- directory: " ./prompts/../prompts "
server:
addr: :8080
`,
filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n",
} {
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
args := []string{"--config", base, "--config-overlay", "local", "--addr", ":9090"}
for _, tc := range []struct {
pointer string
want []string
}{
{pointer: "/server/addr", want: []string{":8080", ":9090", "--addr"}},
{pointer: "/promptkit_sources/0/directory", want: []string{"normalized application path", "prompts"}},
{pointer: "", want: []string{"promptkit_sources", "server", "addr"}},
} {
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", tc.pointer}, args...))
if code != ExitOK || stderr != "" {
t.Fatalf("explain %q failed: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
}
for _, want := range tc.want {
if !strings.Contains(stdout, want) {
t.Fatalf("explain %q lacks %q: %q", tc.pointer, want, stdout)
}
}
}
}
func TestConfigExplainRejectsUnknownOrMalformedPaths(t *testing.T) {
path := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"explain", "--config", path},
{"explain", "--config", path, "--path", "not-a-pointer"},
{"explain", "--config", path, "--path", "/missing"},
} {
code, stdout, stderr := runCLICommand(t, configCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid explanation path accepted: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
}
}
var stderr bytes.Buffer
if code := configCommand([]string{"explain", "--config", path, "--path", ""}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
t.Fatalf("explanation writer failure not reported: code=%d err=%q", code, stderr.String())
}
}

View File

@@ -1,7 +1,9 @@
package cli package cli
import ( import (
"encoding/json"
"fmt" "fmt"
"go.maximumdirect.net/promptkit"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -41,37 +43,46 @@ func TestCollectionCheckoutDiscoveryAcrossInvocations(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
} }
config := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkPath)) config := filepath.Join(root, "app.yml")
writeNativeFile(t, config, fmt.Sprintf("promptkit:\n configuration: %q\n", frameworkPath))
input := filepath.Join(root, "transcript.md") input := filepath.Join(root, "transcript.md")
if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil { if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil {
t.Fatal(err) t.Fatal(err)
} }
args := func(name string) []string { args := func(name string) []string {
if name == "support-repo" { if name == "support-repo" {
return []string{"--config", config, "--prompt", "support/answer", "--profile", "support/fast", "--input", "transcript=" + input} return []string{"support/answer", "--config", config, "--profile", "support/fast", "--input-file", "transcript=" + input}
} }
return []string{"--config", config, "--prompt", "review/answer", "--profile", "review/base"} return []string{"review/answer", "--config", config, "--profile", "review/base"}
} }
check := func(name string) { check := func(name string) {
t.Helper() t.Helper()
code, out, stderr := runCLICommand(t, renderCommand, args(name)) command := append([]string{"promptkit", "show", "prepared"}, args(name)...)
if code != ExitOK { outcome, out, stderr := executeNative(t, append(command, "--format=json")...)
t.Fatalf("render %s: code=%d stderr=%s", name, code, stderr) if outcome.Status != 0 {
t.Fatalf("prepare %s: outcome=%+v stderr=%s", name, outcome, stderr)
}
var prepared promptkit.PreparedRun
if err := json.Unmarshal([]byte(out), &prepared); err != nil {
t.Fatal(err)
} }
if name == "support-repo" { if name == "support-repo" {
for _, want := range []string{"prompt: support/answer", "selected_profile_id: support/fast", "model: support-example", "max_tokens: 256", "Answer the support request", "synthetic discovery transcript"} { for _, want := range []string{"support/answer", "support/fast", "support-example", "Answer the support request", "synthetic discovery transcript"} {
if !strings.Contains(out, want) { if !strings.Contains(out, want) {
t.Fatalf("missing %q: %s", want, out) t.Fatalf("missing %q: %s", want, out)
} }
} }
} else if !strings.Contains(out, "prompt: review/answer") || !strings.Contains(out, "model: review-example") || !strings.Contains(out, "Review this material") { if prepared.EffectiveModelParams.MaxTokens != 256 {
t.Fatalf("inherited profile setting missing: %+v", prepared)
}
} else if !strings.Contains(out, "review/answer") || !strings.Contains(out, "review-example") || !strings.Contains(out, "Review this material") {
t.Fatalf("unexpected review render: %s", out) t.Fatalf("unexpected review render: %s", out)
} }
} }
install(tc.first) install(tc.first)
check(tc.first) check(tc.first)
code, _, _ := runCLICommand(t, renderCommand, args(tc.second)) outcome, _, _ := executeNative(t, append([]string{"promptkit", "show", "prepared"}, args(tc.second)...)...)
if code != ExitRuntimeError { if outcome.Status != 1 {
t.Fatal("uninstalled collection unexpectedly selected") t.Fatal("uninstalled collection unexpectedly selected")
} }
install(tc.second) install(tc.second)

View File

@@ -0,0 +1,42 @@
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestMaintainedApplicationExamplesUseNativeLoading(t *testing.T) {
root, err := filepath.Abs(filepath.Join("..", "..", ".."))
if err != nil {
t.Fatal(err)
}
t.Chdir(root)
entries, err := os.ReadDir("examples")
if err != nil {
t.Fatal(err)
}
for _, entry := range entries {
if entry.IsDir() || !strings.HasPrefix(entry.Name(), "config") || !strings.HasSuffix(entry.Name(), ".yml") {
continue
}
path := filepath.Join("examples", entry.Name())
for _, scope := range []string{"app", "promptkit"} {
outcome, _, diagnostics := executeNative(t, "config", "validate", "--config="+path, "--scope="+scope)
if outcome.Status != 0 {
t.Fatalf("%s/%s: %+v %q", path, scope, outcome, diagnostics)
}
}
}
for _, test := range []struct{ config, overlay string }{{"examples/config.yml", ""}, {"examples/config.composed.yml", "local"}, {"examples/config.collections.yml", "collection"}, {"examples/config.inheritance.yml", ""}} {
args := []string{"promptkit", "show", "prepared", "generic.markdown_summary", "--config=" + test.config, "--profile=local-gpu", "--input-file=transcript=examples/fixtures/transcript.md", "--input-file=glossary=examples/fixtures/glossary.yml", "--format=json"}
if test.overlay != "" {
args = append(args, "--overlay="+test.overlay)
}
outcome, output, diagnostics := executeNative(t, args...)
if outcome.Status != 0 || !strings.Contains(output, "generic.markdown_summary") || !strings.Contains(output, "Session Transcript") {
t.Fatalf("%+v: %+v %q %q", test, outcome, output, diagnostics)
}
}
}

View File

@@ -1,159 +0,0 @@
package cli
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"go.maximumdirect.net/configkit"
)
type frameworkConfigurationConfig struct {
mode, configPath, configOverlay, schemaDir string
path string
configExplicit bool
pathExplicit bool
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
type frameworkConfigurationSource struct {
ID string `json:"id"`
Label string `json:"label"`
Layer configkit.Layer `json:"layer"`
Kind configkit.SourceKind `json:"kind"`
UnderlayIndex int `json:"underlay_index"`
Path string `json:"path,omitempty"`
ImportedFrom *frameworkConfigurationLocation `json:"imported_from,omitempty"`
}
type frameworkConfigurationLocation struct {
SourceID string `json:"source_id"`
Line int `json:"line"`
Column int `json:"column"`
}
func inspectFrameworkConfigurationCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseFrameworkConfigurationArgs(args)
if err != nil {
writeApplicationError(stderr, "inspect configuration parse", err)
return ExitRuntimeError
}
var explanationPath configkit.Path
if cfg.mode == "explain" {
explanationPath, err = configkit.ParsePath(cfg.path)
if err != nil {
writeApplicationError(stderr, "inspect configuration path", err)
return ExitRuntimeError
}
}
settings, err := resolveAppSettingsForFrameworkConfiguration(cfg)
if err != nil {
writeApplicationError(stderr, "inspect configuration", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
view, err := engine.InspectConfiguration()
if err != nil {
writeApplicationError(stderr, "inspect configuration", err)
return ExitRuntimeError
}
if err := configkit.WriteDiagnostics(stderr, view.Diagnostics()); err != nil {
return ExitRuntimeError
}
if cfg.mode == "validate" {
return ExitOK
}
if cfg.mode == "explain" {
if err := view.WriteExplanation(stdout, explanationPath); err != nil {
fmt.Fprintf(stderr, "inspect configuration explain %q: %v\n", cfg.path, err)
return ExitRuntimeError
}
return ExitOK
}
switch cfg.mode {
case "show":
err = view.WriteEffective(stdout)
case "sources":
sources := frameworkConfigurationSources(view.Sources())
err = json.NewEncoder(stdout).Encode(sources)
}
if err != nil {
fmt.Fprintf(stderr, "inspect configuration output error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseFrameworkConfigurationArgs(args []string) (*frameworkConfigurationConfig, error) {
if len(args) == 0 {
return nil, errors.New("configuration operation is required")
}
cfg := &frameworkConfigurationConfig{mode: args[0]}
switch cfg.mode {
case "validate", "show", "sources", "explain":
default:
return nil, fmt.Errorf("unknown configuration operation %q", cfg.mode)
}
fs := flag.NewFlagSet("inspect configuration "+cfg.mode, flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
fs.StringVar(&cfg.path, "path", "", "JSON Pointer to explain (required for explain)")
if err := fs.Parse(args[1:]); err != nil {
return nil, err
}
if fs.NArg() != 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.pathExplicit = flagWasSet(fs, "path")
if cfg.mode == "explain" && !cfg.pathExplicit {
return nil, errors.New("--path is required for explain")
}
cfg.overrides = applicationOverrides(fs)
return cfg, nil
}
func resolveAppSettingsForFrameworkConfiguration(cfg *frameworkConfigurationConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{
promptkitSources: settings.PromptKitSources,
schemaDir: settings.SchemaDir,
transientRetries: settings.TransientRetries,
}, nil
}
func frameworkConfigurationSources(sources []configkit.SourceInfo) []frameworkConfigurationSource {
result := make([]frameworkConfigurationSource, 0, len(sources))
for _, source := range sources {
item := frameworkConfigurationSource{
ID: source.ID, Label: source.Label, Layer: source.Layer,
Kind: source.Kind, UnderlayIndex: source.UnderlayIndex, Path: source.Path,
}
if source.ImportedFrom != nil {
item.ImportedFrom = &frameworkConfigurationLocation{
SourceID: source.ImportedFrom.SourceID,
Line: source.ImportedFrom.Line,
Column: source.ImportedFrom.Column,
}
}
result = append(result, item)
}
return result
}

View File

@@ -1,391 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
configkit "go.maximumdirect.net/configkit"
)
func TestFrameworkConfigurationCommandsNeedNoPromptOrCredential(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
"configuration", "validate", "--config", configPath,
})
if code != ExitOK || stdout != "" || stderr != "" {
t.Fatalf("source-free validation failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
t.Setenv("SCRIPTORIUM_FRAMEWORK_KEY", "")
root := t.TempDir()
frameworkFile := filepath.Join(root, "framework.yml")
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_FRAMEWORK_KEY
profiles:
- id: protected
backend: local
model: offline-model
api_key_required: true
`)
configPath = writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
for _, mode := range []string{"validate", "show", "sources"} {
code, stdout, stderr = runCLICommand(t, inspectCommand, []string{
"configuration", mode, "--config", configPath,
})
if code != ExitOK || stderr != "" {
t.Fatalf("%s required a prompt or credential: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
}
if mode == "validate" && stdout != "" {
t.Fatalf("validation wrote success output: %q", stdout)
}
if mode == "show" && (!strings.Contains(stdout, "profiles:") || !strings.Contains(stdout, "offline-model")) {
t.Fatalf("effective framework YAML is missing: %q", stdout)
}
}
}
func TestFrameworkConfigurationSourcesProjectsOrderedNativeRecords(t *testing.T) {
root := t.TempDir()
importDir := filepath.Join(root, "catalog")
if err := os.MkdirAll(importDir, 0o755); err != nil {
t.Fatal(err)
}
mainPath := writeFrameworkSourceFile(t, root, "main.yml", `config:
imports:
- path: catalog/fragment.yml
required: true
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
`)
fragmentPath := writeFrameworkSourceFile(t, importDir, "fragment.yml", `profiles:
- id: imported
model: imported-model
`)
directory := t.TempDir()
directoryPath := writeFrameworkSourceFile(t, directory, "directory.yml", `profiles:
- id: directory
model: directory-model
`)
configuredMissing := filepath.Join(root, "configured-but-replaced.yml")
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", configuredMissing))
args := []string{
"configuration", "sources", "--config", configPath,
"--promptkit-source", mainPath,
"--promptkit-source", directory,
}
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
t.Fatalf("framework sources command failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
var sources []frameworkConfigurationSource
if err := json.Unmarshal([]byte(stdout), &sources); err != nil {
t.Fatalf("decode framework source DTOs: %v\n%s", err, stdout)
}
var rawSources []map[string]any
if err := json.Unmarshal([]byte(stdout), &rawSources); err != nil {
t.Fatalf("decode framework source fields: %v", err)
}
mainIndex := findFrameworkSource(t, sources, mainPath)
fragmentIndex := findFrameworkSource(t, sources, fragmentPath)
directoryIndex := findFrameworkSource(t, sources, directoryPath)
main := sources[mainIndex]
fragment := sources[fragmentIndex]
directorySource := sources[directoryIndex]
if !(mainIndex < fragmentIndex && fragmentIndex < directoryIndex) {
t.Fatalf("captured source order changed: main=%d fragment=%d directory=%d", mainIndex, fragmentIndex, directoryIndex)
}
if main.ID == "" || main.Label == "" || main.Layer != configkit.LayerUnderlay || main.Kind != configkit.SourceKindFile || main.UnderlayIndex < 1 {
t.Fatalf("explicit file source fields are incomplete: %+v", main)
}
for _, key := range []string{"id", "label", "layer", "kind", "underlay_index", "path"} {
if _, exists := rawSources[mainIndex][key]; !exists {
t.Fatalf("source DTO omitted %q: %#v", key, rawSources[mainIndex])
}
}
for _, key := range []string{"ID", "Label", "Layer", "Kind", "UnderlayIndex", "filesystem"} {
if _, exists := rawSources[mainIndex][key]; exists {
t.Fatalf("source DTO exposed implementation field %q: %#v", key, rawSources[mainIndex])
}
}
if fragment.Layer != configkit.LayerUnderlay || fragment.Kind != configkit.SourceKindFile || fragment.ImportedFrom == nil ||
fragment.ImportedFrom.SourceID != main.ID || fragment.ImportedFrom.Line < 1 || fragment.ImportedFrom.Column < 1 {
t.Fatalf("import source fields are incomplete: %+v", fragment)
}
if importedFrom, ok := rawSources[fragmentIndex]["imported_from"].(map[string]any); !ok || importedFrom["source_id"] != main.ID || importedFrom["line"] == nil || importedFrom["column"] == nil {
t.Fatalf("import location DTO fields are incomplete: %#v", rawSources[fragmentIndex])
}
if directorySource.Layer != configkit.LayerUnderlay || directorySource.Kind != configkit.SourceKindFile || directorySource.UnderlayIndex <= main.UnderlayIndex {
t.Fatalf("directory-selected source fields/order are incomplete: %+v (main %+v)", directorySource, main)
}
}
func TestFrameworkConfigurationShowFailsBeforeOutputForInvalidFinalDefinition(t *testing.T) {
root := t.TempDir()
frameworkFile := writeFrameworkSourceFile(t, root, "invalid.yml", `prompts:
- id: unused
messages:
- role: user
content: "{{"
output:
format: text
validation_mode: none
`)
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", configPath})
if code != ExitOK || stdout != "" || stderr != "" {
t.Fatalf("application validation loaded framework definitions: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
for _, args := range [][]string{
{"configuration", "validate", "--config", configPath},
{"configuration", "show", "--config", configPath},
{"configuration", "sources", "--config", configPath},
{"configuration", "explain", "--config", configPath, "--path", ""},
} {
code, stdout, stderr = runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid final framework definition was reported as success: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
}
func TestFrameworkConfigurationCommandsRejectExtraArgumentsAndReportWriterErrors(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"configuration"},
{"configuration", "explain"},
{"configuration", "show", "unexpected"},
{"configuration", "sources", "--format", "json"},
} {
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid inspection arguments were accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
writerError := errors.New("synthetic framework output failure")
var stderr bytes.Buffer
code := inspectCommand([]string{"configuration", "show", "--config", configPath}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
t.Fatalf("effective-output writer failure was not reported: code=%d stderr=%q", code, stderr.String())
}
stderr.Reset()
code = inspectCommand([]string{"configuration", "explain", "--config", configPath, "--path", ""}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
t.Fatalf("explanation writer failure was not reported: code=%d stderr=%q", code, stderr.String())
}
}
func TestFrameworkConfigurationExplainUsesLiteralJSONPointersAndNativeHistory(t *testing.T) {
root := t.TempDir()
lower := writeFrameworkSourceFile(t, root, "lower.yml", `collections:
support:
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: backend-token-displaced-canary
password: backend-password-canary
visible_option: backend-visible-value
provider.option: dotted-visible-value
profiles:
- id: summary
backend: support/local
model: lower-model
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: profile-token-displaced-canary
password: profile-password-displaced-canary
visible_option: profile-visible-value
provider.option: dotted-visible-value
prompts:
- id: explainable
default_profile: support/summary
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
upperRoot := t.TempDir()
upper := writeFrameworkSourceFile(t, upperRoot, "upper.yml", `collections:
support:
backends:
- id: local
endpoint: http://127.0.0.1:1/v1
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: backend-token-effective-canary
password: backend-password-effective-canary
visible_option: backend-visible-value
provider.option: dotted-visible-value
profiles:
- id: summary
backend: support/local
model: effective-model
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
extra_params:
token: profile-token-effective-canary
password: profile-password-effective-canary
visible_option: profile-visible-value
provider.option: dotted-visible-value
`)
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n - file: %q\n", lower, upper))
const secret = "environment-value-must-never-be-emitted"
t.Setenv("SCRIPTORIUM_EXPLAIN_REFERENCE", secret)
for _, tc := range []struct {
path string
want []string
unwant []string
}{
{
path: "/collections/support/backends/0/extra_params/token",
unwant: []string{"backend-token-displaced-canary", "backend-token-effective-canary"},
},
{
path: "/collections/support/backends/0/extra_params/password",
unwant: []string{"backend-password-displaced-canary", "backend-password-effective-canary"},
},
{
path: "/collections/support/profiles/0/model",
want: []string{"effective-model", "lower-model"},
unwant: []string{},
},
{
path: "/collections/support/profiles/0/extra_params/token",
unwant: []string{"profile-token-displaced-canary", "profile-token-effective-canary"},
},
{
path: "/collections/support/profiles/0/extra_params/password",
unwant: []string{"profile-password-displaced-canary", "profile-password-effective-canary"},
},
{
path: "/collections/support/profiles/0/api_key_env",
want: []string{"SCRIPTORIUM_EXPLAIN_REFERENCE"},
unwant: []string{secret},
},
{
path: "/collections/support/profiles/0/extra_params/provider.option",
want: []string{"dotted-visible-value"},
unwant: []string{"provider/option"},
},
{
path: "",
want: []string{"profiles:", "effective-model"},
unwant: []string{"profile-token-displaced-canary", "backend-password-effective-canary"},
},
} {
args := []string{"configuration", "explain", "--config", configPath, "--path", tc.path}
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitOK || stderr != "" {
t.Fatalf("explain %q failed: code=%d stdout=%q stderr=%q", tc.path, code, stdout, stderr)
}
if stdout == "" {
t.Fatalf("explain %q produced no output", tc.path)
}
for _, value := range tc.want {
if !strings.Contains(stdout, value) {
t.Errorf("explain %q omitted %q: %s", tc.path, value, stdout)
}
}
for _, value := range tc.unwant {
if strings.Contains(stdout, value) {
t.Errorf("explain %q disclosed %q: %s", tc.path, value, stdout)
}
}
if strings.Contains(stdout, secret) {
t.Errorf("explain %q disclosed the resolved environment value: %s", tc.path, stdout)
}
}
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{"configuration", "show", "--config", configPath})
if code != ExitOK || stderr != "" {
t.Fatalf("framework show failed: code=%d stderr=%q", code, stderr)
}
for _, value := range []string{"SCRIPTORIUM_EXPLAIN_REFERENCE", "backend-visible-value", "profile-visible-value", "dotted-visible-value"} {
if !strings.Contains(stdout, value) {
t.Errorf("effective report omitted visible configuration %q: %s", value, stdout)
}
}
for _, value := range []string{
"backend-token-displaced-canary", "backend-token-effective-canary",
"backend-password-displaced-canary", "backend-password-effective-canary",
"profile-token-displaced-canary", "profile-token-effective-canary",
"profile-password-displaced-canary", "profile-password-effective-canary", secret,
} {
if strings.Contains(stdout, value) {
t.Errorf("effective report disclosed classified or resolved value %q: %s", value, stdout)
}
}
profileCode, profileOutput, profileError := runCLICommand(t, inspectCommand, []string{
"profile", "--config", configPath, "--profile", "support/summary", "--format", "json",
})
if profileCode != ExitOK || profileError != "" {
t.Fatalf("existing profile inspection failed: code=%d stdout=%q stderr=%q", profileCode, profileOutput, profileError)
}
var profile renderformat.ProfileInspection
if err := json.Unmarshal([]byte(profileOutput), &profile); err != nil {
t.Fatalf("decode profile inspection: %v\n%s", err, profileOutput)
}
if pointerValue(profile.ProfileModelParams.Model) != "effective-model" || pointerValue(profile.ProfileModelParams.ExtraParams)["visible_option"] != "profile-visible-value" {
t.Fatalf("existing profile inspection changed configured provider data: %+v", profile.ProfileModelParams)
}
if strings.Contains(profileOutput, secret) {
t.Fatalf("profile inspection emitted resolved environment data: %s", profileOutput)
}
code, rendered, stderr := runCLICommand(t, renderCommand, []string{"--config", configPath, "--prompt", "support/explainable"})
if code != ExitOK || stderr != "" {
t.Fatalf("prepared output failed: code=%d stdout=%q stderr=%q", code, rendered, stderr)
}
if !strings.Contains(rendered, "profile-visible-value") || strings.Contains(rendered, secret) {
t.Fatalf("prepared output did not preserve ordinary provider data or disclosed environment data: %s", rendered)
}
}
func TestFrameworkConfigurationExplainRejectsMissingMalformedAndUnknownPaths(t *testing.T) {
configPath := writeAppConfigFile(t, "{}\n")
for _, args := range [][]string{
{"configuration", "explain"},
{"configuration", "explain", "--path", "not-a-pointer"},
{"configuration", "explain", "--path", "/profiles/team~2summary/model"},
{"configuration", "explain", "--config", configPath, "--path", "/profiles/missing/model"},
} {
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
if code != ExitRuntimeError || stdout != "" || stderr == "" {
t.Fatalf("invalid explanation path was accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
}
}
}
func findFrameworkSource(t *testing.T, sources []frameworkConfigurationSource, path string) int {
t.Helper()
for index, source := range sources {
if filepath.Clean(source.Path) == filepath.Clean(path) {
return index
}
}
t.Fatalf("selected source path %q was not present: %+v", path, sources)
return -1
}
type frameworkConfigurationFailingWriter struct {
err error
}
func (writer frameworkConfigurationFailingWriter) Write([]byte) (int, error) {
return 0, writer.err
}

View File

@@ -1,17 +0,0 @@
package cli
import (
"fmt"
"io"
"go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
func writeFrameworkDiagnostics(output io.Writer, engine *promptkit.Engine) error {
view, err := engine.InspectConfiguration()
if err != nil {
return fmt.Errorf("inspect captured PromptKit diagnostics: %w", err)
}
return configkit.WriteDiagnostics(output, view.Diagnostics())
}

View File

@@ -1,386 +0,0 @@
package cli
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"testing"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
configkit "go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
func TestFrameworkSourceGroupsComposeAndValidateEffectiveDefinitions(t *testing.T) {
conflicting := t.TempDir()
writeFrameworkSourceFile(t, conflicting, "first.yml", `profiles:
- id: selected
model: first
`)
writeFrameworkSourceFile(t, conflicting, "second.yml", `profiles:
- id: selected
model: second
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{Directory: conflicting}}}); err == nil {
t.Fatal("conflicting files in one source group unexpectedly constructed an engine")
}
lower := t.TempDir()
upper := t.TempDir()
empty := t.TempDir()
writeFrameworkSourceFile(t, lower, "lower.yml", `profiles:
- id: selected
endpoint: not-a-url
model: lower-model
temperature: 10
`)
writeFrameworkSourceFile(t, upper, "upper.yml", `profiles:
- id: selected
endpoint: http://127.0.0.1:1/v1
model: upper-model
temperature: 0.5
`)
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{
{Directory: lower}, {Directory: empty}, {Directory: upper},
}, "")
inspection, err := engine.InspectProfile(context.Background(), "selected")
if err != nil {
t.Fatalf("inspect profile after later source replacement: %v", err)
}
if pointerValue(inspection.Endpoint) != "http://127.0.0.1:1/v1" || pointerValue(inspection.Model) != "upper-model" || pointerValue(inspection.Temperature) != 0.5 {
t.Fatalf("later source did not replace lower values before validation: %+v", inspection)
}
}
func TestFrameworkSourceConstructionUsesCombinedDocumentAndRejectsInvalidSources(t *testing.T) {
root := t.TempDir()
combined := filepath.Join(root, "combined.yml")
writeFrameworkSourceFile(t, root, "combined.yml", `backends:
- id: local
endpoint: http://127.0.0.1:1/v1
profiles:
- id: selected
backend: local
model: combined-model
prompts:
- id: combined.prompt
default_profile: selected
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
client := &frameworkSourceCaptureClient{}
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: combined}}, "", promptkit.WithLLMClient(client))
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "combined.prompt"}); err != nil {
t.Fatalf("run definitions from one combined document: %v", err)
}
if len(client.requests) != 1 || client.requests[0].Target.BackendID != "local" || client.requests[0].Target.Model != "combined-model" {
t.Fatalf("combined backend/profile/prompt values did not reach generation: %+v", client.requests)
}
malformed := filepath.Join(root, "malformed.yml")
writeFrameworkSourceFile(t, root, "malformed.yml", "not: [valid\n")
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: malformed}}}); err == nil {
t.Fatal("malformed framework source unexpectedly constructed an engine")
}
invalidUnused := filepath.Join(root, "invalid-unused.yml")
writeFrameworkSourceFile(t, root, "invalid-unused.yml", `prompts:
- id: unused.prompt
messages:
- role: user
content: "{{"
output:
format: text
validation_mode: none
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: invalidUnused}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("invalid unused definition did not fail as framework configuration: %v", err)
}
}
func TestFrameworkRejectsObsoleteVersionKeyedPrompt(t *testing.T) {
root := t.TempDir()
path := writeFrameworkSourceFile(t, root, "obsolete.yml", `prompts:
old-prompt:
"1":
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: path}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
t.Fatalf("obsolete version-keyed definition was not rejected by PromptKit: %v", err)
}
}
func TestFrameworkImportsRetainResourceOriginsAndCaptureAtConstruction(t *testing.T) {
root := t.TempDir()
catalogDir := filepath.Join(root, "catalog")
if err := os.MkdirAll(catalogDir, 0o755); err != nil {
t.Fatal(err)
}
schemaDir := filepath.Join(root, "schemas")
if err := os.MkdirAll(schemaDir, 0o755); err != nil {
t.Fatal(err)
}
mainFile := filepath.Join(root, "main.yml")
writeFrameworkSourceFile(t, root, "main.yml", `config:
imports:
- path: catalog/base.yml
required: true
`)
writeFrameworkSourceFile(t, catalogDir, "base.yml", `profiles:
- id: profile-primary
endpoint: http://127.0.0.1:1/v1
model: imported-model
prompts:
- id: team-prompt
default_profile: profile-primary
inputs:
- name: source
required: true
messages:
- role: user
content_file: ./body.md
- role: user
content: 'Artifact: {{input "source"}}'
output:
format: json
validation_mode: json_schema
schema_path: response.schema.json
`)
writeFrameworkSourceFile(t, catalogDir, "body.md", "Captured message for {{.audience}}")
artifactPath := filepath.Join(root, "request-artifact.txt")
writeFrameworkSourceFile(t, root, "request-artifact.txt", "initial artifact")
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"initial schema"}`)
selections := []appconfig.SourceSelection{{File: mainFile}}
engine := newFrameworkSourceTestEngine(t, selections, schemaDir)
writeFrameworkSourceFile(t, catalogDir, "body.md", "edited message for {{.audience}}")
writeFrameworkSourceFile(t, root, "request-artifact.txt", "prepared artifact")
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"prepared schema"}`)
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "team-prompt",
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
Vars: map[string]string{"audience": "reader"},
})
if err != nil {
t.Fatalf("prepare inherited imported prompt: %v", err)
}
if len(prepared.Messages) != 2 || prepared.Messages[0].Content != "Captured message for reader" || prepared.Messages[1].Content != "Artifact: prepared artifact" {
t.Fatalf("resource origin or capture/preparation behavior changed: %+v", prepared.Messages)
}
if prepared.SelectedProfileID != "profile-primary" {
t.Fatalf("profile ID was not preserved: %+v", prepared)
}
if prepared.StructuredOutput == nil || prepared.StructuredOutput.JSONSchema == nil || prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)["title"] != "prepared schema" {
t.Fatalf("schema was not loaded during preparation: %+v", prepared.StructuredOutput)
}
newEngine := newFrameworkSourceTestEngine(t, selections, schemaDir)
newPrepared, err := newEngine.Prepare(context.Background(), promptkit.RunRequest{
PromptID: "team-prompt",
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
Vars: map[string]string{"audience": "reader"},
})
if err != nil {
t.Fatalf("prepare with newly captured message: %v", err)
}
if newPrepared.Messages[0].Content != "edited message for reader" {
t.Fatalf("new engine did not observe the message edit: %+v", newPrepared.Messages[0])
}
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\nschema_dir: %q\n", mainFile, schemaDir))
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
"--config", configPath,
"--prompt", "team-prompt",
"--input", "source=" + artifactPath,
"--var", "audience=reader",
"--format", "json",
})
if code != ExitOK {
t.Fatalf("render exact literal IDs through CLI: code=%d stderr=%q", code, stderr)
}
var rendered promptkit.PreparedRun
if err := json.Unmarshal([]byte(stdout), &rendered); err != nil {
t.Fatalf("decode rendered literal-ID fixture: %v", err)
}
if rendered.PromptID != "team-prompt" || rendered.SelectedProfileID != "profile-primary" {
t.Fatalf("CLI changed exact prompt/profile IDs: %+v", rendered)
}
}
func TestNativeProfileInheritanceAndPresenceReachTheClientBoundary(t *testing.T) {
root := t.TempDir()
frameworkFile := filepath.Join(root, "framework.yml")
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
- id: first
endpoint: https://first.example/v1
api_key_env: FIRST_BACKEND_KEY
extra_params:
source: first
nested:
first_backend: true
- id: second
endpoint: https://second.example/v1
api_key_env: SECOND_BACKEND_KEY
extra_params:
source: second
nested:
second_backend: true
profiles:
- id: parent
backend: first
model: parent-model
temperature: 0.7
top_p: 0.6
max_tokens: 45
timeout_seconds: 30
service_tier: flex
extra_params:
nested:
parent_profile: true
retained: "yes"
- id: backend-child
extends: {id: parent}
backend: second
model: child-model
- id: presence-child
extends: {id: parent}
backend: second
api_key_env: null
temperature: null
max_tokens: 0
timeout_seconds: 0
extra_params:
nested:
explicit_false: false
- id: map-null
backend: second
model: map-model
extra_params: null
- id: map-empty
backend: second
model: map-model
extra_params: {}
prompts:
- id: execution.prompt
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
client := &frameworkSourceCaptureClient{}
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: frameworkFile}}, "", promptkit.WithLLMClient(client))
for _, profileID := range []string{"backend-child", "presence-child", "map-null", "map-empty"} {
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "execution.prompt", ProfileID: profileID}); err != nil {
t.Fatalf("run profile %q: %v", profileID, err)
}
}
if len(client.requests) != 4 {
t.Fatalf("expected four provider-bound requests, got %d", len(client.requests))
}
backendChild := client.requests[0].Target
wantBackendParams := map[string]any{"source": "second", "nested": map[string]any{
"second_backend": true, "parent_profile": true, "retained": "yes",
}}
if backendChild.BackendID != "second" || backendChild.Endpoint != "https://second.example/v1" || backendChild.APIKeyEnv != "SECOND_BACKEND_KEY" || backendChild.Model != "child-model" ||
backendChild.Temperature != 0.7 || backendChild.TopP != 0.6 || backendChild.MaxTokens != 45 || backendChild.TimeoutSeconds != 30 || backendChild.ServiceTier != "flex" || !reflect.DeepEqual(backendChild.ExtraParams, wantBackendParams) {
t.Fatalf("backend change lost backend-derived or explicit inherited values: %+v", backendChild)
}
presenceRequest := client.requests[1]
presenceTarget := presenceRequest.Target
if presenceTarget.APIKeyEnv != "SECOND_BACKEND_KEY" || presenceTarget.Temperature != 0 || presenceTarget.MaxTokens != 0 || presenceTarget.TimeoutSeconds != 0 ||
!presenceRequest.TargetPresence.MaxTokens || !presenceRequest.TargetPresence.TimeoutSeconds || presenceRequest.TargetPresence.Temperature || !presenceRequest.TargetPresence.TopP {
t.Fatalf("explicit null/zero presence did not reach the injected client: target=%+v presence=%+v", presenceTarget, presenceRequest.TargetPresence)
}
wantPresenceParams := map[string]any{"source": "second", "nested": map[string]any{
"second_backend": true, "parent_profile": true, "retained": "yes", "explicit_false": false,
}}
if !reflect.DeepEqual(presenceTarget.ExtraParams, wantPresenceParams) {
t.Fatalf("recursive provider-map retention changed: %#v", presenceTarget.ExtraParams)
}
if !reflect.DeepEqual(client.requests[2].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
t.Fatalf("whole-map null did not fall back to backend defaults: %#v", client.requests[2].Target.ExtraParams)
}
if !reflect.DeepEqual(client.requests[3].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
t.Fatalf("empty provider map unexpectedly cleared backend defaults: %#v", client.requests[3].Target.ExtraParams)
}
}
func TestCLICompletesPartialFrameworkProfileWithRequestOverrides(t *testing.T) {
root := t.TempDir()
frameworkFile := writeFrameworkSourceFile(t, root, "partial.yml", `profiles:
- id: partial
prompts:
- id: partial.prompt
default_profile: partial
messages:
- role: user
content: hello
output:
format: text
validation_mode: none
`)
provider := newTestLLMServer("completed partial profile", nil)
defer provider.Close()
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
code, stdout, stderr := runCLICommand(t, runCommand, []string{
"--config", configPath,
"--prompt", "partial.prompt",
"--llm-base-url", provider.URL + "/v1",
"--model", "request-model",
})
if code != ExitOK || stdout != "completed partial profile" {
t.Fatalf("request overrides did not complete the partial profile: code=%d stdout=%q stderr=%q", code, stdout, stderr)
}
}
func newFrameworkSourceTestEngine(t *testing.T, selections []appconfig.SourceSelection, schemaDir string, options ...promptkit.Option) *promptkit.Engine {
t.Helper()
engine, err := newEngine(engineSettings{promptkitSources: selections, schemaDir: schemaDir}, options...)
if err != nil {
var loadError *configkit.LoadError
if errors.As(err, &loadError) {
t.Fatalf("construct engine from selected framework sources: %v (%+v)", err, loadError.Diagnostics())
}
t.Fatalf("construct engine from selected framework sources: %v", err)
}
return engine
}
func writeFrameworkSourceFile(t *testing.T, directory, name, contents string) string {
t.Helper()
if err := os.MkdirAll(directory, 0o755); err != nil {
t.Fatalf("create framework source directory: %v", err)
}
path := filepath.Join(directory, name)
if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
t.Fatalf("write framework source %q: %v", path, err)
}
return path
}
type frameworkSourceCaptureClient struct {
requests []promptkit.GenerateRequest
}
func (client *frameworkSourceCaptureClient) Generate(_ context.Context, request promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
client.requests = append(client.requests, request)
return &promptkit.GenerateResponse{Content: "ok"}, nil
}

View File

@@ -1,212 +0,0 @@
package cli
import (
"context"
"errors"
"flag"
"fmt"
"io"
"path/filepath"
"strings"
appformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
"go.maximumdirect.net/configkit"
)
type promptInspectionConfig struct {
configPath, promptID, schemaDir, outputPath string
configOverlay string
configExplicit bool
outputFormat appformat.OutputFormat
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
type profileInspectionConfig struct {
configPath, profileID, schemaDir, outputPath string
configOverlay string
configExplicit bool
outputFormat appformat.OutputFormat
overrides []configkit.Override
diagnostics []configkit.Diagnostic
}
func inspectCommand(args []string, stdout, stderr io.Writer) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "inspect parse error: inspection mode is required")
return ExitRuntimeError
}
if args[0] == "profile" {
return inspectProfileCommand(args[1:], stdout, stderr)
}
if args[0] == "configuration" {
return inspectFrameworkConfigurationCommand(args[1:], stdout, stderr)
}
if args[0] != "prompt" {
fmt.Fprintln(stderr, "inspect parse error: unknown inspection mode")
return ExitRuntimeError
}
cfg, err := parsePromptInspectionArgs(args[1:])
if err != nil {
writeApplicationError(stderr, "inspect parse", err)
return ExitRuntimeError
}
settings, err := resolveAppSettingsForPromptInspection(cfg)
if err != nil {
writeApplicationError(stderr, "inspect", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
data, err := appformat.FormatPromptInspection(inspection, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func inspectProfileCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseProfileInspectionArgs(args)
if err != nil {
writeApplicationError(stderr, "inspect parse", err)
return ExitRuntimeError
}
settings, err := resolveAppSettingsForProfileInspection(cfg)
if err != nil {
writeApplicationError(stderr, "inspect", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
engine, err := newEngine(settings)
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
inspection, err := engine.InspectProfile(context.Background(), cfg.profileID)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
data, err := appformat.FormatProfileInspection(inspection, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "inspect error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseProfileInspectionArgs(args []string) (*profileInspectionConfig, error) {
cfg := &profileInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
fs := flag.NewFlagSet("inspect profile", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
var promptkitSources promptKitSourceFlagValue
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
fs.StringVar(&cfg.profileID, "profile", "", "profile ID to inspect")
rawFormat := ""
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.overrides = applicationOverrides(fs)
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
if strings.TrimSpace(cfg.profileID) == "" {
return nil, errors.New("--profile is required")
}
format, err := appformat.ParseOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
return cfg, nil
}
func resolveAppSettingsForProfileInspection(cfg *profileInspectionConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
}
func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
cfg := &promptInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
fs := flag.NewFlagSet("inspect prompt", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
var promptkitSources promptKitSourceFlagValue
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to inspect")
rawFormat := ""
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
cfg.configExplicit = flagWasSet(fs, "config")
cfg.overrides = applicationOverrides(fs)
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
if strings.TrimSpace(cfg.promptID) == "" {
return nil, errors.New("--prompt is required")
}
format, err := appformat.ParseOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
return cfg, nil
}
func resolveAppSettingsForPromptInspection(cfg *promptInspectionConfig) (engineSettings, error) {
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
if err != nil {
return engineSettings{}, err
}
cfg.diagnostics = diagnostics
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
}

View File

@@ -1,636 +0,0 @@
package cli
import (
"context"
"errors"
"flag"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"time"
httpadapter "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
"go.maximumdirect.net/commandkit"
"go.maximumdirect.net/configkit"
"go.maximumdirect.net/promptkit"
)
const (
ExitOK = 0
ExitRuntimeError = 1
ExitValidationFailed = 2
)
type runConfig struct {
configPath string
promptkitSources []appconfig.SourceSelection
promptID string
profileID string
sessionID string
inputRaw listFlag
varRaw listFlag
outputPath string
executionReportPath string
llmBaseURL string
apiKeyEnv string
model string
temperature float64
maxTokens int
topP float64
reasoningEffort string
schemaDir string
timeout time.Duration
runTimeout time.Duration
transientRetries int
configuredTransientRetries *int
diagnostics []configkit.Diagnostic
configOverlay string
defaultRenderFormat renderformat.PreparedRunOutputFormat
llmBaseURLSet bool
apiKeyEnvSet bool
modelSet bool
temperatureSet bool
maxTokensSet bool
topPSet bool
reasoningEffortSet bool
timeoutSet bool
transientRetriesSet bool
}
type renderConfig struct {
runConfig
outputFormat renderformat.PreparedRunOutputFormat
}
type serveConfig struct {
configPath string
addr string
promptkitSources []appconfig.SourceSelection
schemaDir string
artifactRoot string
maxRequestBytes int64
maxArtifactBytes int64
maxResponseBytes int64
transientRetries *int
diagnostics []configkit.Diagnostic
configOverlay string
}
type commonCommandSettings struct {
promptkitSources []appconfig.SourceSelection
schemaDir string
serverAddr string
artifactRoot string
maxRequestBytes int64
maxArtifactBytes int64
maxResponseBytes int64
defaultRenderFormat renderformat.PreparedRunOutputFormat
transientRetries *int
diagnostics []configkit.Diagnostic
}
type listFlag []string
func (l *listFlag) String() string {
return strings.Join(*l, ",")
}
func (l *listFlag) Set(v string) error {
*l = append(*l, v)
return nil
}
func Run(args []string, stdout, stderr io.Writer) int {
return Execute(context.Background(), commandkit.Identity{Name: "scriptorium", Version: "dev"}, args, stdout, stderr).Status
}
func runCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseRunArgs(args)
if err != nil {
writeApplicationError(stderr, "run parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
req, err := buildRunRequestFromConfig(cfg)
if err != nil {
fmt.Fprintf(stderr, "run parse error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.engineSettings())
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
return executeRun(cfg, req, engine.Run, stdout, stderr)
}
func renderCommand(args []string, stdout, stderr io.Writer) int {
cfg, err := parseRenderArgs(args)
if err != nil {
writeApplicationError(stderr, "render parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
req, err := buildRunRequestFromConfig(&cfg.runConfig)
if err != nil {
fmt.Fprintf(stderr, "render parse error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.runConfig.engineSettings())
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
prepared, prepErr := engine.Prepare(context.Background(), req)
if prepErr != nil {
fmt.Fprintf(stderr, "render error: %v\n", prepErr)
return ExitRuntimeError
}
out, err := renderformat.FormatPreparedRun(prepared, cfg.outputFormat)
if err != nil {
fmt.Fprintf(stderr, "render format error: %v\n", err)
return ExitRuntimeError
}
if err := writeOutput(stdout, cfg.outputPath, out); err != nil {
fmt.Fprintf(stderr, "output write error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func serveCommand(args []string, stderr io.Writer) int {
cfg, err := parseServeArgs(args)
if err != nil {
writeApplicationError(stderr, "serve parse", err)
return ExitRuntimeError
}
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
return ExitRuntimeError
}
artifactReader, err := httpadapter.NewRestrictedArtifactReader(cfg.artifactRoot, cfg.maxArtifactBytes)
if err != nil {
fmt.Fprintf(stderr, "artifact root error: %v\n", err)
return ExitRuntimeError
}
engine, err := newEngine(cfg.engineSettings(), promptkit.WithArtifactReader(artifactReader))
if err != nil {
writeApplicationError(stderr, "engine", err)
return ExitRuntimeError
}
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
return ExitRuntimeError
}
h := httpadapter.NewHandlerWithOptions(engine, httpadapter.HandlerOptions{
MaxRequestBytes: cfg.maxRequestBytes,
MaxResponseBytes: cfg.maxResponseBytes,
})
srv := &http.Server{
Addr: cfg.addr,
Handler: h,
ReadHeaderTimeout: defaults.HTTPReadHeaderTimeoutDefault,
}
fmt.Fprintf(stderr, "serving on %s\n", cfg.addr)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
fmt.Fprintf(stderr, "server error: %v\n", err)
return ExitRuntimeError
}
return ExitOK
}
func parseRunArgs(args []string) (*runConfig, error) {
cfg := &runConfig{}
fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
registerExecutionRequestFlags(fs, cfg)
fs.DurationVar(&cfg.runTimeout, "run-timeout", 0, "whole-run execution deadline")
fs.StringVar(&cfg.executionReportPath, "execution-report", "", "JSON execution report file path")
if err := fs.Parse(args); err != nil {
return nil, err
}
if err := finalizeExecutionRequestConfig(fs, cfg); err != nil {
return nil, err
}
if cfg.runTimeout < 0 {
return nil, errors.New("--run-timeout must be greater than or equal to 0")
}
if flagWasSet(fs, "execution-report") {
if cfg.executionReportPath == "" {
return nil, errors.New("--execution-report requires a nonempty path")
}
cfg.executionReportPath = filepath.Clean(cfg.executionReportPath)
if cfg.outputPath != "" {
collision, err := outputPathsCollide(cfg.outputPath, cfg.executionReportPath)
if err != nil {
return nil, err
}
if collision {
return nil, errors.New("--out and --execution-report select the same file")
}
}
}
return cfg, nil
}
func outputPathsCollide(first, second string) (bool, error) {
firstAbs, err := filepath.Abs(first)
if err != nil {
return false, err
}
secondAbs, err := filepath.Abs(second)
if err != nil {
return false, err
}
if firstAbs == secondAbs {
return true, nil
}
firstInfo, firstErr := os.Stat(firstAbs)
secondInfo, secondErr := os.Stat(secondAbs)
return firstErr == nil && secondErr == nil && os.SameFile(firstInfo, secondInfo), nil
}
func parseRenderArgs(args []string) (*renderConfig, error) {
cfg := &renderConfig{
outputFormat: renderformat.DefaultPreparedRunOutputFormat,
}
fs := flag.NewFlagSet("render", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
registerExecutionRequestFlags(fs, &cfg.runConfig)
var rawFormat string
fs.StringVar(&rawFormat, "format", "", "render output format (text|json)")
if err := fs.Parse(args); err != nil {
return nil, err
}
if err := finalizeExecutionRequestConfig(fs, &cfg.runConfig); err != nil {
return nil, err
}
if flagWasSet(fs, "format") {
format, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
if err != nil {
return nil, err
}
cfg.outputFormat = format
} else {
cfg.outputFormat = cfg.defaultRenderFormat
}
return cfg, nil
}
func parseServeArgs(args []string) (*serveConfig, error) {
cfg := &serveConfig{}
fs := flag.NewFlagSet("serve", flag.ContinueOnError)
fs.SetOutput(io.Discard)
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
fs.StringVar(&cfg.addr, "addr", "", "HTTP listen address")
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
fs.Int64Var(&cfg.maxArtifactBytes, "max-artifact-bytes", 0, "maximum HTTP file artifact bytes; 0 disables the limit")
fs.Int64Var(&cfg.maxResponseBytes, "max-response-bytes", 0, "maximum HTTP response body bytes; 0 disables the limit")
if err := fs.Parse(args); err != nil {
return nil, err
}
if fs.NArg() > 0 {
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
if err != nil {
return nil, err
}
cfg.promptkitSources = settings.promptkitSources
cfg.schemaDir = settings.schemaDir
cfg.addr = settings.serverAddr
cfg.artifactRoot = settings.artifactRoot
cfg.maxRequestBytes = settings.maxRequestBytes
cfg.maxArtifactBytes = settings.maxArtifactBytes
cfg.maxResponseBytes = settings.maxResponseBytes
cfg.transientRetries = settings.transientRetries
cfg.diagnostics = settings.diagnostics
return cfg, nil
}
func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
fs.StringVar(&cfg.sessionID, "session-id", "", "optional session ID")
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1")
fs.StringVar(&cfg.apiKeyEnv, "api-key-env", "", "environment variable name containing API key")
fs.StringVar(&cfg.model, "model", "", "model name")
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
fs.Float64Var(&cfg.topP, "top-p", 0, "optional top_p override")
fs.StringVar(&cfg.reasoningEffort, "reasoning-effort", "", "optional reasoning effort override")
fs.DurationVar(&cfg.timeout, "timeout", 0, "LLM request timeout")
fs.IntVar(&cfg.transientRetries, "transient-retries", 0, "additional transient generation attempts")
fs.StringVar(&cfg.promptID, "prompt-id", "", "deprecated alias for --prompt")
fs.StringVar(&cfg.profileID, "profile-id", "", "deprecated alias for --profile")
}
func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
if fs.NArg() > 0 {
return fmt.Errorf("unexpected positional args: %v", fs.Args())
}
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
if err != nil {
return err
}
cfg.promptkitSources = settings.promptkitSources
cfg.schemaDir = settings.schemaDir
cfg.defaultRenderFormat = settings.defaultRenderFormat
cfg.configuredTransientRetries = settings.transientRetries
cfg.diagnostics = settings.diagnostics
if strings.TrimSpace(cfg.promptID) == "" {
return errors.New("--prompt is required")
}
if cfg.outputPath != "" {
cfg.outputPath = filepath.Clean(cfg.outputPath)
}
cfg.llmBaseURLSet = flagWasSet(fs, "llm-base-url")
cfg.apiKeyEnvSet = flagWasSet(fs, "api-key-env")
cfg.modelSet = flagWasSet(fs, "model")
cfg.temperatureSet = flagWasSet(fs, "temperature")
cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
cfg.topPSet = flagWasSet(fs, "top-p")
cfg.reasoningEffortSet = flagWasSet(fs, "reasoning-effort")
cfg.timeoutSet = flagWasSet(fs, "timeout")
cfg.transientRetriesSet = flagWasSet(fs, "transient-retries")
if cfg.transientRetriesSet && cfg.transientRetries < 0 {
return errors.New("--transient-retries must be greater than or equal to 0")
}
return nil
}
func registerConfigSelectionFlags(fs *flag.FlagSet, target, overlay *string) {
fs.StringVar(
target,
"config",
"",
fmt.Sprintf(
"application config path (default search: %s, then %s)",
appconfig.DefaultConfigPathLocal,
appconfig.DefaultConfigPath,
),
)
fs.StringVar(overlay, "config-overlay", "", "selected application configuration overlay")
}
func resolveCommonSettings(fs *flag.FlagSet, configPath, overlay string) (commonCommandSettings, error) {
settings, diagnostics, err := loadApplicationSettings(configPath, overlay, flagWasSet(fs, "config"), applicationOverrides(fs))
if err != nil {
return commonCommandSettings{}, err
}
return commonCommandSettings{
promptkitSources: settings.PromptKitSources,
schemaDir: settings.SchemaDir,
serverAddr: settings.ServerAddr,
artifactRoot: settings.ArtifactRoot,
maxRequestBytes: settings.MaxRequestBytes,
maxArtifactBytes: settings.MaxArtifactBytes,
maxResponseBytes: settings.MaxResponseBytes,
defaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
transientRetries: settings.TransientRetries,
diagnostics: diagnostics,
}, nil
}
type engineSettings struct {
promptkitSources []appconfig.SourceSelection
schemaDir string
transientRetries *int
}
func (c runConfig) engineSettings() engineSettings {
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.configuredTransientRetries}
}
func (c serveConfig) engineSettings() engineSettings {
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.transientRetries}
}
func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
var inputMappings map[string]string
var err error
if len(cfg.inputRaw) > 0 {
inputMappings, err = parseMappings(cfg.inputRaw, false)
if err != nil {
return promptkit.RunRequest{}, fmt.Errorf("input parse error: %w", err)
}
}
varMappings := map[string]string{}
if len(cfg.varRaw) > 0 {
varMappings, err = parseMappings(cfg.varRaw, false)
if err != nil {
return promptkit.RunRequest{}, fmt.Errorf("var parse error: %w", err)
}
}
var inputs map[string]promptkit.ArtifactRef
if len(inputMappings) > 0 {
inputs = make(map[string]promptkit.ArtifactRef, len(inputMappings))
for name, path := range inputMappings {
inputs[name] = promptkit.File(path)
}
}
var modelOverride *promptkit.ExecutionTargetOverride
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.reasoningEffortSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
modelOverride = &promptkit.ExecutionTargetOverride{
Endpoint: cfg.llmBaseURL,
Model: cfg.model,
APIKeyEnv: cfg.apiKeyEnv,
}
if cfg.temperatureSet {
modelOverride.Temperature = &cfg.temperature
}
if cfg.maxTokensSet {
modelOverride.MaxTokens = &cfg.maxTokens
}
if cfg.topPSet {
modelOverride.TopP = &cfg.topP
}
if cfg.reasoningEffortSet {
modelOverride.ReasoningEffort = &cfg.reasoningEffort
}
if cfg.timeoutSet {
timeoutSeconds := int(cfg.timeout.Seconds())
modelOverride.TimeoutSeconds = &timeoutSeconds
}
}
var transientRetries *int
if cfg.transientRetriesSet {
transientRetries = &cfg.transientRetries
}
return promptkit.RunRequest{
PromptID: cfg.promptID,
ProfileID: cfg.profileID,
SessionID: cfg.sessionID,
Inputs: inputs,
Vars: varMappings,
Execution: modelOverride,
TransientRetries: transientRetries,
}, nil
}
func parseMappings(raw []string, allowEmptyValue bool) (map[string]string, error) {
out := make(map[string]string)
for _, entry := range raw {
for _, piece := range strings.Split(entry, ",") {
piece = strings.TrimSpace(piece)
if piece == "" {
continue
}
key, value, err := parseMapping(piece)
if err != nil {
return nil, err
}
if !allowEmptyValue && strings.TrimSpace(value) == "" {
return nil, fmt.Errorf("mapping %q has empty value", piece)
}
out[key] = value
}
}
if len(out) == 0 {
return nil, errors.New("no valid mappings provided")
}
return out, nil
}
func parseMapping(value string) (string, string, error) {
idx := strings.IndexRune(value, '=')
if idx <= 0 {
return "", "", fmt.Errorf("invalid mapping %q, expected name=value", value)
}
key := strings.TrimSpace(value[:idx])
val := strings.TrimSpace(value[idx+1:])
if key == "" {
return "", "", fmt.Errorf("invalid mapping %q, empty name", value)
}
return key, val, nil
}
func flagWasSet(fs *flag.FlagSet, name string) bool {
set := false
fs.Visit(func(f *flag.Flag) {
if f.Name == name {
set = true
}
})
return set
}
func writeOutput(stdout io.Writer, outputPath string, body []byte) error {
if outputPath == "" {
_, err := stdout.Write(body)
return err
}
return os.WriteFile(outputPath, body, 0644)
}
func determineExitCode(runErr error, result *promptkit.RunResult) int {
if runErr != nil {
return ExitRuntimeError
}
if result != nil && result.Validation.Status == promptkit.ValidationFailed {
return ExitValidationFailed
}
return ExitOK
}
func printSummary(stderr io.Writer, res *promptkit.RunResult) {
if res == nil {
return
}
fmt.Fprintf(stderr, "prompt=%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
res.PromptID,
res.SelectedProfileID,
res.ModelName,
res.Validation.Status,
res.Validation.Mode,
len(res.Validation.Errors),
res.RenderedPromptHash,
len(res.InputHashes),
res.Usage.PromptTokens,
res.Usage.CompletionTokens,
res.Usage.TotalTokens,
)
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
}
if res.SelectedBackendID != "" {
fmt.Fprintf(stderr, " backend=%s", res.SelectedBackendID)
}
fmt.Fprintln(stderr)
}
func runErrorMessage(err error) string {
if errors.Is(err, promptkit.ErrCapacityExceeded) {
return "run error: model backend capacity is exhausted"
}
return fmt.Sprintf("run error: %v", err)
}
func printUsage(w io.Writer) {
fmt.Fprintln(w, "usage: scriptorium <run|render|serve|inspect|config> ...")
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]")
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]")
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--addr %s] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
fmt.Fprintln(w, " inspect configuration: scriptorium inspect configuration <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--path JSON_POINTER (required for explain)]")
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [application overrides] [--path /pointer for explain]")
}

File diff suppressed because it is too large Load Diff

View File

@@ -153,7 +153,7 @@ import (
_ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http" _ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/config" _ "gitea.maximumdirect.net/eric/scriptorium/internal/config"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults" _ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
_ "gitea.maximumdirect.net/eric/scriptorium/internal/format" _ "gitea.maximumdirect.net/eric/scriptorium/internal/report"
) )
` `
if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil { if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {

View File

@@ -1,179 +0,0 @@
package format
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"strconv"
"go.maximumdirect.net/promptkit"
)
type PromptInspection struct {
PromptID string `json:"prompt_id"`
PromptHash string `json:"prompt_hash"`
DefaultProfileID *string `json:"default_profile_id,omitempty"`
Inputs *[]PromptInspectionInput `json:"inputs,omitempty"`
OutputContract OutputContract `json:"output_contract"`
}
type PromptInspectionInput struct {
Name string `json:"name"`
Required *bool `json:"required,omitempty"`
ContentType *string `json:"content_type,omitempty"`
Description *string `json:"description,omitempty"`
}
type OutputContract struct {
Format string `json:"format"`
ValidationMode string `json:"validation_mode"`
SchemaPath *string `json:"schema_path,omitempty"`
RepairAttempts *int `json:"repair_attempts,omitempty"`
}
type ProfileInspection struct {
ProfileID string `json:"profile_id"`
ProfileModelParams ProfileModelParams `json:"profile_model_params"`
APIKeyRequired *bool `json:"api_key_required,omitempty"`
}
type ProfileModelParams struct {
BackendID *string `json:"backend_id,omitempty"`
Endpoint *string `json:"endpoint,omitempty"`
Model *string `json:"model,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
ServiceTier *string `json:"service_tier,omitempty"`
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
APIKeyEnv *string `json:"api_key_env,omitempty"`
ExtraParams *map[string]any `json:"extra_params,omitempty"`
}
func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat OutputFormat) ([]byte, error) {
if value == nil {
return nil, errors.New("prompt inspection is nil")
}
dto := PromptInspection{
PromptID: value.PromptID, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
OutputContract: OutputContract{Format: string(value.OutputContract.Format), ValidationMode: string(value.OutputContract.ValidationMode), SchemaPath: value.OutputContract.SchemaPath, RepairAttempts: value.OutputContract.RepairAttempts},
}
if value.Inputs != nil {
inputs := make([]PromptInspectionInput, len(*value.Inputs))
for i, input := range *value.Inputs {
inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
}
dto.Inputs = &inputs
}
switch outputFormat {
case OutputFormatJSON:
data, err := json.MarshalIndent(dto, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
case OutputFormatText:
var b bytes.Buffer
fmt.Fprintf(&b, "prompt_id: %s\nprompt_hash: %s\n", dto.PromptID, dto.PromptHash)
if dto.DefaultProfileID != nil {
fmt.Fprintf(&b, "default_profile_id: %s\n", *dto.DefaultProfileID)
}
if dto.Inputs != nil {
if len(*dto.Inputs) == 0 {
fmt.Fprintln(&b, "inputs: []")
} else {
fmt.Fprintln(&b, "inputs:")
for _, input := range *dto.Inputs {
fmt.Fprintf(&b, " - name: %s\n", input.Name)
if input.Required != nil {
fmt.Fprintf(&b, " required: %t\n", *input.Required)
}
if input.ContentType != nil {
fmt.Fprintf(&b, " content_type: %s\n", *input.ContentType)
}
if input.Description != nil {
fmt.Fprintf(&b, " description: %s\n", strconv.Quote(*input.Description))
}
}
}
}
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode)
if dto.OutputContract.SchemaPath != nil {
fmt.Fprintf(&b, " schema_path: %s\n", *dto.OutputContract.SchemaPath)
}
if dto.OutputContract.RepairAttempts != nil {
fmt.Fprintf(&b, " repair_attempts: %d\n", *dto.OutputContract.RepairAttempts)
}
return b.Bytes(), nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}
func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat OutputFormat) ([]byte, error) {
if value == nil {
return nil, errors.New("profile inspection is nil")
}
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, ProfileModelParams: ProfileModelParams{
BackendID: value.Backend, Endpoint: value.Endpoint, Model: value.Model, Temperature: value.Temperature,
MaxTokens: value.MaxTokens, TopP: value.TopP, TimeoutSeconds: value.TimeoutSeconds, ServiceTier: value.ServiceTier,
ReasoningEffort: value.ReasoningEffort, APIKeyEnv: value.APIKeyEnv, ExtraParams: value.ExtraParams,
}}
switch outputFormat {
case OutputFormatJSON:
data, err := json.MarshalIndent(dto, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
case OutputFormatText:
var b bytes.Buffer
fmt.Fprintf(&b, "profile_id: %s\nprofile_model_params:\n", dto.ProfileID)
p := dto.ProfileModelParams
if p.BackendID != nil {
fmt.Fprintf(&b, " backend_id: %s\n", *p.BackendID)
}
if p.Endpoint != nil {
fmt.Fprintf(&b, " endpoint: %s\n", *p.Endpoint)
}
if p.Model != nil {
fmt.Fprintf(&b, " model: %s\n", *p.Model)
}
if p.Temperature != nil {
fmt.Fprintf(&b, " temperature: %g\n", *p.Temperature)
}
if p.MaxTokens != nil {
fmt.Fprintf(&b, " max_tokens: %d\n", *p.MaxTokens)
}
if p.TopP != nil {
fmt.Fprintf(&b, " top_p: %g\n", *p.TopP)
}
if p.TimeoutSeconds != nil {
fmt.Fprintf(&b, " timeout_seconds: %d\n", *p.TimeoutSeconds)
}
if p.ServiceTier != nil {
fmt.Fprintf(&b, " service_tier: %s\n", *p.ServiceTier)
}
if p.ReasoningEffort != nil {
fmt.Fprintf(&b, " reasoning_effort: %s\n", *p.ReasoningEffort)
}
if p.APIKeyEnv != nil {
fmt.Fprintf(&b, " api_key_env: %s\n", *p.APIKeyEnv)
}
if p.ExtraParams != nil {
paramsJSON, err := json.Marshal(p.ExtraParams)
if err != nil {
return nil, err
}
fmt.Fprintf(&b, " extra_params: %s\n", paramsJSON)
}
if dto.APIKeyRequired != nil {
fmt.Fprintf(&b, "api_key_required: %t\n", *dto.APIKeyRequired)
}
return b.Bytes(), nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}

View File

@@ -1,95 +0,0 @@
package format
import (
"encoding/json"
"strings"
"testing"
"go.maximumdirect.net/promptkit"
)
func ptr[T any](value T) *T { return &value }
func TestPromptInspectionOptionalFieldsAndDisclosure(t *testing.T) {
inputs := []promptkit.PromptInputInspection{{Name: "first", Required: ptr(false), ContentType: ptr(""), Description: ptr("")}, {Name: "second"}}
value := &promptkit.PromptInspection{PromptID: "support/p", PromptHash: "hash", DefaultProfileID: ptr(""), Inputs: &inputs,
Messages: []promptkit.PromptMessageInspection{{Role: "user", Content: "private message"}}, Description: ptr("private description"), SessionID: ptr("private session"),
OutputContract: promptkit.PromptOutputInspection{Format: "text", ValidationMode: promptkit.ValidationNone, SchemaPath: ptr(""), RepairAttempts: ptr(0)}}
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatPromptInspection(value, format)
if err != nil {
t.Fatal(err)
}
s := string(output)
for _, forbidden := range []string{"private message", "private description", "private session", "prompt_version"} {
if strings.Contains(s, forbidden) {
t.Fatalf("inspection exposed %q: %s", forbidden, s)
}
}
if strings.Index(s, "first") > strings.Index(s, "second") {
t.Fatalf("input order changed: %s", s)
}
if !strings.Contains(s, "default_profile_id") || !strings.Contains(s, "repair_attempts") || !strings.Contains(s, "required") {
t.Fatalf("present empty values missing: %s", s)
}
}
value.DefaultProfileID, value.Inputs = nil, nil
value.OutputContract.SchemaPath, value.OutputContract.RepairAttempts = nil, nil
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatPromptInspection(value, format)
if err != nil {
t.Fatal(err)
}
for _, absent := range []string{"default_profile_id", "inputs", "schema_path", "repair_attempts"} {
if strings.Contains(string(output), absent) {
t.Fatalf("absent %s rendered: %s", absent, output)
}
}
}
empty := []promptkit.PromptInputInspection{}
value.Inputs = &empty
output, err := FormatPromptInspection(value, OutputFormatJSON)
if err != nil || !strings.Contains(string(output), `"inputs": []`) {
t.Fatalf("empty inputs lost: %s %v", output, err)
}
if _, err := FormatPromptInspection(nil, OutputFormatText); err == nil {
t.Fatal("nil prompt accepted")
}
}
func TestProfileInspectionPreservesOnlyPresentValues(t *testing.T) {
params := map[string]any{}
value := &promptkit.ProfileInspection{ProfileID: "support/fast", Backend: ptr("backend"), Model: ptr(""), Temperature: ptr(0.0), MaxTokens: ptr(0), APIKeyRequired: ptr(false), ExtraParams: &params}
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
output, err := FormatProfileInspection(value, format)
if err != nil {
t.Fatal(err)
}
s := string(output)
for _, key := range []string{"profile_model_params", "backend_id", "model", "temperature", "max_tokens", "api_key_required", "extra_params"} {
if !strings.Contains(s, key) {
t.Fatalf("present %s missing: %s", key, s)
}
}
for _, key := range []string{"effective_model_params", "endpoint", "api_key_env", "top_p"} {
if strings.Contains(s, key) {
t.Fatalf("absent %s appeared: %s", key, s)
}
}
}
output, err := FormatProfileInspection(value, OutputFormatJSON)
if err != nil {
t.Fatal(err)
}
var decoded map[string]any
if err := json.Unmarshal(output, &decoded); err != nil {
t.Fatal(err)
}
p := decoded["profile_model_params"].(map[string]any)
if p["backend_id"] != "backend" || p["model"] != "" || p["temperature"] != float64(0) || decoded["api_key_required"] != false || len(p["extra_params"].(map[string]any)) != 0 {
t.Fatalf("wrong values: %#v", decoded)
}
if _, err := FormatProfileInspection(nil, OutputFormatText); err == nil {
t.Fatal("nil profile accepted")
}
}

View File

@@ -1,210 +0,0 @@
// Package format formats already-prepared public data for adapters.
package format
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"go.maximumdirect.net/promptkit"
)
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
// OutputFormat selects an application-owned textual or JSON representation.
type OutputFormat string
const (
OutputFormatText OutputFormat = "text"
OutputFormatJSON OutputFormat = "json"
DefaultOutputFormat OutputFormat = OutputFormatText
)
// PreparedRunOutputFormat is the output format for prepared render data.
type PreparedRunOutputFormat = OutputFormat
const (
PreparedRunFormatText = OutputFormatText
PreparedRunFormatJSON = OutputFormatJSON
DefaultPreparedRunOutputFormat = DefaultOutputFormat
)
// PreparedRunFormatter serializes a prepared run without performing use case work.
type PreparedRunFormatter interface {
Format(prepared *promptkit.PreparedRun) ([]byte, error)
}
// ParsePreparedRunOutputFormat parses a format name.
func ParsePreparedRunOutputFormat(raw string) (PreparedRunOutputFormat, error) {
return ParseOutputFormat(raw)
}
// ParseOutputFormat parses the shared inspection and prepared-run format names.
func ParseOutputFormat(raw string) (OutputFormat, error) {
switch OutputFormat(strings.ToLower(strings.TrimSpace(raw))) {
case "":
return DefaultOutputFormat, nil
case OutputFormatText:
return OutputFormatText, nil
case OutputFormatJSON:
return OutputFormatJSON, nil
default:
return "", fmt.Errorf("%w: %q (supported: %s, %s)", ErrUnknownPreparedRunFormat, raw, OutputFormatText, OutputFormatJSON)
}
}
// FormatterForPreparedRun returns a formatter strategy for the chosen format.
func FormatterForPreparedRun(outputFormat PreparedRunOutputFormat) (PreparedRunFormatter, error) {
switch outputFormat {
case PreparedRunFormatText:
return textPreparedRunFormatter{}, nil
case PreparedRunFormatJSON:
return jsonPreparedRunFormatter{}, nil
default:
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
}
}
// FormatPreparedRun formats a prepared run using the selected format.
func FormatPreparedRun(prepared *promptkit.PreparedRun, outputFormat PreparedRunOutputFormat) ([]byte, error) {
formatter, err := FormatterForPreparedRun(outputFormat)
if err != nil {
return nil, err
}
return formatter.Format(prepared)
}
// FormatPreparedRunByName parses a format name and formats a prepared run.
func FormatPreparedRunByName(prepared *promptkit.PreparedRun, rawFormat string) ([]byte, error) {
outputFormat, err := ParsePreparedRunOutputFormat(rawFormat)
if err != nil {
return nil, err
}
return FormatPreparedRun(prepared, outputFormat)
}
type jsonPreparedRunFormatter struct{}
func (jsonPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
if prepared == nil {
return nil, errors.New("prepared run is nil")
}
return json.MarshalIndent(prepared, "", " ")
}
type textPreparedRunFormatter struct{}
func (textPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
if prepared == nil {
return nil, errors.New("prepared run is nil")
}
var b bytes.Buffer
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
if prepared.SelectedBackendID != "" {
fmt.Fprintf(&b, "selected_backend_id: %s\n", prepared.SelectedBackendID)
}
if prepared.PromptHash != "" {
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
}
if prepared.SessionID != "" {
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
}
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
target := prepared.EffectiveModelParams
fmt.Fprintln(&b, "effective_model_params:")
fmt.Fprintf(&b, " endpoint: %s\n", target.Endpoint)
fmt.Fprintf(&b, " model: %s\n", target.Model)
fmt.Fprintf(&b, " temperature: %g\n", target.Temperature)
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
if target.ServiceTier != "" {
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
}
if target.ReasoningEffort != "" {
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
}
if target.APIKeyEnv != "" {
fmt.Fprintf(&b, " api_key_env: %s\n", target.APIKeyEnv)
}
if len(target.ExtraParams) > 0 {
fmt.Fprintln(&b, " extra_params:")
keys := make([]string, 0, len(target.ExtraParams))
for k := range target.ExtraParams {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
if err != nil {
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
}
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
}
}
fmt.Fprintln(&b, "input_hashes:")
inputKeys := make([]string, 0, len(prepared.InputHashes))
for k := range prepared.InputHashes {
inputKeys = append(inputKeys, k)
}
sort.Strings(inputKeys)
for _, k := range inputKeys {
fmt.Fprintf(&b, " %s: %s\n", k, prepared.InputHashes[k])
}
fmt.Fprintln(&b, "messages:")
roleOrder := make([]string, 0)
byRole := make(map[string][]promptkit.RenderedMessage)
for _, msg := range prepared.Messages {
if _, exists := byRole[msg.Role]; !exists {
roleOrder = append(roleOrder, msg.Role)
}
byRole[msg.Role] = append(byRole[msg.Role], msg)
}
for _, role := range roleOrder {
fmt.Fprintf(&b, " %s:\n", role)
messages := byRole[role]
for i, msg := range messages {
fmt.Fprintf(&b, " - message: %d\n", i+1)
if msg.CacheControl != nil {
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
if msg.CacheControl.TTL != "" {
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
}
fmt.Fprintln(&b)
}
fmt.Fprintln(&b, " content: |")
content := msg.Content
if content == "" {
fmt.Fprintln(&b, " ")
continue
}
for _, line := range strings.Split(content, "\n") {
fmt.Fprintf(&b, " %s\n", line)
}
}
}
return b.Bytes(), nil
}
func formatExtraParamTextValue(value any) (string, error) {
if s, ok := value.(string); ok {
return s, nil
}
b, err := json.Marshal(value)
if err != nil {
return "", err
}
return string(b), nil
}

View File

@@ -1,375 +0,0 @@
package format
import (
"encoding/json"
"errors"
"strings"
"testing"
"go.maximumdirect.net/promptkit"
)
func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if strings.Contains(s, "prompt_version") {
t.Fatalf("unexpected prompt version in text output: %s", s)
}
for _, want := range []string{
"prompt: prompt.id",
"selected_profile_id: local-fast",
"selected_backend_id: local",
"endpoint: http://llm/v1",
"model: gpt-test",
"temperature: 0.4",
"max_tokens: 256",
"top_p: 0.8",
"timeout_seconds: 45",
"service_tier: priority",
"reasoning_effort: medium",
"api_key_env: SCRIPTORIUM_API_KEY",
"prompt_hash: prompt-hash",
"rendered_prompt_hash: rendered-hash",
"glossary: hash-glossary",
"transcript: hash-transcript",
"messages:",
" system:",
" user:",
"System guidance.",
"Summarize the transcript.",
"Include key entities.",
"Second user message.",
} {
if !strings.Contains(s, want) {
t.Fatalf("expected text output to include %q, got:\n%s", want, s)
}
}
}
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
prepared := samplePreparedRun()
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"z_string": "enabled",
"b_number": 42,
"a_object": map[string]any{
"nested": "value",
"count": 2,
},
"c_array": []any{"first", 3, false},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
want := strings.Join([]string{
" extra_params:",
" a_object: {\"count\":2,\"nested\":\"value\"}",
" b_number: 42",
" c_array: [\"first\",3,false]",
" z_string: enabled",
}, "\n")
if !strings.Contains(s, want) {
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
}
}
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), secret) {
t.Fatalf("text output should not include resolved api key value: %s", out)
}
}
func TestTextFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
const directKey = "direct-format-key"
// PreparedRun intentionally has no field for direct API keys.
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), directKey) {
t.Fatalf("text output should not include direct api key value: %s", out)
}
}
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
t.Fatalf("expected system message cache control before content, got:\n%s", s)
}
if strings.Count(s, "cache_control:") != 1 {
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
}
}
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if !strings.Contains(string(out), "session_id: session-123\n") {
t.Fatalf("expected session_id in text output, got:\n%s", out)
}
}
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
},
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " cache_control: ephemeral\n") {
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
}
if strings.Contains(s, "ttl=") {
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
}
}
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
prepared := samplePreparedRun()
prepared.SessionID = "session-123"
prepared.EffectiveModelParams.ExtraParams = map[string]any{
"number": 42,
"nested": map[string]any{
"enabled": true,
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded map[string]any
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if decoded["prompt_id"] != "prompt.id" {
t.Fatalf("expected prompt_id in json output, got %#v", decoded["prompt_id"])
}
if _, ok := decoded["prompt_version"]; ok {
t.Fatalf("unexpected prompt_version in json output: %#v", decoded)
}
if decoded["selected_profile_id"] != "local-fast" {
t.Fatalf("expected selected_profile_id in json output, got %#v", decoded["selected_profile_id"])
}
if decoded["rendered_prompt_hash"] != "rendered-hash" {
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
}
if decoded["session_id"] != "session-123" {
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
}
modelParams, ok := decoded["effective_model_params"].(map[string]any)
if !ok {
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
}
extraParams, ok := modelParams["extra_params"].(map[string]any)
if !ok {
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
}
if extraParams["number"] != float64(42) {
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
}
nested, ok := extraParams["nested"].(map[string]any)
if !ok || nested["enabled"] != true {
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
}
if _, ok := decoded["input_hashes"]; !ok {
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
}
if _, ok := decoded["messages"]; !ok {
t.Fatalf("expected messages in json output, got %#v", decoded)
}
}
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []promptkit.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &promptkit.CacheControl{
Type: promptkit.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(promptkit.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret)
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), secret) {
t.Fatalf("json output should not include resolved api key value: %s", out)
}
}
func TestJSONFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
const directKey = "direct-format-key"
// PreparedRun intentionally has no field for direct API keys.
prepared := samplePreparedRun()
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if strings.Contains(string(out), directKey) {
t.Fatalf("json output should not include direct api key value: %s", out)
}
}
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
tests := []struct {
name string
input string
want PreparedRunOutputFormat
}{
{name: "default empty", input: "", want: DefaultPreparedRunOutputFormat},
{name: "text", input: "text", want: PreparedRunFormatText},
{name: "json", input: "json", want: PreparedRunFormatJSON},
{name: "trim and case", input: " JSON ", want: PreparedRunFormatJSON},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, err := ParsePreparedRunOutputFormat(tc.input)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if got != tc.want {
t.Fatalf("expected %q, got %q", tc.want, got)
}
})
}
}
func TestParsePreparedRunOutputFormatUnknownFails(t *testing.T) {
_, err := ParsePreparedRunOutputFormat("yaml")
if err == nil {
t.Fatal("expected error for unknown format")
}
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
}
}
func TestFormatPreparedRunByNameUnknownFailsClearly(t *testing.T) {
_, err := FormatPreparedRunByName(samplePreparedRun(), "yaml")
if err == nil {
t.Fatal("expected unknown format error")
}
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
}
}
func samplePreparedRun() *promptkit.PreparedRun {
return &promptkit.PreparedRun{
PromptID: "prompt.id",
PromptHash: "prompt-hash",
SelectedProfileID: "local-fast",
SelectedBackendID: "local",
EffectiveModelParams: promptkit.ExecutionTarget{
BackendID: "local",
Endpoint: "http://llm/v1",
Model: "gpt-test",
Temperature: 0.4,
MaxTokens: 256,
TopP: 0.8,
TimeoutSeconds: 45,
ServiceTier: "priority",
ReasoningEffort: "medium",
APIKeyEnv: "SCRIPTORIUM_API_KEY",
},
InputHashes: map[string]string{
"transcript": "hash-transcript",
"glossary": "hash-glossary",
},
RenderedPromptHash: "rendered-hash",
Messages: []promptkit.RenderedMessage{
{Role: "system", Content: "System guidance."},
{Role: "user", Content: "Summarize the transcript.\nInclude key entities."},
{Role: "user", Content: "Second user message."},
},
}
}