It accepts named input artifacts, renders prompt templates, calls an LLM, validates output, optionally performs bounded structured-output repair, and returns an artifact with metadata.
Scriptorium also supports rendering/preparing a prompt without calling an LLM. This allows users to inspect the fully rendered prompt messages and effective runtime settings before executing a run.
-`internal/domain`: Core domain contracts, including `PromptDefinition`, `ExecutionProfile`, `PreparedRun`, `RunResult`, and related metadata.
-`internal/usecase`: `Runner` use case logic, including prompt preparation, profile selection, runtime override resolution, full run execution, validation, and bounded repair.
-`internal/format` or equivalent: Formatting of prepared/rendered prompt output for CLI or other adapters, if formatting grows beyond simple CLI-local helpers.
Exact package names may evolve, but the architectural boundaries should remain stable.
## 3. Core Execution Model
Scriptorium has two closely related execution paths:
1. Prepare/render path.
2. Full run path.
The full run path should reuse the prepare path rather than duplicating its logic.
6.**Resolve Artifacts**: Load all named input artifacts defined in the request.
7.**Render Prompt**: Apply template variables and input artifacts to the prompt templates.
8.**Compute Metadata**: Compute hashes, selected profile ID, prompt ID/version, effective runtime settings, input hashes, rendered prompt hash, and timing information as appropriate.
9.**Return PreparedRun**: Return a `PreparedRun` containing the rendered messages, effective runtime settings, resolved metadata, and input/prompt hashes.
The prepare path must not call the LLM.
The prepare path must not validate model output, because there is no model output.
The prepare path must not perform structured-output repair, because repair only applies after model output exists.
The prepare path should not resolve or expose raw API key values. It may include the selected `api_key_env` name in effective runtime settings or metadata, but never the environment variable value.
### 3.2 Full Run Data Flow
The `Runner.Run(ctx, RunRequest)` flow should reuse the prepare path:
1.**Prepare**: Call the shared prepare flow to load the prompt, select the profile, resolve artifacts, render prompt messages, and compute pre-run metadata.
2.**Call LLM**: Execute the generation request using the effective runtime settings from the prepared run.
3.**Build Output Artifact**: Convert the model response into the configured output artifact.
4.**Validate Output**:
- Validate the model output against the prompt definition's output contract.
- Validation content failures remain successful run results with `validation.status=failed`.
- Validator runtime/config errors are run errors.
5.**Repair If Configured**:
- If structured validation fails and `repair_attempts > 0`, perform bounded repair attempts.
- Re-validate after each repair attempt.
- Repair loops must remain strictly bounded.
6.**Return RunResult**: Produce a `RunResult` containing the final artifact, validation status, raw model output, usage information, and metadata.
`Runner.Run` should not duplicate profile selection, artifact resolution, or prompt rendering logic that already exists in `Runner.Prepare`.
-`PromptDefinition`: Defines the "what" of the task: templates, inputs, output contract, validation settings, repair settings, and optional `default_profile`.
-`ExecutionProfile`: Defines the "how" of execution: endpoint, model, generation parameters, timeout, reasoning effort, and `api_key_env`.
-`RunRequest`: The intent to execute or prepare a prompt, including `prompt_id`, optional `profile_id`, inputs, variables, and optional runtime overrides.
-`PreparedRun`: The result of the prepare/render phase. Contains rendered messages, effective runtime settings, selected profile ID, prompt metadata, input hashes, prompt hash, and other pre-LLM metadata.
-`RunResult`: The result of a full run. Contains the generated `Artifact`, `ValidationResult`, raw model output, token usage, and `RunMetadata`.
-`RunMetadata`: Detailed tracing information, including prompt ID/version, selected profile ID, effective model parameters, usage tokens, hashes, timestamps, validation status, and repair attempts where applicable.
Scriptorium should expose at least these commands:
-`scriptorium run`: Executes a prompt by preparing it, calling the LLM, validating output, optionally repairing structured output, and returning an artifact.
-`scriptorium render`: Prepares and renders a prompt without calling the LLM.
-`scriptorium serve`: Starts the HTTP API using infrastructure-only flags.
### `scriptorium run`
`run` uses flags such as:
-`--prompt-dir`
-`--profile-dir`
-`--prompt`
-`--profile`
-`--input`
-`--var`
-`--out`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
`run` should produce the generated artifact as its primary output.
### `scriptorium render`
`render` prepares and renders a prompt without calling an LLM.
It should use the same prompt/profile/input/variable/runtime override flags as `run` where applicable:
-`--prompt-dir`
-`--profile-dir`
-`--prompt`
-`--profile`
-`--input`
-`--var`
- runtime overrides such as `--model`, `--llm-base-url`, `--temperature`, `--max-tokens`, `--top-p`, `--timeout`, and `--api-key-env` if supported.
-`--format`, with initial support for `text` and `json`.
Default render output format should be `text`.
`render` must not call the LLM.
`render` should show the same rendered messages and effective runtime settings that `run` would use.
`render` should include enough information to debug:
- prompt ID
- prompt version
- selected profile ID
- effective runtime settings
- input hashes
- prompt hash
- rendered messages
`render` must not include resolved API key values.
`render` may include the `api_key_env` name.
### Render Output Formats
Initial render output formats:
-`text`: Human-readable default format.
-`json`: Machine-readable structured representation of the prepared run.
Additional formats, such as `markdown`, may be added later.
Render output formatting should be modular. Adding a new output format should not require changing the prepare/run core logic.
The output formatting layer should consume a `PreparedRun` and produce bytes or text for the adapter. It should not reload prompts, re-resolve artifacts, re-render templates, call the LLM, or perform validation.
### `scriptorium serve`
`serve` starts the HTTP API.
It should use infrastructure-only flags such as:
-`--addr`
-`--prompt-dir`
-`--profile-dir`
-`--schema-dir`
`serve` should not introduce a server-level model/runtime precedence layer unless explicitly documented and intentionally implemented.
Request may include runtime overrides under `model` and an `include_raw_output` boolean.
`raw_model_output` is exposed only when explicitly requested with `include_raw_output=true`.
A future HTTP prepare/render endpoint may be added, such as `POST /v1/renders` or `POST /v1/runs/prepare`, but the initial render feature may be CLI-only. If added later, it should call the same usecase-level prepare path as `scriptorium render`.
The prepare flow should be the only implementation of:
- prompt loading
- profile selection
- runtime override resolution
- artifact resolution
- prompt rendering
- pre-run metadata/hash calculation
### CLI Shape
The preferred command name is `render`.
The command should support:
-`--format text`
-`--format json`
Default format:
-`text`
Unknown formats should produce a clear error.
Formatting should be centralized through a small formatter registry, strategy, switch, or interface so new formats can be added without modifying usecase logic.
### Text Output Expectations
Text output should be optimized for human inspection.
It should include, at minimum:
- prompt ID and version
- selected profile ID
- model name
- endpoint
- effective generation settings
- input hashes
- rendered prompt hash
- rendered messages grouped by role
Text output should be readable and deterministic enough for tests.
It should not include raw API key values.
### JSON Output Expectations
JSON output should be a structured representation of `PreparedRun` or a DTO derived from it.
It should include, at minimum:
- prompt ID and version
- selected profile ID
- effective runtime settings
- input hashes
- rendered prompt hash
- rendered messages
JSON output should not include raw API key values.
JSON output should remain stable enough to be useful for automation and integration tests.
