Retire the legacy CLI and document native operator workflows
This commit is contained in:
11
README.md
11
README.md
@@ -5,6 +5,9 @@ an HTTP service. It prepares prompt requests, runs them against
|
|||||||
OpenAI-compatible model endpoints, and returns generated output with validation
|
OpenAI-compatible model endpoints, and returns generated output with validation
|
||||||
metadata.
|
metadata.
|
||||||
|
|
||||||
|
CommandKit supplies native configuration, resource discovery, preview, reporting,
|
||||||
|
and completion interfaces alongside application generation and HTTP startup.
|
||||||
|
|
||||||
The application uses
|
The application uses
|
||||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/)
|
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/)
|
||||||
for prompt, profile, schema, preparation, generation, and validation behavior.
|
for prompt, profile, schema, preparation, generation, and validation behavior.
|
||||||
@@ -16,11 +19,10 @@ behavior, output presentation, and HTTP artifact-containment policy.
|
|||||||
From the repository root:
|
From the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
go run ./cmd/scriptorium render \
|
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
|
||||||
--config ./examples/config.yml \
|
--config ./examples/config.yml \
|
||||||
--prompt generic.markdown_summary \
|
--input-file transcript=./examples/fixtures/transcript.md \
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
--input-file glossary=./examples/fixtures/glossary.yml \
|
||||||
--input glossary=./examples/fixtures/glossary.yml \
|
|
||||||
--format json
|
--format json
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -38,6 +40,7 @@ a model. For complete invocation and output behavior, see the
|
|||||||
- [Operations guide](docs/operations.md)
|
- [Operations guide](docs/operations.md)
|
||||||
- [Consumer integration overview](docs/consumers/api.md)
|
- [Consumer integration overview](docs/consumers/api.md)
|
||||||
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
|
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
|
||||||
|
- [Native CLI migration](docs/consumers/upgrading-cli.md)
|
||||||
- [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md)
|
- [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md)
|
||||||
- [Subprocess integration](docs/integrations/subprocess.md)
|
- [Subprocess integration](docs/integrations/subprocess.md)
|
||||||
- [Architecture policy](docs/policy/architecture.md)
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
|
|||||||
487
docs/cli.md
487
docs/cli.md
@@ -1,377 +1,186 @@
|
|||||||
# CLI Reference
|
# CLI Reference
|
||||||
|
|
||||||
This is the canonical contract for invoking Scriptorium. Configuration discovery,
|
Scriptorium uses CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
|
||||||
precedence, application source locations, and server settings are defined in the
|
This document owns application invocation and delivery behavior. The pinned
|
||||||
[configuration reference](config.md). The [HTTP API reference](api.md) owns
|
[upstream CLI contract][native-cli] owns standard family grammar and reports.
|
||||||
service request and response behavior.
|
[Configuration](config.md) owns discovery, settings, and source composition;
|
||||||
|
the [HTTP API](api.md) owns the service contract.
|
||||||
## Shortest Useful Command
|
Existing callers should read the [CLI migration guide](consumers/upgrading-cli.md).
|
||||||
|
|
||||||
```bash
|
|
||||||
go run ./cmd/scriptorium render \
|
|
||||||
--config ./examples/config.yml \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
|
||||||
--input glossary=./examples/fixtures/glossary.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
`render` prepares a request without calling an LLM.
|
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
- `scriptorium run`: prepare a prompt, call the configured LLM, and write the
|
|
||||||
generated artifact.
|
|
||||||
- `scriptorium render`: prepare a prompt and write prepared-run output.
|
|
||||||
- `scriptorium serve`: start the HTTP server.
|
|
||||||
- `scriptorium inspect prompt`: inspect one prompt definition without model execution.
|
|
||||||
- `scriptorium inspect profile`: inspect one resolved profile without model execution.
|
|
||||||
- `scriptorium inspect configuration`: validate or inspect Promptkit's captured framework configuration.
|
|
||||||
- `scriptorium config validate`: validate application configuration without building an engine.
|
|
||||||
- `scriptorium config show`: show the resolved application configuration.
|
|
||||||
- `scriptorium config sources`: show the ordered configuration sources.
|
|
||||||
- `scriptorium config explain`: explain one effective or historical configuration path.
|
|
||||||
|
|
||||||
All command families accept `--config <path>` and `--config-overlay <name>`;
|
|
||||||
positional arguments are rejected. `--config-overlay` selects one application
|
|
||||||
configuration overlay; it is separate from `--profile`, which selects a
|
|
||||||
Promptkit execution profile. See [configuration composition](config.md#explicit-imports-and-selected-overlays)
|
|
||||||
for name, root, and absent-main behavior. `run`, `render`, `serve`, `inspect prompt`,
|
|
||||||
`inspect profile`, and `inspect configuration` accept repeated
|
|
||||||
`--promptkit-source <path>` flags and `--schema-dir <dir>` where applicable.
|
|
||||||
They replace the configured source sequence in flag order. `--prompt-dir` and
|
|
||||||
`--profile-dir` are no longer supported; use `promptkit_sources` in application
|
|
||||||
configuration or the repeated source flags. Application `config` commands
|
|
||||||
remain independent of framework source loading. Explicit application flags,
|
|
||||||
including empty values, override file settings. Configuration load diagnostics
|
|
||||||
go to stderr and exit with status 1; successful load warnings are printed once
|
|
||||||
before command work.
|
|
||||||
|
|
||||||
## `scriptorium run`
|
|
||||||
|
|
||||||
```text
|
|
||||||
scriptorium run [flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
Required flags:
|
|
||||||
|
|
||||||
| Flag | Meaning |
|
|
||||||
| --- | --- |
|
|
||||||
| `--prompt <id>` | Prompt ID to execute. |
|
|
||||||
|
|
||||||
Optional flags:
|
|
||||||
|
|
||||||
| Flag | Meaning |
|
|
||||||
| --- | --- |
|
|
||||||
| `--config <path>` | Application configuration file. |
|
|
||||||
| `--config-overlay <name>` | Selected application configuration overlay. |
|
|
||||||
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
|
|
||||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
|
||||||
| `--profile <id>` | Execution-profile override. |
|
|
||||||
| `--session-id <id>` | Optional direct session identifier. |
|
|
||||||
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
|
|
||||||
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
|
||||||
| `--out <path>` | Write generated content to this file instead of stdout. |
|
|
||||||
| `--execution-report <path>` | Write an opt-in terminal execution report as newline-terminated JSON to a separate file. |
|
|
||||||
| `--llm-base-url <url>` | Runtime endpoint override. |
|
|
||||||
| `--model <name>` | Runtime model override. |
|
|
||||||
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
|
|
||||||
| `--temperature <float>` | Runtime temperature override. |
|
|
||||||
| `--max-tokens <int>` | Runtime maximum-token override. |
|
|
||||||
| `--top-p <float>` | Runtime top-p override. |
|
|
||||||
| `--reasoning-effort <value>` | Runtime reasoning-effort override. |
|
|
||||||
| `--timeout <duration>` | Runtime timeout override using Go duration syntax. |
|
|
||||||
| `--run-timeout <duration>` | Optional whole-run engine deadline using Go duration syntax; zero or omission adds no deadline, negative values are invalid. |
|
|
||||||
| `--transient-retries <integer>` | Nonnegative additional transient attempt count for this request; `0` disables additional attempts. |
|
|
||||||
|
|
||||||
Deprecated aliases: `--prompt-id` for `--prompt`, and `--profile-id` for
|
|
||||||
`--profile`.
|
|
||||||
|
|
||||||
Omitted numeric runtime flags preserve the selected effective value; explicit
|
|
||||||
zero values override it. `--timeout 0s` disables the per-attempt generation deadline
|
|
||||||
only; the caller context and configured transport cap remain active. CLI
|
|
||||||
durations are converted to whole seconds by truncation toward zero, so any
|
|
||||||
duration whose absolute value is below one second becomes an explicit
|
|
||||||
zero-second override. The timeout layers are defined in the
|
|
||||||
[Promptkit outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
|
||||||
With the built-in client, each transient retry gets a fresh generation timeout;
|
|
||||||
this flag does not limit the complete run.
|
|
||||||
`--run-timeout` keeps subsecond precision and starts immediately before engine
|
|
||||||
execution. It bounds preparation, capacity waits, retry waits, generation, and
|
|
||||||
repairs where they honor context. Parsing, engine construction, and output
|
|
||||||
writes are outside that interval. A failed run identified by PromptKit as an
|
|
||||||
operation timeout exits 1 with a safe deadline diagnostic; a completed invalid
|
|
||||||
output still exits 2. Synchronous work may not stop immediately.
|
|
||||||
Omitting `--transient-retries` inherits the application setting or Promptkit's
|
|
||||||
current default of two additional attempts. The flag overrides only this
|
|
||||||
request; it does not alter the configured engine. Structural repair generations
|
|
||||||
have separate transient retry budgets.
|
|
||||||
|
|
||||||
There is no raw API-key flag. Use `--api-key-env`.
|
|
||||||
|
|
||||||
Prompt and profile IDs are passed unchanged, including collection-qualified
|
|
||||||
IDs. The removed `--prompt-version` flag is rejected. `--input` is optional
|
|
||||||
at the CLI boundary: PromptKit decides whether the selected definition requires
|
|
||||||
declared inputs or template-referenced values.
|
|
||||||
|
|
||||||
`--session-id` supplies a direct, non-secret session identifier. A nonblank
|
|
||||||
value replaces a definition-rendered session ID; an omitted or blank value lets
|
|
||||||
the definition supply one. Promptkit trims direct values and limits them to 256
|
|
||||||
Unicode code points. Use stable, non-sensitive identifiers because effective
|
|
||||||
session IDs may appear in prepared output, run metadata, and provider-facing
|
|
||||||
requests.
|
|
||||||
|
|
||||||
`--reasoning-effort` is presence-aware: omitting it inherits the selected
|
|
||||||
profile value, a nonblank value replaces that value, and
|
|
||||||
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
|
|
||||||
nonblank values as provider-specific opaque strings.
|
|
||||||
|
|
||||||
### Execution report file
|
|
||||||
|
|
||||||
`--execution-report` requires a nonempty path. It is available only for `run`;
|
|
||||||
`-` is an ordinary filename. Parent directories are not created. An existing
|
|
||||||
file is replaced through ordinary file writing, without rollback or an atomic
|
|
||||||
replacement guarantee. Generated content still goes to stdout or `--out`.
|
|
||||||
Equal output/report paths, including existing paths that refer to the same
|
|
||||||
file, are rejected before execution.
|
|
||||||
|
|
||||||
The file is written after engine execution when Promptkit supplies a terminal
|
|
||||||
event, including for a failed run or invalid generated output. A failure before
|
|
||||||
invocation leaves the report path untouched. If no terminal event is supplied,
|
|
||||||
Scriptorium leaves the path untouched and emits a diagnostic; an existing file
|
|
||||||
may therefore contain an older report. Artifact output is attempted first on a
|
|
||||||
successful run, and report writing is attempted even if artifact writing fails.
|
|
||||||
Any output or report write failure exits 1; already-written output is not
|
|
||||||
undone. Without a write failure, the actual run keeps its usual exit status.
|
|
||||||
|
|
||||||
The [HTTP API reference](api.md#execution-report) defines the shared report
|
|
||||||
fields and usage-presence semantics. The file contains that JSON object followed
|
|
||||||
by a newline.
|
|
||||||
|
|
||||||
The report excludes content, credentials, URLs, headers, and raw errors, but
|
|
||||||
its identifiers can still be sensitive. The caller owns file retention and
|
|
||||||
access.
|
|
||||||
|
|
||||||
## `scriptorium render`
|
|
||||||
|
|
||||||
```text
|
|
||||||
scriptorium render [flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
`--prompt <id>` is required. The following optional flags are supported:
|
|
||||||
`--config`, `--config-overlay`, repeatable `--promptkit-source`, `--schema-dir`, `--profile`,
|
|
||||||
`--input`, `--var`, `--out`, `--llm-base-url`,
|
|
||||||
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
|
|
||||||
`--reasoning-effort`, `--session-id`, `--timeout`, `--transient-retries`, and `--format text|json`. Their meanings match the corresponding
|
|
||||||
`run` flags; `--format` selects prepared-run output and otherwise uses
|
|
||||||
`defaults.render_format`.
|
|
||||||
|
|
||||||
The same deprecated aliases and numeric/timeout behavior as `run` apply.
|
|
||||||
The same session and reasoning inheritance, replacement, and clearing behavior
|
|
||||||
also applies.
|
|
||||||
`render` validates the retry override but does not generate model output or
|
|
||||||
include a retry field in prepared output.
|
|
||||||
`render` resolves profiles and schemas as part of preparation but
|
|
||||||
does not call an LLM.
|
|
||||||
|
|
||||||
To render from independently maintained collection checkouts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go run ./cmd/scriptorium render --config ./examples/config.discovery.yml \
|
|
||||||
--prompt support/answer --profile support/fast \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md
|
|
||||||
```
|
|
||||||
|
|
||||||
The [discovery example](../examples/discovery/framework.yml) also provides
|
|
||||||
`review/answer`; see [configuration discovery](config.md#collection-checkout-discovery).
|
|
||||||
|
|
||||||
## `scriptorium serve`
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
scriptorium run PROMPT_ID [flags]
|
||||||
scriptorium serve [flags]
|
scriptorium serve [flags]
|
||||||
|
scriptorium config scopes
|
||||||
|
scriptorium config validate|show|sources|explain|diff [flags]
|
||||||
|
scriptorium promptkit list prompts|profiles|backends|namespaces|collections [flags]
|
||||||
|
scriptorium promptkit show prompt|profile|backend ID [flags]
|
||||||
|
scriptorium promptkit requirements show|validate PROMPT_ID [flags]
|
||||||
|
scriptorium promptkit show prepared PROMPT_ID [flags]
|
||||||
|
scriptorium promptkit test KIND [flags]
|
||||||
|
scriptorium help [COMMAND ...]
|
||||||
|
scriptorium version
|
||||||
|
scriptorium completion bash|zsh|fish
|
||||||
```
|
```
|
||||||
|
|
||||||
Optional flags:
|
Native help is also available with `--help`. Help, version, scope discovery,
|
||||||
|
and static completion do not load configuration or contact a provider.
|
||||||
|
Version/revision metadata is supplied by the build; ordinary builds identify
|
||||||
|
themselves as `dev` with no revision. Release build mechanics belong to the
|
||||||
|
[release procedure](release.md).
|
||||||
|
|
||||||
|
Commands that consume configuration accept `--config PATH` and `--overlay NAME`.
|
||||||
|
An explicitly empty config value remains explicit; overlay names follow
|
||||||
|
[ConfigKit selection](config.md#explicit-imports-and-selected-overlays).
|
||||||
|
There are no source/schema convenience flags or legacy command aliases.
|
||||||
|
Native shared `--format`, `--quiet`, `--verbose`, `--debug`, and progress
|
||||||
|
controls retain their upstream meanings. Diagnostics use stderr, not stdout.
|
||||||
|
Verbose exposes emitted Info and Warning messages without changing result bytes.
|
||||||
|
|
||||||
|
## Generation
|
||||||
|
|
||||||
|
`run` requires exactly one positional prompt ID. It has no duplicate prompt
|
||||||
|
flag or direct backend-selection flag; choose a profile for backend routing.
|
||||||
|
It uses these native bindings, with the preferred aliases below. Canonical
|
||||||
|
spellings prepend `llm-` (for example, `--llm-profile`).
|
||||||
|
|
||||||
|
| Preferred flags | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `--profile ID`, `--session-id ID` | Profile selection and optional non-secret session context. |
|
||||||
|
| `--var KEY=VALUE` | Repeatable caller-variable mappings. |
|
||||||
|
| `--input-text NAME=TEXT`, `--input NAME=TEXT` | Repeatable literal input mappings; `--llm-input` is also a literal-text alias. |
|
||||||
|
| `--input-file NAME=PATH` | Repeatable file references. |
|
||||||
|
| `--endpoint URL`, `--model NAME` | Native endpoint/model overrides. |
|
||||||
|
| `--temperature N`, `--max-tokens N`, `--top-p N` | Presence-aware numeric overrides, including zero. |
|
||||||
|
| `--timeout-seconds N` | Integer per-attempt generation timeout; zero disables that deadline, not caller cancellation or the transport cap. |
|
||||||
|
| `--reasoning-effort VALUE`, `--service-tier VALUE` | Native provider controls; an explicit empty reasoning value is preserved. |
|
||||||
|
| `--api-key-env NAME` | Credential environment-variable name, never raw key bytes. |
|
||||||
|
| `--extra-params JSON` | Native JSON-object provider parameters. |
|
||||||
|
| `--transient-retries N` | Request retry override; omission leaves the engine default. |
|
||||||
|
| `--prompt-namespace NAME`, `--profile-namespace NAME`, `--backend-namespace NAME` | Independent native namespace views. Qualified resource IDs remain exact; stored references are not rewritten. |
|
||||||
|
|
||||||
|
Mappings split at the first `=`; commas and subsequent equals signs are literal.
|
||||||
|
Repeat flags for multiple mappings. The last text/file occurrence for the same
|
||||||
|
input wins in command-line order. PromptKit validates selected definitions,
|
||||||
|
inputs, namespaces, provider settings, and credential requirements; Scriptorium
|
||||||
|
does not add an execution-readiness preflight or configurable default namespace.
|
||||||
|
Native string/pointer override semantics remain upstream-owned.
|
||||||
|
|
||||||
|
Application-owned generation controls:
|
||||||
|
|
||||||
| Flag | Meaning |
|
| Flag | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `--config <path>` | Application configuration file. |
|
| `--run-timeout DURATION` | Go duration passed unchanged to PromptKit's operation budget. Positive values cover engine preparation/execution, including cooperative waits and repairs. Zero, omission, and negative values add no budget. Parsing, configuration/engine setup, and delivery are outside it. |
|
||||||
| `--config-overlay <name>` | Selected application configuration overlay. |
|
| `--report PATH` | Additionally write the returned native result to a separate file. |
|
||||||
| `--addr <listen-address>` | HTTP listen-address override. |
|
| `--report-format text\|yaml\|json` | Explicit separate-report format; omission leaves native automatic rendering selected. |
|
||||||
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
|
|
||||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
|
||||||
| `--artifact-root <dir>` | Root for HTTP `file` input references. |
|
|
||||||
| `--max-request-bytes <n>` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
|
|
||||||
| `--max-artifact-bytes <n>` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
|
||||||
| `--max-response-bytes <n>` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
|
||||||
|
|
||||||
`serve` accepts no runtime model override flags. HTTP request fields, response
|
The operation budget is independent of the integer per-attempt timeout.
|
||||||
schemas, and error codes are defined in the [HTTP API reference](api.md).
|
Earlier parent cancellation/deadlines still apply. Cancellation is cooperative;
|
||||||
|
synchronous work that ignores context cannot be forcibly interrupted.
|
||||||
|
|
||||||
## `scriptorium inspect prompt`
|
### Results And Separate Reports
|
||||||
|
|
||||||
```text
|
The main report goes through the invocation reporter to stdout. Automatic/text
|
||||||
scriptorium inspect prompt --prompt ID
|
output is exactly `Artifact.Body`, with no added newline, header, escaping, or
|
||||||
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
|
summary. JSON/YAML output is the full native PromptKit `RunResult`, without a
|
||||||
```
|
Scriptorium envelope. In native structured serialization, artifact bytes use
|
||||||
|
the upstream representation (JSON base64), not an application text projection.
|
||||||
|
|
||||||
`--prompt` is required. Inspection uses normal configuration discovery and the
|
Save main output with shell redirection or the subprocess stdout destination.
|
||||||
selected Promptkit sources, defaults to text regardless of
|
There is no `--output` or `--out` flag. Omit `--quiet` when capturing the main
|
||||||
`defaults.render_format`, and writes to stdout unless `--out` is supplied.
|
result: native quiet suppresses it even when stdout is redirected.
|
||||||
|
|
||||||
Both definition-inspection commands first construct the same engine used for
|
`--report` renders the same result through CommandKit without the artifact-only
|
||||||
execution from the complete selected source sequence. Construction resolves
|
text template. Its format is independent of stdout's `--format` and the filename
|
||||||
and validates all effective definitions, including unused ones, and captures
|
extension. Omission passes an unset format to the upstream renderer; it currently
|
||||||
referenced message-file contents and static template syntax. Invalid unused
|
renders structured values as YAML in automatic/text mode. Choose an explicit
|
||||||
definitions or unreadable message resources can therefore prevent inspection.
|
format when required rather than relying on that upstream default.
|
||||||
No prompt collection is mandatory, but selected prompts are not ignored.
|
|
||||||
|
|
||||||
Inspection does not prepare an execution request, load request-time schemas or
|
The requested file is written even in quiet mode. Omission creates no separate
|
||||||
artifacts, resolve credential values, reserve backend capacity, or call a model.
|
report. An explicitly empty path is a usage error before generation; `-` is an
|
||||||
Prompt inspection does not select an execution profile or render request-specific
|
ordinary filename. New files use mode 0644 subject to umask, existing files use
|
||||||
templates.
|
normal OS replacement behavior, and parent directories are not created.
|
||||||
Prompt output includes ID, hash, optional default profile and input metadata,
|
Render, write, and close failures are reported.
|
||||||
and the output contract. Absent optional values are omitted; explicitly empty
|
|
||||||
or zero values are retained. Captured message text is not projected.
|
|
||||||
|
|
||||||
## `scriptorium inspect profile`
|
Only a returned result is eligible for delivery. A resultless failure leaves an
|
||||||
|
existing report untouched; it does not synthesize an invocation-error result.
|
||||||
|
Main delivery is attempted first, then the requested report even if main
|
||||||
|
delivery fails. Writes are nontransactional; bytes already written remain.
|
||||||
|
Callers own redirected paths, retention, and keeping main and report destinations
|
||||||
|
separate. A shell may truncate its redirected file before the process starts.
|
||||||
|
|
||||||
```text
|
## Application Configuration Inspection
|
||||||
scriptorium inspect profile --profile ID
|
|
||||||
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
|
|
||||||
```
|
|
||||||
|
|
||||||
`--profile` is required. After the [full-source engine construction described
|
The native `config` family defaults to `app`; select `--scope promptkit` for
|
||||||
above](#scriptorium-inspect-prompt), inspection selects one resolved profile
|
captured framework configuration or use native scope discovery/all-scope modes.
|
||||||
with inheritance. `profile_model_params` contains only fields present on that
|
`app` uses the actual completed ConfigKit result and does not construct an engine.
|
||||||
profile; it does not fill backend defaults. `api_key_required` is a separate
|
|
||||||
optional field. Explicit zero and empty values remain visible. Use `render`
|
|
||||||
for request-completed model settings.
|
|
||||||
|
|
||||||
## Application configuration inspection
|
Native `--set POINTER=JSON` is enabled only for `app`. Records go directly to
|
||||||
|
ConfigKit: duplicate and ancestor/descendant-overlapping paths are rejected,
|
||||||
|
not resolved by flag order. No application-specific configuration flags or
|
||||||
|
custom `--against-*` counterparts are installed.
|
||||||
|
|
||||||
```text
|
Native diff uses independent `--against-config`, `--against-overlay`, and
|
||||||
scriptorium config validate [flags]
|
supported `--against-set` selections. Omitted right-side values do not inherit
|
||||||
scriptorium config show [flags]
|
the left side. See the [upstream contract][native-cli] for focused paths,
|
||||||
scriptorium config sources [flags]
|
metadata discovery, source IDs, findings, explanations, and detailed comparisons.
|
||||||
scriptorium config explain --path JSON_POINTER [flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
Each accepts `--config`, `--config-overlay`, `--schema-dir`, `--addr`,
|
## Framework Configuration Inspection
|
||||||
`--artifact-root`, `--max-request-bytes`,
|
|
||||||
`--max-artifact-bytes`, and `--max-response-bytes`. These are application
|
|
||||||
configuration selectors and overrides; request, credential, backend-editing,
|
|
||||||
output-format, and output-file flags are not accepted. Positional arguments
|
|
||||||
are rejected.
|
|
||||||
|
|
||||||
`validate` writes no success output; status 0 means the application configuration
|
The `promptkit` config scope loads application settings selected by `--config`
|
||||||
loaded and passed its own validation. It does not establish that selected
|
and `--overlay`, constructs an engine, and exposes its captured native view.
|
||||||
Promptkit sources can construct an engine, credentials are present, or a
|
It does not accept temporary overrides; native capability checks also apply
|
||||||
provider is available. `show` writes ConfigKit's resolved YAML snapshot. `sources`
|
when an all-scope operation requests them.
|
||||||
writes one newline-terminated JSON array of source records in load order. Each
|
|
||||||
record has `id`, `label`, and `layer`, plus `path` when available. Application
|
|
||||||
main-file and import records use the `main` layer; explicit CLI
|
|
||||||
overrides appear as synthetic sources. Defaults-only input without overrides
|
|
||||||
produces `[]` for `sources`.
|
|
||||||
|
|
||||||
`explain` requires an explicitly supplied `--path` in JSON Pointer syntax.
|
Framework effective/explanation views retain native classified-field redaction,
|
||||||
The empty string selects the root; `/server/addr` selects a field; `~1` and
|
including history. Environment-reference names and ordinary values remain
|
||||||
`~0` escape `/` and `~` inside mapping keys. Numeric tokens are interpreted
|
visible. This is not arbitrary secret detection and does not redact all prompt,
|
||||||
according to the containing mapping or sequence. Dotted paths, file paths,
|
prepared, application, or generated content. Protect reports accordingly.
|
||||||
and URL fragments are not accepted. ConfigKit writes a human-readable account
|
|
||||||
of the effective value and its chronological default, file/import, overlay,
|
|
||||||
CLI, normalization, and removed-value history. This is not a JSON schema or a
|
|
||||||
byte-for-byte stable prose format. Unknown and malformed paths exit 1 without
|
|
||||||
successful explanation output.
|
|
||||||
|
|
||||||
Diagnostics and warnings go to stderr, never into successful stdout. Parse,
|
## Discovery, Preview, And Explicit Smoke Testing
|
||||||
load, output, and diagnostic-write failures exit 1. Ordinary output writes may
|
|
||||||
leave partial data on failure. Application inspection can contain paths and
|
|
||||||
configured values; protect its output accordingly. Raw API-key values are not
|
|
||||||
accepted by the application configuration schema.
|
|
||||||
|
|
||||||
## Framework configuration inspection
|
Inventories retain native defaults and `--namespace`/`--all-namespaces` filters.
|
||||||
|
Named inspections expose native prompt messages and profile/backend values;
|
||||||
|
prepared preview exposes rendered content, not a summary projection.
|
||||||
|
|
||||||
```text
|
Requirements discovery works before input artifacts exist and without schema
|
||||||
scriptorium inspect configuration validate [flags]
|
loading, credential resolution, message rendering, or provider access. It still
|
||||||
scriptorium inspect configuration show [flags]
|
loads configuration and captured message resources. Reports preserve exact names,
|
||||||
scriptorium inspect configuration sources [flags]
|
metadata, variable presence (including empty values), completeness, and issue
|
||||||
scriptorium inspect configuration explain --path JSON_POINTER [flags]
|
codes. Partial discovery may succeed with status zero; completeness is not
|
||||||
```
|
binding satisfaction or execution readiness. Concrete validation and preview
|
||||||
|
keep their native preparation-time prerequisites and make no generation call.
|
||||||
|
|
||||||
Each operation accepts `--config`, `--config-overlay`, repeatable
|
Smoke-test kinds are `reachability`, `authentication`, `text_inference`,
|
||||||
`--promptkit-source`, and `--schema-dir`. These flags select the application
|
`structured_output`, or `all`, with native aliases as documented upstream.
|
||||||
settings and framework sources; positional arguments, `--format`, and `--out`
|
These explicit commands may contact providers and incur cost. Their presence
|
||||||
are not accepted. Unlike `config validate`, framework `validate` constructs
|
does not add a startup/provider probe. Read the native findings: unsuccessful or
|
||||||
the Promptkit engine and validates all effective definitions, including
|
unsupported checks are not automatically invocation failures.
|
||||||
unused ones. It writes nothing on success and does not prepare a request,
|
|
||||||
resolve a credential, or contact a provider.
|
|
||||||
|
|
||||||
`show` writes Promptkit's native effective YAML, an inspection snapshot rather
|
## Server And Statuses
|
||||||
than a promised reloadable file. `sources` writes a
|
|
||||||
newline-terminated JSON array in captured source order. Each source record has
|
|
||||||
`id`, `label`, `layer`, `kind`, and `underlay_index`, plus `path` when available
|
|
||||||
and `imported_from` for imported sources. That nested object has `source_id`,
|
|
||||||
`line`, and `column`. Values and output structure are a projection of the
|
|
||||||
captured Promptkit view, not Scriptorium's application-source records.
|
|
||||||
|
|
||||||
`explain` requires an explicitly supplied `--path` in ConfigKit JSON Pointer
|
`serve` accepts configuration selection and presence-aware HTTP controls:
|
||||||
syntax; `--path ''` selects the root. For example,
|
`--addr`, `--artifact-root`, `--max-request-bytes`, `--max-artifact-bytes`,
|
||||||
`/collections/support/profiles/0/model` addresses the first support profile's
|
and `--max-response-bytes`. It has no generation request controls. Values and
|
||||||
model in the captured list. Escape `/` as `~1` and `~` as `~0`; dots in a key
|
defaults belong to [configuration](config.md).
|
||||||
remain literal. Malformed and unknown pointers exit 1 without a successful
|
|
||||||
report. Explanation prose is native and is not a stable machine-readable
|
|
||||||
format.
|
|
||||||
|
|
||||||
Framework `show` and `explain` use Promptkit's native redaction for its
|
Interrupt/SIGTERM cancels the invocation context. The server closes its listener
|
||||||
classified credential-shaped provider fields, including historical values.
|
and active connections; canceled shutdown returns operational status, with no
|
||||||
Environment-variable reference names and ordinary provider parameters remain
|
graceful-drain deadline or guarantee that context-ignoring work has finished.
|
||||||
visible. Arbitrary prompt or provider content is not classified as secret by
|
|
||||||
name inspection, so treat these reports as sensitive. This redaction policy is
|
|
||||||
limited to framework configuration reports; it does not add redaction to
|
|
||||||
`inspect profile` or prepared-run output. Credentials are not resolved by any
|
|
||||||
of these inspection operations. Promptkit owns detailed field and source
|
|
||||||
semantics; see its tagged [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
|
||||||
|
|
||||||
Warnings and errors go to stderr. Successful `validate` writes nothing to
|
|
||||||
stdout; successful `show`, `sources`, and `explain` write their reports there.
|
|
||||||
Parse, load, path, and output failures exit 1. A native writer error may leave
|
|
||||||
partial output.
|
|
||||||
|
|
||||||
## Input And Variable Syntax
|
|
||||||
|
|
||||||
`--input name=path` maps an input name to a local file; `--var name=value`
|
|
||||||
maps a template variable to a string. Both flags can be repeated or contain
|
|
||||||
comma-separated mappings. Values may contain `=` after the first separator.
|
|
||||||
Empty names and values are rejected.
|
|
||||||
|
|
||||||
CLI inputs are file references. HTTP inline inputs are defined by the
|
|
||||||
[HTTP API reference](api.md).
|
|
||||||
|
|
||||||
## Output And Exit Behavior
|
|
||||||
|
|
||||||
- `run` writes generated content to stdout, or to `--out` when supplied, and
|
|
||||||
writes a concise summary to stderr. The summary includes `backend=<id>` when
|
|
||||||
Promptkit selected a backend; endpoint-only profiles omit it.
|
|
||||||
- `render` writes prepared-run output to stdout, or to `--out` when supplied,
|
|
||||||
without a success summary. Text output includes `selected_backend_id` after
|
|
||||||
`selected_profile_id` when Promptkit selected one; endpoint-only profiles
|
|
||||||
omit it.
|
|
||||||
- `serve` writes startup and server errors to stderr.
|
|
||||||
|
|
||||||
Exit statuses:
|
|
||||||
|
|
||||||
| Status | Meaning |
|
| Status | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `0` | Success. |
|
| 0 | Native success, subject to successful presentation/delivery. |
|
||||||
| `1` | Parse, configuration, loading, rendering, generation, output-write, or other runtime error. |
|
| 1 | Operational failure, including lookup/preparation/generation/capacity, cancellation/timeouts, unclassified load errors, and delivery failure for an otherwise successful run. |
|
||||||
| `2` | `run` generated and wrote output, but validation failed. |
|
| 2 | Native usage/declaration-input errors, or configuration errors classified as invalid input by native diagnostic evidence. |
|
||||||
|
| 3 | Native `config diff --check` found differences. |
|
||||||
|
| 4 | Completed generation returned `ValidationFailed`; artifact and native result remain deliverable. |
|
||||||
|
|
||||||
A backend admission rejection is a runtime error and prints `run error: model
|
Native finalization handles presentation errors: success/usage become operational;
|
||||||
backend capacity is exhausted`. The HTTP capacity response is defined in the
|
an already nonzero domain outcome retains its classification/status. A report-file
|
||||||
[HTTP API reference](api.md).
|
failure preserves completed-validation status 4 while adding its explanation and
|
||||||
|
cause. An operational failure to perform validation is status 1, not status 4.
|
||||||
|
JSON selection does not turn failures into a separate JSON error envelope.
|
||||||
|
|
||||||
## Workflows And Examples
|
[native-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md
|
||||||
|
|
||||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
|
||||||
copyable render workflow. The [HTTP request example](../examples/http-run.json)
|
|
||||||
is for a running `serve` process.
|
|
||||||
|
|||||||
382
docs/config.md
382
docs/config.md
@@ -1,87 +1,126 @@
|
|||||||
# Configuration Reference
|
# Configuration Reference
|
||||||
|
|
||||||
This is the canonical reference for Scriptorium application settings. Prompt,
|
This document owns Scriptorium's application settings, discovery, and source
|
||||||
profile, backend, schema, execution-setting, built-in profile, and framework
|
selection. [CLI syntax](cli.md) and the [HTTP API](api.md) have separate owners.
|
||||||
credential semantics are defined by the
|
Framework definitions and execution semantics belong to the pinned
|
||||||
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
[PromptKit v0.23.3 formats][formats]; ConfigKit v1.2.1 owns
|
||||||
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and
|
[native composition and inspection][configkit].
|
||||||
outcomes, see the [HTTP API reference](api.md). For migration examples, see
|
|
||||||
[Upgrading Promptkit sources and definitions](consumers/upgrading-promptkit.md).
|
|
||||||
|
|
||||||
## Discovery And Precedence
|
## Discovery And Precedence
|
||||||
|
|
||||||
ConfigKit assembles the selected main source, imports, and overlay, resolves
|
An explicit `--config` selects that file, including an explicitly empty value
|
||||||
inheritance, and decodes the result with Scriptorium's typed defaults. It then
|
(which fails rather than falling back). With no selection, ConfigKit searches
|
||||||
applies explicitly supplied application CLI values, records application
|
these native presets in order, selecting one main file:
|
||||||
normalization, and validates the result. Source, composition, inheritance, and
|
|
||||||
decoding failures happen before CLI assignment and cannot be repaired by it;
|
|
||||||
an override can correct a decoded value before application validation.
|
|
||||||
|
|
||||||
When `--config` is omitted, Scriptorium searches
|
| Preset | Directories in order | Filename groups |
|
||||||
`/usr/local/etc/scriptorium/config.yml` and then `/etc/scriptorium/config.yml`.
|
| --- | --- | --- |
|
||||||
ConfigKit performs this ordered search for each load. If neither exists, it
|
| WorkingDirectory | Current working directory | Named only |
|
||||||
uses built-in defaults; an error in a selected candidate does not cause a
|
| UserConfigDirectory | Go user-config directory / `scriptorium`, then the user-config directory | Named then Generic in the application directory; Named only in the base |
|
||||||
fallback. An explicit `--config` path must exist and decode successfully,
|
| HomeDirectory | Go home directory / `.scriptorium`, then the home directory | Named then Generic in the application directory; Named only in the base |
|
||||||
including when an empty path is explicitly supplied.
|
| UnixSystemDirectories | `/usr/local/etc/scriptorium`, `/etc/scriptorium`, then `/etc` | Named then Generic in application directories; Named only in `/etc` |
|
||||||
|
|
||||||
The maintained [minimal configuration](../examples/config.yml),
|
Named means `scriptorium.yml`, then `scriptorium.yaml`. Generic means
|
||||||
[complete configuration](../examples/config.full.yml),
|
`config.yml`, `config.yaml`, `main.yml`, then `main.yaml`.
|
||||||
[composed configuration](../examples/config.composed.yml),
|
ConfigKit uses native Go directory resolution (including its user-config/home
|
||||||
[collection configuration](../examples/config.collections.yml), and
|
environment behavior) and checks every filename in one directory before the next.
|
||||||
[framework inheritance configuration](../examples/config.inheritance.yml) are
|
Candidates are not configuration layers. Optional exhaustion uses application
|
||||||
copyable examples.
|
defaults; a selected-file acquisition/decoding error is terminal.
|
||||||
|
|
||||||
## Application Configuration File
|
ConfigKit assembles the main, enabled imports, and selected overlay, applies its
|
||||||
|
inheritance/composition rules, creates typed defaults and strictly decodes, applies
|
||||||
|
explicit override records, then runs application normalization and validation.
|
||||||
|
An override can correct a decoded value before validation, but cannot repair
|
||||||
|
source/composition/decoding errors. The completed result is authoritative:
|
||||||
|
runtime settings conversion does not merge, default, or normalize again.
|
||||||
|
|
||||||
Application configuration is decoded by ConfigKit v1.2.1 as strict YAML.
|
## Application Settings
|
||||||
Unknown fields, empty files, multiple YAML documents, and YAML aliases/anchors
|
|
||||||
are rejected.
|
|
||||||
An explicit empty string is a value and may clear a file-provided optional
|
|
||||||
string. A scalar `null` is not an empty string and is rejected for non-nullable
|
|
||||||
fields; nullable fields such as `transient_retries` retain their documented
|
|
||||||
meaning. Raw API-key fields and unknown application fields are rejected.
|
|
||||||
|
|
||||||
Only explicitly supplied application CLI flags override decoded settings. Paths
|
Complete copyable configurations are maintained under
|
||||||
in the application document remain relative to the process working directory, not to the file's
|
[examples](../examples/config.yml), including [full settings](../examples/config.full.yml),
|
||||||
directory. Load diagnostics are printed safely on stderr; invalid application
|
[application imports](../examples/config.composed.yml),
|
||||||
configuration exits with status 1.
|
[framework fragment composition](../examples/config.collections.yml),
|
||||||
|
[collection checkout discovery](../examples/config.discovery.yml), and
|
||||||
### Promptkit source selectors
|
[framework inheritance](../examples/config.inheritance.yml).
|
||||||
|
|
||||||
`promptkit_sources` is an ordered sequence of explicit Promptkit source
|
|
||||||
selectors. Each item must set exactly one of `file` or `directory`:
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
promptkit_sources:
|
promptkit:
|
||||||
- directory: ./examples/prompts
|
configuration: ./examples/promptkit.yml
|
||||||
- directory: ./examples/profiles
|
schema_dir: ./examples/schemas
|
||||||
- file: ./examples/framework/backends.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
```
|
```
|
||||||
|
|
||||||
An empty or null final resolved `promptkit_sources` list selects no
|
The `promptkit` section contains acquisition settings, not embedded framework
|
||||||
application-supplied framework sources; PromptKit's bundled OpenRouter underlay
|
definitions. Profiles, prompts, backends, and collections belong in the separate
|
||||||
still applies. A later sequence item overlays an earlier
|
framework document. The retired `promptkit_sources`, top-level `schema_dir`,
|
||||||
one. Files within a selected directory are composed together, so conflicting
|
and `defaults.render_format` fields have no compatibility conversion and are
|
||||||
definitions in that group fail instead of using filename order as precedence.
|
rejected by strict decoding, as are unknown/raw credential fields.
|
||||||
An explicit file may use ConfigKit imports. The complete framework format and
|
|
||||||
source rules belong to the [PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
|
||||||
|
|
||||||
Selector paths are relative to the process working directory, not to the
|
| Field | Default | Meaning |
|
||||||
application configuration file. Application validation checks the selector
|
| --- | --- | --- |
|
||||||
shape but does not open or validate Promptkit sources. `config validate`
|
| `promptkit.configuration` | empty | One external framework file; omission/empty selects no external main. |
|
||||||
therefore does not prove that selected sources can build an engine; use
|
| `promptkit.schema_dir` | `.` | Schema base directory passed to PromptKit. |
|
||||||
`inspect configuration validate` for that separate check. Repeated
|
| `transient_retries` | upstream default | Optional nonnegative engine retry count. Omitted/null delegates upstream; zero disables additional transient attempts. |
|
||||||
`--promptkit-source` flags replace the entire configured selector sequence
|
| `server.addr` | `:8080` | HTTP listen address. |
|
||||||
and preserve flag order. Without an authored collection policy, a selected
|
| `server.artifact_root` | empty | Root enabling HTTP file-artifact references. |
|
||||||
overlay replaces `promptkit_sources` as a whole; a declared sequence policy
|
| `server.max_request_bytes` | `16777216` | Encoded HTTP request limit; zero is unlimited. |
|
||||||
may instead combine the lists.
|
| `server.max_artifact_bytes` | `16777216` | HTTP file-artifact limit; zero is unlimited. |
|
||||||
|
| `server.max_response_bytes` | `16777216` | Encoded HTTP response limit; zero is unlimited. |
|
||||||
|
|
||||||
### Collection checkout discovery
|
Size fields are nonnegative signed 64-bit values. [HTTP behavior](api.md) defines
|
||||||
|
their enforcement; [operations](operations.md) covers safe deployment choices.
|
||||||
|
|
||||||
Select a framework discovery file using the existing file selector, as in
|
ConfigKit strictly decodes application YAML: empty files, unknown fields,
|
||||||
[config.discovery.yml](../examples/config.discovery.yml). Its native imports
|
multiple documents, aliases/anchors, and incompatible/null values fail under its
|
||||||
select both supported collection-entry spellings:
|
schema rules. Explicit empty strings and numeric zeros remain values. The
|
||||||
|
nullable retry field preserves null. Application paths are trimmed/cleaned and
|
||||||
|
the server address is trimmed within ConfigKit's resolver, with native
|
||||||
|
normalization evidence. Paths remain relative to process cwd, not the application
|
||||||
|
file's directory. No general environment-variable override layer is installed.
|
||||||
|
|
||||||
|
## Explicit Imports And Selected Overlays
|
||||||
|
|
||||||
|
Application imports use native root `config.imports` and are enabled during
|
||||||
|
loading. ConfigKit owns paths/directories/globs, requiredness, destination/item
|
||||||
|
placement, inheritance, sequence policy, and source-root confinement; consult its
|
||||||
|
[pinned contract][configkit]. Scriptorium does not widen the selected file's
|
||||||
|
canonical source root or implement another traversal/merge policy.
|
||||||
|
|
||||||
|
`--overlay NAME` passes the name unchanged to ConfigKit. At the selected release,
|
||||||
|
empty and exact lowercase `default` select none; other names use letters,
|
||||||
|
digits, underscores, or hyphens and select `overlays/NAME.yml` beneath the main
|
||||||
|
source root. A missing selected overlay fails. It is not a filesystem-path flag,
|
||||||
|
inherited stack, or fallback-root mechanism. An active overlay after absent-main
|
||||||
|
search exhaustion has no independent filesystem root in this application.
|
||||||
|
|
||||||
|
Only `serve` has named application override flags, for its HTTP settings.
|
||||||
|
Native application-scope `--set` records are passed directly to ConfigKit.
|
||||||
|
Nonconflicting records are supported; duplicate and ancestor/descendant paths
|
||||||
|
are rejected rather than resolved by order. Configuration commands have no
|
||||||
|
application-defined source/schema/HTTP selectors or comparison counterparts.
|
||||||
|
Native comparison selections and supported overrides are independent on each side.
|
||||||
|
|
||||||
|
## Framework Source Mapping
|
||||||
|
|
||||||
|
A nonempty resolved `promptkit.configuration` becomes an import-enabled
|
||||||
|
`configkit.File` main input to PromptKit. An empty value leaves its main absent.
|
||||||
|
The native bundled OpenRouter underlay remains enabled; no application source
|
||||||
|
list, directory expansion, or local precedence interpreter is installed.
|
||||||
|
|
||||||
|
Imports in that external document compose framework definitions using PromptKit's
|
||||||
|
schema and ConfigKit's native rules. The [maintained entry file](../examples/promptkit.yml)
|
||||||
|
imports the application's maintained prompt/profile fragments and backend
|
||||||
|
definitions. Peer conflicts are errors, not filename-order precedence.
|
||||||
|
Native whole-record replacement and inheritance remain framework-owned.
|
||||||
|
Application settings do not embed or re-decode framework definitions.
|
||||||
|
|
||||||
|
PromptKit captures message-file resources with their defining source context
|
||||||
|
when constructing the engine. Schema/artifact reads, credential resolution, and
|
||||||
|
provider work retain native preparation/runtime lifecycles. The server keeps one
|
||||||
|
engine until restarted; later CLI invocations perform fresh loads.
|
||||||
|
|
||||||
|
### Collection Checkout Discovery
|
||||||
|
|
||||||
|
The [discovery configuration](../examples/config.discovery.yml) selects
|
||||||
|
[one framework entry](../examples/discovery/framework.yml) with two optional imports:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
config:
|
config:
|
||||||
@@ -90,195 +129,30 @@ config:
|
|||||||
- glob: collections/*/collection.yaml
|
- glob: collections/*/collection.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Both globs are optional, so either spelling may have no matches. Paths are
|
Each independently maintained checkout declares its namespace and explicitly
|
||||||
relative to the framework file; this example's `*` selects one checkout
|
imports its own definitions relative to its entry file. Checkout names do not
|
||||||
level and is not a general layout restriction. Each entry declares its own
|
determine resource IDs. The [support example](../examples/discovery/collections/support-repo/collection.yml)
|
||||||
collection namespace and imports only its intended resource fragments. The
|
imports multiple profiles/prompts and a source-relative message; the
|
||||||
checkout directory name does not determine the namespace. New checkouts are
|
[review example](../examples/discovery/collections/review-repo/collection.yaml)
|
||||||
available on the next invocation without editing this file; a running server
|
uses the other spelling. Arbitrary checkout YAML is not loaded implicitly.
|
||||||
does not reload them. See the [maintained collection examples](../examples/discovery/collections/).
|
|
||||||
|
|
||||||
Entries imported by one discovery file are composed together as peers.
|
Either glob may have no matches. New checkouts become visible on a later
|
||||||
Conflicting definitions of the same qualified resource ID fail to load;
|
invocation, not through server hot reload. Entries in one discovery document are
|
||||||
filename, checkout, and glob order do not set precedence. To override a
|
peers: conflicting qualified IDs fail instead of relying on glob order.
|
||||||
discovered record, select a separate framework file later in
|
|
||||||
`promptkit_sources` and supply the complete replacement record.
|
|
||||||
|
|
||||||
### Explicit imports and selected overlays
|
## Inspection And Credentials
|
||||||
|
|
||||||
Application files declare imports under root `config.imports`. Native `path`,
|
The native `app` scope exposes the actual application result without acquiring
|
||||||
`dir`, and `glob` selectors, filters, recursion, destinations, and item
|
framework definitions or credentials. The `promptkit` scope exposes the captured
|
||||||
placement are available as described in the [pinned ConfigKit contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md).
|
framework result selected by the application config/overlay. They are not merged
|
||||||
Imports are optional by default; set `required: true` when absence must fail.
|
into one synthetic document. See [CLI inspection](cli.md#application-configuration-inspection)
|
||||||
The old `optional` field is not supported. Directory selection has no default
|
for operations and independent comparison.
|
||||||
filename filter; author a pattern when only YAML files are intended. ConfigKit
|
|
||||||
composes peer sources strictly, rather than using filename order to resolve
|
|
||||||
conflicts. Imports are resolved relative to their declaring file within the
|
|
||||||
native source root. Scriptorium does not widen that root.
|
|
||||||
|
|
||||||
Root `config.sequences` can declare an identity key and merge mode for a
|
PromptKit owns profiles, namespaces, provider controls, repairs, retries, and
|
||||||
collection. Scriptorium does not register or infer an identity for
|
credential resolution. Store credential environment-variable names, not raw
|
||||||
`promptkit_sources`: absent an authored policy, unkeyed lists replace across
|
values; supply the selected environment to the process. Native framework
|
||||||
precedence boundaries. The [collection example](../examples/config.collections.yml)
|
redaction is limited to classified fields and history, not arbitrary sensitive
|
||||||
declares `file` identity with `upsert`, combines imported file selectors, and
|
content. Protect application, prompt, prepared, and generated reports as data.
|
||||||
lets an overlay add another. CLI list overrides still replace the whole list.
|
|
||||||
|
|
||||||
At application precedence boundaries, omission preserves lower-precedence
|
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md
|
||||||
selectors. Under ordinary list replacement, `[]` clears the selection and
|
[configkit]: https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md
|
||||||
`null` resolves to no selectors. Under authored `append` or `upsert`, `[]`
|
|
||||||
contributes no records and `null` is invalid. To clear a collection with an
|
|
||||||
inherited policy, declare `mode: replace` at
|
|
||||||
`destination: [promptkit_sources]` in the higher-precedence source and supply
|
|
||||||
`promptkit_sources: []`. See
|
|
||||||
[ConfigKit's collection policy contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#yaml-declarations-and-policy-lifetime)
|
|
||||||
for policy lifetime and combination rules.
|
|
||||||
|
|
||||||
These application root `config` controls do not change the separate
|
|
||||||
PromptKit v0.23.3 framework format: its selected framework files use the
|
|
||||||
[Promptkit contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
|
||||||
and their own native ConfigKit `config` controls.
|
|
||||||
|
|
||||||
`--config-overlay NAME` selects one file at `overlays/NAME.yml` below that
|
|
||||||
root. The name must use letters, digits, `_`, or `-`; it is not a path. Empty
|
|
||||||
and exact lowercase `default` select no overlay. There is no fallback,
|
|
||||||
case folding, inherited stack, or automatic selection; a missing selected
|
|
||||||
overlay fails. Mappings merge; sequence contributions and nulls at collection
|
|
||||||
destinations follow the effective collection policy described above. Other
|
|
||||||
explicit nulls replace lower values subject to the application schema.
|
|
||||||
Defaults-only loading can enable imports, but an absent main has no import
|
|
||||||
declarations. This CLI supplies no independent overlay root, so an active
|
|
||||||
overlay cannot be located without a selected main context. That is not a general
|
|
||||||
ConfigKit restriction.
|
|
||||||
|
|
||||||
The [composed example](../examples/config.composed.yml) imports the application
|
|
||||||
[source-selection fragment](../examples/config-fragments/sources.yml) and has
|
|
||||||
a selectable [local overlay](../examples/overlays/local.yml). That fragment
|
|
||||||
selects the [framework backend file](../examples/framework/backends.yml) through
|
|
||||||
`promptkit_sources`; backend definitions are not imported into the application
|
|
||||||
document. From the repository root:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go run ./cmd/scriptorium render \
|
|
||||||
--config ./examples/config.composed.yml --config-overlay local \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
|
||||||
--input glossary=./examples/fixtures/glossary.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Inspect the collection overlay and render without contacting a provider:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go run ./cmd/scriptorium config show --config ./examples/config.collections.yml --config-overlay collection
|
|
||||||
go run ./cmd/scriptorium render --config ./examples/config.collections.yml --config-overlay collection \
|
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
|
||||||
--input glossary=./examples/fixtures/glossary.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Import and overlay file locations are constrained by ConfigKit; application
|
|
||||||
path values such as `promptkit_sources`, `schema_dir`, and `server.artifact_root`
|
|
||||||
remain relative to the process working directory even when authored in a
|
|
||||||
fragment or overlay. These source constraints are separate from Promptkit
|
|
||||||
message-resource paths and HTTP artifact containment.
|
|
||||||
|
|
||||||
| Field | Default | Meaning |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `promptkit_sources` | empty sequence | Ordered file/directory sources for Promptkit backends, profiles, and prompts. |
|
|
||||||
| `schema_dir` | `.` | Promptkit schema source directory for relative schema paths. |
|
|
||||||
| `server.addr` | `:8080` | Address used by `serve`. |
|
|
||||||
| `server.artifact_root` | unset | Root that enables HTTP `file` input references. |
|
|
||||||
| `server.max_request_bytes` | `16777216` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
|
|
||||||
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
|
||||||
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
|
||||||
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
|
|
||||||
| `transient_retries` | Promptkit default | Optional nonnegative count of additional built-in transient generation attempts. Omitted or `null` delegates to Promptkit; `0` disables additional attempts. |
|
|
||||||
|
|
||||||
The size fields must be zero or greater. The [HTTP API](api.md) defines how
|
|
||||||
each limit is enforced and reported. `server.artifact_root` configures an HTTP
|
|
||||||
deployment boundary; see [operations](operations.md) for deployment handling.
|
|
||||||
|
|
||||||
The `config validate`, `config show`, `config sources`, and `config explain`
|
|
||||||
commands inspect this application configuration without creating a Promptkit
|
|
||||||
engine. `show` is a resolved inspection snapshot, not necessarily a reloadable
|
|
||||||
file. Application validation does not check framework sources, backend
|
|
||||||
connectivity, or credential availability. Use `inspect configuration` for
|
|
||||||
Promptkit's separately captured framework configuration; its four operations
|
|
||||||
are documented in the [CLI reference](cli.md#framework-configuration-inspection).
|
|
||||||
`config explain --path /server/addr` uses JSON Pointer syntax to show the
|
|
||||||
selected field's current value and history from defaults, files, overlays,
|
|
||||||
explicit CLI overrides, and normalization. `--path ''` selects the root.
|
|
||||||
History includes replaced and removed values; `show` presents the final
|
|
||||||
resolved snapshot. The application schema rejects raw credential fields, but
|
|
||||||
do not treat inspection as a general-purpose filter for sensitive strings.
|
|
||||||
|
|
||||||
## Framework Source Mapping
|
|
||||||
|
|
||||||
Scriptorium assembles selected Promptkit source files and directories in the
|
|
||||||
order described above, then constructs the Promptkit engine. Framework
|
|
||||||
definitions and custom backends belong in those selected source files, not in
|
|
||||||
the Scriptorium application configuration. Message `content_file` paths use
|
|
||||||
the defining source's resource context and are captured when the engine is
|
|
||||||
constructed; schema and request-artifact loading remain preparation-time
|
|
||||||
operations. Scriptorium does not redefine or independently parse framework
|
|
||||||
file formats.
|
|
||||||
|
|
||||||
- Prompt selection by ID, message templates, inputs, output contracts, and
|
|
||||||
session IDs are Promptkit contracts.
|
|
||||||
- Profile fields, numeric ranges, execution defaults, overlay precedence,
|
|
||||||
built-in profiles, and credential rules are Promptkit contracts.
|
|
||||||
- Schema path behavior and generated-content validation are Promptkit
|
|
||||||
contracts.
|
|
||||||
|
|
||||||
See the
|
|
||||||
[tagged Promptkit format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
|
||||||
for all of those definitions. The files under
|
|
||||||
[`examples/prompts`](../examples/prompts/),
|
|
||||||
[`examples/profiles`](../examples/profiles/), and
|
|
||||||
[`examples/schemas`](../examples/schemas/) are maintained Scriptorium
|
|
||||||
application inputs using that tagged format.
|
|
||||||
|
|
||||||
PromptKit supplies a default-on OpenRouter bundle as the lowest-priority
|
|
||||||
underlay. Choose an actual bundled profile such as `auto`, or a complete
|
|
||||||
application-owned profile; old catalog IDs are not aliases. Caller definitions
|
|
||||||
can replace bundled records by normal composition. Bundle availability is not
|
|
||||||
a readiness check or a promise of a fixed provider model.
|
|
||||||
|
|
||||||
## Credentials And Outbound Behavior
|
|
||||||
|
|
||||||
Scriptorium maps `--api-key-env` and HTTP `model.api_key_env` into Promptkit
|
|
||||||
request overrides. Keep secret values in environment variables and store only
|
|
||||||
their names in configuration or framework source files. Do not place raw keys
|
|
||||||
in configuration, prompts, profiles, CLI arguments, examples, or HTTP
|
|
||||||
payloads.
|
|
||||||
|
|
||||||
An optional `api_key_env` whose environment value is absent or empty can result
|
|
||||||
in an unauthenticated provider request. A profile that declares credentials
|
|
||||||
required still fails as an invalid request when no credential source is
|
|
||||||
selected, and fails with `ErrAPIKeyEnvMissing` when its selected environment
|
|
||||||
source is absent or empty. Scriptorium never reads or emits the environment
|
|
||||||
value itself.
|
|
||||||
|
|
||||||
Unset optional provider controls are omitted from compatible provider requests.
|
|
||||||
A positive Promptkit `repair_attempts` budget can add provider calls, latency,
|
|
||||||
token use, and cost; see the tagged format reference for its permitted values
|
|
||||||
and validation-mode requirements.
|
|
||||||
|
|
||||||
Promptkit's built-in client makes up to two additional attempts for eligible
|
|
||||||
transient HTTP statuses by default. The application `transient_retries` setting
|
|
||||||
changes that budget for its constructed engine; an explicit `run` request
|
|
||||||
override takes precedence. This can increase latency, request count, and cost.
|
|
||||||
Each initial or repair generation has its own transient budget; an injected
|
|
||||||
client retains its own retry behavior. The tagged integration contract owns
|
|
||||||
eligibility and timing.
|
|
||||||
|
|
||||||
Promptkit's
|
|
||||||
[OpenAI-compatible integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
|
||||||
defines outbound authentication, provider request mapping, transport limits,
|
|
||||||
and timeout layering.
|
|
||||||
|
|
||||||
## Related References
|
|
||||||
|
|
||||||
- [CLI reference](cli.md)
|
|
||||||
- [HTTP API reference](api.md)
|
|
||||||
- [Operations guide](operations.md)
|
|
||||||
- [Promptkit framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ and the HTTP service according to the boundary your application needs.
|
|||||||
- CLI subprocess: [subprocess integration](../integrations/subprocess.md)
|
- CLI subprocess: [subprocess integration](../integrations/subprocess.md)
|
||||||
- HTTP service: [HTTP API reference](../api.md)
|
- HTTP service: [HTTP API reference](../api.md)
|
||||||
- Application configuration: [configuration reference](../config.md)
|
- Application configuration: [configuration reference](../config.md)
|
||||||
|
- Existing CLI callers: [native CLI migration](upgrading-cli.md)
|
||||||
|
|
||||||
Go applications that need an in-process prompt framework should import
|
Go applications that need an in-process prompt framework should import
|
||||||
`go.maximumdirect.net/promptkit` directly. The tagged
|
`go.maximumdirect.net/promptkit` directly. The tagged
|
||||||
@@ -27,7 +28,7 @@ Consumers are responsible for:
|
|||||||
- supplying required inputs and template variables;
|
- supplying required inputs and template variables;
|
||||||
- supplying credentials through the chosen interface;
|
- supplying credentials through the chosen interface;
|
||||||
- protecting rendered prompts and generated artifacts as potentially
|
- protecting rendered prompts and generated artifacts as potentially
|
||||||
sensitive;
|
sensitive, including native inspection and full result reports;
|
||||||
- deciding whether validation-failed output is usable; and
|
- deciding whether validation-failed output is usable; and
|
||||||
- retrying only when another model call is acceptable.
|
- retrying only when another model call is acceptable.
|
||||||
|
|
||||||
|
|||||||
95
docs/consumers/upgrading-cli.md
Normal file
95
docs/consumers/upgrading-cli.md
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
# Migrate To The Native CLI
|
||||||
|
|
||||||
|
This guide covers Scriptorium's breaking CommandKit-native CLI cutover. It does
|
||||||
|
not select a release version or change historical release notes. The
|
||||||
|
[CLI reference](../cli.md) owns current syntax, streams, and statuses;
|
||||||
|
[configuration](../config.md) owns source discovery and settings.
|
||||||
|
The earlier [PromptKit upgrade](upgrading-promptkit.md) covers framework formats,
|
||||||
|
not a compatibility layer for this interface.
|
||||||
|
|
||||||
|
## Replace Commands And Selections
|
||||||
|
|
||||||
|
| Former interface | Current interface |
|
||||||
|
| --- | --- |
|
||||||
|
| `run --prompt ID` / `--prompt-id` | `run ID`; exactly one positional prompt ID. |
|
||||||
|
| `render --prompt ID` | `promptkit show prepared ID`. |
|
||||||
|
| `inspect prompt --prompt ID` | `promptkit show prompt ID`. |
|
||||||
|
| `inspect profile --profile ID` | `promptkit show profile ID`. |
|
||||||
|
| `inspect configuration ...` | Native `config ... --scope promptkit`. |
|
||||||
|
| Application `config ...` | Native `config ... --scope app` (the default scope), including diff. |
|
||||||
|
| `--config-overlay` | `--overlay NAME`; a native overlay name, not an arbitrary path. |
|
||||||
|
| `--input NAME=PATH` | `--input-file NAME=PATH`; `--input` now means literal text. |
|
||||||
|
| `--llm-base-url` | `--endpoint` (canonical `--llm-endpoint`). |
|
||||||
|
| Duration-valued `--timeout` | Integer `--timeout-seconds` for each generation attempt. |
|
||||||
|
| `--out` / output-file flags | Caller stdout redirection. |
|
||||||
|
| `--execution-report` | `--report PATH` with native result rendering. |
|
||||||
|
|
||||||
|
Prefer native short aliases such as `--profile`, `--var`, `--input-text`, and
|
||||||
|
`--input-file`; canonical `--llm-*` spellings remain available.
|
||||||
|
There is no duplicate prompt flag, direct backend-ID override, or retained
|
||||||
|
`--profile-id` alias on generation.
|
||||||
|
|
||||||
|
Repeat mappings rather than comma-separating them. The first equals sign splits
|
||||||
|
each mapping; commas and later equals signs are literal. Input text/file
|
||||||
|
occurrences share native last-occurrence precedence.
|
||||||
|
|
||||||
|
## Migrate Application Configuration
|
||||||
|
|
||||||
|
Replace the source-selector list and top-level schema directory with
|
||||||
|
`promptkit.configuration` and `promptkit.schema_dir`.
|
||||||
|
The [maintained example](../../examples/config.yml) selects a separate framework
|
||||||
|
entry file that imports prompt/profile/backend definitions through native
|
||||||
|
ConfigKit composition. No source/schema CLI flags or compatibility conversion
|
||||||
|
remain. Do not embed framework definitions inside the application section.
|
||||||
|
|
||||||
|
Discovery now uses native cwd, user-config, home, then Unix-system presets and
|
||||||
|
their filename conventions. It is no longer limited to the two former
|
||||||
|
`config.yml` paths. Set an explicit config path when migrating unattended jobs.
|
||||||
|
Overlays remain native name selections. Application-scope native `--set` records
|
||||||
|
retain conflict rejection; order does not make duplicate paths authoritative.
|
||||||
|
There are no custom source/schema/HTTP selectors under `config`.
|
||||||
|
|
||||||
|
## Capture Artifacts And Reports
|
||||||
|
|
||||||
|
For an artifact file plus full native report, from the repository root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scriptorium run generic.markdown_summary --config examples/config.yml \
|
||||||
|
--input-file transcript=examples/fixtures/transcript.md \
|
||||||
|
--input-file glossary=examples/fixtures/glossary.yml \
|
||||||
|
--report run-result.json --report-format json > summary.md
|
||||||
|
```
|
||||||
|
|
||||||
|
This generation example contacts the configured provider. Use native prepared
|
||||||
|
preview instead when only checking inputs/settings.
|
||||||
|
|
||||||
|
Main automatic/text output is the artifact alone, with no added newline.
|
||||||
|
Main JSON/YAML is the native result; artifact bytes use native structured
|
||||||
|
serialization. Separate reports contain that full native result, not the old
|
||||||
|
observer-derived metadata shape. `--report-format` independently chooses
|
||||||
|
text/YAML/JSON; omission leaves the format unset for native automatic rendering
|
||||||
|
(currently YAML for structured values). Extensions do not select formats.
|
||||||
|
|
||||||
|
Omit quiet to capture main output, including redirected stdout. Quiet still
|
||||||
|
allows an explicitly requested report file. Reports are available only when a
|
||||||
|
result is returned; resultless failures do not replace an existing report.
|
||||||
|
Partial delivery can leave one or both destinations written. Keep paths separate
|
||||||
|
and check status/diagnostics before treating files as fresh.
|
||||||
|
|
||||||
|
## Adjust Status And Sensitivity Handling
|
||||||
|
|
||||||
|
Completed generated-content validation failure is now domain status 4, not the
|
||||||
|
old status 2. Native usage/invalid-configuration input is 2, diff-check
|
||||||
|
differences are 3, and operational failures are 1. Presentation/file failures
|
||||||
|
can change success into failure while preserving an already nonzero domain
|
||||||
|
classification. The [CLI status table](../cli.md#server-and-statuses) owns details.
|
||||||
|
|
||||||
|
Native prompt inspection includes messages; prepared reports include rendered
|
||||||
|
content; full run reports include generated content and metadata. Framework
|
||||||
|
classified-field redaction does not make arbitrary report data safe to expose.
|
||||||
|
|
||||||
|
Requirements discovery is separate from validation/preparation: it can run
|
||||||
|
before artifacts exist, and a partial report may succeed. Read its native
|
||||||
|
completeness/issue fields. Smoke tests are explicit provider operations, not
|
||||||
|
startup checks. See [subprocess guidance](../integrations/subprocess.md) for
|
||||||
|
stream capture, cancellation, and caller responsibilities.
|
||||||
@@ -1,33 +1,23 @@
|
|||||||
# Upgrade PromptKit Sources And Definitions
|
# Upgrade PromptKit Sources And Definitions
|
||||||
|
|
||||||
This guide covers the CLI and framework configuration cutover to
|
This guide covers the earlier framework configuration cutover to
|
||||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md)
|
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md)
|
||||||
and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||||
for the full upstream contract.
|
for the full upstream contract. The subsequent [native CLI migration](upgrading-cli.md)
|
||||||
|
owns command, reporting, application-discovery, and source-setting changes.
|
||||||
|
|
||||||
Removing version selectors and changing framework definition shapes are
|
Removing version selectors and changing framework definition shapes are
|
||||||
intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility).
|
intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility).
|
||||||
|
|
||||||
## Application configuration
|
## Application configuration
|
||||||
|
|
||||||
Existing application fields that already select the desired sources retain
|
Framework definition migration and application acquisition are separate changes.
|
||||||
their meaning. For example, this application configuration works both before
|
Rewrite older framework files as below, then select their entry file using the
|
||||||
and after the cutover:
|
current [application configuration](../config.md#application-settings).
|
||||||
|
The old source-selector list and top-level schema root are not current aliases;
|
||||||
```yaml
|
follow the [CLI migration guide](upgrading-cli.md#migrate-application-configuration).
|
||||||
promptkit_sources:
|
Scriptorium does not provide a legacy definition reader or embedded-definition
|
||||||
- directory: ./examples/prompts
|
extraction/reload path.
|
||||||
- directory: ./examples/profiles
|
|
||||||
- file: ./examples/framework/backends.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
```
|
|
||||||
|
|
||||||
The selected framework files must still be rewritten as described below.
|
|
||||||
Scriptorium uses one native application schema: discovery, imports, selected
|
|
||||||
overlays, source order, explicit overrides, and working-directory-relative
|
|
||||||
selector paths remain application configuration behavior. It loads framework
|
|
||||||
sources separately with PromptKit's schema. No second application schema or
|
|
||||||
legacy definition reader is provided. See the [configuration reference](../config.md).
|
|
||||||
|
|
||||||
## Rewrite framework definitions
|
## Rewrite framework definitions
|
||||||
|
|
||||||
@@ -82,9 +72,9 @@ Convert framework source controls from `configkit:` to `config:` and add
|
|||||||
header, if present, is `version: 2`. Use sibling inheritance such as
|
header, if present, is `version: 2`. Use sibling inheritance such as
|
||||||
`extends: {id: base}`.
|
`extends: {id: base}`.
|
||||||
|
|
||||||
Across ordered `promptkit_sources` entries, a later framework underlay replaces
|
The single external framework entry uses native ConfigKit import composition;
|
||||||
the whole record for the same ID. Supply the complete intended replacement,
|
there is no application-managed source-list precedence. Native record replacement
|
||||||
or create a distinct inherited record. Checkouts imported by the same
|
and inheritance remain upstream-owned. Checkouts imported by the same
|
||||||
[discovery file](../config.md#collection-checkout-discovery) are peers:
|
[discovery file](../config.md#collection-checkout-discovery) are peers:
|
||||||
conflicting definitions of the same ID fail instead of overriding one another.
|
conflicting definitions of the same ID fail instead of overriding one another.
|
||||||
|
|
||||||
@@ -106,5 +96,5 @@ records can replace it through ordinary composition.
|
|||||||
For multiple independently maintained repositories, see the
|
For multiple independently maintained repositories, see the
|
||||||
[collection-discovery example](../../examples/config.discovery.yml). Its
|
[collection-discovery example](../../examples/config.discovery.yml). Its
|
||||||
entry files declare their own namespaces; checkout names do not become IDs.
|
entry files declare their own namespaces; checkout names do not become IDs.
|
||||||
The [CLI reference](../cli.md) demonstrates qualified rendering and the
|
The [CLI reference](../cli.md) defines qualified selection and prepared preview; the
|
||||||
[operations guide](../operations.md) describes when updates take effect.
|
[operations guide](../operations.md) describes when updates take effect.
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ application that consumes the public
|
|||||||
[PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md);
|
[PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md);
|
||||||
framework implementation work belongs in Promptkit. Application configuration
|
framework implementation work belongs in Promptkit. Application configuration
|
||||||
uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local.
|
uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local.
|
||||||
|
Operator interfaces use CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0;
|
||||||
|
standard command families and presentation remain upstream-owned.
|
||||||
PromptKit uses the same ConfigKit release to load framework sources under its
|
PromptKit uses the same ConfigKit release to load framework sources under its
|
||||||
own separate schema and format contract.
|
own separate schema and format contract.
|
||||||
|
|
||||||
@@ -46,8 +48,8 @@ Before recommending, planning, or implementing work:
|
|||||||
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) |
|
| CLI commands, flags, output, or exit behavior | [CLI contract](cli.md) and [adapter internals](internal/adapters.md) |
|
||||||
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||||
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests |
|
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests |
|
||||||
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` |
|
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and the [native CLI contract](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md) |
|
||||||
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#promptkit-source-selectors), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
|
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#framework-source-mapping), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
|
||||||
| Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) |
|
| Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) |
|
||||||
| OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) |
|
| OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) |
|
||||||
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
||||||
|
|||||||
@@ -1,64 +1,67 @@
|
|||||||
# Subprocess Integration
|
# Subprocess Integration
|
||||||
|
|
||||||
This document covers process-boundary behavior for callers that invoke
|
Use the executable when a local process boundary is appropriate.
|
||||||
Scriptorium as a child process. Command syntax, flags, output, and exit codes
|
The [CLI contract](../cli.md) owns exact command/flag/status syntax; the
|
||||||
are defined by the [CLI reference](../cli.md). Interface selection belongs in
|
[consumer overview](../consumers/api.md) compares subprocess and HTTP use.
|
||||||
the [consumer integration overview](../consumers/api.md).
|
|
||||||
|
|
||||||
## Process Contract
|
## Artifact-First Execution And Inspection
|
||||||
|
|
||||||
Use `scriptorium render` when the caller needs prepared output without a model
|
Select a stored prompt and capture its generated artifact from stdout. Use native
|
||||||
call, and `scriptorium run` for generation. Pass an explicit `--config` or
|
requirements discovery before generated input files exist, concrete validation
|
||||||
make the configuration search paths available to the child process; configuration
|
once bindings exist, and prepared preview to render without a model call.
|
||||||
discovery, fields, profile selection, and credential mechanisms are defined in
|
The [CLI migration guide](../consumers/upgrading-cli.md) explains replacements
|
||||||
the [configuration reference](../config.md).
|
for older command and file-output conventions.
|
||||||
|
|
||||||
Pass required API-key environment variables through the child environment. Do
|
Pass an explicit application configuration or deliberately arrange the child's
|
||||||
not place raw API keys in arguments. Keep the environment limited to the values
|
[configuration discovery](../config.md). Supply only the environment needed for
|
||||||
needed for the selected profile.
|
selected credentials; never put raw keys in arguments.
|
||||||
|
|
||||||
For application-only preflight, invoke `scriptorium config validate`; it exits
|
Discovery completeness is distinct from binding satisfaction and execution
|
||||||
0 with empty stdout when the application configuration loads, but does not
|
readiness. Partial reports can succeed. Consumers own their binding graph and
|
||||||
prove that Promptkit sources or credentials are usable. `config show` emits
|
policy for incomplete findings; Scriptorium supplies no workflow state.
|
||||||
resolved YAML and `config sources` emits newline-terminated JSON provenance.
|
|
||||||
All three use the same source selection and explicit overrides as execution
|
|
||||||
commands and require separate stderr capture for diagnostics. Their output can
|
|
||||||
include source paths, identifiers, credential reference names, and ordinary
|
|
||||||
backend parameters; treat it as potentially sensitive.
|
|
||||||
|
|
||||||
## Streams And Output Ownership
|
## Streams, Reports, And Availability
|
||||||
|
|
||||||
Capture stdout and stderr separately. Stdout contains the requested artifact or
|
Capture stdout and stderr separately. Diagnostics are not a machine-readable
|
||||||
prepared output unless the caller selects an output file; stderr contains
|
error protocol, even when structured result output is selected. Prefer exit
|
||||||
summaries, diagnostics, and server messages. The exact destinations and status
|
classification and native report fields; do not parse diagnostic prose.
|
||||||
meanings are part of the [CLI reference](../cli.md), not a stable stderr data
|
|
||||||
protocol.
|
|
||||||
|
|
||||||
When using `--out`, the caller owns the output path, its permissions, and
|
The default generated result is artifact content. Native structured results and
|
||||||
cleanup. Treat rendered prompts, generated artifacts, stdout, and stderr as
|
optional separate reports carry the full returned result, including content and
|
||||||
potentially sensitive.
|
metadata. They are not the former metadata-only CLI observation report. Native
|
||||||
|
prompt/prepared inspection also carries content. Treat all captured streams and
|
||||||
|
reports as potentially sensitive.
|
||||||
|
|
||||||
For `run`, an optional [execution report file](../cli.md) contains terminal
|
To save main output, supply a file as the subprocess stdout destination or use
|
||||||
metadata separate from generated stdout or `--out`. The caller owns that
|
shell redirection. Omit quiet when that main result is needed: quiet suppresses
|
||||||
path's permissions and retention. Output and report files are not a
|
it even for redirected stdout. A separately requested report remains an
|
||||||
transaction: one may be written when the other fails, and a preexisting report
|
application file effect independent of quiet and main format selection.
|
||||||
may remain when the invocation has no terminal event. Check the exit status
|
|
||||||
and diagnostics rather than assuming a report path contains this run's data.
|
A completed content-validation failure can still deliver an artifact/result;
|
||||||
|
consumers decide whether it is usable. An operational failure with no result
|
||||||
|
produces diagnostics and a nonzero status, not a synthesized result/error
|
||||||
|
envelope. An existing separate report may then describe an older invocation.
|
||||||
|
|
||||||
|
## Partial Delivery And Caller Responsibility
|
||||||
|
|
||||||
|
Main output is attempted before a requested report, and report delivery is
|
||||||
|
attempted even after main presentation failure. Either destination may contain
|
||||||
|
partial data after an error. File delivery is not transactional and has no
|
||||||
|
rollback or atomic-replacement promise.
|
||||||
|
|
||||||
|
Callers own redirected file creation/truncation, permissions, retention, and
|
||||||
|
keeping destinations separate. A shell may truncate its file before the process
|
||||||
|
starts, unlike a separate report that is opened only after a result exists.
|
||||||
|
Check the finalized exit status and diagnostic stream; do not infer success or
|
||||||
|
freshness from a file's existence.
|
||||||
|
|
||||||
## Cancellation And Recovery
|
## Cancellation And Recovery
|
||||||
|
|
||||||
A CLI invocation performs one synchronous request and creates no durable run
|
Interrupt/SIGTERM cancels the child's invocation context. Engine cancellation is
|
||||||
state. A supervising process that needs cancellation must terminate the child
|
cooperative. The [whole-operation budget](../cli.md#generation) is separate from
|
||||||
process according to its own process-management policy. A later invocation is a
|
per-attempt provider timeouts and excludes process/configuration setup and
|
||||||
new request and can make another model call; there is no resume or checkpoint
|
delivery. Supervisors still own forced process termination when required.
|
||||||
protocol.
|
|
||||||
|
|
||||||
For `run`, callers may set the [whole-run deadline](../cli.md) to bound the
|
A new invocation is a new request that may incur another provider call/cost.
|
||||||
engine invocation. It is separate from the per-attempt generation timeout and
|
There is no checkpoint, archive, conversation, or resume protocol. Deployment and
|
||||||
does not include child startup or output writing. A typed operation deadline
|
sensitive-data handling belong to [operations](../operations.md).
|
||||||
failure exits 1.
|
|
||||||
Supervisors still own process termination when they need to stop the child
|
|
||||||
outside that engine interval.
|
|
||||||
|
|
||||||
For deployment, filesystem permissions, and sensitive-artifact handling, see
|
|
||||||
the [operations guide](../operations.md).
|
|
||||||
|
|||||||
@@ -1,157 +1,89 @@
|
|||||||
# Adapter Internals
|
# Adapter Internals
|
||||||
|
|
||||||
## Purpose
|
Adapters map executable inputs and effects through public upstream values.
|
||||||
|
The [CLI](../cli.md), [HTTP API](../api.md), and [architecture policy](../policy/architecture.md)
|
||||||
|
own their external contracts and boundaries.
|
||||||
|
|
||||||
Scriptorium adapters translate executable inputs into Promptkit public requests
|
## CLI Composition And Acquisition
|
||||||
and translate Promptkit results or errors back to CLI or HTTP behavior. They
|
|
||||||
own IO and presentation mechanics, not framework decisions.
|
|
||||||
|
|
||||||
External contracts are canonical in the [CLI reference](../cli.md) and
|
`app.go` composes application run/serve declarations with native configuration,
|
||||||
[HTTP API reference](../api.md). Promptkit's public engine contract is
|
PromptKit, completion, help, and version support. CommandKit owns parsing,
|
||||||
described by its tagged
|
declaration-input checks, reporting selection, presentation, and finalization.
|
||||||
[Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md).
|
|
||||||
|
|
||||||
## Components And Collaborators
|
`selection.go` maps explicit native config/overlay choices and server-only HTTP
|
||||||
|
bindings into `internal/config.Load`. It preserves supplied empty/zero values.
|
||||||
|
Native application `--set` records remain upstream records with native conflict
|
||||||
|
handling. No application selectors are installed under configuration commands.
|
||||||
|
|
||||||
- `cmd/scriptorium` passes process arguments and streams to
|
`native_config.go` adapts the actual application result through
|
||||||
`internal/adapter/cli`.
|
`ckconfig.FromConfigKit` and framework inspection through `ckprompt.ConfigScope`.
|
||||||
- `internal/adapter/cli` resolves settings through `internal/config`,
|
Native views own effective values, paths, metadata, sources, findings, explanations,
|
||||||
constructs `promptkit.Engine`, maps CLI values to `promptkit.RunRequest`,
|
redaction, and comparison. Application inspection does not construct an engine.
|
||||||
exposes application and captured framework configuration inspection, and
|
Comparison loads each side from that side's selection; there is no config/engine
|
||||||
owns output files, summaries, and exit codes.
|
cache or effective-document reconstruction.
|
||||||
- `internal/config` delegates application YAML decoding, default resolution,
|
|
||||||
overrides, and validation ordering to ConfigKit v1.2.1. The CLI supplies only
|
|
||||||
explicitly visited application flags as overrides and renders safe structured
|
|
||||||
load diagnostics on stderr. PromptKit uses the same ConfigKit release to
|
|
||||||
load framework sources under its own definition and format contract. Both
|
|
||||||
domains use ConfigKit's native diagnostic writer.
|
|
||||||
- `internal/adapter/http` strictly decodes request DTOs, maps them to Promptkit
|
|
||||||
public values, calls its adapter-owned `Runner` interface, and maps results
|
|
||||||
and errors to HTTP DTOs.
|
|
||||||
- `internal/format` renders `promptkit.PreparedRun` values as deterministic text
|
|
||||||
or JSON.
|
|
||||||
- `internal/report` maps Promptkit's terminal execution event to a small
|
|
||||||
application-owned value without invoking the engine or writing output.
|
|
||||||
|
|
||||||
## Wiring Flows
|
`acquisitionState` contains application diagnostic evidence only, never loaded
|
||||||
|
configuration or engines. Callback decorators forward it through native
|
||||||
|
`Acquired` channels. An application LoadError is transported separately while
|
||||||
|
the framework adapter runs, then its original typed cause is restored before
|
||||||
|
native outcome classification. This prevents app findings from becoming
|
||||||
|
framework findings. Native framework evidence remains unchanged; comparison
|
||||||
|
application messages identify left/right context. Callback marks prevent replay
|
||||||
|
when operations share invocation state.
|
||||||
|
|
||||||
### CLI
|
`native_prompt.go` installs all twelve ckprompt callbacks, nil request builders,
|
||||||
|
and empty-default native namespace views. Inventories, named resources,
|
||||||
|
requirements, validation, preparation, and smoke testing retain their upstream
|
||||||
|
lifecycles and reports.
|
||||||
|
|
||||||
`run` calls `promptkit.Engine.Run`; `render` calls
|
## Generation And Delivery
|
||||||
`promptkit.Engine.Prepare`. Both share request mapping for exact prompt ID
|
|
||||||
and profile selection, optional file inputs, variables, and presence-aware
|
|
||||||
execution overrides. Omitted framework settings remain zero values so Promptkit
|
|
||||||
resolves its own defaults and definition-required inputs.
|
|
||||||
The application retry setting enters shared engine construction, while a
|
|
||||||
run/render flag enters only the mapped request. The server uses the same engine
|
|
||||||
setting and retains one backend admission state. Render prepares without
|
|
||||||
generation.
|
|
||||||
|
|
||||||
`inspect prompt` maps the selected ID to
|
`native_run.go` binds the required operand and reusable resource/input/execution
|
||||||
`promptkit.Engine.InspectPrompt`, then formats Scriptorium-owned inspection
|
flags into a native RunRequest. Ordinary typed namespace flags feed
|
||||||
data. `inspect profile` maps an explicit profile ID to `Engine.InspectProfile`
|
`Engine.WithNamespaces`; stored references are not rewritten. One `Engine.Run`
|
||||||
and formats Scriptorium-owned resolved profile data without backend completion.
|
call receives the invocation context and unchanged operation duration. PromptKit
|
||||||
|
owns preparation, credentials, admission, retries, repairs, and content validation.
|
||||||
|
|
||||||
Both paths construct the same full-source engine as execution before inspecting
|
`outcomes.go` maps typed failures and native configuration evidence to native
|
||||||
the chosen definition; selected prompts are not filtered out. They do not
|
outcomes, preserving causes while keeping provider payloads out of terminal
|
||||||
perform request preparation or generation. The [CLI definition-inspection
|
explanations. A returned completed validation failure remains a domain outcome;
|
||||||
contract](../cli.md#scriptorium-inspect-prompt) owns the loading and runtime
|
an unexpected nil successful result is an integration failure.
|
||||||
boundaries, including the distinction between an optional prompt collection
|
|
||||||
and validating any prompts that were selected.
|
|
||||||
|
|
||||||
`inspect configuration` builds the same engine from the selected native
|
`delivery.go` submits the actual RunResult to the invocation reporter with a
|
||||||
Promptkit source sequence, reads its captured configuration view, and delegates
|
reusable artifact-byte text template. The optional file renders that same value
|
||||||
effective output and explanation to Promptkit's native writers. The adapter
|
without the template through CommandKit's writer API, outside reporter callbacks
|
||||||
projects captured source records into its small JSON DTO. Its validation
|
to avoid reentrancy. File failures add explanations/causes while reporter
|
||||||
operation constructs the engine without preparing a request. Framework source
|
presentation failures remain native. See [delivery semantics](../cli.md#results-and-separate-reports).
|
||||||
semantics and report redaction are owned by Promptkit; the [CLI
|
|
||||||
contract](../cli.md#framework-configuration-inspection) documents the exposed
|
|
||||||
operations and their deliberately limited redaction scope.
|
|
||||||
|
|
||||||
The `config` family loads the same application ConfigKit result once and uses
|
## Server And HTTP
|
||||||
its validation, effective-output, source, and explanation views directly. It
|
|
||||||
does not construct a Promptkit engine. The CLI maps source records into a small
|
|
||||||
JSON DTO for `sources`; `show` and `explain` use ConfigKit's own writers.
|
|
||||||
|
|
||||||
`serve` constructs Scriptorium's restricted HTTP artifact reader, injects it
|
`server.go` loads application settings, injects the restricted artifact reader
|
||||||
with `promptkit.WithArtifactReader`, passes the engine through the HTTP
|
into one engine, installs the existing HTTP handler/limits, and opens a listener.
|
||||||
adapter's consumer-owned `Runner` interface, and starts the server.
|
It retains the header timeout and engine retry defaults. Context cancellation
|
||||||
|
calls Server.Close; callback coordination finishes connection/listener closing
|
||||||
|
before return, without waiting for arbitrary context-ignoring application work.
|
||||||
|
|
||||||
The command paths assemble the engine from application-selected framework
|
The HTTP handler strictly decodes DTOs and applies transport policy before
|
||||||
source files and directories plus the schema directory and retry setting.
|
mapping to PromptKit requests. Its Runner interface is a consumer-side external
|
||||||
Individual files become import-enabled ConfigKit inputs; directory selections
|
boundary using native values. Request controls do not mutate shared engine
|
||||||
are expanded through Promptkit's directory-input helper. Scriptorium supplies
|
defaults; shared backend capacity accounting remains in that engine.
|
||||||
those ordered underlays through Promptkit's public configuration API rather
|
|
||||||
than parsing or mapping framework definitions itself. The server constructs
|
|
||||||
one engine, which retains the framework's backend registry and associated
|
|
||||||
capacity state across requests.
|
|
||||||
|
|
||||||
### HTTP
|
HTTP optional reports use one per-request `internal/report.Collector`, attached
|
||||||
|
only for opted-in requests. It copies terminal aggregates, not event history,
|
||||||
The handler enforces transport limits and strict JSON decoding before mapping
|
and is never the CLI result report. Success/error attachment and encoded-size
|
||||||
DTOs into `promptkit.RunRequest`, `promptkit.ArtifactRef`, and
|
handling retain the [HTTP contract](../api.md#execution-report). Reader sentinel
|
||||||
`promptkit.ExecutionTargetOverride`. On success it reads Promptkit artifact,
|
identities survive PromptKit wrapping for status mapping.
|
||||||
validation, model, usage, and metadata values directly.
|
|
||||||
The optional HTTP retry count maps to `RunRequest.TransientRetries` without
|
|
||||||
mutating the shared engine. Negative values are rejected before invocation;
|
|
||||||
Promptkit controls retries and backend admission for concurrent requests.
|
|
||||||
CLI and HTTP operation budgets map to PromptKit's `OperationTimeout`; HTTP
|
|
||||||
passes the caller context through unchanged. The public sentinels determine
|
|
||||||
timeout classification.
|
|
||||||
|
|
||||||
Failure mapping uses `errors.Is` against Promptkit's public sentinels and the
|
|
||||||
HTTP reader's Scriptorium-owned containment and size errors. Wrapped reader
|
|
||||||
errors preserve their identity through Promptkit's artifact-load boundary.
|
|
||||||
|
|
||||||
The report collector is per invocation. It ignores nonterminal events and
|
|
||||||
copies terminal aggregates once, including usage presence and completeness.
|
|
||||||
It retains no history or shared state; adapters decide whether to attach it and
|
|
||||||
how to present its snapshot.
|
|
||||||
CLI `run` attaches one only when a report path is selected, writes generated
|
|
||||||
output first on success, and then writes the terminal snapshot when present.
|
|
||||||
Its report path remains untouched for pre-invocation failures or absent
|
|
||||||
terminal observations; report and artifact writes are independent.
|
|
||||||
HTTP `POST /v1/runs` attaches a separate collector to each invocation only when
|
|
||||||
`include_execution_report` is true. Its optional top-level `execution` field
|
|
||||||
appears in success or error envelopes only when Promptkit emits a terminal
|
|
||||||
event. Pre-invocation adapter errors and absent terminal events leave it out.
|
|
||||||
The complete serialized success or report-bearing error response is subject to
|
|
||||||
the configured response-size limit; overflow returns the compact 413 error
|
|
||||||
without a report. Field meanings are owned by the [API contract](../api.md#execution-report).
|
|
||||||
|
|
||||||
## Package-Local Guarantees
|
|
||||||
|
|
||||||
- Adapters contain no copied framework types or orchestration.
|
|
||||||
- Configuration is resolved before Promptkit engine construction.
|
|
||||||
- Explicit numeric overrides preserve presence, including zero.
|
|
||||||
- HTTP DTO and error mapping remains stable and transport-owned.
|
|
||||||
- Resolved secrets are not serialized or printed.
|
|
||||||
- No adapter creates durable run state.
|
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
Inspect:
|
Native composition/configuration/prompt/run/delivery/lifecycle tests protect the
|
||||||
|
application's integration choices using real libraries and controlled local
|
||||||
|
providers. Checkout-discovery tests cover fresh invocations and entry-file
|
||||||
|
spellings. Entrypoint tests protect identity/status mapping; HTTP reader/DTO/
|
||||||
|
report/capacity regressions and the architecture guard remain their own owners.
|
||||||
|
Do not duplicate upstream parsers, analyzers, serializers, or exhaustive format
|
||||||
|
matrices to change an adapter.
|
||||||
|
|
||||||
- `internal/adapter/cli/run_test.go`
|
When changing behavior, update its canonical external contract and focused
|
||||||
- `internal/adapter/http/handler_test.go`
|
integration protection. Required shared capabilities belong upstream before
|
||||||
- `internal/adapter/http/artifact_reader_test.go`
|
tagged adoption, not in a resurrected application framework facade.
|
||||||
- `internal/format/prepared_run_test.go`
|
|
||||||
- `internal/adapter/dependency_test.go`
|
|
||||||
|
|
||||||
The adapter tests protect parsing, configuration mapping, output, status
|
|
||||||
mapping, restricted artifacts, and representative real Promptkit-engine
|
|
||||||
workflows. The dependency test protects the repository boundary.
|
|
||||||
|
|
||||||
## Change Recipes
|
|
||||||
|
|
||||||
For a CLI or HTTP change:
|
|
||||||
|
|
||||||
1. identify the Scriptorium-owned external contract;
|
|
||||||
2. map through Promptkit public values without copying framework semantics;
|
|
||||||
3. add or update the narrow application-owned test;
|
|
||||||
4. update the canonical Scriptorium contract; and
|
|
||||||
5. coordinate and tag Promptkit first if a required public capability is
|
|
||||||
genuinely absent.
|
|
||||||
|
|
||||||
Update [source internals](sources.md) when application source locations or HTTP
|
|
||||||
artifact containment changes.
|
|
||||||
|
|||||||
@@ -1,19 +1,18 @@
|
|||||||
# Internal Component Overview
|
# Internal Component Overview
|
||||||
|
|
||||||
This is the complete inventory of Scriptorium's implemented Go components.
|
This inventory describes implemented Go components. The
|
||||||
The [architecture policy](../policy/architecture.md) owns normative boundaries;
|
[architecture policy](../policy/architecture.md) owns normative boundaries;
|
||||||
public behavior belongs in the linked contracts.
|
public behavior belongs in the linked contracts.
|
||||||
|
|
||||||
| Component | Implemented responsibility | References |
|
| Component | Implemented responsibility | References |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `cmd/scriptorium` | Process entrypoint that delegates arguments and streams to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
| `cmd/scriptorium` | Build identity, signal-aware context, native status-to-process exit. | [CLI](../cli.md), [adapters](adapters.md) |
|
||||||
| `internal/adapter/cli` | Parses execution, application-configuration, and captured framework-configuration inspection commands; resolves settings; constructs Promptkit engines for execution, definition inspection, and framework inspection; maps requests; and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
| `internal/adapter/cli` | CommandKit composition, ckconfig/ckprompt acquisition, application loading, native request mapping, run/report effects, and HTTP process lifecycle. | [CLI](../cli.md), [adapters](adapters.md), [sources](sources.md) |
|
||||||
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
| `internal/adapter/http` | Routes, strict DTO decoding, transport limits, public request/result/error mapping, and restricted artifact reader. | [HTTP API](../api.md), [adapters](adapters.md), [sources](sources.md) |
|
||||||
| `internal/config` | Selects application sources through ConfigKit v1.2.1, supplies the application definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
| `internal/config` | ConfigKit application schema, discovery/selection, defaults/resolution/validation callbacks, and completed-value conversion. | [Configuration](../config.md), [sources](sources.md) |
|
||||||
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) |
|
| `internal/defaults` | Application and HTTP default constants. | [Configuration](../config.md) |
|
||||||
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output; framework-configuration output uses Promptkit's native captured view and writers. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
| `internal/report` | HTTP terminal-observation metadata projection. It is not used for CLI native result reports. | [HTTP execution report](../api.md#execution-report), [adapters](adapters.md) |
|
||||||
| `internal/report` | Projects one terminal Promptkit observation into Scriptorium-owned execution metadata. | [adapter internals](adapters.md) |
|
|
||||||
|
|
||||||
Framework implementation packages are provided by
|
CommandKit, ckconfig, ckprompt, ConfigKit, and PromptKit are released upstream
|
||||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
|
dependencies, not local framework packages. There is no custom CLI formatter
|
||||||
and are not part of this repository.
|
package or compatibility dispatch surface.
|
||||||
|
|||||||
@@ -1,117 +1,55 @@
|
|||||||
# Source Internals
|
# Source Internals
|
||||||
|
|
||||||
## Purpose
|
This document covers application acquisition and the restricted HTTP artifact
|
||||||
|
reader. [Configuration](../config.md) owns field/discovery/composition contracts;
|
||||||
|
the [PromptKit formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||||
|
own framework resources and execution semantics.
|
||||||
|
|
||||||
This document covers Scriptorium-owned source locations and the restricted HTTP
|
## Application Boundary And Engine Construction
|
||||||
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
|
|
||||||
owned by the tagged
|
|
||||||
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
|
||||||
|
|
||||||
## Application Source Locations
|
`internal/config.Load` selects an explicit File or the ordered native SearchConfig
|
||||||
|
presets and performs one ConfigKit load with imports, overlay selection, and
|
||||||
|
explicit overrides. It returns the actual Result with native inspection evidence.
|
||||||
|
Definition callbacks supply application defaults, record actual path/address
|
||||||
|
normalizations through ConfigKit, and validate retry/HTTP-size domains. Settings
|
||||||
|
conversion consumes completed values without another merge or normalization pass.
|
||||||
|
|
||||||
`internal/config` resolves the ordered `promptkit_sources` sequence and
|
The typed application PromptKit section selects acquisition only. In
|
||||||
`schema_dir` from application configuration and explicit CLI overrides. It
|
`engine.go`, a nonempty resolved configuration path becomes an import-enabled
|
||||||
owns selector-shape validation, not framework file parsing. During engine
|
main Input; an empty path leaves Main nil. The schema base and retry default are
|
||||||
construction, `internal/adapter/cli` turns file selections into import-enabled
|
passed directly, retaining the bundled underlay. There is no source-list parser,
|
||||||
ConfigKit inputs and expands directory selections with
|
directory classification/expansion, or one-underlay-per-selector algorithm.
|
||||||
`promptkit.DirectoryInputs`, then passes the ordered groups as Promptkit
|
|
||||||
configuration underlays, one per selector. Directory inputs retain the
|
|
||||||
helper's strict-composition grouping; no selected source becomes an implicit
|
|
||||||
framework main. The shared Promptkit engine receives the selected schema
|
|
||||||
directory and retry setting, and the default OpenRouter bundle remains enabled.
|
|
||||||
|
|
||||||
Promptkit uses ConfigKit to load and compose those framework inputs according
|
Framework imports resolve within the native source context. Message resources
|
||||||
to Promptkit's own schema and format. Scriptorium does not parse or validate
|
are captured from their defining contexts at engine construction. Schema and
|
||||||
Promptkit definitions. Promptkit owns definition selection and validation,
|
request-artifact acquisition retain their preparation-time behavior.
|
||||||
profile built-ins and overlays, schema resolution, ordinary file artifacts,
|
Independent collection checkouts are discovered by native file import globs,
|
||||||
captured framework configuration inspection, and related error identities.
|
not application traversal. New invocations reload normally; a server retains
|
||||||
|
its single engine until restart.
|
||||||
|
|
||||||
The [configuration reference](../config.md) owns Scriptorium's source-location
|
Application-only inspection uses its completed result without framework
|
||||||
fields and precedence. Maintained files under `examples/` are application
|
acquisition. Framework inspection exposes the native captured view separately.
|
||||||
inputs that use Promptkit's tagged formats.
|
Diagnostic decoration in the [CLI adapter](adapters.md) carries application
|
||||||
|
messages without combining evidence trees or effective documents. There is no
|
||||||
## Application Configuration Boundary
|
per-invocation engine/config cache or cross-invocation snapshot guarantee.
|
||||||
|
|
||||||
`internal/config` contains the application-owned ConfigKit-native `Document`
|
|
||||||
definition. Its fresh defaults come from Scriptorium's existing default
|
|
||||||
constants; its resolver records only actual leaf-level trimming, path cleaning, and
|
|
||||||
render-format normalization. The validator reports selector-shape, retry,
|
|
||||||
size-limit, and format errors without mutating the document or exposing
|
|
||||||
rejected values. Application path values keep working-directory-relative
|
|
||||||
meaning. ConfigKit v1.2.1 owns strict application YAML decoding and presence
|
|
||||||
semantics. Framework configuration is separately loaded by PromptKit using
|
|
||||||
the same ConfigKit release under PromptKit's own schema and source contract.
|
|
||||||
|
|
||||||
The `Load` boundary selects an explicit file or constructs a native ConfigKit
|
|
||||||
search over the ordered implicit paths. Discovery exhaustion uses defaults;
|
|
||||||
a selected-file error stops loading. One ConfigKit load enables imports,
|
|
||||||
applies explicitly supplied application overrides after decoding and before
|
|
||||||
resolution and validation, and returns the document and its frozen inspection
|
|
||||||
records. `Settings` copies selectors in their resolved order into runtime
|
|
||||||
settings; it does not read files or apply another merge. ConfigKit uses the selected file's
|
|
||||||
canonical directory as the source root; the application does not widen it.
|
|
||||||
Application path values retain process-working-directory-relative meaning.
|
|
||||||
|
|
||||||
Runtime conversion consumes the resolved application values through the
|
|
||||||
existing `Settings` path. Promptkit receives selected framework sources only
|
|
||||||
after application configuration has loaded and validated; its source merge,
|
|
||||||
definition validation, and captured inspection view remain framework-owned.
|
|
||||||
|
|
||||||
Every configuration-loading command uses this boundary. The CLI maps only
|
|
||||||
explicitly visited application flags to ConfigKit overrides and prints
|
|
||||||
structured load diagnostics before engine construction. Application and
|
|
||||||
framework load errors and warnings use their respective native writers. The old application
|
|
||||||
YAML decoder and merge path have been removed.
|
|
||||||
Application `config` inspection reuses the result's frozen effective,
|
|
||||||
provenance, and history views without constructing an engine or reloading
|
|
||||||
sources. It does not resolve credentials. `inspect configuration` separately
|
|
||||||
constructs an engine and exposes Promptkit's captured framework view; only its
|
|
||||||
framework `show` and `explain` reports receive Promptkit's native,
|
|
||||||
credential-shaped redaction. Neither inspection family is a general-purpose
|
|
||||||
filter for arbitrary sensitive values.
|
|
||||||
|
|
||||||
## Restricted HTTP Artifact Reader
|
## Restricted HTTP Artifact Reader
|
||||||
|
|
||||||
`internal/adapter/http` implements `promptkit.ArtifactReader` for HTTP
|
The HTTP adapter implements `promptkit.ArtifactReader`; server construction
|
||||||
requests. The `serve` path injects it with
|
injects it through the public option. Ordinary CLI artifact loading remains
|
||||||
`promptkit.WithArtifactReader`, replacing Promptkit's ordinary reader for
|
upstream-owned.
|
||||||
inbound HTTP inputs.
|
|
||||||
|
|
||||||
The reader:
|
The reader accepts inline content without a root. File references need a root;
|
||||||
|
relative paths resolve below it and absolute paths must be lexically contained.
|
||||||
|
Traversal outside is rejected and the configured file byte limit applies, with
|
||||||
|
zero unlimited. Reads preserve native artifact metadata and honor cancellation.
|
||||||
|
|
||||||
- accepts inline references without an artifact root;
|
Containment does not resolve symlinks. The OS may follow a symlink inside the
|
||||||
- denies file references when no root is configured;
|
root to an outside target; this is not a filesystem sandbox or a defense against
|
||||||
- resolves relative paths below the configured root;
|
privileged concurrent filesystem modification. Reader errors retain identity
|
||||||
- accepts absolute paths only when they are lexically within that root;
|
through upstream artifact-load wrapping, enabling the existing HTTP mapping.
|
||||||
- rejects lexical traversal outside the root;
|
|
||||||
- applies the configured file byte limit, with zero meaning unlimited;
|
|
||||||
- preserves content type, body, size, hash, name, and URI metadata; and
|
|
||||||
- honors context cancellation.
|
|
||||||
|
|
||||||
Containment is lexical and does not resolve symlinks. The operating system
|
The [HTTP API](../api.md) owns outcomes and [operations](../operations.md) owns
|
||||||
follows symlinks after the check. The [HTTP API](../api.md) owns observable
|
safe root/permission selection. Configuration, CLI acquisition/engine, and HTTP
|
||||||
request outcomes, and [operations](../operations.md) owns safe deployment
|
reader/handler tests own their respective boundaries; upstream owns exhaustive
|
||||||
permissions and root selection.
|
source traversal, format parsing, ordinary artifacts, and inspection semantics.
|
||||||
|
|
||||||
Reader errors remain identifiable after Promptkit wraps them as artifact-load
|
|
||||||
failures, allowing the HTTP adapter to preserve Scriptorium status and error
|
|
||||||
codes.
|
|
||||||
|
|
||||||
## Verification And Change Recipe
|
|
||||||
|
|
||||||
Inspect:
|
|
||||||
|
|
||||||
- `internal/config/document_test.go`
|
|
||||||
- `internal/config/load_test.go`
|
|
||||||
- `internal/adapter/cli/run_test.go`
|
|
||||||
- `internal/adapter/http/artifact_reader_test.go`
|
|
||||||
- `internal/adapter/http/handler_test.go`
|
|
||||||
|
|
||||||
When changing an application source location or HTTP artifact policy:
|
|
||||||
|
|
||||||
1. preserve strict configuration precedence and the Promptkit public boundary;
|
|
||||||
2. keep containment and size policy in Scriptorium;
|
|
||||||
3. update focused configuration, reader, and handler tests;
|
|
||||||
4. update the [configuration](../config.md), [HTTP](../api.md), and
|
|
||||||
[operations](../operations.md) contracts as applicable; and
|
|
||||||
5. do not duplicate Promptkit loaders, formats, or ordinary artifact behavior.
|
|
||||||
|
|||||||
@@ -1,243 +1,120 @@
|
|||||||
# Operations Guide
|
# Operations Guide
|
||||||
|
|
||||||
## Scope And References
|
This runbook covers deployment, normal operation, capacity, sensitive data, and
|
||||||
|
recovery. Exact [CLI syntax/statuses](cli.md), [configuration](config.md), and
|
||||||
|
[HTTP wire behavior](api.md) remain in their canonical references.
|
||||||
|
See the [CLI migration guide](consumers/upgrading-cli.md) for changed jobs and
|
||||||
|
the [framework upgrade guide](consumers/upgrading-promptkit.md) for definitions.
|
||||||
|
|
||||||
This runbook covers deployment, normal operation, capacity planning, and safe
|
## Deploy Sources And Process
|
||||||
recovery for Scriptorium. It does not redefine invocation syntax, configuration
|
|
||||||
fields, or HTTP wire behavior.
|
|
||||||
|
|
||||||
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
Provide readable application configuration, its selected framework entry file,
|
||||||
- [Configuration reference](config.md): application settings, source
|
and explicitly imported definitions/resources. Set cwd deliberately: application
|
||||||
locations, defaults, and credential mapping.
|
path values are cwd-relative, while framework imports and message resources keep
|
||||||
- [Promptkit source upgrade guide](consumers/upgrading-promptkit.md): concise
|
their native source contexts. ConfigKit confines source imports/overlays to the
|
||||||
source-selector and definition migration examples.
|
selected main's canonical root; Scriptorium does not widen it.
|
||||||
- [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md):
|
|
||||||
prompt, profile, schema, execution-setting, and framework credential
|
|
||||||
contracts.
|
|
||||||
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
|
||||||
limits, and HTTP artifact access.
|
|
||||||
- [Consumer integration overview](consumers/api.md): caller responsibilities.
|
|
||||||
|
|
||||||
## Operational Model And State
|
Engine construction validates selected effective definitions and captures message
|
||||||
|
resources/template syntax, including definitions not used by a particular request.
|
||||||
|
Schema/artifact reads, credential resolution, and model work retain their native
|
||||||
|
preparation/runtime lifecycles. A server constructs one engine at startup and
|
||||||
|
does not hot-reload changed files; restart to capture updates. Separate CLI
|
||||||
|
invocations load afresh rather than sharing an immutable snapshot.
|
||||||
|
|
||||||
Scriptorium handles one prompt request for each CLI invocation or HTTP request.
|
For independent collections, install checkouts beneath the layout selected by
|
||||||
It has no durable run store, archive, checkpoint, cache, or resume mechanism.
|
the [discovery entry](../examples/discovery/framework.yml). Each checkout declares
|
||||||
A failed or interrupted request is recovered by correcting its inputs,
|
its namespace and explicit fragment imports; arbitrary YAML is not a framework
|
||||||
configuration, or environment and submitting a new request.
|
source. The maintained [discovery example](../examples/config.discovery.yml)
|
||||||
|
supports both entry-file spellings and multiple definitions. New checkouts or
|
||||||
|
updates become visible to later invocations or a restarted server.
|
||||||
|
|
||||||
Generated artifacts, rendered prompts, model output, and run metadata are
|
Supply only needed credential environment variables through a supervisor or
|
||||||
caller-owned data. Retention, encryption, backup, and deletion are deployment
|
secret-management mechanism. Confirm presence without printing their values.
|
||||||
responsibilities.
|
Raw keys do not belong in arguments or checked-in configuration.
|
||||||
Optional [CLI execution reports](cli.md) also belong to the caller. They can
|
|
||||||
contain sensitive identifiers even though they omit prompt and response bodies.
|
|
||||||
Keep report paths separate from generated output and apply the same access and
|
|
||||||
retention controls. Output and report writes are not transactional; an older
|
|
||||||
report may remain when no terminal observation was produced.
|
|
||||||
HTTP callers may also request a [terminal execution report](api.md#execution-report).
|
|
||||||
It shares the same identifier sensitivity and should receive response access
|
|
||||||
and retention controls. An opted-in report increases the encoded response size;
|
|
||||||
an oversized success or report-bearing error becomes the compact 413 response
|
|
||||||
without a report.
|
|
||||||
|
|
||||||
## Deploy The Filesystem And Process
|
## Normal Operator Workflow
|
||||||
|
|
||||||
Provide the process with readable selected Promptkit source files/directories
|
1. Validate application configuration through the native application scope.
|
||||||
and schema sources that follow the tagged framework formats. At engine
|
This does not acquire framework sources or prove provider readiness.
|
||||||
construction, PromptKit v0.23.3 validates every selected effective definition,
|
2. Inspect framework configuration/resources and discover requirements for the
|
||||||
including unused definitions, and captures referenced message-file contents
|
selected prompt/profile/namespaces. Discovery can precede generated input
|
||||||
and static template syntax. `serve` constructs one engine at startup; it does
|
artifacts; completeness does not establish satisfied bindings.
|
||||||
not reload changed configuration or message content. Restart the process to
|
3. Once artifacts exist, validate the concrete request or inspect prepared
|
||||||
validate and capture a new configuration. Schema and request-artifact reads,
|
content. These are provider-free preparation checks with native prerequisites.
|
||||||
credential resolution, and provider work remain request-preparation or runtime
|
4. Invoke a smoke test only when its provider effects are intended. Findings
|
||||||
concerns. For an HTTP deployment that accepts file artifacts, use a dedicated,
|
can be unsupported or unsuccessful while the command itself succeeds.
|
||||||
narrow artifact directory rather than a general-purpose or sensitive filesystem
|
Smoke tests are not an automatic startup or release prerequisite.
|
||||||
tree.
|
5. Generate with the same selections and bindings, then inspect the finalized
|
||||||
|
status and any returned validation result before accepting the output.
|
||||||
|
|
||||||
If application configuration uses explicit imports or a selected overlay,
|
Use the [maintained preview script](../examples/render-markdown-summary.sh)
|
||||||
deploy those files beneath the selected configuration file's canonical
|
for a provider-free example, and the [CLI contract](cli.md) for commands.
|
||||||
directory. ConfigKit keeps imported and overlay files within that root;
|
A complete discovery report can still list required inputs that are not bound;
|
||||||
Scriptorium does not widen it. Application path values inside those files still
|
a partial report can still be useful. Consumers own their policy for incomplete
|
||||||
resolve from the process working directory, so set that directory deliberately
|
findings and changed resources between invocations.
|
||||||
in the service manager. This configuration-source boundary is distinct from
|
|
||||||
the HTTP artifact root and does not constrain prompt or schema directories.
|
|
||||||
|
|
||||||
For independently maintained collections, place checkouts beneath the
|
## Sensitive Data And Retention
|
||||||
directory selected by your framework discovery file. The maintained
|
|
||||||
[discovery example](../examples/config.discovery.yml) selects optional
|
|
||||||
`collection.yml` and `collection.yaml` entry files; each checkout owns its
|
|
||||||
namespace and explicit fragment imports. Update a checkout independently and
|
|
||||||
invoke the CLI again, or restart a running server to capture the update. Do
|
|
||||||
not treat arbitrary YAML in a checkout as a framework source. When upgrading
|
|
||||||
definitions, review exact profile IDs and replace retired catalog references
|
|
||||||
as described in the [migration guide](consumers/upgrading-promptkit.md).
|
|
||||||
|
|
||||||
Run Scriptorium under an identity that can:
|
Artifacts, prompt messages, prepared content, native run reports, metadata,
|
||||||
|
diagnostics, and redirected streams are caller-owned potentially sensitive data.
|
||||||
|
The separate CLI report contains the full returned result, not metadata alone.
|
||||||
|
Control access, retention, encryption, backup, and deletion accordingly.
|
||||||
|
Framework effective/explanation views apply native classified-field redaction,
|
||||||
|
including history, but do not detect arbitrary secrets. Other content-bearing
|
||||||
|
reports are not automatically made safe for publication.
|
||||||
|
|
||||||
- read only the prompt, profile, schema, and allowed input-artifact paths it
|
Keep main output and report destinations separate. Delivery is nontransactional:
|
||||||
needs;
|
one destination or partial bytes can remain after failure. An existing report
|
||||||
- read the required credential environment variables without writing them to
|
can remain on a resultless error; check status and report identity, not just file
|
||||||
files or logs; and
|
existence. A shell can truncate a redirected destination before the child starts.
|
||||||
- write only caller-selected output locations when CLI output files are used.
|
Omit quiet if main stdout must be captured; the separate report effect remains
|
||||||
|
independent. Exact delivery behavior belongs to [CLI results](cli.md#results-and-separate-reports).
|
||||||
|
|
||||||
Do not make the HTTP artifact directory writable by untrusted users. The HTTP
|
HTTP's optional [execution report](api.md#execution-report) remains a distinct
|
||||||
artifact containment behavior is lexical and the operating system follows
|
terminal-observation projection. Protect its identifiers and any response data.
|
||||||
symlinks; account for that when choosing ownership and mount boundaries. See
|
It increases encoded response size and remains subject to the configured limit.
|
||||||
the [HTTP API reference](api.md) for the externally observable behavior.
|
|
||||||
|
|
||||||
## Supply Credentials And Protect Runtime Data
|
## HTTP Deployment And Shutdown
|
||||||
|
|
||||||
Set secret values in the process environment and configure only their
|
The service has no built-in authentication/authorization. Deploy on a trusted
|
||||||
environment-variable names. Do not put raw keys in configuration, prompt or
|
network or behind appropriate authenticated proxy/gateway controls. Use a narrow,
|
||||||
profile files, process arguments, HTTP payloads, captured command lines, or
|
dedicated artifact root with controlled ownership and permissions when accepting
|
||||||
debug dumps.
|
file inputs; this is lexical containment, not a symlink sandbox.
|
||||||
|
|
||||||
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP
|
Use a supervisor for cwd, environment, restart policy, and stream capture.
|
||||||
responses as potentially sensitive. Send service logs to a controlled collector
|
Startup information uses native Info reporting, not stdout banners.
|
||||||
and apply the same retention and access rules as for model input and output.
|
Interrupt/SIGTERM cancels the invocation and closes the listener and active
|
||||||
|
connections. Canceled shutdown is nonzero operational termination. There is no
|
||||||
|
graceful-drain deadline or promise that context-ignoring work has finished;
|
||||||
|
supervisors own forced termination when necessary.
|
||||||
|
|
||||||
## Run A Normal Workflow
|
## Capacity, Limits, And Recovery
|
||||||
|
|
||||||
Before changing production inputs, profiles, or schemas:
|
One server engine shares backend admission/queue state across concurrent requests.
|
||||||
|
That state is transient operational coordination, not a durable workflow or
|
||||||
|
queue of resumable runs. HTTP capacity exhaustion retains its
|
||||||
|
[API outcome](api.md); callers decide whether another model call is safe.
|
||||||
|
|
||||||
1. use `config validate`, `config show`, or `config sources` to check
|
Plan for input/output sizes, schema complexity, provider latency, retries, and
|
||||||
Scriptorium's application configuration; these do not open Promptkit sources
|
repair costs. Native transient attempts and repair generations can add calls and
|
||||||
or construct an engine;
|
waiting; engine defaults and per-request overrides belong to
|
||||||
2. use `inspect configuration validate` to construct the engine and validate
|
[configuration](config.md) and the [CLI](cli.md)/[HTTP](api.md) contracts.
|
||||||
all selected framework definitions; this does not prepare a request, check
|
Whole-operation budgets are distinct from per-attempt generation timeouts.
|
||||||
credentials, load request-time schemas/artifacts, or contact a provider;
|
They remain cooperative and do not include configuration/setup or delivery.
|
||||||
3. confirm the deployed configuration selects the intended sources and
|
|
||||||
environment-variable names;
|
|
||||||
4. use [`render`](cli.md) with the same request inputs and variables to confirm
|
|
||||||
preparation without a model call;
|
|
||||||
5. use [`run`](cli.md) for generation; and
|
|
||||||
6. retain or discard validation-failed output according to the caller's
|
|
||||||
policy.
|
|
||||||
|
|
||||||
Application configuration inspection can include paths and configured values.
|
Before increasing HTTP byte limits, measure representative requests/artifacts/
|
||||||
Framework `inspect configuration show` and `explain` use Promptkit's native
|
responses, confirm memory/network/provider capacity, retain authentication and
|
||||||
redaction for classified credential-shaped provider fields, including
|
upstream request controls, and test a representative workload. Do not widen the
|
||||||
historical values, while leaving environment-reference names and ordinary
|
artifact root just to make an arbitrary host path accessible.
|
||||||
provider parameters visible. Neither report identifies arbitrary sensitive
|
|
||||||
content by value or name. Protect inspection output accordingly; Scriptorium
|
|
||||||
stores environment-variable names, not the resolved credential bytes.
|
|
||||||
|
|
||||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
For failures, capture diagnostics without publishing sensitive data. Verify
|
||||||
copyable preflight example. The CLI reference owns its complete invocation and
|
selected configuration, resource identity, readable inputs, profile/credentials,
|
||||||
exit semantics.
|
and schema deployment. Use native prepared preview to isolate pre-generation
|
||||||
|
problems. A completed content-validation failure is distinct from an operational
|
||||||
|
failure to validate; consumers choose whether failed content is usable.
|
||||||
|
Correct the cause, then issue a new request only when another provider call and
|
||||||
|
its cost are acceptable.
|
||||||
|
|
||||||
## Expose The HTTP Service
|
There is no durable run store, archive, checkpoint, or resume mechanism.
|
||||||
|
Cleanup concerns caller-owned files/logs only. Rotate/delete them under the
|
||||||
The HTTP service has no built-in authentication or authorization. Place it on a
|
deployment's normal retention policy; recovery starts from a new request.
|
||||||
trusted network or behind an authenticated reverse proxy, API gateway, or
|
|
||||||
equivalent access control. Restrict who can reach it and who can read the
|
|
||||||
artifact root.
|
|
||||||
|
|
||||||
Use a service manager or supervisor appropriate to the deployment to manage
|
|
||||||
process lifetime, restart policy, log capture, and environment injection. The
|
|
||||||
[HTTP API reference](api.md) owns client request shapes, status behavior, and
|
|
||||||
artifact-access outcomes.
|
|
||||||
|
|
||||||
## Plan Capacity And Limits
|
|
||||||
|
|
||||||
Capacity is primarily determined by concurrent model calls, input and output
|
|
||||||
sizes, schema complexity, provider latency, and network behavior. Size limits
|
|
||||||
protect request bodies, HTTP file artifacts, and encoded responses; configure
|
|
||||||
them through the [configuration reference](config.md) and rely on the
|
|
||||||
[HTTP API reference](api.md) for their response effects.
|
|
||||||
|
|
||||||
Configured backend concurrency and queue capacity are enforced per constructed
|
|
||||||
Promptkit engine. A `serve` process constructs one engine for its handler, so
|
|
||||||
concurrent HTTP requests share that transient admission state. Scriptorium does
|
|
||||||
not retain workflow state: capacity is neither durable nor a queue of resumable
|
|
||||||
runs. When admission is exhausted, HTTP returns `503 capacity_exceeded` without
|
|
||||||
retry timing; callers choose any retry policy that is safe for another model
|
|
||||||
call.
|
|
||||||
|
|
||||||
Promptkit's built-in client now allows two additional attempts for eligible
|
|
||||||
transient provider statuses by default. Account for added calls, latency, and
|
|
||||||
cost when planning capacity. Each repair generation has its own transient
|
|
||||||
budget, separate from the structural repair count. Configure the engine budget
|
|
||||||
with [`transient_retries`](config.md); `run` and `render` accept a request
|
|
||||||
override described in the [CLI contract](cli.md), and HTTP callers can select
|
|
||||||
one through the [API contract](api.md). Omission inherits the engine
|
|
||||||
setting and then Promptkit's default, while explicit zero disables additional
|
|
||||||
attempts. Existing generation timeouts apply
|
|
||||||
per attempt, while the caller context covers the complete operation. Provider
|
|
||||||
`Retry-After` timing can exceed Promptkit's ordinary backoff cap; consult the
|
|
||||||
[outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
|
||||||
for details.
|
|
||||||
|
|
||||||
CLI `run` can set a [whole-run deadline](cli.md) when total execution time must
|
|
||||||
be bounded. The existing generation timeout applies afresh to each attempt and
|
|
||||||
does not cap retry waits or the complete invocation. The whole-run interval
|
|
||||||
covers the engine call, including capacity waiting and repairs as supported by
|
|
||||||
context cancellation; it excludes parsing, engine setup, and output writes.
|
|
||||||
HTTP callers can use [`run_timeout_seconds`](api.md) to set the engine's
|
|
||||||
operation budget for the same interval. The API distinguishes a typed expired
|
|
||||||
execution deadline from caller cancellation and from a provider-attempt
|
|
||||||
timeout. A canceled capacity waiter releases its Promptkit admission state.
|
|
||||||
|
|
||||||
Before increasing a limit:
|
|
||||||
|
|
||||||
1. measure representative input, generated-output, and optional raw-output
|
|
||||||
sizes;
|
|
||||||
2. confirm memory, network, and upstream-provider capacity;
|
|
||||||
3. retain an upstream request-size and authentication boundary; and
|
|
||||||
4. test the intended workload in a non-production environment.
|
|
||||||
|
|
||||||
For large local inputs, prefer a controlled file-artifact directory over
|
|
||||||
placing arbitrary paths on the service host. Avoid disabling a limit unless an
|
|
||||||
equivalent trusted control exists elsewhere.
|
|
||||||
|
|
||||||
## Diagnose And Recover
|
|
||||||
|
|
||||||
### Preparation Or Configuration Failure
|
|
||||||
|
|
||||||
Capture the CLI diagnostic or HTTP error response, then verify the selected
|
|
||||||
configuration, prompt ID, profile selection, source readability, and input
|
|
||||||
mapping. Use `render` with the same request when it is unclear whether failure
|
|
||||||
occurs before model execution. Consult the [CLI reference](cli.md), the
|
|
||||||
[configuration reference](config.md), and the [HTTP API reference](api.md) for
|
|
||||||
the exact interface contract.
|
|
||||||
|
|
||||||
### Credential Or Provider Failure
|
|
||||||
|
|
||||||
Confirm that the process environment contains the configured credential name
|
|
||||||
without printing the secret. Check endpoint reachability and provider health
|
|
||||||
from the process network. If preparation succeeds but generation fails, inspect
|
|
||||||
the selected model settings in prepared output and the service's controlled
|
|
||||||
logs. Correct the deployment or provider issue, then submit a new request.
|
|
||||||
|
|
||||||
### Artifact Or Permission Failure
|
|
||||||
|
|
||||||
Verify that the process can read the intended local input. For HTTP file
|
|
||||||
artifacts, verify the deployment's artifact root, ownership, path layout, and
|
|
||||||
file size. Do not widen filesystem permissions or the allowed root merely to
|
|
||||||
make an arbitrary path work; move or copy the required artifact into the
|
|
||||||
controlled location instead.
|
|
||||||
|
|
||||||
### Validation Failure
|
|
||||||
|
|
||||||
A generated-content validation failure is distinct from a runtime failure.
|
|
||||||
CLI `run` reports the validation result and error count in its success summary;
|
|
||||||
it does not print the individual validation messages. For HTTP, inspect the
|
|
||||||
validation object in the response according to the [HTTP API reference](api.md).
|
|
||||||
|
|
||||||
Use rendered input and generated output to determine whether prompt instructions,
|
|
||||||
the selected model, or the schema needs correction. If schema loading or
|
|
||||||
compilation itself fails, correct the source deployment or schema document
|
|
||||||
before rerunning.
|
|
||||||
|
|
||||||
### HTTP Limit Or Request Failure
|
|
||||||
|
|
||||||
Compare the request, artifact, or expected response size with the deployed
|
|
||||||
configuration, and validate the request against the [HTTP API reference](api.md).
|
|
||||||
Reduce the payload, use an appropriate controlled artifact source, omit
|
|
||||||
unneeded raw output, or adjust the deployment limit after capacity review.
|
|
||||||
|
|
||||||
## Cleanup And Reruns
|
|
||||||
|
|
||||||
Because no run state is retained, cleanup concerns caller-owned output files,
|
|
||||||
logs, and artifacts only. Remove or rotate them using the deployment's normal
|
|
||||||
retention policy. After a correction, rerun the request from the beginning;
|
|
||||||
there is no safe resume point.
|
|
||||||
|
|||||||
@@ -39,8 +39,8 @@ Do not promote incidental implementation choices into architectural invariants.
|
|||||||
|
|
||||||
## Scriptorium's Purpose And System Shape
|
## Scriptorium's Purpose And System Shape
|
||||||
|
|
||||||
Scriptorium is an executable application with CLI `run`, `render`,
|
Scriptorium is an executable application with CLI `run`, native `promptkit`
|
||||||
`inspect`, and `config` command families and an HTTP service started by
|
and `config` families, and an HTTP service started by
|
||||||
`serve`. It offers simple, request-independent execution of predefined
|
`serve`. It offers simple, request-independent execution of predefined
|
||||||
prompts, not a reusable Scriptorium root Go API or a workflow framework.
|
prompts, not a reusable Scriptorium root Go API or a workflow framework.
|
||||||
The [internal overview](../internal/overview.md) owns the concrete package
|
The [internal overview](../internal/overview.md) owns the concrete package
|
||||||
@@ -71,9 +71,12 @@ externally observable application behavior; PromptKit owns framework formats.
|
|||||||
|
|
||||||
As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
|
As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
|
||||||
Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to
|
Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to
|
||||||
neutrally expose upstream features and semantics. The current CLI still uses
|
neutrally expose upstream features and semantics. CommandKit owns shared command,
|
||||||
custom presentation; the [adoption roadmap](../roadmap/commandkit-adoption.md)
|
parsing, presentation, and outcome mechanics; application commands and effects
|
||||||
tracks its replacement. This decision complements ADRs 0004 and 0005.
|
remain local. Standard configuration inspection uses native adapters without
|
||||||
|
custom effective documents or evidence reconciliation. This decision complements
|
||||||
|
ADRs 0004 and 0005; the [adoption roadmap](../roadmap/commandkit-adoption.md)
|
||||||
|
records the implementation work.
|
||||||
|
|
||||||
The CLI exposes Promptkit's captured framework configuration inspection. Its
|
The CLI exposes Promptkit's captured framework configuration inspection. Its
|
||||||
`show` and `explain` reports use Promptkit's native redaction for classified
|
`show` and `explain` reports use Promptkit's native redaction for classified
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
**Base Policy Version:** 1.3
|
**Base Policy Version:** 1.3
|
||||||
|
|
||||||
**Last Updated**: 2026-11-17
|
**Last Updated**: 2026-11-19
|
||||||
|
|
||||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md
|
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md
|
||||||
|
|
||||||
@@ -178,12 +178,15 @@ Give each layer its distinct responsibility and keep expectation updates localiz
|
|||||||
diagnostics. ConfigKit owns exhaustive YAML grammar, source traversal,
|
diagnostics. ConfigKit owns exhaustive YAML grammar, source traversal,
|
||||||
import and overlay composition, provenance, and inspection renderers.
|
import and overlay composition, provenance, and inspection renderers.
|
||||||
- CLI tests own argument mapping, streams, output, exit behavior, and
|
- CLI tests own argument mapping, streams, output, exit behavior, and
|
||||||
representative run, render, inspect, serve, and config workflows.
|
representative native run, PromptKit resource/requirements/preview/smoke,
|
||||||
|
serve, and configuration comparison workflows.
|
||||||
- HTTP tests own DTO decoding, transport limits, status and error mapping,
|
- HTTP tests own DTO decoding, transport limits, status and error mapping,
|
||||||
restricted artifact reading, and representative server workflows.
|
restricted artifact reading, and representative server workflows.
|
||||||
- Formatting tests own prepared-run and definition-inspection text and JSON
|
- CommandKit and its integrations own standard parsing and presentation.
|
||||||
presentation. Architecture guards own dependency direction and removed
|
Scriptorium's integration tests protect native adoption, byte-preserving main
|
||||||
framework-package boundaries.
|
output, separate report effects, outcomes, and acquisition boundaries without
|
||||||
|
reproducing the upstream formatting matrix. Architecture guards own dependency
|
||||||
|
direction and removed framework-package boundaries.
|
||||||
- Promptkit owns framework parsing, orchestration, validation, profiles, and
|
- Promptkit owns framework parsing, orchestration, validation, profiles, and
|
||||||
model-client behavior. Scriptorium's definition and execution fixtures
|
model-client behavior. Scriptorium's definition and execution fixtures
|
||||||
protect adoption through its public adapter boundary without duplicating
|
protect adoption through its public adapter boundary without duplicating
|
||||||
|
|||||||
@@ -643,6 +643,25 @@ version output. Test-local fallback deadlines prevent regressions from hanging.
|
|||||||
|
|
||||||
## Stage 8: Retire Superseded Machinery And Update Canonical Documentation
|
## Stage 8: Retire Superseded Machinery And Update Canonical Documentation
|
||||||
|
|
||||||
|
**Status: Complete.** Removed legacy parsing/dispatch, custom inspection/source
|
||||||
|
DTOs and writers, retired CLI tests, and the unused formatter package. Retained
|
||||||
|
architecture boundaries, native integration protection, checkout-discovery
|
||||||
|
coverage, and the unchanged HTTP/report implementation and regression suites.
|
||||||
|
Maintained examples now select external framework entries with native imports;
|
||||||
|
both collection-entry spellings and multi-definition discovery remain present.
|
||||||
|
Canonical CLI/configuration/process/operations/internal/development/policy owners
|
||||||
|
and navigation reflect the cutover, with a focused CLI migration guide distinct
|
||||||
|
from the earlier framework-format guide. Accepted ADRs and release-note history
|
||||||
|
are unchanged; deferred HTTP redesign and dynamic completion remain deferred.
|
||||||
|
|
||||||
|
Verified on Go 1.27.4: full tests and race suite, vet, ordinary build, module
|
||||||
|
tidiness, formatting/diff checks, all maintained application configs in both
|
||||||
|
native scopes, application overlays, maintained preview scripts, a structured
|
||||||
|
schema preview, and local Markdown links/anchors (42 files). Added maintained
|
||||||
|
example loading/preparation protection and adapted fresh-checkout discovery to
|
||||||
|
native preview. Entrypoint package tests now run and pass. Linked identity,
|
||||||
|
release-target builds, and the final assembled acceptance audit remain Stage 9.
|
||||||
|
|
||||||
**Primary files:** old CLI parser/dispatch/inspection files and tests,
|
**Primary files:** old CLI parser/dispatch/inspection files and tests,
|
||||||
`internal/format`, documentation and maintained examples listed below.
|
`internal/format`, documentation and maintained examples listed below.
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# Prompt input requirements for Narratio
|
# Prompt input requirements for Narratio
|
||||||
|
|
||||||
Status: accepted direction, pending implementation through the CommandKit
|
Status: the native interface and focused consumer scenarios are implemented.
|
||||||
migration. This proposal records Narratio's consumer needs and their mapping
|
The [implementation plan](implementation.md) tracks final assembled verification.
|
||||||
to the native interface; it does not describe shipped Scriptorium behavior.
|
This proposal records Narratio's consumer needs and their mapping to the native
|
||||||
|
interface; the [CLI contract](../cli.md) owns current behavior.
|
||||||
The [CommandKit notepad](commandkit-notepad.md) records the migration decisions.
|
The [CommandKit notepad](commandkit-notepad.md) records the migration decisions.
|
||||||
|
|
||||||
## Consumer need
|
## Consumer need
|
||||||
@@ -27,8 +28,8 @@ Default and explicit text output use the native YAML presentation.
|
|||||||
The [upstream CLI contract][requirements-cli] owns syntax and report details.
|
The [upstream CLI contract][requirements-cli] owns syntax and report details.
|
||||||
|
|
||||||
Use the same application configuration and framework source selection as
|
Use the same application configuration and framework source selection as
|
||||||
execution, including `--config`, `--overlay`, and applicable application
|
execution, including `--config` and `--overlay`. There are no application-defined
|
||||||
selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
|
source/schema selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
|
||||||
prompt/profile namespace flags supply the relevant discovery context.
|
prompt/profile namespace flags supply the relevant discovery context.
|
||||||
Preserve upstream selection semantics, including the prompt's default profile
|
Preserve upstream selection semantics, including the prompt's default profile
|
||||||
when an explicit profile is omitted. Qualified IDs identify installed resources;
|
when an explicit profile is omitted. Qualified IDs identify installed resources;
|
||||||
|
|||||||
6
examples/collection-gpu.yml
Normal file
6
examples/collection-gpu.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
config:
|
||||||
|
imports:
|
||||||
|
- path: collections.yml
|
||||||
|
required: true
|
||||||
|
- path: profiles/local-gpu.yml
|
||||||
|
required: true
|
||||||
5
examples/collections.yml
Normal file
5
examples/collections.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
config:
|
||||||
|
imports:
|
||||||
|
- dir: config-fragments/collections
|
||||||
|
pattern: '*.yml'
|
||||||
|
required: true
|
||||||
@@ -1,2 +1,4 @@
|
|||||||
promptkit_sources:
|
config:
|
||||||
- file: ./examples/framework/backends.yml
|
imports:
|
||||||
|
- path: ../../framework/backends.yml
|
||||||
|
required: true
|
||||||
|
|||||||
@@ -1,2 +1,4 @@
|
|||||||
promptkit_sources:
|
config:
|
||||||
- file: ./examples/profiles/local-fast.yaml
|
imports:
|
||||||
|
- path: ../../profiles/local-fast.yaml
|
||||||
|
required: true
|
||||||
|
|||||||
@@ -1,2 +1,4 @@
|
|||||||
promptkit_sources:
|
config:
|
||||||
- file: ./examples/prompts/generic.markdown_summary.yaml
|
imports:
|
||||||
|
- path: ../../prompts/generic.markdown_summary.yaml
|
||||||
|
required: true
|
||||||
|
|||||||
@@ -1,5 +1,3 @@
|
|||||||
promptkit_sources:
|
promptkit:
|
||||||
- directory: ./examples/prompts
|
configuration: ./examples/promptkit.yml
|
||||||
- directory: ./examples/profiles
|
schema_dir: ./examples/schemas
|
||||||
- file: ./examples/framework/backends.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
|
|||||||
@@ -1,10 +1,3 @@
|
|||||||
config:
|
promptkit:
|
||||||
sequences:
|
configuration: ./examples/collections.yml
|
||||||
- destination: [promptkit_sources]
|
schema_dir: ./examples/schemas
|
||||||
key: file
|
|
||||||
mode: upsert
|
|
||||||
imports:
|
|
||||||
- dir: config-fragments/collections
|
|
||||||
pattern: '*.yml'
|
|
||||||
required: true
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
|
|||||||
@@ -1,2 +1,2 @@
|
|||||||
promptkit_sources:
|
promptkit:
|
||||||
- file: ./examples/discovery/framework.yml
|
configuration: ./examples/discovery/framework.yml
|
||||||
|
|||||||
@@ -1,8 +1,6 @@
|
|||||||
promptkit_sources:
|
promptkit:
|
||||||
- directory: ./examples/prompts
|
configuration: ./examples/promptkit.yml
|
||||||
- directory: ./examples/profiles
|
schema_dir: ./examples/schemas
|
||||||
- file: ./examples/framework/backends.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
|
|
||||||
server:
|
server:
|
||||||
addr: 127.0.0.1:8080
|
addr: 127.0.0.1:8080
|
||||||
@@ -10,6 +8,3 @@ server:
|
|||||||
max_request_bytes: 16777216
|
max_request_bytes: 16777216
|
||||||
max_artifact_bytes: 16777216
|
max_artifact_bytes: 16777216
|
||||||
max_response_bytes: 16777216
|
max_response_bytes: 16777216
|
||||||
|
|
||||||
defaults:
|
|
||||||
render_format: text
|
|
||||||
|
|||||||
@@ -1,6 +1,4 @@
|
|||||||
# Application paths are relative to the repository invocation directory.
|
# Application paths are relative to the repository invocation directory.
|
||||||
promptkit_sources:
|
promptkit:
|
||||||
- directory: ./examples/prompts
|
configuration: ./examples/inheritance.yml
|
||||||
- directory: ./examples/profiles
|
schema_dir: ./examples/schemas
|
||||||
- file: ./examples/framework/inheritance.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
|
|||||||
@@ -1,12 +1,7 @@
|
|||||||
promptkit_sources:
|
promptkit:
|
||||||
- directory: ./examples/prompts
|
configuration: ./examples/promptkit.yml
|
||||||
- directory: ./examples/profiles
|
schema_dir: ./examples/schemas
|
||||||
- file: ./examples/framework/backends.yml
|
|
||||||
schema_dir: ./examples/schemas
|
|
||||||
|
|
||||||
server:
|
server:
|
||||||
addr: :8080
|
addr: :8080
|
||||||
artifact_root: .
|
artifact_root: .
|
||||||
|
|
||||||
defaults:
|
|
||||||
render_format: text
|
|
||||||
|
|||||||
8
examples/definitions.yml
Normal file
8
examples/definitions.yml
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
config:
|
||||||
|
imports:
|
||||||
|
- dir: prompts
|
||||||
|
pattern: '*.yaml'
|
||||||
|
required: true
|
||||||
|
- dir: profiles
|
||||||
|
pattern: '*.y*ml'
|
||||||
|
required: true
|
||||||
6
examples/inheritance.yml
Normal file
6
examples/inheritance.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
config:
|
||||||
|
imports:
|
||||||
|
- path: definitions.yml
|
||||||
|
required: true
|
||||||
|
- path: framework/inheritance.yml
|
||||||
|
required: true
|
||||||
@@ -1,2 +1,2 @@
|
|||||||
promptkit_sources:
|
promptkit:
|
||||||
- file: ./examples/profiles/local-gpu.yml
|
configuration: ./examples/collection-gpu.yml
|
||||||
|
|||||||
@@ -1,4 +1,2 @@
|
|||||||
defaults:
|
|
||||||
render_format: json
|
|
||||||
server:
|
server:
|
||||||
addr: 127.0.0.1:8080
|
addr: 127.0.0.1:8080
|
||||||
|
|||||||
6
examples/promptkit.yml
Normal file
6
examples/promptkit.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
config:
|
||||||
|
imports:
|
||||||
|
- path: definitions.yml
|
||||||
|
required: true
|
||||||
|
- path: framework/backends.yml
|
||||||
|
required: true
|
||||||
@@ -5,9 +5,8 @@ repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|||||||
|
|
||||||
cd "$repo_root"
|
cd "$repo_root"
|
||||||
|
|
||||||
go run ./cmd/scriptorium render \
|
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
|
||||||
--config ./examples/config.yml \
|
--config ./examples/config.yml \
|
||||||
--prompt generic.markdown_summary \
|
--input-file transcript=./examples/fixtures/transcript.md \
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
--input-file glossary=./examples/fixtures/glossary.yml \
|
||||||
--input glossary=./examples/fixtures/glossary.yml \
|
|
||||||
--format text
|
--format text
|
||||||
|
|||||||
@@ -4,21 +4,18 @@ set -euo pipefail
|
|||||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
cd "$repo_root"
|
cd "$repo_root"
|
||||||
|
|
||||||
go run ./cmd/scriptorium render \
|
go run ./cmd/scriptorium promptkit show prepared generic.markdown_summary \
|
||||||
--config ./examples/config.full.yml \
|
--config ./examples/config.full.yml \
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--profile local-gpu \
|
--profile local-gpu \
|
||||||
--session-id example-session \
|
--session-id example-session \
|
||||||
--reasoning-effort= \
|
--reasoning-effort= \
|
||||||
--input transcript=./examples/fixtures/transcript.md \
|
--input-file transcript=./examples/fixtures/transcript.md \
|
||||||
--input glossary=./examples/fixtures/glossary.yml
|
--input-file glossary=./examples/fixtures/glossary.yml
|
||||||
|
|
||||||
go run ./cmd/scriptorium inspect prompt \
|
go run ./cmd/scriptorium promptkit show prompt generic.markdown_summary \
|
||||||
--config ./examples/config.full.yml \
|
--config ./examples/config.full.yml \
|
||||||
--prompt generic.markdown_summary \
|
|
||||||
--format json
|
--format json
|
||||||
|
|
||||||
go run ./cmd/scriptorium inspect profile \
|
go run ./cmd/scriptorium promptkit show profile local-gpu \
|
||||||
--config ./examples/config.full.yml \
|
--config ./examples/config.full.yml \
|
||||||
--profile local-gpu \
|
|
||||||
--format json
|
--format json
|
||||||
|
|||||||
2
go.mod
2
go.mod
@@ -8,6 +8,7 @@ require (
|
|||||||
go.maximumdirect.net/commandkit v0.7.0
|
go.maximumdirect.net/commandkit v0.7.0
|
||||||
go.maximumdirect.net/configkit v1.2.1
|
go.maximumdirect.net/configkit v1.2.1
|
||||||
go.maximumdirect.net/promptkit v0.23.3
|
go.maximumdirect.net/promptkit v0.23.3
|
||||||
|
go.yaml.in/yaml/v4 v4.0.0
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
@@ -15,6 +16,5 @@ require (
|
|||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect
|
||||||
github.com/spf13/cobra v1.10.2 // indirect
|
github.com/spf13/cobra v1.10.2 // indirect
|
||||||
github.com/spf13/pflag v1.1.0 // indirect
|
github.com/spf13/pflag v1.1.0 // indirect
|
||||||
go.yaml.in/yaml/v4 v4.0.0 // indirect
|
|
||||||
golang.org/x/text v0.44.0 // indirect
|
golang.org/x/text v0.44.0 // indirect
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,39 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
|
|
||||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func loadApplicationSettings(configPath, overlay string, explicit bool, overrides []configkit.Override) (appconfig.AppSettings, []configkit.Diagnostic, error) {
|
|
||||||
result, err := appconfig.Load(appconfig.LoadOptions{
|
|
||||||
Path: configPath, Overlay: overlay, Explicit: explicit, Overrides: overrides,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return appconfig.AppSettings{}, nil, err
|
|
||||||
}
|
|
||||||
return appconfig.Settings(result.Config), result.Diagnostics(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeApplicationError(output io.Writer, label string, err error) {
|
|
||||||
var loadError *configkit.LoadError
|
|
||||||
if errors.As(err, &loadError) {
|
|
||||||
_, _ = fmt.Fprintf(output, "%s error:\n", label)
|
|
||||||
if writeErr := configkit.WriteDiagnostics(output, loadError.Diagnostics()); writeErr != nil {
|
|
||||||
_, _ = fmt.Fprintf(output, "%s diagnostic write error: %v\n", label, writeErr)
|
|
||||||
}
|
|
||||||
return
|
|
||||||
}
|
|
||||||
_, _ = fmt.Fprintf(output, "%s error: %v\n", label, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeApplicationWarnings(output io.Writer, diagnostics []configkit.Diagnostic) error {
|
|
||||||
if len(diagnostics) == 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return configkit.WriteDiagnostics(output, diagnostics)
|
|
||||||
}
|
|
||||||
@@ -1,101 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
|
|
||||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
type configurationSource struct {
|
|
||||||
ID string `json:"id"`
|
|
||||||
Label string `json:"label"`
|
|
||||||
Layer configkit.Layer `json:"layer"`
|
|
||||||
Path string `json:"path,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func configCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
if len(args) == 0 {
|
|
||||||
fmt.Fprintln(stderr, "config parse error: configuration mode is required")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
mode := args[0]
|
|
||||||
if mode != "validate" && mode != "show" && mode != "sources" && mode != "explain" {
|
|
||||||
fmt.Fprintln(stderr, "config parse error: unknown configuration mode")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
fs := flag.NewFlagSet("config "+mode, flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
var path, overlay, pointer string
|
|
||||||
registerConfigSelectionFlags(fs, &path, &overlay)
|
|
||||||
registerConfigurationOverrides(fs)
|
|
||||||
if mode == "explain" {
|
|
||||||
fs.StringVar(&pointer, "path", "", "configuration path as a JSON Pointer")
|
|
||||||
}
|
|
||||||
if err := fs.Parse(args[1:]); err != nil {
|
|
||||||
writeApplicationError(stderr, "config parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
fmt.Fprintln(stderr, "config parse error: positional arguments are not accepted")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
var explanationPath configkit.Path
|
|
||||||
if mode == "explain" {
|
|
||||||
if !flagWasSet(fs, "path") {
|
|
||||||
fmt.Fprintln(stderr, "config parse error: --path is required")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
var err error
|
|
||||||
explanationPath, err = configkit.ParsePath(pointer)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintln(stderr, "config parse error: --path must be a JSON Pointer")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
result, err := appconfig.Load(appconfig.LoadOptions{
|
|
||||||
Path: path, Explicit: flagWasSet(fs, "config"), Overlay: overlay,
|
|
||||||
Overrides: applicationOverrides(fs),
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "config", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, result.Diagnostics()); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
switch mode {
|
|
||||||
case "validate":
|
|
||||||
return ExitOK
|
|
||||||
case "show":
|
|
||||||
err = result.WriteEffective(stdout)
|
|
||||||
case "sources":
|
|
||||||
sources := make([]configurationSource, 0, len(result.Sources()))
|
|
||||||
for _, source := range result.Sources() {
|
|
||||||
sources = append(sources, configurationSource{
|
|
||||||
ID: source.ID, Label: source.Label, Layer: source.Layer, Path: source.Path,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
err = json.NewEncoder(stdout).Encode(sources)
|
|
||||||
case "explain":
|
|
||||||
err = result.WriteExplanation(stdout, explanationPath)
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "config output error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func registerConfigurationOverrides(fs *flag.FlagSet) {
|
|
||||||
fs.String("addr", "", "HTTP listen address override")
|
|
||||||
fs.String("artifact-root", "", "HTTP artifact root override")
|
|
||||||
fs.Int64("max-request-bytes", 0, "HTTP request byte limit override")
|
|
||||||
fs.Int64("max-artifact-bytes", 0, "HTTP artifact byte limit override")
|
|
||||||
fs.Int64("max-response-bytes", 0, "HTTP response byte limit override")
|
|
||||||
}
|
|
||||||
@@ -1,226 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"reflect"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestConfigCommandsInspectApplicationWithoutEngine(t *testing.T) {
|
|
||||||
path := writeAppConfigFile(t, "{}\n")
|
|
||||||
for _, mode := range []string{"validate", "show", "sources"} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("%s failed: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
switch mode {
|
|
||||||
case "validate":
|
|
||||||
if stdout != "" {
|
|
||||||
t.Fatalf("validation wrote stdout: %q", stdout)
|
|
||||||
}
|
|
||||||
case "show":
|
|
||||||
if !strings.Contains(stdout, "render_format: text") || !strings.Contains(stdout, "max_request_bytes: 16777216") {
|
|
||||||
t.Fatalf("effective defaults missing: %q", stdout)
|
|
||||||
}
|
|
||||||
case "sources":
|
|
||||||
var sources []configurationSource
|
|
||||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "main" {
|
|
||||||
t.Fatalf("unexpected source records: %q (%v)", stdout, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCollectionExampleLoadsAndRenders(t *testing.T) {
|
|
||||||
t.Chdir(filepath.Join("..", "..", ".."))
|
|
||||||
want := []appconfig.SourceSelection{
|
|
||||||
{File: "examples/framework/backends.yml"},
|
|
||||||
{File: "examples/profiles/local-fast.yaml"},
|
|
||||||
{File: "examples/prompts/generic.markdown_summary.yaml"},
|
|
||||||
}
|
|
||||||
for _, overlay := range []string{"", "collection"} {
|
|
||||||
result, err := appconfig.Load(appconfig.LoadOptions{Path: "examples/config.collections.yml", Explicit: true, Overlay: overlay})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
expected := append([]appconfig.SourceSelection{}, want...)
|
|
||||||
if overlay != "" {
|
|
||||||
expected = append(expected, appconfig.SourceSelection{File: "examples/profiles/local-gpu.yml"})
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(result.Config.PromptKitSources, expected) {
|
|
||||||
t.Fatalf("overlay %q selectors = %+v, want %+v", overlay, result.Config.PromptKitSources, expected)
|
|
||||||
}
|
|
||||||
args := []string{"--config", "examples/config.collections.yml", "--prompt", "generic.markdown_summary", "--input", "transcript=examples/fixtures/transcript.md", "--input", "glossary=examples/fixtures/glossary.yml"}
|
|
||||||
if overlay != "" {
|
|
||||||
args = append(args, "--config-overlay", overlay)
|
|
||||||
}
|
|
||||||
code, stdout, stderr := runCLICommand(t, renderCommand, args)
|
|
||||||
if code != ExitOK || !strings.Contains(stdout, "generic.markdown_summary") || !strings.Contains(stdout, "local-fast") {
|
|
||||||
t.Fatalf("collection render failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
base := filepath.Join(dir, "config.yml")
|
|
||||||
for path, body := range map[string]string{
|
|
||||||
base: "config:\n imports:\n - path: fragment.yml\n required: true\npromptkit_sources:\n - directory: ' ./before/../prompts '\nschema_dir: ./schemas\n",
|
|
||||||
filepath.Join(dir, "fragment.yml"): "server:\n addr: :8081\n",
|
|
||||||
filepath.Join(dir, "overlays", "local.yml"): "promptkit_sources:\n - directory: ./overlay-prompts\nserver:\n addr: :9090\n",
|
|
||||||
} {
|
|
||||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
args := []string{"--config", base, "--config-overlay", "local", "--schema-dir=", "--addr", ":7070"}
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"show"}, args...))
|
|
||||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "directory: overlay-prompts") || !strings.Contains(stdout, "schema_dir: ''") || !strings.Contains(stdout, "addr: :7070") || strings.Contains(stdout, "before/../prompts") {
|
|
||||||
t.Fatalf("resolved effective output changed: code=%d out=%q err=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
code, stdout, stderr = runCLICommand(t, configCommand, append([]string{"sources"}, args...))
|
|
||||||
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
|
|
||||||
t.Fatalf("source command failed: code=%d out=%q err=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
var sources []configurationSource
|
|
||||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 5 {
|
|
||||||
t.Fatalf("unexpected composed source records: %+v (%v)", sources, err)
|
|
||||||
}
|
|
||||||
for i, layer := range []string{"main", "main", "overlay", "cli", "cli"} {
|
|
||||||
if string(sources[i].Layer) != layer {
|
|
||||||
t.Fatalf("source %d layer = %q, want %q", i, sources[i].Layer, layer)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestApplicationInspectionDoesNotAcquireFrameworkSources(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
frameworkPath := filepath.Join(dir, "invalid-framework.yml")
|
|
||||||
if err := os.WriteFile(frameworkPath, []byte("not: [valid\n"), 0o600); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
for _, frameworkSource := range []struct {
|
|
||||||
name string
|
|
||||||
path string
|
|
||||||
}{
|
|
||||||
{name: "invalid definition", path: frameworkPath},
|
|
||||||
{name: "missing path", path: filepath.Join(dir, "missing-framework.yml")},
|
|
||||||
} {
|
|
||||||
t.Run(frameworkSource.name, func(t *testing.T) {
|
|
||||||
path := writeAppConfigFile(t, "promptkit_sources:\n - file: "+frameworkSource.path+"\n")
|
|
||||||
for _, args := range [][]string{
|
|
||||||
{"validate"},
|
|
||||||
{"show"},
|
|
||||||
{"sources"},
|
|
||||||
{"explain", "--path", "/promptkit_sources/0/file"},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, append(args, "--config", path))
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("application inspection acquired configured framework source: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestConfigCommandsRejectRetiredFrameworkFields(t *testing.T) {
|
|
||||||
for _, field := range []string{"prompt_dir", "profile_dir", "backends"} {
|
|
||||||
t.Run(field, func(t *testing.T) {
|
|
||||||
path := writeAppConfigFile(t, field+": {}\n")
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", path})
|
|
||||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
|
||||||
t.Fatalf("retired application field %q was accepted: code=%d out=%q err=%q", field, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type errorOutputWriter struct{}
|
|
||||||
|
|
||||||
func (errorOutputWriter) Write([]byte) (int, error) { return 0, errors.New("output unavailable") }
|
|
||||||
|
|
||||||
func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) {
|
|
||||||
path := writeAppConfigFile(t, "server:\n max_request_bytes: -1\n")
|
|
||||||
for _, mode := range []string{"validate", "show", "sources"} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path})
|
|
||||||
if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "validate.error") {
|
|
||||||
t.Fatalf("%s misclassified invalid config: code=%d out=%q err=%q", mode, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
path = writeAppConfigFile(t, "{}\n")
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
if code := configCommand([]string{"show", "--config", path}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
|
|
||||||
t.Fatalf("writer error not reported: code=%d stderr=%q", code, stderr.String())
|
|
||||||
}
|
|
||||||
if code := configCommand([]string{"sources", "--config", path}, errorOutputWriter{}, io.Discard); code != ExitRuntimeError {
|
|
||||||
t.Fatalf("source writer error returned %d", code)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestConfigExplainUsesResolvedHistory(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
base := filepath.Join(dir, "config.yml")
|
|
||||||
for path, body := range map[string]string{
|
|
||||||
base: `promptkit_sources:
|
|
||||||
- directory: " ./prompts/../prompts "
|
|
||||||
server:
|
|
||||||
addr: :8080
|
|
||||||
`,
|
|
||||||
filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n",
|
|
||||||
} {
|
|
||||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
args := []string{"--config", base, "--config-overlay", "local", "--addr", ":9090"}
|
|
||||||
for _, tc := range []struct {
|
|
||||||
pointer string
|
|
||||||
want []string
|
|
||||||
}{
|
|
||||||
{pointer: "/server/addr", want: []string{":8080", ":9090", "--addr"}},
|
|
||||||
{pointer: "/promptkit_sources/0/directory", want: []string{"normalized application path", "prompts"}},
|
|
||||||
{pointer: "", want: []string{"promptkit_sources", "server", "addr"}},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", tc.pointer}, args...))
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("explain %q failed: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
for _, want := range tc.want {
|
|
||||||
if !strings.Contains(stdout, want) {
|
|
||||||
t.Fatalf("explain %q lacks %q: %q", tc.pointer, want, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestConfigExplainRejectsUnknownOrMalformedPaths(t *testing.T) {
|
|
||||||
path := writeAppConfigFile(t, "{}\n")
|
|
||||||
for _, args := range [][]string{
|
|
||||||
{"explain", "--config", path},
|
|
||||||
{"explain", "--config", path, "--path", "not-a-pointer"},
|
|
||||||
{"explain", "--config", path, "--path", "/missing"},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, args)
|
|
||||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
|
||||||
t.Fatalf("invalid explanation path accepted: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
if code := configCommand([]string{"explain", "--config", path, "--path", ""}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
|
|
||||||
t.Fatalf("explanation writer failure not reported: code=%d err=%q", code, stderr.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,7 +1,9 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"go.maximumdirect.net/promptkit"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -41,37 +43,46 @@ func TestCollectionCheckoutDiscoveryAcrossInvocations(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
config := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkPath))
|
config := filepath.Join(root, "app.yml")
|
||||||
|
writeNativeFile(t, config, fmt.Sprintf("promptkit:\n configuration: %q\n", frameworkPath))
|
||||||
input := filepath.Join(root, "transcript.md")
|
input := filepath.Join(root, "transcript.md")
|
||||||
if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil {
|
if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
args := func(name string) []string {
|
args := func(name string) []string {
|
||||||
if name == "support-repo" {
|
if name == "support-repo" {
|
||||||
return []string{"--config", config, "--prompt", "support/answer", "--profile", "support/fast", "--input", "transcript=" + input}
|
return []string{"support/answer", "--config", config, "--profile", "support/fast", "--input-file", "transcript=" + input}
|
||||||
}
|
}
|
||||||
return []string{"--config", config, "--prompt", "review/answer", "--profile", "review/base"}
|
return []string{"review/answer", "--config", config, "--profile", "review/base"}
|
||||||
}
|
}
|
||||||
check := func(name string) {
|
check := func(name string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
code, out, stderr := runCLICommand(t, renderCommand, args(name))
|
command := append([]string{"promptkit", "show", "prepared"}, args(name)...)
|
||||||
if code != ExitOK {
|
outcome, out, stderr := executeNative(t, append(command, "--format=json")...)
|
||||||
t.Fatalf("render %s: code=%d stderr=%s", name, code, stderr)
|
if outcome.Status != 0 {
|
||||||
|
t.Fatalf("prepare %s: outcome=%+v stderr=%s", name, outcome, stderr)
|
||||||
|
}
|
||||||
|
var prepared promptkit.PreparedRun
|
||||||
|
if err := json.Unmarshal([]byte(out), &prepared); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if name == "support-repo" {
|
if name == "support-repo" {
|
||||||
for _, want := range []string{"prompt: support/answer", "selected_profile_id: support/fast", "model: support-example", "max_tokens: 256", "Answer the support request", "synthetic discovery transcript"} {
|
for _, want := range []string{"support/answer", "support/fast", "support-example", "Answer the support request", "synthetic discovery transcript"} {
|
||||||
if !strings.Contains(out, want) {
|
if !strings.Contains(out, want) {
|
||||||
t.Fatalf("missing %q: %s", want, out)
|
t.Fatalf("missing %q: %s", want, out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else if !strings.Contains(out, "prompt: review/answer") || !strings.Contains(out, "model: review-example") || !strings.Contains(out, "Review this material") {
|
if prepared.EffectiveModelParams.MaxTokens != 256 {
|
||||||
|
t.Fatalf("inherited profile setting missing: %+v", prepared)
|
||||||
|
}
|
||||||
|
} else if !strings.Contains(out, "review/answer") || !strings.Contains(out, "review-example") || !strings.Contains(out, "Review this material") {
|
||||||
t.Fatalf("unexpected review render: %s", out)
|
t.Fatalf("unexpected review render: %s", out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
install(tc.first)
|
install(tc.first)
|
||||||
check(tc.first)
|
check(tc.first)
|
||||||
code, _, _ := runCLICommand(t, renderCommand, args(tc.second))
|
outcome, _, _ := executeNative(t, append([]string{"promptkit", "show", "prepared"}, args(tc.second)...)...)
|
||||||
if code != ExitRuntimeError {
|
if outcome.Status != 1 {
|
||||||
t.Fatal("uninstalled collection unexpectedly selected")
|
t.Fatal("uninstalled collection unexpectedly selected")
|
||||||
}
|
}
|
||||||
install(tc.second)
|
install(tc.second)
|
||||||
|
|||||||
42
internal/adapter/cli/examples_test.go
Normal file
42
internal/adapter/cli/examples_test.go
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMaintainedApplicationExamplesUseNativeLoading(t *testing.T) {
|
||||||
|
root, err := filepath.Abs(filepath.Join("..", "..", ".."))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Chdir(root)
|
||||||
|
entries, err := os.ReadDir("examples")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() || !strings.HasPrefix(entry.Name(), "config") || !strings.HasSuffix(entry.Name(), ".yml") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
path := filepath.Join("examples", entry.Name())
|
||||||
|
for _, scope := range []string{"app", "promptkit"} {
|
||||||
|
outcome, _, diagnostics := executeNative(t, "config", "validate", "--config="+path, "--scope="+scope)
|
||||||
|
if outcome.Status != 0 {
|
||||||
|
t.Fatalf("%s/%s: %+v %q", path, scope, outcome, diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, test := range []struct{ config, overlay string }{{"examples/config.yml", ""}, {"examples/config.composed.yml", "local"}, {"examples/config.collections.yml", "collection"}, {"examples/config.inheritance.yml", ""}} {
|
||||||
|
args := []string{"promptkit", "show", "prepared", "generic.markdown_summary", "--config=" + test.config, "--profile=local-gpu", "--input-file=transcript=examples/fixtures/transcript.md", "--input-file=glossary=examples/fixtures/glossary.yml", "--format=json"}
|
||||||
|
if test.overlay != "" {
|
||||||
|
args = append(args, "--overlay="+test.overlay)
|
||||||
|
}
|
||||||
|
outcome, output, diagnostics := executeNative(t, args...)
|
||||||
|
if outcome.Status != 0 || !strings.Contains(output, "generic.markdown_summary") || !strings.Contains(output, "Session Transcript") {
|
||||||
|
t.Fatalf("%+v: %+v %q %q", test, outcome, output, diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,159 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
type frameworkConfigurationConfig struct {
|
|
||||||
mode, configPath, configOverlay, schemaDir string
|
|
||||||
path string
|
|
||||||
configExplicit bool
|
|
||||||
pathExplicit bool
|
|
||||||
overrides []configkit.Override
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
}
|
|
||||||
|
|
||||||
type frameworkConfigurationSource struct {
|
|
||||||
ID string `json:"id"`
|
|
||||||
Label string `json:"label"`
|
|
||||||
Layer configkit.Layer `json:"layer"`
|
|
||||||
Kind configkit.SourceKind `json:"kind"`
|
|
||||||
UnderlayIndex int `json:"underlay_index"`
|
|
||||||
Path string `json:"path,omitempty"`
|
|
||||||
ImportedFrom *frameworkConfigurationLocation `json:"imported_from,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type frameworkConfigurationLocation struct {
|
|
||||||
SourceID string `json:"source_id"`
|
|
||||||
Line int `json:"line"`
|
|
||||||
Column int `json:"column"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func inspectFrameworkConfigurationCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
cfg, err := parseFrameworkConfigurationArgs(args)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect configuration parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
var explanationPath configkit.Path
|
|
||||||
if cfg.mode == "explain" {
|
|
||||||
explanationPath, err = configkit.ParsePath(cfg.path)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect configuration path", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
}
|
|
||||||
settings, err := resolveAppSettingsForFrameworkConfiguration(cfg)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect configuration", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
engine, err := newEngine(settings)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
view, err := engine.InspectConfiguration()
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect configuration", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := configkit.WriteDiagnostics(stderr, view.Diagnostics()); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if cfg.mode == "validate" {
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
if cfg.mode == "explain" {
|
|
||||||
if err := view.WriteExplanation(stdout, explanationPath); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect configuration explain %q: %v\n", cfg.path, err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
switch cfg.mode {
|
|
||||||
case "show":
|
|
||||||
err = view.WriteEffective(stdout)
|
|
||||||
case "sources":
|
|
||||||
sources := frameworkConfigurationSources(view.Sources())
|
|
||||||
err = json.NewEncoder(stdout).Encode(sources)
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect configuration output error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseFrameworkConfigurationArgs(args []string) (*frameworkConfigurationConfig, error) {
|
|
||||||
if len(args) == 0 {
|
|
||||||
return nil, errors.New("configuration operation is required")
|
|
||||||
}
|
|
||||||
cfg := &frameworkConfigurationConfig{mode: args[0]}
|
|
||||||
switch cfg.mode {
|
|
||||||
case "validate", "show", "sources", "explain":
|
|
||||||
default:
|
|
||||||
return nil, fmt.Errorf("unknown configuration operation %q", cfg.mode)
|
|
||||||
}
|
|
||||||
|
|
||||||
fs := flag.NewFlagSet("inspect configuration "+cfg.mode, flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
fs.StringVar(&cfg.path, "path", "", "JSON Pointer to explain (required for explain)")
|
|
||||||
if err := fs.Parse(args[1:]); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
|
||||||
}
|
|
||||||
cfg.configExplicit = flagWasSet(fs, "config")
|
|
||||||
cfg.pathExplicit = flagWasSet(fs, "path")
|
|
||||||
if cfg.mode == "explain" && !cfg.pathExplicit {
|
|
||||||
return nil, errors.New("--path is required for explain")
|
|
||||||
}
|
|
||||||
cfg.overrides = applicationOverrides(fs)
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func resolveAppSettingsForFrameworkConfiguration(cfg *frameworkConfigurationConfig) (engineSettings, error) {
|
|
||||||
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
|
|
||||||
if err != nil {
|
|
||||||
return engineSettings{}, err
|
|
||||||
}
|
|
||||||
cfg.diagnostics = diagnostics
|
|
||||||
return engineSettings{
|
|
||||||
promptkitSources: settings.PromptKitSources,
|
|
||||||
schemaDir: settings.SchemaDir,
|
|
||||||
transientRetries: settings.TransientRetries,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func frameworkConfigurationSources(sources []configkit.SourceInfo) []frameworkConfigurationSource {
|
|
||||||
result := make([]frameworkConfigurationSource, 0, len(sources))
|
|
||||||
for _, source := range sources {
|
|
||||||
item := frameworkConfigurationSource{
|
|
||||||
ID: source.ID, Label: source.Label, Layer: source.Layer,
|
|
||||||
Kind: source.Kind, UnderlayIndex: source.UnderlayIndex, Path: source.Path,
|
|
||||||
}
|
|
||||||
if source.ImportedFrom != nil {
|
|
||||||
item.ImportedFrom = &frameworkConfigurationLocation{
|
|
||||||
SourceID: source.ImportedFrom.SourceID,
|
|
||||||
Line: source.ImportedFrom.Line,
|
|
||||||
Column: source.ImportedFrom.Column,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
result = append(result, item)
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
@@ -1,391 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
|
||||||
configkit "go.maximumdirect.net/configkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationCommandsNeedNoPromptOrCredential(t *testing.T) {
|
|
||||||
configPath := writeAppConfigFile(t, "{}\n")
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
|
|
||||||
"configuration", "validate", "--config", configPath,
|
|
||||||
})
|
|
||||||
if code != ExitOK || stdout != "" || stderr != "" {
|
|
||||||
t.Fatalf("source-free validation failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Setenv("SCRIPTORIUM_FRAMEWORK_KEY", "")
|
|
||||||
root := t.TempDir()
|
|
||||||
frameworkFile := filepath.Join(root, "framework.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
|
|
||||||
- id: local
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
api_key_env: SCRIPTORIUM_FRAMEWORK_KEY
|
|
||||||
profiles:
|
|
||||||
- id: protected
|
|
||||||
backend: local
|
|
||||||
model: offline-model
|
|
||||||
api_key_required: true
|
|
||||||
`)
|
|
||||||
configPath = writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
|
||||||
for _, mode := range []string{"validate", "show", "sources"} {
|
|
||||||
code, stdout, stderr = runCLICommand(t, inspectCommand, []string{
|
|
||||||
"configuration", mode, "--config", configPath,
|
|
||||||
})
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("%s required a prompt or credential: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
if mode == "validate" && stdout != "" {
|
|
||||||
t.Fatalf("validation wrote success output: %q", stdout)
|
|
||||||
}
|
|
||||||
if mode == "show" && (!strings.Contains(stdout, "profiles:") || !strings.Contains(stdout, "offline-model")) {
|
|
||||||
t.Fatalf("effective framework YAML is missing: %q", stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationSourcesProjectsOrderedNativeRecords(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
importDir := filepath.Join(root, "catalog")
|
|
||||||
if err := os.MkdirAll(importDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
mainPath := writeFrameworkSourceFile(t, root, "main.yml", `config:
|
|
||||||
imports:
|
|
||||||
- path: catalog/fragment.yml
|
|
||||||
required: true
|
|
||||||
backends:
|
|
||||||
- id: local
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
`)
|
|
||||||
fragmentPath := writeFrameworkSourceFile(t, importDir, "fragment.yml", `profiles:
|
|
||||||
- id: imported
|
|
||||||
model: imported-model
|
|
||||||
`)
|
|
||||||
directory := t.TempDir()
|
|
||||||
directoryPath := writeFrameworkSourceFile(t, directory, "directory.yml", `profiles:
|
|
||||||
- id: directory
|
|
||||||
model: directory-model
|
|
||||||
`)
|
|
||||||
configuredMissing := filepath.Join(root, "configured-but-replaced.yml")
|
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", configuredMissing))
|
|
||||||
args := []string{
|
|
||||||
"configuration", "sources", "--config", configPath,
|
|
||||||
"--promptkit-source", mainPath,
|
|
||||||
"--promptkit-source", directory,
|
|
||||||
}
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
|
||||||
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
|
|
||||||
t.Fatalf("framework sources command failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
var sources []frameworkConfigurationSource
|
|
||||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil {
|
|
||||||
t.Fatalf("decode framework source DTOs: %v\n%s", err, stdout)
|
|
||||||
}
|
|
||||||
var rawSources []map[string]any
|
|
||||||
if err := json.Unmarshal([]byte(stdout), &rawSources); err != nil {
|
|
||||||
t.Fatalf("decode framework source fields: %v", err)
|
|
||||||
}
|
|
||||||
mainIndex := findFrameworkSource(t, sources, mainPath)
|
|
||||||
fragmentIndex := findFrameworkSource(t, sources, fragmentPath)
|
|
||||||
directoryIndex := findFrameworkSource(t, sources, directoryPath)
|
|
||||||
main := sources[mainIndex]
|
|
||||||
fragment := sources[fragmentIndex]
|
|
||||||
directorySource := sources[directoryIndex]
|
|
||||||
if !(mainIndex < fragmentIndex && fragmentIndex < directoryIndex) {
|
|
||||||
t.Fatalf("captured source order changed: main=%d fragment=%d directory=%d", mainIndex, fragmentIndex, directoryIndex)
|
|
||||||
}
|
|
||||||
if main.ID == "" || main.Label == "" || main.Layer != configkit.LayerUnderlay || main.Kind != configkit.SourceKindFile || main.UnderlayIndex < 1 {
|
|
||||||
t.Fatalf("explicit file source fields are incomplete: %+v", main)
|
|
||||||
}
|
|
||||||
for _, key := range []string{"id", "label", "layer", "kind", "underlay_index", "path"} {
|
|
||||||
if _, exists := rawSources[mainIndex][key]; !exists {
|
|
||||||
t.Fatalf("source DTO omitted %q: %#v", key, rawSources[mainIndex])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for _, key := range []string{"ID", "Label", "Layer", "Kind", "UnderlayIndex", "filesystem"} {
|
|
||||||
if _, exists := rawSources[mainIndex][key]; exists {
|
|
||||||
t.Fatalf("source DTO exposed implementation field %q: %#v", key, rawSources[mainIndex])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if fragment.Layer != configkit.LayerUnderlay || fragment.Kind != configkit.SourceKindFile || fragment.ImportedFrom == nil ||
|
|
||||||
fragment.ImportedFrom.SourceID != main.ID || fragment.ImportedFrom.Line < 1 || fragment.ImportedFrom.Column < 1 {
|
|
||||||
t.Fatalf("import source fields are incomplete: %+v", fragment)
|
|
||||||
}
|
|
||||||
if importedFrom, ok := rawSources[fragmentIndex]["imported_from"].(map[string]any); !ok || importedFrom["source_id"] != main.ID || importedFrom["line"] == nil || importedFrom["column"] == nil {
|
|
||||||
t.Fatalf("import location DTO fields are incomplete: %#v", rawSources[fragmentIndex])
|
|
||||||
}
|
|
||||||
if directorySource.Layer != configkit.LayerUnderlay || directorySource.Kind != configkit.SourceKindFile || directorySource.UnderlayIndex <= main.UnderlayIndex {
|
|
||||||
t.Fatalf("directory-selected source fields/order are incomplete: %+v (main %+v)", directorySource, main)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationShowFailsBeforeOutputForInvalidFinalDefinition(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
frameworkFile := writeFrameworkSourceFile(t, root, "invalid.yml", `prompts:
|
|
||||||
- id: unused
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
|
||||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", configPath})
|
|
||||||
if code != ExitOK || stdout != "" || stderr != "" {
|
|
||||||
t.Fatalf("application validation loaded framework definitions: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
for _, args := range [][]string{
|
|
||||||
{"configuration", "validate", "--config", configPath},
|
|
||||||
{"configuration", "show", "--config", configPath},
|
|
||||||
{"configuration", "sources", "--config", configPath},
|
|
||||||
{"configuration", "explain", "--config", configPath, "--path", ""},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr = runCLICommand(t, inspectCommand, args)
|
|
||||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
|
||||||
t.Fatalf("invalid final framework definition was reported as success: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationCommandsRejectExtraArgumentsAndReportWriterErrors(t *testing.T) {
|
|
||||||
configPath := writeAppConfigFile(t, "{}\n")
|
|
||||||
for _, args := range [][]string{
|
|
||||||
{"configuration"},
|
|
||||||
{"configuration", "explain"},
|
|
||||||
{"configuration", "show", "unexpected"},
|
|
||||||
{"configuration", "sources", "--format", "json"},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
|
||||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
|
||||||
t.Fatalf("invalid inspection arguments were accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
writerError := errors.New("synthetic framework output failure")
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
code := inspectCommand([]string{"configuration", "show", "--config", configPath}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
|
|
||||||
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
|
|
||||||
t.Fatalf("effective-output writer failure was not reported: code=%d stderr=%q", code, stderr.String())
|
|
||||||
}
|
|
||||||
stderr.Reset()
|
|
||||||
code = inspectCommand([]string{"configuration", "explain", "--config", configPath, "--path", ""}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
|
|
||||||
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
|
|
||||||
t.Fatalf("explanation writer failure was not reported: code=%d stderr=%q", code, stderr.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationExplainUsesLiteralJSONPointersAndNativeHistory(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
lower := writeFrameworkSourceFile(t, root, "lower.yml", `collections:
|
|
||||||
support:
|
|
||||||
backends:
|
|
||||||
- id: local
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
|
||||||
extra_params:
|
|
||||||
token: backend-token-displaced-canary
|
|
||||||
password: backend-password-canary
|
|
||||||
visible_option: backend-visible-value
|
|
||||||
provider.option: dotted-visible-value
|
|
||||||
profiles:
|
|
||||||
- id: summary
|
|
||||||
backend: support/local
|
|
||||||
model: lower-model
|
|
||||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
|
||||||
extra_params:
|
|
||||||
token: profile-token-displaced-canary
|
|
||||||
password: profile-password-displaced-canary
|
|
||||||
visible_option: profile-visible-value
|
|
||||||
provider.option: dotted-visible-value
|
|
||||||
prompts:
|
|
||||||
- id: explainable
|
|
||||||
default_profile: support/summary
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: hello
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
upperRoot := t.TempDir()
|
|
||||||
upper := writeFrameworkSourceFile(t, upperRoot, "upper.yml", `collections:
|
|
||||||
support:
|
|
||||||
backends:
|
|
||||||
- id: local
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
|
||||||
extra_params:
|
|
||||||
token: backend-token-effective-canary
|
|
||||||
password: backend-password-effective-canary
|
|
||||||
visible_option: backend-visible-value
|
|
||||||
provider.option: dotted-visible-value
|
|
||||||
profiles:
|
|
||||||
- id: summary
|
|
||||||
backend: support/local
|
|
||||||
model: effective-model
|
|
||||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
|
||||||
extra_params:
|
|
||||||
token: profile-token-effective-canary
|
|
||||||
password: profile-password-effective-canary
|
|
||||||
visible_option: profile-visible-value
|
|
||||||
provider.option: dotted-visible-value
|
|
||||||
`)
|
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n - file: %q\n", lower, upper))
|
|
||||||
const secret = "environment-value-must-never-be-emitted"
|
|
||||||
t.Setenv("SCRIPTORIUM_EXPLAIN_REFERENCE", secret)
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
path string
|
|
||||||
want []string
|
|
||||||
unwant []string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
path: "/collections/support/backends/0/extra_params/token",
|
|
||||||
unwant: []string{"backend-token-displaced-canary", "backend-token-effective-canary"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/backends/0/extra_params/password",
|
|
||||||
unwant: []string{"backend-password-displaced-canary", "backend-password-effective-canary"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/profiles/0/model",
|
|
||||||
want: []string{"effective-model", "lower-model"},
|
|
||||||
unwant: []string{},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/profiles/0/extra_params/token",
|
|
||||||
unwant: []string{"profile-token-displaced-canary", "profile-token-effective-canary"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/profiles/0/extra_params/password",
|
|
||||||
unwant: []string{"profile-password-displaced-canary", "profile-password-effective-canary"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/profiles/0/api_key_env",
|
|
||||||
want: []string{"SCRIPTORIUM_EXPLAIN_REFERENCE"},
|
|
||||||
unwant: []string{secret},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "/collections/support/profiles/0/extra_params/provider.option",
|
|
||||||
want: []string{"dotted-visible-value"},
|
|
||||||
unwant: []string{"provider/option"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
path: "",
|
|
||||||
want: []string{"profiles:", "effective-model"},
|
|
||||||
unwant: []string{"profile-token-displaced-canary", "backend-password-effective-canary"},
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
args := []string{"configuration", "explain", "--config", configPath, "--path", tc.path}
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("explain %q failed: code=%d stdout=%q stderr=%q", tc.path, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
if stdout == "" {
|
|
||||||
t.Fatalf("explain %q produced no output", tc.path)
|
|
||||||
}
|
|
||||||
for _, value := range tc.want {
|
|
||||||
if !strings.Contains(stdout, value) {
|
|
||||||
t.Errorf("explain %q omitted %q: %s", tc.path, value, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for _, value := range tc.unwant {
|
|
||||||
if strings.Contains(stdout, value) {
|
|
||||||
t.Errorf("explain %q disclosed %q: %s", tc.path, value, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.Contains(stdout, secret) {
|
|
||||||
t.Errorf("explain %q disclosed the resolved environment value: %s", tc.path, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{"configuration", "show", "--config", configPath})
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("framework show failed: code=%d stderr=%q", code, stderr)
|
|
||||||
}
|
|
||||||
for _, value := range []string{"SCRIPTORIUM_EXPLAIN_REFERENCE", "backend-visible-value", "profile-visible-value", "dotted-visible-value"} {
|
|
||||||
if !strings.Contains(stdout, value) {
|
|
||||||
t.Errorf("effective report omitted visible configuration %q: %s", value, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for _, value := range []string{
|
|
||||||
"backend-token-displaced-canary", "backend-token-effective-canary",
|
|
||||||
"backend-password-displaced-canary", "backend-password-effective-canary",
|
|
||||||
"profile-token-displaced-canary", "profile-token-effective-canary",
|
|
||||||
"profile-password-displaced-canary", "profile-password-effective-canary", secret,
|
|
||||||
} {
|
|
||||||
if strings.Contains(stdout, value) {
|
|
||||||
t.Errorf("effective report disclosed classified or resolved value %q: %s", value, stdout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
profileCode, profileOutput, profileError := runCLICommand(t, inspectCommand, []string{
|
|
||||||
"profile", "--config", configPath, "--profile", "support/summary", "--format", "json",
|
|
||||||
})
|
|
||||||
if profileCode != ExitOK || profileError != "" {
|
|
||||||
t.Fatalf("existing profile inspection failed: code=%d stdout=%q stderr=%q", profileCode, profileOutput, profileError)
|
|
||||||
}
|
|
||||||
var profile renderformat.ProfileInspection
|
|
||||||
if err := json.Unmarshal([]byte(profileOutput), &profile); err != nil {
|
|
||||||
t.Fatalf("decode profile inspection: %v\n%s", err, profileOutput)
|
|
||||||
}
|
|
||||||
if pointerValue(profile.ProfileModelParams.Model) != "effective-model" || pointerValue(profile.ProfileModelParams.ExtraParams)["visible_option"] != "profile-visible-value" {
|
|
||||||
t.Fatalf("existing profile inspection changed configured provider data: %+v", profile.ProfileModelParams)
|
|
||||||
}
|
|
||||||
if strings.Contains(profileOutput, secret) {
|
|
||||||
t.Fatalf("profile inspection emitted resolved environment data: %s", profileOutput)
|
|
||||||
}
|
|
||||||
|
|
||||||
code, rendered, stderr := runCLICommand(t, renderCommand, []string{"--config", configPath, "--prompt", "support/explainable"})
|
|
||||||
if code != ExitOK || stderr != "" {
|
|
||||||
t.Fatalf("prepared output failed: code=%d stdout=%q stderr=%q", code, rendered, stderr)
|
|
||||||
}
|
|
||||||
if !strings.Contains(rendered, "profile-visible-value") || strings.Contains(rendered, secret) {
|
|
||||||
t.Fatalf("prepared output did not preserve ordinary provider data or disclosed environment data: %s", rendered)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkConfigurationExplainRejectsMissingMalformedAndUnknownPaths(t *testing.T) {
|
|
||||||
configPath := writeAppConfigFile(t, "{}\n")
|
|
||||||
for _, args := range [][]string{
|
|
||||||
{"configuration", "explain"},
|
|
||||||
{"configuration", "explain", "--path", "not-a-pointer"},
|
|
||||||
{"configuration", "explain", "--path", "/profiles/team~2summary/model"},
|
|
||||||
{"configuration", "explain", "--config", configPath, "--path", "/profiles/missing/model"},
|
|
||||||
} {
|
|
||||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
|
||||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
|
||||||
t.Fatalf("invalid explanation path was accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func findFrameworkSource(t *testing.T, sources []frameworkConfigurationSource, path string) int {
|
|
||||||
t.Helper()
|
|
||||||
for index, source := range sources {
|
|
||||||
if filepath.Clean(source.Path) == filepath.Clean(path) {
|
|
||||||
return index
|
|
||||||
}
|
|
||||||
}
|
|
||||||
t.Fatalf("selected source path %q was not present: %+v", path, sources)
|
|
||||||
return -1
|
|
||||||
}
|
|
||||||
|
|
||||||
type frameworkConfigurationFailingWriter struct {
|
|
||||||
err error
|
|
||||||
}
|
|
||||||
|
|
||||||
func (writer frameworkConfigurationFailingWriter) Write([]byte) (int, error) {
|
|
||||||
return 0, writer.err
|
|
||||||
}
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func writeFrameworkDiagnostics(output io.Writer, engine *promptkit.Engine) error {
|
|
||||||
view, err := engine.InspectConfiguration()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("inspect captured PromptKit diagnostics: %w", err)
|
|
||||||
}
|
|
||||||
return configkit.WriteDiagnostics(output, view.Diagnostics())
|
|
||||||
}
|
|
||||||
@@ -1,386 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
|
||||||
configkit "go.maximumdirect.net/configkit"
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestFrameworkSourceGroupsComposeAndValidateEffectiveDefinitions(t *testing.T) {
|
|
||||||
conflicting := t.TempDir()
|
|
||||||
writeFrameworkSourceFile(t, conflicting, "first.yml", `profiles:
|
|
||||||
- id: selected
|
|
||||||
model: first
|
|
||||||
`)
|
|
||||||
writeFrameworkSourceFile(t, conflicting, "second.yml", `profiles:
|
|
||||||
- id: selected
|
|
||||||
model: second
|
|
||||||
`)
|
|
||||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{Directory: conflicting}}}); err == nil {
|
|
||||||
t.Fatal("conflicting files in one source group unexpectedly constructed an engine")
|
|
||||||
}
|
|
||||||
|
|
||||||
lower := t.TempDir()
|
|
||||||
upper := t.TempDir()
|
|
||||||
empty := t.TempDir()
|
|
||||||
writeFrameworkSourceFile(t, lower, "lower.yml", `profiles:
|
|
||||||
- id: selected
|
|
||||||
endpoint: not-a-url
|
|
||||||
model: lower-model
|
|
||||||
temperature: 10
|
|
||||||
`)
|
|
||||||
writeFrameworkSourceFile(t, upper, "upper.yml", `profiles:
|
|
||||||
- id: selected
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
model: upper-model
|
|
||||||
temperature: 0.5
|
|
||||||
`)
|
|
||||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{
|
|
||||||
{Directory: lower}, {Directory: empty}, {Directory: upper},
|
|
||||||
}, "")
|
|
||||||
inspection, err := engine.InspectProfile(context.Background(), "selected")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("inspect profile after later source replacement: %v", err)
|
|
||||||
}
|
|
||||||
if pointerValue(inspection.Endpoint) != "http://127.0.0.1:1/v1" || pointerValue(inspection.Model) != "upper-model" || pointerValue(inspection.Temperature) != 0.5 {
|
|
||||||
t.Fatalf("later source did not replace lower values before validation: %+v", inspection)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkSourceConstructionUsesCombinedDocumentAndRejectsInvalidSources(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
combined := filepath.Join(root, "combined.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "combined.yml", `backends:
|
|
||||||
- id: local
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
profiles:
|
|
||||||
- id: selected
|
|
||||||
backend: local
|
|
||||||
model: combined-model
|
|
||||||
prompts:
|
|
||||||
- id: combined.prompt
|
|
||||||
default_profile: selected
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: hello
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
client := &frameworkSourceCaptureClient{}
|
|
||||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: combined}}, "", promptkit.WithLLMClient(client))
|
|
||||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "combined.prompt"}); err != nil {
|
|
||||||
t.Fatalf("run definitions from one combined document: %v", err)
|
|
||||||
}
|
|
||||||
if len(client.requests) != 1 || client.requests[0].Target.BackendID != "local" || client.requests[0].Target.Model != "combined-model" {
|
|
||||||
t.Fatalf("combined backend/profile/prompt values did not reach generation: %+v", client.requests)
|
|
||||||
}
|
|
||||||
|
|
||||||
malformed := filepath.Join(root, "malformed.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "malformed.yml", "not: [valid\n")
|
|
||||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: malformed}}}); err == nil {
|
|
||||||
t.Fatal("malformed framework source unexpectedly constructed an engine")
|
|
||||||
}
|
|
||||||
|
|
||||||
invalidUnused := filepath.Join(root, "invalid-unused.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "invalid-unused.yml", `prompts:
|
|
||||||
- id: unused.prompt
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: "{{"
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: invalidUnused}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
|
|
||||||
t.Fatalf("invalid unused definition did not fail as framework configuration: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkRejectsObsoleteVersionKeyedPrompt(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
path := writeFrameworkSourceFile(t, root, "obsolete.yml", `prompts:
|
|
||||||
old-prompt:
|
|
||||||
"1":
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: hello
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: path}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
|
|
||||||
t.Fatalf("obsolete version-keyed definition was not rejected by PromptKit: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFrameworkImportsRetainResourceOriginsAndCaptureAtConstruction(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
catalogDir := filepath.Join(root, "catalog")
|
|
||||||
if err := os.MkdirAll(catalogDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
schemaDir := filepath.Join(root, "schemas")
|
|
||||||
if err := os.MkdirAll(schemaDir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
mainFile := filepath.Join(root, "main.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "main.yml", `config:
|
|
||||||
imports:
|
|
||||||
- path: catalog/base.yml
|
|
||||||
required: true
|
|
||||||
`)
|
|
||||||
writeFrameworkSourceFile(t, catalogDir, "base.yml", `profiles:
|
|
||||||
- id: profile-primary
|
|
||||||
endpoint: http://127.0.0.1:1/v1
|
|
||||||
model: imported-model
|
|
||||||
prompts:
|
|
||||||
- id: team-prompt
|
|
||||||
default_profile: profile-primary
|
|
||||||
inputs:
|
|
||||||
- name: source
|
|
||||||
required: true
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content_file: ./body.md
|
|
||||||
- role: user
|
|
||||||
content: 'Artifact: {{input "source"}}'
|
|
||||||
output:
|
|
||||||
format: json
|
|
||||||
validation_mode: json_schema
|
|
||||||
schema_path: response.schema.json
|
|
||||||
`)
|
|
||||||
writeFrameworkSourceFile(t, catalogDir, "body.md", "Captured message for {{.audience}}")
|
|
||||||
artifactPath := filepath.Join(root, "request-artifact.txt")
|
|
||||||
writeFrameworkSourceFile(t, root, "request-artifact.txt", "initial artifact")
|
|
||||||
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"initial schema"}`)
|
|
||||||
selections := []appconfig.SourceSelection{{File: mainFile}}
|
|
||||||
engine := newFrameworkSourceTestEngine(t, selections, schemaDir)
|
|
||||||
|
|
||||||
writeFrameworkSourceFile(t, catalogDir, "body.md", "edited message for {{.audience}}")
|
|
||||||
writeFrameworkSourceFile(t, root, "request-artifact.txt", "prepared artifact")
|
|
||||||
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"prepared schema"}`)
|
|
||||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
|
||||||
PromptID: "team-prompt",
|
|
||||||
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
|
|
||||||
Vars: map[string]string{"audience": "reader"},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("prepare inherited imported prompt: %v", err)
|
|
||||||
}
|
|
||||||
if len(prepared.Messages) != 2 || prepared.Messages[0].Content != "Captured message for reader" || prepared.Messages[1].Content != "Artifact: prepared artifact" {
|
|
||||||
t.Fatalf("resource origin or capture/preparation behavior changed: %+v", prepared.Messages)
|
|
||||||
}
|
|
||||||
if prepared.SelectedProfileID != "profile-primary" {
|
|
||||||
t.Fatalf("profile ID was not preserved: %+v", prepared)
|
|
||||||
}
|
|
||||||
if prepared.StructuredOutput == nil || prepared.StructuredOutput.JSONSchema == nil || prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)["title"] != "prepared schema" {
|
|
||||||
t.Fatalf("schema was not loaded during preparation: %+v", prepared.StructuredOutput)
|
|
||||||
}
|
|
||||||
|
|
||||||
newEngine := newFrameworkSourceTestEngine(t, selections, schemaDir)
|
|
||||||
newPrepared, err := newEngine.Prepare(context.Background(), promptkit.RunRequest{
|
|
||||||
PromptID: "team-prompt",
|
|
||||||
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
|
|
||||||
Vars: map[string]string{"audience": "reader"},
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("prepare with newly captured message: %v", err)
|
|
||||||
}
|
|
||||||
if newPrepared.Messages[0].Content != "edited message for reader" {
|
|
||||||
t.Fatalf("new engine did not observe the message edit: %+v", newPrepared.Messages[0])
|
|
||||||
}
|
|
||||||
|
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\nschema_dir: %q\n", mainFile, schemaDir))
|
|
||||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
|
||||||
"--config", configPath,
|
|
||||||
"--prompt", "team-prompt",
|
|
||||||
"--input", "source=" + artifactPath,
|
|
||||||
"--var", "audience=reader",
|
|
||||||
"--format", "json",
|
|
||||||
})
|
|
||||||
if code != ExitOK {
|
|
||||||
t.Fatalf("render exact literal IDs through CLI: code=%d stderr=%q", code, stderr)
|
|
||||||
}
|
|
||||||
var rendered promptkit.PreparedRun
|
|
||||||
if err := json.Unmarshal([]byte(stdout), &rendered); err != nil {
|
|
||||||
t.Fatalf("decode rendered literal-ID fixture: %v", err)
|
|
||||||
}
|
|
||||||
if rendered.PromptID != "team-prompt" || rendered.SelectedProfileID != "profile-primary" {
|
|
||||||
t.Fatalf("CLI changed exact prompt/profile IDs: %+v", rendered)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNativeProfileInheritanceAndPresenceReachTheClientBoundary(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
frameworkFile := filepath.Join(root, "framework.yml")
|
|
||||||
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
|
|
||||||
- id: first
|
|
||||||
endpoint: https://first.example/v1
|
|
||||||
api_key_env: FIRST_BACKEND_KEY
|
|
||||||
extra_params:
|
|
||||||
source: first
|
|
||||||
nested:
|
|
||||||
first_backend: true
|
|
||||||
- id: second
|
|
||||||
endpoint: https://second.example/v1
|
|
||||||
api_key_env: SECOND_BACKEND_KEY
|
|
||||||
extra_params:
|
|
||||||
source: second
|
|
||||||
nested:
|
|
||||||
second_backend: true
|
|
||||||
profiles:
|
|
||||||
- id: parent
|
|
||||||
backend: first
|
|
||||||
model: parent-model
|
|
||||||
temperature: 0.7
|
|
||||||
top_p: 0.6
|
|
||||||
max_tokens: 45
|
|
||||||
timeout_seconds: 30
|
|
||||||
service_tier: flex
|
|
||||||
extra_params:
|
|
||||||
nested:
|
|
||||||
parent_profile: true
|
|
||||||
retained: "yes"
|
|
||||||
- id: backend-child
|
|
||||||
extends: {id: parent}
|
|
||||||
backend: second
|
|
||||||
model: child-model
|
|
||||||
- id: presence-child
|
|
||||||
extends: {id: parent}
|
|
||||||
backend: second
|
|
||||||
api_key_env: null
|
|
||||||
temperature: null
|
|
||||||
max_tokens: 0
|
|
||||||
timeout_seconds: 0
|
|
||||||
extra_params:
|
|
||||||
nested:
|
|
||||||
explicit_false: false
|
|
||||||
- id: map-null
|
|
||||||
backend: second
|
|
||||||
model: map-model
|
|
||||||
extra_params: null
|
|
||||||
- id: map-empty
|
|
||||||
backend: second
|
|
||||||
model: map-model
|
|
||||||
extra_params: {}
|
|
||||||
prompts:
|
|
||||||
- id: execution.prompt
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: hello
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
client := &frameworkSourceCaptureClient{}
|
|
||||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: frameworkFile}}, "", promptkit.WithLLMClient(client))
|
|
||||||
|
|
||||||
for _, profileID := range []string{"backend-child", "presence-child", "map-null", "map-empty"} {
|
|
||||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "execution.prompt", ProfileID: profileID}); err != nil {
|
|
||||||
t.Fatalf("run profile %q: %v", profileID, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(client.requests) != 4 {
|
|
||||||
t.Fatalf("expected four provider-bound requests, got %d", len(client.requests))
|
|
||||||
}
|
|
||||||
|
|
||||||
backendChild := client.requests[0].Target
|
|
||||||
wantBackendParams := map[string]any{"source": "second", "nested": map[string]any{
|
|
||||||
"second_backend": true, "parent_profile": true, "retained": "yes",
|
|
||||||
}}
|
|
||||||
if backendChild.BackendID != "second" || backendChild.Endpoint != "https://second.example/v1" || backendChild.APIKeyEnv != "SECOND_BACKEND_KEY" || backendChild.Model != "child-model" ||
|
|
||||||
backendChild.Temperature != 0.7 || backendChild.TopP != 0.6 || backendChild.MaxTokens != 45 || backendChild.TimeoutSeconds != 30 || backendChild.ServiceTier != "flex" || !reflect.DeepEqual(backendChild.ExtraParams, wantBackendParams) {
|
|
||||||
t.Fatalf("backend change lost backend-derived or explicit inherited values: %+v", backendChild)
|
|
||||||
}
|
|
||||||
|
|
||||||
presenceRequest := client.requests[1]
|
|
||||||
presenceTarget := presenceRequest.Target
|
|
||||||
if presenceTarget.APIKeyEnv != "SECOND_BACKEND_KEY" || presenceTarget.Temperature != 0 || presenceTarget.MaxTokens != 0 || presenceTarget.TimeoutSeconds != 0 ||
|
|
||||||
!presenceRequest.TargetPresence.MaxTokens || !presenceRequest.TargetPresence.TimeoutSeconds || presenceRequest.TargetPresence.Temperature || !presenceRequest.TargetPresence.TopP {
|
|
||||||
t.Fatalf("explicit null/zero presence did not reach the injected client: target=%+v presence=%+v", presenceTarget, presenceRequest.TargetPresence)
|
|
||||||
}
|
|
||||||
wantPresenceParams := map[string]any{"source": "second", "nested": map[string]any{
|
|
||||||
"second_backend": true, "parent_profile": true, "retained": "yes", "explicit_false": false,
|
|
||||||
}}
|
|
||||||
if !reflect.DeepEqual(presenceTarget.ExtraParams, wantPresenceParams) {
|
|
||||||
t.Fatalf("recursive provider-map retention changed: %#v", presenceTarget.ExtraParams)
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(client.requests[2].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
|
|
||||||
t.Fatalf("whole-map null did not fall back to backend defaults: %#v", client.requests[2].Target.ExtraParams)
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(client.requests[3].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
|
|
||||||
t.Fatalf("empty provider map unexpectedly cleared backend defaults: %#v", client.requests[3].Target.ExtraParams)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCLICompletesPartialFrameworkProfileWithRequestOverrides(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
frameworkFile := writeFrameworkSourceFile(t, root, "partial.yml", `profiles:
|
|
||||||
- id: partial
|
|
||||||
prompts:
|
|
||||||
- id: partial.prompt
|
|
||||||
default_profile: partial
|
|
||||||
messages:
|
|
||||||
- role: user
|
|
||||||
content: hello
|
|
||||||
output:
|
|
||||||
format: text
|
|
||||||
validation_mode: none
|
|
||||||
`)
|
|
||||||
provider := newTestLLMServer("completed partial profile", nil)
|
|
||||||
defer provider.Close()
|
|
||||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
|
||||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
|
||||||
"--config", configPath,
|
|
||||||
"--prompt", "partial.prompt",
|
|
||||||
"--llm-base-url", provider.URL + "/v1",
|
|
||||||
"--model", "request-model",
|
|
||||||
})
|
|
||||||
if code != ExitOK || stdout != "completed partial profile" {
|
|
||||||
t.Fatalf("request overrides did not complete the partial profile: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func newFrameworkSourceTestEngine(t *testing.T, selections []appconfig.SourceSelection, schemaDir string, options ...promptkit.Option) *promptkit.Engine {
|
|
||||||
t.Helper()
|
|
||||||
engine, err := newEngine(engineSettings{promptkitSources: selections, schemaDir: schemaDir}, options...)
|
|
||||||
if err != nil {
|
|
||||||
var loadError *configkit.LoadError
|
|
||||||
if errors.As(err, &loadError) {
|
|
||||||
t.Fatalf("construct engine from selected framework sources: %v (%+v)", err, loadError.Diagnostics())
|
|
||||||
}
|
|
||||||
t.Fatalf("construct engine from selected framework sources: %v", err)
|
|
||||||
}
|
|
||||||
return engine
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeFrameworkSourceFile(t *testing.T, directory, name, contents string) string {
|
|
||||||
t.Helper()
|
|
||||||
if err := os.MkdirAll(directory, 0o755); err != nil {
|
|
||||||
t.Fatalf("create framework source directory: %v", err)
|
|
||||||
}
|
|
||||||
path := filepath.Join(directory, name)
|
|
||||||
if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
|
|
||||||
t.Fatalf("write framework source %q: %v", path, err)
|
|
||||||
}
|
|
||||||
return path
|
|
||||||
}
|
|
||||||
|
|
||||||
type frameworkSourceCaptureClient struct {
|
|
||||||
requests []promptkit.GenerateRequest
|
|
||||||
}
|
|
||||||
|
|
||||||
func (client *frameworkSourceCaptureClient) Generate(_ context.Context, request promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
|
||||||
client.requests = append(client.requests, request)
|
|
||||||
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
|
||||||
}
|
|
||||||
@@ -1,212 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
appformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
type promptInspectionConfig struct {
|
|
||||||
configPath, promptID, schemaDir, outputPath string
|
|
||||||
configOverlay string
|
|
||||||
configExplicit bool
|
|
||||||
outputFormat appformat.OutputFormat
|
|
||||||
overrides []configkit.Override
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
}
|
|
||||||
|
|
||||||
type profileInspectionConfig struct {
|
|
||||||
configPath, profileID, schemaDir, outputPath string
|
|
||||||
configOverlay string
|
|
||||||
configExplicit bool
|
|
||||||
outputFormat appformat.OutputFormat
|
|
||||||
overrides []configkit.Override
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
}
|
|
||||||
|
|
||||||
func inspectCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
if len(args) == 0 {
|
|
||||||
fmt.Fprintln(stderr, "inspect parse error: inspection mode is required")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if args[0] == "profile" {
|
|
||||||
return inspectProfileCommand(args[1:], stdout, stderr)
|
|
||||||
}
|
|
||||||
if args[0] == "configuration" {
|
|
||||||
return inspectFrameworkConfigurationCommand(args[1:], stdout, stderr)
|
|
||||||
}
|
|
||||||
if args[0] != "prompt" {
|
|
||||||
fmt.Fprintln(stderr, "inspect parse error: unknown inspection mode")
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
cfg, err := parsePromptInspectionArgs(args[1:])
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
settings, err := resolveAppSettingsForPromptInspection(cfg)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
engine, err := newEngine(settings)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
data, err := appformat.FormatPromptInspection(inspection, cfg.outputFormat)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "output write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func inspectProfileCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
cfg, err := parseProfileInspectionArgs(args)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
settings, err := resolveAppSettingsForProfileInspection(cfg)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "inspect", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
engine, err := newEngine(settings)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
inspection, err := engine.InspectProfile(context.Background(), cfg.profileID)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
data, err := appformat.FormatProfileInspection(inspection, cfg.outputFormat)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeOutput(stdout, cfg.outputPath, data); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "output write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseProfileInspectionArgs(args []string) (*profileInspectionConfig, error) {
|
|
||||||
cfg := &profileInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
|
|
||||||
fs := flag.NewFlagSet("inspect profile", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
var promptkitSources promptKitSourceFlagValue
|
|
||||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
|
||||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
|
||||||
fs.StringVar(&cfg.profileID, "profile", "", "profile ID to inspect")
|
|
||||||
rawFormat := ""
|
|
||||||
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
|
||||||
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cfg.configExplicit = flagWasSet(fs, "config")
|
|
||||||
cfg.overrides = applicationOverrides(fs)
|
|
||||||
if fs.NArg() > 0 {
|
|
||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.profileID) == "" {
|
|
||||||
return nil, errors.New("--profile is required")
|
|
||||||
}
|
|
||||||
format, err := appformat.ParseOutputFormat(rawFormat)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cfg.outputFormat = format
|
|
||||||
if cfg.outputPath != "" {
|
|
||||||
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
|
||||||
}
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func resolveAppSettingsForProfileInspection(cfg *profileInspectionConfig) (engineSettings, error) {
|
|
||||||
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
|
|
||||||
if err != nil {
|
|
||||||
return engineSettings{}, err
|
|
||||||
}
|
|
||||||
cfg.diagnostics = diagnostics
|
|
||||||
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
|
|
||||||
cfg := &promptInspectionConfig{outputFormat: appformat.DefaultOutputFormat}
|
|
||||||
fs := flag.NewFlagSet("inspect prompt", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
var promptkitSources promptKitSourceFlagValue
|
|
||||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
|
||||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
|
||||||
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to inspect")
|
|
||||||
rawFormat := ""
|
|
||||||
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
|
||||||
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cfg.configExplicit = flagWasSet(fs, "config")
|
|
||||||
cfg.overrides = applicationOverrides(fs)
|
|
||||||
if fs.NArg() > 0 {
|
|
||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(cfg.promptID) == "" {
|
|
||||||
return nil, errors.New("--prompt is required")
|
|
||||||
}
|
|
||||||
format, err := appformat.ParseOutputFormat(rawFormat)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cfg.outputFormat = format
|
|
||||||
if cfg.outputPath != "" {
|
|
||||||
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
|
||||||
}
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func resolveAppSettingsForPromptInspection(cfg *promptInspectionConfig) (engineSettings, error) {
|
|
||||||
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
|
|
||||||
if err != nil {
|
|
||||||
return engineSettings{}, err
|
|
||||||
}
|
|
||||||
cfg.diagnostics = diagnostics
|
|
||||||
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
|
|
||||||
}
|
|
||||||
@@ -1,636 +0,0 @@
|
|||||||
package cli
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
httpadapter "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
|
||||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
|
||||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
|
||||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
|
||||||
"go.maximumdirect.net/commandkit"
|
|
||||||
"go.maximumdirect.net/configkit"
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
ExitOK = 0
|
|
||||||
ExitRuntimeError = 1
|
|
||||||
ExitValidationFailed = 2
|
|
||||||
)
|
|
||||||
|
|
||||||
type runConfig struct {
|
|
||||||
configPath string
|
|
||||||
|
|
||||||
promptkitSources []appconfig.SourceSelection
|
|
||||||
promptID string
|
|
||||||
profileID string
|
|
||||||
sessionID string
|
|
||||||
inputRaw listFlag
|
|
||||||
varRaw listFlag
|
|
||||||
outputPath string
|
|
||||||
executionReportPath string
|
|
||||||
llmBaseURL string
|
|
||||||
apiKeyEnv string
|
|
||||||
model string
|
|
||||||
temperature float64
|
|
||||||
maxTokens int
|
|
||||||
topP float64
|
|
||||||
reasoningEffort string
|
|
||||||
schemaDir string
|
|
||||||
timeout time.Duration
|
|
||||||
runTimeout time.Duration
|
|
||||||
transientRetries int
|
|
||||||
configuredTransientRetries *int
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
configOverlay string
|
|
||||||
|
|
||||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
|
||||||
|
|
||||||
llmBaseURLSet bool
|
|
||||||
apiKeyEnvSet bool
|
|
||||||
modelSet bool
|
|
||||||
temperatureSet bool
|
|
||||||
maxTokensSet bool
|
|
||||||
topPSet bool
|
|
||||||
reasoningEffortSet bool
|
|
||||||
timeoutSet bool
|
|
||||||
transientRetriesSet bool
|
|
||||||
}
|
|
||||||
|
|
||||||
type renderConfig struct {
|
|
||||||
runConfig
|
|
||||||
outputFormat renderformat.PreparedRunOutputFormat
|
|
||||||
}
|
|
||||||
|
|
||||||
type serveConfig struct {
|
|
||||||
configPath string
|
|
||||||
|
|
||||||
addr string
|
|
||||||
promptkitSources []appconfig.SourceSelection
|
|
||||||
schemaDir string
|
|
||||||
artifactRoot string
|
|
||||||
maxRequestBytes int64
|
|
||||||
maxArtifactBytes int64
|
|
||||||
maxResponseBytes int64
|
|
||||||
transientRetries *int
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
configOverlay string
|
|
||||||
}
|
|
||||||
|
|
||||||
type commonCommandSettings struct {
|
|
||||||
promptkitSources []appconfig.SourceSelection
|
|
||||||
schemaDir string
|
|
||||||
serverAddr string
|
|
||||||
artifactRoot string
|
|
||||||
maxRequestBytes int64
|
|
||||||
maxArtifactBytes int64
|
|
||||||
maxResponseBytes int64
|
|
||||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
|
||||||
transientRetries *int
|
|
||||||
diagnostics []configkit.Diagnostic
|
|
||||||
}
|
|
||||||
|
|
||||||
type listFlag []string
|
|
||||||
|
|
||||||
func (l *listFlag) String() string {
|
|
||||||
return strings.Join(*l, ",")
|
|
||||||
}
|
|
||||||
|
|
||||||
func (l *listFlag) Set(v string) error {
|
|
||||||
*l = append(*l, v)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func Run(args []string, stdout, stderr io.Writer) int {
|
|
||||||
return Execute(context.Background(), commandkit.Identity{Name: "scriptorium", Version: "dev"}, args, stdout, stderr).Status
|
|
||||||
}
|
|
||||||
|
|
||||||
func runCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
cfg, err := parseRunArgs(args)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "run parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
req, err := buildRunRequestFromConfig(cfg)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "run parse error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
engine, err := newEngine(cfg.engineSettings())
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return executeRun(cfg, req, engine.Run, stdout, stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
func renderCommand(args []string, stdout, stderr io.Writer) int {
|
|
||||||
cfg, err := parseRenderArgs(args)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "render parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
req, err := buildRunRequestFromConfig(&cfg.runConfig)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "render parse error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
engine, err := newEngine(cfg.runConfig.engineSettings())
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
prepared, prepErr := engine.Prepare(context.Background(), req)
|
|
||||||
if prepErr != nil {
|
|
||||||
fmt.Fprintf(stderr, "render error: %v\n", prepErr)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := renderformat.FormatPreparedRun(prepared, cfg.outputFormat)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "render format error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := writeOutput(stdout, cfg.outputPath, out); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "output write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func serveCommand(args []string, stderr io.Writer) int {
|
|
||||||
cfg, err := parseServeArgs(args)
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "serve parse", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
artifactReader, err := httpadapter.NewRestrictedArtifactReader(cfg.artifactRoot, cfg.maxArtifactBytes)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(stderr, "artifact root error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
engine, err := newEngine(cfg.engineSettings(), promptkit.WithArtifactReader(artifactReader))
|
|
||||||
if err != nil {
|
|
||||||
writeApplicationError(stderr, "engine", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
|
||||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
|
|
||||||
h := httpadapter.NewHandlerWithOptions(engine, httpadapter.HandlerOptions{
|
|
||||||
MaxRequestBytes: cfg.maxRequestBytes,
|
|
||||||
MaxResponseBytes: cfg.maxResponseBytes,
|
|
||||||
})
|
|
||||||
srv := &http.Server{
|
|
||||||
Addr: cfg.addr,
|
|
||||||
Handler: h,
|
|
||||||
ReadHeaderTimeout: defaults.HTTPReadHeaderTimeoutDefault,
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintf(stderr, "serving on %s\n", cfg.addr)
|
|
||||||
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
|
||||||
fmt.Fprintf(stderr, "server error: %v\n", err)
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseRunArgs(args []string) (*runConfig, error) {
|
|
||||||
cfg := &runConfig{}
|
|
||||||
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
registerExecutionRequestFlags(fs, cfg)
|
|
||||||
fs.DurationVar(&cfg.runTimeout, "run-timeout", 0, "whole-run execution deadline")
|
|
||||||
fs.StringVar(&cfg.executionReportPath, "execution-report", "", "JSON execution report file path")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if err := finalizeExecutionRequestConfig(fs, cfg); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if cfg.runTimeout < 0 {
|
|
||||||
return nil, errors.New("--run-timeout must be greater than or equal to 0")
|
|
||||||
}
|
|
||||||
if flagWasSet(fs, "execution-report") {
|
|
||||||
if cfg.executionReportPath == "" {
|
|
||||||
return nil, errors.New("--execution-report requires a nonempty path")
|
|
||||||
}
|
|
||||||
cfg.executionReportPath = filepath.Clean(cfg.executionReportPath)
|
|
||||||
if cfg.outputPath != "" {
|
|
||||||
collision, err := outputPathsCollide(cfg.outputPath, cfg.executionReportPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if collision {
|
|
||||||
return nil, errors.New("--out and --execution-report select the same file")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func outputPathsCollide(first, second string) (bool, error) {
|
|
||||||
firstAbs, err := filepath.Abs(first)
|
|
||||||
if err != nil {
|
|
||||||
return false, err
|
|
||||||
}
|
|
||||||
secondAbs, err := filepath.Abs(second)
|
|
||||||
if err != nil {
|
|
||||||
return false, err
|
|
||||||
}
|
|
||||||
if firstAbs == secondAbs {
|
|
||||||
return true, nil
|
|
||||||
}
|
|
||||||
firstInfo, firstErr := os.Stat(firstAbs)
|
|
||||||
secondInfo, secondErr := os.Stat(secondAbs)
|
|
||||||
return firstErr == nil && secondErr == nil && os.SameFile(firstInfo, secondInfo), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseRenderArgs(args []string) (*renderConfig, error) {
|
|
||||||
cfg := &renderConfig{
|
|
||||||
outputFormat: renderformat.DefaultPreparedRunOutputFormat,
|
|
||||||
}
|
|
||||||
fs := flag.NewFlagSet("render", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
registerExecutionRequestFlags(fs, &cfg.runConfig)
|
|
||||||
|
|
||||||
var rawFormat string
|
|
||||||
fs.StringVar(&rawFormat, "format", "", "render output format (text|json)")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if err := finalizeExecutionRequestConfig(fs, &cfg.runConfig); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if flagWasSet(fs, "format") {
|
|
||||||
format, err := renderformat.ParsePreparedRunOutputFormat(rawFormat)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cfg.outputFormat = format
|
|
||||||
} else {
|
|
||||||
cfg.outputFormat = cfg.defaultRenderFormat
|
|
||||||
}
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseServeArgs(args []string) (*serveConfig, error) {
|
|
||||||
cfg := &serveConfig{}
|
|
||||||
fs := flag.NewFlagSet("serve", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
|
||||||
fs.StringVar(&cfg.addr, "addr", "", "HTTP listen address")
|
|
||||||
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
|
|
||||||
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
|
|
||||||
fs.Int64Var(&cfg.maxArtifactBytes, "max-artifact-bytes", 0, "maximum HTTP file artifact bytes; 0 disables the limit")
|
|
||||||
fs.Int64Var(&cfg.maxResponseBytes, "max-response-bytes", 0, "maximum HTTP response body bytes; 0 disables the limit")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if fs.NArg() > 0 {
|
|
||||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
|
||||||
}
|
|
||||||
|
|
||||||
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg.promptkitSources = settings.promptkitSources
|
|
||||||
cfg.schemaDir = settings.schemaDir
|
|
||||||
cfg.addr = settings.serverAddr
|
|
||||||
cfg.artifactRoot = settings.artifactRoot
|
|
||||||
cfg.maxRequestBytes = settings.maxRequestBytes
|
|
||||||
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
|
||||||
cfg.maxResponseBytes = settings.maxResponseBytes
|
|
||||||
cfg.transientRetries = settings.transientRetries
|
|
||||||
cfg.diagnostics = settings.diagnostics
|
|
||||||
|
|
||||||
return cfg, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
|
|
||||||
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
|
|
||||||
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
|
|
||||||
fs.StringVar(&cfg.sessionID, "session-id", "", "optional session ID")
|
|
||||||
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
|
|
||||||
fs.Var(&cfg.varRaw, "var", "variable mapping(s): name=value (repeatable, comma-separated)")
|
|
||||||
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
|
||||||
fs.StringVar(&cfg.llmBaseURL, "llm-base-url", "", "OpenAI-compatible base URL including /v1")
|
|
||||||
fs.StringVar(&cfg.apiKeyEnv, "api-key-env", "", "environment variable name containing API key")
|
|
||||||
fs.StringVar(&cfg.model, "model", "", "model name")
|
|
||||||
fs.Float64Var(&cfg.temperature, "temperature", 0, "optional temperature override")
|
|
||||||
fs.IntVar(&cfg.maxTokens, "max-tokens", 0, "optional max tokens override")
|
|
||||||
fs.Float64Var(&cfg.topP, "top-p", 0, "optional top_p override")
|
|
||||||
fs.StringVar(&cfg.reasoningEffort, "reasoning-effort", "", "optional reasoning effort override")
|
|
||||||
fs.DurationVar(&cfg.timeout, "timeout", 0, "LLM request timeout")
|
|
||||||
fs.IntVar(&cfg.transientRetries, "transient-retries", 0, "additional transient generation attempts")
|
|
||||||
fs.StringVar(&cfg.promptID, "prompt-id", "", "deprecated alias for --prompt")
|
|
||||||
fs.StringVar(&cfg.profileID, "profile-id", "", "deprecated alias for --profile")
|
|
||||||
}
|
|
||||||
|
|
||||||
func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
|
||||||
if fs.NArg() > 0 {
|
|
||||||
return fmt.Errorf("unexpected positional args: %v", fs.Args())
|
|
||||||
}
|
|
||||||
|
|
||||||
settings, err := resolveCommonSettings(fs, cfg.configPath, cfg.configOverlay)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg.promptkitSources = settings.promptkitSources
|
|
||||||
cfg.schemaDir = settings.schemaDir
|
|
||||||
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
|
||||||
cfg.configuredTransientRetries = settings.transientRetries
|
|
||||||
cfg.diagnostics = settings.diagnostics
|
|
||||||
|
|
||||||
if strings.TrimSpace(cfg.promptID) == "" {
|
|
||||||
return errors.New("--prompt is required")
|
|
||||||
}
|
|
||||||
if cfg.outputPath != "" {
|
|
||||||
cfg.outputPath = filepath.Clean(cfg.outputPath)
|
|
||||||
}
|
|
||||||
cfg.llmBaseURLSet = flagWasSet(fs, "llm-base-url")
|
|
||||||
cfg.apiKeyEnvSet = flagWasSet(fs, "api-key-env")
|
|
||||||
cfg.modelSet = flagWasSet(fs, "model")
|
|
||||||
cfg.temperatureSet = flagWasSet(fs, "temperature")
|
|
||||||
cfg.maxTokensSet = flagWasSet(fs, "max-tokens")
|
|
||||||
cfg.topPSet = flagWasSet(fs, "top-p")
|
|
||||||
cfg.reasoningEffortSet = flagWasSet(fs, "reasoning-effort")
|
|
||||||
cfg.timeoutSet = flagWasSet(fs, "timeout")
|
|
||||||
cfg.transientRetriesSet = flagWasSet(fs, "transient-retries")
|
|
||||||
if cfg.transientRetriesSet && cfg.transientRetries < 0 {
|
|
||||||
return errors.New("--transient-retries must be greater than or equal to 0")
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func registerConfigSelectionFlags(fs *flag.FlagSet, target, overlay *string) {
|
|
||||||
fs.StringVar(
|
|
||||||
target,
|
|
||||||
"config",
|
|
||||||
"",
|
|
||||||
fmt.Sprintf(
|
|
||||||
"application config path (default search: %s, then %s)",
|
|
||||||
appconfig.DefaultConfigPathLocal,
|
|
||||||
appconfig.DefaultConfigPath,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
fs.StringVar(overlay, "config-overlay", "", "selected application configuration overlay")
|
|
||||||
}
|
|
||||||
|
|
||||||
func resolveCommonSettings(fs *flag.FlagSet, configPath, overlay string) (commonCommandSettings, error) {
|
|
||||||
settings, diagnostics, err := loadApplicationSettings(configPath, overlay, flagWasSet(fs, "config"), applicationOverrides(fs))
|
|
||||||
if err != nil {
|
|
||||||
return commonCommandSettings{}, err
|
|
||||||
}
|
|
||||||
return commonCommandSettings{
|
|
||||||
promptkitSources: settings.PromptKitSources,
|
|
||||||
schemaDir: settings.SchemaDir,
|
|
||||||
serverAddr: settings.ServerAddr,
|
|
||||||
artifactRoot: settings.ArtifactRoot,
|
|
||||||
maxRequestBytes: settings.MaxRequestBytes,
|
|
||||||
maxArtifactBytes: settings.MaxArtifactBytes,
|
|
||||||
maxResponseBytes: settings.MaxResponseBytes,
|
|
||||||
defaultRenderFormat: renderformat.DefaultPreparedRunOutputFormat,
|
|
||||||
transientRetries: settings.TransientRetries,
|
|
||||||
diagnostics: diagnostics,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
type engineSettings struct {
|
|
||||||
promptkitSources []appconfig.SourceSelection
|
|
||||||
schemaDir string
|
|
||||||
transientRetries *int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c runConfig) engineSettings() engineSettings {
|
|
||||||
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.configuredTransientRetries}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c serveConfig) engineSettings() engineSettings {
|
|
||||||
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.transientRetries}
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
|
||||||
var inputMappings map[string]string
|
|
||||||
var err error
|
|
||||||
if len(cfg.inputRaw) > 0 {
|
|
||||||
inputMappings, err = parseMappings(cfg.inputRaw, false)
|
|
||||||
if err != nil {
|
|
||||||
return promptkit.RunRequest{}, fmt.Errorf("input parse error: %w", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
varMappings := map[string]string{}
|
|
||||||
if len(cfg.varRaw) > 0 {
|
|
||||||
varMappings, err = parseMappings(cfg.varRaw, false)
|
|
||||||
if err != nil {
|
|
||||||
return promptkit.RunRequest{}, fmt.Errorf("var parse error: %w", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var inputs map[string]promptkit.ArtifactRef
|
|
||||||
if len(inputMappings) > 0 {
|
|
||||||
inputs = make(map[string]promptkit.ArtifactRef, len(inputMappings))
|
|
||||||
for name, path := range inputMappings {
|
|
||||||
inputs[name] = promptkit.File(path)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var modelOverride *promptkit.ExecutionTargetOverride
|
|
||||||
if cfg.llmBaseURLSet || cfg.modelSet || cfg.temperatureSet || cfg.maxTokensSet || cfg.topPSet || cfg.reasoningEffortSet || cfg.apiKeyEnvSet || cfg.timeoutSet {
|
|
||||||
modelOverride = &promptkit.ExecutionTargetOverride{
|
|
||||||
Endpoint: cfg.llmBaseURL,
|
|
||||||
Model: cfg.model,
|
|
||||||
APIKeyEnv: cfg.apiKeyEnv,
|
|
||||||
}
|
|
||||||
if cfg.temperatureSet {
|
|
||||||
modelOverride.Temperature = &cfg.temperature
|
|
||||||
}
|
|
||||||
if cfg.maxTokensSet {
|
|
||||||
modelOverride.MaxTokens = &cfg.maxTokens
|
|
||||||
}
|
|
||||||
if cfg.topPSet {
|
|
||||||
modelOverride.TopP = &cfg.topP
|
|
||||||
}
|
|
||||||
if cfg.reasoningEffortSet {
|
|
||||||
modelOverride.ReasoningEffort = &cfg.reasoningEffort
|
|
||||||
}
|
|
||||||
if cfg.timeoutSet {
|
|
||||||
timeoutSeconds := int(cfg.timeout.Seconds())
|
|
||||||
modelOverride.TimeoutSeconds = &timeoutSeconds
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
var transientRetries *int
|
|
||||||
if cfg.transientRetriesSet {
|
|
||||||
transientRetries = &cfg.transientRetries
|
|
||||||
}
|
|
||||||
return promptkit.RunRequest{
|
|
||||||
PromptID: cfg.promptID,
|
|
||||||
ProfileID: cfg.profileID,
|
|
||||||
SessionID: cfg.sessionID,
|
|
||||||
Inputs: inputs,
|
|
||||||
Vars: varMappings,
|
|
||||||
Execution: modelOverride,
|
|
||||||
TransientRetries: transientRetries,
|
|
||||||
}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseMappings(raw []string, allowEmptyValue bool) (map[string]string, error) {
|
|
||||||
out := make(map[string]string)
|
|
||||||
for _, entry := range raw {
|
|
||||||
for _, piece := range strings.Split(entry, ",") {
|
|
||||||
piece = strings.TrimSpace(piece)
|
|
||||||
if piece == "" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
key, value, err := parseMapping(piece)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
if !allowEmptyValue && strings.TrimSpace(value) == "" {
|
|
||||||
return nil, fmt.Errorf("mapping %q has empty value", piece)
|
|
||||||
}
|
|
||||||
out[key] = value
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(out) == 0 {
|
|
||||||
return nil, errors.New("no valid mappings provided")
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func parseMapping(value string) (string, string, error) {
|
|
||||||
idx := strings.IndexRune(value, '=')
|
|
||||||
if idx <= 0 {
|
|
||||||
return "", "", fmt.Errorf("invalid mapping %q, expected name=value", value)
|
|
||||||
}
|
|
||||||
key := strings.TrimSpace(value[:idx])
|
|
||||||
val := strings.TrimSpace(value[idx+1:])
|
|
||||||
if key == "" {
|
|
||||||
return "", "", fmt.Errorf("invalid mapping %q, empty name", value)
|
|
||||||
}
|
|
||||||
return key, val, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func flagWasSet(fs *flag.FlagSet, name string) bool {
|
|
||||||
set := false
|
|
||||||
fs.Visit(func(f *flag.Flag) {
|
|
||||||
if f.Name == name {
|
|
||||||
set = true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
return set
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeOutput(stdout io.Writer, outputPath string, body []byte) error {
|
|
||||||
if outputPath == "" {
|
|
||||||
_, err := stdout.Write(body)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return os.WriteFile(outputPath, body, 0644)
|
|
||||||
}
|
|
||||||
|
|
||||||
func determineExitCode(runErr error, result *promptkit.RunResult) int {
|
|
||||||
if runErr != nil {
|
|
||||||
return ExitRuntimeError
|
|
||||||
}
|
|
||||||
if result != nil && result.Validation.Status == promptkit.ValidationFailed {
|
|
||||||
return ExitValidationFailed
|
|
||||||
}
|
|
||||||
return ExitOK
|
|
||||||
}
|
|
||||||
|
|
||||||
func printSummary(stderr io.Writer, res *promptkit.RunResult) {
|
|
||||||
if res == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
fmt.Fprintf(stderr, "prompt=%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
|
|
||||||
res.PromptID,
|
|
||||||
res.SelectedProfileID,
|
|
||||||
res.ModelName,
|
|
||||||
res.Validation.Status,
|
|
||||||
res.Validation.Mode,
|
|
||||||
len(res.Validation.Errors),
|
|
||||||
res.RenderedPromptHash,
|
|
||||||
len(res.InputHashes),
|
|
||||||
res.Usage.PromptTokens,
|
|
||||||
res.Usage.CompletionTokens,
|
|
||||||
res.Usage.TotalTokens,
|
|
||||||
)
|
|
||||||
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
|
|
||||||
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
|
|
||||||
}
|
|
||||||
if res.SelectedBackendID != "" {
|
|
||||||
fmt.Fprintf(stderr, " backend=%s", res.SelectedBackendID)
|
|
||||||
}
|
|
||||||
fmt.Fprintln(stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
func runErrorMessage(err error) string {
|
|
||||||
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
|
||||||
return "run error: model backend capacity is exhausted"
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("run error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
func printUsage(w io.Writer) {
|
|
||||||
fmt.Fprintln(w, "usage: scriptorium <run|render|serve|inspect|config> ...")
|
|
||||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]")
|
|
||||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]")
|
|
||||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--addr %s] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
|
||||||
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
|
|
||||||
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
|
|
||||||
fmt.Fprintln(w, " inspect configuration: scriptorium inspect configuration <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--path JSON_POINTER (required for explain)]")
|
|
||||||
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [application overrides] [--path /pointer for explain]")
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -153,7 +153,7 @@ import (
|
|||||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
_ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
||||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
_ "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
_ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
_ "gitea.maximumdirect.net/eric/scriptorium/internal/report"
|
||||||
)
|
)
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {
|
if err := os.WriteFile(sourcePath, []byte(source), 0o644); err != nil {
|
||||||
|
|||||||
@@ -1,179 +0,0 @@
|
|||||||
package format
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"strconv"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
type PromptInspection struct {
|
|
||||||
PromptID string `json:"prompt_id"`
|
|
||||||
PromptHash string `json:"prompt_hash"`
|
|
||||||
DefaultProfileID *string `json:"default_profile_id,omitempty"`
|
|
||||||
Inputs *[]PromptInspectionInput `json:"inputs,omitempty"`
|
|
||||||
OutputContract OutputContract `json:"output_contract"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type PromptInspectionInput struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
Required *bool `json:"required,omitempty"`
|
|
||||||
ContentType *string `json:"content_type,omitempty"`
|
|
||||||
Description *string `json:"description,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type OutputContract struct {
|
|
||||||
Format string `json:"format"`
|
|
||||||
ValidationMode string `json:"validation_mode"`
|
|
||||||
SchemaPath *string `json:"schema_path,omitempty"`
|
|
||||||
RepairAttempts *int `json:"repair_attempts,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ProfileInspection struct {
|
|
||||||
ProfileID string `json:"profile_id"`
|
|
||||||
ProfileModelParams ProfileModelParams `json:"profile_model_params"`
|
|
||||||
APIKeyRequired *bool `json:"api_key_required,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type ProfileModelParams struct {
|
|
||||||
BackendID *string `json:"backend_id,omitempty"`
|
|
||||||
Endpoint *string `json:"endpoint,omitempty"`
|
|
||||||
Model *string `json:"model,omitempty"`
|
|
||||||
Temperature *float64 `json:"temperature,omitempty"`
|
|
||||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
|
||||||
TopP *float64 `json:"top_p,omitempty"`
|
|
||||||
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
|
||||||
ServiceTier *string `json:"service_tier,omitempty"`
|
|
||||||
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
|
|
||||||
APIKeyEnv *string `json:"api_key_env,omitempty"`
|
|
||||||
ExtraParams *map[string]any `json:"extra_params,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat OutputFormat) ([]byte, error) {
|
|
||||||
if value == nil {
|
|
||||||
return nil, errors.New("prompt inspection is nil")
|
|
||||||
}
|
|
||||||
dto := PromptInspection{
|
|
||||||
PromptID: value.PromptID, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
|
|
||||||
OutputContract: OutputContract{Format: string(value.OutputContract.Format), ValidationMode: string(value.OutputContract.ValidationMode), SchemaPath: value.OutputContract.SchemaPath, RepairAttempts: value.OutputContract.RepairAttempts},
|
|
||||||
}
|
|
||||||
if value.Inputs != nil {
|
|
||||||
inputs := make([]PromptInspectionInput, len(*value.Inputs))
|
|
||||||
for i, input := range *value.Inputs {
|
|
||||||
inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
|
|
||||||
}
|
|
||||||
dto.Inputs = &inputs
|
|
||||||
}
|
|
||||||
switch outputFormat {
|
|
||||||
case OutputFormatJSON:
|
|
||||||
data, err := json.MarshalIndent(dto, "", " ")
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return append(data, '\n'), nil
|
|
||||||
case OutputFormatText:
|
|
||||||
var b bytes.Buffer
|
|
||||||
fmt.Fprintf(&b, "prompt_id: %s\nprompt_hash: %s\n", dto.PromptID, dto.PromptHash)
|
|
||||||
if dto.DefaultProfileID != nil {
|
|
||||||
fmt.Fprintf(&b, "default_profile_id: %s\n", *dto.DefaultProfileID)
|
|
||||||
}
|
|
||||||
if dto.Inputs != nil {
|
|
||||||
if len(*dto.Inputs) == 0 {
|
|
||||||
fmt.Fprintln(&b, "inputs: []")
|
|
||||||
} else {
|
|
||||||
fmt.Fprintln(&b, "inputs:")
|
|
||||||
for _, input := range *dto.Inputs {
|
|
||||||
fmt.Fprintf(&b, " - name: %s\n", input.Name)
|
|
||||||
if input.Required != nil {
|
|
||||||
fmt.Fprintf(&b, " required: %t\n", *input.Required)
|
|
||||||
}
|
|
||||||
if input.ContentType != nil {
|
|
||||||
fmt.Fprintf(&b, " content_type: %s\n", *input.ContentType)
|
|
||||||
}
|
|
||||||
if input.Description != nil {
|
|
||||||
fmt.Fprintf(&b, " description: %s\n", strconv.Quote(*input.Description))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode)
|
|
||||||
if dto.OutputContract.SchemaPath != nil {
|
|
||||||
fmt.Fprintf(&b, " schema_path: %s\n", *dto.OutputContract.SchemaPath)
|
|
||||||
}
|
|
||||||
if dto.OutputContract.RepairAttempts != nil {
|
|
||||||
fmt.Fprintf(&b, " repair_attempts: %d\n", *dto.OutputContract.RepairAttempts)
|
|
||||||
}
|
|
||||||
return b.Bytes(), nil
|
|
||||||
default:
|
|
||||||
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat OutputFormat) ([]byte, error) {
|
|
||||||
if value == nil {
|
|
||||||
return nil, errors.New("profile inspection is nil")
|
|
||||||
}
|
|
||||||
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, ProfileModelParams: ProfileModelParams{
|
|
||||||
BackendID: value.Backend, Endpoint: value.Endpoint, Model: value.Model, Temperature: value.Temperature,
|
|
||||||
MaxTokens: value.MaxTokens, TopP: value.TopP, TimeoutSeconds: value.TimeoutSeconds, ServiceTier: value.ServiceTier,
|
|
||||||
ReasoningEffort: value.ReasoningEffort, APIKeyEnv: value.APIKeyEnv, ExtraParams: value.ExtraParams,
|
|
||||||
}}
|
|
||||||
switch outputFormat {
|
|
||||||
case OutputFormatJSON:
|
|
||||||
data, err := json.MarshalIndent(dto, "", " ")
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return append(data, '\n'), nil
|
|
||||||
case OutputFormatText:
|
|
||||||
var b bytes.Buffer
|
|
||||||
fmt.Fprintf(&b, "profile_id: %s\nprofile_model_params:\n", dto.ProfileID)
|
|
||||||
p := dto.ProfileModelParams
|
|
||||||
if p.BackendID != nil {
|
|
||||||
fmt.Fprintf(&b, " backend_id: %s\n", *p.BackendID)
|
|
||||||
}
|
|
||||||
if p.Endpoint != nil {
|
|
||||||
fmt.Fprintf(&b, " endpoint: %s\n", *p.Endpoint)
|
|
||||||
}
|
|
||||||
if p.Model != nil {
|
|
||||||
fmt.Fprintf(&b, " model: %s\n", *p.Model)
|
|
||||||
}
|
|
||||||
if p.Temperature != nil {
|
|
||||||
fmt.Fprintf(&b, " temperature: %g\n", *p.Temperature)
|
|
||||||
}
|
|
||||||
if p.MaxTokens != nil {
|
|
||||||
fmt.Fprintf(&b, " max_tokens: %d\n", *p.MaxTokens)
|
|
||||||
}
|
|
||||||
if p.TopP != nil {
|
|
||||||
fmt.Fprintf(&b, " top_p: %g\n", *p.TopP)
|
|
||||||
}
|
|
||||||
if p.TimeoutSeconds != nil {
|
|
||||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", *p.TimeoutSeconds)
|
|
||||||
}
|
|
||||||
if p.ServiceTier != nil {
|
|
||||||
fmt.Fprintf(&b, " service_tier: %s\n", *p.ServiceTier)
|
|
||||||
}
|
|
||||||
if p.ReasoningEffort != nil {
|
|
||||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", *p.ReasoningEffort)
|
|
||||||
}
|
|
||||||
if p.APIKeyEnv != nil {
|
|
||||||
fmt.Fprintf(&b, " api_key_env: %s\n", *p.APIKeyEnv)
|
|
||||||
}
|
|
||||||
if p.ExtraParams != nil {
|
|
||||||
paramsJSON, err := json.Marshal(p.ExtraParams)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
fmt.Fprintf(&b, " extra_params: %s\n", paramsJSON)
|
|
||||||
}
|
|
||||||
if dto.APIKeyRequired != nil {
|
|
||||||
fmt.Fprintf(&b, "api_key_required: %t\n", *dto.APIKeyRequired)
|
|
||||||
}
|
|
||||||
return b.Bytes(), nil
|
|
||||||
default:
|
|
||||||
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
package format
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func ptr[T any](value T) *T { return &value }
|
|
||||||
|
|
||||||
func TestPromptInspectionOptionalFieldsAndDisclosure(t *testing.T) {
|
|
||||||
inputs := []promptkit.PromptInputInspection{{Name: "first", Required: ptr(false), ContentType: ptr(""), Description: ptr("")}, {Name: "second"}}
|
|
||||||
value := &promptkit.PromptInspection{PromptID: "support/p", PromptHash: "hash", DefaultProfileID: ptr(""), Inputs: &inputs,
|
|
||||||
Messages: []promptkit.PromptMessageInspection{{Role: "user", Content: "private message"}}, Description: ptr("private description"), SessionID: ptr("private session"),
|
|
||||||
OutputContract: promptkit.PromptOutputInspection{Format: "text", ValidationMode: promptkit.ValidationNone, SchemaPath: ptr(""), RepairAttempts: ptr(0)}}
|
|
||||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
|
||||||
output, err := FormatPromptInspection(value, format)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
s := string(output)
|
|
||||||
for _, forbidden := range []string{"private message", "private description", "private session", "prompt_version"} {
|
|
||||||
if strings.Contains(s, forbidden) {
|
|
||||||
t.Fatalf("inspection exposed %q: %s", forbidden, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.Index(s, "first") > strings.Index(s, "second") {
|
|
||||||
t.Fatalf("input order changed: %s", s)
|
|
||||||
}
|
|
||||||
if !strings.Contains(s, "default_profile_id") || !strings.Contains(s, "repair_attempts") || !strings.Contains(s, "required") {
|
|
||||||
t.Fatalf("present empty values missing: %s", s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
value.DefaultProfileID, value.Inputs = nil, nil
|
|
||||||
value.OutputContract.SchemaPath, value.OutputContract.RepairAttempts = nil, nil
|
|
||||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
|
||||||
output, err := FormatPromptInspection(value, format)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
for _, absent := range []string{"default_profile_id", "inputs", "schema_path", "repair_attempts"} {
|
|
||||||
if strings.Contains(string(output), absent) {
|
|
||||||
t.Fatalf("absent %s rendered: %s", absent, output)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
empty := []promptkit.PromptInputInspection{}
|
|
||||||
value.Inputs = &empty
|
|
||||||
output, err := FormatPromptInspection(value, OutputFormatJSON)
|
|
||||||
if err != nil || !strings.Contains(string(output), `"inputs": []`) {
|
|
||||||
t.Fatalf("empty inputs lost: %s %v", output, err)
|
|
||||||
}
|
|
||||||
if _, err := FormatPromptInspection(nil, OutputFormatText); err == nil {
|
|
||||||
t.Fatal("nil prompt accepted")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestProfileInspectionPreservesOnlyPresentValues(t *testing.T) {
|
|
||||||
params := map[string]any{}
|
|
||||||
value := &promptkit.ProfileInspection{ProfileID: "support/fast", Backend: ptr("backend"), Model: ptr(""), Temperature: ptr(0.0), MaxTokens: ptr(0), APIKeyRequired: ptr(false), ExtraParams: ¶ms}
|
|
||||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
|
||||||
output, err := FormatProfileInspection(value, format)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
s := string(output)
|
|
||||||
for _, key := range []string{"profile_model_params", "backend_id", "model", "temperature", "max_tokens", "api_key_required", "extra_params"} {
|
|
||||||
if !strings.Contains(s, key) {
|
|
||||||
t.Fatalf("present %s missing: %s", key, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for _, key := range []string{"effective_model_params", "endpoint", "api_key_env", "top_p"} {
|
|
||||||
if strings.Contains(s, key) {
|
|
||||||
t.Fatalf("absent %s appeared: %s", key, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
output, err := FormatProfileInspection(value, OutputFormatJSON)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
var decoded map[string]any
|
|
||||||
if err := json.Unmarshal(output, &decoded); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
p := decoded["profile_model_params"].(map[string]any)
|
|
||||||
if p["backend_id"] != "backend" || p["model"] != "" || p["temperature"] != float64(0) || decoded["api_key_required"] != false || len(p["extra_params"].(map[string]any)) != 0 {
|
|
||||||
t.Fatalf("wrong values: %#v", decoded)
|
|
||||||
}
|
|
||||||
if _, err := FormatProfileInspection(nil, OutputFormatText); err == nil {
|
|
||||||
t.Fatal("nil profile accepted")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,210 +0,0 @@
|
|||||||
// Package format formats already-prepared public data for adapters.
|
|
||||||
package format
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"sort"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
|
|
||||||
|
|
||||||
// OutputFormat selects an application-owned textual or JSON representation.
|
|
||||||
type OutputFormat string
|
|
||||||
|
|
||||||
const (
|
|
||||||
OutputFormatText OutputFormat = "text"
|
|
||||||
OutputFormatJSON OutputFormat = "json"
|
|
||||||
|
|
||||||
DefaultOutputFormat OutputFormat = OutputFormatText
|
|
||||||
)
|
|
||||||
|
|
||||||
// PreparedRunOutputFormat is the output format for prepared render data.
|
|
||||||
type PreparedRunOutputFormat = OutputFormat
|
|
||||||
|
|
||||||
const (
|
|
||||||
PreparedRunFormatText = OutputFormatText
|
|
||||||
PreparedRunFormatJSON = OutputFormatJSON
|
|
||||||
|
|
||||||
DefaultPreparedRunOutputFormat = DefaultOutputFormat
|
|
||||||
)
|
|
||||||
|
|
||||||
// PreparedRunFormatter serializes a prepared run without performing use case work.
|
|
||||||
type PreparedRunFormatter interface {
|
|
||||||
Format(prepared *promptkit.PreparedRun) ([]byte, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ParsePreparedRunOutputFormat parses a format name.
|
|
||||||
func ParsePreparedRunOutputFormat(raw string) (PreparedRunOutputFormat, error) {
|
|
||||||
return ParseOutputFormat(raw)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ParseOutputFormat parses the shared inspection and prepared-run format names.
|
|
||||||
func ParseOutputFormat(raw string) (OutputFormat, error) {
|
|
||||||
switch OutputFormat(strings.ToLower(strings.TrimSpace(raw))) {
|
|
||||||
case "":
|
|
||||||
return DefaultOutputFormat, nil
|
|
||||||
case OutputFormatText:
|
|
||||||
return OutputFormatText, nil
|
|
||||||
case OutputFormatJSON:
|
|
||||||
return OutputFormatJSON, nil
|
|
||||||
default:
|
|
||||||
return "", fmt.Errorf("%w: %q (supported: %s, %s)", ErrUnknownPreparedRunFormat, raw, OutputFormatText, OutputFormatJSON)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// FormatterForPreparedRun returns a formatter strategy for the chosen format.
|
|
||||||
func FormatterForPreparedRun(outputFormat PreparedRunOutputFormat) (PreparedRunFormatter, error) {
|
|
||||||
switch outputFormat {
|
|
||||||
case PreparedRunFormatText:
|
|
||||||
return textPreparedRunFormatter{}, nil
|
|
||||||
case PreparedRunFormatJSON:
|
|
||||||
return jsonPreparedRunFormatter{}, nil
|
|
||||||
default:
|
|
||||||
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// FormatPreparedRun formats a prepared run using the selected format.
|
|
||||||
func FormatPreparedRun(prepared *promptkit.PreparedRun, outputFormat PreparedRunOutputFormat) ([]byte, error) {
|
|
||||||
formatter, err := FormatterForPreparedRun(outputFormat)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return formatter.Format(prepared)
|
|
||||||
}
|
|
||||||
|
|
||||||
// FormatPreparedRunByName parses a format name and formats a prepared run.
|
|
||||||
func FormatPreparedRunByName(prepared *promptkit.PreparedRun, rawFormat string) ([]byte, error) {
|
|
||||||
outputFormat, err := ParsePreparedRunOutputFormat(rawFormat)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return FormatPreparedRun(prepared, outputFormat)
|
|
||||||
}
|
|
||||||
|
|
||||||
type jsonPreparedRunFormatter struct{}
|
|
||||||
|
|
||||||
func (jsonPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
|
|
||||||
if prepared == nil {
|
|
||||||
return nil, errors.New("prepared run is nil")
|
|
||||||
}
|
|
||||||
return json.MarshalIndent(prepared, "", " ")
|
|
||||||
}
|
|
||||||
|
|
||||||
type textPreparedRunFormatter struct{}
|
|
||||||
|
|
||||||
func (textPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte, error) {
|
|
||||||
if prepared == nil {
|
|
||||||
return nil, errors.New("prepared run is nil")
|
|
||||||
}
|
|
||||||
|
|
||||||
var b bytes.Buffer
|
|
||||||
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
|
|
||||||
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
|
|
||||||
if prepared.SelectedBackendID != "" {
|
|
||||||
fmt.Fprintf(&b, "selected_backend_id: %s\n", prepared.SelectedBackendID)
|
|
||||||
}
|
|
||||||
if prepared.PromptHash != "" {
|
|
||||||
fmt.Fprintf(&b, "prompt_hash: %s\n", prepared.PromptHash)
|
|
||||||
}
|
|
||||||
if prepared.SessionID != "" {
|
|
||||||
fmt.Fprintf(&b, "session_id: %s\n", prepared.SessionID)
|
|
||||||
}
|
|
||||||
fmt.Fprintf(&b, "rendered_prompt_hash: %s\n", prepared.RenderedPromptHash)
|
|
||||||
|
|
||||||
target := prepared.EffectiveModelParams
|
|
||||||
fmt.Fprintln(&b, "effective_model_params:")
|
|
||||||
fmt.Fprintf(&b, " endpoint: %s\n", target.Endpoint)
|
|
||||||
fmt.Fprintf(&b, " model: %s\n", target.Model)
|
|
||||||
fmt.Fprintf(&b, " temperature: %g\n", target.Temperature)
|
|
||||||
fmt.Fprintf(&b, " max_tokens: %d\n", target.MaxTokens)
|
|
||||||
fmt.Fprintf(&b, " top_p: %g\n", target.TopP)
|
|
||||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", target.TimeoutSeconds)
|
|
||||||
if target.ServiceTier != "" {
|
|
||||||
fmt.Fprintf(&b, " service_tier: %s\n", target.ServiceTier)
|
|
||||||
}
|
|
||||||
if target.ReasoningEffort != "" {
|
|
||||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", target.ReasoningEffort)
|
|
||||||
}
|
|
||||||
if target.APIKeyEnv != "" {
|
|
||||||
fmt.Fprintf(&b, " api_key_env: %s\n", target.APIKeyEnv)
|
|
||||||
}
|
|
||||||
if len(target.ExtraParams) > 0 {
|
|
||||||
fmt.Fprintln(&b, " extra_params:")
|
|
||||||
keys := make([]string, 0, len(target.ExtraParams))
|
|
||||||
for k := range target.ExtraParams {
|
|
||||||
keys = append(keys, k)
|
|
||||||
}
|
|
||||||
sort.Strings(keys)
|
|
||||||
for _, k := range keys {
|
|
||||||
renderedValue, err := formatExtraParamTextValue(target.ExtraParams[k])
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to format extra_params.%s: %w", k, err)
|
|
||||||
}
|
|
||||||
fmt.Fprintf(&b, " %s: %s\n", k, renderedValue)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintln(&b, "input_hashes:")
|
|
||||||
inputKeys := make([]string, 0, len(prepared.InputHashes))
|
|
||||||
for k := range prepared.InputHashes {
|
|
||||||
inputKeys = append(inputKeys, k)
|
|
||||||
}
|
|
||||||
sort.Strings(inputKeys)
|
|
||||||
for _, k := range inputKeys {
|
|
||||||
fmt.Fprintf(&b, " %s: %s\n", k, prepared.InputHashes[k])
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintln(&b, "messages:")
|
|
||||||
roleOrder := make([]string, 0)
|
|
||||||
byRole := make(map[string][]promptkit.RenderedMessage)
|
|
||||||
for _, msg := range prepared.Messages {
|
|
||||||
if _, exists := byRole[msg.Role]; !exists {
|
|
||||||
roleOrder = append(roleOrder, msg.Role)
|
|
||||||
}
|
|
||||||
byRole[msg.Role] = append(byRole[msg.Role], msg)
|
|
||||||
}
|
|
||||||
for _, role := range roleOrder {
|
|
||||||
fmt.Fprintf(&b, " %s:\n", role)
|
|
||||||
messages := byRole[role]
|
|
||||||
for i, msg := range messages {
|
|
||||||
fmt.Fprintf(&b, " - message: %d\n", i+1)
|
|
||||||
if msg.CacheControl != nil {
|
|
||||||
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
|
|
||||||
if msg.CacheControl.TTL != "" {
|
|
||||||
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
|
|
||||||
}
|
|
||||||
fmt.Fprintln(&b)
|
|
||||||
}
|
|
||||||
fmt.Fprintln(&b, " content: |")
|
|
||||||
content := msg.Content
|
|
||||||
if content == "" {
|
|
||||||
fmt.Fprintln(&b, " ")
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
for _, line := range strings.Split(content, "\n") {
|
|
||||||
fmt.Fprintf(&b, " %s\n", line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return b.Bytes(), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func formatExtraParamTextValue(value any) (string, error) {
|
|
||||||
if s, ok := value.(string); ok {
|
|
||||||
return s, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
b, err := json.Marshal(value)
|
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return string(b), nil
|
|
||||||
}
|
|
||||||
@@ -1,375 +0,0 @@
|
|||||||
package format
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"go.maximumdirect.net/promptkit"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
s := string(out)
|
|
||||||
if strings.Contains(s, "prompt_version") {
|
|
||||||
t.Fatalf("unexpected prompt version in text output: %s", s)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, want := range []string{
|
|
||||||
"prompt: prompt.id",
|
|
||||||
"selected_profile_id: local-fast",
|
|
||||||
"selected_backend_id: local",
|
|
||||||
"endpoint: http://llm/v1",
|
|
||||||
"model: gpt-test",
|
|
||||||
"temperature: 0.4",
|
|
||||||
"max_tokens: 256",
|
|
||||||
"top_p: 0.8",
|
|
||||||
"timeout_seconds: 45",
|
|
||||||
"service_tier: priority",
|
|
||||||
"reasoning_effort: medium",
|
|
||||||
"api_key_env: SCRIPTORIUM_API_KEY",
|
|
||||||
"prompt_hash: prompt-hash",
|
|
||||||
"rendered_prompt_hash: rendered-hash",
|
|
||||||
"glossary: hash-glossary",
|
|
||||||
"transcript: hash-transcript",
|
|
||||||
"messages:",
|
|
||||||
" system:",
|
|
||||||
" user:",
|
|
||||||
"System guidance.",
|
|
||||||
"Summarize the transcript.",
|
|
||||||
"Include key entities.",
|
|
||||||
"Second user message.",
|
|
||||||
} {
|
|
||||||
if !strings.Contains(s, want) {
|
|
||||||
t.Fatalf("expected text output to include %q, got:\n%s", want, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterRendersExtraParamsDeterministically(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
|
||||||
"z_string": "enabled",
|
|
||||||
"b_number": 42,
|
|
||||||
"a_object": map[string]any{
|
|
||||||
"nested": "value",
|
|
||||||
"count": 2,
|
|
||||||
},
|
|
||||||
"c_array": []any{"first", 3, false},
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
s := string(out)
|
|
||||||
|
|
||||||
want := strings.Join([]string{
|
|
||||||
" extra_params:",
|
|
||||||
" a_object: {\"count\":2,\"nested\":\"value\"}",
|
|
||||||
" b_number: 42",
|
|
||||||
" c_array: [\"first\",3,false]",
|
|
||||||
" z_string: enabled",
|
|
||||||
}, "\n")
|
|
||||||
if !strings.Contains(s, want) {
|
|
||||||
t.Fatalf("expected deterministic extra_params block %q, got:\n%s", want, s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
|
||||||
const secret = "super-secret-api-key"
|
|
||||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(out), secret) {
|
|
||||||
t.Fatalf("text output should not include resolved api key value: %s", out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
|
||||||
const directKey = "direct-format-key"
|
|
||||||
// PreparedRun intentionally has no field for direct API keys.
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(out), directKey) {
|
|
||||||
t.Fatalf("text output should not include direct api key value: %s", out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.Messages = []promptkit.RenderedMessage{
|
|
||||||
{
|
|
||||||
Role: "system",
|
|
||||||
Content: "System guidance.",
|
|
||||||
CacheControl: &promptkit.CacheControl{
|
|
||||||
Type: promptkit.CacheControlEphemeral,
|
|
||||||
TTL: "1h",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{Role: "user", Content: "Summarize the transcript."},
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
s := string(out)
|
|
||||||
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
|
|
||||||
t.Fatalf("expected system message cache control before content, got:\n%s", s)
|
|
||||||
}
|
|
||||||
if strings.Count(s, "cache_control:") != 1 {
|
|
||||||
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterIncludesSessionIDWhenPresent(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.SessionID = "session-123"
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if !strings.Contains(string(out), "session_id: session-123\n") {
|
|
||||||
t.Fatalf("expected session_id in text output, got:\n%s", out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.Messages = []promptkit.RenderedMessage{
|
|
||||||
{
|
|
||||||
Role: "system",
|
|
||||||
Content: "System guidance.",
|
|
||||||
CacheControl: &promptkit.CacheControl{
|
|
||||||
Type: promptkit.CacheControlEphemeral,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
s := string(out)
|
|
||||||
if !strings.Contains(s, " cache_control: ephemeral\n") {
|
|
||||||
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
|
|
||||||
}
|
|
||||||
if strings.Contains(s, "ttl=") {
|
|
||||||
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.SessionID = "session-123"
|
|
||||||
prepared.EffectiveModelParams.ExtraParams = map[string]any{
|
|
||||||
"number": 42,
|
|
||||||
"nested": map[string]any{
|
|
||||||
"enabled": true,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var decoded map[string]any
|
|
||||||
if err := json.Unmarshal(out, &decoded); err != nil {
|
|
||||||
t.Fatalf("expected valid json output, got %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if decoded["prompt_id"] != "prompt.id" {
|
|
||||||
t.Fatalf("expected prompt_id in json output, got %#v", decoded["prompt_id"])
|
|
||||||
}
|
|
||||||
if _, ok := decoded["prompt_version"]; ok {
|
|
||||||
t.Fatalf("unexpected prompt_version in json output: %#v", decoded)
|
|
||||||
}
|
|
||||||
if decoded["selected_profile_id"] != "local-fast" {
|
|
||||||
t.Fatalf("expected selected_profile_id in json output, got %#v", decoded["selected_profile_id"])
|
|
||||||
}
|
|
||||||
if decoded["rendered_prompt_hash"] != "rendered-hash" {
|
|
||||||
t.Fatalf("expected rendered_prompt_hash in json output, got %#v", decoded["rendered_prompt_hash"])
|
|
||||||
}
|
|
||||||
if decoded["session_id"] != "session-123" {
|
|
||||||
t.Fatalf("expected session_id in json output, got %#v", decoded["session_id"])
|
|
||||||
}
|
|
||||||
modelParams, ok := decoded["effective_model_params"].(map[string]any)
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("expected effective_model_params in json output, got %#v", decoded)
|
|
||||||
}
|
|
||||||
extraParams, ok := modelParams["extra_params"].(map[string]any)
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("expected extra_params in json output, got %#v", modelParams["extra_params"])
|
|
||||||
}
|
|
||||||
if extraParams["number"] != float64(42) {
|
|
||||||
t.Fatalf("unexpected numeric extra param in json output: %#v", extraParams["number"])
|
|
||||||
}
|
|
||||||
nested, ok := extraParams["nested"].(map[string]any)
|
|
||||||
if !ok || nested["enabled"] != true {
|
|
||||||
t.Fatalf("unexpected nested extra param in json output: %#v", extraParams["nested"])
|
|
||||||
}
|
|
||||||
if _, ok := decoded["input_hashes"]; !ok {
|
|
||||||
t.Fatalf("expected input_hashes in json output, got %#v", decoded)
|
|
||||||
}
|
|
||||||
if _, ok := decoded["messages"]; !ok {
|
|
||||||
t.Fatalf("expected messages in json output, got %#v", decoded)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
prepared.Messages = []promptkit.RenderedMessage{
|
|
||||||
{
|
|
||||||
Role: "system",
|
|
||||||
Content: "System guidance.",
|
|
||||||
CacheControl: &promptkit.CacheControl{
|
|
||||||
Type: promptkit.CacheControlEphemeral,
|
|
||||||
TTL: "1h",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{Role: "user", Content: "Summarize the transcript."},
|
|
||||||
}
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var decoded struct {
|
|
||||||
Messages []map[string]any `json:"messages"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(out, &decoded); err != nil {
|
|
||||||
t.Fatalf("expected valid json output, got %v", err)
|
|
||||||
}
|
|
||||||
if len(decoded.Messages) != 2 {
|
|
||||||
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
|
|
||||||
}
|
|
||||||
|
|
||||||
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
|
|
||||||
}
|
|
||||||
if cacheControl["type"] != string(promptkit.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
|
|
||||||
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
|
|
||||||
}
|
|
||||||
if _, ok := decoded.Messages[1]["cache_control"]; ok {
|
|
||||||
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
|
|
||||||
const secret = "super-secret-api-key"
|
|
||||||
t.Setenv("SCRIPTORIUM_API_KEY", secret)
|
|
||||||
|
|
||||||
out, err := FormatPreparedRun(samplePreparedRun(), PreparedRunFormatJSON)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(out), secret) {
|
|
||||||
t.Fatalf("json output should not include resolved api key value: %s", out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestJSONFormatterDoesNotIncludeDirectAPIKeyValue(t *testing.T) {
|
|
||||||
const directKey = "direct-format-key"
|
|
||||||
// PreparedRun intentionally has no field for direct API keys.
|
|
||||||
prepared := samplePreparedRun()
|
|
||||||
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(string(out), directKey) {
|
|
||||||
t.Fatalf("json output should not include direct api key value: %s", out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestParsePreparedRunOutputFormatRecognizesSupportedNames(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
input string
|
|
||||||
want PreparedRunOutputFormat
|
|
||||||
}{
|
|
||||||
{name: "default empty", input: "", want: DefaultPreparedRunOutputFormat},
|
|
||||||
{name: "text", input: "text", want: PreparedRunFormatText},
|
|
||||||
{name: "json", input: "json", want: PreparedRunFormatJSON},
|
|
||||||
{name: "trim and case", input: " JSON ", want: PreparedRunFormatJSON},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
got, err := ParsePreparedRunOutputFormat(tc.input)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("expected no error, got %v", err)
|
|
||||||
}
|
|
||||||
if got != tc.want {
|
|
||||||
t.Fatalf("expected %q, got %q", tc.want, got)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestParsePreparedRunOutputFormatUnknownFails(t *testing.T) {
|
|
||||||
_, err := ParsePreparedRunOutputFormat("yaml")
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected error for unknown format")
|
|
||||||
}
|
|
||||||
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
|
|
||||||
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFormatPreparedRunByNameUnknownFailsClearly(t *testing.T) {
|
|
||||||
_, err := FormatPreparedRunByName(samplePreparedRun(), "yaml")
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected unknown format error")
|
|
||||||
}
|
|
||||||
if !errors.Is(err, ErrUnknownPreparedRunFormat) {
|
|
||||||
t.Fatalf("expected ErrUnknownPreparedRunFormat, got %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func samplePreparedRun() *promptkit.PreparedRun {
|
|
||||||
return &promptkit.PreparedRun{
|
|
||||||
PromptID: "prompt.id",
|
|
||||||
PromptHash: "prompt-hash",
|
|
||||||
SelectedProfileID: "local-fast",
|
|
||||||
SelectedBackendID: "local",
|
|
||||||
EffectiveModelParams: promptkit.ExecutionTarget{
|
|
||||||
BackendID: "local",
|
|
||||||
Endpoint: "http://llm/v1",
|
|
||||||
Model: "gpt-test",
|
|
||||||
Temperature: 0.4,
|
|
||||||
MaxTokens: 256,
|
|
||||||
TopP: 0.8,
|
|
||||||
TimeoutSeconds: 45,
|
|
||||||
ServiceTier: "priority",
|
|
||||||
ReasoningEffort: "medium",
|
|
||||||
APIKeyEnv: "SCRIPTORIUM_API_KEY",
|
|
||||||
},
|
|
||||||
InputHashes: map[string]string{
|
|
||||||
"transcript": "hash-transcript",
|
|
||||||
"glossary": "hash-glossary",
|
|
||||||
},
|
|
||||||
RenderedPromptHash: "rendered-hash",
|
|
||||||
Messages: []promptkit.RenderedMessage{
|
|
||||||
{Role: "system", Content: "System guidance."},
|
|
||||||
{Role: "user", Content: "Summarize the transcript.\nInclude key entities."},
|
|
||||||
{Role: "user", Content: "Second user message."},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Reference in New Issue
Block a user