Compare commits
85 Commits
9849c6bda2
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 69d2ba7038 | |||
| fbe7dd56e3 | |||
| 99b0ebfc03 | |||
| bcb1a76e34 | |||
| 6ed4ac624a | |||
| c7b9c30c43 | |||
| 5f32a4f2ad | |||
| 8078c6ecc7 | |||
| 66a0954b52 | |||
| 5eeb09d1ec | |||
| f31dec61ef | |||
| 293374d7a7 | |||
| b3a0a73416 | |||
| 3f5e441756 | |||
| 23108bd2f0 | |||
| d97e2b8e30 | |||
| 3105f4d62f | |||
| 6562d63a02 | |||
| dd1915735f | |||
| a2f3588639 | |||
| 3f481ea432 | |||
| ac858be2a2 | |||
| cf50650e55 | |||
| 7afe6ba333 | |||
| 16197ae9dd | |||
| c5e54edaa5 | |||
| 78bcf86619 | |||
| 5d372dd37d | |||
| 9bd1629a9e | |||
| 50c65adac4 | |||
| a2205db152 | |||
| 27db6b6f47 | |||
| f858d57322 | |||
| de7c866662 | |||
| 77f168c660 | |||
| e7a8b0db04 | |||
| ae5344a77d | |||
| 247b67312e | |||
| ca7c7faf5c | |||
| a0134ea49a | |||
| 95a94249f1 | |||
| 87d7240f31 | |||
| 1b1d967fc7 | |||
| 932fd6da25 | |||
| 65bd2da51a | |||
| eec4fa2d91 | |||
| 2c3d8e4c8b | |||
| 79dfe0a3ef | |||
| 03b479b4d7 | |||
| 5c780a4cc0 | |||
| d79e23739f | |||
| 49461b18e4 | |||
| 222f4056e6 | |||
| a5d8394313 | |||
| ba24b7a879 | |||
| 723fee9c46 | |||
| 0cd95f6e8e | |||
| c233d03371 | |||
| 907fc51236 | |||
| cd0811a49e | |||
| 1c8494ec20 | |||
| 58cb02f973 | |||
| beaaea2f1a | |||
| b913ef0d07 | |||
| 411a67b241 | |||
| 2ba2ce1b91 | |||
| c2e2b74f83 | |||
| 9ea9d0706d | |||
| 5c302f18ac | |||
| 54c8e9011e | |||
| 2027b80311 | |||
| 0df2aad3a8 | |||
| 0d5ecfa842 | |||
| c8e2b972f3 | |||
| b1bb55a9a0 | |||
| 82b0f231b6 | |||
| 15e7c1aed9 | |||
| bb5006517b | |||
| bcf2ddf246 | |||
| ddcba9c639 | |||
| 70f7e96d02 | |||
| ea964707f7 | |||
| bbb473a3d9 | |||
| e1ce526114 | |||
| 8b6c12f912 |
@@ -3,11 +3,13 @@ when:
|
||||
|
||||
steps:
|
||||
- name: build-release-assets
|
||||
image: golang:1.25
|
||||
image: golang:1.27.4
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
|
||||
export GOTOOLCHAIN=local
|
||||
|
||||
version="$CI_COMMIT_TAG"
|
||||
dist="dist"
|
||||
pkg="gitea.maximumdirect.net/eric/scriptorium/cmd/scriptorium"
|
||||
|
||||
13
README.md
13
README.md
@@ -6,7 +6,7 @@ OpenAI-compatible model endpoints, and returns generated output with validation
|
||||
metadata.
|
||||
|
||||
The application uses
|
||||
[Promptkit v0.11.2](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/)
|
||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/)
|
||||
for prompt, profile, schema, preparation, generation, and validation behavior.
|
||||
Scriptorium owns executable configuration, CLI and HTTP mapping, process
|
||||
behavior, output presentation, and HTTP artifact-containment policy.
|
||||
@@ -33,20 +33,25 @@ a model. For complete invocation and output behavior, see the
|
||||
- [CLI reference](docs/cli.md)
|
||||
- [Configuration reference](docs/config.md)
|
||||
- [Application configuration inspection](docs/cli.md#application-configuration-inspection)
|
||||
- [Framework configuration inspection](docs/cli.md#framework-configuration-inspection)
|
||||
- [HTTP API reference](docs/api.md)
|
||||
- [Operations guide](docs/operations.md)
|
||||
- [Consumer integration overview](docs/consumers/api.md)
|
||||
- [Migration from the former Go package](docs/consumers/migrating-to-promptkit.md)
|
||||
- [Upgrade Promptkit source selectors and definitions](docs/consumers/upgrading-promptkit.md)
|
||||
- [Subprocess integration](docs/integrations/subprocess.md)
|
||||
- [Architecture policy](docs/policy/architecture.md)
|
||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md)
|
||||
- [Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md)
|
||||
- [Promptkit framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
- [Promptkit Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
|
||||
|
||||
## Examples
|
||||
|
||||
- [Minimal configuration](examples/config.yml) and
|
||||
[complete configuration](examples/config.full.yml), plus a
|
||||
[composed import/overlay example](examples/config.composed.yml)
|
||||
[composed import/overlay example](examples/config.composed.yml), a
|
||||
[collection composition example](examples/config.collections.yml), a
|
||||
[collection checkout discovery example](examples/config.discovery.yml), and a
|
||||
[backend inheritance example](examples/config.inheritance.yml)
|
||||
- [Prompt definitions](examples/prompts/), [execution profiles](examples/profiles/),
|
||||
[schemas](examples/schemas/), and [synthetic input fixtures](examples/fixtures/)
|
||||
- [Render script](examples/render-markdown-summary.sh)
|
||||
|
||||
71
docs/adr/0005-configkit-boundary.md
Normal file
71
docs/adr/0005-configkit-boundary.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# ADR 0005: Adopt ConfigKit's Configuration Boundary
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Date
|
||||
|
||||
2026-10-04.
|
||||
|
||||
## Context
|
||||
|
||||
Scriptorium uses ConfigKit to resolve application configuration. Adopting its
|
||||
standard semantics across the application suite gives operators a consistent
|
||||
configuration model and keeps configuration processing under a single owner.
|
||||
|
||||
The application supplies its schema and domain rules through ConfigKit's public
|
||||
definition API. ConfigKit returns the resolved configuration and inspection
|
||||
information for use by Scriptorium.
|
||||
|
||||
## Decision
|
||||
|
||||
ConfigKit owns configuration loading and resolution semantics, including
|
||||
decoding, imports, composition, precedence, overlays, inheritance, and
|
||||
inspection. Scriptorium adopts those semantics through ConfigKit's supported
|
||||
public APIs.
|
||||
|
||||
Scriptorium owns its application schema, defaults, value normalization, and
|
||||
domain validation. It supplies these rules through ConfigKit's definition
|
||||
callbacks so they participate in the library's resolution and validation flow.
|
||||
Scriptorium consumes the final, fully resolved configuration returned by that
|
||||
flow.
|
||||
|
||||
Scriptorium also owns application-facing choices: default discovery locations,
|
||||
source and option selection, and the mapping of CLI inputs to ConfigKit options
|
||||
and overrides. ConfigKit determines how the selected inputs are processed.
|
||||
|
||||
Scriptorium may expose ConfigKit's inspection APIs through operator interfaces
|
||||
when demand warrants them. ConfigKit owns the underlying inspection semantics;
|
||||
Scriptorium owns their application-facing presentation.
|
||||
|
||||
Resolution establishes configuration values. Runtime credentials, artifacts,
|
||||
and downstream resources retain their respective preparation and execution
|
||||
lifecycle. Promptkit owns its framework schema and domain rules when using
|
||||
ConfigKit, consistent with [ADR 0004](0004-definition-boundary.md).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Application-Owned Configuration Processing
|
||||
|
||||
Scriptorium could implement loading and composition locally. This offers
|
||||
application-specific flexibility, but duplicates ConfigKit's responsibilities
|
||||
and creates recurring maintenance and semantic drift across applications.
|
||||
|
||||
### ConfigKit-Owned Application Choices
|
||||
|
||||
Scriptorium could delegate discovery conventions and domain defaults to the
|
||||
library. This reduces local decisions, but places application-specific behavior
|
||||
in a shared configuration mechanism. The selected boundary keeps those choices
|
||||
with the application while using the library's processing semantics.
|
||||
|
||||
## Consequences
|
||||
|
||||
Configuration changes follow the ownership boundary: processing capabilities
|
||||
belong in ConfigKit, and application rules use its supported extension points.
|
||||
ConfigKit upgrades are evaluated for their effects on Scriptorium's selected
|
||||
inputs, domain callbacks, and operator interfaces.
|
||||
|
||||
The [architecture policy](../policy/architecture.md) owns current architectural
|
||||
constraints, and the [configuration contract](../config.md) owns observable
|
||||
application behavior. This ADR records the rationale for the ownership boundary.
|
||||
53
docs/adr/0006-adopt-commandkit-native-operator-interfaces.md
Normal file
53
docs/adr/0006-adopt-commandkit-native-operator-interfaces.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# ADR 0006: Adopt CommandKit's Native Operator Interfaces
|
||||
|
||||
## Status
|
||||
|
||||
Accepted. Complements [ADR 0004](0004-definition-boundary.md) and
|
||||
[ADR 0005](0005-configkit-boundary.md). Both remain accepted and are not
|
||||
superseded.
|
||||
|
||||
## Date
|
||||
|
||||
2026-11-19.
|
||||
|
||||
## Context
|
||||
|
||||
Scriptorium is adopting CommandKit's native operator interfaces within the
|
||||
PromptKit and ConfigKit boundaries established by ADRs 0004 and 0005.
|
||||
|
||||
## Decision
|
||||
|
||||
Scriptorium adopts CommandKit's native operator interfaces.
|
||||
|
||||
ConfigKit owns configuration policy and everything in the configuration
|
||||
lifecycle before the final, fully resolved configuration handoff. Application
|
||||
schema and domain callbacks operate through ConfigKit's supported extension
|
||||
points within that lifecycle.
|
||||
|
||||
PromptKit owns prompt definitions, preparation, execution, validation, and
|
||||
inspection semantics. Its configuration is resolved through ConfigKit, with
|
||||
PromptKit supplying its domain schema and rules through ConfigKit's supported
|
||||
extension points.
|
||||
|
||||
Scriptorium owns the domain. After the handoff, it interprets the resolved
|
||||
configuration values and uses them to perform useful work that satisfies the
|
||||
user's request. It does not revise what constitutes effective configuration.
|
||||
It supplies requests through PromptKit's supported APIs and uses the returned
|
||||
results without redefining their meaning.
|
||||
|
||||
CommandKit's `ckconfig` and `ckprompt` integrations expose upstream ConfigKit
|
||||
and PromptKit features. Scriptorium neutrally makes those features available
|
||||
through the native interfaces, preserving upstream semantics.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
Retaining custom configuration and prompt inspection would duplicate upstream
|
||||
behavior and create another owner for its semantics. Use the native integrations
|
||||
instead.
|
||||
|
||||
## Consequences
|
||||
|
||||
Standard inspection exposes the upstream result without constructing a different
|
||||
effective document, reconciling separate explanation trees, or reinterpreting
|
||||
upstream paths. The [adoption roadmap](../roadmap/commandkit-adoption.md) tracks
|
||||
implementation.
|
||||
38
docs/api.md
38
docs/api.md
@@ -43,7 +43,6 @@ copyable shape. At the HTTP adapter boundary, the smallest valid shape is:
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `prompt_id` | yes | Non-blank prompt ID. |
|
||||
| `prompt_version` | no | Prompt version filter. |
|
||||
| `profile_id` | no | Execution-profile ID; otherwise the prompt must set `default_profile`. |
|
||||
| `session_id` | no | Optional direct, non-secret session identifier. |
|
||||
| `inputs` | no | Optional object mapping input names to references. Promptkit decides whether the selected definition needs them. |
|
||||
@@ -52,7 +51,7 @@ copyable shape. At the HTTP adapter boundary, the smallest valid shape is:
|
||||
| `include_raw_output` | no | Include `raw_model_output` when true. |
|
||||
| `include_execution_report` | no | Include a terminal `execution` report when true and Promptkit supplies a terminal observation. Omitted, `null`, or false leaves reporting off. |
|
||||
| `transient_retries` | no | Nonnegative additional transient attempt count for this run. Omitted or `null` inherits the configured engine value and then Promptkit's default; `0` disables additional attempts. |
|
||||
| `run_timeout_seconds` | no | Optional signed 64-bit whole-second limit for engine execution. Omitted, `null`, or `0` adds no deadline; positive values shorten the caller context deadline. |
|
||||
| `run_timeout_seconds` | no | Optional signed 64-bit whole-second engine operation budget. Omitted, `null`, or `0` adds no deadline. |
|
||||
|
||||
An input reference has a required `type` of `file` or `inline`. A `file`
|
||||
reference requires `uri`; an `inline` reference requires `body`.
|
||||
@@ -68,16 +67,17 @@ The optional `model` object accepts `endpoint`, `model`, `temperature`,
|
||||
`max_tokens`, `top_p`, `timeout_seconds`, `service_tier`,
|
||||
`reasoning_effort`, `api_key_env`, and `extra_params`. Numeric ranges and
|
||||
framework credential semantics are defined by the
|
||||
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md).
|
||||
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
Explicit zero values for the numeric fields are overrides; zero
|
||||
`timeout_seconds` disables the per-attempt generation deadline only, retaining the
|
||||
request context and configured transport cap. The timeout layers are defined in
|
||||
the [Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||
the [PromptKit v0.23.3 outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||
Each built-in transient retry gets a fresh generation timeout; the caller's
|
||||
request context remains the outer cancellation boundary.
|
||||
|
||||
Raw API-key values are not accepted. `api_key` and any other unknown model
|
||||
field cause `400 invalid_json`.
|
||||
Raw API-key values are not accepted. `api_key`, the removed `prompt_version`,
|
||||
and any other unknown field cause `400 invalid_json`. Prompt and profile IDs
|
||||
are passed unchanged, including qualified collection IDs.
|
||||
|
||||
`model.reasoning_effort` is an optional JSON string with three states: omission
|
||||
inherits the selected profile, a non-empty string replaces its value, and an
|
||||
@@ -116,14 +116,14 @@ validation contract. The response contains:
|
||||
optional `uri`;
|
||||
- `validation`: `status`, `mode`, `repair_attempts`, `is_valid`, plus
|
||||
optional `errors` and `schema_path`;
|
||||
- `metadata`: run, prompt, rendered-prompt, profile, optional backend identity, model, input-hash, usage,
|
||||
timing, validation, and repair-attempt metadata; and
|
||||
- `metadata`: run, prompt, rendered-prompt, profile, optional backend identity,
|
||||
model parameters, input hashes, usage, and timing; and
|
||||
- optional `raw_model_output` when requested.
|
||||
- optional `execution` when requested and a terminal observation exists.
|
||||
|
||||
`metadata.model_params` has `endpoint`, `model`, `temperature`,
|
||||
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
|
||||
`backend_id`, `service_tier`, `reasoning_effort`, `api_key_env`, and `extra_params`.
|
||||
`service_tier`, `reasoning_effort`, `api_key_env`, and `extra_params`.
|
||||
`metadata.usage` always includes `prompt_tokens`, `completion_tokens`,
|
||||
`total_tokens`, `cached_tokens`, and `cache_write_tokens`; unavailable
|
||||
cache usage is reported as zero.
|
||||
@@ -132,10 +132,11 @@ When Promptkit resolves a direct or definition-rendered session ID,
|
||||
`metadata.session_id` contains that effective result value. It is omitted when
|
||||
no effective session ID exists.
|
||||
|
||||
`metadata.selected_backend_id` and `metadata.model_params.backend_id` report
|
||||
the corresponding Promptkit result fields independently when present. Both are
|
||||
omitted for an endpoint-only profile; Scriptorium does not infer backend
|
||||
identity from an endpoint.
|
||||
`metadata.selected_backend_id` is omitted for an endpoint-only profile;
|
||||
Scriptorium does not infer backend identity from an endpoint. The redundant
|
||||
`prompt_version`, `model_name`, `endpoint`, `validation_mode`,
|
||||
`validation_status`, `repair_attempts_used`, and `model_params.backend_id`
|
||||
metadata members are not returned: use `model_params` and `validation`.
|
||||
|
||||
A validation failure has `validation.status: "failed"`, `is_valid: false`,
|
||||
and any available diagnostic errors, while still returning the artifact and
|
||||
@@ -150,7 +151,7 @@ call was dispatched; if there is no terminal event, `execution` is omitted.
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
|
||||
| `run_id`, `prompt_id`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
|
||||
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
|
||||
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
|
||||
| `status_code` | Provider status, omitted when unavailable. |
|
||||
@@ -201,7 +202,7 @@ exceed it, the response is the compact `413 response_too_large` error without
|
||||
| `400` | `prompt_render_failed` | Prompt template rendering failed. |
|
||||
| `400` | `api_key_env_missing` | The selected credential environment variable is unset or empty. |
|
||||
| `404` | `not_found` | Route does not exist. |
|
||||
| `404` | `prompt_not_found` | Prompt ID or version does not exist. |
|
||||
| `404` | `prompt_not_found` | Prompt ID does not exist. |
|
||||
| `404` | `profile_not_found` | Profile ID does not exist. |
|
||||
| `405` | `method_not_allowed` | The route does not accept the method. |
|
||||
| `408` | `request_canceled` | The caller context was canceled during a failed run. |
|
||||
@@ -212,10 +213,11 @@ exceed it, the response is the compact `413 response_too_large` error without
|
||||
| `500` | `internal_error` | Unclassified server failure. |
|
||||
| `502` | `llm_failed` | Outbound model request failed. |
|
||||
| `503` | `capacity_exceeded` | The selected model backend has no admission capacity. No retry timing is supplied. |
|
||||
| `504` | `deadline_exceeded` | The execution context's deadline expired during a failed run. |
|
||||
| `504` | `deadline_exceeded` | PromptKit reports an operation timeout, or the parent deadline expired and the failed run wraps that deadline. |
|
||||
|
||||
These context outcomes require the execution context to have ended. A provider
|
||||
attempt timeout while it remains live retains `502 llm_failed`. A successful
|
||||
An operation timeout is checked before parent cancellation; a failed run
|
||||
wrapping cancellation returns `408` when the parent is canceled. A provider
|
||||
attempt timeout while the parent remains live retains `502 llm_failed`. A successful
|
||||
run remains successful even if cancellation races with response formatting;
|
||||
delivery after a client disconnect cannot be guaranteed.
|
||||
|
||||
|
||||
175
docs/cli.md
175
docs/cli.md
@@ -24,20 +24,24 @@ go run ./cmd/scriptorium render \
|
||||
- `scriptorium render`: prepare a prompt and write prepared-run output.
|
||||
- `scriptorium serve`: start the HTTP server.
|
||||
- `scriptorium inspect prompt`: inspect one prompt definition without model execution.
|
||||
- `scriptorium inspect profile`: inspect one effective profile without model execution.
|
||||
- `scriptorium inspect profile`: inspect one resolved profile without model execution.
|
||||
- `scriptorium inspect configuration`: validate or inspect Promptkit's captured framework configuration.
|
||||
- `scriptorium config validate`: validate application configuration without building an engine.
|
||||
- `scriptorium config show`: show the resolved application configuration.
|
||||
- `scriptorium config sources`: show the ordered configuration sources.
|
||||
- `scriptorium config explain`: explain one effective or historical configuration path.
|
||||
|
||||
All commands accept `--config <path>` and `--config-overlay <name>` and reject
|
||||
positional arguments. `--config-overlay` selects one application configuration
|
||||
overlay; it is separate from `--profile`, which selects a Promptkit execution
|
||||
profile. See [configuration composition](config.md#explicit-imports-and-selected-overlays)
|
||||
for name, root, and no-base behavior. `run`,
|
||||
`render`, `serve`, and `inspect prompt` require an effective `prompt_dir`;
|
||||
`inspect profile` does not. Supply the directory through the configuration
|
||||
contract or a supported `--prompt-dir` flag. Explicit application flags,
|
||||
All command families accept `--config <path>` and `--config-overlay <name>`;
|
||||
positional arguments are rejected. `--config-overlay` selects one application
|
||||
configuration overlay; it is separate from `--profile`, which selects a
|
||||
Promptkit execution profile. See [configuration composition](config.md#explicit-imports-and-selected-overlays)
|
||||
for name, root, and absent-main behavior. `run`, `render`, `serve`, `inspect prompt`,
|
||||
`inspect profile`, and `inspect configuration` accept repeated
|
||||
`--promptkit-source <path>` flags and `--schema-dir <dir>` where applicable.
|
||||
They replace the configured source sequence in flag order. `--prompt-dir` and
|
||||
`--profile-dir` are no longer supported; use `promptkit_sources` in application
|
||||
configuration or the repeated source flags. Application `config` commands
|
||||
remain independent of framework source loading. Explicit application flags,
|
||||
including empty values, override file settings. Configuration load diagnostics
|
||||
go to stderr and exit with status 1; successful load warnings are printed once
|
||||
before command work.
|
||||
@@ -60,10 +64,8 @@ Optional flags:
|
||||
| --- | --- |
|
||||
| `--config <path>` | Application configuration file. |
|
||||
| `--config-overlay <name>` | Selected application configuration overlay. |
|
||||
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
||||
| `--profile-dir <dir>` | Custom profile-directory override. |
|
||||
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
|
||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
||||
| `--prompt-version <version>` | Optional prompt-definition version selector. |
|
||||
| `--profile <id>` | Execution-profile override. |
|
||||
| `--session-id <id>` | Optional direct session identifier. |
|
||||
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
|
||||
@@ -90,16 +92,15 @@ only; the caller context and configured transport cap remain active. CLI
|
||||
durations are converted to whole seconds by truncation toward zero, so any
|
||||
duration whose absolute value is below one second becomes an explicit
|
||||
zero-second override. The timeout layers are defined in the
|
||||
[Promptkit outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||
[Promptkit outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md#timeout-and-cancellation).
|
||||
With the built-in client, each transient retry gets a fresh generation timeout;
|
||||
this flag does not limit the complete run.
|
||||
`--run-timeout` keeps subsecond precision and starts immediately before engine
|
||||
execution. It bounds preparation, capacity waits, retry waits, generation, and
|
||||
repairs where they honor context. Parsing, engine construction, and output
|
||||
writes are outside that interval. A failed run whose execution context reaches
|
||||
this deadline exits 1 with a safe deadline diagnostic; a completed invalid
|
||||
output still exits 2. It does not promise immediate interruption of synchronous
|
||||
work.
|
||||
writes are outside that interval. A failed run identified by PromptKit as an
|
||||
operation timeout exits 1 with a safe deadline diagnostic; a completed invalid
|
||||
output still exits 2. Synchronous work may not stop immediately.
|
||||
Omitting `--transient-retries` inherits the application setting or Promptkit's
|
||||
current default of two additional attempts. The flag overrides only this
|
||||
request; it does not alter the configured engine. Structural repair generations
|
||||
@@ -107,10 +108,10 @@ have separate transient retry budgets.
|
||||
|
||||
There is no raw API-key flag. Use `--api-key-env`.
|
||||
|
||||
`--prompt-version` is passed directly to Promptkit. When it is omitted, the
|
||||
selected prompt ID must have exactly one available version. `--input` is
|
||||
optional at the CLI boundary: Promptkit decides whether the selected definition
|
||||
requires declared inputs or template-referenced values.
|
||||
Prompt and profile IDs are passed unchanged, including collection-qualified
|
||||
IDs. The removed `--prompt-version` flag is rejected. `--input` is optional
|
||||
at the CLI boundary: PromptKit decides whether the selected definition requires
|
||||
declared inputs or template-referenced values.
|
||||
|
||||
`--session-id` supplies a direct, non-secret session identifier. A nonblank
|
||||
value replaces a definition-rendered session ID; an omitted or blank value lets
|
||||
@@ -157,7 +158,7 @@ scriptorium render [flags]
|
||||
```
|
||||
|
||||
`--prompt <id>` is required. The following optional flags are supported:
|
||||
`--config`, `--config-overlay`, `--prompt-dir`, `--profile-dir`, `--schema-dir`, `--prompt-version`, `--profile`,
|
||||
`--config`, `--config-overlay`, repeatable `--promptkit-source`, `--schema-dir`, `--profile`,
|
||||
`--input`, `--var`, `--out`, `--llm-base-url`,
|
||||
`--model`, `--api-key-env`, `--temperature`, `--max-tokens`, `--top-p`,
|
||||
`--reasoning-effort`, `--session-id`, `--timeout`, `--transient-retries`, and `--format text|json`. Their meanings match the corresponding
|
||||
@@ -172,6 +173,17 @@ include a retry field in prepared output.
|
||||
`render` resolves profiles and schemas as part of preparation but
|
||||
does not call an LLM.
|
||||
|
||||
To render from independently maintained collection checkouts:
|
||||
|
||||
```bash
|
||||
go run ./cmd/scriptorium render --config ./examples/config.discovery.yml \
|
||||
--prompt support/answer --profile support/fast \
|
||||
--input transcript=./examples/fixtures/transcript.md
|
||||
```
|
||||
|
||||
The [discovery example](../examples/discovery/framework.yml) also provides
|
||||
`review/answer`; see [configuration discovery](config.md#collection-checkout-discovery).
|
||||
|
||||
## `scriptorium serve`
|
||||
|
||||
```text
|
||||
@@ -185,8 +197,7 @@ Optional flags:
|
||||
| `--config <path>` | Application configuration file. |
|
||||
| `--config-overlay <name>` | Selected application configuration overlay. |
|
||||
| `--addr <listen-address>` | HTTP listen-address override. |
|
||||
| `--prompt-dir <dir>` | Prompt-definition directory override. |
|
||||
| `--profile-dir <dir>` | Custom profile-directory override. |
|
||||
| `--promptkit-source <path>` | PromptKit source file or directory; repeat to select ordered sources. |
|
||||
| `--schema-dir <dir>` | Schema base-directory override. |
|
||||
| `--artifact-root <dir>` | Root for HTTP `file` input references. |
|
||||
| `--max-request-bytes <n>` | Maximum encoded HTTP request-body bytes; `0` disables the limit. |
|
||||
@@ -199,27 +210,42 @@ schemas, and error codes are defined in the [HTTP API reference](api.md).
|
||||
## `scriptorium inspect prompt`
|
||||
|
||||
```text
|
||||
scriptorium inspect prompt --prompt ID [--prompt-version VERSION]
|
||||
[--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--format text|json] [--out PATH]
|
||||
scriptorium inspect prompt --prompt ID
|
||||
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
|
||||
```
|
||||
|
||||
`--prompt` is required. Inspection uses normal configuration discovery and a
|
||||
`--prompt-dir` override, defaults to text regardless of `defaults.render_format`,
|
||||
and writes to stdout unless `--out` is supplied. It loads and normalizes the
|
||||
selected definition but does not resolve a profile, load a schema, render a
|
||||
template, reserve backend capacity, or call a model.
|
||||
`--prompt` is required. Inspection uses normal configuration discovery and the
|
||||
selected Promptkit sources, defaults to text regardless of
|
||||
`defaults.render_format`, and writes to stdout unless `--out` is supplied.
|
||||
|
||||
Both definition-inspection commands first construct the same engine used for
|
||||
execution from the complete selected source sequence. Construction resolves
|
||||
and validates all effective definitions, including unused ones, and captures
|
||||
referenced message-file contents and static template syntax. Invalid unused
|
||||
definitions or unreadable message resources can therefore prevent inspection.
|
||||
No prompt collection is mandatory, but selected prompts are not ignored.
|
||||
|
||||
Inspection does not prepare an execution request, load request-time schemas or
|
||||
artifacts, resolve credential values, reserve backend capacity, or call a model.
|
||||
Prompt inspection does not select an execution profile or render request-specific
|
||||
templates.
|
||||
Prompt output includes ID, hash, optional default profile and input metadata,
|
||||
and the output contract. Absent optional values are omitted; explicitly empty
|
||||
or zero values are retained. Captured message text is not projected.
|
||||
|
||||
## `scriptorium inspect profile`
|
||||
|
||||
```text
|
||||
scriptorium inspect profile --profile ID
|
||||
[--config PATH] [--config-overlay NAME] [--profile-dir DIR] [--format text|json] [--out PATH]
|
||||
[--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out PATH]
|
||||
```
|
||||
|
||||
`--profile` is required; built-in profiles need no prompt directory. Inspection
|
||||
resolves profile inheritance and backend defaults, but never reads a credential
|
||||
value, loads a prompt, reserves capacity, or calls a model. Unset provider
|
||||
controls are shown as zero values when Promptkit leaves them unspecified.
|
||||
`--profile` is required. After the [full-source engine construction described
|
||||
above](#scriptorium-inspect-prompt), inspection selects one resolved profile
|
||||
with inheritance. `profile_model_params` contains only fields present on that
|
||||
profile; it does not fill backend defaults. `api_key_required` is a separate
|
||||
optional field. Explicit zero and empty values remain visible. Use `render`
|
||||
for request-completed model settings.
|
||||
|
||||
## Application configuration inspection
|
||||
|
||||
@@ -230,19 +256,20 @@ scriptorium config sources [flags]
|
||||
scriptorium config explain --path JSON_POINTER [flags]
|
||||
```
|
||||
|
||||
Each accepts `--config`, `--config-overlay`, `--prompt-dir`, `--profile-dir`,
|
||||
`--schema-dir`, `--addr`, `--artifact-root`, `--max-request-bytes`,
|
||||
Each accepts `--config`, `--config-overlay`, `--schema-dir`, `--addr`,
|
||||
`--artifact-root`, `--max-request-bytes`,
|
||||
`--max-artifact-bytes`, and `--max-response-bytes`. These are application
|
||||
configuration selectors and overrides; request, credential, backend-editing,
|
||||
output-format, and output-file flags are not accepted. Positional arguments
|
||||
are rejected.
|
||||
|
||||
`validate` writes no success output; status 0 means the application configuration
|
||||
loaded and passed its own validation. It does not establish that a prompt
|
||||
directory exists, credentials are present, or Promptkit can construct an
|
||||
engine. `show` writes ConfigKit's resolved, redacted YAML snapshot. `sources`
|
||||
loaded and passed its own validation. It does not establish that selected
|
||||
Promptkit sources can construct an engine, credentials are present, or a
|
||||
provider is available. `show` writes ConfigKit's resolved YAML snapshot. `sources`
|
||||
writes one newline-terminated JSON array of source records in load order. Each
|
||||
record has `id`, `label`, and `layer`, plus `path` when available; explicit CLI
|
||||
record has `id`, `label`, and `layer`, plus `path` when available. Application
|
||||
main-file and import records use the `main` layer; explicit CLI
|
||||
overrides appear as synthetic sources. Defaults-only input without overrides
|
||||
produces `[]` for `sources`.
|
||||
|
||||
@@ -251,18 +278,64 @@ The empty string selects the root; `/server/addr` selects a field; `~1` and
|
||||
`~0` escape `/` and `~` inside mapping keys. Numeric tokens are interpreted
|
||||
according to the containing mapping or sequence. Dotted paths, file paths,
|
||||
and URL fragments are not accepted. ConfigKit writes a human-readable account
|
||||
of the effective value and its chronological default, base, overlay, CLI,
|
||||
and normalization history, including removed historical values. This is not a
|
||||
JSON schema or a byte-for-byte stable prose format. Unknown and malformed paths
|
||||
exit 1 without successful explanation output.
|
||||
of the effective value and its chronological default, file/import, overlay,
|
||||
CLI, normalization, and removed-value history. This is not a JSON schema or a
|
||||
byte-for-byte stable prose format. Unknown and malformed paths exit 1 without
|
||||
successful explanation output.
|
||||
|
||||
Diagnostics and warnings go to stderr, never into successful stdout. Parse,
|
||||
load, output, and diagnostic-write failures exit 1. Ordinary output writes may
|
||||
leave partial data on failure. The inspection output can contain paths,
|
||||
identifiers, `api_key_env` reference names, and ordinary `extra_params` values,
|
||||
including displaced historical values in explanations;
|
||||
protect it accordingly. Raw API-key values are not read or emitted by these
|
||||
commands.
|
||||
leave partial data on failure. Application inspection can contain paths and
|
||||
configured values; protect its output accordingly. Raw API-key values are not
|
||||
accepted by the application configuration schema.
|
||||
|
||||
## Framework configuration inspection
|
||||
|
||||
```text
|
||||
scriptorium inspect configuration validate [flags]
|
||||
scriptorium inspect configuration show [flags]
|
||||
scriptorium inspect configuration sources [flags]
|
||||
scriptorium inspect configuration explain --path JSON_POINTER [flags]
|
||||
```
|
||||
|
||||
Each operation accepts `--config`, `--config-overlay`, repeatable
|
||||
`--promptkit-source`, and `--schema-dir`. These flags select the application
|
||||
settings and framework sources; positional arguments, `--format`, and `--out`
|
||||
are not accepted. Unlike `config validate`, framework `validate` constructs
|
||||
the Promptkit engine and validates all effective definitions, including
|
||||
unused ones. It writes nothing on success and does not prepare a request,
|
||||
resolve a credential, or contact a provider.
|
||||
|
||||
`show` writes Promptkit's native effective YAML, an inspection snapshot rather
|
||||
than a promised reloadable file. `sources` writes a
|
||||
newline-terminated JSON array in captured source order. Each source record has
|
||||
`id`, `label`, `layer`, `kind`, and `underlay_index`, plus `path` when available
|
||||
and `imported_from` for imported sources. That nested object has `source_id`,
|
||||
`line`, and `column`. Values and output structure are a projection of the
|
||||
captured Promptkit view, not Scriptorium's application-source records.
|
||||
|
||||
`explain` requires an explicitly supplied `--path` in ConfigKit JSON Pointer
|
||||
syntax; `--path ''` selects the root. For example,
|
||||
`/collections/support/profiles/0/model` addresses the first support profile's
|
||||
model in the captured list. Escape `/` as `~1` and `~` as `~0`; dots in a key
|
||||
remain literal. Malformed and unknown pointers exit 1 without a successful
|
||||
report. Explanation prose is native and is not a stable machine-readable
|
||||
format.
|
||||
|
||||
Framework `show` and `explain` use Promptkit's native redaction for its
|
||||
classified credential-shaped provider fields, including historical values.
|
||||
Environment-variable reference names and ordinary provider parameters remain
|
||||
visible. Arbitrary prompt or provider content is not classified as secret by
|
||||
name inspection, so treat these reports as sensitive. This redaction policy is
|
||||
limited to framework configuration reports; it does not add redaction to
|
||||
`inspect profile` or prepared-run output. Credentials are not resolved by any
|
||||
of these inspection operations. Promptkit owns detailed field and source
|
||||
semantics; see its tagged [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
|
||||
Warnings and errors go to stderr. Successful `validate` writes nothing to
|
||||
stdout; successful `show`, `sources`, and `explain` write their reports there.
|
||||
Parse, load, path, and output failures exit 1. A native writer error may leave
|
||||
partial output.
|
||||
|
||||
## Input And Variable Syntax
|
||||
|
||||
|
||||
267
docs/config.md
267
docs/config.md
@@ -1,68 +1,161 @@
|
||||
# Configuration Reference
|
||||
|
||||
This is the canonical reference for Scriptorium application settings. Prompt,
|
||||
profile, schema, execution-setting, built-in profile, and framework credential
|
||||
semantics are defined by the
|
||||
[Promptkit v0.11.2 format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md).
|
||||
profile, backend, schema, execution-setting, built-in profile, and framework
|
||||
credential semantics are defined by the
|
||||
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
For command syntax, see the [CLI reference](cli.md); for HTTP request shapes and
|
||||
outcomes, see the [HTTP API reference](api.md).
|
||||
outcomes, see the [HTTP API reference](api.md). For migration examples, see
|
||||
[Upgrading Promptkit sources and definitions](consumers/upgrading-promptkit.md).
|
||||
|
||||
## Discovery And Precedence
|
||||
|
||||
Application settings are resolved in this order:
|
||||
|
||||
1. built-in Scriptorium defaults;
|
||||
2. a configuration file; then
|
||||
3. CLI overrides.
|
||||
ConfigKit assembles the selected main source, imports, and overlay, resolves
|
||||
inheritance, and decodes the result with Scriptorium's typed defaults. It then
|
||||
applies explicitly supplied application CLI values, records application
|
||||
normalization, and validates the result. Source, composition, inheritance, and
|
||||
decoding failures happen before CLI assignment and cannot be repaired by it;
|
||||
an override can correct a decoded value before application validation.
|
||||
|
||||
When `--config` is omitted, Scriptorium searches
|
||||
`/usr/local/etc/scriptorium/config.yml` and then `/etc/scriptorium/config.yml`.
|
||||
If neither exists, it uses built-in defaults. An explicit `--config` path must
|
||||
exist and decode successfully.
|
||||
ConfigKit performs this ordered search for each load. If neither exists, it
|
||||
uses built-in defaults; an error in a selected candidate does not cause a
|
||||
fallback. An explicit `--config` path must exist and decode successfully,
|
||||
including when an empty path is explicitly supplied.
|
||||
|
||||
The maintained [minimal configuration](../examples/config.yml),
|
||||
[complete configuration](../examples/config.full.yml), and
|
||||
[composed configuration](../examples/config.composed.yml) are copyable examples.
|
||||
[complete configuration](../examples/config.full.yml),
|
||||
[composed configuration](../examples/config.composed.yml),
|
||||
[collection configuration](../examples/config.collections.yml), and
|
||||
[framework inheritance configuration](../examples/config.inheritance.yml) are
|
||||
copyable examples.
|
||||
|
||||
## Application Configuration File
|
||||
|
||||
Configuration is decoded by ConfigKit v0.4.2 as strict YAML. Unknown fields,
|
||||
empty files, multiple YAML documents, and YAML aliases/anchors are rejected.
|
||||
An explicit empty string is a value: it clears a file-provided optional path
|
||||
or, for required `prompt_dir`, fails the command's prerequisite. A scalar
|
||||
`null` is not an empty string and is rejected for scalar fields; nullable
|
||||
fields such as `transient_retries` retain their documented meaning. Raw
|
||||
API-key fields are not accepted.
|
||||
Application configuration is decoded by ConfigKit v1.2.1 as strict YAML.
|
||||
Unknown fields, empty files, multiple YAML documents, and YAML aliases/anchors
|
||||
are rejected.
|
||||
An explicit empty string is a value and may clear a file-provided optional
|
||||
string. A scalar `null` is not an empty string and is rejected for non-nullable
|
||||
fields; nullable fields such as `transient_retries` retain their documented
|
||||
meaning. Raw API-key fields and unknown application fields are rejected.
|
||||
|
||||
ConfigKit applies defaults, the selected file, then only explicitly supplied
|
||||
application CLI flags before validating the result. Paths in the application
|
||||
document remain relative to the process working directory, not to the file's
|
||||
Only explicitly supplied application CLI flags override decoded settings. Paths
|
||||
in the application document remain relative to the process working directory, not to the file's
|
||||
directory. Load diagnostics are printed safely on stderr; invalid application
|
||||
configuration exits with status 1.
|
||||
|
||||
### Promptkit source selectors
|
||||
|
||||
`promptkit_sources` is an ordered sequence of explicit Promptkit source
|
||||
selectors. Each item must set exactly one of `file` or `directory`:
|
||||
|
||||
```yaml
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/backends.yml
|
||||
schema_dir: ./examples/schemas
|
||||
```
|
||||
|
||||
An empty or null final resolved `promptkit_sources` list selects no
|
||||
application-supplied framework sources; PromptKit's bundled OpenRouter underlay
|
||||
still applies. A later sequence item overlays an earlier
|
||||
one. Files within a selected directory are composed together, so conflicting
|
||||
definitions in that group fail instead of using filename order as precedence.
|
||||
An explicit file may use ConfigKit imports. The complete framework format and
|
||||
source rules belong to the [PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
|
||||
Selector paths are relative to the process working directory, not to the
|
||||
application configuration file. Application validation checks the selector
|
||||
shape but does not open or validate Promptkit sources. `config validate`
|
||||
therefore does not prove that selected sources can build an engine; use
|
||||
`inspect configuration validate` for that separate check. Repeated
|
||||
`--promptkit-source` flags replace the entire configured selector sequence
|
||||
and preserve flag order. Without an authored collection policy, a selected
|
||||
overlay replaces `promptkit_sources` as a whole; a declared sequence policy
|
||||
may instead combine the lists.
|
||||
|
||||
### Collection checkout discovery
|
||||
|
||||
Select a framework discovery file using the existing file selector, as in
|
||||
[config.discovery.yml](../examples/config.discovery.yml). Its native imports
|
||||
select both supported collection-entry spellings:
|
||||
|
||||
```yaml
|
||||
config:
|
||||
imports:
|
||||
- glob: collections/*/collection.yml
|
||||
- glob: collections/*/collection.yaml
|
||||
```
|
||||
|
||||
Both globs are optional, so either spelling may have no matches. Paths are
|
||||
relative to the framework file; this example's `*` selects one checkout
|
||||
level and is not a general layout restriction. Each entry declares its own
|
||||
collection namespace and imports only its intended resource fragments. The
|
||||
checkout directory name does not determine the namespace. New checkouts are
|
||||
available on the next invocation without editing this file; a running server
|
||||
does not reload them. See the [maintained collection examples](../examples/discovery/collections/).
|
||||
|
||||
Entries imported by one discovery file are composed together as peers.
|
||||
Conflicting definitions of the same qualified resource ID fail to load;
|
||||
filename, checkout, and glob order do not set precedence. To override a
|
||||
discovered record, select a separate framework file later in
|
||||
`promptkit_sources` and supply the complete replacement record.
|
||||
|
||||
### Explicit imports and selected overlays
|
||||
|
||||
A file-backed configuration may declare `configkit.imports` with relative
|
||||
`path` or `glob` entries. Each entry is explicit; Scriptorium does not scan
|
||||
for fragments on its own. ConfigKit composes the selected file and imports by
|
||||
conflict, so two fragments cannot define the same field merely to establish
|
||||
precedence. A required missing import fails; an entry may use
|
||||
`optional: true` to suppress target nonexistence. Imports are resolved from
|
||||
the declaring file's canonical directory and remain within the selected
|
||||
file's canonical directory by default. Scriptorium does not widen that root.
|
||||
Application files declare imports under root `config.imports`. Native `path`,
|
||||
`dir`, and `glob` selectors, filters, recursion, destinations, and item
|
||||
placement are available as described in the [pinned ConfigKit contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md).
|
||||
Imports are optional by default; set `required: true` when absence must fail.
|
||||
The old `optional` field is not supported. Directory selection has no default
|
||||
filename filter; author a pattern when only YAML files are intended. ConfigKit
|
||||
composes peer sources strictly, rather than using filename order to resolve
|
||||
conflicts. Imports are resolved relative to their declaring file within the
|
||||
native source root. Scriptorium does not widen that root.
|
||||
|
||||
Root `config.sequences` can declare an identity key and merge mode for a
|
||||
collection. Scriptorium does not register or infer an identity for
|
||||
`promptkit_sources`: absent an authored policy, unkeyed lists replace across
|
||||
precedence boundaries. The [collection example](../examples/config.collections.yml)
|
||||
declares `file` identity with `upsert`, combines imported file selectors, and
|
||||
lets an overlay add another. CLI list overrides still replace the whole list.
|
||||
|
||||
At application precedence boundaries, omission preserves lower-precedence
|
||||
selectors. Under ordinary list replacement, `[]` clears the selection and
|
||||
`null` resolves to no selectors. Under authored `append` or `upsert`, `[]`
|
||||
contributes no records and `null` is invalid. To clear a collection with an
|
||||
inherited policy, declare `mode: replace` at
|
||||
`destination: [promptkit_sources]` in the higher-precedence source and supply
|
||||
`promptkit_sources: []`. See
|
||||
[ConfigKit's collection policy contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#yaml-declarations-and-policy-lifetime)
|
||||
for policy lifetime and combination rules.
|
||||
|
||||
These application root `config` controls do not change the separate
|
||||
PromptKit v0.23.3 framework format: its selected framework files use the
|
||||
[Promptkit contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
and their own native ConfigKit `config` controls.
|
||||
|
||||
`--config-overlay NAME` selects one file at `overlays/NAME.yml` below that
|
||||
root. The name must use letters, digits, `_`, or `-`; it is not a path. Empty
|
||||
and exact lowercase `default` select no overlay. There is no fallback,
|
||||
case folding, inherited stack, or automatic selection; a missing selected
|
||||
overlay fails. ConfigKit merges mapping fields and replaces other values,
|
||||
including sequences and explicit nulls, before applying explicit CLI flags.
|
||||
An active overlay needs a selected base file. No-source defaults-only loads
|
||||
cannot use imports or an active overlay.
|
||||
overlay fails. Mappings merge; sequence contributions and nulls at collection
|
||||
destinations follow the effective collection policy described above. Other
|
||||
explicit nulls replace lower values subject to the application schema.
|
||||
Defaults-only loading can enable imports, but an absent main has no import
|
||||
declarations. This CLI supplies no independent overlay root, so an active
|
||||
overlay cannot be located without a selected main context. That is not a general
|
||||
ConfigKit restriction.
|
||||
|
||||
The [composed example](../examples/config.composed.yml) imports two fragments
|
||||
and has a selectable [local overlay](../examples/overlays/local.yml). From the
|
||||
repository root:
|
||||
The [composed example](../examples/config.composed.yml) imports the application
|
||||
[source-selection fragment](../examples/config-fragments/sources.yml) and has
|
||||
a selectable [local overlay](../examples/overlays/local.yml). That fragment
|
||||
selects the [framework backend file](../examples/framework/backends.yml) through
|
||||
`promptkit_sources`; backend definitions are not imported into the application
|
||||
document. From the repository root:
|
||||
|
||||
```bash
|
||||
go run ./cmd/scriptorium render \
|
||||
@@ -72,15 +165,25 @@ go run ./cmd/scriptorium render \
|
||||
--input glossary=./examples/fixtures/glossary.yml
|
||||
```
|
||||
|
||||
Import and overlay file locations are constrained by ConfigKit;
|
||||
*application path values* such as `prompt_dir` remain relative
|
||||
to the process working directory even when authored in a fragment or overlay.
|
||||
These source constraints are separate from HTTP artifact containment.
|
||||
Inspect the collection overlay and render without contacting a provider:
|
||||
|
||||
```bash
|
||||
go run ./cmd/scriptorium config show --config ./examples/config.collections.yml --config-overlay collection
|
||||
go run ./cmd/scriptorium render --config ./examples/config.collections.yml --config-overlay collection \
|
||||
--prompt generic.markdown_summary \
|
||||
--input transcript=./examples/fixtures/transcript.md \
|
||||
--input glossary=./examples/fixtures/glossary.yml
|
||||
```
|
||||
|
||||
Import and overlay file locations are constrained by ConfigKit; application
|
||||
path values such as `promptkit_sources`, `schema_dir`, and `server.artifact_root`
|
||||
remain relative to the process working directory even when authored in a
|
||||
fragment or overlay. These source constraints are separate from Promptkit
|
||||
message-resource paths and HTTP artifact containment.
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `prompt_dir` | unset | Promptkit prompt-definition source directory. `run`, `render`, `serve`, and `inspect prompt` require an effective value. |
|
||||
| `profile_dir` | unset | Optional custom Promptkit profile source directory overlaid on Promptkit built-ins. |
|
||||
| `promptkit_sources` | empty sequence | Ordered file/directory sources for Promptkit backends, profiles, and prompts. |
|
||||
| `schema_dir` | `.` | Promptkit schema source directory for relative schema paths. |
|
||||
| `server.addr` | `:8080` | Address used by `serve`. |
|
||||
| `server.artifact_root` | unset | Root that enables HTTP `file` input references. |
|
||||
@@ -88,7 +191,6 @@ These source constraints are separate from HTTP artifact containment.
|
||||
| `server.max_artifact_bytes` | `16777216` | Maximum HTTP file-input artifact bytes; `0` disables the limit. |
|
||||
| `server.max_response_bytes` | `16777216` | Maximum encoded HTTP response bytes; `0` disables the limit. |
|
||||
| `defaults.render_format` | `text` | Default prepared-run output format: `text` or `json`. |
|
||||
| `backends` | unset | Optional mapping of custom Promptkit backend IDs to engine-scoped connection and capacity settings. |
|
||||
| `transient_retries` | Promptkit default | Optional nonnegative count of additional built-in transient generation attempts. Omitted or `null` delegates to Promptkit; `0` disables additional attempts. |
|
||||
|
||||
The size fields must be zero or greater. The [HTTP API](api.md) defines how
|
||||
@@ -96,55 +198,31 @@ each limit is enforced and reported. `server.artifact_root` configures an HTTP
|
||||
deployment boundary; see [operations](operations.md) for deployment handling.
|
||||
|
||||
The `config validate`, `config show`, `config sources`, and `config explain`
|
||||
commands inspect this
|
||||
application configuration without creating a Promptkit engine. `show` is a
|
||||
resolved inspection snapshot, not necessarily a reloadable file. Application
|
||||
validation does not check framework sources, backend connectivity, or credential
|
||||
availability. `api_key_env` is an environment-variable *name* and remains
|
||||
visible in inspection; its environment value is never read by ConfigKit.
|
||||
Ordinary backend `extra_params` values also remain visible, so keep secrets out
|
||||
of those fields and protect inspection output as potentially sensitive.
|
||||
commands inspect this application configuration without creating a Promptkit
|
||||
engine. `show` is a resolved inspection snapshot, not necessarily a reloadable
|
||||
file. Application validation does not check framework sources, backend
|
||||
connectivity, or credential availability. Use `inspect configuration` for
|
||||
Promptkit's separately captured framework configuration; its four operations
|
||||
are documented in the [CLI reference](cli.md#framework-configuration-inspection).
|
||||
`config explain --path /server/addr` uses JSON Pointer syntax to show the
|
||||
selected field's current value and history from defaults, files, overlays,
|
||||
explicit CLI overrides, and normalization. `--path ''` selects the root.
|
||||
History includes replaced and removed values; `show` presents the final
|
||||
resolved snapshot. Neither renderer looks up credential bytes, and neither
|
||||
claims to identify arbitrary secrets in unclassified parameter values.
|
||||
|
||||
## Custom Backends
|
||||
|
||||
Use `backends` when a profile selects an application-defined backend ID:
|
||||
|
||||
```yaml
|
||||
backends:
|
||||
local-gpu:
|
||||
endpoint: http://localhost:11434/v1
|
||||
api_key_env: LOCAL_GPU_API_KEY
|
||||
extra_params:
|
||||
provider_option: enabled
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 0
|
||||
```
|
||||
|
||||
Each mapping key is the case-sensitive backend ID. `endpoint` is required;
|
||||
`api_key_env`, `extra_params`, `concurrency_limit`, and `queue_capacity` are
|
||||
optional. `concurrency_limit: 0` leaves the backend unlimited. Omitting
|
||||
`queue_capacity` lets Promptkit use its default for a limited backend, while
|
||||
an explicit `queue_capacity: 0` disables queueing.
|
||||
|
||||
Configuration strictly owns the YAML shape and rejects unknown fields. Promptkit
|
||||
validates backend IDs, endpoints, environment-variable names, extra parameters,
|
||||
and capacity relationships when Scriptorium constructs its engine. There are no
|
||||
backend command-line overrides. Store only an environment-variable name in
|
||||
`api_key_env`; raw API-key fields are not accepted.
|
||||
resolved snapshot. The application schema rejects raw credential fields, but
|
||||
do not treat inspection as a general-purpose filter for sensitive strings.
|
||||
|
||||
## Framework Source Mapping
|
||||
|
||||
Scriptorium passes `prompt_dir`, `profile_dir`, and `schema_dir` to Promptkit
|
||||
when constructing its engine. Scriptorium does not redefine or independently
|
||||
parse those framework file formats.
|
||||
Scriptorium assembles selected Promptkit source files and directories in the
|
||||
order described above, then constructs the Promptkit engine. Framework
|
||||
definitions and custom backends belong in those selected source files, not in
|
||||
the Scriptorium application configuration. Message `content_file` paths use
|
||||
the defining source's resource context and are captured when the engine is
|
||||
constructed; schema and request-artifact loading remain preparation-time
|
||||
operations. Scriptorium does not redefine or independently parse framework
|
||||
file formats.
|
||||
|
||||
- Prompt selection, versions, message templates, inputs, output contracts, and
|
||||
- Prompt selection by ID, message templates, inputs, output contracts, and
|
||||
session IDs are Promptkit contracts.
|
||||
- Profile fields, numeric ranges, execution defaults, overlay precedence,
|
||||
built-in profiles, and credential rules are Promptkit contracts.
|
||||
@@ -152,19 +230,18 @@ parse those framework file formats.
|
||||
contracts.
|
||||
|
||||
See the
|
||||
[tagged Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md)
|
||||
[tagged Promptkit format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
for all of those definitions. The files under
|
||||
[`examples/prompts`](../examples/prompts/),
|
||||
[`examples/profiles`](../examples/profiles/), and
|
||||
[`examples/schemas`](../examples/schemas/) are maintained Scriptorium
|
||||
application inputs using that tagged format.
|
||||
|
||||
Promptkit supplies the opt-in profile IDs `promptkit-openrouter-standard` and
|
||||
`promptkit-openrouter-auto`. They can be selected as a prompt default or request
|
||||
profile, and custom profile sources retain precedence. Their effective model
|
||||
settings may change with a catalog upgrade; use a complete application-owned
|
||||
profile when fixed settings are required. See the tagged format reference for
|
||||
the maintained-profile contract.
|
||||
PromptKit supplies a default-on OpenRouter bundle as the lowest-priority
|
||||
underlay. Choose an actual bundled profile such as `auto`, or a complete
|
||||
application-owned profile; old catalog IDs are not aliases. Caller definitions
|
||||
can replace bundled records by normal composition. Bundle availability is not
|
||||
a readiness check or a promise of a fixed provider model.
|
||||
|
||||
## Credentials And Outbound Behavior
|
||||
|
||||
@@ -195,7 +272,7 @@ client retains its own retry behavior. The tagged integration contract owns
|
||||
eligibility and timing.
|
||||
|
||||
Promptkit's
|
||||
[OpenAI-compatible integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md)
|
||||
[OpenAI-compatible integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
||||
defines outbound authentication, provider request mapping, transport limits,
|
||||
and timeout layering.
|
||||
|
||||
@@ -204,4 +281,4 @@ and timeout layering.
|
||||
- [CLI reference](cli.md)
|
||||
- [HTTP API reference](api.md)
|
||||
- [Operations guide](operations.md)
|
||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md)
|
||||
- [Promptkit framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
|
||||
@@ -13,8 +13,8 @@ and the HTTP service according to the boundary your application needs.
|
||||
- Application configuration: [configuration reference](../config.md)
|
||||
|
||||
Go applications that need an in-process prompt framework should import
|
||||
Promptkit directly. The tagged
|
||||
[Promptkit Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md)
|
||||
`go.maximumdirect.net/promptkit` directly. The tagged
|
||||
[Promptkit Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
|
||||
owns that interface; Scriptorium does not provide a Go library package.
|
||||
Consumers arriving from the former Scriptorium Go API should follow the
|
||||
[migration guide](migrating-to-promptkit.md).
|
||||
@@ -35,4 +35,4 @@ Scriptorium does not persist run state. A retry can produce different output
|
||||
and can incur another provider request. CLI exits belong to the
|
||||
[CLI reference](../cli.md), HTTP status behavior belongs to the
|
||||
[HTTP API reference](../api.md), and framework semantics belong to
|
||||
[Promptkit v0.11.2](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md).
|
||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# Migrate From Scriptorium To Promptkit
|
||||
|
||||
This historical first hop targets Promptkit v0.11.1 and its former module
|
||||
path. For the current public module `go.maximumdirect.net/promptkit` and the
|
||||
ID-only configuration cutover, follow the
|
||||
[current upgrade guide](upgrading-promptkit.md) afterward and consult the
|
||||
[current Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md).
|
||||
|
||||
## Supported Migration Boundary
|
||||
|
||||
Scriptorium `v0.11.1` at
|
||||
|
||||
110
docs/consumers/upgrading-promptkit.md
Normal file
110
docs/consumers/upgrading-promptkit.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Upgrade PromptKit Sources And Definitions
|
||||
|
||||
This guide covers the CLI and framework configuration cutover to
|
||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/configuration-migration.md)
|
||||
and ConfigKit v1.2.1. See the [framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
for the full upstream contract.
|
||||
|
||||
Removing version selectors and changing framework definition shapes are
|
||||
intentional incompatible changes under the [release policy](../policy/release.md#versioning-and-compatibility).
|
||||
|
||||
## Application configuration
|
||||
|
||||
Existing application fields that already select the desired sources retain
|
||||
their meaning. For example, this application configuration works both before
|
||||
and after the cutover:
|
||||
|
||||
```yaml
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/backends.yml
|
||||
schema_dir: ./examples/schemas
|
||||
```
|
||||
|
||||
The selected framework files must still be rewritten as described below.
|
||||
Scriptorium uses one native application schema: discovery, imports, selected
|
||||
overlays, source order, explicit overrides, and working-directory-relative
|
||||
selector paths remain application configuration behavior. It loads framework
|
||||
sources separately with PromptKit's schema. No second application schema or
|
||||
legacy definition reader is provided. See the [configuration reference](../config.md).
|
||||
|
||||
## Rewrite framework definitions
|
||||
|
||||
Replace version-keyed prompts and keyed profile/backend mappings with lists
|
||||
of records containing `id`. For example, before:
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
report:
|
||||
"1":
|
||||
default_profile: fast
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize this note.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
profiles:
|
||||
fast:
|
||||
endpoint: https://example.invalid/v1
|
||||
model: synthetic-model
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
- id: report
|
||||
default_profile: fast
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize this note.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
profiles:
|
||||
- id: fast
|
||||
endpoint: https://example.invalid/v1
|
||||
model: synthetic-model
|
||||
```
|
||||
|
||||
Select prompts by exact ID; `--prompt-version` is rejected. If multiple former
|
||||
versions must remain independently selectable, give them distinct IDs and
|
||||
update callers. Qualified IDs such as `support/answer` are passed unchanged.
|
||||
PromptKit resolves `content_file` relative to the source that supplied the
|
||||
field, including imported or inherited definitions. The
|
||||
[framework format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md#messages-and-templates)
|
||||
describes which paths are allowed within the source root.
|
||||
|
||||
Convert framework source controls from `configkit:` to `config:` and add
|
||||
`required: true` for imports that must exist. An optional framework source
|
||||
header, if present, is `version: 2`. Use sibling inheritance such as
|
||||
`extends: {id: base}`.
|
||||
|
||||
Across ordered `promptkit_sources` entries, a later framework underlay replaces
|
||||
the whole record for the same ID. Supply the complete intended replacement,
|
||||
or create a distinct inherited record. Checkouts imported by the same
|
||||
[discovery file](../config.md#collection-checkout-discovery) are peers:
|
||||
conflicting definitions of the same ID fail instead of overriding one another.
|
||||
|
||||
Null profile endpoint, credential, or provider-map settings can fall
|
||||
back to the selected backend; review behavior rather than assuming null clears
|
||||
backend values. Keep JSON schemas self-contained instead of depending on remote
|
||||
schema references. Application selectors remain separate from these framework
|
||||
semantics.
|
||||
|
||||
## Replace catalog selections
|
||||
|
||||
The default-on OpenRouter bundle supplies actual profiles such as `auto` and
|
||||
`gpt-sol-latest`. Former catalog and `promptkit-openrouter-*` IDs are not
|
||||
compatibility aliases. Choose an intended current bundled profile explicitly,
|
||||
or define a complete local profile with the endpoint, model, and settings you
|
||||
need. Do not assume a new default model. The bundle is an underlay and caller
|
||||
records can replace it through ordinary composition.
|
||||
|
||||
For multiple independently maintained repositories, see the
|
||||
[collection-discovery example](../../examples/config.discovery.yml). Its
|
||||
entry files declare their own namespaces; checkout names do not become IDs.
|
||||
The [CLI reference](../cli.md) demonstrates qualified rendering and the
|
||||
[operations guide](../operations.md) describes when updates take effect.
|
||||
@@ -2,9 +2,11 @@
|
||||
|
||||
This is the contributor entry point for Scriptorium. Scriptorium is an
|
||||
application that consumes the public
|
||||
[Promptkit v0.11.2 package](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md);
|
||||
[PromptKit v0.23.3 package](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md);
|
||||
framework implementation work belongs in Promptkit. Application configuration
|
||||
uses ConfigKit v0.4.2; keep Scriptorium's schema and source selection local.
|
||||
uses ConfigKit v1.2.1; keep Scriptorium's schema and source selection local.
|
||||
PromptKit uses the same ConfigKit release to load framework sources under its
|
||||
own separate schema and format contract.
|
||||
|
||||
## Initial Orientation
|
||||
|
||||
@@ -45,21 +47,31 @@ Before recommending, planning, or implementing work:
|
||||
| HTTP routes, DTOs, limits, status mapping, or artifact policy | [HTTP API contract](api.md), [adapter internals](internal/adapters.md), and [source internals](internal/sources.md) |
|
||||
| Application configuration or precedence | [Configuration contract](config.md), [adapter internals](internal/adapters.md), [source internals](internal/sources.md), and `internal/config` definition/load tests |
|
||||
| Prepared-run presentation | [CLI contract](cli.md), [adapter internals](internal/adapters.md), and `internal/format` |
|
||||
| Prompt, profile, schema, generation, or validation semantics | [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md) and the [Promptkit consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md) |
|
||||
| OpenAI-compatible outbound behavior or timeout layering | [Promptkit integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md) |
|
||||
| Promptkit source selection or captured framework inspection | [Configuration contract](config.md#promptkit-source-selectors), [framework configuration CLI contract](cli.md#framework-configuration-inspection), and [source internals](internal/sources.md) |
|
||||
| Prompt, profile, schema, generation, or validation semantics | [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md) and the [Promptkit consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md) |
|
||||
| OpenAI-compatible outbound behavior or timeout layering | [PromptKit v0.23.3 integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md) |
|
||||
| Subprocess behavior | [Subprocess integration](integrations/subprocess.md) and [CLI contract](cli.md) |
|
||||
| Runtime operation or recovery | [Operations](operations.md) |
|
||||
| Dependencies, copied assets, or attribution | The [OSS policy](policy/oss.md), [component attributions](oss/attribution.md), and [architecture policy](policy/architecture.md) |
|
||||
| Release documentation, packaging, or publication | The [release policy](policy/release.md), [OSS policy](policy/oss.md), [release procedure](release.md), [hosted release workflow](../.woodpecker/release.yml), and [architecture policy](policy/architecture.md) |
|
||||
| Examples or copyable assets | The owning Scriptorium contract, the relevant [Promptkit format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md), and the related files under `examples/` |
|
||||
| Examples or copyable assets | The owning Scriptorium contract, the relevant [PromptKit v0.23.3 format contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md), and the related files under `examples/` |
|
||||
| Architecture decisions or future work | The [documentation policy](policy/documentation.md), relevant accepted ADRs, and relevant roadmap documents |
|
||||
|
||||
Cross-project changes land and release in Promptkit before Scriptorium adopts
|
||||
the tagged version. Do not commit a Go workspace, local replacement, vendored
|
||||
Promptkit source, or an import of a Promptkit `internal` package.
|
||||
The module graph can include upstream maintainer tools such as ReleaseKit and
|
||||
their dependencies even though those graph-only packages are not linked into
|
||||
the Scriptorium runtime; use `go list -deps ./cmd/scriptorium` to distinguish
|
||||
the linked packages.
|
||||
|
||||
## Baseline Validation
|
||||
|
||||
The Go consumer minimum is the version declared in `go.mod`, currently 1.27.0,
|
||||
required by application ConfigKit v1.2.1. Running a published binary does not
|
||||
require a Go installation. Release validation and builds use
|
||||
the separately selected toolchain in the [release procedure](release.md).
|
||||
|
||||
For code changes, run:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -55,8 +55,9 @@ protocol.
|
||||
|
||||
For `run`, callers may set the [whole-run deadline](../cli.md) to bound the
|
||||
engine invocation. It is separate from the per-attempt generation timeout and
|
||||
does not include child startup or output writing. A deadline failure exits 1;
|
||||
supervisors still own process termination when they need to stop the child
|
||||
does not include child startup or output writing. A typed operation deadline
|
||||
failure exits 1.
|
||||
Supervisors still own process termination when they need to stop the child
|
||||
outside that engine interval.
|
||||
|
||||
For deployment, filesystem permissions, and sensitive-artifact handling, see
|
||||
|
||||
@@ -9,7 +9,7 @@ own IO and presentation mechanics, not framework decisions.
|
||||
External contracts are canonical in the [CLI reference](../cli.md) and
|
||||
[HTTP API reference](../api.md). Promptkit's public engine contract is
|
||||
described by its tagged
|
||||
[Go consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md).
|
||||
[Go consumer guide](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md).
|
||||
|
||||
## Components And Collaborators
|
||||
|
||||
@@ -17,11 +17,14 @@ described by its tagged
|
||||
`internal/adapter/cli`.
|
||||
- `internal/adapter/cli` resolves settings through `internal/config`,
|
||||
constructs `promptkit.Engine`, maps CLI values to `promptkit.RunRequest`,
|
||||
and owns output files, summaries, and exit codes.
|
||||
exposes application and captured framework configuration inspection, and
|
||||
owns output files, summaries, and exit codes.
|
||||
- `internal/config` delegates application YAML decoding, default resolution,
|
||||
overrides, and validation ordering to ConfigKit. The CLI supplies only
|
||||
overrides, and validation ordering to ConfigKit v1.2.1. The CLI supplies only
|
||||
explicitly visited application flags as overrides and renders safe structured
|
||||
load diagnostics on stderr.
|
||||
load diagnostics on stderr. PromptKit uses the same ConfigKit release to
|
||||
load framework sources under its own definition and format contract. Both
|
||||
domains use ConfigKit's native diagnostic writer.
|
||||
- `internal/adapter/http` strictly decodes request DTOs, maps them to Promptkit
|
||||
public values, calls its adapter-owned `Runner` interface, and maps results
|
||||
and errors to HTTP DTOs.
|
||||
@@ -35,7 +38,7 @@ described by its tagged
|
||||
### CLI
|
||||
|
||||
`run` calls `promptkit.Engine.Run`; `render` calls
|
||||
`promptkit.Engine.Prepare`. Both share request mapping for prompt ID/version
|
||||
`promptkit.Engine.Prepare`. Both share request mapping for exact prompt ID
|
||||
and profile selection, optional file inputs, variables, and presence-aware
|
||||
execution overrides. Omitted framework settings remain zero values so Promptkit
|
||||
resolves its own defaults and definition-required inputs.
|
||||
@@ -44,14 +47,26 @@ run/render flag enters only the mapped request. The server uses the same engine
|
||||
setting and retains one backend admission state. Render prepares without
|
||||
generation.
|
||||
|
||||
`inspect prompt` maps the selected ID and optional version to
|
||||
`inspect prompt` maps the selected ID to
|
||||
`promptkit.Engine.InspectPrompt`, then formats Scriptorium-owned inspection
|
||||
data. It constructs the same configuration-aware engine but does not perform
|
||||
preparation or generation.
|
||||
data. `inspect profile` maps an explicit profile ID to `Engine.InspectProfile`
|
||||
and formats Scriptorium-owned resolved profile data without backend completion.
|
||||
|
||||
`inspect profile` maps an explicit profile ID to `Engine.InspectProfile` and
|
||||
formats a safe application-owned effective profile view. It intentionally does
|
||||
not require prompt, schema, or artifact sources.
|
||||
Both paths construct the same full-source engine as execution before inspecting
|
||||
the chosen definition; selected prompts are not filtered out. They do not
|
||||
perform request preparation or generation. The [CLI definition-inspection
|
||||
contract](../cli.md#scriptorium-inspect-prompt) owns the loading and runtime
|
||||
boundaries, including the distinction between an optional prompt collection
|
||||
and validating any prompts that were selected.
|
||||
|
||||
`inspect configuration` builds the same engine from the selected native
|
||||
Promptkit source sequence, reads its captured configuration view, and delegates
|
||||
effective output and explanation to Promptkit's native writers. The adapter
|
||||
projects captured source records into its small JSON DTO. Its validation
|
||||
operation constructs the engine without preparing a request. Framework source
|
||||
semantics and report redaction are owned by Promptkit; the [CLI
|
||||
contract](../cli.md#framework-configuration-inspection) documents the exposed
|
||||
operations and their deliberately limited redaction scope.
|
||||
|
||||
The `config` family loads the same application ConfigKit result once and uses
|
||||
its validation, effective-output, source, and explanation views directly. It
|
||||
@@ -62,11 +77,14 @@ JSON DTO for `sources`; `show` and `explain` use ConfigKit's own writers.
|
||||
with `promptkit.WithArtifactReader`, passes the engine through the HTTP
|
||||
adapter's consumer-owned `Runner` interface, and starts the server.
|
||||
|
||||
The command paths assemble the engine from the same resolved prompt, profile,
|
||||
and schema directories plus configured custom backends. Each backend is mapped
|
||||
to Promptkit's public `Backend` value and registered during engine construction,
|
||||
so one constructed server engine retains one immutable backend registry and its
|
||||
associated capacity state.
|
||||
The command paths assemble the engine from application-selected framework
|
||||
source files and directories plus the schema directory and retry setting.
|
||||
Individual files become import-enabled ConfigKit inputs; directory selections
|
||||
are expanded through Promptkit's directory-input helper. Scriptorium supplies
|
||||
those ordered underlays through Promptkit's public configuration API rather
|
||||
than parsing or mapping framework definitions itself. The server constructs
|
||||
one engine, which retains the framework's backend registry and associated
|
||||
capacity state across requests.
|
||||
|
||||
### HTTP
|
||||
|
||||
@@ -77,6 +95,9 @@ validation, model, usage, and metadata values directly.
|
||||
The optional HTTP retry count maps to `RunRequest.TransientRetries` without
|
||||
mutating the shared engine. Negative values are rejected before invocation;
|
||||
Promptkit controls retries and backend admission for concurrent requests.
|
||||
CLI and HTTP operation budgets map to PromptKit's `OperationTimeout`; HTTP
|
||||
passes the caller context through unchanged. The public sentinels determine
|
||||
timeout classification.
|
||||
|
||||
Failure mapping uses `errors.Is` against Promptkit's public sentinels and the
|
||||
HTTP reader's Scriptorium-owned containment and size errors. Wrapped reader
|
||||
|
||||
@@ -7,13 +7,13 @@ public behavior belongs in the linked contracts.
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| `cmd/scriptorium` | Process entrypoint that delegates arguments and streams to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/adapter/cli` | Parses execution and application-configuration inspection commands, resolves settings, constructs Promptkit engines only for execution/definition inspection, maps requests, and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/adapter/cli` | Parses execution, application-configuration, and captured framework-configuration inspection commands; resolves settings; constructs Promptkit engines for execution, definition inspection, and framework inspection; maps requests; and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
||||
| `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
||||
| `internal/config` | Selects application sources, supplies the ConfigKit definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md) |
|
||||
| `internal/config` | Selects application sources through ConfigKit v1.2.1, supplies the application definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md), [source internals](sources.md) |
|
||||
| `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) |
|
||||
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output; framework-configuration output uses Promptkit's native captured view and writers. | [CLI contract](../cli.md), [adapter internals](adapters.md) |
|
||||
| `internal/report` | Projects one terminal Promptkit observation into Scriptorium-owned execution metadata. | [adapter internals](adapters.md) |
|
||||
|
||||
Framework implementation packages are provided by
|
||||
[Promptkit v0.11.2](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md)
|
||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
|
||||
and are not part of this repository.
|
||||
|
||||
@@ -5,20 +5,26 @@
|
||||
This document covers Scriptorium-owned source locations and the restricted HTTP
|
||||
artifact reader. Prompt, profile, schema, and ordinary artifact semantics are
|
||||
owned by the tagged
|
||||
[Promptkit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md).
|
||||
[PromptKit v0.23.3 format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md).
|
||||
|
||||
## Application Source Locations
|
||||
|
||||
`internal/config` resolves `prompt_dir`, `profile_dir`, and `schema_dir` from
|
||||
Scriptorium defaults, configuration files, and CLI overrides. It also resolves
|
||||
the application-owned `backends` mapping into sorted engine settings.
|
||||
`internal/adapter/cli` passes the directories into `promptkit.Config` and maps
|
||||
each configured backend to Promptkit's public engine registration when
|
||||
constructing an engine shared by the command path.
|
||||
`internal/config` resolves the ordered `promptkit_sources` sequence and
|
||||
`schema_dir` from application configuration and explicit CLI overrides. It
|
||||
owns selector-shape validation, not framework file parsing. During engine
|
||||
construction, `internal/adapter/cli` turns file selections into import-enabled
|
||||
ConfigKit inputs and expands directory selections with
|
||||
`promptkit.DirectoryInputs`, then passes the ordered groups as Promptkit
|
||||
configuration underlays, one per selector. Directory inputs retain the
|
||||
helper's strict-composition grouping; no selected source becomes an implicit
|
||||
framework main. The shared Promptkit engine receives the selected schema
|
||||
directory and retry setting, and the default OpenRouter bundle remains enabled.
|
||||
|
||||
Scriptorium does not search, parse, validate, or overlay framework source files
|
||||
itself. Promptkit owns prompt selection, profile built-ins and overlays, schema
|
||||
resolution, ordinary file artifacts, and the related error identities.
|
||||
Promptkit uses ConfigKit to load and compose those framework inputs according
|
||||
to Promptkit's own schema and format. Scriptorium does not parse or validate
|
||||
Promptkit definitions. Promptkit owns definition selection and validation,
|
||||
profile built-ins and overlays, schema resolution, ordinary file artifacts,
|
||||
captured framework configuration inspection, and related error identities.
|
||||
|
||||
The [configuration reference](../config.md) owns Scriptorium's source-location
|
||||
fields and precedence. Maintained files under `examples/` are application
|
||||
@@ -26,33 +32,43 @@ inputs that use Promptkit's tagged formats.
|
||||
|
||||
## Application Configuration Boundary
|
||||
|
||||
`internal/config` contains a ConfigKit-native `Document` definition. Its
|
||||
fresh defaults come from Scriptorium's existing default constants; its resolver
|
||||
records only actual trimming, path cleaning, and render-format normalization.
|
||||
The validator reports application retry, size-limit, and format errors without
|
||||
mutating the document or exposing rejected values. Application path values keep
|
||||
working-directory-relative meaning. ConfigKit owns strict YAML decoding and
|
||||
presence semantics, including dynamic integer values in backend parameters.
|
||||
`internal/config` contains the application-owned ConfigKit-native `Document`
|
||||
definition. Its fresh defaults come from Scriptorium's existing default
|
||||
constants; its resolver records only actual leaf-level trimming, path cleaning, and
|
||||
render-format normalization. The validator reports selector-shape, retry,
|
||||
size-limit, and format errors without mutating the document or exposing
|
||||
rejected values. Application path values keep working-directory-relative
|
||||
meaning. ConfigKit v1.2.1 owns strict application YAML decoding and presence
|
||||
semantics. Framework configuration is separately loaded by PromptKit using
|
||||
the same ConfigKit release under PromptKit's own schema and source contract.
|
||||
|
||||
The `Load` boundary selects an explicit file, the first existing implicit
|
||||
configuration file, or ConfigKit's defaults-only source. Only absence advances
|
||||
implicit discovery. One ConfigKit load applies explicit application overrides
|
||||
before validation and returns a result containing the resolved document and
|
||||
its frozen inspection records. `Settings` converts that document into the
|
||||
existing runtime settings with backend IDs sorted; it does not read files or
|
||||
apply another merge. File-backed loads enable explicit ConfigKit imports and
|
||||
select one named overlay when requested. ConfigKit uses the selected file's
|
||||
The `Load` boundary selects an explicit file or constructs a native ConfigKit
|
||||
search over the ordered implicit paths. Discovery exhaustion uses defaults;
|
||||
a selected-file error stops loading. One ConfigKit load enables imports,
|
||||
applies explicitly supplied application overrides after decoding and before
|
||||
resolution and validation, and returns the document and its frozen inspection
|
||||
records. `Settings` copies selectors in their resolved order into runtime
|
||||
settings; it does not read files or apply another merge. ConfigKit uses the selected file's
|
||||
canonical directory as the source root; the application does not widen it.
|
||||
Application path values retain process-working-directory-relative meaning.
|
||||
|
||||
Runtime conversion consumes the resolved application values through the
|
||||
existing `Settings` path. Promptkit receives selected framework sources only
|
||||
after application configuration has loaded and validated; its source merge,
|
||||
definition validation, and captured inspection view remain framework-owned.
|
||||
|
||||
Every configuration-loading command uses this boundary. The CLI maps only
|
||||
explicitly visited application flags to ConfigKit overrides and prints
|
||||
structured load diagnostics before engine construction. The old application
|
||||
structured load diagnostics before engine construction. Application and
|
||||
framework load errors and warnings use their respective native writers. The old application
|
||||
YAML decoder and merge path have been removed.
|
||||
Application `config` inspection reuses the result's frozen effective,
|
||||
provenance, and history views without constructing an engine or reloading
|
||||
sources. It does not resolve API-key environment values; ordinary backend
|
||||
parameters remain visible by the selected inspection policy.
|
||||
sources. It does not resolve credentials. `inspect configuration` separately
|
||||
constructs an engine and exposes Promptkit's captured framework view; only its
|
||||
framework `show` and `explain` reports receive Promptkit's native,
|
||||
credential-shaped redaction. Neither inspection family is a general-purpose
|
||||
filter for arbitrary sensitive values.
|
||||
|
||||
## Restricted HTTP Artifact Reader
|
||||
|
||||
|
||||
@@ -9,7 +9,9 @@ fields, or HTTP wire behavior.
|
||||
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
||||
- [Configuration reference](config.md): application settings, source
|
||||
locations, defaults, and credential mapping.
|
||||
- [Promptkit framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md):
|
||||
- [Promptkit source upgrade guide](consumers/upgrading-promptkit.md): concise
|
||||
source-selector and definition migration examples.
|
||||
- [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md):
|
||||
prompt, profile, schema, execution-setting, and framework credential
|
||||
contracts.
|
||||
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
||||
@@ -39,10 +41,17 @@ without a report.
|
||||
|
||||
## Deploy The Filesystem And Process
|
||||
|
||||
Provide the process with readable configured Promptkit prompt, profile, and
|
||||
schema sources that follow the tagged framework formats. For an HTTP deployment
|
||||
that accepts file artifacts, use a dedicated, narrow artifact directory rather
|
||||
than a general-purpose or sensitive filesystem tree.
|
||||
Provide the process with readable selected Promptkit source files/directories
|
||||
and schema sources that follow the tagged framework formats. At engine
|
||||
construction, PromptKit v0.23.3 validates every selected effective definition,
|
||||
including unused definitions, and captures referenced message-file contents
|
||||
and static template syntax. `serve` constructs one engine at startup; it does
|
||||
not reload changed configuration or message content. Restart the process to
|
||||
validate and capture a new configuration. Schema and request-artifact reads,
|
||||
credential resolution, and provider work remain request-preparation or runtime
|
||||
concerns. For an HTTP deployment that accepts file artifacts, use a dedicated,
|
||||
narrow artifact directory rather than a general-purpose or sensitive filesystem
|
||||
tree.
|
||||
|
||||
If application configuration uses explicit imports or a selected overlay,
|
||||
deploy those files beneath the selected configuration file's canonical
|
||||
@@ -52,12 +61,15 @@ resolve from the process working directory, so set that directory deliberately
|
||||
in the service manager. This configuration-source boundary is distinct from
|
||||
the HTTP artifact root and does not constrain prompt or schema directories.
|
||||
|
||||
When upgrading existing definitions, review selected profile IDs.
|
||||
`deepseek-3-2`, `deepseek-4-flash`, and `deepseek-4-pro` were removed from the
|
||||
selected catalog. Choose an available catalog profile, one of Promptkit's
|
||||
maintained names, or a complete application-owned profile. New DeepSeek names
|
||||
are not aliases for the removed ones; see the
|
||||
[v0.10 migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.10.0/docs/releases/v0.10.0.md).
|
||||
For independently maintained collections, place checkouts beneath the
|
||||
directory selected by your framework discovery file. The maintained
|
||||
[discovery example](../examples/config.discovery.yml) selects optional
|
||||
`collection.yml` and `collection.yaml` entry files; each checkout owns its
|
||||
namespace and explicit fragment imports. Update a checkout independently and
|
||||
invoke the CLI again, or restart a running server to capture the update. Do
|
||||
not treat arbitrary YAML in a checkout as a framework source. When upgrading
|
||||
definitions, review exact profile IDs and replace retired catalog references
|
||||
as described in the [migration guide](consumers/upgrading-promptkit.md).
|
||||
|
||||
Run Scriptorium under an identity that can:
|
||||
|
||||
@@ -87,25 +99,27 @@ and apply the same retention and access rules as for model input and output.
|
||||
|
||||
Before changing production inputs, profiles, or schemas:
|
||||
|
||||
1. use `config validate`, `config show`, or `config sources` to check the
|
||||
application configuration and its selected sources without constructing an
|
||||
engine; this does not verify prompt files, provider access, or credentials;
|
||||
2. confirm the deployed configuration selects the intended sources and model
|
||||
credentials;
|
||||
3. use [`render`](cli.md) with the same request inputs and variables to confirm
|
||||
1. use `config validate`, `config show`, or `config sources` to check
|
||||
Scriptorium's application configuration; these do not open Promptkit sources
|
||||
or construct an engine;
|
||||
2. use `inspect configuration validate` to construct the engine and validate
|
||||
all selected framework definitions; this does not prepare a request, check
|
||||
credentials, load request-time schemas/artifacts, or contact a provider;
|
||||
3. confirm the deployed configuration selects the intended sources and
|
||||
environment-variable names;
|
||||
4. use [`render`](cli.md) with the same request inputs and variables to confirm
|
||||
preparation without a model call;
|
||||
4. use [`run`](cli.md) for generation; and
|
||||
5. retain or discard validation-failed output according to the caller's
|
||||
5. use [`run`](cli.md) for generation; and
|
||||
6. retain or discard validation-failed output according to the caller's
|
||||
policy.
|
||||
|
||||
Inspection output can include configuration paths, names, and ordinary backend
|
||||
parameters. `config explain --path` uses JSON Pointer paths and can show
|
||||
displaced historical values as well as the final value. Protect both the
|
||||
selected paths and the output, especially when backend parameters contain
|
||||
sensitive identifiers; ConfigKit does not infer arbitrary secrets in those
|
||||
parameters. Apply
|
||||
access and retention controls; the application stores only API-key environment
|
||||
reference names, not the resolved credential bytes.
|
||||
Application configuration inspection can include paths and configured values.
|
||||
Framework `inspect configuration show` and `explain` use Promptkit's native
|
||||
redaction for classified credential-shaped provider fields, including
|
||||
historical values, while leaving environment-reference names and ordinary
|
||||
provider parameters visible. Neither report identifies arbitrary sensitive
|
||||
content by value or name. Protect inspection output accordingly; Scriptorium
|
||||
stores environment-variable names, not the resolved credential bytes.
|
||||
|
||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
||||
copyable preflight example. The CLI reference owns its complete invocation and
|
||||
@@ -150,7 +164,7 @@ setting and then Promptkit's default, while explicit zero disables additional
|
||||
attempts. Existing generation timeouts apply
|
||||
per attempt, while the caller context covers the complete operation. Provider
|
||||
`Retry-After` timing can exceed Promptkit's ordinary backoff cap; consult the
|
||||
[outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md)
|
||||
[outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
||||
for details.
|
||||
|
||||
CLI `run` can set a [whole-run deadline](cli.md) when total execution time must
|
||||
@@ -158,8 +172,8 @@ be bounded. The existing generation timeout applies afresh to each attempt and
|
||||
does not cap retry waits or the complete invocation. The whole-run interval
|
||||
covers the engine call, including capacity waiting and repairs as supported by
|
||||
context cancellation; it excludes parsing, engine setup, and output writes.
|
||||
HTTP callers can use [`run_timeout_seconds`](api.md) to shorten their request
|
||||
context for the same engine interval. The API distinguishes an expired
|
||||
HTTP callers can use [`run_timeout_seconds`](api.md) to set the engine's
|
||||
operation budget for the same interval. The API distinguishes a typed expired
|
||||
execution deadline from caller cancellation and from a provider-attempt
|
||||
timeout. A canceled capacity waiter releases its Promptkit admission state.
|
||||
|
||||
|
||||
@@ -5,23 +5,22 @@
|
||||
Material: `docs/policy/oss.md`, `docs/policy/meta.md`,
|
||||
`docs/policy/agents.md`, `docs/policy/documentation.md`,
|
||||
`docs/policy/architecture.md`, `docs/policy/testing.md`,
|
||||
`docs/policy/release.md`, `AGENTS.md`, and the
|
||||
Policy Reading Guide section of `docs/development.md`, adapted for this
|
||||
repository.
|
||||
`docs/policy/release.md`, `AGENTS.md`, and the Policy Reading Guide section
|
||||
of `docs/development.md`, adapted for this repository.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
License and notices: [CC BY-SA 4.0](licenses/CC-BY-SA-4.0.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/LICENSE).
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843).
|
||||
[Source](https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4).
|
||||
|
||||
## Promptkit
|
||||
## PromptKit
|
||||
|
||||
Selected module: `gitea.maximumdirect.net/eric/promptkit` at `v0.11.2`.
|
||||
Selected module: `go.maximumdirect.net/promptkit` at `v0.23.3`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
@@ -29,14 +28,14 @@ SPDX-License-Identifier: LGPL-3.0-or-later
|
||||
|
||||
License and notices: [LGPL 3.0](licenses/LGPL-3.0.txt) |
|
||||
[GPL 3.0](licenses/GPL-3.0.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit/src/commit/12d6d80024e80c552d122a076c22429bf6f2f6c2/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/promptkit/src/commit/12d6d80024e80c552d122a076c22429bf6f2f6c2/LICENSE).
|
||||
[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/COPYRIGHT) |
|
||||
[LICENSE](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/promptkit/src/commit/12d6d80024e80c552d122a076c22429bf6f2f6c2).
|
||||
[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f).
|
||||
|
||||
## ConfigKit
|
||||
|
||||
Selected module: `gitea.maximumdirect.net/eric/ConfigKit` at `v0.4.2`.
|
||||
Selected module: `go.maximumdirect.net/configkit` at `v1.2.1`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
@@ -44,87 +43,39 @@ SPDX-License-Identifier: LGPL-3.0-or-later
|
||||
|
||||
License and notices: [LGPL 3.0](licenses/LGPL-3.0.txt) |
|
||||
[GPL 3.0](licenses/GPL-3.0.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f/LICENSE).
|
||||
[COPYRIGHT](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/COPYRIGHT) |
|
||||
[LICENSE](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/67c03690181c5af8dde6be325abac241dae6732f).
|
||||
|
||||
## Promptkit OpenRouter Catalog
|
||||
|
||||
Selected module: `gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2` at `v2.20260926.0`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
SPDX-License-Identifier: MIT
|
||||
|
||||
License and notices: [MIT](licenses/MIT.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/4f731d217e0bc39151c010f50fb5912706e7b6b3/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/4f731d217e0bc39151c010f50fb5912706e7b6b3/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/4f731d217e0bc39151c010f50fb5912706e7b6b3).
|
||||
|
||||
## Promptkit Rakestrawhome Catalog
|
||||
|
||||
Selected module: `gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2` at `v2.20260926.0`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
SPDX-License-Identifier: MIT
|
||||
|
||||
License and notices: [MIT](licenses/MIT.txt) |
|
||||
[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/48918a3ad39fd99843cda94a86133b1b3881d596/COPYRIGHT) |
|
||||
[LICENSE](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/48918a3ad39fd99843cda94a86133b1b3881d596/LICENSE).
|
||||
|
||||
[Source](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/48918a3ad39fd99843cda94a86133b1b3881d596).
|
||||
[Source](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b).
|
||||
|
||||
## jsonschema/v6
|
||||
|
||||
Selected module: `github.com/santhosh-tekuri/jsonschema/v6` at `v6.0.2`.
|
||||
Selected module: `github.com/santhosh-tekuri/jsonschema/v6` at `v6.0.3`.
|
||||
|
||||
SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
License and notices: [Apache 2.0](licenses/Apache-2.0.txt) |
|
||||
[LICENSE](https://github.com/santhosh-tekuri/jsonschema/blob/29cbed948d24a04700eb94436416b18a07953b71/LICENSE).
|
||||
[LICENSE](https://github.com/santhosh-tekuri/jsonschema/blob/b0fc661f4939578bc429f408c18202573dbafcc4/LICENSE).
|
||||
|
||||
[Source](https://github.com/santhosh-tekuri/jsonschema/tree/29cbed948d24a04700eb94436416b18a07953b71).
|
||||
[Source](https://github.com/santhosh-tekuri/jsonschema/tree/b0fc661f4939578bc429f408c18202573dbafcc4).
|
||||
|
||||
## golang.org/x/text
|
||||
|
||||
Selected module: `golang.org/x/text` at `v0.14.0`.
|
||||
Selected module: `golang.org/x/text` at `v0.44.0`.
|
||||
|
||||
Copyright (c) 2009 The Go Authors. All rights reserved.
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
SPDX-License-Identifier: BSD-3-Clause
|
||||
|
||||
License and notices: [BSD 3-Clause](licenses/BSD-3-Clause.txt) |
|
||||
[LICENSE](https://go.googlesource.com/text/+/6c97a165dd661335ff7bce6104a008558123c353/LICENSE).
|
||||
License and notices: [BSD 3-Clause](licenses/go-x-text-BSD-3-Clause.txt) |
|
||||
[LICENSE](https://go.googlesource.com/text/+/6070ac32abcdd1670599eeb3ee4d513c8184a513/LICENSE) |
|
||||
[PATENTS](https://go.googlesource.com/text/+/6070ac32abcdd1670599eeb3ee4d513c8184a513/PATENTS).
|
||||
|
||||
[Source](https://go.googlesource.com/text/+/6c97a165dd661335ff7bce6104a008558123c353).
|
||||
|
||||
## gopkg.in/yaml.v3
|
||||
|
||||
Selected module: `gopkg.in/yaml.v3` at `v3.0.1`.
|
||||
|
||||
Copyright (c) 2006-2010 Kirill Simonov
|
||||
|
||||
Copyright (c) 2006-2011 Kirill Simonov
|
||||
|
||||
Copyright (c) 2011-2019 Canonical Ltd
|
||||
|
||||
Copyright 2011-2016 Canonical Ltd.
|
||||
|
||||
SPDX-License-Identifier: MIT AND Apache-2.0
|
||||
|
||||
License and notices: [MIT](licenses/MIT.txt) |
|
||||
[Apache 2.0](licenses/Apache-2.0.txt) |
|
||||
[LICENSE](https://github.com/go-yaml/yaml/blob/f6f7691b1fdeb513f56608cd2c32c51f8194bf51/LICENSE) |
|
||||
[NOTICE](https://github.com/go-yaml/yaml/blob/f6f7691b1fdeb513f56608cd2c32c51f8194bf51/NOTICE).
|
||||
|
||||
[Source](https://github.com/go-yaml/yaml/tree/f6f7691b1fdeb513f56608cd2c32c51f8194bf51).
|
||||
[Source](https://go.googlesource.com/text/+/6070ac32abcdd1670599eeb3ee4d513c8184a513).
|
||||
|
||||
## go.yaml.in/yaml/v4
|
||||
|
||||
Selected module: `go.yaml.in/yaml/v4` at `v4.0.0-rc.6`.
|
||||
Selected module: `go.yaml.in/yaml/v4` at `v4.0.0`.
|
||||
|
||||
Copyright 2006-2010 Kirill Simonov
|
||||
|
||||
@@ -136,7 +87,7 @@ SPDX-License-Identifier: MIT AND Apache-2.0
|
||||
|
||||
License and notices: [MIT](licenses/MIT.txt) |
|
||||
[Apache 2.0](licenses/Apache-2.0.txt) |
|
||||
[LICENSE](https://github.com/yaml/go-yaml/blob/1c17b9cee72bf81ae73a8be50fbd7dd3a9985125/LICENSE) |
|
||||
[NOTICE](https://github.com/yaml/go-yaml/blob/1c17b9cee72bf81ae73a8be50fbd7dd3a9985125/NOTICE).
|
||||
[LICENSE](https://github.com/yaml/go-yaml/blob/f2f9563503f11b7a60da0c532175dcc693ce8d5a/LICENSE) |
|
||||
[NOTICE](https://github.com/yaml/go-yaml/blob/f2f9563503f11b7a60da0c532175dcc693ce8d5a/NOTICE).
|
||||
|
||||
[Source](https://github.com/yaml/go-yaml/tree/1c17b9cee72bf81ae73a8be50fbd7dd3a9985125).
|
||||
[Source](https://github.com/yaml/go-yaml/tree/f2f9563503f11b7a60da0c532175dcc693ce8d5a).
|
||||
|
||||
27
docs/oss/licenses/go-x-text-BSD-3-Clause.txt
Normal file
27
docs/oss/licenses/go-x-text-BSD-3-Clause.txt
Normal file
@@ -0,0 +1,27 @@
|
||||
Copyright 2009 The Go Authors.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google LLC nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Base Policy Version:** 1.3
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/agents.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/agents.md
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Base Policy Version:** 1.3
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-19
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/architecture.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/architecture.md
|
||||
|
||||
## Purpose And Ownership
|
||||
|
||||
@@ -50,22 +50,36 @@ inventory; [adapter internals](../internal/adapters.md) and
|
||||
## Framework And Application Boundaries
|
||||
|
||||
Scriptorium consumes
|
||||
[Promptkit v0.11.2](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/consumers/pkg-promptkit.md)
|
||||
[PromptKit v0.23.3](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/consumers/pkg-promptkit.md)
|
||||
through its supported public root package. Promptkit owns prompt execution,
|
||||
preparation, prompt and profile formats, built-in profiles, model-client
|
||||
behavior, validation, and framework defaults. Scriptorium owns application
|
||||
configuration, executable CLI and HTTP adapters, prepared-run and
|
||||
definition-inspection presentation, process behavior, and HTTP deployment
|
||||
policy. The [format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/formats.md)
|
||||
and [outbound integration contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.2/docs/integrations/openai-compatible-chat.md)
|
||||
behavior, validation, inspection semantics, and framework defaults. Scriptorium
|
||||
owns the application domain, executable CLI and HTTP adapters, process behavior,
|
||||
and HTTP deployment policy. It interprets resolved configuration and uses
|
||||
PromptKit's supported APIs and results to satisfy user requests. The [format
|
||||
reference](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md)
|
||||
and [outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
||||
remain Promptkit's contracts.
|
||||
|
||||
Scriptorium uses ConfigKit v0.4.2 for strict application YAML decoding,
|
||||
resolution, and validation flow. Scriptorium retains its application schema,
|
||||
defaults, normalization, validation rules, source selection, and CLI override
|
||||
mapping. ConfigKit does not own Promptkit framework sources or HTTP artifact
|
||||
containment. The [configuration contract](../config.md) owns externally
|
||||
observable configuration behavior.
|
||||
ConfigKit v1.2.1 owns configuration policy and the lifecycle through the final,
|
||||
fully resolved configuration handoff. Scriptorium supplies application schema
|
||||
and domain callbacks through ConfigKit's supported extension points and consumes
|
||||
the completed result without redefining effective configuration. PromptKit's
|
||||
configuration is also resolved through ConfigKit, with PromptKit supplying its
|
||||
domain schema and rules. The [configuration contract](../config.md) owns
|
||||
externally observable application behavior; PromptKit owns framework formats.
|
||||
|
||||
As decided in [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
|
||||
Scriptorium adopts CommandKit's native `ckconfig` and `ckprompt` interfaces to
|
||||
neutrally expose upstream features and semantics. The current CLI still uses
|
||||
custom presentation; the [adoption roadmap](../roadmap/commandkit-adoption.md)
|
||||
tracks its replacement. This decision complements ADRs 0004 and 0005.
|
||||
|
||||
The CLI exposes Promptkit's captured framework configuration inspection. Its
|
||||
`show` and `explain` reports use Promptkit's native redaction for classified
|
||||
credential-shaped fields, including historical values. This scoped behavior is
|
||||
not a general-purpose filter for arbitrary sensitive content and does not
|
||||
extend to application configuration or existing prompt/profile inspection.
|
||||
|
||||
Application code may import Promptkit's public root package, but not its
|
||||
`internal` packages or catalog modules. It must not recreate the removed
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Documentation Policy
|
||||
|
||||
**Base Policy Version:** 1.11
|
||||
**Base Policy Version:** 1.12
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/documentation.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/documentation.md
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -87,8 +87,8 @@ secret values.
|
||||
| Scriptorium-owned integration contracts | `docs/integrations/` | Scriptorium-owned executable and process contracts, logical output paths and schemas, media types, and compatibility. | PromptKit framework formats and outbound provider protocols, physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
|
||||
| Inventory of implemented internal packages | `docs/internal/overview.md` | Implemented packages and components, responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
|
||||
| Behavior and architecture of internal packages | Other files under `docs/internal/` | Implementation flow, collaborators, state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
|
||||
| Architectural decision records | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
|
||||
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status, sequencing, and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||
| Architectural decision records | `docs/adr/` | Records of significant architectural decisions and supporting rationale. | Ordinary design choices, current behavior reference, implementation status, task sequencing. |
|
||||
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, or future work; implementation status, sequencing, and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||
| Release-note history | `docs/releases/` | One checked-in summary per source release under the procedure; the note at its immutable tag is the release record. | Current commands, behavior, contracts, and compatibility definitions. |
|
||||
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other copyable or runnable files. | Field-by-field reference, command reference, prose explanation. |
|
||||
|
||||
@@ -154,8 +154,15 @@ Use sequentially numbered ADR filenames such as
|
||||
|
||||
Treat the decision content of an accepted ADR as immutable. When a decision
|
||||
changes, create a new ADR and update the earlier ADR's status to superseded.
|
||||
Rejected architectural alternatives belong in the ADR; rejected product ideas
|
||||
belong in the roadmap.
|
||||
|
||||
ADRs should be reserved for major architectural decisions, such as ownership
|
||||
boundaries, large-scale architectural features, or invariants that rise to
|
||||
the level of architectural significance. Routine choices, implementation
|
||||
decisions, and rejected suggestions do not warrant an ADR unless they
|
||||
otherwise represent a significant architectural decision.
|
||||
|
||||
Roadmaps should preserve plausible future work, not rejected product
|
||||
ideas.
|
||||
|
||||
## Maintenance
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Base Policy Version:** 1.0
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/meta.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/meta.md
|
||||
|
||||
## Origin And Local Authority
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Open Source Licensing And Attribution Policy
|
||||
|
||||
**Base Policy Version:** 1.8
|
||||
**Base Policy Version:** 1.9
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/oss.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/oss.md
|
||||
|
||||
## Purpose And Scope
|
||||
|
||||
@@ -64,6 +64,9 @@ individual packages.
|
||||
Use the upstream [application COPYRIGHT example](https://gitea.maximumdirect.net/eric/policy/src/commit/a469273adda336625f96b9d769ea27b33999f146/examples/licensing/application/COPYRIGHT)
|
||||
or [library COPYRIGHT example](https://gitea.maximumdirect.net/eric/policy/src/commit/a469273adda336625f96b9d769ea27b33999f146/examples/licensing/library/COPYRIGHT)
|
||||
to create the project’s root `COPYRIGHT` file.
|
||||
Use the COPYRIGHT examples exactly as provided, except for placeholder
|
||||
substitutions and changes expressly required or authorized by this policy or
|
||||
the author.
|
||||
Use verified project names, contribution years, and copyright holders.
|
||||
For another selected license, adapt the grant, SPDX expression, disclaimer, and
|
||||
text references together to reflect the author's selected grant.
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Release Policy
|
||||
|
||||
**Base Policy Version:** 1.4
|
||||
**Base Policy Version:** 1.7
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/release.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/release.md
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -12,6 +12,9 @@ This policy defines a proportionate workflow for validating and publishing an
|
||||
intended source revision under a fresh version. Releases should require little
|
||||
routine intervention, avoid repeated work, and preserve published contents.
|
||||
|
||||
Adopt stronger verification or reproducibility requirements through an explicit
|
||||
project-specific decision justified by the intended audience and support needs.
|
||||
|
||||
The repository-specific commands belong in `docs/release.md`. Historical release
|
||||
summaries belong in `docs/releases/`. The procedure must agree with the adopted
|
||||
policy and link to it rather than introduce additional release requirements.
|
||||
@@ -59,9 +62,14 @@ requires a deliberate update to the policy and procedure where needed.
|
||||
For releases at or after `v1.0.0`:
|
||||
|
||||
- a major release may make intentional incompatible changes;
|
||||
- a minor release may add backward-compatible behavior; and
|
||||
- a minor release may add backward-compatible behavior or raise the Go consumer
|
||||
minimum as described below; and
|
||||
- a patch release may make backward-compatible fixes.
|
||||
|
||||
In this context, "backward-compatible" means only that pre-existing, valid
|
||||
configurations will continue to work following the relase. Behavior need not
|
||||
remain strictly identical.
|
||||
|
||||
Before `v1.0.0`, a minor release may change a documented public, operational,
|
||||
configuration, data, integration, or compatibility contract when the release
|
||||
note explains the impact and required consumer action. A patch release must
|
||||
@@ -71,6 +79,36 @@ Additional compatibility promises and deliberate exceptions belong in the
|
||||
adopted policy or canonical contract documentation. Release notes explain
|
||||
changes; they do not redefine current compatibility contracts.
|
||||
|
||||
## Consumer Minimum And Release Toolchain
|
||||
|
||||
Distinguish the minimum toolchain supported for consumers from the toolchain
|
||||
maintainers use to build and validate a release. Record the consumer minimum in
|
||||
the language's standard metadata and explain its support meaning in canonical
|
||||
documentation. Define the selected release toolchain and its setup commands in
|
||||
`docs/release.md`. Toolchain selection and upgrade cadence are repository-specific
|
||||
choices.
|
||||
|
||||
For Go repositories, normally declare the initial stable release of the selected
|
||||
series as the consumer minimum, such as `go 1.26.0`. Use a higher patch minimum
|
||||
when a dependency or an identified correctness fix requires it, and document
|
||||
the reason.
|
||||
|
||||
For release-policy purposes, successful build and release validation on the
|
||||
selected Go patch release are sufficient to presume support across that Go
|
||||
series, subject to the declared consumer minimum and concrete evidence of
|
||||
incompatibility. Investigate specific compatibility concerns as they arise.
|
||||
|
||||
Use the selected release toolchain for candidate validation and release builds,
|
||||
including CI jobs performing those tasks. A newer validation patch version alone
|
||||
leaves the consumer minimum unchanged.
|
||||
|
||||
A minor release may raise the Go consumer minimum, including from one Go series
|
||||
to the next, both before and after `v1.0.0`. This is an explicit exception to the
|
||||
backward-compatibility requirement for minor releases. Patch releases preserve
|
||||
the consumer minimum. Explain an increase's impact and required consumer action
|
||||
in the release note. Other incompatible changes remain subject to the versioning
|
||||
rules above.
|
||||
|
||||
## Release Documentation And Preparation
|
||||
|
||||
Every new release must include `docs/releases/<version>.md` in the tagged
|
||||
@@ -186,7 +224,8 @@ release without replacing its contents or reusing its version.
|
||||
Maintain `docs/release.md` as a short, executable maintainer runbook. It must
|
||||
identify:
|
||||
|
||||
1. prerequisites, primary branch, and authoritative remote;
|
||||
1. prerequisites, selected release toolchain, primary branch, and authoritative
|
||||
remote;
|
||||
2. version and release-note preparation;
|
||||
3. one publication command with its selected local checks and automated guards;
|
||||
4. declared deliverables, required build targets, and automation responsibilities;
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Base Policy Version:** 1.3
|
||||
|
||||
**Last Updated**: 2026-09-28
|
||||
**Last Updated**: 2026-11-17
|
||||
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/testing.md
|
||||
**Upstream Source:** https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/testing.md
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -9,11 +9,31 @@ discoverability requirements.
|
||||
## Prepare The Candidate
|
||||
|
||||
Use a POSIX shell, Git with a configured tagger identity and access to
|
||||
`origin`, and a Go toolchain compatible with `go.mod`. The native host must
|
||||
`origin`, and the selected release toolchain, Go 1.27.4. The native host must
|
||||
support Go's race detector and have its required C compiler. If Git is
|
||||
configured to sign tags, its signing tool and key must be available. Run from the
|
||||
repository root. Provision the toolchain and module cache before offline
|
||||
validation; when dependencies are not cached:
|
||||
repository root. The [developer guide](development.md#baseline-validation)
|
||||
explains the separate Go consumer minimum; selecting a release toolchain does
|
||||
not raise that minimum.
|
||||
|
||||
Install Go 1.27.4 from the [official downloads](https://go.dev/dl/) or an
|
||||
equivalent package source before validation. Set `release_go_root` to its
|
||||
installation directory, replacing the illustrative path below, and select it
|
||||
in the release shell:
|
||||
|
||||
```sh
|
||||
release_go_root=/path/to/go1.27.4
|
||||
export PATH="$release_go_root/bin:$PATH"
|
||||
export GOTOOLCHAIN=local
|
||||
go version
|
||||
```
|
||||
|
||||
The version output should identify `go1.27.4`. Keep this selection and the
|
||||
hosted workflow's Go image in step when deliberately updating the release
|
||||
toolchain. Ordinary contributor checks may use another compatible toolchain.
|
||||
|
||||
Provision the module cache before offline validation; when dependencies are
|
||||
not cached:
|
||||
|
||||
```sh
|
||||
GOWORK=off go mod download
|
||||
|
||||
334
docs/roadmap/commandkit-adoption.md
Normal file
334
docs/roadmap/commandkit-adoption.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# CommandKit Adoption
|
||||
|
||||
## Status And Intent
|
||||
|
||||
Initial roadmap, prepared 2026-11-19. The product direction is agreed;
|
||||
implementation has not begun. This document owns the target and product intent,
|
||||
incorporating the decisions in the [migration notepad](commandkit-notepad.md).
|
||||
The notepad remains the discussion
|
||||
record. The [Narratio proposal](narratio-feature-request.md) supplies consumer
|
||||
acceptance scenarios for requirements discovery. The
|
||||
[implementation plan](implementation.md) owns the ordered stages and concrete
|
||||
adapter choices.
|
||||
|
||||
[ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md) records
|
||||
the native-interface adoption and ownership boundaries, complementing ADRs 0004
|
||||
and 0005.
|
||||
|
||||
Make Scriptorium a CommandKit-native executable interface over PromptKit,
|
||||
including ckconfig and ckprompt. Most subprocess consumers should be able to
|
||||
select a stored prompt and consume its resulting artifact directly. Operators
|
||||
and consumers needing inspection or metadata should use native upstream
|
||||
interfaces and reports.
|
||||
|
||||
This is a full CLI cutover. Existing command names, flag spellings, parsing,
|
||||
report shapes, and exit conventions are not compatibility constraints. Release
|
||||
migration guidance replaces a compatibility layer. Implementation may proceed
|
||||
in reviewable increments; this does not require shipping both interfaces.
|
||||
|
||||
## Current Baseline And Upstream References
|
||||
|
||||
Reviewed Scriptorium baseline: `5f32a4f2adb2d382a6cfce3a787ada4576be7705`.
|
||||
The module uses Go 1.27.0, PromptKit v0.23.3, and ConfigKit v1.2.1. CommandKit
|
||||
is not yet a dependency.
|
||||
|
||||
| Current implementation | Migration work |
|
||||
| --- | --- |
|
||||
| `internal/adapter/cli` dispatches and parses commands, resolves configuration, builds engines and requests, writes diagnostics, and returns integer statuses. | Replace dispatch and shared CLI mechanics with CommandKit; retain application loading, request mapping, and effects through its extension points. |
|
||||
| `run` uses flag-based prompt selection and writes generated content; `--execution-report` writes an observer-derived Scriptorium report. | Adopt positional prompt selection, native request flags, and native result reporting. |
|
||||
| `render`, `inspect prompt`, and `inspect profile` use custom presentation in `internal/format`. | Replace with native ckprompt operations and remove unused presentation code. |
|
||||
| `config` and `inspect configuration` separately expose application and framework configuration. | Install one ckconfig family with `app` and `promptkit` scopes, including comparison. |
|
||||
| `internal/config` owns application schema, discovery, defaults, and conversion through ConfigKit. | Adapt its loader and explicit overrides to standard selections and scope capabilities. |
|
||||
| `serve` constructs a restricted HTTP artifact reader and one shared PromptKit engine. | Register it as an application command and preserve existing HTTP behavior. |
|
||||
| `internal/report` also supplies HTTP's optional execution report. | Remove its superseded CLI use; retain functionality still used by HTTP. |
|
||||
|
||||
The [current CLI](../cli.md), [configuration contract](../config.md),
|
||||
[adapter internals](../internal/adapters.md), and
|
||||
[subprocess contract](../integrations/subprocess.md) describe the starting
|
||||
behavior, not a compatibility checklist for this migration.
|
||||
|
||||
The reviewed upstream combination is CommandKit v0.7.0, ckconfig v0.9.0,
|
||||
and ckprompt v0.10.0, documented at revision
|
||||
`d330d7637f2e7507eca931a899397222c82bc892`:
|
||||
|
||||
- [Consumer guide][upstream-guide]: composition, loading, reusable bindings,
|
||||
reporting, and outcomes.
|
||||
- [CLI reference][upstream-cli]: native command grammar and observable behavior.
|
||||
|
||||
The selected released combination retains Scriptorium's adopted PromptKit
|
||||
v0.23.3 fixes and ConfigKit v1.2.1. The reviewed ckprompt release's PromptKit
|
||||
v0.23.2 minimum does not require a downgrade. Supported public APIs and released
|
||||
dependencies remain subject to the existing
|
||||
[architecture policy](../policy/architecture.md).
|
||||
|
||||
No selected-format accessor is required for this migration. The native
|
||||
invocation reporter handles the main result on stdout; the existing writer API
|
||||
handles the separate report with automatic or explicitly selected rendering.
|
||||
The upstream accessor proposal remains optional API work, independent of
|
||||
Scriptorium's adoption.
|
||||
|
||||
## Target End State
|
||||
|
||||
### Responsibilities And Scope
|
||||
|
||||
CommandKit owns command declarations, dispatch, common parsing, help, reporting,
|
||||
and outcome mechanics. ckconfig and ckprompt supply their standard families.
|
||||
ConfigKit owns configuration policy through the fully resolved handoff,
|
||||
including resolution of PromptKit's configuration using PromptKit's schema and
|
||||
rules. PromptKit owns definition, preparation, execution, validation, and
|
||||
inspection semantics. Scriptorium owns the application domain: it interprets
|
||||
resolved values and uses upstream APIs and results to satisfy user requests.
|
||||
Application commands, process lifecycle, and artifact delivery remain local.
|
||||
|
||||
This preserves Scriptorium's request-independent execution model and support
|
||||
for valid definitions under [ADR 0004](../adr/0004-definition-boundary.md).
|
||||
It adds no workflow, conversation, archive, or resume system. HTTP design review
|
||||
remains separate work before v1.0; migrating its startup command does not
|
||||
redesign the HTTP API or its execution-report representation.
|
||||
|
||||
Use shared upstream capabilities where available. A concrete missing shared
|
||||
capability should prompt an upstream discussion before a local substitute.
|
||||
Application-specific loading and effects remain ordinary application work.
|
||||
|
||||
### Command Surface
|
||||
|
||||
| Surface | Target |
|
||||
| --- | --- |
|
||||
| Generation | Application command `run PROMPT_ID`, with a required positional prompt ID and no duplicate prompt flag. |
|
||||
| HTTP service | Application command `serve`. |
|
||||
| Configuration | Native `config` family with `app` and `promptkit` scopes: scope discovery, validation, effective values, sources, explanations, and diff. |
|
||||
| Resource inspection | Native prompt, profile, backend, namespace, and collection inventories and applicable named inspection. |
|
||||
| Dependency discovery | Native `promptkit requirements show PROMPT_ID`. |
|
||||
| Concrete validation | Native `promptkit requirements validate` capability. |
|
||||
| Prepared preview | Native `promptkit show prepared PROMPT_ID`, replacing `render`. |
|
||||
| Provider diagnostics | Native smoke-test commands, invoked explicitly. |
|
||||
| Executable support | Native help, version, reporting modes, outcomes, and shell completion. |
|
||||
|
||||
Install application commands outside the reserved standard families. Enable
|
||||
the native convenience aliases consistently. Prefer shorter spellings such as
|
||||
`--profile`, `--input-file`, `--input-text`, and `--var` in examples and guidance
|
||||
where applicable and unambiguous; canonical `--llm-*` spellings remain available.
|
||||
Aliases retain upstream semantics: `--input` supplies literal text, and file
|
||||
references use `--input-file`.
|
||||
|
||||
Reuse applicable ckprompt resource, input, and execution bindings for `run`.
|
||||
Preserve explicit-value presence, including zero and empty values, and native
|
||||
repeated mapping semantics. Do not carry forward comma splitting or obsolete
|
||||
aliases merely because the old parser accepted them. Keep application source
|
||||
selectors, server settings, output destinations, and whole-operation deadlines
|
||||
distinct from framework request controls.
|
||||
|
||||
Completion initially uses native command, flag, static-value, and path
|
||||
completion, with suitable application hints. Dynamic installed-resource
|
||||
completion is not required. Smoke tests are explicit operator operations;
|
||||
their inclusion does not make provider access a startup or release prerequisite.
|
||||
Preserve native diagnostic findings, including successful invocations whose
|
||||
reports describe unsupported or unsuccessful checks.
|
||||
|
||||
### Configuration Authority And Native Inspection
|
||||
|
||||
The migration implements the ownership decision in
|
||||
[ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md) through
|
||||
these two boundaries:
|
||||
|
||||
1. **ConfigKit's completed result is authoritative configuration.** Scriptorium
|
||||
supplies its schema, domain defaults, validation, and genuine application
|
||||
configuration-resolution logic through ConfigKit's supported extension
|
||||
points. Those callbacks operate within ConfigKit's lifecycle and evidence
|
||||
model, including recording resolution changes through the supported APIs.
|
||||
Scriptorium consumes the completed result without subsequently revising what
|
||||
effective configuration means. Runtime conversion and request construction
|
||||
may consume resolved values; they do not introduce another configuration
|
||||
merge, defaulting, or normalization pass. This follows the configuration
|
||||
boundary established in [ADR 0005](../adr/0005-configkit-boundary.md).
|
||||
2. **Standard inspection exposes that result through native adapters.**
|
||||
Scriptorium selects and acquires the requested configuration and declares
|
||||
supported capabilities. Use `ckconfig.FromConfigKit` for the application
|
||||
result and `ckprompt.ConfigScope` for PromptKit's captured configuration
|
||||
inspection within the native `ckconfig` family. Scriptorium does not assemble
|
||||
a different effective document, reconcile separate explanation trees, or
|
||||
reinterpret upstream path semantics. Effective values, sources, explanations,
|
||||
metadata, and comparisons come from the acquired upstream result through
|
||||
those adapters. CommandKit owns their standard presentation.
|
||||
|
||||
Application selection and supported configuration overrides feed the loading
|
||||
lifecycle before its completed result is exposed. Adapting loaders or supplying
|
||||
scopes below does not authorize a second configuration-resolution or inspection
|
||||
model. The `app` and `promptkit` scopes retain their respective upstream evidence;
|
||||
they are not combined into a synthetic effective configuration.
|
||||
|
||||
### Configuration, Collections, And Request Context
|
||||
|
||||
Expose application configuration through `app` and captured framework
|
||||
configuration through `promptkit`. Collections are resources within the
|
||||
framework configuration, not separate configuration scopes. Application-only
|
||||
inspection must work without constructing a PromptKit engine.
|
||||
|
||||
`--config` selects Scriptorium application configuration on every applicable
|
||||
command, including framework inspection. That configuration selects framework
|
||||
sources. Adopt native `--overlay` selection and upstream selection presence
|
||||
semantics. For ConfigKit v1.2.1 the supplied value selects an overlay by name,
|
||||
not an arbitrary file path. Map selections through the application's ConfigKit
|
||||
loader rather than retaining the old flag parser as a compatibility layer.
|
||||
Preserve useful application source and server controls through appropriate selectors.
|
||||
|
||||
Enable temporary configuration overrides where the loader supports them,
|
||||
passing native override records into loading. The reviewed PromptKit scope
|
||||
does not support these overrides; do not emulate them locally. Configuration
|
||||
comparison must support independent selections for both sides in both scopes,
|
||||
including corresponding application selectors where needed.
|
||||
|
||||
Keep the established collection-discovery model: ConfigKit imports discover
|
||||
`collection.yml` and `collection.yaml` entry points; each independently
|
||||
maintained collection declares its namespace and imports its own definitions
|
||||
through relative references. Retain the
|
||||
[discovery example](../../examples/config.discovery.yml) and representative
|
||||
multi-definition collection documentation. No Git update manager or custom
|
||||
collection traversal is part of this migration.
|
||||
|
||||
Use native prompt, profile, and backend namespace selectors independently on
|
||||
applicable commands. Short caller-supplied IDs use their selected namespace;
|
||||
qualified IDs remain exact. Selection does not rewrite stored references.
|
||||
Retain upstream inventory filters and defaults, including `--all-namespaces`.
|
||||
Do not add a configurable default namespace in this migration.
|
||||
|
||||
Share applicable configuration selection, engine loading, and request mapping
|
||||
across execution, preview, validation, and discovery. Preserve each operation's
|
||||
upstream lifecycle: discovery can work before artifacts exist; preparation
|
||||
loads and renders without generation. This is consistency of selection and
|
||||
mapping, not a promise that separate invocations see an immutable snapshot.
|
||||
|
||||
### Generation, Reports, And Process Outcomes
|
||||
|
||||
Use a main CommandKit `Report` whose value is PromptKit's native `RunResult` and
|
||||
whose text template emits only the resulting artifact content. The template
|
||||
should emit artifact bytes without presentation decoration. Default/text serves raw
|
||||
artifact consumers; JSON and YAML expose the full native result. The native
|
||||
structured serialization, including its representation of artifact bytes,
|
||||
remains upstream-owned. There is no Scriptorium result envelope.
|
||||
|
||||
| Selection | Behavior |
|
||||
| --- | --- |
|
||||
| Main result | Goes through the native invocation reporter to stdout using the selected format. |
|
||||
| `--format text` or default | Main result contains only the artifact. |
|
||||
| `--format json` or `--format yaml` | Main result contains the native result and metadata. |
|
||||
| `--report PATH` | Additionally write the native result through CommandKit's renderer, independently of the main format and destination. Without `--report-format`, use native automatic rendering. |
|
||||
| `--report-format text\|yaml\|json` | Explicitly select the separate report's native rendering format. |
|
||||
|
||||
The separate report contains the same native `RunResult`, without the main
|
||||
report's artifact-only text template. Omission of `--report-format` leaves the
|
||||
format unset so CommandKit decides how to render it. Its current automatic
|
||||
rendering for structured values is YAML; future upstream default changes flow
|
||||
through. Scriptorium does not set a default report format.
|
||||
|
||||
There is no `--output` or legacy `--out` flag. Callers save the main result by
|
||||
redirecting stdout in a shell or supplying a file as the subprocess stdout
|
||||
destination. Diagnostics remain on stderr. Callers wanting main output omit
|
||||
`--quiet`, including when stdout is redirected. Filename extensions do not
|
||||
select formats. The separate report includes the native result's content as
|
||||
well as metadata; it replaces the CLI's old
|
||||
observer-derived metadata projection. A caller wanting an artifact file plus
|
||||
metadata redirects stdout and selects a separate `--report` path. The caller
|
||||
manages redirected destinations and retention; there is no automatic reporting
|
||||
directory or report collection.
|
||||
|
||||
Native result reporting is available when PromptKit returns a result.
|
||||
Operational errors return nonzero status and native failure diagnostics, without
|
||||
synthesizing replacement results. Preserve PromptKit's distinction between a
|
||||
completed content check reporting `ValidationFailed` and an operational failure
|
||||
to perform the check. Neither extra content judgments nor a structured JSON
|
||||
invocation-error envelope is part of this migration.
|
||||
|
||||
Adopt CommandKit's quiet and verbosity semantics, including rendering and
|
||||
presentation failures. Console results use the invocation reporter; diagnostics
|
||||
use its diagnostic channel. An explicit `--report` write is an application
|
||||
effect rendered through the writer API and still occurs in quiet mode.
|
||||
Quiet does not cancel generation; it suppresses the main result even when
|
||||
stdout is redirected, while failure explanations remain visible. Move any
|
||||
retained generation summary into an appropriate native message category.
|
||||
|
||||
Map typed upstream failures and configuration diagnostics into native outcomes,
|
||||
without parsing error text. Standard statuses remain upstream-owned. Use an
|
||||
application domain outcome for completed validation failure, distinct from
|
||||
usage and diff-check outcomes. The implementation plan specifies the mapping.
|
||||
Delivery failures must produce a nonzero outcome and explanation, preserving
|
||||
native presentation-failure behavior where applicable, including an already
|
||||
nonzero domain outcome. Requested file errors must not disappear merely because
|
||||
they occur outside the invocation reporter.
|
||||
|
||||
For `--report`, use ordinary file replacement without creating parents. Attempt
|
||||
the main output followed by the report, and attempt the report even if main
|
||||
delivery fails. Open the report only when a native result exists. Redirection
|
||||
and its file-creation or truncation behavior belong to the caller. Scriptorium
|
||||
does not manage the main output file or check it against the report path. This
|
||||
adds no rollback or transactional delivery; written bytes may remain after
|
||||
failure.
|
||||
|
||||
Wire signal cancellation through the invocation context. Keep the application
|
||||
whole-operation budget separate from the native per-attempt timeout binding.
|
||||
Use `--run-timeout` with Go duration values mapped to PromptKit's
|
||||
`OperationTimeout`, preserving upstream zero, negative-value, and timeout
|
||||
behavior without a second timeout policy.
|
||||
|
||||
### Preview And Consumer Inspection
|
||||
|
||||
Remove `render`, its compatibility aliases, `defaults.render_format`, and
|
||||
unused prepared-output formatters. Native prepared preview uses its upstream
|
||||
report, default/text YAML presentation, standard formats, request flags,
|
||||
diagnostics, and outcomes. It does not call a model. Consumers save preview
|
||||
output through stdout capture or redirection, as they do for generation.
|
||||
|
||||
Adopt native inspection reports, including captured prompt messages and rendered
|
||||
prepared content, without recreating old summary projections. Native schemas
|
||||
and adopted upstream versions govern structured output; this migration adds no
|
||||
separate schema-version wrapper or compatibility guarantee.
|
||||
|
||||
Implement the [Narratio acceptance scenarios](narratio-feature-request.md#consumer-acceptance-scenarios).
|
||||
Requirements discovery exposes native classification, completeness, and issue
|
||||
codes. Complete discovery is distinct from satisfied bindings and execution
|
||||
readiness; partial discovery can succeed with status zero. Narratio owns its
|
||||
pipeline, binding graph, and policy for incomplete findings. Preserve native
|
||||
metadata without inventing prompt versions, variable defaults, or input schemas.
|
||||
|
||||
## Acceptance And Scope Boundaries
|
||||
|
||||
The target is met when Scriptorium exposes the native command surface and
|
||||
application behavior above, superseded CLI machinery is retired, and maintained
|
||||
documentation and examples describe the shipped interface. In particular:
|
||||
|
||||
- Configuration callbacks, runtime consumers, and native inspection agree on
|
||||
ConfigKit's completed values and evidence. Application-only inspection needs
|
||||
no PromptKit engine, and comparison sides retain independent selections.
|
||||
- Applicable selection and request context remain consistent across execution,
|
||||
preparation, discovery, and inspection of multiple independently maintained
|
||||
collections. Their upstream lifecycle differences remain visible.
|
||||
- Generation defaults to the artifact alone; structured output and separate
|
||||
reports expose the native result. Main output uses stdout and native quiet
|
||||
semantics; the optional report file uses native automatic or explicitly
|
||||
selected rendering, independently of quiet, with useful failure outcomes and
|
||||
nontransactional delivery.
|
||||
- Narratio's linked acceptance scenarios are supported, including useful partial
|
||||
discovery and discovery before generated artifacts exist.
|
||||
- Native help, aliases, completion, and explicit smoke testing are available.
|
||||
Including smoke testing adds no provider access prerequisite to startup or
|
||||
release validation.
|
||||
- Server startup reaches the existing HTTP adapter with its transport,
|
||||
restricted-reading, execution-report, and shared-capacity behavior intact.
|
||||
- Canonical contracts, examples, dependency attribution, and migration guidance
|
||||
reflect the cutover. The CLI migration includes positional prompt selection,
|
||||
command replacements, literal versus file inputs, repeated mappings, overlay
|
||||
selection, native endpoint/timeout controls, stdout redirection in place of
|
||||
output-file flags, content-bearing reports, optional report-format selection,
|
||||
report availability, quiet behavior, and changed exit statuses.
|
||||
|
||||
The [implementation plan](implementation.md) owns task sequencing, concrete
|
||||
adapter mappings, verification steps, and sprint status. There are no unresolved
|
||||
product decisions or selected-format accessor release prerequisite.
|
||||
|
||||
The [release policy](../policy/release.md#versioning-and-compatibility) governs
|
||||
the eventual breaking release version. This migration does not itself select
|
||||
or publish a release, introduce a compatibility layer, or settle the separate
|
||||
pre-v1.0 HTTP design review.
|
||||
|
||||
[upstream-guide]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/consumers/guide.md
|
||||
[upstream-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md
|
||||
331
docs/roadmap/commandkit-notepad.md
Normal file
331
docs/roadmap/commandkit-notepad.md
Normal file
@@ -0,0 +1,331 @@
|
||||
# CommandKit migration notepad
|
||||
|
||||
Status: discussion record incorporated into the
|
||||
[adoption roadmap](commandkit-adoption.md) and
|
||||
[ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md).
|
||||
The roadmap owns implementation work; the ADR records the ownership decision.
|
||||
|
||||
## Product context
|
||||
|
||||
Scriptorium is a neutral executable interface over PromptKit. Its primary
|
||||
consumers are applications that need to run stored prompts and consume their
|
||||
output without importing PromptKit or building an internal prompting workflow.
|
||||
The CLI and its subprocess contract are therefore central to this migration.
|
||||
|
||||
An installation may load many independently maintained collections. The
|
||||
existing collection model remains the starting point: ConfigKit imports discover
|
||||
`collection.yml` or `collection.yaml` entry points, each collection declares its
|
||||
own namespace, and relative imports assemble its profiles and prompts. See the
|
||||
[configuration contract](../config.md) and
|
||||
[discovery example](../../examples/config.discovery.yml).
|
||||
|
||||
HTTP remains secondary to the CLI discussion. A broader HTTP design review is
|
||||
planned before v1.0; adopting CommandKit does not itself settle that design.
|
||||
|
||||
## Agreed direction
|
||||
|
||||
### Full native cutover
|
||||
|
||||
Decision recorded 2026-11-19: become a CommandKit-native application, including
|
||||
`ckconfig` and `ckprompt`, and adopt upstream public conventions and
|
||||
recommendations as fully as possible.
|
||||
|
||||
Backward compatibility with existing CLI conventions is an explicit non-goal.
|
||||
Legacy command names, flag spellings, parsing rules, report shapes, and exit
|
||||
semantics do not constrain the new interface. Do not add compatibility wrappers,
|
||||
legacy aliases, or duplicate inspection implementations solely to preserve the
|
||||
old CLI. Adopt upstream convenience aliases as described below.
|
||||
|
||||
Provide migration guidance in release documentation so consumers can update
|
||||
their invocations and output handling. Guidance replaces a compatibility layer;
|
||||
it does not require a transition period with both interfaces. The
|
||||
[release policy](../policy/release.md#versioning-and-compatibility) governs
|
||||
release version selection and documentation of breaking changes.
|
||||
|
||||
### Ownership and integration
|
||||
|
||||
Use CommandKit for command declarations, parsing, help, reporting, and outcomes;
|
||||
use `ckconfig` and `ckprompt` for their standardized operator interfaces.
|
||||
Scriptorium should retire the corresponding custom inspection machinery as
|
||||
those interfaces take over.
|
||||
|
||||
ConfigKit owns configuration policy through the fully resolved handoff,
|
||||
including PromptKit configuration resolution. PromptKit owns prompting and
|
||||
inspection semantics. Scriptorium interprets resolved values and uses upstream
|
||||
APIs and results to satisfy user requests. CommandKit adoption does not introduce
|
||||
workflow or conversation state. ADR 0006 records these boundaries.
|
||||
|
||||
Prefer shared upstream capabilities over new Scriptorium-specific equivalents.
|
||||
If a required shared capability is missing, discuss an upstream request before
|
||||
designing a local substitute. Application-specific behavior still belongs in
|
||||
Scriptorium.
|
||||
|
||||
### Initial capability set
|
||||
|
||||
Decision recorded 2026-11-19: include configuration comparison, provider smoke
|
||||
tests, and shell completion in the initial migration, alongside application and
|
||||
PromptKit configuration scopes, resource inventories and inspection,
|
||||
requirements discovery, concrete request validation, and prepared-request
|
||||
preview.
|
||||
|
||||
Use the existing upstream capabilities. Configuration comparison should cover
|
||||
both application and PromptKit scopes, with independent selections for each
|
||||
side. Provider smoke tests run when explicitly invoked; including them does not
|
||||
make them startup checks or release prerequisites. Document provider usage and
|
||||
the distinction between successful diagnostic execution and positive findings.
|
||||
Shell completion should use upstream command, flag, static-value, and path
|
||||
completion, with appropriate hints for application-owned inputs. Dynamic
|
||||
completion of installed resource names is a separate possible enhancement.
|
||||
|
||||
Application-specific wiring remains to be designed, but inclusion of these
|
||||
capabilities is settled. The prepared-request preview decision is recorded below.
|
||||
|
||||
### Generation and subprocess requirements
|
||||
|
||||
Decision recorded 2026-11-19: optimize the default generation response for the
|
||||
majority of consumers, who want only the raw resulting artifact. Metadata is
|
||||
optional and available through the native result reporting choices below.
|
||||
|
||||
Prefer native CommandKit reporting and PromptKit result types before introducing
|
||||
a Scriptorium-specific response format. A failure must produce a nonzero process
|
||||
status and information explaining the failure. This does not require a new
|
||||
structured error envelope: native diagnostics are an acceptable way to deliver
|
||||
that information.
|
||||
|
||||
Scriptorium remains a neutral interface to PromptKit. Pass through the errors,
|
||||
validation outcomes, and result information that PromptKit supplies rather than
|
||||
adding independent judgments about generated content or redefining upstream
|
||||
failure semantics. Adapting those outcomes to CommandKit and process exit codes
|
||||
is application work; deciding what constitutes a PromptKit failure is not.
|
||||
|
||||
### Native result reporting and separate report files
|
||||
|
||||
Decision revised 2026-11-19: use a main CommandKit `Report` containing PromptKit's
|
||||
native `RunResult`, with a text template that emits only generated content.
|
||||
Default/text output serves artifact consumers; standard JSON/YAML formats expose
|
||||
the native result and its metadata. This composes existing upstream APIs;
|
||||
ckprompt does not supply a general generation command or prescribe this template.
|
||||
|
||||
Include an optional `--report PATH` flag requiring an explicit file destination.
|
||||
Omitting it creates no separate report file. When requested, pass the native
|
||||
`RunResult` to CommandKit's renderer without an artifact-only text template.
|
||||
Optional `--report-format text|yaml|json` selects the native rendering format.
|
||||
When omitted, leave the format unset and let CommandKit choose; its current
|
||||
structured-value default is YAML, and future upstream changes flow through.
|
||||
There is no Scriptorium default or filename-based selection. Report rendering
|
||||
is independent of the main result's `--format`, representation, and destination.
|
||||
Failure to write a requested report is a delivery failure and must produce a
|
||||
nonzero outcome and diagnostic.
|
||||
|
||||
Do not include a configured reporting directory or automatic report collection
|
||||
in the initial migration. The caller selects the report path and manages its
|
||||
lifecycle. A configured directory may be considered later if automatic
|
||||
collection becomes useful; an explicit report path would take precedence.
|
||||
|
||||
Native result reports are available when PromptKit returns a result. Operational
|
||||
failures produce a nonzero exit and native diagnostics. This follows PromptKit's
|
||||
capabilities and is the intended boundary, not a gap for Scriptorium to address.
|
||||
Do not synthesize replacement results or add an alternative failure-report
|
||||
format to compensate for the absence of a native result.
|
||||
|
||||
### Output destinations and verbosity
|
||||
|
||||
Decision revised 2026-11-19: remove the previously selected `--output` flag.
|
||||
The main result uses the native invocation reporter and stdout. Callers save it
|
||||
through shell redirection or a subprocess stdout destination; diagnostics remain
|
||||
on stderr. Do not retain the old `--out` flag.
|
||||
|
||||
`--format` selects the stdout representation independently of redirection:
|
||||
default/text emits the raw artifact, while JSON/YAML emits the native result
|
||||
with metadata. The filename does not select the format. `--report PATH` remains
|
||||
an additional native result report, so callers can save the raw artifact and its
|
||||
report to separate files by redirecting stdout and supplying `--report`.
|
||||
|
||||
Adopt CommandKit's upstream verbosity and `--quiet` semantics fully, including
|
||||
mode selection, message visibility, rendering, and presentation failures. Do
|
||||
not add Scriptorium-specific exceptions to normal command-output suppression.
|
||||
Quiet mode does not cancel generation, and failure explanations remain visible
|
||||
through the native diagnostic channel.
|
||||
|
||||
An explicit `--report` write remains an application-owned effect in quiet mode,
|
||||
using the existing writer API. Quiet suppresses the main result even when stdout
|
||||
is redirected; callers wanting that result omit quiet. No
|
||||
selected-format accessor is needed, so the upstream proposal is no longer a
|
||||
migration prerequisite. The [upstream reporting contract][upstream-cli] owns the
|
||||
shared verbosity rules.
|
||||
|
||||
### Prepared-request preview
|
||||
|
||||
Decision recorded 2026-11-19: replace `render` entirely with the native
|
||||
`promptkit show prepared PROMPT_ID` command, enabled through ckprompt's request
|
||||
preparation integration. Remove `render` without a compatibility alias.
|
||||
|
||||
Use the native prepared-run report, standard request flags, diagnostics,
|
||||
verbosity, and outcomes. Default and explicit text output use the upstream YAML
|
||||
presentation; JSON remains available through `--format json`. Remove
|
||||
`defaults.render_format` and any custom prepared-output formatting code left
|
||||
unused by the migration.
|
||||
|
||||
Share application configuration selection, collection loading, and applicable
|
||||
request construction with generation so previews describe the corresponding
|
||||
execution request. PromptKit continues to own preparation semantics.
|
||||
|
||||
Use the native result stream for preview output; consumers can capture or
|
||||
redirect stdout to save it, as with generation. File-output convenience does
|
||||
not justify retaining a separate preview command. Update maintained examples
|
||||
and release migration guidance to use `promptkit show prepared`.
|
||||
|
||||
### Configuration selection and namespaces
|
||||
|
||||
Decision recorded 2026-11-19: fully adopt native configuration and namespace
|
||||
conventions. Expose the `app` and `promptkit` configuration scopes through
|
||||
ckconfig. `app` inspects Scriptorium's application settings; `promptkit`
|
||||
inspects the framework configuration assembled from the selected sources.
|
||||
Collections do not become separate configuration scopes. Application-only
|
||||
inspection must not require constructing a PromptKit engine.
|
||||
|
||||
`--config` consistently selects Scriptorium's application configuration,
|
||||
including when inspecting the `promptkit` scope. That configuration selects
|
||||
the framework sources and collection-discovery file. Adopt the standard
|
||||
`--overlay` spelling and use consistent source-selection and loading rules
|
||||
across generation, preview, inspection, requirements, and other applicable
|
||||
commands. Keep the established ConfigKit collection-import model.
|
||||
|
||||
Enable the native prompt, profile, and backend namespace selectors on
|
||||
applicable commands. They independently scope short caller-supplied resource
|
||||
IDs; qualified IDs remain exact selections. Namespace selection does not
|
||||
rewrite stored references or change collection discovery. Keep configuration
|
||||
scope, resource namespace, and profile selection distinct.
|
||||
|
||||
Use native inventory filters, including namespace selection and
|
||||
`--all-namespaces`, and retain upstream default listing behavior. Support
|
||||
standard temporary configuration overrides where the upstream loader supports
|
||||
them. The reviewed PromptKit configuration scope does not support those
|
||||
overrides; do not add local override semantics to compensate.
|
||||
|
||||
Do not add configurable default namespaces in the initial migration. Use
|
||||
qualified IDs and explicit namespace flags when selecting namespaced resources,
|
||||
while retaining upstream behavior for unqualified IDs. Configurable defaults
|
||||
can be considered later if a need arises.
|
||||
|
||||
### Native inspection reports and consumer requirements
|
||||
|
||||
Decision recorded 2026-11-19: adopt native inspection reports and the native
|
||||
`promptkit requirements show` interface for Narratio's early dependency
|
||||
discovery. Its consumer needs and acceptance scenarios are recorded in the
|
||||
[aligned Narratio proposal](narratio-feature-request.md). This direction is
|
||||
accepted; Scriptorium integration remains to be implemented.
|
||||
|
||||
Consume native report fields without a Scriptorium-specific schema-version
|
||||
envelope. Use native nonzero outcomes and diagnostics for invocation failures;
|
||||
JSON report selection does not require JSON errors. Neither a separately
|
||||
versioned envelope nor programmatic invocation-failure categories blocks this
|
||||
migration. A future concrete need for either belongs in an upstream discussion.
|
||||
|
||||
PromptKit owns requirement classification, discovery completeness, and structured
|
||||
discovery issues. Completeness is distinct from satisfied bindings or execution
|
||||
readiness; Narratio owns comparison against its planned inputs. Keep early
|
||||
discovery separate from concrete request validation. Use current exact resource
|
||||
identities and native input/variable metadata, without restoring obsolete prompt
|
||||
versions or inventing input schemas or variable-default semantics.
|
||||
|
||||
Standard prompt inspection includes captured messages; prepared preview includes
|
||||
rendered content. Adopt these native reports without rebuilding Scriptorium's
|
||||
previous summary projections, and explain the changed presentation in migration
|
||||
guidance. Requirements discovery retains its distinct upstream content boundary.
|
||||
|
||||
### Application commands and standard request flags
|
||||
|
||||
Decision recorded 2026-11-19: use `run PROMPT_ID` for generation, with the prompt
|
||||
as a required positional operand and no duplicate prompt-selection flag. Keep
|
||||
`serve` as the application command for the HTTP service. Both are CommandKit
|
||||
commands owned by Scriptorium, outside the reserved `config` and `promptkit`
|
||||
families. The CLI migration does not redesign the HTTP API.
|
||||
|
||||
Reuse ckprompt's applicable resource, input, and execution flag bindings.
|
||||
Preserve native parsing, explicit-value presence, repeated mapping semantics,
|
||||
and PromptKit's interpretation of request values. Keep application-specific
|
||||
configuration/source selection, `--report`/`--report-format`, server options,
|
||||
and the whole-operation deadline separate from framework request controls.
|
||||
|
||||
Enable upstream convenience aliases consistently on applicable commands,
|
||||
including `run`. Prefer their shorter spellings in user-facing documentation,
|
||||
examples, migration guidance, and subprocess integration examples wherever they
|
||||
are applicable and unambiguous. For example, prefer `--profile`, `--input-file`,
|
||||
`--input-text`, and `--var` over their `--llm-*` equivalents. Canonical upstream
|
||||
spellings remain available with identical semantics; the preference does not
|
||||
remove or rename them.
|
||||
|
||||
Aliases retain upstream meanings. In particular, `--input` supplies literal
|
||||
text; file references use `--input-file`. Migration guidance must explain this
|
||||
change along with endpoint and timeout naming and repeated mapping values.
|
||||
Use the canonical spelling where a shorter alias would be ambiguous; do not
|
||||
assign an upstream alias a different application-specific meaning.
|
||||
|
||||
## Upstream baseline
|
||||
|
||||
The initial review covered CommandKit **v0.7.0**, ckconfig **v0.9.0**, and
|
||||
ckprompt **v0.10.0**. These are a review baseline, not a final dependency choice
|
||||
for implementation. The [upstream consumer guide][upstream-guide] describes
|
||||
their composition and supported extension points.
|
||||
|
||||
| Component | Relevant capabilities |
|
||||
| --- | --- |
|
||||
| CommandKit | Typed flags and operands with explicit-value presence, generated help and version, shared reporting and outcomes, optional shell completion. |
|
||||
| ckconfig | Configuration scopes; validation, effective values, sources, explanations, and comparisons; temporary overrides where supported by the loader. |
|
||||
| ckprompt | Prompt, profile, backend, namespace, and collection inventories; resource inspection; requirements discovery; optional concrete request validation, prepared-request previews, and provider smoke tests. Reusable request flags also serve application-owned commands. |
|
||||
|
||||
Scriptorium currently uses PromptKit v0.23.3 and ConfigKit v1.2.1. The reviewed
|
||||
ckprompt release declares PromptKit v0.23.2 and ConfigKit v1.2.1. Validate the
|
||||
chosen dependency combination while retaining Scriptorium's adopted PromptKit
|
||||
fixes; the older declared minimum is not a reason to downgrade.
|
||||
|
||||
The [upstream CLI reference][upstream-cli] owns the detailed conventions. The
|
||||
[current Scriptorium CLI](../cli.md) and
|
||||
[subprocess contract](../integrations/subprocess.md) describe the starting point,
|
||||
not requirements to reproduce during migration.
|
||||
|
||||
## Details for the feature roadmap
|
||||
|
||||
The product and interface direction is settled. The following details need
|
||||
concrete implementation choices in the roadmap, but do not require another
|
||||
architectural discussion before drafting it.
|
||||
|
||||
### Outcomes and process lifecycle
|
||||
|
||||
Map PromptKit's typed errors and reported outcomes into CommandKit conventions,
|
||||
preserving its distinction between a completed content check reporting
|
||||
`ValidationFailed` and an operational error. Select any necessary domain exit
|
||||
code without reusing the current validation-failure exit 2, which conflicts with
|
||||
CommandKit's usage convention. Also specify artifact/report delivery failures
|
||||
using CommandKit's presentation-failure behavior where applicable. These are
|
||||
adapter mappings, not a new application validation policy.
|
||||
|
||||
Signal cancellation and the whole-operation deadline remain application
|
||||
responsibilities. Specify their mapping to the CommandKit invocation context
|
||||
and PromptKit operation timeout, including the application deadline flag and
|
||||
units, separately from the standard per-attempt timeout flag.
|
||||
|
||||
### File delivery
|
||||
|
||||
The roadmap and implementation plan specify ordinary report-file handling and
|
||||
delivery ordering relative to stdout. The caller owns redirected destinations;
|
||||
Scriptorium does not infer them or check for collisions with the report path.
|
||||
Partial delivery follows native outcomes and the agreed result-availability
|
||||
boundary.
|
||||
|
||||
### Adoption and verification
|
||||
|
||||
Confirm the released dependency combination, complete the application-specific
|
||||
flag and loader mappings, and sequence replacement and removal of CLI code.
|
||||
Define appropriate verification for the selected behavior and update canonical
|
||||
documentation, examples, and release migration guidance as implementation lands.
|
||||
|
||||
## Roadmap handoff
|
||||
|
||||
The [adoption roadmap](commandkit-adoption.md) incorporates these decisions and
|
||||
integration details. Keep implementation status there; retain this notepad as
|
||||
the discussion record.
|
||||
|
||||
[upstream-guide]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/consumers/guide.md
|
||||
[upstream-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md
|
||||
@@ -1,232 +0,0 @@
|
||||
# ConfigKit Adoption
|
||||
|
||||
## Status And Intent
|
||||
|
||||
Implemented feature direction, drafted on 2026-09-26 and delivered through the
|
||||
historical plan at
|
||||
`git show 8452b54809b99425a97940da485daaa48c77aeb8:docs/roadmap/implementation.md`.
|
||||
This roadmap records the target behavior and changes from the reviewed baseline;
|
||||
that Git record owns sequencing, concrete interfaces, and verification evidence.
|
||||
|
||||
The user is adopting ConfigKit across an application suite to standardize
|
||||
configuration semantics. Scriptorium will make a clean cutover to ConfigKit's
|
||||
native conventions, including intentional breaking changes. Preserving the
|
||||
legacy loader's behavior is not a goal: there will be no parallel legacy loader,
|
||||
compatibility mode, or recreation of the old merging rules.
|
||||
|
||||
Alignment applies to application configuration, not to Promptkit-owned prompt,
|
||||
profile, or schema definitions. Departures from ConfigKit conventions require an
|
||||
application-specific reason or a genuine ownership boundary, not backward
|
||||
compatibility alone. The [architecture policy](../policy/architecture.md) and
|
||||
[ADR 0004](../adr/0004-definition-boundary.md) remain the scope boundaries.
|
||||
|
||||
## Reviewed Baseline (Historical)
|
||||
|
||||
Scriptorium consumes Promptkit v0.11.0. Its `internal/config` package uses
|
||||
yaml.v3 directly and manually applies defaults, file settings, and CLI overrides.
|
||||
It has a parser-specific integer decoder, ignores empty string overrides, and
|
||||
treats an empty configuration file as defaults-only input. It has no application
|
||||
configuration imports, selected overlays, or provenance/inspection interface.
|
||||
The [configuration reference](../config.md), [CLI reference](../cli.md), and
|
||||
[source internals](../internal/sources.md) own the current behavior.
|
||||
|
||||
The reviewed ConfigKit release is v0.4.0 at commit
|
||||
`e6ced0d3d1194b58e0b3dc8970089471ba175258`. Adopt that published release as the
|
||||
initial target, rather than a sibling checkout or unpublished replacement.
|
||||
Its canonical references are:
|
||||
|
||||
- [Configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/e6ced0d3d1194b58e0b3dc8970089471ba175258/docs/config.md).
|
||||
- [Consumer adoption guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/e6ced0d3d1194b58e0b3dc8970089471ba175258/docs/consumers/README.md).
|
||||
- [Release notes](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/e6ced0d3d1194b58e0b3dc8970089471ba175258/docs/releases/v0.4.0.md).
|
||||
|
||||
ConfigKit requires Go 1.25, within Scriptorium's existing toolchain requirement.
|
||||
It selects `go.yaml.in/yaml/v4 v4.0.0-rc.6`. Removing Scriptorium's direct parser
|
||||
usage does not remove yaml.v3 from the dependency graph: Promptkit still uses it.
|
||||
The initial review passed ConfigKit's default test suite and Scriptorium's
|
||||
configuration and CLI tests; that is baseline evidence, not migration validation.
|
||||
|
||||
## Target Ownership And Loading Model
|
||||
|
||||
| Owner | Responsibility |
|
||||
| --- | --- |
|
||||
| ConfigKit | Application YAML policy, source acquisition, composition, overlays, typed decoding, explicit override application, provenance, and redacted inspection. |
|
||||
| Scriptorium | Source discovery, application schema and defaults, CLI parsing, recorded application normalization, semantic validation, command prerequisites, and runtime conversion. |
|
||||
| Promptkit | Framework definition formats, backend semantics, credential lookup, engine defaults, and prompt execution. |
|
||||
|
||||
`internal/config` becomes a small application-specific ConfigKit integration,
|
||||
not a second general-purpose configuration framework. Execution and inspection
|
||||
share one configuration definition and loading pipeline. Defaults, selected base
|
||||
and imports, selected overlay, and explicit application CLI overrides participate
|
||||
in ConfigKit's lifecycle; resolution and validation follow its upstream rules.
|
||||
Normalization that changes inspected values is recorded through its resolver.
|
||||
Runtime conversion does not perform a hidden second merge or apply new defaults
|
||||
after the inspection snapshot has been created.
|
||||
|
||||
Scriptorium retains its application field names and domain meanings unless a
|
||||
specific change below requires otherwise. ConfigKit's public root API replaces
|
||||
direct YAML decoding and parser-specific hooks. Application defaults remain
|
||||
Scriptorium-owned; omitted framework settings still delegate to Promptkit.
|
||||
Backend conversion retains deterministic registration and engine-scoped shared
|
||||
capacity state. This work does not change the HTTP request contract or move
|
||||
configuration loading into each HTTP request.
|
||||
|
||||
## Native Configuration Semantics
|
||||
|
||||
The target accepts ConfigKit's upstream YAML grammar, schema binding, limits,
|
||||
composition, and value-presence rules without a legacy preprocessing layer.
|
||||
Migration-visible changes include:
|
||||
|
||||
- An existing empty file is invalid; `{}` expresses an empty application mapping.
|
||||
Defaults-only operation with no discovered file uses `NoSource` instead.
|
||||
- Explicit empty strings, false values, and zeroes replace defaults. They are
|
||||
not silently treated as omission. Application validation still decides whether
|
||||
a resulting value is meaningful for its field or selected command.
|
||||
- Null follows the application's declared nullable types. It is not a universal
|
||||
spelling for omission. Scalars and structs do not accept null merely because
|
||||
the former parser tolerated it.
|
||||
- Anchors, aliases, merge keys, explicit tags, additional documents, unsupported
|
||||
scalar spellings, and coercions rejected by ConfigKit are rejected here too.
|
||||
- ConfigKit's reserved `configkit` key and resource limits apply, including in
|
||||
dynamic application data. These constraints do not apply to independently
|
||||
loaded Promptkit definitions or HTTP request bodies.
|
||||
- Dynamic integers in backend `extra_params` use ConfigKit's `int64`
|
||||
representation. Runtime mapping preserves their supported value semantics
|
||||
rather than reserializing through YAML or demanding the old Go scalar type.
|
||||
|
||||
Presence remains meaningful at the framework boundary: omitted/null
|
||||
`transient_retries` delegates to Promptkit, while explicit zero disables retries;
|
||||
omitted backend queue capacity differs from explicit zero. Application HTTP
|
||||
size defaults and zero-as-unlimited behavior remain. No new value ranges or
|
||||
backend restrictions are introduced merely to adopt the library.
|
||||
|
||||
## Discovery, Imports, Overlays, And Paths
|
||||
|
||||
Source discovery remains Scriptorium-owned. The target retains the documented
|
||||
explicit `--config` selection and ordered system-location search. An explicitly
|
||||
empty config path is invalid rather than an implicit request for a default file.
|
||||
If neither implicit candidate exists, loading uses defaults only; a selected
|
||||
file that fails loading is not silently replaced with defaults. No implicit
|
||||
working-directory configuration search, environment overlay, or directory scan
|
||||
is added.
|
||||
|
||||
Application configuration supports ConfigKit's explicit path and glob imports.
|
||||
Imports compose disjoint definitions; repeated non-mapping definitions are
|
||||
conflicts, not last-writer-wins overrides. There is no implicit `conf.d` scan.
|
||||
|
||||
An optional `--config-overlay` selects ConfigKit's one named overlay using its
|
||||
standard `overlays/<name>.yml` location, naming rules, and merge behavior. No
|
||||
overlay is selected by default. Selecting an active overlay without a base
|
||||
source follows ConfigKit's rejection of overlays with `NoSource`; Scriptorium
|
||||
does not invent a root. Application overlays do not replace `--profile`,
|
||||
`profile_dir`, or Promptkit's execution-profile selection.
|
||||
|
||||
Use the selected file's canonical directory as ConfigKit's default import and
|
||||
overlay root. This root governs configuration source traversal only. Values of
|
||||
`prompt_dir`, `profile_dir`, `schema_dir`, and `server.artifact_root` retain
|
||||
working-directory-relative interpretation; absolute values remain absolute.
|
||||
This is an explicit application path policy, not an upstream requirement or a
|
||||
legacy parsing mode. ConfigKit's consumer examples demonstrate alternatives but
|
||||
do not mandate source-file-relative application paths. This adoption does not
|
||||
extend import containment to application data or alter HTTP artifact containment.
|
||||
|
||||
## CLI Overrides And Configuration Inspection
|
||||
|
||||
Only flags explicitly supplied by the user become ConfigKit application
|
||||
override records. Parser defaults do not overwrite file or overlay settings.
|
||||
Explicit empty values are not dropped. The existing invocation-specific prompt,
|
||||
profile, input, model, timeout, output, and reporting options remain invocation
|
||||
state rather than new application configuration fields. In particular, the
|
||||
run/render retry flag remains a request override; it does not mutate the engine's
|
||||
application retry setting or backend state.
|
||||
|
||||
Expose application configuration validation, effective redacted YAML, source
|
||||
provenance, and explanation of an individual configuration path. These are
|
||||
included features, not merely library capabilities available for future work.
|
||||
They use ConfigKit diagnostics and inspection records from the shared loading
|
||||
pipeline, including defaults, imports, overlays, explicit CLI values, and
|
||||
recorded normalization. No separate inspection-only interpretation is maintained.
|
||||
|
||||
Configuration inspection does not execute prompts, make provider calls, or
|
||||
resolve API-key values. Application validation is not a promise that every
|
||||
Promptkit backend or definition is valid or that a provider is reachable.
|
||||
Command-specific execution prerequisites remain with the relevant command;
|
||||
inspection can explain an incomplete application configuration without requiring
|
||||
an executable prompt selection. The interface is a CLI configuration command
|
||||
family, not a second CLI framework or an HTTP configuration endpoint; the
|
||||
implementation plan specifies its concrete commands and mappings.
|
||||
|
||||
Safe diagnostics must not restore raw parser errors or configuration values
|
||||
that ConfigKit deliberately suppresses. Effective output and explanation use
|
||||
the upstream redacted renderers; they are inspection output, not guaranteed
|
||||
reloadable configuration. Under the maintainer's selected policy, backend
|
||||
`extra_params` are visible in effective output and explanation, including
|
||||
historical values. There is no blanket container masking, sensitive-key heuristic,
|
||||
or new selective-redaction configuration surface.
|
||||
|
||||
## Credential Boundary And Deferred Features
|
||||
|
||||
Existing `api_key_env` values remain environment-name strings passed to Promptkit.
|
||||
They do not become ConfigKit `Secret` fields. ConfigKit eagerly resolves effective
|
||||
typed references and requires nonempty values; that would change Promptkit's
|
||||
optional/required and selected-credential behavior. This departure from the
|
||||
secret-reference feature is an ownership decision, not compatibility plumbing.
|
||||
Raw API-key fields remain unsupported under the existing architecture policy.
|
||||
|
||||
API-key values are the only secrets requiring ConfigKit redaction in this
|
||||
adoption. The application configuration holds environment-variable names, not
|
||||
those values, so inspection neither resolves nor contains the actual keys.
|
||||
Reference names remain visible. Do not tag `api_key_env` as secret merely to
|
||||
hide its name, add an eager key lookup, or mask unrelated application settings.
|
||||
The existing raw-key prohibition and Promptkit boundary provide the protection
|
||||
without introducing a redundant credential-bearing configuration field.
|
||||
|
||||
Secrets-directory bootstrap and application schema versioning are deferred until
|
||||
there is a concrete application need. Their availability in ConfigKit does not
|
||||
make them mandatory. Hot reload, runtime resource replacement, remote imports,
|
||||
automatic migrations, and general environment-variable overlays are outside
|
||||
this adoption. These are scope limits for this feature, not new permanent
|
||||
prohibitions on future explicitly requested work.
|
||||
|
||||
## Required Changes And Completion Criteria
|
||||
|
||||
- Replace manual decoding/default/override machinery with the shared ConfigKit
|
||||
integration and retain only application-specific discovery, rules, and mapping.
|
||||
- Support imports and selected overlays across commands that load application
|
||||
configuration, and deliver the configuration inspection capabilities above.
|
||||
- Update application configuration types, presence handling, CLI explicit-set
|
||||
mapping, and diagnostics to match the selected native semantics.
|
||||
- Update maintained examples and canonical configuration, CLI, operations,
|
||||
architecture, and internal documentation when behavior lands. Upstream generic
|
||||
contracts remain linked rather than copied into a competing specification.
|
||||
- Add proportionate integration coverage for the cutover: defaults-only input,
|
||||
explicit values and nulls, imports/conflicts, overlays, CLI precedence,
|
||||
inspection agreement and API-key non-disclosure, visible dynamic backend
|
||||
parameters, unchanged credential ownership, and representative
|
||||
execution/server startup.
|
||||
ConfigKit owns exhaustive parser and composition tests; Scriptorium tests its
|
||||
adoption and application boundaries.
|
||||
- Pin the published dependency and maintain ConfigKit and YAML attribution under
|
||||
the [OSS policy](../policy/oss.md), using imported-commit source/notice links
|
||||
and applicable standard and companion license texts.
|
||||
- Document migration in a release appropriate for the intentional breaking
|
||||
configuration changes under the [release policy](../policy/release.md).
|
||||
Release selection and publication remain separate preparation work; this
|
||||
roadmap adds no release gate, network probe, or new distribution format.
|
||||
|
||||
Completion means one ConfigKit-native application configuration path serves
|
||||
execution and inspection, the selected user-facing features work, and migrated
|
||||
examples and regression coverage demonstrate the documented semantics. It does
|
||||
not require adopting every ConfigKit feature or changing Promptkit definitions.
|
||||
|
||||
## Resolved Inspection Policy
|
||||
|
||||
On 2026-09-26 the maintainer selected no redaction for `extra_params` and limited
|
||||
the relevant secret to the API key. Inspection therefore preserves parameter
|
||||
values and environment-reference names, while actual API-key values remain
|
||||
outside the application configuration pipeline. Visibility is intentional;
|
||||
this is not a promise to detect secrets placed inside arbitrary parameter values.
|
||||
|
||||
The delivered `config show` and `config explain` commands use ConfigKit's
|
||||
inspection writers without secret tags or a second redaction path. Both
|
||||
effective and historical parameter values remain visible; `api_key_env` names
|
||||
remain visible while Promptkit alone resolves selected credential values.
|
||||
@@ -1,303 +1,562 @@
|
||||
# Policy Alignment Implementation Plan
|
||||
# CommandKit Adoption Implementation Plan
|
||||
|
||||
## Status, Scope, And Authority
|
||||
## Purpose And Execution
|
||||
|
||||
Planned on 2026-09-28 against Scriptorium
|
||||
`8452b54809b99425a97940da485daaa48c77aeb8`. All five stages are complete.
|
||||
The ordered stages below retain the implementation instructions and completion
|
||||
evidence for this delivered work set.
|
||||
Implement the target in the [CommandKit adoption roadmap](commandkit-adoption.md).
|
||||
This plan owns the implementation steps and concrete adapter choices; the roadmap
|
||||
owns product intent and target behavior. The [migration notepad](commandkit-notepad.md)
|
||||
records the discussion. [ADR 0006](../adr/0006-adopt-commandkit-native-operator-interfaces.md),
|
||||
[ADR 0005](../adr/0005-configkit-boundary.md), and
|
||||
[ADR 0004](../adr/0004-definition-boundary.md) remain authoritative.
|
||||
|
||||
The [feature roadmap](policy.md) owns the agreed end state, scope, and reasons
|
||||
for local exceptions. This plan owns execution steps and acceptance checks.
|
||||
This is documentation work, not application, dependency, or release-tool work.
|
||||
Audience: a Luna coding agent implementing one numbered stage per prompt, in
|
||||
order, as one end-to-end sprint. Intermediate stages may leave compilation or
|
||||
tests temporarily broken. Do not build compatibility dispatch, parallel CLI
|
||||
surfaces, feature toggles, or temporary abstractions to make intermediate stages
|
||||
shippable. The final assembled application is the completion boundary. Add the
|
||||
stage's relevant tests with its implementation; run focused checks when runnable
|
||||
and record deferred checks for Stage 9. Follow the agent policy for focused
|
||||
commits, accurately identifying incomplete validation.
|
||||
|
||||
The selected public upstream revision is
|
||||
`d84a38ebab26f048cafbe59a2246700c35d29843`.
|
||||
Start with its
|
||||
[adoption guide](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/adoption.md).
|
||||
Retrieve source files at this revision, not whatever main contains when a
|
||||
stage runs. A verified checkout of this exact revision is acceptable; no
|
||||
sibling checkout is a prerequisite. If source retrieval fails, report that
|
||||
blocker rather than inventing source content or provenance.
|
||||
Start with [development guidance](../development.md). Read the applicable
|
||||
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[testing](../policy/testing.md), [documentation](../policy/documentation.md), and
|
||||
[OSS](../policy/oss.md) policies. Use the [release policy](../policy/release.md)
|
||||
for release-related documentation and build metadata. Preserve unrelated work.
|
||||
This plan does not authorize release publication, upstream repository changes,
|
||||
or rewriting accepted ADR decisions.
|
||||
|
||||
Read the current local agent policy and developer orientation before work,
|
||||
then policies governing the stage. Once Stage 2 lands, use the new selective
|
||||
reading guide. Reuse unchanged policy text already available in context.
|
||||
Planning baseline: `bcb1a76e345ef7f3720dcbce7c446beb0d9db23b`.
|
||||
The current application uses Go 1.27.0, ConfigKit v1.2.1, and PromptKit v0.23.3,
|
||||
with validation/build toolchain Go 1.27.4. Relevant starting components are
|
||||
`cmd/scriptorium`, `internal/adapter/cli`, `internal/config`, `internal/format`,
|
||||
`internal/report`, and the unchanged `internal/adapter/http`.
|
||||
|
||||
## Historical Implementation Records
|
||||
## Released Dependencies And Selected Integration
|
||||
|
||||
This filename previously held completed application plans. Preserve their
|
||||
records through Git rather than carrying their implementation steps forward:
|
||||
The reviewed public APIs are documented at CommandKit revision
|
||||
`d330d7637f2e7507eca931a899397222c82bc892` in its
|
||||
[consumer guide][upstream-guide] and [CLI contract][upstream-cli]. The released
|
||||
baseline is:
|
||||
|
||||
- ConfigKit adoption and release handoff:
|
||||
`git show 8452b54809b99425a97940da485daaa48c77aeb8:docs/roadmap/implementation.md`.
|
||||
- Earlier PromptKit plans are referenced within that historical document.
|
||||
| Module | Reviewed release | Release source commit |
|
||||
| --- | --- | --- |
|
||||
| `go.maximumdirect.net/commandkit` | v0.7.0 | `b57bfe176cff05a892e6cb07ebb5f350239e55d4` |
|
||||
| `go.maximumdirect.net/ckconfig` | v0.9.0 | `9962368f0c1ea1f6ad010638e94621091105d745` |
|
||||
| `go.maximumdirect.net/ckprompt` | v0.10.0 | `c9a56e1b15f2169881a91fca4e1f4a1c475e057a` |
|
||||
|
||||
Those stages must not be rerun. Existing release notes, including
|
||||
[the v0.14.0 note](../releases/v0.14.0.md), record prior delivered features.
|
||||
Do not infer a new release obligation from an old prospective handoff.
|
||||
The new policy and licensing changes still need a concise summary in the next
|
||||
release, including the already-selected GPL-3.0-or-later application grant and
|
||||
the separately licensed policy material. Version selection and publication
|
||||
are not part of this plan.
|
||||
Use the reviewed released combination and verify it in Stage 1. Preserve
|
||||
ConfigKit v1.2.1 and PromptKit v0.23.3; ckprompt's PromptKit v0.23.2 minimum is
|
||||
not a reason to downgrade. Raise any demonstrated compatibility problem before
|
||||
expanding this sprint into another framework upgrade.
|
||||
|
||||
## Common Execution Rules
|
||||
No selected-format accessor is required. Main results go through the native
|
||||
invocation reporter to stdout, with caller-controlled redirection. The separate
|
||||
`--report` file uses the existing `RenderReport(writer, report, format)` API,
|
||||
with the format unset unless `--report-format` is supplied.
|
||||
The upstream accessor proposal is optional and does not block any stage. Do not
|
||||
add a second reporting-flag parser or private-API workaround.
|
||||
|
||||
1. Inspect status and relevant diffs before each stage. Preserve unrelated
|
||||
changes; do not silently include them in the stage commit.
|
||||
2. Read the selected upstream files completely before adopting them. Compare
|
||||
directly with local files; do not reconstruct historical adoption baselines.
|
||||
3. Start from upstream text, remove the complete Base Policy Template section,
|
||||
and apply only the customizations selected below. Preserve shared language
|
||||
verbatim, including its meaning and strength. Do not retain old paraphrases
|
||||
merely because they are familiar or add prohibitions for hypothetical risks.
|
||||
4. For each policy adopted in that stage, set the target Base Policy Version,
|
||||
the actual adoption date, and an Upstream Source link:
|
||||
`https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/FILE.md`.
|
||||
Substitute the actual filename. Do not stamp a source on a policy before
|
||||
incorporating it. Intermediate commits may legitimately have mixed versions.
|
||||
5. Update the single policy-attribution entry as its copied/adapted scope grows.
|
||||
Do not claim later-stage material has already been adopted.
|
||||
6. Validate affected text, links, notices, and diff; record concise completion
|
||||
evidence under the completed stage here. Commit each coherent stage with a
|
||||
short plain-English message under the agent policy. Do not push or publish.
|
||||
## Settled Application Choices
|
||||
|
||||
Do not change Go code, module pins, tests, release scripts, CI, accepted ADR
|
||||
decisions, or published release notes. Do not introduce recurring sync checks,
|
||||
license approval rounds, release gates, or blanket source headers. Preserve
|
||||
the application grant and dependency terms. Standard license text must remain
|
||||
verbatim even if a generic whitespace check dislikes publisher formatting.
|
||||
### Command And Selection Mapping
|
||||
|
||||
## Stage 1: Adopt OSS Rules And Establish Licensing Notices
|
||||
Install the native `config` and `promptkit` families, plus application commands
|
||||
`run PROMPT_ID` and `serve`. Use `ckprompt.Declarations` to install native
|
||||
convenience aliases. Install equivalent command-scoped aliases for applicable
|
||||
reused bindings on `run`; do not add legacy command aliases or a duplicate prompt
|
||||
flag. Native help, version, completion, format, verbosity, and outcomes replace
|
||||
their old counterparts without a compatibility parser.
|
||||
|
||||
**Status: Complete.** OSS language was compared with the pinned source after
|
||||
the approved metadata and example changes. The CC-BY-SA text matches its pinned
|
||||
source byte-for-byte; attribution and navigation were checked against the
|
||||
specified format and local paths.
|
||||
| Input | Application mapping |
|
||||
| --- | --- |
|
||||
| `--config` | Application main-source selection on every configuration-consuming command, including the `promptkit` scope and family. Preserve explicit presence, including an empty value. |
|
||||
| `--overlay` | Pass the supplied value to ConfigKit's `Options.Overlay`. At v1.2.1 this is a name, not a filesystem path: native empty/default selection and `overlays/NAME.yml` lookup apply. Do not add a second overlay interpreter or a fallback root. |
|
||||
| Repeated `--promptkit-source PATH` | Replace the complete `promptkit_sources` list during ConfigKit loading with the selected files/directories in occurrence order, as a native override. Retain the useful existing file/directory classification, without comma splitting. |
|
||||
| `--schema-dir` | Presence-aware override of application `schema_dir`. |
|
||||
| `--addr`, `--artifact-root`, `--max-request-bytes`, `--max-artifact-bytes`, `--max-response-bytes` | Presence-aware application `server` overrides; available on `serve` and application-scope configuration inspection. |
|
||||
| `config ... --set POINTER=JSON` | Ordered native override records for `app`; append after named application-selector overrides so native assignments are authoritative when both address the same value. Enable the feature only on `app`. |
|
||||
| `config diff --against-*` | Independent right-side selections and corresponding application-selector overrides, initialized independently rather than inherited from the left. |
|
||||
| `--run-timeout DURATION` on `run` | `time.ParseDuration`, then unchanged into `RunRequest.OperationTimeout`. Positive values add the native operation budget; zero and negative values add none under PromptKit v0.23.3. No second application timer or positivity restriction. |
|
||||
| Native `--llm-timeout-seconds` / `--timeout-seconds` | Integer per-attempt timeout through the ckprompt execution binding; no old duration conversion. |
|
||||
|
||||
Files: `docs/policy/oss.md`, `docs/oss/attribution.md`,
|
||||
`docs/oss/licenses/CC-BY-SA-4.0.txt`, `COPYRIGHT`, `README.md`,
|
||||
and the footer portion of `docs/release.md`.
|
||||
Framework selectors (`promptkit-source`, `schema-dir`) apply to both config
|
||||
scopes and to the native PromptKit family. Server selectors apply to the `app`
|
||||
scope only. Provide each config selector with its matching `AgainstName`, such
|
||||
as `against-promptkit-source`, `against-schema-dir`, or `against-addr`.
|
||||
`run` accepts the framework selectors, request controls, and delivery controls;
|
||||
it does not need server-only flags. `serve` accepts framework and server
|
||||
selectors, not generation request controls. The configured `transient_retries`
|
||||
value remains an engine default; the native execution flag overrides a request.
|
||||
|
||||
1. Read upstream OSS v1.8, COPYRIGHT, the canonical
|
||||
`examples/licensing/policy-attribution.md`, and referenced licensing
|
||||
examples at their pinned URLs. Adopt OSS v1.8 with metadata as above.
|
||||
Remove template-only content; retain upstream common wording and its
|
||||
existing commit-pinned example links. Customize the README example to
|
||||
Scriptorium and GPL v3 or later. No other OSS exception is selected.
|
||||
2. Copy the complete CC-BY-SA-4.0 reference text from the selected revision
|
||||
into the named local license file and compare it byte-for-byte with that
|
||||
source. Do not copy the rest of the reference collection or add CC0 merely
|
||||
because example formatting is used.
|
||||
3. Reformat the attribution document to the exact v1.8 House Format. Use
|
||||
`# Open Source Attribution Notices`, immediately followed by entries.
|
||||
Place Standard Repository Policies first, using the canonical entry,
|
||||
with its COPYRIGHT, LICENSE, and Source URLs pinned to the selected full
|
||||
commit. In this stage identify `docs/policy/oss.md` as the newly adapted
|
||||
material. Do not research or make claims about prior adoption provenance.
|
||||
4. Preserve the eight existing dependency entries and their imported commits,
|
||||
copyright wording, SPDX expressions, and required notices. Optional module
|
||||
identification paragraphs may retain versions currently in source labels.
|
||||
Remove the introductory prose, missing-copyright filler, and editorial
|
||||
YAML license explanations. Preserve the factual MIT AND Apache-2.0
|
||||
expressions. Use actual upstream filename labels, local license names,
|
||||
terminal periods, and the exact `[Source](COMMIT_URL).` source form.
|
||||
Do not upgrade dependencies or relabel their grants.
|
||||
5. Add a scoped paragraph to COPYRIGHT stating that material adapted from
|
||||
Standard Repository Policies remains CC-BY-SA-4.0, with
|
||||
`docs/oss/attribution.md` identifying affected material and links. This
|
||||
covers the scoped material as stages add it; it does not relicense unrelated
|
||||
application code or documentation. Keep the existing GPL grant and LICENSE.
|
||||
6. Correct the README ending to link GPL v3 or later to COPYRIGHT and keep
|
||||
the attribution link. Change the release-procedure footer to absolute
|
||||
immutable-tag COPYRIGHT, LICENSE, and ATTRIBUTION links, updating the
|
||||
references to the number of URLs. Change no publication commands.
|
||||
Do not retrofit published notes or create a release note/version now.
|
||||
ConfigKit remains responsible for loading and all precedence, resolution, and
|
||||
inspection semantics. The override ordering above selects library inputs; it
|
||||
does not authorize post-load merging. Keep cwd-relative application path values,
|
||||
the existing discovery locations, and source-relative PromptKit resources.
|
||||
|
||||
Acceptance: OSS text matches the pinned source except the approved project
|
||||
example and metadata; attribution follows the prescribed format; the added
|
||||
license text is faithful; project and dependency grants remain unchanged.
|
||||
Check affected local paths and upstream notice/example destinations. Missing
|
||||
copyright fields are omitted, not explained with agent-authored commentary.
|
||||
### Outcomes And Delivery
|
||||
|
||||
## Stage 2: Adopt Provenance, Agent Guidance, And Documentation Ownership
|
||||
| Condition | Outcome |
|
||||
| --- | --- |
|
||||
| Completed run without `ValidationFailed` | Native success, status 0, subject to delivery. |
|
||||
| Completed run with `ValidationFailed` | `commandkit.Domain("validation_failed", 4)` with a useful explanation; deliver the returned artifact and native result. |
|
||||
| Syntax or declaration-input errors, including an explicitly empty report path | Native usage, status 2. |
|
||||
| Configuration load errors | Use `ckconfig.ClassifyConfigurationError` and native diagnostic evidence: usage 2 for classified invalid configuration, otherwise operational 1. |
|
||||
| Lookup, preparation, generation, capacity, operation-timeout, cancellation, or other operational errors | Operational 1 with safe explanation and preserved typed cause. Request failures are not completed content-validation failures. |
|
||||
| Native `config diff --check` differences | Native status 3, unchanged. |
|
||||
| Result or diagnostic presentation failure | Preserve CommandKit's finalization rules: success/usage become operational 1; a nonzero domain status remains nonzero and retains its classification. |
|
||||
| Report file delivery failure | Operational 1 for an otherwise successful run. Preserve an already nonzero domain status 4 while adding the delivery explanation/cause. |
|
||||
|
||||
**Status: Complete.** The three policies carry the selected versions, source
|
||||
links, and local adoption date; shared text and the orientation template were
|
||||
compared with the pinned source. The seven-link reading guide and local links
|
||||
were checked, and attribution names only the incorporated material.
|
||||
Use `errors.Is`/`errors.As`, not error-text matching. Apply cancellation and
|
||||
typed operation-timeout handling before configuration classification. Do not
|
||||
classify every `ErrInvalidConfig` wrapper as invalid user input without native
|
||||
evidence. Retain safe explanations for provider failures; causes are inspectable
|
||||
Go errors, not permission to print credential-bearing provider payloads.
|
||||
|
||||
Files: new `docs/policy/meta.md`, `docs/policy/agents.md`,
|
||||
`docs/policy/documentation.md`, root `AGENTS.md`,
|
||||
`docs/development.md`, and the policy attribution entry.
|
||||
Use a main `commandkit.Report` with the actual `*promptkit.RunResult` as `Value`.
|
||||
Its reusable text template emits `Artifact.Body` as bytes represented as a string,
|
||||
with no added newline, header, escaping, or summary. Native JSON/YAML serialization
|
||||
remains upstream-owned, including byte representation and native metadata.
|
||||
|
||||
1. Adopt meta v1.0 and agents v1.3 verbatim after template-section removal
|
||||
and metadata substitution. Keep meta's live project and main-branch
|
||||
adoption-guide discovery URLs; its own Upstream Source is commit-pinned.
|
||||
Retain local relative links to developer guidance and attribution.
|
||||
2. Adopt documentation v1.11. Customize only applicable topic-map rows:
|
||||
configuration remains application configuration, framework formats belong
|
||||
to PromptKit, consumers cover executable use and migration rather than a
|
||||
Scriptorium public Go API, and integrations cover Scriptorium-owned
|
||||
executable/process contracts. Keep upstream rows for meta, COPYRIGHT,
|
||||
LICENSE, and attribution. Remove instructional “if applicable” qualifiers
|
||||
from selected rows. Preserve the rest of the shared language.
|
||||
3. Use the upstream ADR lifecycle and examples guidance without retaining
|
||||
generic local expansions. This does not edit accepted ADRs or change their
|
||||
decisions. Metadata maintenance follows the adoption guide through meta,
|
||||
rather than a duplicate custom rule set.
|
||||
4. Adapt upstream `agents/AGENTS.md` into root AGENTS.md. Preserve any actual
|
||||
repository-specific code-discovery instructions; do not copy environment-
|
||||
injected instructions into the file just because they appeared in a session.
|
||||
Do not copy upstream's own root AGENTS.md.
|
||||
5. Add the adoption guide's Policy Reading Guide table and selection bullets
|
||||
verbatim to the developer guide with the seven local links. Replace its
|
||||
unconditional architecture-reading instruction and duplicated policy-reading
|
||||
prose with this selective guide. Retain project orientation, task-specific
|
||||
contract rows, dependency boundaries, and baseline validation commands.
|
||||
Avoid maintaining competing policy trigger rules in multiple places.
|
||||
6. Expand the policy entry's Material sentence to identify the adopted OSS,
|
||||
meta, agents, and documentation policies plus adapted root AGENTS.md and
|
||||
the copied Policy Reading Guide section of docs/development.md. Mark them
|
||||
adapted for this repository; do not imply the whole developer guide or
|
||||
unrelated tooling instructions have changed license.
|
||||
Submit the main report through `Invocation.Reporter().Report`. There is no
|
||||
`--output` or legacy `--out` flag. Callers save stdout using shell redirection or
|
||||
their subprocess API; Scriptorium keeps diagnostics on stderr. Native quiet
|
||||
suppresses the main result even when stdout is redirected. Consumers wanting
|
||||
that result omit `--quiet`.
|
||||
|
||||
Acceptance: all stage policies have their own correct source link and version;
|
||||
orientation selects policies by task, including meta for adoption work; local
|
||||
links resolve; no upstream handbook or example collection was copied.
|
||||
Generic language differences from upstream are eliminated, not rephrased.
|
||||
`--report PATH` additionally renders `commandkit.Report{Value: result}` through
|
||||
`commandkit.RenderReport`, without the main report's artifact-only text template.
|
||||
Optional `--report-format text|yaml|json` selects its native format independently
|
||||
of stdout's `--format`. When omitted, pass the zero `commandkit.Format` value
|
||||
(`""`) for native automatic rendering. Do not substitute JSON, YAML, or text.
|
||||
CommandKit currently renders structured values as YAML in automatic and text
|
||||
modes; future upstream defaults flow through without application policy changes.
|
||||
Filename extensions do not select the format.
|
||||
|
||||
## Stage 3: Adopt Architecture And Testing With Narrow Local Decisions
|
||||
This application-owned file write still occurs in quiet mode. Omitting `--report`
|
||||
creates no separate report. Use ordinary create/truncate operations (mode 0644
|
||||
for newly created files, subject to umask), preserve normal OS behavior for
|
||||
existing files, and handle render/write/close errors. Do not create parent
|
||||
directories. An explicit empty report path is a usage error; `-` is an ordinary
|
||||
filename. The caller owns redirected destinations, including keeping the main
|
||||
output file separate from the report. Do not add stdout-destination discovery
|
||||
or collision checks in Scriptorium.
|
||||
|
||||
**Status: Complete.** The three shared architecture sections and the common
|
||||
testing text match the pinned source. Local architecture decisions were checked
|
||||
against the accepted ADRs and internal guides; focused test ownership and the
|
||||
expanded attribution scope were reviewed without changing application code.
|
||||
Once a result is available, attempt main delivery first, then the separate
|
||||
report, even if the first attempt fails. Open the report only after a result
|
||||
exists; a resultless failure leaves an existing report untouched. This does not
|
||||
govern a caller's redirection: a shell may create or truncate its destination
|
||||
before Scriptorium runs. Delivery is nontransactional; written bytes remain
|
||||
after failure. No rollback, atomic replacement, symlink sandbox, or additional
|
||||
filesystem hardening is requested.
|
||||
|
||||
Files: `docs/policy/architecture.md`, `docs/policy/testing.md`,
|
||||
policy attribution entry, and focused internal docs only if moving existing
|
||||
implementation detail requires it.
|
||||
## Stage 1: Select Released Dependencies And Establish Command Composition
|
||||
|
||||
1. Read the existing architecture, relevant accepted ADRs, source/adapter
|
||||
internal guides, and the pinned architecture v1.3 template. Retain upstream
|
||||
Purpose And Ownership, Architectural Approach, and Adopted Constraints And
|
||||
Implemented Structure text verbatim. Replace the Application-Specific
|
||||
Topics placeholder with Scriptorium's existing substantive decisions.
|
||||
2. Preserve public PromptKit/ConfigKit responsibility boundaries, definition
|
||||
support, strict decoding, presence handling, HTTP artifact containment,
|
||||
request independence and concurrency-safe shared capacity, credential/error
|
||||
boundaries, dependency restrictions, and explicit workflow non-goals.
|
||||
Retain the existing lexical containment and privileged-host threat scope,
|
||||
not a stronger filesystem-sandbox promise. Customize the trust discussion
|
||||
to those decisions rather than retaining instructional example prose.
|
||||
3. Correct system shape to include run, render, serve, inspect, and config.
|
||||
Link to internal documentation for concrete package inventory/mechanics.
|
||||
Where material moves, preserve it in its canonical internal owner without
|
||||
duplicating it. Do not introduce new wrappers, ports, orchestration, or
|
||||
package constraints merely to “implement” the architectural recommendation.
|
||||
4. Adopt testing v1.3 common text verbatim. Add one clearly identified local
|
||||
test-ownership section for configuration, CLI, HTTP, formatting, architecture
|
||||
guards, and PromptKit/ConfigKit boundaries. Preserve the existing distinction
|
||||
between adoption/integration fixtures and upstream exhaustive tests.
|
||||
Retain a short application-specific note explaining HTTP containment and
|
||||
shared-state risks where useful; do not replace upstream risk-based coverage
|
||||
guidance or create percentage targets, quotas, or new test requirements.
|
||||
5. Add accurate source/version/date metadata and extend the policy attribution
|
||||
scope to architecture and testing. Record any moved adapted prose in the
|
||||
Material scope if it is placed outside the already identified files.
|
||||
**Primary files:** `go.mod`, `go.sum`, `internal/adapter/cli` composition,
|
||||
`docs/oss/attribution.md`, and applicable `docs/oss/licenses/` texts.
|
||||
|
||||
Acceptance: direct comparison isolates only the roadmap's justified local
|
||||
decisions; shared text matches upstream. Architecture describes existing
|
||||
behavior accurately without implementation work. Testing preserves ownership
|
||||
without duplicating upstream matrices or imposing new coverage gates.
|
||||
1. Verify the reviewed released module combination above. Record the actual
|
||||
versions and complete source commits in this plan's implementation status
|
||||
and attribution entries as appropriate; use released modules, not sibling
|
||||
checkouts, workspaces, vendoring, or replacements.
|
||||
2. Add CommandKit, ckconfig, and ckprompt dependencies. Keep the existing Go
|
||||
minimum and validation toolchain unless the selected dependencies supply a
|
||||
concrete contrary requirement; raise such an expansion rather than guessing.
|
||||
Verify selected PromptKit/ConfigKit versions and tidy the module metadata.
|
||||
3. Create the CLI composition entry point in `internal/adapter/cli`. Accept a
|
||||
context, application identity, argv, and output streams; let `App.Execute`
|
||||
own parsing and finalization and return its outcome/status to the entrypoint.
|
||||
Keep package/file choices straightforward rather than introducing a new
|
||||
framework or interface for each factory.
|
||||
4. Establish declarations for `run`, `serve`, the native configuration suite,
|
||||
native PromptKit suite, and `commandkit.Completion("scriptorium")`. Fill the
|
||||
application loaders and handlers in later stages. Native root/group/help
|
||||
behavior replaces the dispatch switch. Declaration construction, help,
|
||||
version, scope discovery, and static completion must not load application
|
||||
or framework configuration or invoke providers.
|
||||
5. Apply the OSS House Format to CommandKit, ckconfig, and ckprompt using their
|
||||
actual release commits and module-subdirectory notice paths. Inspect linked
|
||||
dependencies, including Cobra, pflag, and any applicable mousetrap use,
|
||||
rather than treating every module-graph tool as a runtime dependency.
|
||||
Reuse existing verbatim license texts and add missing applicable ones.
|
||||
Preserve current project licensing and notice navigation.
|
||||
|
||||
## Stage 4: Adopt Release Policy Without Changing Publication Behavior
|
||||
**Evidence:** selected module metadata has no replacements or unintended
|
||||
PromptKit downgrade; declaration construction is valid when wired; help/version
|
||||
can be exercised without a readable application configuration. Check notice
|
||||
paths against the selected commits. Final composed behavior is checked in Stage 9.
|
||||
|
||||
**Status: Complete.** The shared release language matches the pinned source;
|
||||
the local selections match the existing script, hosted workflow, and runbook.
|
||||
Commands, build targets, recovery behavior, and the three-link footer remain
|
||||
unchanged. Attribution now names all seven adopted policies.
|
||||
## Stage 2: Adapt Authoritative Application Loading And Shared Selectors
|
||||
|
||||
Files: `docs/policy/release.md`, `docs/release.md`, attribution entry.
|
||||
**Primary files:** `internal/config/{load,document,settings}.go`, their tests,
|
||||
and CLI application-loading/source-selector helpers.
|
||||
|
||||
1. Read the current release policy, procedure, script, and hosted workflow;
|
||||
inspect them read-only. Adopt upstream release v1.4 common wording.
|
||||
2. Add a compact Scriptorium-specific section preserving the established
|
||||
selections: main/origin, branch-before-tag publication with exact remote
|
||||
candidate agreement, annotated tags derived from the checked-in note,
|
||||
stable releases only, asynchronous Linux binaries/checksums, one race-enabled
|
||||
offline test invocation, and source-boundary guards. Keep rationale for
|
||||
selected gates in policy and command/target details in the procedure.
|
||||
3. Retain upstream's general alternatives verbatim but state which existing
|
||||
approach Scriptorium selects. Atomic push and source-only release remain
|
||||
template-supported alternatives, not newly supported Scriptorium workflows.
|
||||
Preserve successful tag publication as completion; no CI polling, artifact
|
||||
availability, or consumer installation gate is added.
|
||||
4. Align the procedure with this allocation without changing its commands,
|
||||
required checks, supported platforms, or failure-recovery behavior. Confirm
|
||||
the Stage 1 three-link footer survives hosted-body reuse. Do not run the
|
||||
publication command or mutate remote state to validate documentation.
|
||||
5. Add metadata and expand the policy attribution scope to all seven policies.
|
||||
The final Material description also identifies adapted AGENTS.md and the
|
||||
developer-guide reading section, plus any actually moved adapted text.
|
||||
1. Retain one `configkit.Load` boundary returning its actual
|
||||
`*configkit.Result[Document]`. Adapt selection into `LoadOptions` or its
|
||||
replacement: explicit config presence chooses `File`, omission chooses the
|
||||
existing ordered `SearchConfig`, imports remain enabled, and overlay and
|
||||
override records go directly into ConfigKit. Preserve wrapped error identity.
|
||||
Respect the invocation context at the application boundary without claiming
|
||||
ConfigKit's synchronous work can be forcibly interrupted.
|
||||
2. Replace `flag.FlagSet.Visit` and parser-owned override collection with typed
|
||||
CommandKit bindings retaining `Supplied.Set`. Convert only supplied
|
||||
application selectors into correctly labeled ConfigKit overrides, then append
|
||||
native `Selection.Overrides` as specified above. Keep source-list order and
|
||||
replacement. Use ordinary OS classification for explicitly supplied source
|
||||
paths; do not load framework definitions during application inspection.
|
||||
Preserve the existing nonnegative `int64` domain for HTTP byte limits,
|
||||
including zero as unlimited; do not narrow it to a platform `int` or add a cap.
|
||||
3. Keep schema, defaults, normalization, and domain validation inside the
|
||||
existing ConfigKit definition callbacks. Record normalization through native
|
||||
`configkit.Normalize`/resolution evidence. `Settings` remains a conversion
|
||||
of completed values, not another resolver or merger.
|
||||
4. Remove `Document.Defaults`, `DefaultsDocument`, `defaults.render_format`,
|
||||
`AppSettings.DefaultRenderFormat`, associated normalization/validation, and
|
||||
the resulting `internal/config` dependency on `internal/format`. Strict
|
||||
ConfigKit decoding handles the retired field; add no compatibility mapping.
|
||||
5. Retain shared engine construction from resolved settings: one ordered
|
||||
underlay per app selector, import-enabled file inputs, directory expansion
|
||||
through `promptkit.DirectoryInputs`, no implicit main, existing schema base
|
||||
and engine retry setting, and the default upstream bundle. Do not change
|
||||
collection traversal, peer composition, or whole-record replacement.
|
||||
6. Use fresh application input per comparison side and invocation-owned state
|
||||
under the native factory lifecycle; comparison may share that state without
|
||||
sharing either side's selected values. Start without an engine/config cache:
|
||||
separately reached loads are ordinary
|
||||
loads, not an immutable-snapshot contract. Server engine lifetime is handled
|
||||
separately in Stage 7. Small invocation state for acquisition diagnostics is
|
||||
acceptable and must not alter effective configuration.
|
||||
|
||||
Acceptance: the existing release workflow still satisfies the resulting policy
|
||||
with no script/CI changes. Source completion, delivery status, immutability,
|
||||
tag type, and validation obligations are unchanged. All seven source links
|
||||
identify their own file at the selected full commit.
|
||||
**Evidence:** real ConfigKit-backed tests cover omitted versus explicit-empty
|
||||
config/overlay selection, a supplied zero or empty app override, native `--set`
|
||||
precedence, ordered source replacement, normalized values and their evidence,
|
||||
and removal of `render_format`. Reuse existing import/search/source tests;
|
||||
do not reproduce ConfigKit's exhaustive composition matrix.
|
||||
|
||||
## Stage 5: Verify Language Parity And Complete The Documentation Handoff
|
||||
## Stage 3: Install Native Configuration Scopes And Independent Comparison
|
||||
|
||||
**Status: Complete.** All seven policies were compared directly with the pinned
|
||||
upstream files; differences are limited to metadata, removed template text,
|
||||
the selected project example and topic rows, and the identified local sections.
|
||||
The copied license matches byte-for-byte. A Markdown-aware check resolved 86
|
||||
local links; historical plan references resolve to Git objects. Attribution,
|
||||
notice scope, release navigation, and the final documentation-only diff were
|
||||
reviewed without changing application or publication artifacts.
|
||||
**Primary files:** CLI configuration-suite composition and loaders; replace
|
||||
`internal/adapter/cli/config.go` and `framework_config.go` responsibilities.
|
||||
|
||||
Files: this plan, the feature roadmap, and only affected documentation needing
|
||||
correction within the preceding stages' scope.
|
||||
1. Construct `ckconfig.Command(ckconfig.Suite[...])` with `app` from
|
||||
`ckconfig.FromConfigKit` and `promptkit` from `ckprompt.ConfigScope`.
|
||||
Enable native validate/show/sources/explain/diff and supported path/metadata
|
||||
modes. Keep native scope discovery and default scope selection.
|
||||
2. Mark only the application scope as supporting temporary overrides, passing
|
||||
those records through Stage 2. Retain the PromptKit scope's native disabled
|
||||
override feature, including native capability checks for all-scope requests.
|
||||
3. Install selectors with their scopes and independent `AgainstName` bindings
|
||||
from the mapping above. Use fresh `NewInput` results, not shared pointers.
|
||||
Each comparison loader must consume that side's `Selection` and input;
|
||||
do not cache the first engine into the second side. Delegate comparisons
|
||||
to the returned native adapters instead of assembling a synthetic document.
|
||||
4. Preserve native view metadata, findings, sources, path discovery, explanation,
|
||||
redaction, diff reports, and failure classification. No local source DTO,
|
||||
JSON Pointer interpretation, custom effective document, explanation-tree
|
||||
reconciliation, or replacement formatter is needed.
|
||||
5. Route application-load diagnostics for framework acquisitions through
|
||||
`ckconfig.AdaptDiagnostics` and `commandkit.Acquired` diagnostics/explanations,
|
||||
retaining the adapter's existing framework evidence. Small callback
|
||||
decoration may attach application acquisition messages; do not insert app
|
||||
findings into the PromptKit configuration view or merge their evidence trees.
|
||||
Retain diagnostics from each reached comparison side and label their context.
|
||||
Application-scope acquisition itself stays with `FromConfigKit`.
|
||||
|
||||
1. Compare all seven adopted documents directly with their selected upstream
|
||||
counterparts. Account for metadata, removed template sections, approved
|
||||
substitutions, and the narrow local sections. Correct every remaining
|
||||
non-substantive wording difference. Do not turn this comparison into a
|
||||
permanent script, CI requirement, or release gate.
|
||||
2. Check there are no adoption placeholders, copied template-only instructions,
|
||||
absent local example links, stale BSD descriptions of Scriptorium itself,
|
||||
or contradictory selective-reading directions. Legitimate BSD dependency
|
||||
attribution remains. Validate links using Markdown-aware handling so
|
||||
illustrative syntax in inline code is not treated as a broken real link.
|
||||
3. Confirm per-document source metadata, the policy entry's exact scope,
|
||||
COPYRIGHT's scoped license treatment, existing dependency grants/notices,
|
||||
and the byte-for-byte new license text. Keep example/source links pinned
|
||||
and public discovery links current. Check footer paths in the candidate;
|
||||
unpublished release tags need not resolve online.
|
||||
4. Review affected roadmap references to implementation.md. Where a link
|
||||
specifically means an old completed plan, replace it with a clear historical
|
||||
Git reference from this plan, not this new policy sequence. Preserve
|
||||
deferred application features and historical completion evidence.
|
||||
Current release directions link to the canonical procedure, not an old
|
||||
two-link footer. Do not rewrite published release notes.
|
||||
5. Mark only actually delivered stages complete with concise evidence here.
|
||||
Update the feature roadmap's status to delivered and keep its end-state
|
||||
focus. Record the next-release summary described above without choosing a
|
||||
version, creating a tag, or claiming publication.
|
||||
6. Run diff/whitespace and affected-document/link checks. Do not reformat
|
||||
verbatim license text to satisfy a style check. No Go test/build rerun is
|
||||
necessary for this documentation-only work; no executable example or
|
||||
publication command needs changing. Inspect status and staged diff, commit,
|
||||
and report the commit and remaining work under the agent policy.
|
||||
**Evidence:** app inspection works even with unreadable framework sources;
|
||||
framework inspection uses sources selected by `--config`/`--overlay`; native
|
||||
show and explain agree with the resolved callback values. Exercise native
|
||||
root/index/escaped-key paths and redaction through representative integration
|
||||
cases, independent left/right configs/overlays/selectors, and diff-check status 3.
|
||||
Verify app overrides are available and framework overrides are not emulated.
|
||||
|
||||
Acceptance: policy language is standardized with only justified local
|
||||
customization; notice/provenance arrangements describe reality; no runtime,
|
||||
dependency, publication, or historical-release change slipped into the diff.
|
||||
## Stage 4: Install Native PromptKit Resources, Requirements, And Preview
|
||||
|
||||
## Open Questions
|
||||
**Primary files:** CLI PromptKit-suite composition, engine-loading adapters,
|
||||
and behavior tests replacing custom definition/prepared inspection tests.
|
||||
|
||||
None. The maintainer approved the public discovery URLs, the standard
|
||||
CC-BY-SA arrangement for copied policy material, the selected upstream revision,
|
||||
and the justified local variations. The existing application grant, dependency
|
||||
grants, and release approach remain unchanged; do not reopen them merely to
|
||||
perform this adoption.
|
||||
1. Compose `ckprompt.FromEngine`, `WithRequestValidation`,
|
||||
`WithRequestPreparation`, and `WithSmokeTesting`, using the common application
|
||||
selection and engine loading from Stage 2. Enable all twelve callbacks:
|
||||
five inventories, three named resource inspections, requirements discovery,
|
||||
concrete validation, preparation, and smoke testing.
|
||||
2. Enable `ckprompt.WithNamespaces` with empty application defaults and install
|
||||
`ckprompt.Declarations`. Let the adapters apply their native engine views;
|
||||
the common engine loader returns an unscoped engine. Retain inventory filters
|
||||
and default listing behavior rather than forcing all namespaces.
|
||||
3. Request builders should be nil where the seed already is the desired
|
||||
request. If application mapping needs a builder, start with the supplied
|
||||
seed and preserve its fields, explicit values, and namespace context; do not
|
||||
reread bindings or merge a different request after the builder returns.
|
||||
The configured retry default belongs in the engine, not a second request pass.
|
||||
4. Carry application acquisition diagnostics through the native callback return
|
||||
channels as in Stage 3, preserving framework messages and typed failures.
|
||||
Let CommandKit control presentation and verbosity; do not write directly to
|
||||
stderr from loaders or suppress native reports based on local interpretations.
|
||||
5. Expose native prompt messages, profile/backend inspection, and prepared-run
|
||||
reports as returned, including their content-bearing nature. No old summary
|
||||
projection or local schema-version envelope. Preview remains native
|
||||
`promptkit show prepared PROMPT_ID`, with native text/YAML/JSON behavior and
|
||||
stdout capture for file saving.
|
||||
6. Keep discovery distinct from preparation: requirements discovery does not
|
||||
need generated input artifacts, model credentials, schemas, provider calls,
|
||||
or a generation output path. Loading configuration and captured message
|
||||
resources is still expected. Concrete validation/preview keep their upstream
|
||||
preparation-time prerequisites. Do not reject partial discovery or convert
|
||||
unsuccessful/unsupported smoke findings into extra application failures.
|
||||
|
||||
**Evidence:** implement the nine linked
|
||||
[Narratio scenarios](narratio-feature-request.md#consumer-acceptance-scenarios)
|
||||
with a small set of real-library fixtures covering exact required names,
|
||||
undeclared references, optional/conditional inputs, variable presence including
|
||||
empty values, complete versus partial reports, profile/prompt identity, and
|
||||
selection failures. Check native inventories/namespace filters, one named report
|
||||
per resource kind, and preparation/validation without generation. Smoke-test
|
||||
wiring uses a controlled local provider or injected public LLM client, never real
|
||||
credentials or paid calls. Reuse fixture coverage rather than duplicating the
|
||||
upstream analysis or rendering suite.
|
||||
|
||||
## Stage 5: Implement Positional Generation, Request Mapping, And Outcomes
|
||||
|
||||
**Primary files:** the new application `run` declaration, request/outcome
|
||||
helpers in `internal/adapter/cli`, and run integration tests.
|
||||
|
||||
1. Declare exactly one required positional `PROMPT_ID` with CommandKit, using
|
||||
shared app config/overlay/framework-selector bindings. Reuse
|
||||
`ckprompt.NewResourceFlags` for profile and session selection,
|
||||
`NewPromptInputFlags` for variables/text/files, and `NewExecutionFlags` for
|
||||
all native execution controls. Do not install the reusable prompt flag:
|
||||
the operand supplies that ID. The native run request has no direct backend-ID
|
||||
override, so do not invent a backend flag or synthetic profile for `run`.
|
||||
2. Map the bound values directly into a native `RunRequest` (profile, session,
|
||||
inputs, vars, execution, retry count). Preserve pointers and explicit values
|
||||
rather than implementing local inheritance/clearing rules. Keep native
|
||||
assignment semantics: first `=` separates a mapping, commas are literal,
|
||||
and the last text/file occurrence for an input wins.
|
||||
3. Install the native short convenience spellings on `run` with
|
||||
`commandkit.FlagAliasFor`, matching the reused binding meanings. Cover
|
||||
profile/session, var, input-text/input-file, all execution controls, and
|
||||
`--input`/`--llm-input` as literal-text aliases. Do not retain `--out`,
|
||||
`--execution-report`, `--llm-base-url`, old `--timeout`, `--prompt-id`, or
|
||||
`--profile-id` merely for compatibility.
|
||||
4. For `run`, declare the three standard namespace flags and convenience aliases
|
||||
through ordinary typed CommandKit string flags. The reviewed reusable
|
||||
resource bindings do not export namespace bindings; use public
|
||||
`Engine.WithNamespaces(promptkit.Namespaces{...})`, which is the supported
|
||||
application-engine-view integration. Leave validation of namespace values
|
||||
and ID selection to PromptKit. No local qualification or rewriting of stored
|
||||
references, and no configurable default namespace.
|
||||
5. Pass the invocation context to `Engine.Run` and map `--run-timeout` unchanged
|
||||
to `OperationTimeout`. Keep generation single-dispatch at the adapter level;
|
||||
retries, repairs, backend admission, credentials, and preparation belong
|
||||
upstream. Do not call `ValidateRequest` first as a new execution prerequisite.
|
||||
6. Implement the outcome table above. A completed invalid result is still
|
||||
delivered and returns domain status 4. A resultless operational error has
|
||||
no synthesized result or alternative report shape. Treat an unexpected nil
|
||||
result without an error as an operational integration failure, not success.
|
||||
Preserve typed causes and safe terminal explanations.
|
||||
7. Build the native result report/text template and submit stdout results
|
||||
through the invocation reporter. Keep any retained generation summary as a
|
||||
native Info message, normally hidden at ordinary verbosity. Stage 6 adds
|
||||
the separate native report without changing the result shape.
|
||||
|
||||
**Evidence:** real-engine/controlled-provider tests exercise the assembled
|
||||
positional command, preferred aliases, literal/file precedence, comma-containing
|
||||
values, numeric zero and reasoning empty values, service tier/extra parameters,
|
||||
qualified IDs and independent namespaces, raw byte output, native structured
|
||||
serialization, completed invalid content/status 4, and resultless operational
|
||||
failure/status 1. Include a small typed-error mapping table rather than parsing
|
||||
error prose. Use native config classification for representative load failures.
|
||||
|
||||
## Stage 6: Add Separate Native Report Delivery
|
||||
|
||||
**Primary files:** CLI generation-delivery helpers and focused delivery tests.
|
||||
|
||||
1. Add `--report` as a presence-aware application file flag with path completion.
|
||||
Add optional `--report-format` with typed binding and completion for the native
|
||||
`text`, `yaml`, and `json` formats; leave its default unset. Reject an explicitly
|
||||
empty path or invalid format as usage errors before executing the model. Do
|
||||
not create files during validation or add a main-output destination flag.
|
||||
2. Main stdout uses Stage 5's report through `Invocation.Reporter().Report`.
|
||||
Render the separate report through `RenderReport` with
|
||||
`commandkit.Report{Value: result}`, sharing the native result but no text
|
||||
template. Pass the empty format when `--report-format` is omitted;
|
||||
otherwise pass the explicitly selected native format. The invocation reporter
|
||||
owns main format selection, including automatic artifact-template output.
|
||||
No selected-format accessor is needed. Extensions never select formats.
|
||||
3. Keep the report file effect independent of console quiet selection. Do not
|
||||
temporarily change reporting mode, rerun `App.Execute`, capture stdout and
|
||||
parse it back, or replace native structured serialization to obtain the report.
|
||||
Keep file rendering outside reporter render callbacks to avoid reentrancy.
|
||||
4. Attempt main delivery and then the requested report even when the first
|
||||
fails, retaining every relevant delivery error. Include useful failure
|
||||
explanations and preserve the selected nonzero domain outcome. Console
|
||||
presentation failures remain recorded by CommandKit finalization; report
|
||||
file errors need their own outcome handling because they bypass its reporter.
|
||||
5. Remove the CLI observer/report collector wiring and old report-file format.
|
||||
Leave `internal/report` and HTTP report attachment intact. Only a returned
|
||||
native run result is eligible for the new CLI report file.
|
||||
|
||||
**Evidence:** cover default/text raw bytes and JSON/YAML through stdout, including
|
||||
a subprocess whose stdout is redirected to a file. Compare omitted and explicit
|
||||
report formats against native `RenderReport` of the same value without a template;
|
||||
use an empty format for the omission expectation, not hard-coded YAML. Confirm
|
||||
the separate report retains the full result and metadata, independently of the
|
||||
main format and filename extension. Quiet suppresses main output, including
|
||||
redirected stdout, while an explicitly requested report is still written. Test
|
||||
empty report paths and invalid formats before model execution, replacement
|
||||
without parent creation, stdout failure followed by report attempt, report failure
|
||||
after successful main delivery, and an unchanged existing report on a resultless
|
||||
operational failure. Use temporary files and a small failing-writer seam for
|
||||
stdout/render failures. Confirm status 4 survives a delivery failure while its
|
||||
explanation identifies the failure. No exhaustive filesystem matrix.
|
||||
|
||||
## Stage 7: Wire Process Metadata, Cancellation, And Server Startup
|
||||
|
||||
**Primary files:** `cmd/scriptorium/main.go`, CLI `serve` declaration,
|
||||
`scripts/release.sh`, `.woodpecker/release.yml`, and lifecycle tests.
|
||||
|
||||
1. Build the CLI identity as `scriptorium`, with caller-supplied version and
|
||||
revision. Add entrypoint string variables defaulting to `dev` and empty
|
||||
revision for ordinary builds, suitable for `-X main.version=...` and
|
||||
`-X main.revision=...`. Supply the release version/source commit in the
|
||||
existing release script and hosted build flags. Keep Go 1.27.4, build targets,
|
||||
publication guards, and release-completion semantics unchanged.
|
||||
2. At the process entrypoint, create a signal-aware context for interrupt and
|
||||
SIGTERM, pass it into execution, clean it up before `os.Exit`, and return
|
||||
CommandKit's finalized status without translating it to the old CLI constants.
|
||||
Keep this separately testable without signaling the test runner itself.
|
||||
3. Implement `serve` as an application command consuming Stage 2's resolved
|
||||
server settings. Construct the existing restricted artifact reader and inject
|
||||
it into one PromptKit engine shared by that server's requests. Install the
|
||||
existing HTTP handler and limits; retain `ReadHeaderTimeout` and engine retry
|
||||
defaults. Do not construct engines per request or remove backend capacity
|
||||
coordination.
|
||||
4. Send startup information through native Info reporting and failures through
|
||||
native explanations/causes. Do not add raw stdout server banners. Integrate
|
||||
context cancellation with `http.Server.Close` so the listener and active
|
||||
connections are closed; treat the resulting `ErrServerClosed` as expected
|
||||
cancellation with a useful nonzero operational outcome. Retain operational
|
||||
failure for unrelated listen/serve errors. This adds no graceful-drain deadline
|
||||
or guarantee that context-ignoring work has finished.
|
||||
5. Leave HTTP routes, DTOs, status mappings, optional execution reports,
|
||||
restricted-reader policy, and concurrency contracts unchanged. Continue using
|
||||
the HTTP observer collector; native CLI result reporting does not replace it.
|
||||
|
||||
**Evidence:** test injected context cancellation reaching a real engine operation
|
||||
without a duplicate timer; distinguish native operation timeout from earlier
|
||||
parent cancellation. Include zero/negative operation budgets and independent
|
||||
per-attempt timeout mapping. Exercise actual `serve` startup with a loopback
|
||||
listener, representative HTTP request, and controlled shutdown using explicit
|
||||
readiness coordination. Reuse the HTTP reader/DTO/report/shared-capacity suite
|
||||
rather than duplicating it. Build a binary with test identity flags and check native
|
||||
version output. Test-local fallback deadlines prevent regressions from hanging.
|
||||
|
||||
## Stage 8: Retire Superseded Machinery And Update Canonical Documentation
|
||||
|
||||
**Primary files:** old CLI parser/dispatch/inspection files and tests,
|
||||
`internal/format`, documentation and maintained examples listed below.
|
||||
|
||||
1. Remove obsolete `flag` parsing, comma mapping, old dispatch/help/exit
|
||||
constants, legacy `render`/`inspect` commands, custom source DTOs and
|
||||
inspection writers, custom prepared/definition presentation, and unused
|
||||
CLI report helpers. Delete `internal/format` if it has no remaining consumers;
|
||||
preserve `internal/report` for HTTP. Keep the useful source-to-underlay wiring
|
||||
in its new CLI home, not a resurrected framework facade.
|
||||
2. Replace obsolete tests with selected native integration protection from
|
||||
Stages 2–7. Retain architectural import boundaries and HTTP regression tests.
|
||||
Do not make intentionally retired snapshots or aliases pass by retaining
|
||||
production compatibility code. Avoid testing CommandKit's complete parser
|
||||
or formatting matrix again in Scriptorium.
|
||||
3. Update canonical owners with implemented behavior:
|
||||
|
||||
| Owner | Required changes |
|
||||
| --- | --- |
|
||||
| `docs/cli.md` | New command tree, applicable flags/aliases, native selection/reporting, stdout redirection, separate report files with optional format selection, statuses, completion, explicit smoke tests. Link upstream for standard family details. |
|
||||
| `docs/config.md` | Removed render default; native overlay-name selection; application overrides and loader authority; app/framework scopes and native comparison. Preserve collection/source/default semantics. |
|
||||
| `docs/integrations/subprocess.md` | Artifact-first execution, native reports and result availability, content-bearing metadata reports, stream capture, partial delivery, cancellation, and caller responsibility. Link exact CLI syntax rather than duplicate it. |
|
||||
| `docs/operations.md` | Updated discovery/preview/execution workflow, explicit smoke-test effects, potentially sensitive reports, file retention, server process lifecycle, unchanged HTTP limits. |
|
||||
| `docs/internal/overview.md`, `adapters.md`, `sources.md` | Actual component inventory and data flow, native acquisition boundaries, CLI result versus retained HTTP observation report, selector/engine lifecycle. |
|
||||
| `docs/development.md`, `docs/policy/architecture.md`, `docs/policy/testing.md` | Accurate component/dependency/reading and test ownership descriptions; retain adopted constraints, policy provenance, and accepted ADR decisions. |
|
||||
| `README.md`, `docs/consumers/api.md`, `docs/consumers/upgrading-promptkit.md` | Current quickstart/navigation and any affected examples; distinguish framework-format migration from the new CLI migration. |
|
||||
|
||||
4. Add a focused CLI migration guide under `docs/consumers/`, linked from the
|
||||
CLI/consumer overview. Cover positional prompt selection; `render` and
|
||||
`inspect` replacements; file versus literal inputs; repeated mappings without
|
||||
comma splitting; `--overlay` names; endpoint and timeout spelling/units;
|
||||
stdout redirection in place of output-file flags; separate native reports,
|
||||
optional report-format selection and upstream automatic rendering, and result
|
||||
availability; quiet behavior (omit quiet to capture main output); status
|
||||
changes; and content-bearing native inspection/reports.
|
||||
This prepares release migration information without selecting a version,
|
||||
editing published notes, tagging, pushing, or adding release gates. Eventual
|
||||
release notes must summarize the breaking change and link the guide under
|
||||
the existing release/OSS policies.
|
||||
5. Update maintained application configs to remove the retired render default.
|
||||
Update `examples/render-markdown-summary.sh`, `render-v0.9-features.sh`,
|
||||
README/config examples, and collection workflows to native prepared preview,
|
||||
inspection, positional IDs, and preferred aliases. Existing example script
|
||||
filenames may remain; they are not CLI compatibility commands. Retain both
|
||||
collection-entry spellings and the multi-definition discovery example.
|
||||
6. Keep historical release notes unchanged. Update roadmap/Narratio status only
|
||||
after their acceptance behavior is implemented and verified. Do not silently
|
||||
retire deferred HTTP redesign or dynamic-resource completion proposals.
|
||||
|
||||
**Evidence:** no production command routes to retired parsing/presentation;
|
||||
remaining imports justify every retained helper/package. Maintained YAML loads
|
||||
under the new schema, scripts run without provider access where intended, and
|
||||
current documentation links resolve. Migration examples match the built CLI.
|
||||
|
||||
## Stage 9: Verify The Complete Native Cutover
|
||||
|
||||
1. Finish any deferred stage checks against the complete application. Use real
|
||||
ConfigKit, PromptKit, and CommandKit with deterministic provider substitutes
|
||||
at the external boundary. Tests remain offline and independent of real keys.
|
||||
2. Run a representative end-to-end collection workflow using
|
||||
`examples/config.discovery.yml`: discover resources in support/review,
|
||||
show requirements before input artifacts exist, validate/preview once inputs
|
||||
exist, and run through a controlled provider with consistent selections.
|
||||
Include native app and framework configuration comparisons with independent
|
||||
sides. Reuse the stage tests to cover Narratio's scenarios, not a parallel
|
||||
test harness. Confirm both optional entry-file globs still work and a new
|
||||
checkout is seen on a later invocation, not a hot-reload promise.
|
||||
3. Verify reachable help/version, preferred/canonical aliases, completion
|
||||
generation for the upstream-supported shells, and controlled smoke testing.
|
||||
Help/version/scope discovery/static completion work without configuration or
|
||||
provider availability. Confirm retired commands/flags are absent or rejected
|
||||
except where a spelling now intentionally has its native meaning (`--input`).
|
||||
4. Review the output/outcome matrix across stdout, caller redirection, report
|
||||
files with automatic/explicit formats, quiet, completed validation failure,
|
||||
operational failures, presentation errors, cancellation, and partial delivery.
|
||||
Verify reports are native values with no Scriptorium wrapper, and HTTP's
|
||||
distinct contract remains unchanged.
|
||||
5. Run the repository baseline on the selected toolchain:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./cmd/scriptorium
|
||||
```
|
||||
|
||||
Check `gofmt`, `git diff --check`, affected examples and Markdown links,
|
||||
`go mod tidy` cleanliness, and the selected module graph. Check linked
|
||||
runtime packages separately with `go list -deps ./cmd/scriptorium` when
|
||||
validating attribution. Build the existing Linux amd64/arm64 targets with
|
||||
the unchanged release build options plus the selected identity metadata;
|
||||
this is migration verification, not a new publication gate.
|
||||
6. Audit the final diff against the roadmap and ADRs. In particular, confirm
|
||||
no post-handoff effective-config rewriting, synthetic inspection evidence,
|
||||
legacy CLI compatibility layer, new HTTP restriction, arbitrary secret
|
||||
redaction, extra resource-readiness prerequisite, automatic provider probe,
|
||||
workflow state, snapshot guarantee, or transactional file-delivery claim
|
||||
was introduced. Upstream limits/semantics remain upstream-owned.
|
||||
7. Record completion and the actual dependency/API selections after successful
|
||||
verification. Commit the finished implementation units under the agent
|
||||
policy and report checks, commits, and any remaining work accurately. Do
|
||||
not mark the migration complete while required acceptance behavior is missing.
|
||||
|
||||
There are no unresolved product questions. The optional upstream selected-format
|
||||
accessor is not a migration dependency or completion gate.
|
||||
|
||||
[upstream-guide]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/consumers/guide.md
|
||||
[upstream-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md
|
||||
|
||||
166
docs/roadmap/narratio-feature-request.md
Normal file
166
docs/roadmap/narratio-feature-request.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Prompt input requirements for Narratio
|
||||
|
||||
Status: accepted direction, pending implementation through the CommandKit
|
||||
migration. This proposal records Narratio's consumer needs and their mapping
|
||||
to the native interface; it does not describe shipped Scriptorium behavior.
|
||||
The [CommandKit notepad](commandkit-notepad.md) records the migration decisions.
|
||||
|
||||
## Consumer need
|
||||
|
||||
Narratio produces transcripts, extraction outputs, and generated artifacts in
|
||||
a staged pipeline. It binds named inputs to a Scriptorium prompt when generating
|
||||
each analysis artifact. A Narratio source ID and a prompt input name are separate
|
||||
identities: for example, the source
|
||||
`narratio.transcript.final_trimmed_markdown` might be supplied under the prompt
|
||||
input name `transcript`.
|
||||
|
||||
Narratio needs to discover expected input names before running expensive
|
||||
upstream stages and before the eventual input files exist. It can then show
|
||||
operators how prompt inputs map to pipeline products and identify missing
|
||||
bindings without reading prompt definitions or reproducing PromptKit's analysis.
|
||||
|
||||
## Selected native operation
|
||||
|
||||
Expose `scriptorium promptkit requirements show PROMPT_ID` through ckprompt.
|
||||
Subprocess consumers select JSON with the standard `--format json` flag.
|
||||
Default and explicit text output use the native YAML presentation.
|
||||
The [upstream CLI contract][requirements-cli] owns syntax and report details.
|
||||
|
||||
Use the same application configuration and framework source selection as
|
||||
execution, including `--config`, `--overlay`, and applicable application
|
||||
selectors. The preferred native aliases `--profile`, `--var`, `--session-id`, and
|
||||
prompt/profile namespace flags supply the relevant discovery context.
|
||||
Preserve upstream selection semantics, including the prompt's default profile
|
||||
when an explicit profile is omitted. Qualified IDs identify installed resources;
|
||||
there is no separate prompt-version selector or reported prompt-version field.
|
||||
|
||||
The selected profile establishes identity, not execution readiness. Selecting
|
||||
another profile does not itself change the prompt's discovered requirements.
|
||||
Scriptorium's loading and request mapping must preserve the same effective
|
||||
selection and applicable caller context across discovery and execution.
|
||||
|
||||
Discovery requires no input bindings or output-artifact path. PromptKit performs
|
||||
no artifact loading, schema loading, credential resolution, message rendering,
|
||||
capacity reservation, or model call for this operation. Reading configuration
|
||||
and prompt resources needed to build the captured engine is expected. The
|
||||
[PromptKit discovery contract][requirements-api] owns the analysis and its
|
||||
limits.
|
||||
|
||||
## Native response and consumer interpretation
|
||||
|
||||
Consume the native JSON report directly, without a Scriptorium-specific wrapper
|
||||
or schema-version field. Successful report delivery writes one JSON document to
|
||||
stdout, with diagnostics on stderr under CommandKit's standard reporting rules.
|
||||
Consumers requesting a report should not enable quiet mode.
|
||||
|
||||
| Field | Consumer meaning |
|
||||
| --- | --- |
|
||||
| `prompt_id`, `profile_id` | Exact selected resource identities. |
|
||||
| `inputs` | Declared and recognizable template-referenced inputs, including exact `name`, requirement `kind`, `declared`, `content_type`, `description`, and `explanation`. |
|
||||
| `variables` | Recognizable caller-variable keys with `name`, requirement `kind`, `supplied`, and `explanation`. |
|
||||
| `complete` | Whether the supported dependency analysis is complete for the supplied context. |
|
||||
| `issues` | Structured discovery limitations with a `code`, associated input or variable names where known, and a human-readable description. |
|
||||
|
||||
PromptKit sorts input and variable entries by exact name, and deduplicates and
|
||||
orders issues by their code and associated names. Preserve empty inventories
|
||||
and false values. Input metadata distinguishes absent/null values from an
|
||||
explicitly empty string. Explanations and issue descriptions are for display;
|
||||
Narratio must use structured fields rather than parse their prose.
|
||||
|
||||
The native input metadata includes content type and description. Current
|
||||
PromptKit inputs have no per-input schema declaration to project. Do not infer
|
||||
constraints from names or treat the output-validation schema as an input schema.
|
||||
Variable `supplied` records exact-key presence, including an explicitly empty
|
||||
value; this interface does not introduce a separate variable-default system.
|
||||
|
||||
Requirements reports exclude artifact payloads, prompt contents, and unrelated
|
||||
supplied variable values. A supplied value used to resolve an input identifier
|
||||
can appear as that identifier. Keep this native behavior without adding a custom
|
||||
projection of the report.
|
||||
|
||||
The upstream report contract and adopted dependency versions govern the shape.
|
||||
Migration documentation must identify relevant consumer changes; this proposal
|
||||
adds no independent response-versioning mechanism or stronger compatibility
|
||||
guarantee.
|
||||
|
||||
## Conditional and incomplete discovery
|
||||
|
||||
PromptKit classifies dependencies as required, optional, or conditional using
|
||||
the supplied context and its supported template analysis. Recognizable references
|
||||
only on known skipped paths can be optional. Explicit required declarations
|
||||
still apply. Unavailable input contents, unresolved variable values or dynamic
|
||||
names, and unsupported dependency-relevant constructs can leave discovery
|
||||
incomplete. Valid templates are not rejected merely because the analysis cannot
|
||||
fully resolve them.
|
||||
|
||||
Narratio must inspect both the requirements and discovery completeness:
|
||||
|
||||
- `complete: true` establishes discovery coverage, not satisfied bindings,
|
||||
valid future input bytes, or execution readiness. Required inputs or missing
|
||||
required variables may still be present.
|
||||
- `complete: false` preserves useful findings and structured issues. An empty
|
||||
partial inventory does not prove that no inputs are needed.
|
||||
- A partial report can return status zero. An issue is not itself an invocation
|
||||
failure. Narratio owns its policy for proceeding with incomplete dependency
|
||||
information.
|
||||
|
||||
Once input artifacts exist, native `promptkit requirements validate` can check
|
||||
a concrete request through preparation. It is a separate operation and must not
|
||||
replace early discovery. Neither operation guarantees later generation success.
|
||||
Changes to configuration or resources between invocations can change findings.
|
||||
|
||||
## Failures
|
||||
|
||||
Use CommandKit's native nonzero outcomes and diagnostic messages for selection,
|
||||
configuration, operational, and report-delivery failures. JSON report selection
|
||||
does not turn diagnostic messages into a categorized JSON error response.
|
||||
Narratio can attach the diagnostic to its own artifact or pipeline context;
|
||||
it must not depend on parsing the message to recover a stable error category.
|
||||
|
||||
Structured discovery issue codes belong to successful inspection reports and
|
||||
are distinct from invocation failures. Follow PromptKit's distinction between
|
||||
partial discovery and failures that return no inspection. Absence of a future
|
||||
input artifact is not a discovery failure because discovery does not load it.
|
||||
|
||||
The accepted consumer requirement is a nonzero return code plus failure
|
||||
information. If programmatic failure categories or separate schema versioning
|
||||
become necessary, discuss them upstream rather than introducing local wrappers.
|
||||
Neither is a prerequisite for this migration.
|
||||
|
||||
## Consumer acceptance scenarios
|
||||
|
||||
1. A prompt requires `transcript`, but Narratio binds only `session_transcript`.
|
||||
The native inventory identifies the exact required name before either file
|
||||
exists, allowing Narratio to identify the missing binding.
|
||||
2. A template references an undeclared input. Recognizable references appear in
|
||||
the inventory with `declared: false`; unsupported dependency analysis is
|
||||
explicitly incomplete rather than silently claiming full coverage.
|
||||
3. An optional glossary is reported with its native classification, allowing
|
||||
Narratio to distinguish it from a required binding.
|
||||
4. Variables resolve a supported conditional dependency. Unknown conditions
|
||||
retain conditional findings or discovery issues according to PromptKit.
|
||||
5. Selecting another qualified prompt or profile reports the corresponding
|
||||
identities. Prompt selection changes the analyzed definition; profile
|
||||
selection alone does not change its requirements.
|
||||
6. Discovery works with no generated input files, model access, or generation
|
||||
credentials when the selected configuration and resources are readable and
|
||||
valid for discovery.
|
||||
7. An unknown prompt or invalid profile selection returns nonzero with native
|
||||
diagnostics suitable for Narratio to associate with the affected artifact.
|
||||
8. Narratio treats a complete report with missing required bindings differently
|
||||
from a complete, satisfied dependency check, and handles partial reports
|
||||
without assuming that status zero establishes completeness.
|
||||
9. Scriptorium's adapter uses consistent configuration, namespaces, and applicable
|
||||
request context across discovery, validation, preview, and execution.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
Narratio owns its source catalog, stage ordering, source-to-input bindings,
|
||||
artifact dependency graph, and session availability checks. Scriptorium exposes
|
||||
PromptKit's native requirements through CommandKit. Narratio source IDs and
|
||||
pipeline concepts do not become part of that interface. A graph renderer, HTTP
|
||||
endpoint, batch operation, and changes to generation behavior are outside this
|
||||
request.
|
||||
|
||||
[requirements-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/commit/d330d7637f2e7507eca931a899397222c82bc892/docs/cli.md#promptkit-requirements
|
||||
[requirements-api]: https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/requirements.go
|
||||
@@ -5,8 +5,8 @@
|
||||
Agreed and delivered on 2026-09-28. This roadmap replaces the
|
||||
earlier licensing-only proposal with alignment of the complete applicable
|
||||
policy set and its supporting documentation. It defines the targeted end state
|
||||
and required changes; the [implementation plan](implementation.md) owns ordered
|
||||
stages and their verification.
|
||||
and required changes. Its completed implementation plan is preserved in Git at
|
||||
`git show 8b6c12f912b28ac00016f047f9846798a5de95d9:docs/roadmap/implementation.md`.
|
||||
|
||||
The next release summary should mention the policy and licensing alignment,
|
||||
the already-selected GPL-3.0-or-later application grant, and the separate
|
||||
|
||||
374
docs/roadmap/promptkit.md
Normal file
374
docs/roadmap/promptkit.md
Normal file
@@ -0,0 +1,374 @@
|
||||
# PromptKit v0.12.0 Adoption
|
||||
|
||||
## Status And Intent
|
||||
|
||||
Drafted on 2026-10-03; source-selection and framework-inspection decisions
|
||||
confirmed by the maintainer on 2026-10-04. The PromptKit v0.12.0 adoption has
|
||||
been implemented and delivered after the complete audit passed on 2026-10-04.
|
||||
This document remains target and decision context, not an implementation-stage
|
||||
log. Release preparation remains separate; no release version has been
|
||||
selected or published. No significant decisions remain open within this scope.
|
||||
|
||||
Adopt PromptKit's ConfigKit-native configuration boundary while preserving
|
||||
Scriptorium's simple, request-independent execution model. As established in
|
||||
[ADR 0004](../adr/0004-definition-boundary.md), valid prompt and profile
|
||||
definitions for the selected PromptKit version should work without a narrower
|
||||
Scriptorium field-level format. Library-only APIs are considered separately;
|
||||
this upgrade does not turn Scriptorium into a workflow application.
|
||||
|
||||
The maintainer selected a native source cutover and full framework configuration
|
||||
inspection. The latter uses upstream's native redaction in this new interface
|
||||
only. These are deliberate product choices, not requirements inferred merely
|
||||
from new library APIs.
|
||||
|
||||
## Reviewed Baseline And Upstream Target
|
||||
|
||||
The reviewed Scriptorium source is commit
|
||||
`bb5006517b6c27dcbe56074f1d97140bf0999c48`; the last release is v0.15.0.
|
||||
The completed ConfigKit adoption selects ConfigKit
|
||||
v0.5.0. PromptKit remains v0.11.2, with the OpenRouter and Rakestrawhome `/v2`
|
||||
catalog modules at v2.20260926.0.
|
||||
|
||||
Application configuration already uses one ConfigKit loading, inheritance,
|
||||
resolution, validation, and inspection pipeline. Framework construction still
|
||||
passes `prompt_dir` and `profile_dir` into the removed PromptKit directory
|
||||
fields and registers application `backends` with `WithBackend`. Profile
|
||||
inspection substitutes an empty prompt filesystem to avoid loading prompts.
|
||||
Maintained prompt/profile files and inline test fixtures use the old embedded
|
||||
`id`, `version`, and `base_profile` syntax.
|
||||
|
||||
The target is the published PromptKit v0.12.0 release at full commit
|
||||
`4b8169da286917fde7875425bb32b799b12c6af3`. The published tag and downloaded
|
||||
module identify that same revision. Canonical references are:
|
||||
|
||||
- [Release notes](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/docs/releases/v0.12.0.md).
|
||||
- [Framework formats](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/docs/formats.md).
|
||||
- [Consumer and migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/docs/consumers/pkg-promptkit.md).
|
||||
- [Configuration ownership decision](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/docs/adr/0001-configuration-boundary.md).
|
||||
|
||||
PromptKit now also selects ConfigKit v0.5.0, so no further ConfigKit upgrade is
|
||||
needed. Go 1.25.5 remains sufficient. Its catalog selections are:
|
||||
|
||||
| Module | Version | Full source commit |
|
||||
| --- | --- | --- |
|
||||
| `promptkit-backend-openrouter/v3` | `v3.20260930.0` | `f17d208cb7ad1fb043cda2266cd3011d11b40aa0` |
|
||||
| `promptkit-backend-rakestrawhome/v3` | `v3.20260930.0` | `993443d88c451901f4c917fa00c204fd3b54d28d` |
|
||||
|
||||
This review examined upstream contracts and the affected Scriptorium adapters,
|
||||
configuration, presentation, examples, and tests. It did not establish that the
|
||||
unmodified application builds against the new, incompatible construction API.
|
||||
|
||||
## Targeted End State
|
||||
|
||||
### Framework Ownership And Source Assembly
|
||||
|
||||
Scriptorium constructs engines through PromptKit's public root package and
|
||||
`Config.Sources`. ConfigKit owns source composition, overlays, inheritance,
|
||||
presence, null values, and resource origins. PromptKit owns final definition
|
||||
validation, catalog contributions, backend completion, content capture,
|
||||
preparation, execution, and capacity coordination.
|
||||
|
||||
An operator-selected source may contain any combination of keyed `backends`,
|
||||
`profiles`, and versioned `prompts`. Do not partition or filter documents by
|
||||
directory name, filename, or a Scriptorium copy of the framework schema.
|
||||
Do not independently preflight domain fields in lower-priority definitions
|
||||
that the final effective configuration replaces. Syntax and composition
|
||||
failures remain upstream loading failures.
|
||||
|
||||
The selected application interface is an ordered `promptkit_sources` sequence.
|
||||
Each entry selects either a `file` or a `directory` and contributes one caller
|
||||
underlay, from lowest to highest priority. A directory uses
|
||||
`promptkit.DirectoryInputs` and strictly composes its recursively discovered
|
||||
YAML files. A file uses `configkit.File`
|
||||
with declared imports enabled, so an entry document can compose fragments.
|
||||
All caller groups follow PromptKit's maintained catalogs and owned defaults.
|
||||
No separately exposed `Sources.Main` setting is needed for this interface.
|
||||
|
||||
For example, the target application shape is:
|
||||
|
||||
```yaml
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./deployment/promptkit.yml
|
||||
schema_dir: ./examples/schemas
|
||||
```
|
||||
|
||||
Directories retain the helper's default of disabled imports. Enabling imports
|
||||
while also recursively discovering their YAML targets would include those
|
||||
files twice in the same group, which ConfigKit rejects. Use a selected entry
|
||||
file with imports for that layout; do not silently deduplicate, skip files, or
|
||||
convert composition conflicts into precedence.
|
||||
|
||||
Under this decision, `prompt_dir`, `profile_dir`, application `backends`,
|
||||
and their directory flags are replaced, not retained as a second configuration
|
||||
model. Framework backend definitions live with the framework sources rather
|
||||
than passing through Scriptorium's scalar backend schema. A repeatable
|
||||
`--promptkit-source PATH` selects a replacement ordered source list for
|
||||
engine-using commands; each path selects a regular file or directory using the
|
||||
same rules. Only an explicitly supplied flag overrides the configured list.
|
||||
|
||||
Application source discovery, its imports and selected overlay, explicit CLI
|
||||
override handling, and working-directory-relative application paths remain
|
||||
unchanged. `--config-overlay` still selects an application overlay; it does not
|
||||
implicitly select a PromptKit layer or profile. Framework priority is explicit
|
||||
in the source list, with import/resource containment supplied by upstream for
|
||||
each selected root. This is not a new global filesystem sandbox.
|
||||
|
||||
### Complete Definition Support
|
||||
|
||||
The target accepts the keyed v0.12.0 formats without compatibility rewriting:
|
||||
profile and backend IDs are mapping keys, prompt IDs contain keyed string
|
||||
versions, and `extends` replaces `base_profile`. Literal IDs and references
|
||||
retain their exact spelling, including punctuation and meaningful whitespace;
|
||||
filenames do not establish identities.
|
||||
|
||||
All supported prompt/profile fields continue to reach PromptKit, including
|
||||
inputs, message roles and cache metadata, inline and file content, session
|
||||
templates, output validation and repair, backend/endpoint selection,
|
||||
credentials by environment name, generation controls, and provider parameters.
|
||||
The upstream format reference, not this list, owns the complete contract.
|
||||
|
||||
Presence must survive the boundary: omission, explicit zero/false/blank, and
|
||||
null are not interchangeable. Inheritance and layers recursively combine maps;
|
||||
an empty map does not clear inherited entries. Backend defaults apply after
|
||||
effective profile inheritance. Changing a model or backend no longer drops
|
||||
unrelated explicitly inherited settings. Profile null suppression and explicit
|
||||
zero controls must affect outbound requests as specified upstream.
|
||||
|
||||
Partial profiles remain valid when request overrides can complete them.
|
||||
Scriptorium must not introduce a mandatory model, credential, default profile,
|
||||
or directory prerequisite that prevents such a request. With the selected
|
||||
source interface, engine construction may have no prompt collection; missing
|
||||
request selections fail through PromptKit's ordinary public errors rather than
|
||||
a required `prompt_dir` check. Source selection does not imply provider access
|
||||
or credential availability.
|
||||
|
||||
### Captured Definitions And Runtime Lifecycle
|
||||
|
||||
Engine construction validates every final effective definition, including
|
||||
unused entries, and captures message-file content and static template syntax.
|
||||
Invalid unused definitions or unreadable message resources may therefore fail
|
||||
startup or inspection before a request selects them. An overwritten domain
|
||||
value is not independently rejected merely because it appears in history.
|
||||
|
||||
ConfigKit's structural directives now also apply to framework input mappings,
|
||||
including dynamic provider maps. Document the reserved `extends` key and the
|
||||
difference from literal runtime provider parameters; do not invent a quoting
|
||||
escape or a second inheritance resolver.
|
||||
|
||||
Content files use their effective field's source context, including imported
|
||||
and inherited content. Retain those contexts by passing actual upstream
|
||||
inputs; do not flatten files into synthetic mappings that lose origins.
|
||||
Application paths remain working-directory-relative, whereas framework message
|
||||
resources follow PromptKit's source-relative contract. Schema and request
|
||||
artifact reads remain preparation-time resources; credentials and provider
|
||||
operations retain their documented runtime boundaries.
|
||||
|
||||
A server retains one constructed engine across its requests, with shared
|
||||
backend capacity and queue accounting. Definition/message edits require a new
|
||||
engine; for this scope, restarting the server is the supported way to obtain
|
||||
one. CLI invocations naturally construct fresh engines. No file watcher,
|
||||
reload endpoint, per-request server reconstruction, or durable state is added.
|
||||
This is not a transactional snapshot of all external files or credentials.
|
||||
|
||||
### Definition And Application Inspection
|
||||
|
||||
`inspect prompt` and `inspect profile` use the same selected configuration
|
||||
snapshot as execution. Remove the empty-prompt-filesystem substitution.
|
||||
Profile inspection still does not need request inputs, schema reads, model
|
||||
calls, or credential values, but constructing its engine now validates and
|
||||
captures all selected definitions, including prompts. A partial target may be
|
||||
inspected without imposing preparation's completeness requirements.
|
||||
|
||||
`config validate`, `show`, `sources`, and `explain` remain application
|
||||
configuration operations without constructing an engine. Under the selected
|
||||
source cutover, they show source selections rather than the contents of
|
||||
PromptKit's separately loaded definitions. They are not framework validation.
|
||||
The separate framework inspection family below supplies framework validation
|
||||
and captured provenance.
|
||||
|
||||
### Captured Framework Configuration Inspection
|
||||
|
||||
Add the CLI command family `inspect configuration` with `validate`, `show`,
|
||||
`sources`, and `explain` operations. It uses the same application settings,
|
||||
source selections, and engine construction as other engine-using commands,
|
||||
without requiring a prompt/profile selection or request inputs. Each invocation
|
||||
constructs one engine; reporting uses that engine's captured public view rather
|
||||
than loading definitions again.
|
||||
|
||||
- `validate` succeeds when the engine constructs successfully and produces no
|
||||
success report. It validates final definitions and captures message resources,
|
||||
not provider access, credential availability, or request-specific preparation.
|
||||
- `show` writes the captured effective YAML through
|
||||
`ConfigurationInspection.WriteEffective`.
|
||||
- `sources` presents structured JSON source records from
|
||||
`ConfigurationInspection.Sources`, including the source kinds, layers, and
|
||||
underlay positions needed to distinguish maintained and operator inputs.
|
||||
- `explain --path JSON_POINTER` uses the native explanation view and writer
|
||||
for effective values and history. Root and literal mapping-key paths follow
|
||||
the same JSON Pointer conventions as application `config explain`; IDs with
|
||||
slashes or punctuation must not become dotted-path lookups.
|
||||
|
||||
Use PromptKit's public `InspectConfiguration` and its native writers and
|
||||
records. Surface applicable load diagnostics safely on stderr; successful
|
||||
reports go to stdout. Inspection does not resolve credential values, contact
|
||||
a provider, or watch/reopen sources after construction. Captured effective
|
||||
configuration is distinct from backend-completed `inspect profile` output and
|
||||
request-specific prepared output; none substitutes for the others.
|
||||
|
||||
The maintainer explicitly accepts upstream's credential-shaped provider-key
|
||||
redaction for this new family, including effective and historical values.
|
||||
Fields such as `token` and `password` can therefore be redacted here while
|
||||
remaining visible in existing profile/prepared views. This is not blanket
|
||||
redaction of `extra_params`, a change to parameters sent to providers, or a
|
||||
change to existing inspection interfaces. Environment-reference names and
|
||||
ordinary unclassified content can remain visible. Reports may contain prompt
|
||||
text, provider data, paths, and identifiers; native redaction is not a promise
|
||||
that the output is safe to disclose.
|
||||
|
||||
### Requests, Presentation, And Disclosure
|
||||
|
||||
Keep CLI/HTTP request independence, presence-aware execution overrides,
|
||||
generation/whole-run timeout separation, transient retries, execution reports,
|
||||
artifact policy, and public error mapping. Preserve existing serialized results
|
||||
unless an upstream change actually requires an application contract change.
|
||||
Prepared/profile numeric values alone are not proof that a zero control was
|
||||
sent; verify upstream presence through generation or recorded outbound requests.
|
||||
The dependency change alone does not require a new presence DTO. Continue
|
||||
mapping errors by public identity rather than message text.
|
||||
|
||||
Environment-variable names remain visible; resolved API keys are never
|
||||
printed or serialized. Ordinary `extra_params` retain their selected visibility
|
||||
where those values are presented. Moving definitions out of application
|
||||
settings changes what `config` can inspect, not their sensitivity classification.
|
||||
Do not silently adopt broader redaction rules for existing profile/prepared
|
||||
inspection.
|
||||
|
||||
## Work Needed From The Baseline
|
||||
|
||||
- **Dependencies and attribution:** select released PromptKit v0.12.0, tidy the
|
||||
module graph, and use its selected `/v3` catalogs without direct application
|
||||
imports. Remove obsolete `/v2` modules and `gopkg.in/yaml.v3` if no remaining
|
||||
imports require them. Update PromptKit/catalog component entries and exact
|
||||
source, COPYRIGHT, and LICENSE links in `docs/oss/attribution.md`. The reviewed
|
||||
grants remain LGPL-3.0-or-later for PromptKit and MIT for the catalogs; reuse
|
||||
the applicable local license texts. Preserve unrelated attribution entries
|
||||
and historical release documents.
|
||||
- **Application settings and CLI:** implement the selected source interface
|
||||
through `internal/config` and shared CLI override mapping. Retire the backend
|
||||
document/runtime conversion and old directory fields, add ordered source
|
||||
selection, and update command help and prerequisites. Application validation
|
||||
checks its schema and selector shape;
|
||||
framework acquisition and domain validation belong to engine construction.
|
||||
- **Shared engine wiring:** replace removed fields/options with upstream
|
||||
inputs and underlays, retain `SchemaDir`, `TransientRetries`, and runtime
|
||||
collaborator injection, and use the same construction path for execution,
|
||||
rendering, definition inspection, and serving. Preserve identifiable
|
||||
PromptKit and ConfigKit causes and safe diagnostics. Remove obsolete
|
||||
prompt-filesystem inspection scaffolding.
|
||||
- **Captured framework inspection:** add the `inspect configuration` family
|
||||
with `validate`, `show`, `sources`, and `explain` using the shared engine path
|
||||
and its public captured inspection view. Reuse native YAML/explanation writers,
|
||||
preserve useful source-layer information in JSON presentation, and keep application
|
||||
`config` independent of framework loading. Apply the approved native
|
||||
redaction exception only to this new family.
|
||||
- **Definitions and fixtures:** migrate every maintained prompt/profile file
|
||||
under `examples/` and `testdata/`, and inline definitions and construction
|
||||
helpers in CLI, HTTP, and formatting tests. Replace old inheritance syntax
|
||||
and move backend examples into framework definitions. Review old explicit
|
||||
zeros for their changed meaning instead of preserving obsolete expectations
|
||||
mechanically. Keep useful feature-adoption coverage; filenames need not be
|
||||
renamed merely to reduce historical naming.
|
||||
- **Focused adoption coverage:** use real ConfigKit/PromptKit boundaries for
|
||||
representative source order, same-group conflicts, file imports, combined
|
||||
documents, inherited source-relative content, exact IDs/version selection,
|
||||
null suppression, explicit zero/false, and partial-profile completion.
|
||||
Verify eager failures for unused final definitions and unchanged captured
|
||||
message content after file edits on an existing engine. Retain shared-server
|
||||
capacity, retry, cancellation, restricted artifact, output, and error tests.
|
||||
Cover framework inspection's effective YAML, source layers, literal-key
|
||||
explanations, diagnostics, and effective/historical native redaction without
|
||||
provider calls. Protect the distinction between application validation and
|
||||
framework validation, and unchanged visibility in existing profile/prepared
|
||||
inspection. Avoid snapshots of incidental writer prose.
|
||||
Own integration risks locally without recreating upstream's exhaustive parser
|
||||
or inheritance matrices; use offline clients or `httptest` for generation.
|
||||
- **Current documentation and migration guidance:** update owning contracts
|
||||
in `docs/config.md`, `docs/cli.md`, and `docs/operations.md`, affected internal
|
||||
docs, `docs/development.md`, architecture version/ownership references,
|
||||
README, examples, and applicable consumer guidance. Link to v0.12.0 framework
|
||||
contracts rather than copying their specification. Correct statements that
|
||||
ConfigKit does not participate in framework loading. Explain the selected
|
||||
source migration, framework-inspection command family and scoped redaction
|
||||
exception, and engine-rebuild requirement. Preserve accepted ADR
|
||||
decision content. Update HTTP docs only where observable behavior changes.
|
||||
|
||||
## Compatibility And Release Impact
|
||||
|
||||
This is an intentional breaking adoption, not a patch-only dependency refresh.
|
||||
Users must migrate definitions, inheritance syntax, source assembly, and
|
||||
assumptions about numeric presence, recursive maps, eager validation, and file
|
||||
reload. Under the selected native interface they must also migrate the
|
||||
application source selectors and move application backend definitions into
|
||||
PromptKit source files. Provide small before/after examples and runnable
|
||||
offline migration checks when implementation lands; no automatic legacy parser
|
||||
or catalog alias layer is proposed.
|
||||
|
||||
PromptKit's maintained profile IDs remain supported, not frozen to fixed model
|
||||
settings. Review used external catalog IDs against the selected catalogs; do
|
||||
not promise live provider availability or preserve every volatile inventory ID.
|
||||
Hashes remain opaque equality values, not cross-version identity guarantees.
|
||||
|
||||
Under the [release policy](../policy/release.md), the eventual release must be
|
||||
a new pre-1.0 minor version. Do not select or publish a release in this work
|
||||
set. The previously completed ConfigKit work may share that release; final
|
||||
version, notes, and roadmap retirement belong to release preparation.
|
||||
|
||||
## Completion Criteria And Scope Limits
|
||||
|
||||
Adoption is complete when the released dependency graph is selected, the
|
||||
approved source interface reaches the canonical framework boundary without
|
||||
losing definition semantics, maintained examples and fixtures use v0.12.0,
|
||||
and CLI/HTTP workflows and inspection meet the target above. The new framework
|
||||
inspection family must use captured native reports with its explicitly scoped
|
||||
redaction behavior, without changing existing inspection disclosure. Run the
|
||||
existing development checks, affected offline examples, and link/diff checks,
|
||||
with focused new integration tests for the actual migration risks.
|
||||
|
||||
This work does not add conversations, appended/follow-up messages,
|
||||
prepared-handle coordination, semantic retry workflows, embedded configuration
|
||||
bundles, framework internals, or application-owned framework validation.
|
||||
Library availability alone does not justify additional commands, endpoints,
|
||||
resource controls, recurring upstream probes, or release gates. Deferred
|
||||
interfaces are scope choices for this adoption, not permanent prohibitions.
|
||||
|
||||
## Confirmed Decisions
|
||||
|
||||
### Native Source Cutover
|
||||
|
||||
The maintainer accepted the ordered `promptkit_sources` file/directory interface
|
||||
and moving backend definitions out of application settings. This exposes
|
||||
combined definitions, explicit priority, imports, and full presence/null
|
||||
semantics through their canonical owner. It removes duplicated backend
|
||||
representation and avoids translating absent scalar fields into presence-aware
|
||||
overlays. The deliberate application schema and CLI migration accompanies the
|
||||
unavoidable framework format migration.
|
||||
|
||||
Retaining the old directories and application backend surface was considered
|
||||
but not selected: it would reduce migration while retaining duplicate backend
|
||||
configuration, presence/null translation, and a less flexible source interface.
|
||||
|
||||
### Native Framework Inspection With A Scoped Redaction Exception
|
||||
|
||||
The maintainer selected `inspect configuration` using the captured native view,
|
||||
including upstream redaction in this new interface only, rather than deferring
|
||||
the feature. Provenance and history make layered configuration easier to
|
||||
diagnose, and native writers avoid maintaining a parallel representation.
|
||||
|
||||
The accepted tradeoff is different provider-parameter visibility between this
|
||||
new view and existing profile/prepared inspection. Existing API-key-only
|
||||
redaction choices remain otherwise unchanged. Deferral or an upstream
|
||||
consumer-selectable sensitivity API could avoid that difference, but neither is
|
||||
a prerequisite for this adoption. Do not reconstruct private configuration to
|
||||
bypass native writers or depend on an unpublished upstream interface.
|
||||
@@ -1,77 +0,0 @@
|
||||
# Promptkit Upgrade Review Remediation
|
||||
|
||||
## Scope And Status
|
||||
|
||||
Review baseline: `e177d7c`, covering implementation commits after `5c3b0cf`.
|
||||
This document records a resolved test-contract issue; it does not authorize
|
||||
unrelated implementation changes. No runtime defect or missing planned feature
|
||||
was identified in this review. Release preparation remains separate work.
|
||||
|
||||
## R1: Remove The Unrequested Context-Identity Invariant
|
||||
|
||||
Priority: P3 (low). Status: Complete.
|
||||
|
||||
Original location: `internal/adapter/http/handler_test.go:334–349`,
|
||||
`TestHandlerRunTimeoutValidationAndContextMapping`, introduced in `7de2b02`.
|
||||
|
||||
The test's `child` field and `(ctx != parent) != tc.child` assertion required
|
||||
the exact parent context object for omitted/null/zero timeouts and a distinct
|
||||
object for a positive timeout. The requested contract concerns effective
|
||||
deadlines and cancellation propagation, not Go interface identity. In
|
||||
particular, a harmless derived context that preserves the parent's deadline,
|
||||
values, and cancellation would fail the omission cases despite satisfying the
|
||||
application contract.
|
||||
|
||||
This elevated an incidental implementation choice into a test-enforced
|
||||
invariant, contrary to the [agent policy](../policy/agents.md) and
|
||||
[testing policy](../policy/testing.md). The current handler's context behavior
|
||||
is correct; this is a maintainability issue, not a runtime cancellation bug.
|
||||
|
||||
### Required Change
|
||||
|
||||
Remove the identity comparison and the identity-only table field. Retain
|
||||
behavioral assertions that an earlier parent deadline is not extended and
|
||||
parent cancellation reaches execution. For omitted/null/zero values, verify
|
||||
that no deadline is introduced when the parent has none. Retain coverage of
|
||||
positive deadlines and the distinction between a canceled execution context
|
||||
and a provider-attempt timeout while the execution context remains live.
|
||||
|
||||
Use the existing runner seam and focused cases; do not introduce a context
|
||||
wrapper framework, timer abstraction, new public guarantee, or production
|
||||
refactor solely to change this test.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Tests protect effective deadline/cancellation behavior without requiring
|
||||
context object identity.
|
||||
- A semantically equivalent derived context does not fail merely because it
|
||||
is a different object.
|
||||
- Existing timeout validation and HTTP outcome coverage remains intact.
|
||||
|
||||
### Resolution And Validation
|
||||
|
||||
Removed the identity-only table field and context comparisons. The test now
|
||||
checks effective deadlines, including omitted/null/zero requests with no
|
||||
parent deadline, and explicitly checks that parent cancellation reaches the
|
||||
runner. Existing positive-deadline integration tests and live-context
|
||||
provider-timeout mapping remain intact. No production code changed.
|
||||
|
||||
Focused HTTP timeout tests, `go test ./...`, `go test -race ./...`,
|
||||
`go vet ./...`, a native build outside the repository, and `git diff --check`
|
||||
all passed after remediation.
|
||||
|
||||
## Original Review Verification
|
||||
|
||||
- `go test -race -count=1 ./...` passed, including retry, shared-capacity,
|
||||
deadline, report, definition-conformance, and architecture-boundary tests.
|
||||
- `go vet ./...` passed.
|
||||
- A native `cmd/scriptorium` build passed, with output outside the repository.
|
||||
- The selected dependency is Promptkit v0.11.0 with its intended v2 catalogs.
|
||||
- Application code, tests, release automation, and historical releases were
|
||||
not changed during this review.
|
||||
|
||||
The runtime ownership boundaries remain as intended: Promptkit owns framework
|
||||
formats, execution, retries, and backend admission; Scriptorium owns executable
|
||||
adapters, application settings, HTTP artifact policy, deadlines, and terminal
|
||||
report presentation. No additional runtime restrictions, durable workflow
|
||||
state, or extra release compliance gates were identified in the reviewed diff.
|
||||
@@ -1,4 +0,0 @@
|
||||
backends:
|
||||
local-gpu:
|
||||
endpoint: http://localhost:11434/v1
|
||||
api_key_env: LOCAL_GPU_API_KEY
|
||||
2
examples/config-fragments/collections/10-backends.yml
Normal file
2
examples/config-fragments/collections/10-backends.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
promptkit_sources:
|
||||
- file: ./examples/framework/backends.yml
|
||||
2
examples/config-fragments/collections/20-profile.yml
Normal file
2
examples/config-fragments/collections/20-profile.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
promptkit_sources:
|
||||
- file: ./examples/profiles/local-fast.yaml
|
||||
2
examples/config-fragments/collections/30-prompt.yml
Normal file
2
examples/config-fragments/collections/30-prompt.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
promptkit_sources:
|
||||
- file: ./examples/prompts/generic.markdown_summary.yaml
|
||||
@@ -1,3 +1,5 @@
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/backends.yml
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
10
examples/config.collections.yml
Normal file
10
examples/config.collections.yml
Normal file
@@ -0,0 +1,10 @@
|
||||
config:
|
||||
sequences:
|
||||
- destination: [promptkit_sources]
|
||||
key: file
|
||||
mode: upsert
|
||||
imports:
|
||||
- dir: config-fragments/collections
|
||||
pattern: '*.yml'
|
||||
required: true
|
||||
schema_dir: ./examples/schemas
|
||||
@@ -1,8 +1,8 @@
|
||||
# Select from the repository root; application paths remain working-directory-relative.
|
||||
configkit:
|
||||
config:
|
||||
imports:
|
||||
- path: config-fragments/sources.yml
|
||||
- path: config-fragments/backend.yml
|
||||
required: true
|
||||
|
||||
server:
|
||||
addr: :8080
|
||||
|
||||
2
examples/config.discovery.yml
Normal file
2
examples/config.discovery.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
promptkit_sources:
|
||||
- file: ./examples/discovery/framework.yml
|
||||
@@ -1,16 +1,9 @@
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/backends.yml
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
backends:
|
||||
local-gpu:
|
||||
endpoint: http://localhost:11434/v1
|
||||
api_key_env: LOCAL_GPU_API_KEY
|
||||
extra_params:
|
||||
provider_option: enabled
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 0
|
||||
|
||||
server:
|
||||
addr: 127.0.0.1:8080
|
||||
artifact_root: .
|
||||
|
||||
6
examples/config.inheritance.yml
Normal file
6
examples/config.inheritance.yml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Application paths are relative to the repository invocation directory.
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/inheritance.yml
|
||||
schema_dir: ./examples/schemas
|
||||
@@ -1,5 +1,7 @@
|
||||
prompt_dir: ./examples/prompts
|
||||
profile_dir: ./examples/profiles
|
||||
promptkit_sources:
|
||||
- directory: ./examples/prompts
|
||||
- directory: ./examples/profiles
|
||||
- file: ./examples/framework/backends.yml
|
||||
schema_dir: ./examples/schemas
|
||||
|
||||
server:
|
||||
|
||||
12
examples/discovery/collections/review-repo/collection.yaml
Normal file
12
examples/discovery/collections/review-repo/collection.yaml
Normal file
@@ -0,0 +1,12 @@
|
||||
config:
|
||||
imports:
|
||||
- path: profiles/base.yaml
|
||||
destination: [collections, review, profiles]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
- path: prompts/answer.yaml
|
||||
destination: [collections, review, prompts]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
collections:
|
||||
review: {}
|
||||
@@ -0,0 +1,3 @@
|
||||
id: base
|
||||
endpoint: https://example.invalid/v1
|
||||
model: review-example
|
||||
@@ -0,0 +1,8 @@
|
||||
id: answer
|
||||
default_profile: review/base
|
||||
messages:
|
||||
- role: user
|
||||
content: Review this material and report your findings.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
20
examples/discovery/collections/support-repo/collection.yml
Normal file
20
examples/discovery/collections/support-repo/collection.yml
Normal file
@@ -0,0 +1,20 @@
|
||||
config:
|
||||
imports:
|
||||
- path: profiles/base.yml
|
||||
destination: [collections, support, profiles]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
- path: profiles/fast.yml
|
||||
destination: [collections, support, profiles]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
- path: prompts/answer.yml
|
||||
destination: [collections, support, prompts]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
- path: prompts/summarize.yml
|
||||
destination: [collections, support, prompts]
|
||||
placement: sequence-item
|
||||
required: true
|
||||
collections:
|
||||
support: {}
|
||||
2
examples/discovery/collections/support-repo/other.yml
Normal file
2
examples/discovery/collections/support-repo/other.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
repository_metadata:
|
||||
purpose: This file is not a PromptKit framework input.
|
||||
@@ -0,0 +1,4 @@
|
||||
id: base
|
||||
endpoint: https://example.invalid/v1
|
||||
model: support-example
|
||||
temperature: 0.2
|
||||
@@ -0,0 +1,3 @@
|
||||
id: fast
|
||||
extends: {id: base}
|
||||
max_tokens: 256
|
||||
@@ -0,0 +1,3 @@
|
||||
Answer the support request using this transcript:
|
||||
|
||||
{{input "transcript"}}
|
||||
@@ -0,0 +1,12 @@
|
||||
id: answer
|
||||
default_profile: support/fast
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
messages:
|
||||
- role: user
|
||||
content_file: ./answer.md
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
@@ -0,0 +1,8 @@
|
||||
id: summarize
|
||||
default_profile: support/fast
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize the support request concisely.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
4
examples/discovery/framework.yml
Normal file
4
examples/discovery/framework.yml
Normal file
@@ -0,0 +1,4 @@
|
||||
config:
|
||||
imports:
|
||||
- glob: collections/*/collection.yml
|
||||
- glob: collections/*/collection.yaml
|
||||
8
examples/framework/backends.yml
Normal file
8
examples/framework/backends.yml
Normal file
@@ -0,0 +1,8 @@
|
||||
backends:
|
||||
- id: local-gpu
|
||||
endpoint: http://localhost:11434/v1
|
||||
api_key_env: LOCAL_GPU_API_KEY
|
||||
extra_params:
|
||||
provider_option: enabled
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 0
|
||||
13
examples/framework/inheritance.yml
Normal file
13
examples/framework/inheritance.yml
Normal file
@@ -0,0 +1,13 @@
|
||||
backends:
|
||||
- id: local-base
|
||||
endpoint: http://localhost:11434/v1
|
||||
api_key_env: LOCAL_GPU_API_KEY
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 4
|
||||
extra_params:
|
||||
provider_option: true
|
||||
- id: local-gpu
|
||||
extends: {id: local-base}
|
||||
queue_capacity: 0
|
||||
extra_params:
|
||||
provider_option: false
|
||||
@@ -14,5 +14,6 @@
|
||||
"vars": {
|
||||
"session_date": "2026-05-04"
|
||||
},
|
||||
"include_raw_output": false
|
||||
"include_raw_output": false,
|
||||
"include_execution_report": false
|
||||
}
|
||||
|
||||
2
examples/overlays/collection.yml
Normal file
2
examples/overlays/collection.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
promptkit_sources:
|
||||
- file: ./examples/profiles/local-gpu.yml
|
||||
@@ -1,7 +1,8 @@
|
||||
id: local-fast
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4o-mini
|
||||
temperature: 0.2
|
||||
max_tokens: 500
|
||||
top_p: 1.0
|
||||
timeout_seconds: 90
|
||||
profiles:
|
||||
- id: local-fast
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4o-mini
|
||||
temperature: 0.2
|
||||
max_tokens: 500
|
||||
top_p: 1.0
|
||||
timeout_seconds: 90
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
id: local-gpu
|
||||
backend: local-gpu
|
||||
model: local-model
|
||||
reasoning_effort: low
|
||||
profiles:
|
||||
- id: local-gpu
|
||||
backend: local-gpu
|
||||
model: local-model
|
||||
reasoning_effort: low
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
id: local-quality
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4.1
|
||||
temperature: 0.0
|
||||
max_tokens: 1200
|
||||
top_p: 1.0
|
||||
timeout_seconds: 180
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
profiles:
|
||||
- id: local-quality
|
||||
endpoint: http://localhost:8000/v1
|
||||
model: gpt-4.1
|
||||
temperature: 0.0
|
||||
max_tokens: 1200
|
||||
top_p: 1.0
|
||||
timeout_seconds: 180
|
||||
api_key_env: SCRIPTORIUM_API_KEY
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
id: dnd.session_recap
|
||||
version: "1.0.0"
|
||||
default_profile: local-quality
|
||||
description: Example D&D session recap prompt definition for demonstration only.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Session source content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./dnd.session_recap.system.md
|
||||
- role: user
|
||||
content_file: ./dnd.session_recap.user.md
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
prompts:
|
||||
- id: dnd.session_recap
|
||||
default_profile: local-quality
|
||||
description: Example D&D session recap prompt definition for demonstration only.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Session source content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./dnd.session_recap.system.md
|
||||
- role: user
|
||||
content_file: ./dnd.session_recap.user.md
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
|
||||
@@ -1,22 +1,22 @@
|
||||
id: generic.markdown_summary
|
||||
version: "1.0.0"
|
||||
default_profile: local-fast
|
||||
description: Generic markdown summary from transcript and glossary.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Source transcript content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./generic.markdown_summary.system.md
|
||||
- role: user
|
||||
content_file: ./generic.markdown_summary.user.md
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
prompts:
|
||||
- id: generic.markdown_summary
|
||||
default_profile: local-fast
|
||||
description: Generic markdown summary from transcript and glossary.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Source transcript content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./generic.markdown_summary.system.md
|
||||
- role: user
|
||||
content_file: ./generic.markdown_summary.user.md
|
||||
output:
|
||||
format: markdown
|
||||
validation_mode: basic
|
||||
repair_attempts: 0
|
||||
|
||||
@@ -1,23 +1,23 @@
|
||||
id: generic.structured_events
|
||||
version: "1.0.0"
|
||||
default_profile: local-quality
|
||||
description: Produce structured event JSON from a transcript.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Source transcript content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./generic.structured_events.system.md
|
||||
- role: user
|
||||
content_file: ./generic.structured_events.user.md
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: structured_events.schema.json
|
||||
repair_attempts: 0
|
||||
prompts:
|
||||
- id: generic.structured_events
|
||||
default_profile: local-quality
|
||||
description: Produce structured event JSON from a transcript.
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: text/markdown
|
||||
description: Source transcript content
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/yaml
|
||||
description: Optional glossary context
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./generic.structured_events.system.md
|
||||
- role: user
|
||||
content_file: ./generic.structured_events.user.md
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: structured_events.schema.json
|
||||
repair_attempts: 0
|
||||
|
||||
@@ -7,7 +7,6 @@ cd "$repo_root"
|
||||
go run ./cmd/scriptorium render \
|
||||
--config ./examples/config.full.yml \
|
||||
--prompt generic.markdown_summary \
|
||||
--prompt-version 1.0.0 \
|
||||
--profile local-gpu \
|
||||
--session-id example-session \
|
||||
--reasoning-effort= \
|
||||
|
||||
15
go.mod
15
go.mod
@@ -1,17 +1,14 @@
|
||||
module gitea.maximumdirect.net/eric/scriptorium
|
||||
|
||||
go 1.25.5
|
||||
go 1.27.0
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2
|
||||
go.maximumdirect.net/configkit v1.2.1
|
||||
go.maximumdirect.net/promptkit v0.23.3
|
||||
)
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260926.0 // indirect
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260926.0 // indirect
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 // indirect
|
||||
golang.org/x/text v0.14.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 // indirect
|
||||
go.yaml.in/yaml/v4 v4.0.0 // indirect
|
||||
golang.org/x/text v0.44.0 // indirect
|
||||
)
|
||||
|
||||
28
go.sum
28
go.sum
@@ -1,20 +1,12 @@
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2 h1:ZMxmv8pOnLtAx2hFTf8Z+pHKUbR89xErkBhwjr4yK24=
|
||||
gitea.maximumdirect.net/eric/ConfigKit v0.4.2/go.mod h1:tYWpsrbLyu05r8+1MapWp5fN6ZrX9I8hU2xEPedYmYk=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2 h1:2RqI+KtflvSh7vLQaTAQCmMaFQscEA7EPwPazyvXRoY=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.11.2/go.mod h1:onm/FBPUKQ8/NSPwTEk26Ut+dVX2iXcq8vEseB+g/0E=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260926.0 h1:wdab+R+lgMZxzYwhbAuCrW7zdSQ1cW9ovBR6UzLmZR4=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v2 v2.20260926.0/go.mod h1:a3VBbh9oaGXD6ruxhtwHj+AWW6DY/SSyby1QtMmYSUo=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260926.0 h1:baCCIB6Ee5t0S4Zc++NTBIPcJEv5UPS6DlSrpZfykwg=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v2 v2.20260926.0/go.mod h1:mtgD1E0SGm0e/I8Ffhz+AYX7GN7RKUWJXehNFIKHs4Y=
|
||||
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
|
||||
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3 h1:1EYB5IzjZawrrnELUi78f9fPu57HuXjmddZPjrls/28=
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.3/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
|
||||
go.maximumdirect.net/configkit v1.2.1 h1:f+/6WjNl6CULbMr6ZSVwi4aty9rmuDfJlvpeU7I+HZM=
|
||||
go.maximumdirect.net/configkit v1.2.1/go.mod h1:ewu6blSokUDCMovy8JBXAdXRikGmlHXAQpFghnPWRfs=
|
||||
go.maximumdirect.net/promptkit v0.23.3 h1:bKbrkluwfgXUsm0e/mxz/TpV9b60yUtJLVUROLsM3VU=
|
||||
go.maximumdirect.net/promptkit v0.23.3/go.mod h1:vPFg76VXcffP0JBFMS2WADCHwV5FYvO2O4bvIoiDlwc=
|
||||
go.yaml.in/yaml/v4 v4.0.0 h1:/LG1pNO84Z4l603j/xMn69CcS5OYhMQ/f7l2LoQUGuw=
|
||||
go.yaml.in/yaml/v4 v4.0.0/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/text v0.44.0 h1:YItdp/mQNGnsmrgpcdA5yQ3bnRxCfo/Tj/iByGW9GGo=
|
||||
golang.org/x/text v0.44.0/go.mod h1:S4FR1waQ9LbjQTls97SyRjDaUUVkaM+cXwwuG/G+MwY=
|
||||
|
||||
@@ -6,8 +6,8 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
// applicationOverrides keeps explicit application flags separate from invocation state.
|
||||
@@ -16,10 +16,8 @@ func applicationOverrides(fs *flag.FlagSet) []configkit.Override {
|
||||
fs.Visit(func(selected *flag.Flag) {
|
||||
var path configkit.Path
|
||||
switch selected.Name {
|
||||
case "prompt-dir":
|
||||
path = configkit.Keys("prompt_dir")
|
||||
case "profile-dir":
|
||||
path = configkit.Keys("profile_dir")
|
||||
case "promptkit-source":
|
||||
path = configkit.Keys("promptkit_sources")
|
||||
case "schema-dir":
|
||||
path = configkit.Keys("schema_dir")
|
||||
case "addr":
|
||||
@@ -55,6 +53,7 @@ func loadApplicationSettings(configPath, overlay string, explicit bool, override
|
||||
func writeApplicationError(output io.Writer, label string, err error) {
|
||||
var loadError *configkit.LoadError
|
||||
if errors.As(err, &loadError) {
|
||||
_, _ = fmt.Fprintf(output, "%s error:\n", label)
|
||||
if writeErr := configkit.WriteDiagnostics(output, loadError.Diagnostics()); writeErr != nil {
|
||||
_, _ = fmt.Fprintf(output, "%s diagnostic write error: %v\n", label, writeErr)
|
||||
}
|
||||
|
||||
@@ -6,8 +6,8 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
type configurationSource struct {
|
||||
@@ -93,8 +93,6 @@ func configCommand(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
|
||||
func registerConfigurationOverrides(fs *flag.FlagSet) {
|
||||
fs.String("prompt-dir", "", "prompt-definition directory override")
|
||||
fs.String("profile-dir", "", "profile directory override")
|
||||
fs.String("schema-dir", "", "schema directory override")
|
||||
fs.String("addr", "", "HTTP listen address override")
|
||||
fs.String("artifact-root", "", "HTTP artifact root override")
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -31,13 +32,43 @@ func TestConfigCommandsInspectApplicationWithoutEngine(t *testing.T) {
|
||||
}
|
||||
case "sources":
|
||||
var sources []configurationSource
|
||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "base" {
|
||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "main" {
|
||||
t.Fatalf("unexpected source records: %q (%v)", stdout, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectionExampleLoadsAndRenders(t *testing.T) {
|
||||
t.Chdir(filepath.Join("..", "..", ".."))
|
||||
want := []appconfig.SourceSelection{
|
||||
{File: "examples/framework/backends.yml"},
|
||||
{File: "examples/profiles/local-fast.yaml"},
|
||||
{File: "examples/prompts/generic.markdown_summary.yaml"},
|
||||
}
|
||||
for _, overlay := range []string{"", "collection"} {
|
||||
result, err := appconfig.Load(appconfig.LoadOptions{Path: "examples/config.collections.yml", Explicit: true, Overlay: overlay})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
expected := append([]appconfig.SourceSelection{}, want...)
|
||||
if overlay != "" {
|
||||
expected = append(expected, appconfig.SourceSelection{File: "examples/profiles/local-gpu.yml"})
|
||||
}
|
||||
if !reflect.DeepEqual(result.Config.PromptKitSources, expected) {
|
||||
t.Fatalf("overlay %q selectors = %+v, want %+v", overlay, result.Config.PromptKitSources, expected)
|
||||
}
|
||||
args := []string{"--config", "examples/config.collections.yml", "--prompt", "generic.markdown_summary", "--input", "transcript=examples/fixtures/transcript.md", "--input", "glossary=examples/fixtures/glossary.yml"}
|
||||
if overlay != "" {
|
||||
args = append(args, "--config-overlay", overlay)
|
||||
}
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, args)
|
||||
if code != ExitOK || !strings.Contains(stdout, "generic.markdown_summary") || !strings.Contains(stdout, "local-fast") {
|
||||
t.Fatalf("collection render failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
|
||||
@@ -45,17 +76,17 @@ func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
|
||||
}
|
||||
base := filepath.Join(dir, "config.yml")
|
||||
for path, body := range map[string]string{
|
||||
base: "configkit:\n imports:\n - path: fragment.yml\nprompt_dir: ' ./before/../prompts '\nprofile_dir: ./profiles\n",
|
||||
base: "config:\n imports:\n - path: fragment.yml\n required: true\npromptkit_sources:\n - directory: ' ./before/../prompts '\nschema_dir: ./schemas\n",
|
||||
filepath.Join(dir, "fragment.yml"): "server:\n addr: :8081\n",
|
||||
filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n",
|
||||
filepath.Join(dir, "overlays", "local.yml"): "promptkit_sources:\n - directory: ./overlay-prompts\nserver:\n addr: :9090\n",
|
||||
} {
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
args := []string{"--config", base, "--config-overlay", "local", "--profile-dir=", "--addr", ":7070"}
|
||||
args := []string{"--config", base, "--config-overlay", "local", "--schema-dir=", "--addr", ":7070"}
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"show"}, args...))
|
||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "prompt_dir: prompts") || !strings.Contains(stdout, "profile_dir: ''") || !strings.Contains(stdout, "addr: :7070") {
|
||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "directory: overlay-prompts") || !strings.Contains(stdout, "schema_dir: ''") || !strings.Contains(stdout, "addr: :7070") || strings.Contains(stdout, "before/../prompts") {
|
||||
t.Fatalf("resolved effective output changed: code=%d out=%q err=%q", code, stdout, stderr)
|
||||
}
|
||||
code, stdout, stderr = runCLICommand(t, configCommand, append([]string{"sources"}, args...))
|
||||
@@ -66,42 +97,52 @@ func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) {
|
||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 5 {
|
||||
t.Fatalf("unexpected composed source records: %+v (%v)", sources, err)
|
||||
}
|
||||
for i, layer := range []string{"base", "base", "overlay", "cli", "cli"} {
|
||||
for i, layer := range []string{"main", "main", "overlay", "cli", "cli"} {
|
||||
if string(sources[i].Layer) != layer {
|
||||
t.Fatalf("source %d layer = %q, want %q", i, sources[i].Layer, layer)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigInspectionKeepsReferencesAndOrdinaryParameters(t *testing.T) {
|
||||
const name = "SCRIPTORIUM_CONFIG_TEST_API_KEY"
|
||||
const secret = "synthetic-secret-value-should-not-appear"
|
||||
path := writeAppConfigFile(t, `backends:
|
||||
synthetic:
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_CONFIG_TEST_API_KEY
|
||||
extra_params:
|
||||
visible_parameter: synthetic-ordinary-value
|
||||
`)
|
||||
t.Setenv(name, secret)
|
||||
for _, present := range []bool{true, false} {
|
||||
if !present {
|
||||
if err := os.Unsetenv(name); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{"show", "--config", path})
|
||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, name) || !strings.Contains(stdout, "synthetic-ordinary-value") || strings.Contains(stdout, secret) {
|
||||
t.Fatalf("credential boundary changed with present=%t: code=%d out=%q err=%q", present, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
result, err := appconfig.Load(appconfig.LoadOptions{Path: path, Explicit: true})
|
||||
if err != nil {
|
||||
func TestApplicationInspectionDoesNotAcquireFrameworkSources(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
frameworkPath := filepath.Join(dir, "invalid-framework.yml")
|
||||
if err := os.WriteFile(frameworkPath, []byte("not: [valid\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
settings := appconfig.Settings(result.Config)
|
||||
if len(settings.Backends) != 1 || settings.Backends[0].ExtraParams["visible_parameter"] != "synthetic-ordinary-value" {
|
||||
t.Fatalf("runtime backend parameters changed: %+v", settings.Backends)
|
||||
for _, frameworkSource := range []struct {
|
||||
name string
|
||||
path string
|
||||
}{
|
||||
{name: "invalid definition", path: frameworkPath},
|
||||
{name: "missing path", path: filepath.Join(dir, "missing-framework.yml")},
|
||||
} {
|
||||
t.Run(frameworkSource.name, func(t *testing.T) {
|
||||
path := writeAppConfigFile(t, "promptkit_sources:\n - file: "+frameworkSource.path+"\n")
|
||||
for _, args := range [][]string{
|
||||
{"validate"},
|
||||
{"show"},
|
||||
{"sources"},
|
||||
{"explain", "--path", "/promptkit_sources/0/file"},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append(args, "--config", path))
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("application inspection acquired configured framework source: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigCommandsRejectRetiredFrameworkFields(t *testing.T) {
|
||||
for _, field := range []string{"prompt_dir", "profile_dir", "backends"} {
|
||||
t.Run(field, func(t *testing.T) {
|
||||
path := writeAppConfigFile(t, field+": {}\n")
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", path})
|
||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
||||
t.Fatalf("retired application field %q was accepted: code=%d out=%q err=%q", field, code, stdout, stderr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,51 +168,36 @@ func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigExplainUsesResolvedHistoryAndKeepsParametersVisible(t *testing.T) {
|
||||
func TestConfigExplainUsesResolvedHistory(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := filepath.Join(dir, "config.yml")
|
||||
for path, body := range map[string]string{
|
||||
base: `prompt_dir: " ./prompts/../prompts "
|
||||
base: `promptkit_sources:
|
||||
- directory: " ./prompts/../prompts "
|
||||
server:
|
||||
addr: :8080
|
||||
backends:
|
||||
"team/blue~1":
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_KEY
|
||||
extra_params:
|
||||
"0": base-parameter
|
||||
`,
|
||||
filepath.Join(dir, "overlays", "local.yml"): `server:
|
||||
addr: :9090
|
||||
backends:
|
||||
"team/blue~1":
|
||||
extra_params:
|
||||
"0": overlay-parameter
|
||||
`,
|
||||
filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n",
|
||||
} {
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
const keyValue = "synthetic-key-bytes-must-not-appear"
|
||||
t.Setenv("SCRIPTORIUM_EXPLAIN_KEY", keyValue)
|
||||
args := []string{"--config", base, "--config-overlay", "local", "--addr", ":9090"}
|
||||
for _, tc := range []struct {
|
||||
pointer string
|
||||
want []string
|
||||
}{
|
||||
{pointer: "/server/addr", want: []string{":8080", ":9090", "--addr"}},
|
||||
{pointer: "/prompt_dir", want: []string{"normalized application path", "prompts"}},
|
||||
{pointer: "/backends/team~1blue~01/extra_params/0", want: []string{"base-parameter", "overlay-parameter"}},
|
||||
{pointer: "/backends/team~1blue~01/api_key_env", want: []string{"SCRIPTORIUM_EXPLAIN_KEY"}},
|
||||
{pointer: "", want: []string{"base-parameter", "overlay-parameter", "SCRIPTORIUM_EXPLAIN_KEY"}},
|
||||
{pointer: "/promptkit_sources/0/directory", want: []string{"normalized application path", "prompts"}},
|
||||
{pointer: "", want: []string{"promptkit_sources", "server", "addr"}},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", tc.pointer}, args...))
|
||||
if code != ExitOK || stderr != "" || strings.Contains(stdout, keyValue) {
|
||||
t.Fatalf("explain %q failed or disclosed key: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("explain %q failed: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
|
||||
}
|
||||
for _, want := range tc.want {
|
||||
if !strings.Contains(stdout, want) {
|
||||
@@ -179,13 +205,6 @@ backends:
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := os.Unsetenv("SCRIPTORIUM_EXPLAIN_KEY"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", "/backends/team~1blue~01/api_key_env"}, args...))
|
||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "SCRIPTORIUM_EXPLAIN_KEY") || strings.Contains(stdout, keyValue) {
|
||||
t.Fatalf("absent credential changed reference explanation: code=%d out=%q err=%q", code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigExplainRejectsUnknownOrMalformedPaths(t *testing.T) {
|
||||
|
||||
82
internal/adapter/cli/discovery_test.go
Normal file
82
internal/adapter/cli/discovery_test.go
Normal file
@@ -0,0 +1,82 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCollectionCheckoutDiscoveryAcrossInvocations(t *testing.T) {
|
||||
fixture, err := filepath.Abs(filepath.Join("..", "..", "..", "examples", "discovery"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
first string
|
||||
second string
|
||||
}{
|
||||
{name: "yml then yaml", first: "support-repo", second: "review-repo"},
|
||||
{name: "yaml then yml", first: "review-repo", second: "support-repo"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
framework, err := os.ReadFile(filepath.Join(fixture, "framework.yml"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
frameworkPath := filepath.Join(root, "framework.yml")
|
||||
if err := os.WriteFile(frameworkPath, framework, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
collections := filepath.Join(root, "collections")
|
||||
if err := os.MkdirAll(collections, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
install := func(name string) {
|
||||
t.Helper()
|
||||
if err := os.CopyFS(filepath.Join(collections, name), os.DirFS(filepath.Join(fixture, "collections", name))); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
config := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkPath))
|
||||
input := filepath.Join(root, "transcript.md")
|
||||
if err := os.WriteFile(input, []byte("synthetic discovery transcript"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := func(name string) []string {
|
||||
if name == "support-repo" {
|
||||
return []string{"--config", config, "--prompt", "support/answer", "--profile", "support/fast", "--input", "transcript=" + input}
|
||||
}
|
||||
return []string{"--config", config, "--prompt", "review/answer", "--profile", "review/base"}
|
||||
}
|
||||
check := func(name string) {
|
||||
t.Helper()
|
||||
code, out, stderr := runCLICommand(t, renderCommand, args(name))
|
||||
if code != ExitOK {
|
||||
t.Fatalf("render %s: code=%d stderr=%s", name, code, stderr)
|
||||
}
|
||||
if name == "support-repo" {
|
||||
for _, want := range []string{"prompt: support/answer", "selected_profile_id: support/fast", "model: support-example", "max_tokens: 256", "Answer the support request", "synthetic discovery transcript"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("missing %q: %s", want, out)
|
||||
}
|
||||
}
|
||||
} else if !strings.Contains(out, "prompt: review/answer") || !strings.Contains(out, "model: review-example") || !strings.Contains(out, "Review this material") {
|
||||
t.Fatalf("unexpected review render: %s", out)
|
||||
}
|
||||
}
|
||||
install(tc.first)
|
||||
check(tc.first)
|
||||
code, _, _ := runCLICommand(t, renderCommand, args(tc.second))
|
||||
if code != ExitRuntimeError {
|
||||
t.Fatal("uninstalled collection unexpectedly selected")
|
||||
}
|
||||
install(tc.second)
|
||||
check(tc.first)
|
||||
check(tc.second)
|
||||
})
|
||||
}
|
||||
}
|
||||
162
internal/adapter/cli/framework_config.go
Normal file
162
internal/adapter/cli/framework_config.go
Normal file
@@ -0,0 +1,162 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
type frameworkConfigurationConfig struct {
|
||||
mode, configPath, configOverlay, schemaDir string
|
||||
path string
|
||||
configExplicit bool
|
||||
pathExplicit bool
|
||||
overrides []configkit.Override
|
||||
diagnostics []configkit.Diagnostic
|
||||
}
|
||||
|
||||
type frameworkConfigurationSource struct {
|
||||
ID string `json:"id"`
|
||||
Label string `json:"label"`
|
||||
Layer configkit.Layer `json:"layer"`
|
||||
Kind configkit.SourceKind `json:"kind"`
|
||||
UnderlayIndex int `json:"underlay_index"`
|
||||
Path string `json:"path,omitempty"`
|
||||
ImportedFrom *frameworkConfigurationLocation `json:"imported_from,omitempty"`
|
||||
}
|
||||
|
||||
type frameworkConfigurationLocation struct {
|
||||
SourceID string `json:"source_id"`
|
||||
Line int `json:"line"`
|
||||
Column int `json:"column"`
|
||||
}
|
||||
|
||||
func inspectFrameworkConfigurationCommand(args []string, stdout, stderr io.Writer) int {
|
||||
cfg, err := parseFrameworkConfigurationArgs(args)
|
||||
if err != nil {
|
||||
writeApplicationError(stderr, "inspect configuration parse", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
var explanationPath configkit.Path
|
||||
if cfg.mode == "explain" {
|
||||
explanationPath, err = configkit.ParsePath(cfg.path)
|
||||
if err != nil {
|
||||
writeApplicationError(stderr, "inspect configuration path", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
}
|
||||
settings, err := resolveAppSettingsForFrameworkConfiguration(cfg)
|
||||
if err != nil {
|
||||
writeApplicationError(stderr, "inspect configuration", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
engine, err := newEngine(settings)
|
||||
if err != nil {
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
view, err := engine.InspectConfiguration()
|
||||
if err != nil {
|
||||
writeApplicationError(stderr, "inspect configuration", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := configkit.WriteDiagnostics(stderr, view.Diagnostics()); err != nil {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if cfg.mode == "validate" {
|
||||
return ExitOK
|
||||
}
|
||||
|
||||
if cfg.mode == "explain" {
|
||||
if err := view.WriteExplanation(stdout, explanationPath); err != nil {
|
||||
fmt.Fprintf(stderr, "inspect configuration explain %q: %v\n", cfg.path, err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
return ExitOK
|
||||
}
|
||||
|
||||
switch cfg.mode {
|
||||
case "show":
|
||||
err = view.WriteEffective(stdout)
|
||||
case "sources":
|
||||
sources := frameworkConfigurationSources(view.Sources())
|
||||
err = json.NewEncoder(stdout).Encode(sources)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "inspect configuration output error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
return ExitOK
|
||||
}
|
||||
|
||||
func parseFrameworkConfigurationArgs(args []string) (*frameworkConfigurationConfig, error) {
|
||||
if len(args) == 0 {
|
||||
return nil, errors.New("configuration operation is required")
|
||||
}
|
||||
cfg := &frameworkConfigurationConfig{mode: args[0]}
|
||||
switch cfg.mode {
|
||||
case "validate", "show", "sources", "explain":
|
||||
default:
|
||||
return nil, fmt.Errorf("unknown configuration operation %q", cfg.mode)
|
||||
}
|
||||
|
||||
fs := flag.NewFlagSet("inspect configuration "+cfg.mode, flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
||||
var promptkitSources promptKitSourceFlagValue
|
||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||
fs.StringVar(&cfg.path, "path", "", "JSON Pointer to explain (required for explain)")
|
||||
if err := fs.Parse(args[1:]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return nil, fmt.Errorf("unexpected positional args: %v", fs.Args())
|
||||
}
|
||||
cfg.configExplicit = flagWasSet(fs, "config")
|
||||
cfg.pathExplicit = flagWasSet(fs, "path")
|
||||
if cfg.mode == "explain" && !cfg.pathExplicit {
|
||||
return nil, errors.New("--path is required for explain")
|
||||
}
|
||||
cfg.overrides = applicationOverrides(fs)
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func resolveAppSettingsForFrameworkConfiguration(cfg *frameworkConfigurationConfig) (engineSettings, error) {
|
||||
settings, diagnostics, err := loadApplicationSettings(cfg.configPath, cfg.configOverlay, cfg.configExplicit, cfg.overrides)
|
||||
if err != nil {
|
||||
return engineSettings{}, err
|
||||
}
|
||||
cfg.diagnostics = diagnostics
|
||||
return engineSettings{
|
||||
promptkitSources: settings.PromptKitSources,
|
||||
schemaDir: settings.SchemaDir,
|
||||
transientRetries: settings.TransientRetries,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func frameworkConfigurationSources(sources []configkit.SourceInfo) []frameworkConfigurationSource {
|
||||
result := make([]frameworkConfigurationSource, 0, len(sources))
|
||||
for _, source := range sources {
|
||||
item := frameworkConfigurationSource{
|
||||
ID: source.ID, Label: source.Label, Layer: source.Layer,
|
||||
Kind: source.Kind, UnderlayIndex: source.UnderlayIndex, Path: source.Path,
|
||||
}
|
||||
if source.ImportedFrom != nil {
|
||||
item.ImportedFrom = &frameworkConfigurationLocation{
|
||||
SourceID: source.ImportedFrom.SourceID,
|
||||
Line: source.ImportedFrom.Line,
|
||||
Column: source.ImportedFrom.Column,
|
||||
}
|
||||
}
|
||||
result = append(result, item)
|
||||
}
|
||||
return result
|
||||
}
|
||||
391
internal/adapter/cli/framework_config_test.go
Normal file
391
internal/adapter/cli/framework_config_test.go
Normal file
@@ -0,0 +1,391 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
configkit "go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
func TestFrameworkConfigurationCommandsNeedNoPromptOrCredential(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, "{}\n")
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{
|
||||
"configuration", "validate", "--config", configPath,
|
||||
})
|
||||
if code != ExitOK || stdout != "" || stderr != "" {
|
||||
t.Fatalf("source-free validation failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||
}
|
||||
|
||||
t.Setenv("SCRIPTORIUM_FRAMEWORK_KEY", "")
|
||||
root := t.TempDir()
|
||||
frameworkFile := filepath.Join(root, "framework.yml")
|
||||
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
|
||||
- id: local
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_FRAMEWORK_KEY
|
||||
profiles:
|
||||
- id: protected
|
||||
backend: local
|
||||
model: offline-model
|
||||
api_key_required: true
|
||||
`)
|
||||
configPath = writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
||||
for _, mode := range []string{"validate", "show", "sources"} {
|
||||
code, stdout, stderr = runCLICommand(t, inspectCommand, []string{
|
||||
"configuration", mode, "--config", configPath,
|
||||
})
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("%s required a prompt or credential: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr)
|
||||
}
|
||||
if mode == "validate" && stdout != "" {
|
||||
t.Fatalf("validation wrote success output: %q", stdout)
|
||||
}
|
||||
if mode == "show" && (!strings.Contains(stdout, "profiles:") || !strings.Contains(stdout, "offline-model")) {
|
||||
t.Fatalf("effective framework YAML is missing: %q", stdout)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkConfigurationSourcesProjectsOrderedNativeRecords(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
importDir := filepath.Join(root, "catalog")
|
||||
if err := os.MkdirAll(importDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mainPath := writeFrameworkSourceFile(t, root, "main.yml", `config:
|
||||
imports:
|
||||
- path: catalog/fragment.yml
|
||||
required: true
|
||||
backends:
|
||||
- id: local
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
`)
|
||||
fragmentPath := writeFrameworkSourceFile(t, importDir, "fragment.yml", `profiles:
|
||||
- id: imported
|
||||
model: imported-model
|
||||
`)
|
||||
directory := t.TempDir()
|
||||
directoryPath := writeFrameworkSourceFile(t, directory, "directory.yml", `profiles:
|
||||
- id: directory
|
||||
model: directory-model
|
||||
`)
|
||||
configuredMissing := filepath.Join(root, "configured-but-replaced.yml")
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", configuredMissing))
|
||||
args := []string{
|
||||
"configuration", "sources", "--config", configPath,
|
||||
"--promptkit-source", mainPath,
|
||||
"--promptkit-source", directory,
|
||||
}
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
||||
if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") {
|
||||
t.Fatalf("framework sources command failed: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||
}
|
||||
var sources []frameworkConfigurationSource
|
||||
if err := json.Unmarshal([]byte(stdout), &sources); err != nil {
|
||||
t.Fatalf("decode framework source DTOs: %v\n%s", err, stdout)
|
||||
}
|
||||
var rawSources []map[string]any
|
||||
if err := json.Unmarshal([]byte(stdout), &rawSources); err != nil {
|
||||
t.Fatalf("decode framework source fields: %v", err)
|
||||
}
|
||||
mainIndex := findFrameworkSource(t, sources, mainPath)
|
||||
fragmentIndex := findFrameworkSource(t, sources, fragmentPath)
|
||||
directoryIndex := findFrameworkSource(t, sources, directoryPath)
|
||||
main := sources[mainIndex]
|
||||
fragment := sources[fragmentIndex]
|
||||
directorySource := sources[directoryIndex]
|
||||
if !(mainIndex < fragmentIndex && fragmentIndex < directoryIndex) {
|
||||
t.Fatalf("captured source order changed: main=%d fragment=%d directory=%d", mainIndex, fragmentIndex, directoryIndex)
|
||||
}
|
||||
if main.ID == "" || main.Label == "" || main.Layer != configkit.LayerUnderlay || main.Kind != configkit.SourceKindFile || main.UnderlayIndex < 1 {
|
||||
t.Fatalf("explicit file source fields are incomplete: %+v", main)
|
||||
}
|
||||
for _, key := range []string{"id", "label", "layer", "kind", "underlay_index", "path"} {
|
||||
if _, exists := rawSources[mainIndex][key]; !exists {
|
||||
t.Fatalf("source DTO omitted %q: %#v", key, rawSources[mainIndex])
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"ID", "Label", "Layer", "Kind", "UnderlayIndex", "filesystem"} {
|
||||
if _, exists := rawSources[mainIndex][key]; exists {
|
||||
t.Fatalf("source DTO exposed implementation field %q: %#v", key, rawSources[mainIndex])
|
||||
}
|
||||
}
|
||||
if fragment.Layer != configkit.LayerUnderlay || fragment.Kind != configkit.SourceKindFile || fragment.ImportedFrom == nil ||
|
||||
fragment.ImportedFrom.SourceID != main.ID || fragment.ImportedFrom.Line < 1 || fragment.ImportedFrom.Column < 1 {
|
||||
t.Fatalf("import source fields are incomplete: %+v", fragment)
|
||||
}
|
||||
if importedFrom, ok := rawSources[fragmentIndex]["imported_from"].(map[string]any); !ok || importedFrom["source_id"] != main.ID || importedFrom["line"] == nil || importedFrom["column"] == nil {
|
||||
t.Fatalf("import location DTO fields are incomplete: %#v", rawSources[fragmentIndex])
|
||||
}
|
||||
if directorySource.Layer != configkit.LayerUnderlay || directorySource.Kind != configkit.SourceKindFile || directorySource.UnderlayIndex <= main.UnderlayIndex {
|
||||
t.Fatalf("directory-selected source fields/order are incomplete: %+v (main %+v)", directorySource, main)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkConfigurationShowFailsBeforeOutputForInvalidFinalDefinition(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
frameworkFile := writeFrameworkSourceFile(t, root, "invalid.yml", `prompts:
|
||||
- id: unused
|
||||
messages:
|
||||
- role: user
|
||||
content: "{{"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, []string{"validate", "--config", configPath})
|
||||
if code != ExitOK || stdout != "" || stderr != "" {
|
||||
t.Fatalf("application validation loaded framework definitions: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||
}
|
||||
for _, args := range [][]string{
|
||||
{"configuration", "validate", "--config", configPath},
|
||||
{"configuration", "show", "--config", configPath},
|
||||
{"configuration", "sources", "--config", configPath},
|
||||
{"configuration", "explain", "--config", configPath, "--path", ""},
|
||||
} {
|
||||
code, stdout, stderr = runCLICommand(t, inspectCommand, args)
|
||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
||||
t.Fatalf("invalid final framework definition was reported as success: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkConfigurationCommandsRejectExtraArgumentsAndReportWriterErrors(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, "{}\n")
|
||||
for _, args := range [][]string{
|
||||
{"configuration"},
|
||||
{"configuration", "explain"},
|
||||
{"configuration", "show", "unexpected"},
|
||||
{"configuration", "sources", "--format", "json"},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
||||
t.Fatalf("invalid inspection arguments were accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
writerError := errors.New("synthetic framework output failure")
|
||||
var stderr bytes.Buffer
|
||||
code := inspectCommand([]string{"configuration", "show", "--config", configPath}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
|
||||
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
|
||||
t.Fatalf("effective-output writer failure was not reported: code=%d stderr=%q", code, stderr.String())
|
||||
}
|
||||
stderr.Reset()
|
||||
code = inspectCommand([]string{"configuration", "explain", "--config", configPath, "--path", ""}, frameworkConfigurationFailingWriter{err: writerError}, &stderr)
|
||||
if code != ExitRuntimeError || !strings.Contains(stderr.String(), writerError.Error()) {
|
||||
t.Fatalf("explanation writer failure was not reported: code=%d stderr=%q", code, stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkConfigurationExplainUsesLiteralJSONPointersAndNativeHistory(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
lower := writeFrameworkSourceFile(t, root, "lower.yml", `collections:
|
||||
support:
|
||||
backends:
|
||||
- id: local
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
||||
extra_params:
|
||||
token: backend-token-displaced-canary
|
||||
password: backend-password-canary
|
||||
visible_option: backend-visible-value
|
||||
provider.option: dotted-visible-value
|
||||
profiles:
|
||||
- id: summary
|
||||
backend: support/local
|
||||
model: lower-model
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
||||
extra_params:
|
||||
token: profile-token-displaced-canary
|
||||
password: profile-password-displaced-canary
|
||||
visible_option: profile-visible-value
|
||||
provider.option: dotted-visible-value
|
||||
prompts:
|
||||
- id: explainable
|
||||
default_profile: support/summary
|
||||
messages:
|
||||
- role: user
|
||||
content: hello
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
upperRoot := t.TempDir()
|
||||
upper := writeFrameworkSourceFile(t, upperRoot, "upper.yml", `collections:
|
||||
support:
|
||||
backends:
|
||||
- id: local
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
||||
extra_params:
|
||||
token: backend-token-effective-canary
|
||||
password: backend-password-effective-canary
|
||||
visible_option: backend-visible-value
|
||||
provider.option: dotted-visible-value
|
||||
profiles:
|
||||
- id: summary
|
||||
backend: support/local
|
||||
model: effective-model
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_REFERENCE
|
||||
extra_params:
|
||||
token: profile-token-effective-canary
|
||||
password: profile-password-effective-canary
|
||||
visible_option: profile-visible-value
|
||||
provider.option: dotted-visible-value
|
||||
`)
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n - file: %q\n", lower, upper))
|
||||
const secret = "environment-value-must-never-be-emitted"
|
||||
t.Setenv("SCRIPTORIUM_EXPLAIN_REFERENCE", secret)
|
||||
|
||||
for _, tc := range []struct {
|
||||
path string
|
||||
want []string
|
||||
unwant []string
|
||||
}{
|
||||
{
|
||||
path: "/collections/support/backends/0/extra_params/token",
|
||||
unwant: []string{"backend-token-displaced-canary", "backend-token-effective-canary"},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/backends/0/extra_params/password",
|
||||
unwant: []string{"backend-password-displaced-canary", "backend-password-effective-canary"},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/profiles/0/model",
|
||||
want: []string{"effective-model", "lower-model"},
|
||||
unwant: []string{},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/profiles/0/extra_params/token",
|
||||
unwant: []string{"profile-token-displaced-canary", "profile-token-effective-canary"},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/profiles/0/extra_params/password",
|
||||
unwant: []string{"profile-password-displaced-canary", "profile-password-effective-canary"},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/profiles/0/api_key_env",
|
||||
want: []string{"SCRIPTORIUM_EXPLAIN_REFERENCE"},
|
||||
unwant: []string{secret},
|
||||
},
|
||||
{
|
||||
path: "/collections/support/profiles/0/extra_params/provider.option",
|
||||
want: []string{"dotted-visible-value"},
|
||||
unwant: []string{"provider/option"},
|
||||
},
|
||||
{
|
||||
path: "",
|
||||
want: []string{"profiles:", "effective-model"},
|
||||
unwant: []string{"profile-token-displaced-canary", "backend-password-effective-canary"},
|
||||
},
|
||||
} {
|
||||
args := []string{"configuration", "explain", "--config", configPath, "--path", tc.path}
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("explain %q failed: code=%d stdout=%q stderr=%q", tc.path, code, stdout, stderr)
|
||||
}
|
||||
if stdout == "" {
|
||||
t.Fatalf("explain %q produced no output", tc.path)
|
||||
}
|
||||
for _, value := range tc.want {
|
||||
if !strings.Contains(stdout, value) {
|
||||
t.Errorf("explain %q omitted %q: %s", tc.path, value, stdout)
|
||||
}
|
||||
}
|
||||
for _, value := range tc.unwant {
|
||||
if strings.Contains(stdout, value) {
|
||||
t.Errorf("explain %q disclosed %q: %s", tc.path, value, stdout)
|
||||
}
|
||||
}
|
||||
if strings.Contains(stdout, secret) {
|
||||
t.Errorf("explain %q disclosed the resolved environment value: %s", tc.path, stdout)
|
||||
}
|
||||
}
|
||||
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, []string{"configuration", "show", "--config", configPath})
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("framework show failed: code=%d stderr=%q", code, stderr)
|
||||
}
|
||||
for _, value := range []string{"SCRIPTORIUM_EXPLAIN_REFERENCE", "backend-visible-value", "profile-visible-value", "dotted-visible-value"} {
|
||||
if !strings.Contains(stdout, value) {
|
||||
t.Errorf("effective report omitted visible configuration %q: %s", value, stdout)
|
||||
}
|
||||
}
|
||||
for _, value := range []string{
|
||||
"backend-token-displaced-canary", "backend-token-effective-canary",
|
||||
"backend-password-displaced-canary", "backend-password-effective-canary",
|
||||
"profile-token-displaced-canary", "profile-token-effective-canary",
|
||||
"profile-password-displaced-canary", "profile-password-effective-canary", secret,
|
||||
} {
|
||||
if strings.Contains(stdout, value) {
|
||||
t.Errorf("effective report disclosed classified or resolved value %q: %s", value, stdout)
|
||||
}
|
||||
}
|
||||
|
||||
profileCode, profileOutput, profileError := runCLICommand(t, inspectCommand, []string{
|
||||
"profile", "--config", configPath, "--profile", "support/summary", "--format", "json",
|
||||
})
|
||||
if profileCode != ExitOK || profileError != "" {
|
||||
t.Fatalf("existing profile inspection failed: code=%d stdout=%q stderr=%q", profileCode, profileOutput, profileError)
|
||||
}
|
||||
var profile renderformat.ProfileInspection
|
||||
if err := json.Unmarshal([]byte(profileOutput), &profile); err != nil {
|
||||
t.Fatalf("decode profile inspection: %v\n%s", err, profileOutput)
|
||||
}
|
||||
if pointerValue(profile.ProfileModelParams.Model) != "effective-model" || pointerValue(profile.ProfileModelParams.ExtraParams)["visible_option"] != "profile-visible-value" {
|
||||
t.Fatalf("existing profile inspection changed configured provider data: %+v", profile.ProfileModelParams)
|
||||
}
|
||||
if strings.Contains(profileOutput, secret) {
|
||||
t.Fatalf("profile inspection emitted resolved environment data: %s", profileOutput)
|
||||
}
|
||||
|
||||
code, rendered, stderr := runCLICommand(t, renderCommand, []string{"--config", configPath, "--prompt", "support/explainable"})
|
||||
if code != ExitOK || stderr != "" {
|
||||
t.Fatalf("prepared output failed: code=%d stdout=%q stderr=%q", code, rendered, stderr)
|
||||
}
|
||||
if !strings.Contains(rendered, "profile-visible-value") || strings.Contains(rendered, secret) {
|
||||
t.Fatalf("prepared output did not preserve ordinary provider data or disclosed environment data: %s", rendered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkConfigurationExplainRejectsMissingMalformedAndUnknownPaths(t *testing.T) {
|
||||
configPath := writeAppConfigFile(t, "{}\n")
|
||||
for _, args := range [][]string{
|
||||
{"configuration", "explain"},
|
||||
{"configuration", "explain", "--path", "not-a-pointer"},
|
||||
{"configuration", "explain", "--path", "/profiles/team~2summary/model"},
|
||||
{"configuration", "explain", "--config", configPath, "--path", "/profiles/missing/model"},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, inspectCommand, args)
|
||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
||||
t.Fatalf("invalid explanation path was accepted: args=%q code=%d stdout=%q stderr=%q", args, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func findFrameworkSource(t *testing.T, sources []frameworkConfigurationSource, path string) int {
|
||||
t.Helper()
|
||||
for index, source := range sources {
|
||||
if filepath.Clean(source.Path) == filepath.Clean(path) {
|
||||
return index
|
||||
}
|
||||
}
|
||||
t.Fatalf("selected source path %q was not present: %+v", path, sources)
|
||||
return -1
|
||||
}
|
||||
|
||||
type frameworkConfigurationFailingWriter struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (writer frameworkConfigurationFailingWriter) Write([]byte) (int, error) {
|
||||
return 0, writer.err
|
||||
}
|
||||
75
internal/adapter/cli/framework_sources.go
Normal file
75
internal/adapter/cli/framework_sources.go
Normal file
@@ -0,0 +1,75 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"go.maximumdirect.net/configkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
type promptKitSourceFlagEntry struct {
|
||||
selector string
|
||||
path string
|
||||
}
|
||||
|
||||
// promptKitSourceFlagValue treats each occurrence as one selected source path.
|
||||
type promptKitSourceFlagValue struct {
|
||||
entries []promptKitSourceFlagEntry
|
||||
}
|
||||
|
||||
func (value *promptKitSourceFlagValue) Set(path string) error {
|
||||
trimmed := strings.TrimSpace(path)
|
||||
if trimmed == "" {
|
||||
return errors.New("--promptkit-source requires a nonblank path")
|
||||
}
|
||||
|
||||
info, err := os.Stat(filepath.Clean(trimmed))
|
||||
if err != nil {
|
||||
reason := "unavailable"
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
reason = "permission denied"
|
||||
}
|
||||
return fmt.Errorf("--promptkit-source %q: %s", path, reason)
|
||||
}
|
||||
|
||||
selector := ""
|
||||
switch {
|
||||
case info.IsDir():
|
||||
selector = "directory"
|
||||
case info.Mode().IsRegular():
|
||||
selector = "file"
|
||||
default:
|
||||
return fmt.Errorf("--promptkit-source %q must be a regular file or directory", path)
|
||||
}
|
||||
value.entries = append(value.entries, promptKitSourceFlagEntry{selector: selector, path: path})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (value *promptKitSourceFlagValue) Get() any {
|
||||
selections := make([]any, 0, len(value.entries))
|
||||
for _, entry := range value.entries {
|
||||
selections = append(selections, map[string]any{entry.selector: entry.path})
|
||||
}
|
||||
return selections
|
||||
}
|
||||
|
||||
func (value *promptKitSourceFlagValue) String() string {
|
||||
paths := make([]string, 0, len(value.entries))
|
||||
for _, entry := range value.entries {
|
||||
paths = append(paths, entry.path)
|
||||
}
|
||||
return fmt.Sprint(paths)
|
||||
}
|
||||
|
||||
func writeFrameworkDiagnostics(output io.Writer, engine *promptkit.Engine) error {
|
||||
view, err := engine.InspectConfiguration()
|
||||
if err != nil {
|
||||
return fmt.Errorf("inspect captured PromptKit diagnostics: %w", err)
|
||||
}
|
||||
return configkit.WriteDiagnostics(output, view.Diagnostics())
|
||||
}
|
||||
386
internal/adapter/cli/framework_sources_test.go
Normal file
386
internal/adapter/cli/framework_sources_test.go
Normal file
@@ -0,0 +1,386 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
configkit "go.maximumdirect.net/configkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestFrameworkSourceGroupsComposeAndValidateEffectiveDefinitions(t *testing.T) {
|
||||
conflicting := t.TempDir()
|
||||
writeFrameworkSourceFile(t, conflicting, "first.yml", `profiles:
|
||||
- id: selected
|
||||
model: first
|
||||
`)
|
||||
writeFrameworkSourceFile(t, conflicting, "second.yml", `profiles:
|
||||
- id: selected
|
||||
model: second
|
||||
`)
|
||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{Directory: conflicting}}}); err == nil {
|
||||
t.Fatal("conflicting files in one source group unexpectedly constructed an engine")
|
||||
}
|
||||
|
||||
lower := t.TempDir()
|
||||
upper := t.TempDir()
|
||||
empty := t.TempDir()
|
||||
writeFrameworkSourceFile(t, lower, "lower.yml", `profiles:
|
||||
- id: selected
|
||||
endpoint: not-a-url
|
||||
model: lower-model
|
||||
temperature: 10
|
||||
`)
|
||||
writeFrameworkSourceFile(t, upper, "upper.yml", `profiles:
|
||||
- id: selected
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
model: upper-model
|
||||
temperature: 0.5
|
||||
`)
|
||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{
|
||||
{Directory: lower}, {Directory: empty}, {Directory: upper},
|
||||
}, "")
|
||||
inspection, err := engine.InspectProfile(context.Background(), "selected")
|
||||
if err != nil {
|
||||
t.Fatalf("inspect profile after later source replacement: %v", err)
|
||||
}
|
||||
if pointerValue(inspection.Endpoint) != "http://127.0.0.1:1/v1" || pointerValue(inspection.Model) != "upper-model" || pointerValue(inspection.Temperature) != 0.5 {
|
||||
t.Fatalf("later source did not replace lower values before validation: %+v", inspection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkSourceConstructionUsesCombinedDocumentAndRejectsInvalidSources(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
combined := filepath.Join(root, "combined.yml")
|
||||
writeFrameworkSourceFile(t, root, "combined.yml", `backends:
|
||||
- id: local
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
profiles:
|
||||
- id: selected
|
||||
backend: local
|
||||
model: combined-model
|
||||
prompts:
|
||||
- id: combined.prompt
|
||||
default_profile: selected
|
||||
messages:
|
||||
- role: user
|
||||
content: hello
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
client := &frameworkSourceCaptureClient{}
|
||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: combined}}, "", promptkit.WithLLMClient(client))
|
||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "combined.prompt"}); err != nil {
|
||||
t.Fatalf("run definitions from one combined document: %v", err)
|
||||
}
|
||||
if len(client.requests) != 1 || client.requests[0].Target.BackendID != "local" || client.requests[0].Target.Model != "combined-model" {
|
||||
t.Fatalf("combined backend/profile/prompt values did not reach generation: %+v", client.requests)
|
||||
}
|
||||
|
||||
malformed := filepath.Join(root, "malformed.yml")
|
||||
writeFrameworkSourceFile(t, root, "malformed.yml", "not: [valid\n")
|
||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: malformed}}}); err == nil {
|
||||
t.Fatal("malformed framework source unexpectedly constructed an engine")
|
||||
}
|
||||
|
||||
invalidUnused := filepath.Join(root, "invalid-unused.yml")
|
||||
writeFrameworkSourceFile(t, root, "invalid-unused.yml", `prompts:
|
||||
- id: unused.prompt
|
||||
messages:
|
||||
- role: user
|
||||
content: "{{"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: invalidUnused}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
|
||||
t.Fatalf("invalid unused definition did not fail as framework configuration: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkRejectsObsoleteVersionKeyedPrompt(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := writeFrameworkSourceFile(t, root, "obsolete.yml", `prompts:
|
||||
old-prompt:
|
||||
"1":
|
||||
messages:
|
||||
- role: user
|
||||
content: hello
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
if _, err := newEngine(engineSettings{promptkitSources: []appconfig.SourceSelection{{File: path}}}); !errors.Is(err, promptkit.ErrInvalidConfig) {
|
||||
t.Fatalf("obsolete version-keyed definition was not rejected by PromptKit: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameworkImportsRetainResourceOriginsAndCaptureAtConstruction(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
catalogDir := filepath.Join(root, "catalog")
|
||||
if err := os.MkdirAll(catalogDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
schemaDir := filepath.Join(root, "schemas")
|
||||
if err := os.MkdirAll(schemaDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mainFile := filepath.Join(root, "main.yml")
|
||||
writeFrameworkSourceFile(t, root, "main.yml", `config:
|
||||
imports:
|
||||
- path: catalog/base.yml
|
||||
required: true
|
||||
`)
|
||||
writeFrameworkSourceFile(t, catalogDir, "base.yml", `profiles:
|
||||
- id: profile-primary
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
model: imported-model
|
||||
prompts:
|
||||
- id: team-prompt
|
||||
default_profile: profile-primary
|
||||
inputs:
|
||||
- name: source
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content_file: ./body.md
|
||||
- role: user
|
||||
content: 'Artifact: {{input "source"}}'
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: response.schema.json
|
||||
`)
|
||||
writeFrameworkSourceFile(t, catalogDir, "body.md", "Captured message for {{.audience}}")
|
||||
artifactPath := filepath.Join(root, "request-artifact.txt")
|
||||
writeFrameworkSourceFile(t, root, "request-artifact.txt", "initial artifact")
|
||||
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"initial schema"}`)
|
||||
selections := []appconfig.SourceSelection{{File: mainFile}}
|
||||
engine := newFrameworkSourceTestEngine(t, selections, schemaDir)
|
||||
|
||||
writeFrameworkSourceFile(t, catalogDir, "body.md", "edited message for {{.audience}}")
|
||||
writeFrameworkSourceFile(t, root, "request-artifact.txt", "prepared artifact")
|
||||
writeFrameworkSourceFile(t, schemaDir, "response.schema.json", `{"type":"object","title":"prepared schema"}`)
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: "team-prompt",
|
||||
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
|
||||
Vars: map[string]string{"audience": "reader"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare inherited imported prompt: %v", err)
|
||||
}
|
||||
if len(prepared.Messages) != 2 || prepared.Messages[0].Content != "Captured message for reader" || prepared.Messages[1].Content != "Artifact: prepared artifact" {
|
||||
t.Fatalf("resource origin or capture/preparation behavior changed: %+v", prepared.Messages)
|
||||
}
|
||||
if prepared.SelectedProfileID != "profile-primary" {
|
||||
t.Fatalf("profile ID was not preserved: %+v", prepared)
|
||||
}
|
||||
if prepared.StructuredOutput == nil || prepared.StructuredOutput.JSONSchema == nil || prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)["title"] != "prepared schema" {
|
||||
t.Fatalf("schema was not loaded during preparation: %+v", prepared.StructuredOutput)
|
||||
}
|
||||
|
||||
newEngine := newFrameworkSourceTestEngine(t, selections, schemaDir)
|
||||
newPrepared, err := newEngine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: "team-prompt",
|
||||
Inputs: map[string]promptkit.ArtifactRef{"source": promptkit.File(artifactPath)},
|
||||
Vars: map[string]string{"audience": "reader"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare with newly captured message: %v", err)
|
||||
}
|
||||
if newPrepared.Messages[0].Content != "edited message for reader" {
|
||||
t.Fatalf("new engine did not observe the message edit: %+v", newPrepared.Messages[0])
|
||||
}
|
||||
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\nschema_dir: %q\n", mainFile, schemaDir))
|
||||
code, stdout, stderr := runCLICommand(t, renderCommand, []string{
|
||||
"--config", configPath,
|
||||
"--prompt", "team-prompt",
|
||||
"--input", "source=" + artifactPath,
|
||||
"--var", "audience=reader",
|
||||
"--format", "json",
|
||||
})
|
||||
if code != ExitOK {
|
||||
t.Fatalf("render exact literal IDs through CLI: code=%d stderr=%q", code, stderr)
|
||||
}
|
||||
var rendered promptkit.PreparedRun
|
||||
if err := json.Unmarshal([]byte(stdout), &rendered); err != nil {
|
||||
t.Fatalf("decode rendered literal-ID fixture: %v", err)
|
||||
}
|
||||
if rendered.PromptID != "team-prompt" || rendered.SelectedProfileID != "profile-primary" {
|
||||
t.Fatalf("CLI changed exact prompt/profile IDs: %+v", rendered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNativeProfileInheritanceAndPresenceReachTheClientBoundary(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
frameworkFile := filepath.Join(root, "framework.yml")
|
||||
writeFrameworkSourceFile(t, root, "framework.yml", `backends:
|
||||
- id: first
|
||||
endpoint: https://first.example/v1
|
||||
api_key_env: FIRST_BACKEND_KEY
|
||||
extra_params:
|
||||
source: first
|
||||
nested:
|
||||
first_backend: true
|
||||
- id: second
|
||||
endpoint: https://second.example/v1
|
||||
api_key_env: SECOND_BACKEND_KEY
|
||||
extra_params:
|
||||
source: second
|
||||
nested:
|
||||
second_backend: true
|
||||
profiles:
|
||||
- id: parent
|
||||
backend: first
|
||||
model: parent-model
|
||||
temperature: 0.7
|
||||
top_p: 0.6
|
||||
max_tokens: 45
|
||||
timeout_seconds: 30
|
||||
service_tier: flex
|
||||
extra_params:
|
||||
nested:
|
||||
parent_profile: true
|
||||
retained: "yes"
|
||||
- id: backend-child
|
||||
extends: {id: parent}
|
||||
backend: second
|
||||
model: child-model
|
||||
- id: presence-child
|
||||
extends: {id: parent}
|
||||
backend: second
|
||||
api_key_env: null
|
||||
temperature: null
|
||||
max_tokens: 0
|
||||
timeout_seconds: 0
|
||||
extra_params:
|
||||
nested:
|
||||
explicit_false: false
|
||||
- id: map-null
|
||||
backend: second
|
||||
model: map-model
|
||||
extra_params: null
|
||||
- id: map-empty
|
||||
backend: second
|
||||
model: map-model
|
||||
extra_params: {}
|
||||
prompts:
|
||||
- id: execution.prompt
|
||||
messages:
|
||||
- role: user
|
||||
content: hello
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
client := &frameworkSourceCaptureClient{}
|
||||
engine := newFrameworkSourceTestEngine(t, []appconfig.SourceSelection{{File: frameworkFile}}, "", promptkit.WithLLMClient(client))
|
||||
|
||||
for _, profileID := range []string{"backend-child", "presence-child", "map-null", "map-empty"} {
|
||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "execution.prompt", ProfileID: profileID}); err != nil {
|
||||
t.Fatalf("run profile %q: %v", profileID, err)
|
||||
}
|
||||
}
|
||||
if len(client.requests) != 4 {
|
||||
t.Fatalf("expected four provider-bound requests, got %d", len(client.requests))
|
||||
}
|
||||
|
||||
backendChild := client.requests[0].Target
|
||||
wantBackendParams := map[string]any{"source": "second", "nested": map[string]any{
|
||||
"second_backend": true, "parent_profile": true, "retained": "yes",
|
||||
}}
|
||||
if backendChild.BackendID != "second" || backendChild.Endpoint != "https://second.example/v1" || backendChild.APIKeyEnv != "SECOND_BACKEND_KEY" || backendChild.Model != "child-model" ||
|
||||
backendChild.Temperature != 0.7 || backendChild.TopP != 0.6 || backendChild.MaxTokens != 45 || backendChild.TimeoutSeconds != 30 || backendChild.ServiceTier != "flex" || !reflect.DeepEqual(backendChild.ExtraParams, wantBackendParams) {
|
||||
t.Fatalf("backend change lost backend-derived or explicit inherited values: %+v", backendChild)
|
||||
}
|
||||
|
||||
presenceRequest := client.requests[1]
|
||||
presenceTarget := presenceRequest.Target
|
||||
if presenceTarget.APIKeyEnv != "SECOND_BACKEND_KEY" || presenceTarget.Temperature != 0 || presenceTarget.MaxTokens != 0 || presenceTarget.TimeoutSeconds != 0 ||
|
||||
!presenceRequest.TargetPresence.MaxTokens || !presenceRequest.TargetPresence.TimeoutSeconds || presenceRequest.TargetPresence.Temperature || !presenceRequest.TargetPresence.TopP {
|
||||
t.Fatalf("explicit null/zero presence did not reach the injected client: target=%+v presence=%+v", presenceTarget, presenceRequest.TargetPresence)
|
||||
}
|
||||
wantPresenceParams := map[string]any{"source": "second", "nested": map[string]any{
|
||||
"second_backend": true, "parent_profile": true, "retained": "yes", "explicit_false": false,
|
||||
}}
|
||||
if !reflect.DeepEqual(presenceTarget.ExtraParams, wantPresenceParams) {
|
||||
t.Fatalf("recursive provider-map retention changed: %#v", presenceTarget.ExtraParams)
|
||||
}
|
||||
if !reflect.DeepEqual(client.requests[2].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
|
||||
t.Fatalf("whole-map null did not fall back to backend defaults: %#v", client.requests[2].Target.ExtraParams)
|
||||
}
|
||||
if !reflect.DeepEqual(client.requests[3].Target.ExtraParams, map[string]any{"source": "second", "nested": map[string]any{"second_backend": true}}) {
|
||||
t.Fatalf("empty provider map unexpectedly cleared backend defaults: %#v", client.requests[3].Target.ExtraParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCLICompletesPartialFrameworkProfileWithRequestOverrides(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
frameworkFile := writeFrameworkSourceFile(t, root, "partial.yml", `profiles:
|
||||
- id: partial
|
||||
prompts:
|
||||
- id: partial.prompt
|
||||
default_profile: partial
|
||||
messages:
|
||||
- role: user
|
||||
content: hello
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)
|
||||
provider := newTestLLMServer("completed partial profile", nil)
|
||||
defer provider.Close()
|
||||
configPath := writeAppConfigFile(t, fmt.Sprintf("promptkit_sources:\n - file: %q\n", frameworkFile))
|
||||
code, stdout, stderr := runCLICommand(t, runCommand, []string{
|
||||
"--config", configPath,
|
||||
"--prompt", "partial.prompt",
|
||||
"--llm-base-url", provider.URL + "/v1",
|
||||
"--model", "request-model",
|
||||
})
|
||||
if code != ExitOK || stdout != "completed partial profile" {
|
||||
t.Fatalf("request overrides did not complete the partial profile: code=%d stdout=%q stderr=%q", code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func newFrameworkSourceTestEngine(t *testing.T, selections []appconfig.SourceSelection, schemaDir string, options ...promptkit.Option) *promptkit.Engine {
|
||||
t.Helper()
|
||||
engine, err := newEngine(engineSettings{promptkitSources: selections, schemaDir: schemaDir}, options...)
|
||||
if err != nil {
|
||||
var loadError *configkit.LoadError
|
||||
if errors.As(err, &loadError) {
|
||||
t.Fatalf("construct engine from selected framework sources: %v (%+v)", err, loadError.Diagnostics())
|
||||
}
|
||||
t.Fatalf("construct engine from selected framework sources: %v", err)
|
||||
}
|
||||
return engine
|
||||
}
|
||||
|
||||
func writeFrameworkSourceFile(t *testing.T, directory, name, contents string) string {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(directory, 0o755); err != nil {
|
||||
t.Fatalf("create framework source directory: %v", err)
|
||||
}
|
||||
path := filepath.Join(directory, name)
|
||||
if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
|
||||
t.Fatalf("write framework source %q: %v", path, err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
type frameworkSourceCaptureClient struct {
|
||||
requests []promptkit.GenerateRequest
|
||||
}
|
||||
|
||||
func (client *frameworkSourceCaptureClient) Generate(_ context.Context, request promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||
client.requests = append(client.requests, request)
|
||||
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||
}
|
||||
@@ -6,40 +6,29 @@ import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
appformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
type promptInspectionConfig struct {
|
||||
configPath, promptDir, promptID, promptVersion, outputPath string
|
||||
configOverlay string
|
||||
configExplicit bool
|
||||
outputFormat appformat.OutputFormat
|
||||
overrides []configkit.Override
|
||||
diagnostics []configkit.Diagnostic
|
||||
configPath, promptID, schemaDir, outputPath string
|
||||
configOverlay string
|
||||
configExplicit bool
|
||||
outputFormat appformat.OutputFormat
|
||||
overrides []configkit.Override
|
||||
diagnostics []configkit.Diagnostic
|
||||
}
|
||||
|
||||
type profileInspectionConfig struct {
|
||||
configPath, profileDir, profileID, outputPath string
|
||||
configOverlay string
|
||||
configExplicit bool
|
||||
outputFormat appformat.OutputFormat
|
||||
overrides []configkit.Override
|
||||
diagnostics []configkit.Diagnostic
|
||||
}
|
||||
|
||||
// emptyPromptDefinitionFS satisfies Promptkit's engine-level prompt-source
|
||||
// requirement without exposing the caller's working directory. Profile
|
||||
// inspection never reads this source.
|
||||
type emptyPromptDefinitionFS struct{}
|
||||
|
||||
func (emptyPromptDefinitionFS) Open(name string) (fs.File, error) {
|
||||
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
|
||||
configPath, profileID, schemaDir, outputPath string
|
||||
configOverlay string
|
||||
configExplicit bool
|
||||
outputFormat appformat.OutputFormat
|
||||
overrides []configkit.Override
|
||||
diagnostics []configkit.Diagnostic
|
||||
}
|
||||
|
||||
func inspectCommand(args []string, stdout, stderr io.Writer) int {
|
||||
@@ -50,6 +39,9 @@ func inspectCommand(args []string, stdout, stderr io.Writer) int {
|
||||
if args[0] == "profile" {
|
||||
return inspectProfileCommand(args[1:], stdout, stderr)
|
||||
}
|
||||
if args[0] == "configuration" {
|
||||
return inspectFrameworkConfigurationCommand(args[1:], stdout, stderr)
|
||||
}
|
||||
if args[0] != "prompt" {
|
||||
fmt.Fprintln(stderr, "inspect parse error: unknown inspection mode")
|
||||
return ExitRuntimeError
|
||||
@@ -69,10 +61,14 @@ func inspectCommand(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
engine, err := newEngine(settings)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID, cfg.promptVersion)
|
||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
inspection, err := engine.InspectPrompt(context.Background(), cfg.promptID)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "inspect error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
@@ -103,9 +99,13 @@ func inspectProfileCommand(args []string, stdout, stderr io.Writer) int {
|
||||
if err := writeApplicationWarnings(stderr, cfg.diagnostics); err != nil {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
engine, err := newEngine(settings, promptkit.WithPromptFS(emptyPromptDefinitionFS{}, "."))
|
||||
engine, err := newEngine(settings)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
inspection, err := engine.InspectProfile(context.Background(), cfg.profileID)
|
||||
@@ -130,7 +130,9 @@ func parseProfileInspectionArgs(args []string) (*profileInspectionConfig, error)
|
||||
fs := flag.NewFlagSet("inspect profile", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profiles")
|
||||
var promptkitSources promptKitSourceFlagValue
|
||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||
fs.StringVar(&cfg.profileID, "profile", "", "profile ID to inspect")
|
||||
rawFormat := ""
|
||||
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
||||
@@ -163,7 +165,7 @@ func resolveAppSettingsForProfileInspection(cfg *profileInspectionConfig) (engin
|
||||
return engineSettings{}, err
|
||||
}
|
||||
cfg.diagnostics = diagnostics
|
||||
return engineSettings{profileDir: settings.ProfileDir, backends: settings.Backends, transientRetries: settings.TransientRetries}, nil
|
||||
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
|
||||
}
|
||||
|
||||
func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
|
||||
@@ -171,9 +173,10 @@ func parsePromptInspectionArgs(args []string) (*promptInspectionConfig, error) {
|
||||
fs := flag.NewFlagSet("inspect prompt", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definitions")
|
||||
var promptkitSources promptKitSourceFlagValue
|
||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to inspect")
|
||||
fs.StringVar(&cfg.promptVersion, "prompt-version", "", "optional prompt version")
|
||||
rawFormat := ""
|
||||
fs.StringVar(&rawFormat, "format", "", "output format: text or json")
|
||||
fs.StringVar(&cfg.outputPath, "out", "", "optional output file path")
|
||||
@@ -205,8 +208,5 @@ func resolveAppSettingsForPromptInspection(cfg *promptInspectionConfig) (engineS
|
||||
return engineSettings{}, err
|
||||
}
|
||||
cfg.diagnostics = diagnostics
|
||||
if strings.TrimSpace(settings.PromptDir) == "" {
|
||||
return engineSettings{}, errors.New(errPromptDirRequired)
|
||||
}
|
||||
return engineSettings{promptDir: settings.PromptDir, backends: settings.Backends, transientRetries: settings.TransientRetries}, nil
|
||||
return engineSettings{promptkitSources: settings.PromptKitSources, schemaDir: settings.SchemaDir, transientRetries: settings.TransientRetries}, nil
|
||||
}
|
||||
|
||||
@@ -13,13 +13,13 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
httpadapter "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
||||
appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
executionreport "gitea.maximumdirect.net/eric/scriptorium/internal/report"
|
||||
"go.maximumdirect.net/configkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -28,17 +28,11 @@ const (
|
||||
ExitValidationFailed = 2
|
||||
)
|
||||
|
||||
const (
|
||||
errPromptDirRequired = "prompt directory is required; provide --prompt-dir or config.yml prompt_dir"
|
||||
)
|
||||
|
||||
type runConfig struct {
|
||||
configPath string
|
||||
|
||||
promptDir string
|
||||
profileDir string
|
||||
promptkitSources []appconfig.SourceSelection
|
||||
promptID string
|
||||
promptVersion string
|
||||
profileID string
|
||||
sessionID string
|
||||
inputRaw listFlag
|
||||
@@ -53,7 +47,6 @@ type runConfig struct {
|
||||
topP float64
|
||||
reasoningEffort string
|
||||
schemaDir string
|
||||
backends []appconfig.BackendSettings
|
||||
timeout time.Duration
|
||||
runTimeout time.Duration
|
||||
transientRetries int
|
||||
@@ -83,22 +76,19 @@ type serveConfig struct {
|
||||
configPath string
|
||||
|
||||
addr string
|
||||
promptDir string
|
||||
profileDir string
|
||||
promptkitSources []appconfig.SourceSelection
|
||||
schemaDir string
|
||||
artifactRoot string
|
||||
maxRequestBytes int64
|
||||
maxArtifactBytes int64
|
||||
maxResponseBytes int64
|
||||
backends []appconfig.BackendSettings
|
||||
transientRetries *int
|
||||
diagnostics []configkit.Diagnostic
|
||||
configOverlay string
|
||||
}
|
||||
|
||||
type commonCommandSettings struct {
|
||||
promptDir string
|
||||
profileDir string
|
||||
promptkitSources []appconfig.SourceSelection
|
||||
schemaDir string
|
||||
serverAddr string
|
||||
artifactRoot string
|
||||
@@ -106,7 +96,6 @@ type commonCommandSettings struct {
|
||||
maxArtifactBytes int64
|
||||
maxResponseBytes int64
|
||||
defaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||
backends []appconfig.BackendSettings
|
||||
transientRetries *int
|
||||
diagnostics []configkit.Diagnostic
|
||||
}
|
||||
@@ -164,7 +153,11 @@ func runCommand(args []string, stdout, stderr io.Writer) int {
|
||||
|
||||
engine, err := newEngine(cfg.engineSettings())
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
return executeRun(cfg, req, engine.Run, stdout, stderr)
|
||||
@@ -176,16 +169,10 @@ func executeRun(cfg *runConfig, req promptkit.RunRequest, run func(context.Conte
|
||||
collector = executionreport.NewCollector()
|
||||
req.Observer = collector.Observe
|
||||
}
|
||||
runContext := context.Background()
|
||||
cancel := func() {}
|
||||
if cfg.runTimeout > 0 {
|
||||
runContext, cancel = context.WithTimeout(runContext, cfg.runTimeout)
|
||||
}
|
||||
res, runErr := run(runContext, req)
|
||||
contextErr := runContext.Err()
|
||||
cancel()
|
||||
req.OperationTimeout = cfg.runTimeout
|
||||
res, runErr := run(context.Background(), req)
|
||||
if runErr != nil {
|
||||
if errors.Is(contextErr, context.DeadlineExceeded) {
|
||||
if errors.Is(runErr, promptkit.ErrOperationTimeout) {
|
||||
fmt.Fprintln(stderr, "run error: execution deadline exceeded")
|
||||
} else {
|
||||
fmt.Fprintln(stderr, runErrorMessage(runErr))
|
||||
@@ -242,7 +229,11 @@ func renderCommand(args []string, stdout, stderr io.Writer) int {
|
||||
|
||||
engine, err := newEngine(cfg.runConfig.engineSettings())
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
@@ -283,7 +274,11 @@ func serveCommand(args []string, stderr io.Writer) int {
|
||||
|
||||
engine, err := newEngine(cfg.engineSettings(), promptkit.WithArtifactReader(artifactReader))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "engine error: %v\n", err)
|
||||
writeApplicationError(stderr, "engine", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
if err := writeFrameworkDiagnostics(stderr, engine); err != nil {
|
||||
fmt.Fprintf(stderr, "engine diagnostic write error: %v\n", err)
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
@@ -400,8 +395,8 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
|
||||
registerConfigSelectionFlags(fs, &cfg.configPath, &cfg.configOverlay)
|
||||
fs.StringVar(&cfg.addr, "addr", "", "HTTP listen address")
|
||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
||||
var promptkitSources promptKitSourceFlagValue
|
||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
||||
fs.StringVar(&cfg.schemaDir, "schema-dir", "", "base directory for validation schemas")
|
||||
fs.StringVar(&cfg.artifactRoot, "artifact-root", "", "base directory for HTTP file input artifacts")
|
||||
fs.Int64Var(&cfg.maxRequestBytes, "max-request-bytes", 0, "maximum HTTP request body bytes; 0 disables the limit")
|
||||
@@ -420,30 +415,23 @@ func parseServeArgs(args []string) (*serveConfig, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.promptkitSources = settings.promptkitSources
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.addr = settings.serverAddr
|
||||
cfg.artifactRoot = settings.artifactRoot
|
||||
cfg.maxRequestBytes = settings.maxRequestBytes
|
||||
cfg.maxArtifactBytes = settings.maxArtifactBytes
|
||||
cfg.maxResponseBytes = settings.maxResponseBytes
|
||||
cfg.backends = settings.backends
|
||||
cfg.transientRetries = settings.transientRetries
|
||||
cfg.diagnostics = settings.diagnostics
|
||||
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func registerExecutionRequestFlags(fs *flag.FlagSet, cfg *runConfig) {
|
||||
fs.StringVar(&cfg.promptDir, "prompt-dir", "", "directory containing prompt definition YAML files")
|
||||
fs.StringVar(&cfg.profileDir, "profile-dir", "", "directory containing execution profile YAML files")
|
||||
var promptkitSources promptKitSourceFlagValue
|
||||
fs.Var(&promptkitSources, "promptkit-source", "selected PromptKit source path (repeatable)")
|
||||
fs.StringVar(&cfg.promptID, "prompt", "", "prompt ID to run")
|
||||
fs.StringVar(&cfg.promptVersion, "prompt-version", "", "optional prompt definition version")
|
||||
fs.StringVar(&cfg.profileID, "profile", "", "optional execution profile ID; if omitted, prompt default_profile is used")
|
||||
fs.StringVar(&cfg.sessionID, "session-id", "", "optional session ID")
|
||||
fs.Var(&cfg.inputRaw, "input", "input mapping(s): name=path (repeatable, comma-separated)")
|
||||
@@ -472,17 +460,12 @@ func finalizeExecutionRequestConfig(fs *flag.FlagSet, cfg *runConfig) error {
|
||||
return err
|
||||
}
|
||||
|
||||
cfg.promptDir = settings.promptDir
|
||||
cfg.profileDir = settings.profileDir
|
||||
cfg.promptkitSources = settings.promptkitSources
|
||||
cfg.schemaDir = settings.schemaDir
|
||||
cfg.defaultRenderFormat = settings.defaultRenderFormat
|
||||
cfg.backends = settings.backends
|
||||
cfg.configuredTransientRetries = settings.transientRetries
|
||||
cfg.diagnostics = settings.diagnostics
|
||||
|
||||
if err := validateRequiredLibraryDirs(cfg.promptDir); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(cfg.promptID) == "" {
|
||||
return errors.New("--prompt is required")
|
||||
}
|
||||
@@ -524,8 +507,7 @@ func resolveCommonSettings(fs *flag.FlagSet, configPath, overlay string) (common
|
||||
return commonCommandSettings{}, err
|
||||
}
|
||||
return commonCommandSettings{
|
||||
promptDir: settings.PromptDir,
|
||||
profileDir: settings.ProfileDir,
|
||||
promptkitSources: settings.PromptKitSources,
|
||||
schemaDir: settings.SchemaDir,
|
||||
serverAddr: settings.ServerAddr,
|
||||
artifactRoot: settings.ArtifactRoot,
|
||||
@@ -533,54 +515,50 @@ func resolveCommonSettings(fs *flag.FlagSet, configPath, overlay string) (common
|
||||
maxArtifactBytes: settings.MaxArtifactBytes,
|
||||
maxResponseBytes: settings.MaxResponseBytes,
|
||||
defaultRenderFormat: settings.DefaultRenderFormat,
|
||||
backends: settings.Backends,
|
||||
transientRetries: settings.TransientRetries,
|
||||
diagnostics: diagnostics,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func validateRequiredLibraryDirs(promptDir string) error {
|
||||
if strings.TrimSpace(promptDir) == "" {
|
||||
return errors.New(errPromptDirRequired)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type engineSettings struct {
|
||||
promptDir string
|
||||
profileDir string
|
||||
promptkitSources []appconfig.SourceSelection
|
||||
schemaDir string
|
||||
backends []appconfig.BackendSettings
|
||||
transientRetries *int
|
||||
}
|
||||
|
||||
func (c runConfig) engineSettings() engineSettings {
|
||||
return engineSettings{promptDir: c.promptDir, profileDir: c.profileDir, schemaDir: c.schemaDir, backends: c.backends, transientRetries: c.configuredTransientRetries}
|
||||
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.configuredTransientRetries}
|
||||
}
|
||||
|
||||
func (c serveConfig) engineSettings() engineSettings {
|
||||
return engineSettings{promptDir: c.promptDir, profileDir: c.profileDir, schemaDir: c.schemaDir, backends: c.backends, transientRetries: c.transientRetries}
|
||||
return engineSettings{promptkitSources: c.promptkitSources, schemaDir: c.schemaDir, transientRetries: c.transientRetries}
|
||||
}
|
||||
|
||||
func newEngine(settings engineSettings, options ...promptkit.Option) (*promptkit.Engine, error) {
|
||||
engineOptions := make([]promptkit.Option, 0, len(settings.backends)+len(options))
|
||||
for _, configured := range settings.backends {
|
||||
engineOptions = append(engineOptions, promptkit.WithBackend(promptkit.Backend{
|
||||
ID: configured.ID,
|
||||
Endpoint: configured.Endpoint,
|
||||
APIKeyEnv: configured.APIKeyEnv,
|
||||
ExtraParams: configured.ExtraParams,
|
||||
ConcurrencyLimit: configured.ConcurrencyLimit,
|
||||
QueueCapacity: configured.QueueCapacity,
|
||||
}))
|
||||
underlays := make([]configkit.Underlay, 0, len(settings.promptkitSources))
|
||||
for index, selection := range settings.promptkitSources {
|
||||
var inputs []configkit.Input
|
||||
switch {
|
||||
case selection.File != "" && selection.Directory == "":
|
||||
inputs = []configkit.Input{{Source: configkit.File(selection.File), Imports: true}}
|
||||
case selection.Directory != "" && selection.File == "":
|
||||
var err error
|
||||
inputs, err = promptkit.DirectoryInputs(selection.Directory)
|
||||
if err != nil {
|
||||
cause := fmt.Errorf("PromptKit source %d directory %q: %w", index+1, selection.Directory, err)
|
||||
return nil, fmt.Errorf("engine initialization from application configuration: %w", errors.Join(promptkit.ErrInvalidConfig, cause))
|
||||
}
|
||||
default:
|
||||
cause := fmt.Errorf("PromptKit source %d must select exactly one file or directory", index+1)
|
||||
return nil, fmt.Errorf("engine initialization from application configuration: %w", errors.Join(promptkit.ErrInvalidConfig, cause))
|
||||
}
|
||||
underlays = append(underlays, configkit.Underlay{Inputs: inputs})
|
||||
}
|
||||
engineOptions = append(engineOptions, options...)
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: settings.promptDir,
|
||||
ProfileDir: settings.profileDir,
|
||||
Sources: promptkit.ConfigurationSources{Underlays: underlays},
|
||||
SchemaDir: settings.schemaDir,
|
||||
TransientRetries: settings.transientRetries,
|
||||
}, engineOptions...)
|
||||
}, options...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("engine initialization from application configuration: %w", err)
|
||||
}
|
||||
@@ -644,7 +622,6 @@ func buildRunRequestFromConfig(cfg *runConfig) (promptkit.RunRequest, error) {
|
||||
}
|
||||
return promptkit.RunRequest{
|
||||
PromptID: cfg.promptID,
|
||||
PromptVersion: cfg.promptVersion,
|
||||
ProfileID: cfg.profileID,
|
||||
SessionID: cfg.sessionID,
|
||||
Inputs: inputs,
|
||||
@@ -723,9 +700,8 @@ func printSummary(stderr io.Writer, res *promptkit.RunResult) {
|
||||
if res == nil {
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
|
||||
fmt.Fprintf(stderr, "prompt=%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
|
||||
res.PromptID,
|
||||
res.PromptVersion,
|
||||
res.SelectedProfileID,
|
||||
res.ModelName,
|
||||
res.Validation.Status,
|
||||
@@ -755,10 +731,11 @@ func runErrorMessage(err error) string {
|
||||
|
||||
func printUsage(w io.Writer) {
|
||||
fmt.Fprintln(w, "usage: scriptorium <run|render|serve|inspect|config> ...")
|
||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]")
|
||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]")
|
||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--prompt-version VERSION] [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--profile-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]")
|
||||
fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] --prompt ID [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]")
|
||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--addr %s] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " inspect configuration: scriptorium inspect configuration <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [--promptkit-source PATH ...] [--schema-dir DIR] [--path JSON_POINTER (required for explain)]")
|
||||
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [application overrides] [--path /pointer for explain]")
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
const (
|
||||
scriptoriumModulePath = "gitea.maximumdirect.net/eric/scriptorium"
|
||||
promptkitInternalPath = "gitea.maximumdirect.net/eric/promptkit/internal"
|
||||
promptkitInternalPath = "go.maximumdirect.net/promptkit/internal"
|
||||
promptkitOpenRouterCatalogPath = "gitea.maximumdirect.net/eric/promptkit-backend-openrouter"
|
||||
promptkitRakestrawhomeCatalogPath = "gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome"
|
||||
)
|
||||
@@ -149,7 +149,7 @@ func TestForbiddenImportScannerAllowsRetainedApplicationPackages(t *testing.T) {
|
||||
source := `package consumer
|
||||
|
||||
import (
|
||||
_ "gitea.maximumdirect.net/eric/promptkit"
|
||||
_ "go.maximumdirect.net/promptkit"
|
||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/adapter/http"
|
||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/config"
|
||||
_ "gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
|
||||
@@ -6,12 +6,13 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -136,7 +137,11 @@ func readArtifactFile(path string, maxBytes int64) (*promptkit.Artifact, error)
|
||||
|
||||
var reader io.Reader = file
|
||||
if maxBytes > 0 {
|
||||
reader = io.LimitReader(file, maxBytes+1)
|
||||
readLimit := maxBytes
|
||||
if readLimit < math.MaxInt64 {
|
||||
readLimit++ // Read one extra byte to detect an oversized file.
|
||||
}
|
||||
reader = io.LimitReader(file, readLimit)
|
||||
}
|
||||
body, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
|
||||
@@ -3,14 +3,45 @@ package httpadapter
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestRestrictedArtifactReaderReturnsDetachedBodies(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "input.txt"), []byte("original"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reader, err := NewRestrictedArtifactReader(root, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, ref := range []promptkit.ArtifactRef{promptkit.Inline("original"), promptkit.File("input.txt")} {
|
||||
first, err := reader.Read(context.Background(), ref)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := reader.Read(context.Background(), ref)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first.Body[0] = 'X'
|
||||
first.Name = "changed"
|
||||
if string(second.Body) != "original" || second.Name == "changed" {
|
||||
t.Fatalf("read results share mutable state: %+v", second)
|
||||
}
|
||||
second.Body[1] = 'Y'
|
||||
if string(first.Body) != "Xriginal" {
|
||||
t.Fatalf("reader retained a body alias: %q", first.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestrictedArtifactReaderReadsContainedFiles(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
@@ -150,6 +181,15 @@ func TestRestrictedArtifactReaderEnforcesLimits(t *testing.T) {
|
||||
t.Fatalf("expected unlimited artifact, got %#v and %v", artifact, err)
|
||||
}
|
||||
|
||||
maxLimited, err := NewRestrictedArtifactReader(root, math.MaxInt64)
|
||||
if err != nil {
|
||||
t.Fatalf("construct reader at maximum limit: %v", err)
|
||||
}
|
||||
artifact, err = maxLimited.Read(context.Background(), promptkit.File("large.txt"))
|
||||
if err != nil || string(artifact.Body) != "123456" || artifact.Size != 6 {
|
||||
t.Fatalf("expected full artifact at maximum limit, got %#v and %v", artifact, err)
|
||||
}
|
||||
|
||||
if _, err := NewRestrictedArtifactReader(root, -1); err == nil {
|
||||
t.Fatal("expected negative limit to fail")
|
||||
}
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
|
||||
type runRequestDTO struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs map[string]inputRefDTO `json:"inputs"`
|
||||
@@ -59,28 +58,21 @@ type artifactDTO struct {
|
||||
type metadataDTO struct {
|
||||
RunID string `json:"run_id"`
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version"`
|
||||
PromptHash string `json:"prompt_hash"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
SelectedBackendID string `json:"selected_backend_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
ModelName string `json:"model_name"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
ModelParams modelParamsDTO `json:"model_params"`
|
||||
InputHashes map[string]string `json:"input_hashes"`
|
||||
Usage tokenUsageDTO `json:"usage"`
|
||||
StartTime time.Time `json:"start_time"`
|
||||
EndTime time.Time `json:"end_time"`
|
||||
DurationMS int64 `json:"duration_ms"`
|
||||
ValidationMode string `json:"validation_mode"`
|
||||
ValidationStatus string `json:"validation_status"`
|
||||
RepairAttemptsUsed int `json:"repair_attempts_used"`
|
||||
}
|
||||
|
||||
type modelParamsDTO struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
BackendID string `json:"backend_id,omitempty"`
|
||||
Model string `json:"model"`
|
||||
Temperature float64 `json:"temperature"`
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
|
||||
@@ -10,9 +10,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
executionreport "gitea.maximumdirect.net/eric/scriptorium/internal/report"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
type Runner interface {
|
||||
@@ -106,38 +106,36 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
model = executionTargetOverrideFromModelOverrideDTO(req.Model)
|
||||
}
|
||||
|
||||
runContext := r.Context()
|
||||
var operationTimeout time.Duration
|
||||
if req.RunTimeoutSeconds != nil && *req.RunTimeoutSeconds > 0 {
|
||||
var cancel context.CancelFunc
|
||||
runContext, cancel = context.WithTimeout(runContext, time.Duration(*req.RunTimeoutSeconds)*time.Second)
|
||||
defer cancel()
|
||||
operationTimeout = time.Duration(*req.RunTimeoutSeconds) * time.Second
|
||||
}
|
||||
mappedRequest := promptkit.RunRequest{
|
||||
PromptID: req.PromptID,
|
||||
PromptVersion: req.PromptVersion,
|
||||
ProfileID: req.ProfileID,
|
||||
SessionID: req.SessionID,
|
||||
Inputs: mappedInputs,
|
||||
Vars: req.Vars,
|
||||
Execution: model,
|
||||
TransientRetries: req.TransientRetries,
|
||||
OperationTimeout: operationTimeout,
|
||||
}
|
||||
var collector *executionreport.Collector
|
||||
if req.IncludeExecutionReport {
|
||||
collector = executionreport.NewCollector()
|
||||
mappedRequest.Observer = collector.Observe
|
||||
}
|
||||
res, err := h.runner.Run(runContext, mappedRequest)
|
||||
res, err := h.runner.Run(r.Context(), mappedRequest)
|
||||
var execution *executionreport.Execution
|
||||
if collector != nil {
|
||||
execution = collector.Snapshot()
|
||||
}
|
||||
if err != nil {
|
||||
switch runContext.Err() {
|
||||
case context.DeadlineExceeded:
|
||||
switch {
|
||||
case errors.Is(err, promptkit.ErrOperationTimeout), errors.Is(err, context.DeadlineExceeded) && errors.Is(r.Context().Err(), context.DeadlineExceeded):
|
||||
writeRunFailure(w, http.StatusGatewayTimeout, "deadline_exceeded", "run execution deadline exceeded", execution, h.options.MaxResponseBytes)
|
||||
return
|
||||
case context.Canceled:
|
||||
case errors.Is(err, context.Canceled) && errors.Is(r.Context().Err(), context.Canceled):
|
||||
writeRunFailure(w, http.StatusRequestTimeout, "request_canceled", "run request canceled", execution, h.options.MaxResponseBytes)
|
||||
return
|
||||
}
|
||||
@@ -159,14 +157,11 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
Metadata: metadataDTO{
|
||||
RunID: res.RunID,
|
||||
PromptID: res.PromptID,
|
||||
PromptVersion: res.PromptVersion,
|
||||
PromptHash: res.PromptHash,
|
||||
RenderedPromptHash: res.RenderedPromptHash,
|
||||
SelectedProfileID: res.SelectedProfileID,
|
||||
SelectedBackendID: res.SelectedBackendID,
|
||||
SessionID: res.SessionID,
|
||||
ModelName: res.ModelName,
|
||||
Endpoint: res.Endpoint,
|
||||
ModelParams: modelParamsDTOFromExecutionTarget(res.EffectiveModelParams),
|
||||
InputHashes: res.InputHashes,
|
||||
Usage: tokenUsageDTO{
|
||||
@@ -176,12 +171,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
CachedTokens: res.Usage.CachedTokens,
|
||||
CacheWriteTokens: res.Usage.CacheWriteTokens,
|
||||
},
|
||||
StartTime: res.StartTime,
|
||||
EndTime: res.EndTime,
|
||||
DurationMS: res.Duration.Milliseconds(),
|
||||
ValidationMode: string(res.Validation.Mode),
|
||||
ValidationStatus: string(res.Validation.Status),
|
||||
RepairAttemptsUsed: res.Validation.RepairAttempts,
|
||||
StartTime: res.StartTime,
|
||||
EndTime: res.EndTime,
|
||||
DurationMS: res.Duration.Milliseconds(),
|
||||
},
|
||||
Execution: execution,
|
||||
}
|
||||
@@ -221,7 +213,6 @@ func executionTargetOverrideFromModelOverrideDTO(dto *modelOverrideRequestDTO) *
|
||||
func modelParamsDTOFromExecutionTarget(target promptkit.ExecutionTarget) modelParamsDTO {
|
||||
return modelParamsDTO{
|
||||
Endpoint: target.Endpoint,
|
||||
BackendID: target.BackendID,
|
||||
Model: target.Model,
|
||||
Temperature: target.Temperature,
|
||||
MaxTokens: target.MaxTokens,
|
||||
@@ -255,16 +246,16 @@ func mapRunError(err error) (int, string, string) {
|
||||
return http.StatusBadRequest, "profile_required", "profile_id is required when prompt default_profile is not set"
|
||||
case errors.Is(err, promptkit.ErrAPIKeyEnvMissing):
|
||||
return http.StatusBadRequest, "api_key_env_missing", "api_key_env is set but the environment variable is missing"
|
||||
case errors.Is(err, ErrFileNotAllowed), errors.Is(err, ErrFileOutsideRoot):
|
||||
return http.StatusBadRequest, "artifact_not_allowed", "file input artifact is not allowed"
|
||||
case errors.Is(err, ErrFileTooLarge):
|
||||
return http.StatusRequestEntityTooLarge, "artifact_too_large", "file input artifact is too large"
|
||||
case errors.Is(err, promptkit.ErrPromptLoad):
|
||||
return http.StatusBadRequest, "prompt_load_failed", "failed to load prompt definition"
|
||||
case errors.Is(err, promptkit.ErrProfileLoad):
|
||||
return http.StatusBadRequest, "profile_load_failed", "failed to load execution profile"
|
||||
case errors.Is(err, promptkit.ErrInvalidRequest):
|
||||
return http.StatusBadRequest, "invalid_request", "invalid run request"
|
||||
case errors.Is(err, ErrFileNotAllowed), errors.Is(err, ErrFileOutsideRoot):
|
||||
return http.StatusBadRequest, "artifact_not_allowed", "file input artifact is not allowed"
|
||||
case errors.Is(err, ErrFileTooLarge):
|
||||
return http.StatusRequestEntityTooLarge, "artifact_too_large", "file input artifact is too large"
|
||||
case errors.Is(err, promptkit.ErrArtifactLoad):
|
||||
return http.StatusBadRequest, "artifact_read_failed", "failed to read input artifact"
|
||||
case errors.Is(err, promptkit.ErrPromptRender):
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -15,7 +16,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/configkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
type fakeRunner struct {
|
||||
@@ -335,11 +337,13 @@ func TestHandlerRunTimeoutValidationAndContextMapping(t *testing.T) {
|
||||
name, body string
|
||||
parent context.Context
|
||||
wantDeadline time.Time
|
||||
wantTimeout time.Duration
|
||||
}{
|
||||
{name: "omitted inherits deadline", body: `{"prompt_id":"p"}`, parent: parent, wantDeadline: parentDeadline},
|
||||
{name: "null inherits deadline", body: `{"prompt_id":"p","run_timeout_seconds":null}`, parent: parent, wantDeadline: parentDeadline},
|
||||
{name: "zero inherits deadline", body: `{"prompt_id":"p","run_timeout_seconds":0}`, parent: parent, wantDeadline: parentDeadline},
|
||||
{name: "positive preserves earlier deadline", body: `{"prompt_id":"p","run_timeout_seconds":10}`, parent: parent, wantDeadline: parentDeadline},
|
||||
{name: "positive preserves earlier deadline", body: `{"prompt_id":"p","run_timeout_seconds":10}`, parent: parent, wantDeadline: parentDeadline, wantTimeout: 10 * time.Second},
|
||||
{name: "positive adds operation budget", body: `{"prompt_id":"p","run_timeout_seconds":2}`, parent: context.Background(), wantTimeout: 2 * time.Second},
|
||||
{name: "omitted adds no deadline", body: `{"prompt_id":"p"}`, parent: context.Background()},
|
||||
{name: "null adds no deadline", body: `{"prompt_id":"p","run_timeout_seconds":null}`, parent: context.Background()},
|
||||
{name: "zero adds no deadline", body: `{"prompt_id":"p","run_timeout_seconds":0}`, parent: context.Background()},
|
||||
@@ -352,6 +356,9 @@ func TestHandlerRunTimeoutValidationAndContextMapping(t *testing.T) {
|
||||
if ok != !tc.wantDeadline.IsZero() || (ok && !deadline.Equal(tc.wantDeadline)) {
|
||||
t.Fatalf("execution deadline: got %v (present=%t), want %v", deadline, ok, tc.wantDeadline)
|
||||
}
|
||||
if req.OperationTimeout != tc.wantTimeout {
|
||||
t.Fatalf("operation timeout = %v, want %v", req.OperationTimeout, tc.wantTimeout)
|
||||
}
|
||||
return &promptkit.RunResult{}, nil
|
||||
}))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -381,6 +388,22 @@ func TestHandlerRunTimeoutValidationAndContextMapping(t *testing.T) {
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p","run_timeout_seconds":10}`)))
|
||||
assertHTTPErrorCode(t, w, http.StatusBadGateway, "llm_failed")
|
||||
|
||||
expired, cancelExpired := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
|
||||
defer cancelExpired()
|
||||
h = NewHandler(runnerFunc(func(ctx context.Context, req promptkit.RunRequest) (*promptkit.RunResult, error) {
|
||||
return nil, fmt.Errorf("run failed: %w", ctx.Err())
|
||||
}))
|
||||
w = httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p"}`)).WithContext(expired))
|
||||
assertHTTPErrorCode(t, w, http.StatusGatewayTimeout, "deadline_exceeded")
|
||||
|
||||
h = NewHandler(runnerFunc(func(ctx context.Context, req promptkit.RunRequest) (*promptkit.RunResult, error) {
|
||||
return nil, errors.Join(promptkit.ErrOperationTimeout, context.Canceled)
|
||||
}))
|
||||
w = httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p"}`)).WithContext(canceled))
|
||||
assertHTTPErrorCode(t, w, http.StatusGatewayTimeout, "deadline_exceeded")
|
||||
|
||||
successContext, cancelSuccess := context.WithCancel(context.Background())
|
||||
h = NewHandler(runnerFunc(func(ctx context.Context, req promptkit.RunRequest) (*promptkit.RunResult, error) {
|
||||
cancelSuccess()
|
||||
@@ -446,7 +469,6 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
},
|
||||
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic, IsValid: true},
|
||||
PromptID: "prompt-1",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptHash: "phash",
|
||||
RenderedPromptHash: "rhash",
|
||||
SelectedProfileID: "exec-default",
|
||||
@@ -507,8 +529,10 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if metadata["prompt_id"] != "prompt-1" {
|
||||
t.Fatalf("unexpected metadata.prompt_id: %#v", metadata["prompt_id"])
|
||||
}
|
||||
if metadata["prompt_version"] != "1.0.0" {
|
||||
t.Fatalf("unexpected metadata.prompt_version: %#v", metadata["prompt_version"])
|
||||
for _, field := range []string{"prompt_version", "model_name", "endpoint", "validation_mode", "validation_status", "repair_attempts_used"} {
|
||||
if _, ok := metadata[field]; ok {
|
||||
t.Fatalf("redundant metadata field %q: %#v", field, metadata)
|
||||
}
|
||||
}
|
||||
if metadata["selected_profile_id"] != "exec-default" {
|
||||
t.Fatalf("unexpected metadata.selected_profile_id: %#v", metadata["selected_profile_id"])
|
||||
@@ -516,9 +540,6 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
if metadata["selected_backend_id"] != "local" {
|
||||
t.Fatalf("unexpected metadata.selected_backend_id: %#v", metadata["selected_backend_id"])
|
||||
}
|
||||
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
|
||||
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
|
||||
}
|
||||
usage := metadata["usage"].(map[string]any)
|
||||
if usage["prompt_tokens"] != float64(1) || usage["completion_tokens"] != float64(2) || usage["total_tokens"] != float64(3) {
|
||||
t.Fatalf("unexpected base usage metadata: %#v", usage)
|
||||
@@ -527,8 +548,8 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
t.Fatalf("unexpected cache usage metadata: %#v", usage)
|
||||
}
|
||||
modelParams := metadata["model_params"].(map[string]any)
|
||||
if modelParams["backend_id"] != "local" {
|
||||
t.Fatalf("expected model_params.backend_id=local, got %#v", modelParams["backend_id"])
|
||||
if _, ok := modelParams["backend_id"]; ok || modelParams["model"] != "m1" || modelParams["endpoint"] != "http://llm/v1" {
|
||||
t.Fatalf("unexpected model parameters: %#v", modelParams)
|
||||
}
|
||||
if modelParams["api_key_env"] != envName {
|
||||
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
|
||||
@@ -560,7 +581,7 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||
func TestHandlerAllowsOmittedInputsAndPreservesQualifiedIDs(t *testing.T) {
|
||||
r := &fakeRunner{result: &promptkit.RunResult{
|
||||
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationNone, IsValid: true},
|
||||
@@ -569,8 +590,8 @@ func TestHandlerAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||
h := NewHandler(r)
|
||||
|
||||
for _, body := range []string{
|
||||
`{"prompt_id":"prompt-1","prompt_version":"2"}`,
|
||||
`{"prompt_id":"prompt-1","prompt_version":"2","inputs":{}}`,
|
||||
`{"prompt_id":"support/prompt-1","profile_id":"support/profile-1"}`,
|
||||
`{"prompt_id":"support/prompt-1","profile_id":"support/profile-1","inputs":{}}`,
|
||||
} {
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/runs", bytes.NewBufferString(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -579,8 +600,8 @@ func TestHandlerAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if r.last.PromptVersion != "2" {
|
||||
t.Fatalf("expected prompt version to be mapped, got %q", r.last.PromptVersion)
|
||||
if r.last.PromptID != "support/prompt-1" || r.last.ProfileID != "support/profile-1" {
|
||||
t.Fatalf("qualified IDs changed: %+v", r.last)
|
||||
}
|
||||
if r.last.Inputs != nil {
|
||||
t.Fatalf("expected omitted or empty inputs to remain nil, got %#v", r.last.Inputs)
|
||||
@@ -588,6 +609,16 @@ func TestHandlerAllowsOmittedInputsAndMapsPromptVersion(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerRejectsRemovedPromptVersion(t *testing.T) {
|
||||
runner := &fakeRunner{result: &promptkit.RunResult{}}
|
||||
w := httptest.NewRecorder()
|
||||
NewHandler(runner).ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p","prompt_version":"1"}`)))
|
||||
assertHTTPErrorCode(t, w, http.StatusBadRequest, "invalid_json")
|
||||
if runner.last.PromptID != "" {
|
||||
t.Fatal("runner invoked for removed selector")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerDelegatesDefinitionInputRequirements(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -597,64 +628,64 @@ func TestHandlerDelegatesDefinitionInputRequirements(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
name: "no declared inputs",
|
||||
definition: `id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
definition: `prompts:
|
||||
- id: p
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`,
|
||||
wantStatus: http.StatusOK,
|
||||
},
|
||||
{
|
||||
name: "optional input omitted",
|
||||
definition: `id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
inputs:
|
||||
- name: note
|
||||
required: false
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
definition: `prompts:
|
||||
- id: p
|
||||
default_profile: exec
|
||||
inputs:
|
||||
- name: note
|
||||
required: false
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`,
|
||||
wantStatus: http.StatusOK,
|
||||
},
|
||||
{
|
||||
name: "required input omitted",
|
||||
definition: `id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
inputs:
|
||||
- name: note
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
definition: `prompts:
|
||||
- id: p
|
||||
default_profile: exec
|
||||
inputs:
|
||||
- name: note
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: "hello"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
wantCode: "prompt_render_failed",
|
||||
},
|
||||
{
|
||||
name: "template input omitted",
|
||||
definition: `id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: '{{input "note"}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
definition: `prompts:
|
||||
- id: p
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: '{{input "note"}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
wantCode: "prompt_render_failed",
|
||||
@@ -781,7 +812,6 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
|
||||
r := &fakeRunner{result: &promptkit.RunResult{
|
||||
Artifact: promptkit.Artifact{Body: []byte("ok")},
|
||||
PromptID: "prompt-1",
|
||||
PromptVersion: "1.0.0",
|
||||
SelectedProfileID: "prompt-default",
|
||||
Validation: promptkit.ValidationResult{Status: promptkit.ValidationPassed, Mode: promptkit.ValidationBasic, IsValid: true},
|
||||
EffectiveModelParams: promptkit.ExecutionTarget{Endpoint: "http://llm/v1", Model: "m1"},
|
||||
@@ -986,24 +1016,21 @@ func TestHandlerSessionIDUsesPromptkitResolution(t *testing.T) {
|
||||
profileDir := t.TempDir()
|
||||
sessionLine := ""
|
||||
if tc.definitionSession != "" {
|
||||
sessionLine = "session_id: " + tc.definitionSession + "\n"
|
||||
sessionLine = " session_id: " + tc.definitionSession + "\n"
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte("id: p\nversion: \"1\"\ndefault_profile: exec\n"+sessionLine+`messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte("prompts:\n - id: p\n default_profile: exec\n"+sessionLine+` messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write prompt fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("id: exec\nendpoint: http://127.0.0.1:1/v1\nmodel: test\n"), 0o644); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("profiles:\n - id: exec\n endpoint: http://127.0.0.1:1/v1\n model: test\n"), 0o644); err != nil {
|
||||
t.Fatalf("write profile fixture: %v", err)
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir}, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||
if err != nil {
|
||||
t.Fatalf("new engine: %v", err)
|
||||
}
|
||||
engine := newHandlerEngineFromDirectories(t, promptDir, profileDir, nil, nil, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||
|
||||
body := `{"prompt_id":"p"}`
|
||||
if tc.requestSession != "" {
|
||||
@@ -1035,39 +1062,29 @@ output:
|
||||
func TestHandlerSharesBackendCapacityAcrossConcurrentRequests(t *testing.T) {
|
||||
promptDir := t.TempDir()
|
||||
profileDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`id: p
|
||||
version: "1"
|
||||
default_profile: limited
|
||||
messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`prompts:
|
||||
- id: p
|
||||
default_profile: limited
|
||||
messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write prompt fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: limited
|
||||
backend: limited
|
||||
model: test
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`profiles:
|
||||
- id: limited
|
||||
backend: limited
|
||||
model: test
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write profile fixture: %v", err)
|
||||
}
|
||||
queueCapacity := 0
|
||||
client := &blockingLLMClient{started: make(chan struct{}, 1), release: make(chan struct{})}
|
||||
engine, err := promptkit.NewEngine(
|
||||
promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir},
|
||||
promptkit.WithBackend(promptkit.Backend{
|
||||
ID: "limited",
|
||||
Endpoint: "http://127.0.0.1:1/v1",
|
||||
ConcurrencyLimit: 1,
|
||||
QueueCapacity: &queueCapacity,
|
||||
}),
|
||||
promptkit.WithLLMClient(client),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("new engine: %v", err)
|
||||
}
|
||||
backend := promptkit.BackendConfig{ID: "limited", Endpoint: stringPointer("http://127.0.0.1:1/v1"), ConcurrencyLimit: intPointer(1), QueueCapacity: &queueCapacity}
|
||||
engine := newHandlerEngineFromDirectories(t, promptDir, profileDir, nil, &backend, promptkit.WithLLMClient(client))
|
||||
h := NewHandler(engine)
|
||||
first := httptest.NewRecorder()
|
||||
firstDone := make(chan struct{})
|
||||
@@ -1198,14 +1215,49 @@ func TestHandlerDeadlineCancelsBuiltInRetryWait(t *testing.T) {
|
||||
}))
|
||||
defer provider.Close()
|
||||
h := newRetryHandler(t, provider.URL+"/v1", intPointer(1), 0, nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p","run_timeout_seconds":1}`)))
|
||||
w, err := serveObservedTimeoutRequest(h, `{"prompt_id":"p","run_timeout_seconds":1}`)
|
||||
assertHTTPErrorCode(t, w, http.StatusGatewayTimeout, "deadline_exceeded")
|
||||
if !errors.Is(err, promptkit.ErrOperationTimeout) {
|
||||
t.Fatalf("engine did not attribute retry wait to operation timeout: %v", err)
|
||||
}
|
||||
if atomic.LoadInt32(&hits) != 1 {
|
||||
t.Fatalf("expected one provider call before canceled retry wait, got %d", atomic.LoadInt32(&hits))
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerDeadlineDuringBuiltInGeneration(t *testing.T) {
|
||||
started := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
started <- struct{}{}
|
||||
<-release
|
||||
}))
|
||||
defer provider.Close()
|
||||
defer close(release)
|
||||
h := newRetryHandler(t, provider.URL+"/v1", intPointer(0), 0, nil)
|
||||
w, err := serveObservedTimeoutRequest(h, `{"prompt_id":"p","run_timeout_seconds":1,"include_execution_report":true}`)
|
||||
select {
|
||||
case <-started:
|
||||
default:
|
||||
t.Fatal("provider call did not start")
|
||||
}
|
||||
if !errors.Is(err, promptkit.ErrOperationTimeout) || !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Errorf("engine did not attribute in-flight generation to operation timeout: %v (deadline cause: %t)", err, errors.Is(err, context.DeadlineExceeded))
|
||||
}
|
||||
var response struct {
|
||||
Execution *struct {
|
||||
Failure string `json:"failure"`
|
||||
} `json:"execution"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &response); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if response.Execution == nil || response.Execution.Failure != "deadline" {
|
||||
t.Errorf("terminal report did not classify deadline: %+v", response.Execution)
|
||||
}
|
||||
assertHTTPErrorCode(t, w, http.StatusGatewayTimeout, "deadline_exceeded")
|
||||
}
|
||||
|
||||
func TestHandlerDeadlineCancelsCapacityWaitAndReleasesAdmission(t *testing.T) {
|
||||
started := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
@@ -1228,20 +1280,24 @@ func TestHandlerDeadlineCancelsCapacityWaitAndReleasesAdmission(t *testing.T) {
|
||||
}()
|
||||
queue := 1
|
||||
h := newRetryHandler(t, provider.URL+"/v1", intPointer(0), 1, &queue)
|
||||
firstCtx, cancelFirst := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancelFirst()
|
||||
first := httptest.NewRecorder()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
h.ServeHTTP(first, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p"}`)))
|
||||
h.ServeHTTP(first, httptest.NewRequestWithContext(firstCtx, http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p"}`)))
|
||||
}()
|
||||
select {
|
||||
case <-started:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("first provider call did not start")
|
||||
}
|
||||
waiting := httptest.NewRecorder()
|
||||
h.ServeHTTP(waiting, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p","run_timeout_seconds":1}`)))
|
||||
waiting, err := serveObservedTimeoutRequest(h, `{"prompt_id":"p","run_timeout_seconds":1}`)
|
||||
assertHTTPErrorCode(t, waiting, http.StatusGatewayTimeout, "deadline_exceeded")
|
||||
if !errors.Is(err, promptkit.ErrOperationTimeout) {
|
||||
t.Fatalf("engine did not attribute capacity wait to operation timeout: %v", err)
|
||||
}
|
||||
if atomic.LoadInt32(&hits) != 1 {
|
||||
t.Fatalf("capacity waiter reached provider: hits=%d", atomic.LoadInt32(&hits))
|
||||
}
|
||||
@@ -1254,27 +1310,48 @@ func TestHandlerDeadlineCancelsCapacityWaitAndReleasesAdmission(t *testing.T) {
|
||||
if first.Code != http.StatusOK {
|
||||
t.Fatalf("first request: status=%d body=%s", first.Code, first.Body.String())
|
||||
}
|
||||
third := httptest.NewRecorder()
|
||||
h.ServeHTTP(third, httptest.NewRequest(http.MethodPost, "/v1/runs", strings.NewReader(`{"prompt_id":"p"}`)))
|
||||
third, _ := serveObservedTimeoutRequest(h, `{"prompt_id":"p"}`)
|
||||
if third.Code != http.StatusOK || atomic.LoadInt32(&hits) != 2 {
|
||||
t.Fatalf("admission was not released: status=%d hits=%d body=%s", third.Code, atomic.LoadInt32(&hits), third.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func serveObservedTimeoutRequest(h *Handler, body string) (*httptest.ResponseRecorder, error) {
|
||||
// Bound the test independently of the shorter operation budget under test.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
var runErr error
|
||||
observed := NewHandlerWithOptions(runnerFunc(func(ctx context.Context, request promptkit.RunRequest) (*promptkit.RunResult, error) {
|
||||
result, err := h.runner.Run(ctx, request)
|
||||
runErr = err
|
||||
return result, err
|
||||
}), h.options)
|
||||
w := httptest.NewRecorder()
|
||||
observed.ServeHTTP(w, httptest.NewRequestWithContext(ctx, http.MethodPost, "/v1/runs", strings.NewReader(body)))
|
||||
return w, runErr
|
||||
}
|
||||
|
||||
func newRetryHandler(t *testing.T, endpoint string, retries *int, concurrency int, queue *int) *Handler {
|
||||
t.Helper()
|
||||
promptDir := t.TempDir()
|
||||
profileDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte("id: p\nversion: \"1\"\ndefault_profile: limited\nmessages:\n - role: user\n content: hi\noutput:\n format: text\n validation_mode: none\n"), 0o644); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`prompts:
|
||||
- id: p
|
||||
default_profile: limited
|
||||
messages:
|
||||
- role: user
|
||||
content: hi
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("id: limited\nbackend: limited\nmodel: test\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir, TransientRetries: retries}, promptkit.WithBackend(promptkit.Backend{ID: "limited", Endpoint: endpoint, ConcurrencyLimit: concurrency, QueueCapacity: queue}))
|
||||
if err != nil {
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("profiles:\n - id: limited\n backend: limited\n model: test\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
backend := promptkit.BackendConfig{ID: "limited", Endpoint: &endpoint, ConcurrencyLimit: &concurrency, QueueCapacity: queue}
|
||||
engine := newHandlerEngineFromDirectories(t, promptDir, profileDir, retries, &backend)
|
||||
return NewHandler(engine)
|
||||
}
|
||||
|
||||
@@ -1626,6 +1703,7 @@ func TestHandlerPublicErrorMapping(t *testing.T) {
|
||||
{name: "file denied", err: ErrFileNotAllowed, status: http.StatusBadRequest, code: "artifact_not_allowed", message: "file input artifact is not allowed"},
|
||||
{name: "file outside root", err: ErrFileOutsideRoot, status: http.StatusBadRequest, code: "artifact_not_allowed", message: "file input artifact is not allowed"},
|
||||
{name: "file too large", err: ErrFileTooLarge, status: http.StatusRequestEntityTooLarge, code: "artifact_too_large", message: "file input artifact is too large"},
|
||||
{name: "file too large with broad category", err: errors.Join(ErrFileTooLarge, promptkit.ErrInvalidRequest, promptkit.ErrArtifactLoad), status: http.StatusRequestEntityTooLarge, code: "artifact_too_large", message: "file input artifact is too large"},
|
||||
{name: "artifact", err: wrap(promptkit.ErrArtifactLoad, fmt.Errorf("read failed")), status: http.StatusBadRequest, code: "artifact_read_failed", message: "failed to read input artifact", avoidCause: "read failed"},
|
||||
{name: "prompt render", err: wrap(promptkit.ErrPromptRender, fmt.Errorf("render failed")), status: http.StatusBadRequest, code: "prompt_render_failed", message: "failed to render prompt", avoidCause: "render failed"},
|
||||
{name: "capacity", err: &promptkit.CapacityError{BackendID: "private-backend"}, status: http.StatusServiceUnavailable, code: "capacity_exceeded", message: "model backend capacity is exhausted", avoidCause: "private-backend"},
|
||||
@@ -1770,34 +1848,28 @@ func newHandlerEngineWithOptions(t *testing.T, options ...promptkit.Option) *pro
|
||||
|
||||
promptDir := t.TempDir()
|
||||
profileDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`id: p
|
||||
version: "1"
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
repair_attempts: 0
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`prompts:
|
||||
- id: p
|
||||
default_profile: exec
|
||||
messages:
|
||||
- role: user
|
||||
content: "hi"
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
repair_attempts: 0
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write prompt fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: exec
|
||||
endpoint: http://example.invalid/v1
|
||||
model: model
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`profiles:
|
||||
- id: exec
|
||||
endpoint: http://example.invalid/v1
|
||||
model: model
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write profile fixture: %v", err)
|
||||
}
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: promptDir,
|
||||
ProfileDir: profileDir,
|
||||
}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("construct public engine: %v", err)
|
||||
}
|
||||
return engine
|
||||
return newHandlerEngineFromDirectories(t, promptDir, profileDir, nil, nil, options...)
|
||||
}
|
||||
|
||||
func newDefinitionHandler(t *testing.T, definition string) *Handler {
|
||||
@@ -1808,21 +1880,46 @@ func newDefinitionHandler(t *testing.T, definition string) *Handler {
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(definition), 0o644); err != nil {
|
||||
t.Fatalf("write prompt fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`id: exec
|
||||
endpoint: http://example.invalid/v1
|
||||
model: model
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte(`profiles:
|
||||
- id: exec
|
||||
endpoint: http://example.invalid/v1
|
||||
model: model
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write profile fixture: %v", err)
|
||||
}
|
||||
|
||||
engine := newHandlerEngineFromDirectories(t, promptDir, profileDir, nil, nil, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||
return NewHandler(engine)
|
||||
}
|
||||
|
||||
func newHandlerEngineFromDirectories(t *testing.T, promptDir, profileDir string, retries *int, backend *promptkit.BackendConfig, options ...promptkit.Option) *promptkit.Engine {
|
||||
t.Helper()
|
||||
|
||||
promptInputs, err := promptkit.DirectoryInputs(promptDir)
|
||||
if err != nil {
|
||||
t.Fatalf("load prompt inputs: %v", err)
|
||||
}
|
||||
profileInputs, err := promptkit.DirectoryInputs(profileDir)
|
||||
if err != nil {
|
||||
t.Fatalf("load profile inputs: %v", err)
|
||||
}
|
||||
underlays := []configkit.Underlay{{Inputs: promptInputs}, {Inputs: profileInputs}}
|
||||
if backend != nil {
|
||||
underlays = append(underlays, configkit.Underlay{Inputs: []configkit.Input{{Source: promptkit.BackendSource("test backend", []promptkit.BackendConfig{*backend})}}})
|
||||
}
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
PromptDir: promptDir,
|
||||
ProfileDir: profileDir,
|
||||
}, promptkit.WithLLMClient(handlerLLMClient{}))
|
||||
Sources: promptkit.ConfigurationSources{Underlays: underlays},
|
||||
TransientRetries: retries,
|
||||
}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("construct public engine: %v", err)
|
||||
}
|
||||
return NewHandler(engine)
|
||||
return engine
|
||||
}
|
||||
|
||||
func stringPointer(value string) *string {
|
||||
return &value
|
||||
}
|
||||
|
||||
func assertHTTPErrorCode(t *testing.T, w *httptest.ResponseRecorder, status int, code string) {
|
||||
|
||||
@@ -4,28 +4,24 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"gitea.maximumdirect.net/eric/scriptorium/internal/defaults"
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
// Document is Scriptorium's ConfigKit-native application configuration shape.
|
||||
type Document struct {
|
||||
PromptDir string `yaml:"prompt_dir"`
|
||||
ProfileDir string `yaml:"profile_dir"`
|
||||
SchemaDir string `yaml:"schema_dir"`
|
||||
TransientRetries *int `yaml:"transient_retries"`
|
||||
Server ServerDocument `yaml:"server"`
|
||||
Defaults DefaultsDocument `yaml:"defaults"`
|
||||
Backends map[string]BackendDocument `yaml:"backends"`
|
||||
PromptKitSources []SourceSelection `yaml:"promptkit_sources"`
|
||||
SchemaDir string `yaml:"schema_dir"`
|
||||
TransientRetries *int `yaml:"transient_retries"`
|
||||
Server ServerDocument `yaml:"server"`
|
||||
Defaults DefaultsDocument `yaml:"defaults"`
|
||||
}
|
||||
|
||||
type BackendDocument struct {
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
APIKeyEnv string `yaml:"api_key_env"`
|
||||
ExtraParams map[string]any `yaml:"extra_params"`
|
||||
ConcurrencyLimit int `yaml:"concurrency_limit"`
|
||||
QueueCapacity *int `yaml:"queue_capacity"`
|
||||
// SourceSelection identifies one PromptKit configuration file or directory.
|
||||
type SourceSelection struct {
|
||||
File string `yaml:"file"`
|
||||
Directory string `yaml:"directory"`
|
||||
}
|
||||
|
||||
type ServerDocument struct {
|
||||
@@ -77,8 +73,11 @@ func resolveDocument(document *Document, resolution *configkit.Resolution) error
|
||||
}
|
||||
}
|
||||
|
||||
normalizePath(configkit.Keys("prompt_dir"), &document.PromptDir)
|
||||
normalizePath(configkit.Keys("profile_dir"), &document.ProfileDir)
|
||||
for i := range document.PromptKitSources {
|
||||
path := configkit.Keys("promptkit_sources").Index(i)
|
||||
normalizePath(path.Key("file"), &document.PromptKitSources[i].File)
|
||||
normalizePath(path.Key("directory"), &document.PromptKitSources[i].Directory)
|
||||
}
|
||||
normalizePath(configkit.Keys("schema_dir"), &document.SchemaDir)
|
||||
normalizePath(configkit.Keys("server", "artifact_root"), &document.Server.ArtifactRoot)
|
||||
normalizeString(configkit.Keys("server", "addr"), &document.Server.Addr,
|
||||
@@ -89,6 +88,13 @@ func resolveDocument(document *Document, resolution *configkit.Resolution) error
|
||||
}
|
||||
|
||||
func validateDocument(document *Document, validation *configkit.Validation) {
|
||||
for i, source := range document.PromptKitSources {
|
||||
hasFile := source.File != ""
|
||||
hasDirectory := source.Directory != ""
|
||||
if hasFile == hasDirectory {
|
||||
validation.Error(configkit.Keys("promptkit_sources").Index(i), "exactly one of file or directory must be non-empty")
|
||||
}
|
||||
}
|
||||
if document.TransientRetries != nil && *document.TransientRetries < 0 {
|
||||
validation.Error(configkit.Keys("transient_retries"), "must be greater than or equal to zero")
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
func loadDocumentBytes(content string) (*configkit.Result[Document], error) {
|
||||
@@ -25,8 +25,9 @@ func TestDocumentDefaultsAndExplicitValues(t *testing.T) {
|
||||
t.Fatalf("unexpected defaults-only result: %+v sources=%+v", defaultsOnly.Config, defaultsOnly.Sources())
|
||||
}
|
||||
|
||||
result, err := loadDocumentBytes(`prompt_dir: ""
|
||||
profile_dir: ""
|
||||
result, err := loadDocumentBytes(`promptkit_sources:
|
||||
- file: " ./bundle/../framework.yml "
|
||||
- directory: "./profiles"
|
||||
schema_dir: ""
|
||||
transient_retries: 0
|
||||
server:
|
||||
@@ -37,12 +38,6 @@ server:
|
||||
max_response_bytes: 0
|
||||
defaults:
|
||||
render_format: " JSON "
|
||||
backends:
|
||||
local:
|
||||
endpoint: "http://fixture.invalid/v1"
|
||||
queue_capacity: 0
|
||||
extra_params:
|
||||
sample_count: 23
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("explicit-value load: %v", err)
|
||||
@@ -54,30 +49,70 @@ backends:
|
||||
if config.TransientRetries == nil || *config.TransientRetries != 0 {
|
||||
t.Fatalf("explicit zero retry budget lost: %+v", config.TransientRetries)
|
||||
}
|
||||
if backend := config.Backends["local"]; backend.QueueCapacity == nil || *backend.QueueCapacity != 0 || backend.ExtraParams["sample_count"] != int64(23) {
|
||||
t.Fatalf("backend presence or dynamic integer changed: %+v", backend)
|
||||
if want := []SourceSelection{{File: "framework.yml"}, {Directory: "profiles"}}; !reflect.DeepEqual(config.PromptKitSources, want) {
|
||||
t.Fatalf("source selectors did not decode: %+v", config.PromptKitSources)
|
||||
}
|
||||
if config.Defaults.RenderFormat != "json" {
|
||||
t.Fatalf("render format normalization failed: %q", config.Defaults.RenderFormat)
|
||||
}
|
||||
|
||||
nullable, err := loadDocumentBytes("transient_retries: null\nbackends:\n local:\n queue_capacity: null\n")
|
||||
nullable, err := loadDocumentBytes("transient_retries: null\npromptkit_sources: null\n")
|
||||
if err != nil {
|
||||
t.Fatalf("nullable-value load: %v", err)
|
||||
}
|
||||
if nullable.Config.TransientRetries != nil || nullable.Config.Backends["local"].QueueCapacity != nil {
|
||||
if nullable.Config.TransientRetries != nil || nullable.Config.PromptKitSources != nil {
|
||||
t.Fatalf("nullable fields did not remain nil: %+v", nullable.Config)
|
||||
}
|
||||
}
|
||||
|
||||
nullBackends, err := loadDocumentBytes("backends: null\n")
|
||||
if err != nil || nullBackends.Config.Backends != nil {
|
||||
t.Fatalf("nullable backend map did not remain nil: %+v error=%v", nullBackends, err)
|
||||
func TestDocumentSourceSelectionsDoNotAcquirePaths(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
content string
|
||||
want []SourceSelection
|
||||
}{
|
||||
{name: "omitted", content: "{}\n"},
|
||||
{name: "null", content: "promptkit_sources: null\n"},
|
||||
{name: "empty", content: "promptkit_sources: []\n"},
|
||||
{name: "nonexistent file", content: "promptkit_sources:\n - file: ./missing/framework.yml\n", want: []SourceSelection{{File: "missing/framework.yml"}}},
|
||||
{name: "nonexistent directory", content: "promptkit_sources:\n - directory: ./missing/framework\n", want: []SourceSelection{{Directory: "missing/framework"}}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
result, err := loadDocumentBytes(tc.content)
|
||||
if err != nil {
|
||||
t.Fatalf("load source selectors: %v", err)
|
||||
}
|
||||
if len(result.Config.PromptKitSources) != len(tc.want) || len(tc.want) > 0 && !reflect.DeepEqual(result.Config.PromptKitSources, tc.want) {
|
||||
t.Fatalf("source selectors changed: got=%+v want=%+v", result.Config.PromptKitSources, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingsCopiesPromptKitSourcesInOrder(t *testing.T) {
|
||||
document := Document{PromptKitSources: []SourceSelection{
|
||||
{File: "./first.yml"},
|
||||
{Directory: "./second"},
|
||||
}}
|
||||
settings := Settings(document)
|
||||
if !reflect.DeepEqual(settings.PromptKitSources, document.PromptKitSources) {
|
||||
t.Fatalf("source ordering changed: got=%+v want=%+v", settings.PromptKitSources, document.PromptKitSources)
|
||||
}
|
||||
document.PromptKitSources[0].File = "changed.yml"
|
||||
if settings.PromptKitSources[0].File != "./first.yml" {
|
||||
t.Fatalf("settings retained the document's source slice: %+v", settings.PromptKitSources)
|
||||
}
|
||||
if empty := Settings(Document{PromptKitSources: []SourceSelection{}}).PromptKitSources; empty == nil {
|
||||
t.Fatal("explicit empty source list was not copied")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDocumentRecordsEffectiveNormalization(t *testing.T) {
|
||||
result, err := loadDocumentBytes(`prompt_dir: " ./one/../two "
|
||||
profile_dir: " "
|
||||
result, err := loadDocumentBytes(`promptkit_sources:
|
||||
- file: " ./one/../two "
|
||||
- file: "./blank.yml"
|
||||
directory: " "
|
||||
- directory: "./three/../three"
|
||||
server:
|
||||
artifact_root: " ./data/../artifacts "
|
||||
addr: " :9000 "
|
||||
@@ -87,19 +122,22 @@ defaults:
|
||||
if err != nil {
|
||||
t.Fatalf("load normalized document: %v", err)
|
||||
}
|
||||
if result.Config.PromptDir != "two" || result.Config.ProfileDir != "" || result.Config.Server.ArtifactRoot != "artifacts" || result.Config.Server.Addr != ":9000" {
|
||||
if want := []SourceSelection{{File: "two"}, {File: "blank.yml"}, {Directory: "three"}}; !reflect.DeepEqual(result.Config.PromptKitSources, want) {
|
||||
t.Fatalf("unexpected normalized source selections: %+v", result.Config.PromptKitSources)
|
||||
}
|
||||
if result.Config.Server.ArtifactRoot != "artifacts" || result.Config.Server.Addr != ":9000" {
|
||||
t.Fatalf("unexpected normalized values: %+v", result.Config)
|
||||
}
|
||||
var effective bytes.Buffer
|
||||
if err := result.WriteEffective(&effective); err != nil {
|
||||
t.Fatalf("write effective configuration: %v", err)
|
||||
}
|
||||
for _, fragment := range []string{"prompt_dir: two", "profile_dir: ''", "artifact_root: artifacts", "addr: :9000", "render_format: json"} {
|
||||
for _, fragment := range []string{"file: two", "directory: ''", "directory: three", "artifact_root: artifacts", "addr: :9000", "render_format: json"} {
|
||||
if !strings.Contains(effective.String(), fragment) {
|
||||
t.Fatalf("effective output omitted %q: %s", fragment, effective.String())
|
||||
}
|
||||
}
|
||||
explanation, known := result.Explain(configkit.Keys("prompt_dir"))
|
||||
explanation, known := result.Explain(configkit.Keys("promptkit_sources").Index(0).Key("file"))
|
||||
if !known || explanation.Effective.YAML != "two\n" {
|
||||
t.Fatalf("explanation does not agree with effective value: %+v known=%t", explanation, known)
|
||||
}
|
||||
@@ -112,10 +150,26 @@ defaults:
|
||||
if !found {
|
||||
t.Fatalf("normalization was not recorded: %+v", explanation.History)
|
||||
}
|
||||
|
||||
unchanged, err := loadDocumentBytes("promptkit_sources:\n - file: framework.yml\n")
|
||||
if err != nil {
|
||||
t.Fatalf("load unchanged source path: %v", err)
|
||||
}
|
||||
unchangedExplanation, known := unchanged.Explain(configkit.Keys("promptkit_sources").Index(0).Key("file"))
|
||||
if !known {
|
||||
t.Fatal("unchanged source path was not inspectable")
|
||||
}
|
||||
for _, change := range unchangedExplanation.History {
|
||||
if change.Origin.Operation == configkit.ChangeNormalization {
|
||||
t.Fatalf("unchanged source path recorded a normalization: %+v", unchangedExplanation.History)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDocumentValidationIsSafeAndPathSpecific(t *testing.T) {
|
||||
for _, tc := range []struct{ name, content, path string }{
|
||||
{"empty source selector", "promptkit_sources:\n - file: ''\n", "/promptkit_sources/0"},
|
||||
{"multiple source selectors", "promptkit_sources:\n - file: secret-path\n directory: private-path\n", "/promptkit_sources/0"},
|
||||
{"negative retry", "transient_retries: -9173\n", "/transient_retries"},
|
||||
{"negative request limit", "server:\n max_request_bytes: -9173\n", "/server/max_request_bytes"},
|
||||
{"negative artifact limit", "server:\n max_artifact_bytes: -9173\n", "/server/max_artifact_bytes"},
|
||||
@@ -134,7 +188,7 @@ func TestDocumentValidationIsSafeAndPathSpecific(t *testing.T) {
|
||||
if diagnostic.Code == configkit.CodeValidateError && diagnostic.Path.String() == tc.path {
|
||||
found = true
|
||||
}
|
||||
if strings.Contains(diagnostic.Message, "9173") || strings.Contains(diagnostic.Message, "private-invalid-value") {
|
||||
if strings.Contains(diagnostic.Message, "9173") || strings.Contains(diagnostic.Message, "private-invalid-value") || strings.Contains(diagnostic.Message, "secret-path") || strings.Contains(diagnostic.Message, "private-path") {
|
||||
t.Fatalf("diagnostic disclosed rejected value: %+v", diagnostic)
|
||||
}
|
||||
}
|
||||
@@ -153,7 +207,13 @@ func TestDocumentRejectsNativeShapeErrors(t *testing.T) {
|
||||
"server:\n max_request_bytes: null\n",
|
||||
"defaults:\n render_format: null\n",
|
||||
"server:\n unknown_field: 1\n",
|
||||
"backends:\n local:\n unknown_field: 1\n",
|
||||
"promptkit_sources:\n - file: ./selected.yml\n unknown_field: 1\n",
|
||||
"promptkit_sources:\n - file: ./selected.yml\n directory: ./also-selected\n",
|
||||
"promptkit_sources:\n - file: null\n",
|
||||
"promptkit_sources: ./not-a-sequence.yml\n",
|
||||
"prompt_dir: ./old-prompts\n",
|
||||
"profile_dir: ./old-profiles\n",
|
||||
"backends:\n local:\n endpoint: http://example.invalid/v1\n",
|
||||
} {
|
||||
_, err := loadDocumentBytes(content)
|
||||
if err == nil {
|
||||
|
||||
@@ -1,13 +1,10 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
renderformat "gitea.maximumdirect.net/eric/scriptorium/internal/format"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
// LoadOptions selects one application source and explicit configuration values.
|
||||
@@ -21,35 +18,15 @@ type LoadOptions struct {
|
||||
// Load resolves one ConfigKit document and retains its inspection snapshot.
|
||||
func Load(options LoadOptions) (*configkit.Result[Document], error) {
|
||||
var source configkit.Source
|
||||
fileBacked := false
|
||||
if options.Explicit {
|
||||
source = configkit.File(options.Path)
|
||||
fileBacked = true
|
||||
} else {
|
||||
selected := ""
|
||||
for _, candidate := range defaultConfigSearchPaths {
|
||||
if _, err := os.Stat(candidate); err == nil {
|
||||
selected = candidate
|
||||
break
|
||||
} else if !errors.Is(err, os.ErrNotExist) {
|
||||
reason := "filesystem error"
|
||||
if errors.Is(err, os.ErrPermission) {
|
||||
reason = "permission denied"
|
||||
}
|
||||
return nil, fmt.Errorf("cannot inspect configuration path %q: %s", candidate, reason)
|
||||
}
|
||||
}
|
||||
if selected == "" {
|
||||
source = configkit.NoSource()
|
||||
} else {
|
||||
source = configkit.File(selected)
|
||||
fileBacked = true
|
||||
}
|
||||
source = configkit.SearchConfig("scriptorium", configkit.SearchPaths(defaultConfigSearchPaths...))
|
||||
}
|
||||
|
||||
result, err := configkit.Load(source, configkit.Options[Document]{
|
||||
Definition: documentDefinition(),
|
||||
Imports: fileBacked,
|
||||
Imports: true,
|
||||
Overlay: options.Overlay,
|
||||
Overrides: options.Overrides,
|
||||
})
|
||||
@@ -62,8 +39,6 @@ func Load(options LoadOptions) (*configkit.Result[Document], error) {
|
||||
// Settings converts a validated document without another merge or normalization.
|
||||
func Settings(document Document) AppSettings {
|
||||
settings := AppSettings{
|
||||
PromptDir: document.PromptDir,
|
||||
ProfileDir: document.ProfileDir,
|
||||
SchemaDir: document.SchemaDir,
|
||||
ServerAddr: document.Server.Addr,
|
||||
ArtifactRoot: document.Server.ArtifactRoot,
|
||||
@@ -73,24 +48,8 @@ func Settings(document Document) AppSettings {
|
||||
DefaultRenderFormat: renderformat.PreparedRunOutputFormat(document.Defaults.RenderFormat),
|
||||
TransientRetries: document.TransientRetries,
|
||||
}
|
||||
if document.Backends != nil {
|
||||
ids := make([]string, 0, len(document.Backends))
|
||||
for id := range document.Backends {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
settings.Backends = make([]BackendSettings, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
backend := document.Backends[id]
|
||||
settings.Backends = append(settings.Backends, BackendSettings{
|
||||
ID: id,
|
||||
Endpoint: backend.Endpoint,
|
||||
APIKeyEnv: backend.APIKeyEnv,
|
||||
ExtraParams: backend.ExtraParams,
|
||||
ConcurrencyLimit: backend.ConcurrencyLimit,
|
||||
QueueCapacity: backend.QueueCapacity,
|
||||
})
|
||||
}
|
||||
if document.PromptKitSources != nil {
|
||||
settings.PromptKitSources = append([]SourceSelection{}, document.PromptKitSources...)
|
||||
}
|
||||
return settings
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
configkit "gitea.maximumdirect.net/eric/ConfigKit"
|
||||
"go.maximumdirect.net/configkit"
|
||||
)
|
||||
|
||||
func useConfigSearchPaths(t *testing.T, paths ...string) {
|
||||
@@ -29,19 +29,19 @@ func TestLoadSelectsOrderedSourceAndKeepsItsSnapshot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
first := filepath.Join(dir, "first.yml")
|
||||
second := filepath.Join(dir, "second.yml")
|
||||
writeApplicationFile(t, first, "prompt_dir: ./first\n")
|
||||
writeApplicationFile(t, second, "prompt_dir: ./second\n")
|
||||
writeApplicationFile(t, first, "promptkit_sources:\n - file: ./first.yml\n")
|
||||
writeApplicationFile(t, second, "promptkit_sources:\n - directory: ./second\n")
|
||||
useConfigSearchPaths(t, first, second)
|
||||
|
||||
result, err := Load(LoadOptions{Path: filepath.Join(dir, "ignored.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("load first implicit source: %v", err)
|
||||
}
|
||||
if result.Config.PromptDir != "first" || len(result.Sources()) != 1 || result.Sources()[0].Path != first {
|
||||
if !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "first.yml"}}) || len(result.Sources()) != 1 || result.Sources()[0].Path != first {
|
||||
t.Fatalf("unexpected implicit source: config=%+v sources=%+v", result.Config, result.Sources())
|
||||
}
|
||||
writeApplicationFile(t, first, "prompt_dir: ./changed\n")
|
||||
if result.Config.PromptDir != "first" {
|
||||
writeApplicationFile(t, first, "promptkit_sources:\n - file: ./changed.yml\n")
|
||||
if !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "first.yml"}}) {
|
||||
t.Fatalf("result did not retain resolved snapshot: %+v", result.Config)
|
||||
}
|
||||
|
||||
@@ -49,7 +49,7 @@ func TestLoadSelectsOrderedSourceAndKeepsItsSnapshot(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err = Load(LoadOptions{})
|
||||
if err != nil || result.Config.PromptDir != "second" || result.Sources()[0].Path != second {
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{Directory: "second"}}) || result.Sources()[0].Path != second {
|
||||
t.Fatalf("second implicit source not selected: result=%+v err=%v", result, err)
|
||||
}
|
||||
|
||||
@@ -113,7 +113,7 @@ func TestLoadDefaultsOnlyAndOverridesBeforeValidation(t *testing.T) {
|
||||
if err != nil || result.Config.Defaults.RenderFormat != "json" {
|
||||
t.Fatalf("override did not precede validation: result=%+v err=%v", result, err)
|
||||
}
|
||||
if len(result.Sources()) != 2 || result.Sources()[0].Layer != configkit.LayerBase || result.Sources()[1].Layer != configkit.LayerCLI {
|
||||
if len(result.Sources()) != 2 || result.Sources()[0].Layer != configkit.LayerMain || result.Sources()[1].Layer != configkit.LayerCLI {
|
||||
t.Fatalf("source order changed: %+v", result.Sources())
|
||||
}
|
||||
writeApplicationFile(t, path, "defaults: [\n")
|
||||
@@ -124,8 +124,11 @@ func TestLoadDefaultsOnlyAndOverridesBeforeValidation(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingsCopiesResolvedValuesAndSortsBackends(t *testing.T) {
|
||||
func TestSettingsCopiesResolvedValuesAndSourceOrder(t *testing.T) {
|
||||
result, err := loadDocumentBytes(`schema_dir: ""
|
||||
promptkit_sources:
|
||||
- directory: ./profiles
|
||||
- file: ./framework.yml
|
||||
transient_retries: 0
|
||||
server:
|
||||
addr: ""
|
||||
@@ -133,13 +136,6 @@ server:
|
||||
max_request_bytes: 0
|
||||
defaults:
|
||||
render_format: json
|
||||
backends:
|
||||
zed:
|
||||
endpoint: http://zed.invalid/v1
|
||||
extra_params:
|
||||
count: 4
|
||||
alpha:
|
||||
queue_capacity: 0
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -151,31 +147,49 @@ backends:
|
||||
if settings.TransientRetries == nil || *settings.TransientRetries != 0 {
|
||||
t.Fatalf("retry presence lost: %+v", settings.TransientRetries)
|
||||
}
|
||||
if len(settings.Backends) != 2 || settings.Backends[0].ID != "alpha" || settings.Backends[1].ID != "zed" {
|
||||
t.Fatalf("backend order changed: %+v", settings.Backends)
|
||||
}
|
||||
if settings.Backends[0].QueueCapacity == nil || *settings.Backends[0].QueueCapacity != 0 || settings.Backends[1].QueueCapacity != nil {
|
||||
t.Fatalf("queue presence changed: %+v", settings.Backends)
|
||||
}
|
||||
if value := settings.Backends[1].ExtraParams["count"]; value != int64(4) {
|
||||
t.Fatalf("dynamic number changed shape: %T %v", value, value)
|
||||
if want := []SourceSelection{{Directory: "profiles"}, {File: "framework.yml"}}; !reflect.DeepEqual(settings.PromptKitSources, want) {
|
||||
t.Fatalf("source ordering changed: got=%+v want=%+v", settings.PromptKitSources, want)
|
||||
}
|
||||
|
||||
if got := Settings(documentDefaults()); got.TransientRetries != nil || got.Backends != nil {
|
||||
if got := Settings(documentDefaults()); got.TransientRetries != nil || got.PromptKitSources != nil {
|
||||
t.Fatalf("default nullable values changed: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadDiscoveryErrorQuotesPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
loop := filepath.Join(dir, "loop-\n.yml")
|
||||
if err := os.Symlink(loop, loop); err != nil {
|
||||
t.Skipf("symlink unavailable: %v", err)
|
||||
path := filepath.Join(dir, "directory-\n.yml")
|
||||
if err := os.Mkdir(path, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
useConfigSearchPaths(t, loop)
|
||||
useConfigSearchPaths(t, path)
|
||||
_, err := Load(LoadOptions{})
|
||||
if err == nil || !strings.Contains(err.Error(), `\n`) || strings.Contains(err.Error(), "loop-\n.yml") {
|
||||
t.Fatalf("discovery error did not quote path safely: %v", err)
|
||||
var loadError *configkit.LoadError
|
||||
if !errors.As(err, &loadError) {
|
||||
t.Fatalf("expected structured error: %v", err)
|
||||
}
|
||||
var diagnostics strings.Builder
|
||||
if err := configkit.WriteDiagnostics(&diagnostics, loadError.Diagnostics()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(diagnostics.String(), string(configkit.CodeSourceNotRegular)) || strings.Contains(diagnostics.String(), "directory-\n.yml") {
|
||||
t.Fatalf("discovery diagnostics exposed an unescaped path: %s", diagnostics.String())
|
||||
}
|
||||
|
||||
// A decoding diagnostic can contain an authored configuration path even
|
||||
// though the nonregular-file diagnostic omits its filesystem path.
|
||||
file := filepath.Join(dir, "invalid.yml")
|
||||
writeApplicationFile(t, file, "server:\n \"bad\\nkey\": value\n")
|
||||
_, err = Load(LoadOptions{Path: file, Explicit: true})
|
||||
if !errors.As(err, &loadError) {
|
||||
t.Fatalf("expected decoding diagnostic: %v", err)
|
||||
}
|
||||
diagnostics.Reset()
|
||||
if err := configkit.WriteDiagnostics(&diagnostics, loadError.Diagnostics()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(diagnostics.String(), `\n`) || strings.Contains(diagnostics.String(), "bad\nkey") {
|
||||
t.Fatalf("diagnostic path was not escaped: %q", diagnostics.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -188,25 +202,29 @@ func TestLoadComposesExplicitImportsAndSelectedOverlay(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := filepath.Join(dir, "config.yml")
|
||||
writeApplicationFile(t, base, "configkit:\n imports:\n - glob: fragments/*.yml\nprompt_dir: ./base-prompts\nserver:\n addr: :8081\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "a.yml"), "backends:\n alpha:\n endpoint: http://alpha.invalid/v1\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "b.yml"), "backends:\n beta:\n endpoint: http://beta.invalid/v1\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "overlays", "local.yml"), "prompt_dir: ./overlay-prompts\nserver:\n addr: :9090\n")
|
||||
writeApplicationFile(t, base, "config:\n imports:\n - glob: fragments/*.yml\n required: true\npromptkit_sources:\n - file: ./base-framework.yml\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "a.yml"), "server:\n addr: :8081\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "b.yml"), "defaults:\n render_format: json\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "overlays", "local.yml"), "promptkit_sources:\n - directory: ./overlay-framework\nserver:\n addr: :9090\n")
|
||||
|
||||
result, err := Load(LoadOptions{Path: base, Explicit: true})
|
||||
if err != nil || result.Config.PromptDir != "base-prompts" || len(result.Config.Backends) != 2 || len(result.Sources()) != 3 {
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "base-framework.yml"}}) || result.Config.Server.Addr != ":8081" || result.Config.Defaults.RenderFormat != "json" || len(result.Sources()) != 3 {
|
||||
t.Fatalf("disjoint imports did not compose: result=%+v err=%v", result, err)
|
||||
}
|
||||
result, err = Load(LoadOptions{Path: base, Explicit: true, Overlay: "local", Overrides: []configkit.Override{{
|
||||
Path: configkit.Keys("server", "addr"), Value: ":7070", Flag: "--addr",
|
||||
Path: configkit.Keys("promptkit_sources"), Value: []any{map[string]any{"file": "./cli-framework.yml"}}, Flag: "--promptkit-source",
|
||||
}}})
|
||||
if err != nil || result.Config.PromptDir != "overlay-prompts" || result.Config.Server.Addr != ":7070" || len(result.Sources()) != 5 {
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "cli-framework.yml"}}) || result.Config.Server.Addr != ":9090" || len(result.Sources()) != 5 {
|
||||
t.Fatalf("overlay or explicit override changed: result=%+v err=%v", result, err)
|
||||
}
|
||||
result, err = Load(LoadOptions{Path: base, Explicit: true, Overlay: "local"})
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{Directory: "overlay-framework"}}) {
|
||||
t.Fatalf("overlay did not replace the source list: result=%+v err=%v", result, err)
|
||||
}
|
||||
if _, err := Load(LoadOptions{Path: base, Explicit: true, Overlay: "missing"}); err == nil {
|
||||
t.Fatal("missing selected overlay accepted")
|
||||
}
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "b.yml"), "backends:\n alpha:\n endpoint: http://conflict.invalid/v1\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "b.yml"), "server:\n addr: :8082\n")
|
||||
if _, err := Load(LoadOptions{Path: base, Explicit: true}); err == nil {
|
||||
t.Fatal("conflicting import accepted")
|
||||
} else {
|
||||
@@ -217,17 +235,103 @@ func TestLoadComposesExplicitImportsAndSelectedOverlay(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsOverlayWithoutBase(t *testing.T) {
|
||||
func TestLoadOverlayWithoutMainContext(t *testing.T) {
|
||||
useConfigSearchPaths(t, filepath.Join(t.TempDir(), "absent.yml"))
|
||||
if _, err := Load(LoadOptions{Overlay: "local"}); err == nil {
|
||||
t.Fatal("overlay without base accepted")
|
||||
t.Fatal("overlay without a main context accepted")
|
||||
} else {
|
||||
var loadError *configkit.LoadError
|
||||
if !errors.As(err, &loadError) {
|
||||
t.Fatalf("no-base overlay lost structured diagnostic: %v", err)
|
||||
t.Fatalf("no-context overlay lost structured diagnostic: %v", err)
|
||||
}
|
||||
}
|
||||
if _, err := Load(LoadOptions{Overlay: "default"}); err != nil {
|
||||
t.Fatalf("default must select no overlay: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadDoesNotFallThroughSelectedInvalidSource(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
first, second := filepath.Join(dir, "first.yml"), filepath.Join(dir, "second.yml")
|
||||
writeApplicationFile(t, first, "server: [\n")
|
||||
writeApplicationFile(t, second, "server:\n addr: ':9090'\n")
|
||||
useConfigSearchPaths(t, first, second)
|
||||
if _, err := Load(LoadOptions{}); err == nil {
|
||||
t.Fatal("malformed selected file fell through to the second candidate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadImportRequirednessAndDestination(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
main := filepath.Join(dir, "config.yml")
|
||||
for _, tc := range []struct {
|
||||
name, declaration string
|
||||
wantError bool
|
||||
}{
|
||||
{"optional by default", "- path: absent.yml", false},
|
||||
{"required", "- path: absent.yml\n required: true", true},
|
||||
{"removed optional field", "- path: absent.yml\n optional: true", true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
writeApplicationFile(t, main, "config:\n imports:\n "+tc.declaration+"\n")
|
||||
_, err := Load(LoadOptions{Path: main, Explicit: true})
|
||||
if (err != nil) != tc.wantError {
|
||||
t.Fatalf("load error = %v, want error %t", err, tc.wantError)
|
||||
}
|
||||
})
|
||||
}
|
||||
fragments := filepath.Join(dir, "fragments")
|
||||
if err := os.Mkdir(fragments, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeApplicationFile(t, filepath.Join(fragments, "server.yml"), "addr: ':8181'\n")
|
||||
writeApplicationFile(t, main, "config:\n imports:\n - dir: fragments\n pattern: '*.yml'\n destination: [server]\n required: true\n")
|
||||
result, err := Load(LoadOptions{Path: main, Explicit: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
info, ok := result.Metadata().Lookup(configkit.Keys("server", "addr"))
|
||||
if result.Config.Server.Addr != ":8181" || !ok || len(info.Origin.Locations) == 0 {
|
||||
t.Fatalf("mounted value or origin missing: config=%+v origin=%+v", result.Config.Server, info)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadAuthoredSelectorCollection(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, subdir := range []string{"fragments", "overlays"} {
|
||||
if err := os.Mkdir(filepath.Join(dir, subdir), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
main := filepath.Join(dir, "config.yml")
|
||||
writeApplicationFile(t, main, "config:\n sequences:\n - destination: [promptkit_sources]\n key: file\n mode: upsert\n imports:\n - dir: fragments\n pattern: '*.yml'\n required: true\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "fragments", "a.yml"), "promptkit_sources:\n - file: a.yml\n")
|
||||
peer := filepath.Join(dir, "fragments", "b.yml")
|
||||
writeApplicationFile(t, peer, "promptkit_sources:\n - file: b.yml\n")
|
||||
writeApplicationFile(t, filepath.Join(dir, "overlays", "local.yml"), "promptkit_sources:\n - file: c.yml\n")
|
||||
result, err := Load(LoadOptions{Path: main, Explicit: true, Overlay: "local"})
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "a.yml"}, {File: "b.yml"}, {File: "c.yml"}}) {
|
||||
t.Fatalf("authored upsert did not retain selectors: result=%+v err=%v", result, err)
|
||||
}
|
||||
result, err = Load(LoadOptions{Path: main, Explicit: true, Overlay: "local", Overrides: []configkit.Override{{
|
||||
Path: configkit.Keys("promptkit_sources"), Value: []any{map[string]any{"file": "cli.yml"}}, Flag: "--promptkit-source",
|
||||
}}})
|
||||
if err != nil || !reflect.DeepEqual(result.Config.PromptKitSources, []SourceSelection{{File: "cli.yml"}}) {
|
||||
t.Fatalf("CLI list did not replace selectors: result=%+v err=%v", result, err)
|
||||
}
|
||||
writeApplicationFile(t, peer, "promptkit_sources:\n - file: a.yml\n")
|
||||
if _, err := Load(LoadOptions{Path: main, Explicit: true}); err == nil {
|
||||
t.Fatal("duplicate authored peer identity accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadOverrideCannotRepairDecodeError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
writeApplicationFile(t, path, "server:\n max_request_bytes: not-a-number\n")
|
||||
_, err := Load(LoadOptions{Path: path, Explicit: true, Overrides: []configkit.Override{{
|
||||
Path: configkit.Keys("server", "max_request_bytes"), Value: int64(12), Flag: "--max-request-bytes",
|
||||
}}})
|
||||
if err == nil {
|
||||
t.Fatal("override repaired an undecodable source value")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,8 +14,7 @@ var defaultConfigSearchPaths = []string{
|
||||
|
||||
// AppSettings holds the validated application values used by executable adapters.
|
||||
type AppSettings struct {
|
||||
PromptDir string
|
||||
ProfileDir string
|
||||
PromptKitSources []SourceSelection
|
||||
SchemaDir string
|
||||
ServerAddr string
|
||||
ArtifactRoot string
|
||||
@@ -23,15 +22,5 @@ type AppSettings struct {
|
||||
MaxArtifactBytes int64
|
||||
MaxResponseBytes int64
|
||||
DefaultRenderFormat renderformat.PreparedRunOutputFormat
|
||||
Backends []BackendSettings
|
||||
TransientRetries *int
|
||||
}
|
||||
|
||||
type BackendSettings struct {
|
||||
ID string
|
||||
Endpoint string
|
||||
APIKeyEnv string
|
||||
ExtraParams map[string]any
|
||||
ConcurrencyLimit int
|
||||
QueueCapacity *int
|
||||
}
|
||||
|
||||
@@ -7,50 +7,49 @@ import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
type PromptInspection struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version"`
|
||||
PromptHash string `json:"prompt_hash"`
|
||||
DefaultProfileID string `json:"default_profile_id"`
|
||||
Inputs []PromptInspectionInput `json:"inputs"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptHash string `json:"prompt_hash"`
|
||||
DefaultProfileID *string `json:"default_profile_id,omitempty"`
|
||||
Inputs *[]PromptInspectionInput `json:"inputs,omitempty"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
}
|
||||
|
||||
type PromptInspectionInput struct {
|
||||
Name string `json:"name"`
|
||||
Required bool `json:"required"`
|
||||
ContentType string `json:"content_type"`
|
||||
Description string `json:"description"`
|
||||
Name string `json:"name"`
|
||||
Required *bool `json:"required,omitempty"`
|
||||
ContentType *string `json:"content_type,omitempty"`
|
||||
Description *string `json:"description,omitempty"`
|
||||
}
|
||||
|
||||
type OutputContract struct {
|
||||
Format string `json:"format"`
|
||||
ValidationMode string `json:"validation_mode"`
|
||||
SchemaPath string `json:"schema_path"`
|
||||
RepairAttempts int `json:"repair_attempts"`
|
||||
Format string `json:"format"`
|
||||
ValidationMode string `json:"validation_mode"`
|
||||
SchemaPath *string `json:"schema_path,omitempty"`
|
||||
RepairAttempts *int `json:"repair_attempts,omitempty"`
|
||||
}
|
||||
|
||||
type ProfileInspection struct {
|
||||
ProfileID string `json:"profile_id"`
|
||||
EffectiveModelParams ProfileModelParams `json:"effective_model_params"`
|
||||
APIKeyRequired bool `json:"api_key_required"`
|
||||
ProfileID string `json:"profile_id"`
|
||||
ProfileModelParams ProfileModelParams `json:"profile_model_params"`
|
||||
APIKeyRequired *bool `json:"api_key_required,omitempty"`
|
||||
}
|
||||
|
||||
type ProfileModelParams struct {
|
||||
BackendID string `json:"backend_id"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Temperature float64 `json:"temperature"`
|
||||
MaxTokens int `json:"max_tokens"`
|
||||
TopP float64 `json:"top_p"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
ServiceTier string `json:"service_tier"`
|
||||
ReasoningEffort string `json:"reasoning_effort"`
|
||||
APIKeyEnv string `json:"api_key_env"`
|
||||
ExtraParams map[string]any `json:"extra_params"`
|
||||
BackendID *string `json:"backend_id,omitempty"`
|
||||
Endpoint *string `json:"endpoint,omitempty"`
|
||||
Model *string `json:"model,omitempty"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
MaxTokens *int `json:"max_tokens,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
TimeoutSeconds *int `json:"timeout_seconds,omitempty"`
|
||||
ServiceTier *string `json:"service_tier,omitempty"`
|
||||
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
|
||||
APIKeyEnv *string `json:"api_key_env,omitempty"`
|
||||
ExtraParams *map[string]any `json:"extra_params,omitempty"`
|
||||
}
|
||||
|
||||
func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat OutputFormat) ([]byte, error) {
|
||||
@@ -58,12 +57,15 @@ func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat Outp
|
||||
return nil, errors.New("prompt inspection is nil")
|
||||
}
|
||||
dto := PromptInspection{
|
||||
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
|
||||
Inputs: make([]PromptInspectionInput, len(value.Inputs)),
|
||||
PromptID: value.PromptID, PromptHash: value.PromptHash, DefaultProfileID: value.DefaultProfileID,
|
||||
OutputContract: OutputContract{Format: string(value.OutputContract.Format), ValidationMode: string(value.OutputContract.ValidationMode), SchemaPath: value.OutputContract.SchemaPath, RepairAttempts: value.OutputContract.RepairAttempts},
|
||||
}
|
||||
for i, input := range value.Inputs {
|
||||
dto.Inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
|
||||
if value.Inputs != nil {
|
||||
inputs := make([]PromptInspectionInput, len(*value.Inputs))
|
||||
for i, input := range *value.Inputs {
|
||||
inputs[i] = PromptInspectionInput{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}
|
||||
}
|
||||
dto.Inputs = &inputs
|
||||
}
|
||||
switch outputFormat {
|
||||
case OutputFormatJSON:
|
||||
@@ -74,16 +76,36 @@ func FormatPromptInspection(value *promptkit.PromptInspection, outputFormat Outp
|
||||
return append(data, '\n'), nil
|
||||
case OutputFormatText:
|
||||
var b bytes.Buffer
|
||||
fmt.Fprintf(&b, "prompt_id: %s\nprompt_version: %s\nprompt_hash: %s\ndefault_profile_id: %s\ninputs:", dto.PromptID, dto.PromptVersion, dto.PromptHash, dto.DefaultProfileID)
|
||||
if len(dto.Inputs) == 0 {
|
||||
fmt.Fprintln(&b, " []")
|
||||
} else {
|
||||
fmt.Fprintln(&b)
|
||||
for _, input := range dto.Inputs {
|
||||
fmt.Fprintf(&b, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", input.Name, input.Required, input.ContentType, strconv.Quote(input.Description))
|
||||
fmt.Fprintf(&b, "prompt_id: %s\nprompt_hash: %s\n", dto.PromptID, dto.PromptHash)
|
||||
if dto.DefaultProfileID != nil {
|
||||
fmt.Fprintf(&b, "default_profile_id: %s\n", *dto.DefaultProfileID)
|
||||
}
|
||||
if dto.Inputs != nil {
|
||||
if len(*dto.Inputs) == 0 {
|
||||
fmt.Fprintln(&b, "inputs: []")
|
||||
} else {
|
||||
fmt.Fprintln(&b, "inputs:")
|
||||
for _, input := range *dto.Inputs {
|
||||
fmt.Fprintf(&b, " - name: %s\n", input.Name)
|
||||
if input.Required != nil {
|
||||
fmt.Fprintf(&b, " required: %t\n", *input.Required)
|
||||
}
|
||||
if input.ContentType != nil {
|
||||
fmt.Fprintf(&b, " content_type: %s\n", *input.ContentType)
|
||||
}
|
||||
if input.Description != nil {
|
||||
fmt.Fprintf(&b, " description: %s\n", strconv.Quote(*input.Description))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode, dto.OutputContract.SchemaPath, dto.OutputContract.RepairAttempts)
|
||||
fmt.Fprintf(&b, "output_contract:\n format: %s\n validation_mode: %s\n", dto.OutputContract.Format, dto.OutputContract.ValidationMode)
|
||||
if dto.OutputContract.SchemaPath != nil {
|
||||
fmt.Fprintf(&b, " schema_path: %s\n", *dto.OutputContract.SchemaPath)
|
||||
}
|
||||
if dto.OutputContract.RepairAttempts != nil {
|
||||
fmt.Fprintf(&b, " repair_attempts: %d\n", *dto.OutputContract.RepairAttempts)
|
||||
}
|
||||
return b.Bytes(), nil
|
||||
default:
|
||||
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
||||
@@ -94,12 +116,11 @@ func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat Ou
|
||||
if value == nil {
|
||||
return nil, errors.New("profile inspection is nil")
|
||||
}
|
||||
target := value.EffectiveModelParams
|
||||
params := map[string]any{}
|
||||
for key, item := range target.ExtraParams {
|
||||
params[key] = item
|
||||
}
|
||||
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, EffectiveModelParams: ProfileModelParams{BackendID: target.BackendID, Endpoint: target.Endpoint, Model: target.Model, Temperature: target.Temperature, MaxTokens: target.MaxTokens, TopP: target.TopP, TimeoutSeconds: target.TimeoutSeconds, ServiceTier: target.ServiceTier, ReasoningEffort: target.ReasoningEffort, APIKeyEnv: target.APIKeyEnv, ExtraParams: params}}
|
||||
dto := ProfileInspection{ProfileID: value.ProfileID, APIKeyRequired: value.APIKeyRequired, ProfileModelParams: ProfileModelParams{
|
||||
BackendID: value.Backend, Endpoint: value.Endpoint, Model: value.Model, Temperature: value.Temperature,
|
||||
MaxTokens: value.MaxTokens, TopP: value.TopP, TimeoutSeconds: value.TimeoutSeconds, ServiceTier: value.ServiceTier,
|
||||
ReasoningEffort: value.ReasoningEffort, APIKeyEnv: value.APIKeyEnv, ExtraParams: value.ExtraParams,
|
||||
}}
|
||||
switch outputFormat {
|
||||
case OutputFormatJSON:
|
||||
data, err := json.MarshalIndent(dto, "", " ")
|
||||
@@ -109,12 +130,48 @@ func FormatProfileInspection(value *promptkit.ProfileInspection, outputFormat Ou
|
||||
return append(data, '\n'), nil
|
||||
case OutputFormatText:
|
||||
var b bytes.Buffer
|
||||
fmt.Fprintf(&b, "profile_id: %s\neffective_model_params:\n backend_id: %s\n endpoint: %s\n model: %s\n temperature: %g\n max_tokens: %d\n top_p: %g\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n extra_params: ", dto.ProfileID, dto.EffectiveModelParams.BackendID, dto.EffectiveModelParams.Endpoint, dto.EffectiveModelParams.Model, dto.EffectiveModelParams.Temperature, dto.EffectiveModelParams.MaxTokens, dto.EffectiveModelParams.TopP, dto.EffectiveModelParams.TimeoutSeconds, dto.EffectiveModelParams.ServiceTier, dto.EffectiveModelParams.ReasoningEffort, dto.EffectiveModelParams.APIKeyEnv)
|
||||
paramsJSON, err := json.Marshal(dto.EffectiveModelParams.ExtraParams)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
fmt.Fprintf(&b, "profile_id: %s\nprofile_model_params:\n", dto.ProfileID)
|
||||
p := dto.ProfileModelParams
|
||||
if p.BackendID != nil {
|
||||
fmt.Fprintf(&b, " backend_id: %s\n", *p.BackendID)
|
||||
}
|
||||
if p.Endpoint != nil {
|
||||
fmt.Fprintf(&b, " endpoint: %s\n", *p.Endpoint)
|
||||
}
|
||||
if p.Model != nil {
|
||||
fmt.Fprintf(&b, " model: %s\n", *p.Model)
|
||||
}
|
||||
if p.Temperature != nil {
|
||||
fmt.Fprintf(&b, " temperature: %g\n", *p.Temperature)
|
||||
}
|
||||
if p.MaxTokens != nil {
|
||||
fmt.Fprintf(&b, " max_tokens: %d\n", *p.MaxTokens)
|
||||
}
|
||||
if p.TopP != nil {
|
||||
fmt.Fprintf(&b, " top_p: %g\n", *p.TopP)
|
||||
}
|
||||
if p.TimeoutSeconds != nil {
|
||||
fmt.Fprintf(&b, " timeout_seconds: %d\n", *p.TimeoutSeconds)
|
||||
}
|
||||
if p.ServiceTier != nil {
|
||||
fmt.Fprintf(&b, " service_tier: %s\n", *p.ServiceTier)
|
||||
}
|
||||
if p.ReasoningEffort != nil {
|
||||
fmt.Fprintf(&b, " reasoning_effort: %s\n", *p.ReasoningEffort)
|
||||
}
|
||||
if p.APIKeyEnv != nil {
|
||||
fmt.Fprintf(&b, " api_key_env: %s\n", *p.APIKeyEnv)
|
||||
}
|
||||
if p.ExtraParams != nil {
|
||||
paramsJSON, err := json.Marshal(p.ExtraParams)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fmt.Fprintf(&b, " extra_params: %s\n", paramsJSON)
|
||||
}
|
||||
if dto.APIKeyRequired != nil {
|
||||
fmt.Fprintf(&b, "api_key_required: %t\n", *dto.APIKeyRequired)
|
||||
}
|
||||
fmt.Fprintf(&b, "%s\napi_key_required: %t\n", paramsJSON, dto.APIKeyRequired)
|
||||
return b.Bytes(), nil
|
||||
default:
|
||||
return nil, fmt.Errorf("%w: %q", ErrUnknownPreparedRunFormat, outputFormat)
|
||||
|
||||
@@ -5,120 +5,91 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestFormatPromptInspectionPreservesDeclaredOrderAndEmptyValues(t *testing.T) {
|
||||
value := &promptkit.PromptInspection{PromptID: "p", PromptVersion: "1", PromptHash: "hash", Inputs: []promptkit.PromptInputDefinition{{Name: "first", Required: true, ContentType: "text/plain", Description: "first input"}, {Name: "second"}}, OutputContract: promptkit.OutputContract{Format: "text", ValidationMode: promptkit.ValidationNone}}
|
||||
text, err := FormatPromptInspection(value, OutputFormatText)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(text), "default_profile_id: \ninputs:\n - name: first") || strings.Index(string(text), "name: first") > strings.Index(string(text), "name: second") {
|
||||
t.Fatalf("unexpected text inspection: %s", text)
|
||||
}
|
||||
jsonOutput, err := FormatPromptInspection(value, OutputFormatJSON)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(string(jsonOutput), "\n") || !strings.Contains(string(jsonOutput), `"default_profile_id": ""`) {
|
||||
t.Fatalf("unexpected json inspection: %s", jsonOutput)
|
||||
}
|
||||
}
|
||||
func ptr[T any](value T) *T { return &value }
|
||||
|
||||
func TestFormatPromptInspectionRejectsNil(t *testing.T) {
|
||||
func TestPromptInspectionOptionalFieldsAndDisclosure(t *testing.T) {
|
||||
inputs := []promptkit.PromptInputInspection{{Name: "first", Required: ptr(false), ContentType: ptr(""), Description: ptr("")}, {Name: "second"}}
|
||||
value := &promptkit.PromptInspection{PromptID: "support/p", PromptHash: "hash", DefaultProfileID: ptr(""), Inputs: &inputs,
|
||||
Messages: []promptkit.PromptMessageInspection{{Role: "user", Content: "private message"}}, Description: ptr("private description"), SessionID: ptr("private session"),
|
||||
OutputContract: promptkit.PromptOutputInspection{Format: "text", ValidationMode: promptkit.ValidationNone, SchemaPath: ptr(""), RepairAttempts: ptr(0)}}
|
||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
||||
output, err := FormatPromptInspection(value, format)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := string(output)
|
||||
for _, forbidden := range []string{"private message", "private description", "private session", "prompt_version"} {
|
||||
if strings.Contains(s, forbidden) {
|
||||
t.Fatalf("inspection exposed %q: %s", forbidden, s)
|
||||
}
|
||||
}
|
||||
if strings.Index(s, "first") > strings.Index(s, "second") {
|
||||
t.Fatalf("input order changed: %s", s)
|
||||
}
|
||||
if !strings.Contains(s, "default_profile_id") || !strings.Contains(s, "repair_attempts") || !strings.Contains(s, "required") {
|
||||
t.Fatalf("present empty values missing: %s", s)
|
||||
}
|
||||
}
|
||||
value.DefaultProfileID, value.Inputs = nil, nil
|
||||
value.OutputContract.SchemaPath, value.OutputContract.RepairAttempts = nil, nil
|
||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
||||
output, err := FormatPromptInspection(value, format)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, absent := range []string{"default_profile_id", "inputs", "schema_path", "repair_attempts"} {
|
||||
if strings.Contains(string(output), absent) {
|
||||
t.Fatalf("absent %s rendered: %s", absent, output)
|
||||
}
|
||||
}
|
||||
}
|
||||
empty := []promptkit.PromptInputInspection{}
|
||||
value.Inputs = &empty
|
||||
output, err := FormatPromptInspection(value, OutputFormatJSON)
|
||||
if err != nil || !strings.Contains(string(output), `"inputs": []`) {
|
||||
t.Fatalf("empty inputs lost: %s %v", output, err)
|
||||
}
|
||||
if _, err := FormatPromptInspection(nil, OutputFormatText); err == nil {
|
||||
t.Fatal("expected nil inspection error")
|
||||
t.Fatal("nil prompt accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatProfileInspectionPreservesSafeEffectiveValues(t *testing.T) {
|
||||
const secret = "sentinel-secret-must-not-appear"
|
||||
t.Setenv("FIXTURE_PROFILE_API_KEY", secret)
|
||||
|
||||
value := &promptkit.ProfileInspection{
|
||||
ProfileID: "custom-derived",
|
||||
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||
BackendID: "fixture-custom",
|
||||
Endpoint: "http://127.0.0.1:11434/v1",
|
||||
Model: "fixture-model",
|
||||
Temperature: 0.25,
|
||||
MaxTokens: 640,
|
||||
TopP: 0.9,
|
||||
TimeoutSeconds: 45,
|
||||
ServiceTier: "flex",
|
||||
ReasoningEffort: "high",
|
||||
APIKeyEnv: "FIXTURE_PROFILE_API_KEY",
|
||||
ExtraParams: map[string]any{
|
||||
"zeta": true,
|
||||
"alpha": map[string]any{"nested": []any{"first", 2.0}},
|
||||
},
|
||||
},
|
||||
APIKeyRequired: true,
|
||||
func TestProfileInspectionPreservesOnlyPresentValues(t *testing.T) {
|
||||
params := map[string]any{}
|
||||
value := &promptkit.ProfileInspection{ProfileID: "support/fast", Backend: ptr("backend"), Model: ptr(""), Temperature: ptr(0.0), MaxTokens: ptr(0), APIKeyRequired: ptr(false), ExtraParams: ¶ms}
|
||||
for _, format := range []OutputFormat{OutputFormatText, OutputFormatJSON} {
|
||||
output, err := FormatProfileInspection(value, format)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := string(output)
|
||||
for _, key := range []string{"profile_model_params", "backend_id", "model", "temperature", "max_tokens", "api_key_required", "extra_params"} {
|
||||
if !strings.Contains(s, key) {
|
||||
t.Fatalf("present %s missing: %s", key, s)
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"effective_model_params", "endpoint", "api_key_env", "top_p"} {
|
||||
if strings.Contains(s, key) {
|
||||
t.Fatalf("absent %s appeared: %s", key, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
jsonOutput, err := FormatProfileInspection(value, OutputFormatJSON)
|
||||
output, err := FormatProfileInspection(value, OutputFormatJSON)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secondJSONOutput, err := FormatProfileInspection(value, OutputFormatJSON)
|
||||
if err != nil {
|
||||
var decoded map[string]any
|
||||
if err := json.Unmarshal(output, &decoded); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(jsonOutput) != string(secondJSONOutput) || !strings.HasSuffix(string(jsonOutput), "\n") {
|
||||
t.Fatalf("expected deterministic newline-terminated JSON, got %q", jsonOutput)
|
||||
}
|
||||
if strings.Contains(string(jsonOutput), secret) {
|
||||
t.Fatalf("inspection exposed an environment secret: %s", jsonOutput)
|
||||
}
|
||||
|
||||
var decoded ProfileInspection
|
||||
if err := json.Unmarshal(jsonOutput, &decoded); err != nil {
|
||||
t.Fatalf("decode profile inspection: %v", err)
|
||||
}
|
||||
if decoded.ProfileID != "custom-derived" || decoded.EffectiveModelParams.BackendID != "fixture-custom" {
|
||||
t.Fatalf("unexpected profile identity: %+v", decoded)
|
||||
}
|
||||
if decoded.EffectiveModelParams.APIKeyEnv != "FIXTURE_PROFILE_API_KEY" || !decoded.APIKeyRequired {
|
||||
t.Fatalf("unexpected credential metadata: %+v", decoded)
|
||||
}
|
||||
alpha, ok := decoded.EffectiveModelParams.ExtraParams["alpha"].(map[string]any)
|
||||
if !ok || len(alpha["nested"].([]any)) != 2 {
|
||||
t.Fatalf("nested extra parameters were not preserved: %#v", decoded.EffectiveModelParams.ExtraParams)
|
||||
}
|
||||
|
||||
textOutput, err := FormatProfileInspection(value, OutputFormatText)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
textValue := string(textOutput)
|
||||
if !strings.Contains(textValue, "backend_id: fixture-custom") ||
|
||||
!strings.Contains(textValue, "api_key_env: FIXTURE_PROFILE_API_KEY") ||
|
||||
!strings.Contains(textValue, `extra_params: {"alpha":{"nested":["first",2]},"zeta":true}`) ||
|
||||
!strings.Contains(textValue, "api_key_required: true") {
|
||||
t.Fatalf("unexpected text inspection: %s", textValue)
|
||||
}
|
||||
if strings.Contains(textValue, secret) {
|
||||
t.Fatalf("text inspection exposed an environment secret: %s", textValue)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatProfileInspectionPreservesEmptyBackendAndRejectsNil(t *testing.T) {
|
||||
output, err := FormatProfileInspection(&promptkit.ProfileInspection{
|
||||
ProfileID: "endpoint-only",
|
||||
EffectiveModelParams: promptkit.ExecutionTarget{
|
||||
Endpoint: "http://127.0.0.1:8000/v1",
|
||||
Model: "fixture-model",
|
||||
},
|
||||
}, OutputFormatJSON)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(output), `"backend_id": ""`) ||
|
||||
!strings.Contains(string(output), `"extra_params": {}`) {
|
||||
t.Fatalf("expected explicit empty backend and object extra params, got %s", output)
|
||||
p := decoded["profile_model_params"].(map[string]any)
|
||||
if p["backend_id"] != "backend" || p["model"] != "" || p["temperature"] != float64(0) || decoded["api_key_required"] != false || len(p["extra_params"].(map[string]any)) != 0 {
|
||||
t.Fatalf("wrong values: %#v", decoded)
|
||||
}
|
||||
if _, err := FormatProfileInspection(nil, OutputFormatText); err == nil {
|
||||
t.Fatal("expected nil profile inspection error")
|
||||
t.Fatal("nil profile accepted")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
var ErrUnknownPreparedRunFormat = errors.New("unknown prepared run format")
|
||||
@@ -106,7 +106,6 @@ func (textPreparedRunFormatter) Format(prepared *promptkit.PreparedRun) ([]byte,
|
||||
|
||||
var b bytes.Buffer
|
||||
fmt.Fprintf(&b, "prompt: %s\n", prepared.PromptID)
|
||||
fmt.Fprintf(&b, "prompt_version: %s\n", prepared.PromptVersion)
|
||||
fmt.Fprintf(&b, "selected_profile_id: %s\n", prepared.SelectedProfileID)
|
||||
if prepared.SelectedBackendID != "" {
|
||||
fmt.Fprintf(&b, "selected_backend_id: %s\n", prepared.SelectedBackendID)
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
@@ -17,10 +17,12 @@ func TestTextFormatterIncludesPreparedRunDetails(t *testing.T) {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
s := string(out)
|
||||
if strings.Contains(s, "prompt_version") {
|
||||
t.Fatalf("unexpected prompt version in text output: %s", s)
|
||||
}
|
||||
|
||||
for _, want := range []string{
|
||||
"prompt: prompt.id",
|
||||
"prompt_version: v1",
|
||||
"selected_profile_id: local-fast",
|
||||
"selected_backend_id: local",
|
||||
"endpoint: http://llm/v1",
|
||||
@@ -194,8 +196,8 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
|
||||
if decoded["prompt_id"] != "prompt.id" {
|
||||
t.Fatalf("expected prompt_id in json output, got %#v", decoded["prompt_id"])
|
||||
}
|
||||
if decoded["prompt_version"] != "v1" {
|
||||
t.Fatalf("expected prompt_version in json output, got %#v", decoded["prompt_version"])
|
||||
if _, ok := decoded["prompt_version"]; ok {
|
||||
t.Fatalf("unexpected prompt_version in json output: %#v", decoded)
|
||||
}
|
||||
if decoded["selected_profile_id"] != "local-fast" {
|
||||
t.Fatalf("expected selected_profile_id in json output, got %#v", decoded["selected_profile_id"])
|
||||
@@ -344,7 +346,6 @@ func TestFormatPreparedRunByNameUnknownFailsClearly(t *testing.T) {
|
||||
func samplePreparedRun() *promptkit.PreparedRun {
|
||||
return &promptkit.PreparedRun{
|
||||
PromptID: "prompt.id",
|
||||
PromptVersion: "v1",
|
||||
PromptHash: "prompt-hash",
|
||||
SelectedProfileID: "local-fast",
|
||||
SelectedBackendID: "local",
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
package report
|
||||
|
||||
import "gitea.maximumdirect.net/eric/promptkit"
|
||||
import "go.maximumdirect.net/promptkit"
|
||||
|
||||
// Execution is the terminal observation exposed by Scriptorium's opt-in adapters.
|
||||
type Execution struct {
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
BackendID string `json:"backend_id,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
@@ -53,7 +52,7 @@ func (collector *Collector) Observe(event promptkit.ExecutionEvent) {
|
||||
return TokenCount{Value: value.Value, Known: value.Known, Complete: value.Complete}
|
||||
}
|
||||
collector.report = &Execution{
|
||||
RunID: event.RunID, PromptID: event.PromptID, PromptVersion: event.PromptVersion,
|
||||
RunID: event.RunID, PromptID: event.PromptID,
|
||||
ProfileID: event.ProfileID, BackendID: event.BackendID, Model: event.Model,
|
||||
Outcome: outcome(event.Outcome), Failure: failure(event.Failure),
|
||||
StatusCode: event.StatusCode, RetriesExhausted: event.RetriesExhausted,
|
||||
|
||||
@@ -9,7 +9,8 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
"go.maximumdirect.net/configkit"
|
||||
"go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func TestCollectorProjectsOnlyTerminalAggregates(t *testing.T) {
|
||||
@@ -23,7 +24,7 @@ func TestCollectorProjectsOnlyTerminalAggregates(t *testing.T) {
|
||||
}
|
||||
collector.Observe(promptkit.ExecutionEvent{
|
||||
Kind: promptkit.ExecutionFinished, RunID: "run-1", PromptID: "prompt-1",
|
||||
PromptVersion: "1", ProfileID: "profile-1", BackendID: "backend-1", Model: "model-1",
|
||||
ProfileID: "profile-1", BackendID: "backend-1", Model: "model-1",
|
||||
Outcome: promptkit.ExecutionOutcomeError, Failure: promptkit.ExecutionFailureProviderStatus,
|
||||
StatusCode: 503, RetriesExhausted: true, Attempts: 3, TransientRetries: 2, RepairCalls: 1,
|
||||
Duration: 1500 * time.Millisecond, CapacityWait: 20 * time.Millisecond,
|
||||
@@ -53,7 +54,7 @@ func TestCollectorProjectsOnlyTerminalAggregates(t *testing.T) {
|
||||
if err := json.Unmarshal(encoded, &fields); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range []string{"run_id", "prompt_id", "prompt_version", "profile_id", "backend_id", "model", "outcome", "failure", "status_code", "retries_exhausted", "attempts", "transient_retries", "repair_calls", "duration_ms", "capacity_wait_ms", "generation_duration_ms", "retry_delay_ms", "usage"} {
|
||||
for _, key := range []string{"run_id", "prompt_id", "profile_id", "backend_id", "model", "outcome", "failure", "status_code", "retries_exhausted", "attempts", "transient_retries", "repair_calls", "duration_ms", "capacity_wait_ms", "generation_duration_ms", "retry_delay_ms", "usage"} {
|
||||
if _, ok := fields[key]; !ok {
|
||||
t.Fatalf("report missing %q: %s", key, encoded)
|
||||
}
|
||||
@@ -64,7 +65,7 @@ func TestCollectorProjectsOnlyTerminalAggregates(t *testing.T) {
|
||||
t.Fatalf("incomplete token count %q: %#v", key, item)
|
||||
}
|
||||
}
|
||||
for _, forbidden := range []string{"session_id", "headers", "endpoint", "api_key", "error", "output", "input"} {
|
||||
for _, forbidden := range []string{"prompt_version", "session_id", "headers", "endpoint", "api_key", "error", "output", "input"} {
|
||||
if _, present := fields[forbidden]; present {
|
||||
t.Fatalf("report exposed %q: %s", forbidden, encoded)
|
||||
}
|
||||
@@ -105,13 +106,32 @@ func (localClient) Generate(ctx context.Context, req promptkit.GenerateRequest)
|
||||
func TestCollectorReceivesRealEngineTerminalEvent(t *testing.T) {
|
||||
promptDir := t.TempDir()
|
||||
profileDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte("id: p\nversion: \"1\"\ndefault_profile: local\nmessages:\n - role: user\n content: hi\noutput:\n format: text\n validation_mode: none\n"), 0o644); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(promptDir, "prompt.yaml"), []byte(`prompts:
|
||||
- id: p
|
||||
default_profile: local
|
||||
messages:
|
||||
- role: user
|
||||
content: hi
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("id: local\nendpoint: http://127.0.0.1:1/v1\nmodel: local-model\n"), 0o644); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(profileDir, "profile.yaml"), []byte("profiles:\n - id: local\n endpoint: http://127.0.0.1:1/v1\n model: local-model\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{PromptDir: promptDir, ProfileDir: profileDir}, promptkit.WithLLMClient(localClient{}))
|
||||
promptInputs, err := promptkit.DirectoryInputs(promptDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profileInputs, err := promptkit.DirectoryInputs(profileDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{
|
||||
Sources: promptkit.ConfigurationSources{Underlays: []configkit.Underlay{{Inputs: promptInputs}, {Inputs: profileInputs}}},
|
||||
}, promptkit.WithLLMClient(localClient{}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
15
testdata/promptkit-v0.9/config.yml
vendored
15
testdata/promptkit-v0.9/config.yml
vendored
@@ -1,12 +1,5 @@
|
||||
prompt_dir: ./testdata/promptkit-v0.9/prompts
|
||||
profile_dir: ./testdata/promptkit-v0.9/profiles
|
||||
promptkit_sources:
|
||||
- directory: ./testdata/promptkit-v0.9/prompts
|
||||
- directory: ./testdata/promptkit-v0.9/profiles
|
||||
- file: ./testdata/promptkit-v0.9/framework/backends.yml
|
||||
schema_dir: ./testdata/promptkit-v0.9/schemas
|
||||
backends:
|
||||
fixture-custom:
|
||||
endpoint: http://127.0.0.1:11434/v1
|
||||
api_key_env: FIXTURE_BACKEND_API_KEY
|
||||
extra_params:
|
||||
backend_option:
|
||||
enabled: true
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 0
|
||||
|
||||
9
testdata/promptkit-v0.9/framework/backends.yml
vendored
Normal file
9
testdata/promptkit-v0.9/framework/backends.yml
vendored
Normal file
@@ -0,0 +1,9 @@
|
||||
backends:
|
||||
- id: fixture-custom
|
||||
endpoint: http://127.0.0.1:11434/v1
|
||||
api_key_env: FIXTURE_BACKEND_API_KEY
|
||||
extra_params:
|
||||
backend_option:
|
||||
enabled: true
|
||||
concurrency_limit: 2
|
||||
queue_capacity: 0
|
||||
@@ -1,16 +1,17 @@
|
||||
id: custom-base
|
||||
backend: fixture-custom
|
||||
model: fixture-base-model
|
||||
temperature: 0.25
|
||||
max_tokens: 640
|
||||
top_p: 0.9
|
||||
timeout_seconds: 45
|
||||
service_tier: flex
|
||||
reasoning_effort: medium
|
||||
api_key_env: FIXTURE_PROFILE_API_KEY
|
||||
extra_params:
|
||||
routing:
|
||||
order:
|
||||
- primary
|
||||
- fallback
|
||||
allow_fallback: true
|
||||
profiles:
|
||||
- id: custom-base
|
||||
backend: fixture-custom
|
||||
model: fixture-base-model
|
||||
temperature: 0.25
|
||||
max_tokens: 640
|
||||
top_p: 0.9
|
||||
timeout_seconds: 45
|
||||
service_tier: flex
|
||||
reasoning_effort: medium
|
||||
api_key_env: FIXTURE_PROFILE_API_KEY
|
||||
extra_params:
|
||||
routing:
|
||||
order:
|
||||
- primary
|
||||
- fallback
|
||||
allow_fallback: true
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user