Retire the legacy CLI and document native operator workflows
This commit is contained in:
@@ -1,243 +1,120 @@
|
||||
# Operations Guide
|
||||
|
||||
## Scope And References
|
||||
This runbook covers deployment, normal operation, capacity, sensitive data, and
|
||||
recovery. Exact [CLI syntax/statuses](cli.md), [configuration](config.md), and
|
||||
[HTTP wire behavior](api.md) remain in their canonical references.
|
||||
See the [CLI migration guide](consumers/upgrading-cli.md) for changed jobs and
|
||||
the [framework upgrade guide](consumers/upgrading-promptkit.md) for definitions.
|
||||
|
||||
This runbook covers deployment, normal operation, capacity planning, and safe
|
||||
recovery for Scriptorium. It does not redefine invocation syntax, configuration
|
||||
fields, or HTTP wire behavior.
|
||||
## Deploy Sources And Process
|
||||
|
||||
- [CLI reference](cli.md): commands, output destinations, and exit codes.
|
||||
- [Configuration reference](config.md): application settings, source
|
||||
locations, defaults, and credential mapping.
|
||||
- [Promptkit source upgrade guide](consumers/upgrading-promptkit.md): concise
|
||||
source-selector and definition migration examples.
|
||||
- [PromptKit v0.23.3 framework formats](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/formats.md):
|
||||
prompt, profile, schema, execution-setting, and framework credential
|
||||
contracts.
|
||||
- [HTTP API reference](api.md): route, request/response schema, status codes,
|
||||
limits, and HTTP artifact access.
|
||||
- [Consumer integration overview](consumers/api.md): caller responsibilities.
|
||||
Provide readable application configuration, its selected framework entry file,
|
||||
and explicitly imported definitions/resources. Set cwd deliberately: application
|
||||
path values are cwd-relative, while framework imports and message resources keep
|
||||
their native source contexts. ConfigKit confines source imports/overlays to the
|
||||
selected main's canonical root; Scriptorium does not widen it.
|
||||
|
||||
## Operational Model And State
|
||||
Engine construction validates selected effective definitions and captures message
|
||||
resources/template syntax, including definitions not used by a particular request.
|
||||
Schema/artifact reads, credential resolution, and model work retain their native
|
||||
preparation/runtime lifecycles. A server constructs one engine at startup and
|
||||
does not hot-reload changed files; restart to capture updates. Separate CLI
|
||||
invocations load afresh rather than sharing an immutable snapshot.
|
||||
|
||||
Scriptorium handles one prompt request for each CLI invocation or HTTP request.
|
||||
It has no durable run store, archive, checkpoint, cache, or resume mechanism.
|
||||
A failed or interrupted request is recovered by correcting its inputs,
|
||||
configuration, or environment and submitting a new request.
|
||||
For independent collections, install checkouts beneath the layout selected by
|
||||
the [discovery entry](../examples/discovery/framework.yml). Each checkout declares
|
||||
its namespace and explicit fragment imports; arbitrary YAML is not a framework
|
||||
source. The maintained [discovery example](../examples/config.discovery.yml)
|
||||
supports both entry-file spellings and multiple definitions. New checkouts or
|
||||
updates become visible to later invocations or a restarted server.
|
||||
|
||||
Generated artifacts, rendered prompts, model output, and run metadata are
|
||||
caller-owned data. Retention, encryption, backup, and deletion are deployment
|
||||
responsibilities.
|
||||
Optional [CLI execution reports](cli.md) also belong to the caller. They can
|
||||
contain sensitive identifiers even though they omit prompt and response bodies.
|
||||
Keep report paths separate from generated output and apply the same access and
|
||||
retention controls. Output and report writes are not transactional; an older
|
||||
report may remain when no terminal observation was produced.
|
||||
HTTP callers may also request a [terminal execution report](api.md#execution-report).
|
||||
It shares the same identifier sensitivity and should receive response access
|
||||
and retention controls. An opted-in report increases the encoded response size;
|
||||
an oversized success or report-bearing error becomes the compact 413 response
|
||||
without a report.
|
||||
Supply only needed credential environment variables through a supervisor or
|
||||
secret-management mechanism. Confirm presence without printing their values.
|
||||
Raw keys do not belong in arguments or checked-in configuration.
|
||||
|
||||
## Deploy The Filesystem And Process
|
||||
## Normal Operator Workflow
|
||||
|
||||
Provide the process with readable selected Promptkit source files/directories
|
||||
and schema sources that follow the tagged framework formats. At engine
|
||||
construction, PromptKit v0.23.3 validates every selected effective definition,
|
||||
including unused definitions, and captures referenced message-file contents
|
||||
and static template syntax. `serve` constructs one engine at startup; it does
|
||||
not reload changed configuration or message content. Restart the process to
|
||||
validate and capture a new configuration. Schema and request-artifact reads,
|
||||
credential resolution, and provider work remain request-preparation or runtime
|
||||
concerns. For an HTTP deployment that accepts file artifacts, use a dedicated,
|
||||
narrow artifact directory rather than a general-purpose or sensitive filesystem
|
||||
tree.
|
||||
1. Validate application configuration through the native application scope.
|
||||
This does not acquire framework sources or prove provider readiness.
|
||||
2. Inspect framework configuration/resources and discover requirements for the
|
||||
selected prompt/profile/namespaces. Discovery can precede generated input
|
||||
artifacts; completeness does not establish satisfied bindings.
|
||||
3. Once artifacts exist, validate the concrete request or inspect prepared
|
||||
content. These are provider-free preparation checks with native prerequisites.
|
||||
4. Invoke a smoke test only when its provider effects are intended. Findings
|
||||
can be unsupported or unsuccessful while the command itself succeeds.
|
||||
Smoke tests are not an automatic startup or release prerequisite.
|
||||
5. Generate with the same selections and bindings, then inspect the finalized
|
||||
status and any returned validation result before accepting the output.
|
||||
|
||||
If application configuration uses explicit imports or a selected overlay,
|
||||
deploy those files beneath the selected configuration file's canonical
|
||||
directory. ConfigKit keeps imported and overlay files within that root;
|
||||
Scriptorium does not widen it. Application path values inside those files still
|
||||
resolve from the process working directory, so set that directory deliberately
|
||||
in the service manager. This configuration-source boundary is distinct from
|
||||
the HTTP artifact root and does not constrain prompt or schema directories.
|
||||
Use the [maintained preview script](../examples/render-markdown-summary.sh)
|
||||
for a provider-free example, and the [CLI contract](cli.md) for commands.
|
||||
A complete discovery report can still list required inputs that are not bound;
|
||||
a partial report can still be useful. Consumers own their policy for incomplete
|
||||
findings and changed resources between invocations.
|
||||
|
||||
For independently maintained collections, place checkouts beneath the
|
||||
directory selected by your framework discovery file. The maintained
|
||||
[discovery example](../examples/config.discovery.yml) selects optional
|
||||
`collection.yml` and `collection.yaml` entry files; each checkout owns its
|
||||
namespace and explicit fragment imports. Update a checkout independently and
|
||||
invoke the CLI again, or restart a running server to capture the update. Do
|
||||
not treat arbitrary YAML in a checkout as a framework source. When upgrading
|
||||
definitions, review exact profile IDs and replace retired catalog references
|
||||
as described in the [migration guide](consumers/upgrading-promptkit.md).
|
||||
## Sensitive Data And Retention
|
||||
|
||||
Run Scriptorium under an identity that can:
|
||||
Artifacts, prompt messages, prepared content, native run reports, metadata,
|
||||
diagnostics, and redirected streams are caller-owned potentially sensitive data.
|
||||
The separate CLI report contains the full returned result, not metadata alone.
|
||||
Control access, retention, encryption, backup, and deletion accordingly.
|
||||
Framework effective/explanation views apply native classified-field redaction,
|
||||
including history, but do not detect arbitrary secrets. Other content-bearing
|
||||
reports are not automatically made safe for publication.
|
||||
|
||||
- read only the prompt, profile, schema, and allowed input-artifact paths it
|
||||
needs;
|
||||
- read the required credential environment variables without writing them to
|
||||
files or logs; and
|
||||
- write only caller-selected output locations when CLI output files are used.
|
||||
Keep main output and report destinations separate. Delivery is nontransactional:
|
||||
one destination or partial bytes can remain after failure. An existing report
|
||||
can remain on a resultless error; check status and report identity, not just file
|
||||
existence. A shell can truncate a redirected destination before the child starts.
|
||||
Omit quiet if main stdout must be captured; the separate report effect remains
|
||||
independent. Exact delivery behavior belongs to [CLI results](cli.md#results-and-separate-reports).
|
||||
|
||||
Do not make the HTTP artifact directory writable by untrusted users. The HTTP
|
||||
artifact containment behavior is lexical and the operating system follows
|
||||
symlinks; account for that when choosing ownership and mount boundaries. See
|
||||
the [HTTP API reference](api.md) for the externally observable behavior.
|
||||
HTTP's optional [execution report](api.md#execution-report) remains a distinct
|
||||
terminal-observation projection. Protect its identifiers and any response data.
|
||||
It increases encoded response size and remains subject to the configured limit.
|
||||
|
||||
## Supply Credentials And Protect Runtime Data
|
||||
## HTTP Deployment And Shutdown
|
||||
|
||||
Set secret values in the process environment and configure only their
|
||||
environment-variable names. Do not put raw keys in configuration, prompt or
|
||||
profile files, process arguments, HTTP payloads, captured command lines, or
|
||||
debug dumps.
|
||||
The service has no built-in authentication/authorization. Deploy on a trusted
|
||||
network or behind appropriate authenticated proxy/gateway controls. Use a narrow,
|
||||
dedicated artifact root with controlled ownership and permissions when accepting
|
||||
file inputs; this is lexical containment, not a symlink sandbox.
|
||||
|
||||
Treat stdout, stderr, prepared-run output, generated artifacts, and HTTP
|
||||
responses as potentially sensitive. Send service logs to a controlled collector
|
||||
and apply the same retention and access rules as for model input and output.
|
||||
Use a supervisor for cwd, environment, restart policy, and stream capture.
|
||||
Startup information uses native Info reporting, not stdout banners.
|
||||
Interrupt/SIGTERM cancels the invocation and closes the listener and active
|
||||
connections. Canceled shutdown is nonzero operational termination. There is no
|
||||
graceful-drain deadline or promise that context-ignoring work has finished;
|
||||
supervisors own forced termination when necessary.
|
||||
|
||||
## Run A Normal Workflow
|
||||
## Capacity, Limits, And Recovery
|
||||
|
||||
Before changing production inputs, profiles, or schemas:
|
||||
One server engine shares backend admission/queue state across concurrent requests.
|
||||
That state is transient operational coordination, not a durable workflow or
|
||||
queue of resumable runs. HTTP capacity exhaustion retains its
|
||||
[API outcome](api.md); callers decide whether another model call is safe.
|
||||
|
||||
1. use `config validate`, `config show`, or `config sources` to check
|
||||
Scriptorium's application configuration; these do not open Promptkit sources
|
||||
or construct an engine;
|
||||
2. use `inspect configuration validate` to construct the engine and validate
|
||||
all selected framework definitions; this does not prepare a request, check
|
||||
credentials, load request-time schemas/artifacts, or contact a provider;
|
||||
3. confirm the deployed configuration selects the intended sources and
|
||||
environment-variable names;
|
||||
4. use [`render`](cli.md) with the same request inputs and variables to confirm
|
||||
preparation without a model call;
|
||||
5. use [`run`](cli.md) for generation; and
|
||||
6. retain or discard validation-failed output according to the caller's
|
||||
policy.
|
||||
Plan for input/output sizes, schema complexity, provider latency, retries, and
|
||||
repair costs. Native transient attempts and repair generations can add calls and
|
||||
waiting; engine defaults and per-request overrides belong to
|
||||
[configuration](config.md) and the [CLI](cli.md)/[HTTP](api.md) contracts.
|
||||
Whole-operation budgets are distinct from per-attempt generation timeouts.
|
||||
They remain cooperative and do not include configuration/setup or delivery.
|
||||
|
||||
Application configuration inspection can include paths and configured values.
|
||||
Framework `inspect configuration show` and `explain` use Promptkit's native
|
||||
redaction for classified credential-shaped provider fields, including
|
||||
historical values, while leaving environment-reference names and ordinary
|
||||
provider parameters visible. Neither report identifies arbitrary sensitive
|
||||
content by value or name. Protect inspection output accordingly; Scriptorium
|
||||
stores environment-variable names, not the resolved credential bytes.
|
||||
Before increasing HTTP byte limits, measure representative requests/artifacts/
|
||||
responses, confirm memory/network/provider capacity, retain authentication and
|
||||
upstream request controls, and test a representative workload. Do not widen the
|
||||
artifact root just to make an arbitrary host path accessible.
|
||||
|
||||
The [maintained render script](../examples/render-markdown-summary.sh) is a
|
||||
copyable preflight example. The CLI reference owns its complete invocation and
|
||||
exit semantics.
|
||||
For failures, capture diagnostics without publishing sensitive data. Verify
|
||||
selected configuration, resource identity, readable inputs, profile/credentials,
|
||||
and schema deployment. Use native prepared preview to isolate pre-generation
|
||||
problems. A completed content-validation failure is distinct from an operational
|
||||
failure to validate; consumers choose whether failed content is usable.
|
||||
Correct the cause, then issue a new request only when another provider call and
|
||||
its cost are acceptable.
|
||||
|
||||
## Expose The HTTP Service
|
||||
|
||||
The HTTP service has no built-in authentication or authorization. Place it on a
|
||||
trusted network or behind an authenticated reverse proxy, API gateway, or
|
||||
equivalent access control. Restrict who can reach it and who can read the
|
||||
artifact root.
|
||||
|
||||
Use a service manager or supervisor appropriate to the deployment to manage
|
||||
process lifetime, restart policy, log capture, and environment injection. The
|
||||
[HTTP API reference](api.md) owns client request shapes, status behavior, and
|
||||
artifact-access outcomes.
|
||||
|
||||
## Plan Capacity And Limits
|
||||
|
||||
Capacity is primarily determined by concurrent model calls, input and output
|
||||
sizes, schema complexity, provider latency, and network behavior. Size limits
|
||||
protect request bodies, HTTP file artifacts, and encoded responses; configure
|
||||
them through the [configuration reference](config.md) and rely on the
|
||||
[HTTP API reference](api.md) for their response effects.
|
||||
|
||||
Configured backend concurrency and queue capacity are enforced per constructed
|
||||
Promptkit engine. A `serve` process constructs one engine for its handler, so
|
||||
concurrent HTTP requests share that transient admission state. Scriptorium does
|
||||
not retain workflow state: capacity is neither durable nor a queue of resumable
|
||||
runs. When admission is exhausted, HTTP returns `503 capacity_exceeded` without
|
||||
retry timing; callers choose any retry policy that is safe for another model
|
||||
call.
|
||||
|
||||
Promptkit's built-in client now allows two additional attempts for eligible
|
||||
transient provider statuses by default. Account for added calls, latency, and
|
||||
cost when planning capacity. Each repair generation has its own transient
|
||||
budget, separate from the structural repair count. Configure the engine budget
|
||||
with [`transient_retries`](config.md); `run` and `render` accept a request
|
||||
override described in the [CLI contract](cli.md), and HTTP callers can select
|
||||
one through the [API contract](api.md). Omission inherits the engine
|
||||
setting and then Promptkit's default, while explicit zero disables additional
|
||||
attempts. Existing generation timeouts apply
|
||||
per attempt, while the caller context covers the complete operation. Provider
|
||||
`Retry-After` timing can exceed Promptkit's ordinary backoff cap; consult the
|
||||
[outbound integration contract](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/804a4d32b23aa2e7973400bb0a322a7016574c9f/docs/integrations/openai-compatible-chat.md)
|
||||
for details.
|
||||
|
||||
CLI `run` can set a [whole-run deadline](cli.md) when total execution time must
|
||||
be bounded. The existing generation timeout applies afresh to each attempt and
|
||||
does not cap retry waits or the complete invocation. The whole-run interval
|
||||
covers the engine call, including capacity waiting and repairs as supported by
|
||||
context cancellation; it excludes parsing, engine setup, and output writes.
|
||||
HTTP callers can use [`run_timeout_seconds`](api.md) to set the engine's
|
||||
operation budget for the same interval. The API distinguishes a typed expired
|
||||
execution deadline from caller cancellation and from a provider-attempt
|
||||
timeout. A canceled capacity waiter releases its Promptkit admission state.
|
||||
|
||||
Before increasing a limit:
|
||||
|
||||
1. measure representative input, generated-output, and optional raw-output
|
||||
sizes;
|
||||
2. confirm memory, network, and upstream-provider capacity;
|
||||
3. retain an upstream request-size and authentication boundary; and
|
||||
4. test the intended workload in a non-production environment.
|
||||
|
||||
For large local inputs, prefer a controlled file-artifact directory over
|
||||
placing arbitrary paths on the service host. Avoid disabling a limit unless an
|
||||
equivalent trusted control exists elsewhere.
|
||||
|
||||
## Diagnose And Recover
|
||||
|
||||
### Preparation Or Configuration Failure
|
||||
|
||||
Capture the CLI diagnostic or HTTP error response, then verify the selected
|
||||
configuration, prompt ID, profile selection, source readability, and input
|
||||
mapping. Use `render` with the same request when it is unclear whether failure
|
||||
occurs before model execution. Consult the [CLI reference](cli.md), the
|
||||
[configuration reference](config.md), and the [HTTP API reference](api.md) for
|
||||
the exact interface contract.
|
||||
|
||||
### Credential Or Provider Failure
|
||||
|
||||
Confirm that the process environment contains the configured credential name
|
||||
without printing the secret. Check endpoint reachability and provider health
|
||||
from the process network. If preparation succeeds but generation fails, inspect
|
||||
the selected model settings in prepared output and the service's controlled
|
||||
logs. Correct the deployment or provider issue, then submit a new request.
|
||||
|
||||
### Artifact Or Permission Failure
|
||||
|
||||
Verify that the process can read the intended local input. For HTTP file
|
||||
artifacts, verify the deployment's artifact root, ownership, path layout, and
|
||||
file size. Do not widen filesystem permissions or the allowed root merely to
|
||||
make an arbitrary path work; move or copy the required artifact into the
|
||||
controlled location instead.
|
||||
|
||||
### Validation Failure
|
||||
|
||||
A generated-content validation failure is distinct from a runtime failure.
|
||||
CLI `run` reports the validation result and error count in its success summary;
|
||||
it does not print the individual validation messages. For HTTP, inspect the
|
||||
validation object in the response according to the [HTTP API reference](api.md).
|
||||
|
||||
Use rendered input and generated output to determine whether prompt instructions,
|
||||
the selected model, or the schema needs correction. If schema loading or
|
||||
compilation itself fails, correct the source deployment or schema document
|
||||
before rerunning.
|
||||
|
||||
### HTTP Limit Or Request Failure
|
||||
|
||||
Compare the request, artifact, or expected response size with the deployed
|
||||
configuration, and validate the request against the [HTTP API reference](api.md).
|
||||
Reduce the payload, use an appropriate controlled artifact source, omit
|
||||
unneeded raw output, or adjust the deployment limit after capacity review.
|
||||
|
||||
## Cleanup And Reruns
|
||||
|
||||
Because no run state is retained, cleanup concerns caller-owned output files,
|
||||
logs, and artifacts only. Remove or rotate them using the deployment's normal
|
||||
retention policy. After a correction, rerun the request from the beginning;
|
||||
there is no safe resume point.
|
||||
There is no durable run store, archive, checkpoint, or resume mechanism.
|
||||
Cleanup concerns caller-owned files/logs only. Rotate/delete them under the
|
||||
deployment's normal retention policy; recovery starts from a new request.
|
||||
|
||||
Reference in New Issue
Block a user