826 lines
44 KiB
Markdown
826 lines
44 KiB
Markdown
# Configuration
|
|
|
|
This is the canonical reference for Notarius configuration. Configuration files
|
|
are YAML; the main file must declare version 5. They select pipelines and their modules;
|
|
the [CLI reference](cli.md) owns invocation syntax, and
|
|
[Operations](operations.md) owns run-state procedures.
|
|
|
|
## Configuration Discovery And Precedence
|
|
|
|
Commands that load configuration choose a file in this order:
|
|
|
|
1. a non-empty **--config** CLI value;
|
|
2. a non-empty **NOTARIUS_CONFIG** environment value;
|
|
3. `notarius.yml`, then `notarius.yaml` in the working directory;
|
|
4. the user configuration directory; then
|
|
5. the Unix system configuration directories.
|
|
|
|
The user search checks `<user-config>/notarius/` for `notarius.yml`,
|
|
`notarius.yaml`, `config.yml`, `config.yaml`, `main.yml`, and `main.yaml`, in
|
|
that order, then `<user-config>/notarius.yml` and `notarius.yaml`. The system
|
|
search uses the same groups under `/usr/local/etc/notarius/` and `/etc/notarius/`,
|
|
then `/etc/notarius.yml` and `/etc/notarius.yaml`. Generic filenames in the
|
|
working directory are not candidates. Home-directory search is not enabled.
|
|
|
|
Discovery selects one main file; it does not layer all candidates. A missing
|
|
candidate advances the search, but an explicitly selected missing file, access
|
|
error, or invalid selected file fails without fallback. An exhausted search
|
|
fails. Use an explicit `--config` for services and subprocesses that need a
|
|
fixed selection independent of their working directory.
|
|
|
|
For configuration values, precedence is:
|
|
|
|
1. built-in defaults;
|
|
2. composed YAML sources, including imports;
|
|
3. the selected ConfigKit overlay and inheritance;
|
|
4. typed decoding and explicitly supplied configuration-valued run flags; and
|
|
5. Notarius resolution and validation.
|
|
|
|
`--config-overlay` selects a ConfigKit overlay from `overlays/<name>.yml`
|
|
beneath the selected configuration root. Empty and `default` select no overlay.
|
|
`profiles/<name>.yml` is not a fallback location, and `--config-profile` is no
|
|
longer accepted. Configuration overlays are distinct from PromptKit LLM profiles selected by
|
|
`llm_profile` or `--llm-profile`. The retired operational environment variables
|
|
have no effect; `NOTARIUS_CONFIG` remains a source-discovery input.
|
|
|
|
## Maintained Examples
|
|
|
|
- [Minimal D&D configuration](../examples/dnd-minimal.config.yml) is a
|
|
single-lane Seriatim-to-spell pipeline.
|
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
|
|
ordered steps, all implemented D&D lanes, generated references, state
|
|
settings, bounded LLM concurrency, and the maintained
|
|
[LLM profile](../examples/llm-profiles/dnd-extraction.yml) and imports its
|
|
pipeline from [a fragment](../examples/pipelines/dnd-session.yml), which imports
|
|
its [first step](../examples/pipelines/dnd-session.describe-session.yml).
|
|
|
|
The complete fragment also defines `dnd-session-development`, which inherits
|
|
the complete pipeline and profile while setting
|
|
`structured_output_repair_attempts: 0`. Select it with:
|
|
|
|
```sh
|
|
notarius run dnd-session-development \
|
|
--config /path/to/dnd-complete.config.yml \
|
|
--input /path/to/transcript.json
|
|
```
|
|
|
|
See the [CLI reference](cli.md#run) for the full invocation contract. This
|
|
variant disables PromptKit structural repairs without changing configured producer
|
|
retry budgets. When using another profile, operators can instead select
|
|
`dnd-session` and set its `llm_profile` to that profile.
|
|
|
|
Use these complete files as starting points rather than combining the
|
|
illustrative fragments in this reference.
|
|
|
|
## File Shape And Defaults
|
|
|
|
Unknown fields, duplicate mapping keys, empty identifiers, and identifiers that
|
|
become duplicates after trimming whitespace are rejected. Every top-level field
|
|
other than **version** is optional.
|
|
|
|
| Field | Type | Default | Rules |
|
|
| --- | --- | --- | --- |
|
|
| **version** | integer or quoted string | none | Required; must be 5. Maintained files use `version: 5`. |
|
|
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
|
|
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
|
|
| **concurrency** | object | see below | Global LLM and extraction limits. |
|
|
| **output** | object | see below | Published output settings. |
|
|
| **cache** | object | see below | Chunk-plan and checkpoint settings. |
|
|
| **debug** | object | see below | Debug-bundle root only; it does not enable capture. |
|
|
|
|
Built-in defaults are:
|
|
|
|
| Field | Default |
|
|
| --- | --- |
|
|
| **concurrency.total_llm** | 16 |
|
|
| **concurrency.stage_workers.extract** | Effective **total_llm** |
|
|
| **output.directory** | **./notarius-output** |
|
|
| **cache.chunk_plans.mode** | **auto** |
|
|
| **cache.chunk_plans.directory** | Empty, selecting the per-user chunk-plan root |
|
|
| **cache.checkpoints.enabled** | false |
|
|
| **cache.checkpoints.directory** | Empty, selecting the per-user checkpoint root |
|
|
| **debug.directory** | **./notarius-debug** |
|
|
|
|
An empty cache directory in YAML deliberately selects the corresponding
|
|
per-user root. An explicit empty output or debug directory is invalid.
|
|
Selecting a chunk-plan or checkpoint directory does not configure filesystem
|
|
access policy. Cache permissions come from deployment provisioning, process
|
|
umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cache-provisioning).
|
|
|
|
Every nonempty relative configured destination or reference path resolves from
|
|
the canonical directory of the file that supplies its surviving declaration.
|
|
This includes imported fragments, selected configuration overlays, and values
|
|
copied through inheritance: the copied value keeps its original declaring
|
|
file. A child-supplied replacement resolves from the child's declaring file.
|
|
Built-in relative destinations use the selected main configuration file's
|
|
directory; explicit `--output-dir` and `--debug-dir` values use the invocation
|
|
working directory. Absolute values remain unchanged.
|
|
|
|
Use source-root `config` controls for imports and collection declarations. For
|
|
example, an essential fragment should explicitly be required:
|
|
|
|
~~~yaml
|
|
config:
|
|
imports:
|
|
- path: pipelines/session.yml
|
|
required: true
|
|
~~~
|
|
|
|
Imports are optional by default. ConfigKit supports paths, lists, directories,
|
|
globs, exclusions, recursive selection, destinations, and sequence-item placement;
|
|
its native input-root rules apply. Imported fragments need not declare a version.
|
|
The complete example places a mapping fragment at a step-list destination. Nested
|
|
application data named `config` is ordinary data, not a control block.
|
|
|
|
When migrating application files, rename genuine root `configkit` control
|
|
blocks to `config` in the main file and fragments. This is not a compatibility
|
|
alias: `configkit` is ordinary data under the new reader. PromptKit profile
|
|
documents still use their separate older engine and retain `configkit` controls;
|
|
see [PromptKit Profiles](#promptkit-profiles).
|
|
|
|
## Sibling Inheritance
|
|
|
|
ConfigKit supports native `extends` composition in mappings and identified
|
|
sequence records. In a mapping, `extends: production` selects the named immediate
|
|
sibling in that same mapping. In an identified sequence, the reference is an
|
|
identity mapping: for steps identified by `id`, `extends: {id: parent}` selects
|
|
a record in the same final sequence. Neither form is a path or cross-scope
|
|
lookup. See the pinned [native inheritance examples](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e/docs/config.md#sibling-inheritance).
|
|
Imports and the selected overlay are composed before
|
|
inheritance and typed decoding. Explicit CLI assignments follow decoding and
|
|
inheritance, before Notarius resolution and validation; they cannot repair an
|
|
earlier structural or decoding error. Every definition, including a parent and definitions not
|
|
selected for a run, remains subject to ordinary validation.
|
|
|
|
Child mappings recursively overlay parent mappings. Scalars and null replace
|
|
inherited values. Unidentified sequences replace as whole values; pipeline
|
|
steps are identified by `id` and use native upsert by default:
|
|
|
|
~~~yaml
|
|
pipelines:
|
|
production:
|
|
input: {module: seriatim}
|
|
llm_profile: dnd-extraction
|
|
# Other complete pipeline settings and ordered steps...
|
|
development:
|
|
extends: production
|
|
structured_output_repair_attempts: 0
|
|
~~~
|
|
|
|
Pipeline names are siblings within `pipelines`; nested mappings such as named
|
|
artifact lanes also have their own sibling namespace. Distinct imported step IDs
|
|
combine in source order; duplicate peer IDs conflict. Across an overlay or
|
|
inheritance, upsert replaces a matching step as a complete record in place and
|
|
appends new IDs. It does not merge individual fields of a matching step. An
|
|
identified record can itself use native `extends`. Authors may select `replace`
|
|
or `append` with a `config.sequences` declaration at the relevant destination;
|
|
an explicit replacement with `[]` clears the list, while empty append or upsert
|
|
does not. Notarius then validates the resulting pipeline, including its nonempty
|
|
step requirement. Validator chains are not automatically identified. Authors
|
|
may declare identities for other lists when needed. CLI list assignment replaces
|
|
the complete decoded list.
|
|
|
|
For example, an overlay can opt into replacing a pipeline's steps:
|
|
|
|
~~~yaml
|
|
config:
|
|
sequences:
|
|
- destination: [pipelines, production, steps]
|
|
mode: replace
|
|
pipelines:
|
|
production:
|
|
steps:
|
|
- id: extract
|
|
artifacts: {spells: {extract: {module: dnd/spells}}}
|
|
~~~
|
|
|
|
Inherited values count as explicit input for Notarius's existing
|
|
presence-sensitive rules: zero, false, and empty-list overrides stay explicit
|
|
instead of restoring a default. Null also replaces the inherited value, but
|
|
each field's ordinary type and validation rules still apply; null is not
|
|
universally valid. Omitting `extends` preserves a relationship supplied by a
|
|
lower-priority source, while `extends: null` cancels that relationship. It
|
|
does not independently clear other inherited fields.
|
|
|
|
`extends` is now a reserved key in every input mapping, including dynamic
|
|
module options and named maps. Existing ordinary data or entry names using that
|
|
key must be renamed along with any consumer that reads them. Configuration
|
|
overlays, pipeline inheritance, and PromptKit's native profile `extends`
|
|
inheritance are separate mechanisms.
|
|
|
|
## PromptKit Profiles
|
|
|
|
The optional **promptkit** object selects one source of profile definitions and
|
|
may register one conventional local OpenAI-compatible backend:
|
|
|
|
~~~yaml
|
|
version: 5
|
|
|
|
promptkit:
|
|
profile_dir: ./profiles
|
|
# profile_file: ./profiles.yml
|
|
transient_retries: 2
|
|
local_backend:
|
|
endpoint: http://localhost:8000/v1
|
|
concurrency_limit: 2
|
|
~~~
|
|
|
|
| Field | Type | Rules |
|
|
| --- | --- | --- |
|
|
| **profile_dir** | string | Non-empty directory containing profile files. |
|
|
| **profile_file** | string | Non-empty profile file. |
|
|
| **transient_retries** | integer | Non-negative additional built-in HTTP attempts for eligible transient provider failures; defaults to 2. Set 0 to disable these retries. |
|
|
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
|
|
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
|
|
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
|
|
|
|
Set at most one of **profile_dir** and **profile_file**. Every nonempty relative
|
|
configured filesystem path is resolved from the canonical directory of the
|
|
file that supplies it. The complete example's
|
|
`./llm-profiles/dnd-extraction.yml` value therefore resolves beside that example.
|
|
Use absolute paths for services and containers when their deployment layout is
|
|
not configuration-relative.
|
|
|
|
An operator source is optional. PromptKit's maintained catalog, Notarius's
|
|
registered prompt and fallback assets, and the selected operator source are
|
|
assembled in that order; a configured `local_backend` contributes its backend
|
|
last. Later source groups recursively overlay earlier groups. Within one group,
|
|
inputs compose strictly; directory filenames do not provide priority. For a
|
|
requested profile ID, an operator definition can partially override the
|
|
embedded fallback while retaining unspecified fields. The maintained
|
|
[`dnd-extraction` LLM profile](../examples/llm-profiles/dnd-extraction.yml)
|
|
is a secret-free deployment artifact. It also demonstrates a derived
|
|
development profile that is not selected by the normal pipeline.
|
|
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
|
|
workflow and credential-handling guidance.
|
|
|
|
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
|
|
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
|
|
allowed. User information, queries, and fragments are rejected. A zero
|
|
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
|
|
a positive value limits simultaneous local generations. The application-wide
|
|
**concurrency.total_llm** limit still applies in both cases. Neither local
|
|
backend field has an environment override. Omitting **local_backend** registers
|
|
nothing and preserves existing built-in and endpoint-only profile behavior.
|
|
|
|
A PromptKit file uses keyed mappings for profiles, backends, and prompts. A
|
|
profile selects a backend by its exact ID:
|
|
|
|
~~~yaml
|
|
profiles:
|
|
local-summary:
|
|
backend: local
|
|
model: example-model
|
|
~~~
|
|
|
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
|
its credential environment variable through `api_key_env`; set that variable
|
|
only in the run environment. `api_key_required` is supported in profile files;
|
|
inspection reports PromptKit's effective metadata but does not check whether
|
|
the named environment variable is set. PromptKit owns the
|
|
[pinned v0.12.0 format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md),
|
|
including native `extends` inheritance and field-presence rules. Notarius
|
|
selects source groups and passes definitions through PromptKit.
|
|
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
|
|
supported package API, and [Operations](operations.md#operational-limits)
|
|
describes the effective concurrency layers.
|
|
|
|
### Native PromptKit Sources And Migration
|
|
|
|
The configured `profile_file` or `profile_dir` is a PromptKit configuration
|
|
source. It can contain profiles and backends, as well as prompt overlays.
|
|
PromptKit's catalog is followed by registered application prompt/fallback
|
|
assets and the selected operator source; Notarius's optional programmatic
|
|
`local` backend is applied last. Composition, recursion, exact-key inheritance,
|
|
and final-value validation are owned by PromptKit and ConfigKit. Explicit
|
|
`null`, zero, false, blank, and omission may have different meanings; an empty
|
|
map does not clear an inherited map.
|
|
|
|
When migrating external files from PromptKit before v0.12, place backend and
|
|
profile entries beneath their `backends` and `profiles` mappings, and place
|
|
prompt versions beneath `prompts.<prompt-id>.v1`. Remove embedded prompt
|
|
`id`/`version` fields. Replace `base_profile` with native `extends`. There is
|
|
no legacy-format parser or alias layer. Keep Notarius's `version: 5` and
|
|
`promptkit.profile_file`/`profile_dir` settings unchanged.
|
|
|
|
Imports use native ConfigKit declarations relative to the file that declares
|
|
them. A file-root layout keeps imported fragments outside a recursively
|
|
selected profile directory, avoiding selection of the same definition both by
|
|
directory discovery and by import:
|
|
|
|
~~~yaml
|
|
# notarius.yml
|
|
version: 5
|
|
promptkit:
|
|
profile_file: ./promptkit/entry.yml
|
|
~~~
|
|
|
|
~~~yaml
|
|
# promptkit/entry.yml
|
|
configkit:
|
|
imports:
|
|
- path: fragments/profiles.yml
|
|
profiles:
|
|
dnd-extraction-development:
|
|
extends: dnd-extraction
|
|
temperature: 0.2
|
|
~~~
|
|
|
|
PromptKit recursively discovers YAML files in a selected directory, and its
|
|
inputs compose strictly; there is no filename-priority merger. Imports retain
|
|
their declaring source roots, so relative message/content files resolve from
|
|
the file that declares them. Use a separate file root for related imports
|
|
when directory selection would otherwise load those files a second time.
|
|
|
|
The same unified document can declare a custom backend or select a catalog
|
|
backend without redeclaring it:
|
|
|
|
~~~yaml
|
|
backends:
|
|
private-compatible:
|
|
endpoint: https://llm.example.invalid/v1
|
|
api_key_env: PRIVATE_LLM_API_KEY
|
|
profiles:
|
|
private-summary:
|
|
backend: private-compatible
|
|
model: summary-model
|
|
catalog-summary:
|
|
backend: openrouter
|
|
model: openai/gpt-5.6-luna
|
|
~~~
|
|
|
|
To override the embedded D&D fallback for a configured local backend, supply
|
|
only the changed fields. Explicitly clear inherited settings that do not apply
|
|
to the local target:
|
|
|
|
~~~yaml
|
|
profiles:
|
|
dnd-extraction:
|
|
backend: local
|
|
model: local-model
|
|
service_tier: null
|
|
~~~
|
|
|
|
The backend can be declared as `backends.local` in the operator source or
|
|
registered through the optional `promptkit.local_backend` convenience object.
|
|
The [source precedence](#native-promptkit-sources-and-migration) applies when
|
|
both contribute. Prompt overlays use the same recursive source composition; for
|
|
example, this changes one output setting while retaining the other embedded
|
|
definition fields:
|
|
|
|
~~~yaml
|
|
prompts:
|
|
dnd.spells:
|
|
v1:
|
|
output:
|
|
repair_attempts: 0
|
|
~~~
|
|
|
|
Sequences such as prompt messages are replaced as a whole when overridden.
|
|
Engine construction validates final effective prompt definitions and captures
|
|
message files and static templates. Request preparation loads response schemas
|
|
and resolves request variables and input artifacts. Profile preflight needs no
|
|
transcript, credential value, or provider call, but can read prompt-content files.
|
|
|
|
`notarius config validate --pipeline <id>` resolves the selected pipeline and
|
|
inspects every explicit effective profile without contacting a provider or
|
|
requiring credential values. It rejects absent, malformed, or incompatible
|
|
profiles before a run prepares modules. Credential availability is checked only
|
|
when a generation is prepared.
|
|
|
|
## Version 5 Migration
|
|
|
|
Version 3 and version 4 files are rejected and are not rewritten at load time.
|
|
Use `version: 5`, rename the former `scriptorium` section to `promptkit`, and
|
|
convert every module binding to an object containing `module`. Convert every
|
|
external reference to `{path: ...}` and move any lane-level references to the
|
|
corresponding extraction binding. Put durable operational values in YAML or
|
|
use the documented explicit run flags; the former field-level environment
|
|
variables are not read.
|
|
|
|
## Concurrency, Output, Cache, And Debug
|
|
|
|
~~~yaml
|
|
concurrency:
|
|
total_llm: 2
|
|
stage_workers:
|
|
extract: 2
|
|
output:
|
|
directory: ./notarius-output
|
|
cache:
|
|
chunk_plans:
|
|
mode: auto
|
|
directory: ./notarius-cache/chunk-plans
|
|
checkpoints:
|
|
enabled: true
|
|
directory: ./notarius-cache/checkpoints
|
|
debug:
|
|
directory: ./notarius-debug
|
|
~~~
|
|
|
|
**concurrency.total_llm** must be greater than zero. The only supported
|
|
**concurrency.stage_workers** key is **extract**; its value must be from 1
|
|
through **total_llm**. When omitted, it is recalculated from the effective
|
|
**total_llm** after imports, overlay selection, and explicit ConfigKit
|
|
overrides.
|
|
|
|
**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**.
|
|
**cache.checkpoints.enabled** is a boolean. The CLI can override the output
|
|
directory and chunk-plan mode for one run; see [CLI reference](cli.md#run).
|
|
Cache directory fields select only locations; Notarius has no cache permission
|
|
field or operational environment override.
|
|
|
|
## Pipelines
|
|
|
|
Each **pipelines** entry has a unique, non-empty ID and the following shape:
|
|
|
|
~~~yaml
|
|
pipelines:
|
|
dnd-session:
|
|
llm_profile: dnd-extraction
|
|
input:
|
|
module: seriatim
|
|
chunk:
|
|
module: generic
|
|
output:
|
|
module: json
|
|
artifacts:
|
|
spells:
|
|
extract:
|
|
module: dnd/spells
|
|
merge:
|
|
module: appendorder
|
|
normalize:
|
|
module: dnd/spells
|
|
~~~
|
|
|
|
| Field | Type | Default | Rules |
|
|
| --- | --- | --- | --- |
|
|
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
|
|
| **structured_output_repair_attempts** | integer | prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for selected LLM-backed bindings and validators. Omission leaves the prompt's declared policy in control; explicit 0 disables structural repair at that scope. |
|
|
| **validation_policy** | object | see below | Optional terminal policy defaults for producer validation. Its fields inherit independently into chunk, extract, merge, and normalize bindings. |
|
|
| **input** | module binding | none | Required. |
|
|
| **chunk** | module binding | **generic** | Optional. |
|
|
| **output** | module binding | **json** | Optional. |
|
|
| **artifacts** | map | none | Compact single-step lane map. |
|
|
| **steps** | list | none | Ordered lane definitions. Mutually exclusive with **artifacts**. |
|
|
| **references** | map | none | External reference defaults for eligible targets. |
|
|
|
|
Use either **artifacts** or **steps**. The compact **artifacts** form is an
|
|
implicit single step. An explicit **steps** list must be non-empty; every step
|
|
needs a unique non-empty **id**, an **artifacts** map, and may have
|
|
**references**. A lane ID must not appear more than once in a pipeline,
|
|
including across explicit steps.
|
|
|
|
For each selected LLM-backed binding or validator, profile selection occurs
|
|
after module, validator, and `--only` lane selection. It uses the
|
|
run-level **--llm-profile** value first, then the binding's **llm_profile**,
|
|
then the pipeline's **llm_profile**, and finally the PromptKit default.
|
|
Deterministic bindings do not receive these defaults or run overrides.
|
|
|
|
Structural output repair is resolved after module, validator, and `--only` lane
|
|
selection. An object's **structured_output_repair_attempts** value takes
|
|
precedence over the pipeline value; otherwise, an LLM-backed binding or
|
|
validator inherits the pipeline value. If both are omitted, PromptKit uses the
|
|
prompt's declared repair policy. The value must be an integer from 0 through 3;
|
|
explicit `null` and non-integer values are invalid. An explicit value on a
|
|
deterministic binding or validator is invalid, while a pipeline value simply
|
|
does not apply to deterministic selections.
|
|
|
|
`validation_policy` controls terminal disposition for one complete producer
|
|
attempt and validator chain. It may appear on a pipeline or a **chunk**,
|
|
**extract**, **merge**, or **normalize** module binding; input, output, and
|
|
validator bindings reject it. Every field is optional and resolves in binding,
|
|
pipeline, then application-default order:
|
|
|
|
| Field | Values | Default |
|
|
| --- | --- | --- |
|
|
| **producer_structural_failure** | **fail_run**, **reject_output** | **fail_run** |
|
|
| **semantic_rejection** | **fail_run**, **reject_output** | **fail_run** |
|
|
| **validator_failure** | **warn_continue**, **fail_run** | **warn_continue** |
|
|
|
|
The policy object and its fields must be non-null, and unknown fields are
|
|
rejected. A deterministic producer may not explicitly set
|
|
**producer_structural_failure** on its binding, although a pipeline-level
|
|
default remains valid for pipelines that include LLM-backed producers.
|
|
|
|
After the producer binding's retry budget is exhausted, an invalid structured
|
|
response uses **producer_structural_failure**. One or more semantic validator
|
|
rejections use **semantic_rejection**; rejection takes precedence over an
|
|
exhausted validator failure or skip. With no rejection, an exhausted validator
|
|
failure or skip uses **validator_failure**. `reject_output` records the
|
|
terminal rejection without advancing that candidate. `warn_continue` is valid
|
|
only for validator execution failure: it advances a structurally valid,
|
|
otherwise unrejected result with incomplete-validation provenance and without
|
|
making it reusable checkpoint state.
|
|
|
|
A lane has these fields:
|
|
|
|
| Field | Type | Default | Rules |
|
|
| --- | --- | --- | --- |
|
|
| **extract** | module binding | none | Required. |
|
|
| **merge** | module binding | **appendorder** | Optional. |
|
|
| **normalize** | module binding | **noop** | Optional. |
|
|
| **references** | map | none | Not supported at lane scope; place references on the eligible binding. |
|
|
| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. |
|
|
|
|
Lane-level **references** are not supported. Put a lane extraction reference
|
|
under **extract.references**.
|
|
|
|
## Module Bindings And Validators
|
|
|
|
Every module and validator binding uses an object, even when it has no fields
|
|
beyond its module key:
|
|
|
|
~~~yaml
|
|
input:
|
|
module: seriatim
|
|
~~~
|
|
|
|
Use an object for fields:
|
|
|
|
~~~yaml
|
|
extract:
|
|
module: dnd/spells
|
|
llm_profile: dnd-extraction
|
|
retries: 2
|
|
references:
|
|
spell_catalog:
|
|
path: ./dnd-spell-catalog.json
|
|
~~~
|
|
|
|
| Binding field | Type | Default | Rules |
|
|
| --- | --- | --- | --- |
|
|
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
|
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
|
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
|
|
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
|
|
| **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. |
|
|
| **options** | object | none | Must satisfy the selected module. |
|
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
|
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
|
|
|
|
Omitting **validators** uses the registered chain. **validators: []** selects
|
|
an empty chain; a non-empty list replaces the chain in the listed order.
|
|
Validator bindings accept only **module**, **llm_profile**,
|
|
**structured_output_repair_attempts**, **retries**, and **options**. Their
|
|
**retries** value is a non-negative additional validator-execution budget and
|
|
is valid only when the selected validator is LLM-backed. A validator retry
|
|
rechecks the same immutable candidate; it never regenerates the producer.
|
|
They reject
|
|
**validation_policy**, **references**, and nested **validators**. Deterministic
|
|
validators reject explicit **llm_profile** and
|
|
**structured_output_repair_attempts**.
|
|
Deterministic module bindings also reject those explicit fields.
|
|
|
|
An LLM-backed chunk, extract, merge, or normalize producer with both a
|
|
non-empty validator chain and positive **retries** must declare the supported
|
|
single-response correction capability. Preparation rejects a configuration
|
|
that could require semantic correction from a producer that cannot provide an
|
|
exact prior response. A deterministic producer, or an LLM attempt that did
|
|
not make a model call, cannot consume a semantic retry after rejection.
|
|
|
|
The **json** output module accepts optional **include_chunk_map** and
|
|
**evidence_context** settings:
|
|
|
|
~~~yaml
|
|
output:
|
|
module: json
|
|
options:
|
|
include_chunk_map: true
|
|
evidence_context:
|
|
enabled: true
|
|
window_units: 3
|
|
lanes:
|
|
- npc-registry
|
|
- spells
|
|
~~~
|
|
|
|
**include_chunk_map** is a boolean and defaults to false. It adds the accepted
|
|
chunk map when one exists; its wire format is defined in the
|
|
[chunk-map contract](integrations/chunk-map.md).
|
|
|
|
Omitting **evidence_context** disables evidence publication. When present, it
|
|
is an object with these strict fields:
|
|
|
|
| Field | Type | Rules |
|
|
| --- | --- | --- |
|
|
| **enabled** | boolean | Required. `false` permits no other evidence fields. |
|
|
| **lanes** | array of strings | Required and non-empty when enabled. Each value is trimmed and must be unique; every value must name a configured pipeline lane. |
|
|
| **window_units** | non-negative integer | Optional when enabled; defaults to 3. Zero retains only directly cited units. |
|
|
|
|
Unknown outer or nested option fields are rejected, as are incompatible YAML
|
|
types. The allowlist remains valid when a run uses lane filtering: a configured
|
|
lane that is not active for that invocation simply contributes no evidence.
|
|
Evidence publication is opt-in because it can persist source text and metadata.
|
|
When enabled, it publishes the selected source-unit excerpt defined by the
|
|
[Published Evidence Context contract](integrations/evidence-context.md).
|
|
|
|
## References And Ordered Handoffs
|
|
|
|
Reference maps bind named slots that the selected target declares. An external
|
|
reference is an object with `path`. Pipeline-level maps accept only external
|
|
paths; step-local and binding-local maps may also select a normalized artifact
|
|
from an earlier step:
|
|
|
|
~~~yaml
|
|
steps:
|
|
- id: describe-session
|
|
artifacts:
|
|
npc-registry:
|
|
extract:
|
|
module: dnd/npc-registry
|
|
normalize:
|
|
module: dnd/npc-registry
|
|
- id: extract-events
|
|
references:
|
|
npc_registry:
|
|
artifact:
|
|
step: describe-session
|
|
lane: npc-registry
|
|
artifacts:
|
|
spells:
|
|
extract:
|
|
module: dnd/spells
|
|
normalize:
|
|
module: dnd/spells
|
|
~~~
|
|
|
|
An artifact selector contains only **step** and **lane**. The producer must be
|
|
an earlier step and the selected artifact must be compatible with the consumer
|
|
slot. A generated binding supplies one accepted normalized artifact; it does
|
|
not name a file. A configured generated dependency remains required even when
|
|
that consumer slot is otherwise optional.
|
|
|
|
Pipeline references are configuration defaults. A matching step-local or
|
|
binding-local external path overrides a pipeline default. CLI reference
|
|
bindings are then operational overrides of configured external paths; their
|
|
pipeline, lane, and binding scopes and precedence are defined by the
|
|
[CLI reference](cli.md#reference-selectors). A CLI file reference cannot
|
|
replace a configured generated artifact handoff. Required slots must be bound
|
|
after configuration and CLI reference actions are applied. A YAML reference
|
|
path resolves from the file that supplies its surviving declaration, including
|
|
an imported fragment or selected overlay. CLI reference paths resolve relative
|
|
to the process working directory.
|
|
|
|
### D&D Reference Slots
|
|
|
|
The following slot names are accepted by the implemented D&D modules when the
|
|
selected target declares them:
|
|
|
|
| Slot | Source and use |
|
|
| --- | --- |
|
|
| **party** | Optional text campaign context. This is the canonical party-roster spelling. |
|
|
| **roster** | Accepted compatibility alias for **party**. |
|
|
| **players** | Optional text player context. |
|
|
| **glossary** | Optional text campaign glossary. |
|
|
| **spell_catalog** | Optional JSON spell-catalog overlay for spell extraction and normalization. See [spell-catalog overlays](integrations/dnd-spell-catalog-overlays.md). |
|
|
| **location_registry** | Required normalized location registry for location-occurrence extraction and normalization. |
|
|
| **item_registry** | Required normalized item registry for item-occurrence extraction and normalization. |
|
|
| **npc_registry** | Normalized NPC registry. Optional for spells and combat turns; required for NPC occurrences and enemy-event extraction and normalization. |
|
|
| **scene_descriptions** | Required normalized scene-description artifact for combat-turn and enemy-event extraction. |
|
|
| **combat_turns** | Required normalized combat-turn artifact for enemy-event extraction. |
|
|
| **npc_occurrences** | Required normalized NPC-occurrence artifact for enemy-event extraction. |
|
|
|
|
Registry-backed occurrence and enemy-event artifact slots have the following
|
|
exact binding contracts. Durable semantics and wire shapes remain in their
|
|
[NPC occurrence](integrations/dnd-npc-occurrence-artifacts.md),
|
|
[location occurrence](integrations/dnd-location-occurrence-artifacts.md),
|
|
[item occurrence](integrations/dnd-item-occurrence-artifacts.md), and
|
|
[enemy-event](integrations/dnd-enemy-event-artifacts.md) contracts.
|
|
|
|
| Slot | Accepted artifact kind | Media type | Maximum size | Required stage |
|
|
| --- | --- | --- | --- | --- |
|
|
| `npc_registry` | `dnd/npc-registry` | `application/json` | 1,048,576 bytes | extract and normalize |
|
|
| `scene_descriptions` | `dnd/scene-description-list` | `application/json` | 1,048,576 bytes | extract only |
|
|
| `combat_turns` | `dnd/combat-turn-list` | `application/json` | 1,048,576 bytes | extract only |
|
|
| `npc_occurrences` | `dnd/npc-occurrence-list` | `application/json` | 1,048,576 bytes | extract only |
|
|
| `location_registry` | `dnd/location-registry` | `application/json` | 1,048,576 bytes | location-occurrence extract and normalize |
|
|
| `item_registry` | `dnd/item-registry` | `application/json` | 1,048,576 bytes | item-occurrence extract and normalize |
|
|
|
|
Scene descriptions accept **party**, **players**, and **glossary**, but not
|
|
**roster**. NPC occurrences require **npc_registry** for both extraction and
|
|
normalization. Combat turns require **scene_descriptions** for extraction; the
|
|
normalized combat-turn module may use optional **npc_registry**. Location occurrences
|
|
require **location_registry** for extraction and normalization. Item occurrences require
|
|
**item_registry** for extraction and normalization. Enemy-event extraction requires all
|
|
four of its JSON artifact slots; its normalizer requires **npc_registry**.
|
|
The [complete example](../examples/dnd-complete.config.yml) shows the ordered
|
|
generated bindings.
|
|
|
|
## Production Module Keys
|
|
|
|
| Kind | Keys |
|
|
| --- | --- |
|
|
| Input | **seriatim** |
|
|
| Chunk | **generic**, **dnd/scenes** |
|
|
| Extract | **dnd/spells**, **dnd/npc-registry**, **dnd/combat-turns**, **dnd/item-occurrences**, **dnd/item-registry**, **dnd/npc-occurrences**, **dnd/scene-descriptions**, **dnd/enemy-events**, **dnd/location-registry**, **dnd/location-occurrences** |
|
|
| Merge | **appendorder** |
|
|
| Normalize | **noop**, **dnd/spells**, **dnd/npc-registry**, **dnd/combat-turns**, **dnd/item-occurrences**, **dnd/item-registry**, **dnd/npc-occurrences**, **dnd/scene-descriptions**, **dnd/enemy-events**, **dnd/location-registry**, **dnd/location-occurrences** |
|
|
| Output | **json** |
|
|
|
|
`dnd/scenes` and every D&D extractor are `llm_backed`. The
|
|
`dnd/npc-registry`, `dnd/location-registry`, and `dnd/item-registry`
|
|
normalizers are also `llm_backed` for bounded duplicate proposals; every other
|
|
D&D normalizer is `deterministic`. LLM-backed bindings use the effective
|
|
[PromptKit profile](#promptkit-profiles). The complete example binds each
|
|
registry in an earlier step before its occurrence consumer.
|
|
|
|
The D&D artifact contracts define each emitted schema:
|
|
[spells](integrations/dnd-spell-artifacts.md),
|
|
[NPC registry](integrations/dnd-npc-registry-artifacts.md),
|
|
[NPC occurrences](integrations/dnd-npc-occurrence-artifacts.md),
|
|
[combat turns](integrations/dnd-combat-turn-artifacts.md),
|
|
[item registry](integrations/dnd-item-registry-artifacts.md),
|
|
[item occurrences](integrations/dnd-item-occurrence-artifacts.md),
|
|
[scene descriptions](integrations/dnd-scene-description-artifacts.md),
|
|
[enemy events](integrations/dnd-enemy-event-artifacts.md),
|
|
[location registry](integrations/dnd-location-registry-artifacts.md), and
|
|
[location occurrences](integrations/dnd-location-occurrence-artifacts.md).
|
|
|
|
## Production Validator Keys And Default Chains
|
|
|
|
Available validator keys are:
|
|
|
|
| Family | Keys |
|
|
| --- | --- |
|
|
| Generic | **generic/always_accept**, **generic/always_reject**, **generic/valid_json**, **generic/valid_json_schema** |
|
|
| Spells | **extract/dnd/spells/shape**, **extract/dnd/spells/catalog**, **extract/dnd/spells/source_refs**, **extract/dnd/spells/source_relatedness** |
|
|
| NPC registry | **extract/dnd/npc-registry/shape**, **extract/dnd/npc-registry/source_refs**, **extract/dnd/npc-registry/source_relatedness**, **normalize/dnd/npc-registry/identity** |
|
|
| Combat turns | **extract/dnd/combat-turns/shape**, **extract/dnd/combat-turns/source_refs**, **extract/dnd/combat-turns/source_relatedness**, **normalize/dnd/combat-turns/invariants** |
|
|
| Item occurrences | **extract/dnd/item-occurrences/shape**, **extract/dnd/item-occurrences/registry**, **extract/dnd/item-occurrences/source_refs**, **extract/dnd/item-occurrences/source_relatedness**, **normalize/dnd/item-occurrences/invariants** |
|
|
| Item registry | **extract/dnd/item-registry/shape**, **extract/dnd/item-registry/source_refs**, **extract/dnd/item-registry/source_relatedness**, **normalize/dnd/item-registry/identity** |
|
|
| NPC occurrences | **extract/dnd/npc-occurrences/shape**, **extract/dnd/npc-occurrences/registry**, **extract/dnd/npc-occurrences/source_refs**, **extract/dnd/npc-occurrences/source_relatedness**, **normalize/dnd/npc-occurrences/invariants** |
|
|
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **extract/dnd/scene-descriptions/combat_semantics** (LLM-backed, opt-in), **normalize/dnd/scene-descriptions/invariants** |
|
|
| Enemy events | **extract/dnd/enemy-events/shape**, **extract/dnd/enemy-events/engagements**, **extract/dnd/enemy-events/source_refs**, **extract/dnd/enemy-events/source_relatedness**, **normalize/dnd/enemy-events/invariants** |
|
|
| Location registry | **extract/dnd/location-registry/shape**, **extract/dnd/location-registry/source_refs**, **extract/dnd/location-registry/source_relatedness**, **normalize/dnd/location-registry/identity** |
|
|
| Location occurrences | **extract/dnd/location-occurrences/shape**, **extract/dnd/location-occurrences/registry**, **extract/dnd/location-occurrences/source_refs**, **extract/dnd/location-occurrences/source_relatedness**, **normalize/dnd/location-occurrences/invariants** |
|
|
|
|
`extract/dnd/scene-descriptions/combat_semantics` is not in a production default chain. To opt in, replace the scene extractor validator chain with the current ordered chain plus the semantic validator last, and set a positive producer retry budget if a rejection should request a corrected scene:
|
|
|
|
~~~yaml
|
|
extract:
|
|
module: dnd/scene-descriptions
|
|
retries: 1
|
|
validators:
|
|
- module: generic/valid_json
|
|
- module: extract/dnd/scene-descriptions/shape
|
|
- module: extract/dnd/scene-descriptions/source_refs
|
|
- module: generic/valid_json_schema
|
|
- module: extract/dnd/scene-descriptions/source_relatedness
|
|
- module: extract/dnd/scene-descriptions/combat_semantics
|
|
retries: 1
|
|
~~~
|
|
|
|
An override replaces, rather than extends, the default chain. See [Module Bindings And Validators](#module-bindings-and-validators) for binding, profile, repair, retry, and failure-policy rules.
|
|
|
|
The validator retry shown above permits one additional execution against the
|
|
same scene candidate when the LLM-backed validator itself fails; it is separate
|
|
from both the extractor's producer retry and PromptKit structural repair.
|
|
|
|
When no override is configured, production D&D bindings use the following
|
|
ordered chains. Each row lists extract then normalize; spell chains are the
|
|
same at both stages.
|
|
|
|
| Lane | Extract | Normalize |
|
|
| --- | --- | --- |
|
|
| Spells | generic/valid_json, extract/dnd/spells/shape, extract/dnd/spells/catalog, extract/dnd/spells/source_refs, generic/valid_json_schema, extract/dnd/spells/source_relatedness | Same as extract |
|
|
| NPC registry | generic/valid_json, extract/dnd/npc-registry/shape, extract/dnd/npc-registry/source_refs, generic/valid_json_schema, extract/dnd/npc-registry/source_relatedness | generic/valid_json, extract/dnd/npc-registry/shape, normalize/dnd/npc-registry/identity, extract/dnd/npc-registry/source_refs, generic/valid_json_schema, extract/dnd/npc-registry/source_relatedness |
|
|
| Combat turns | generic/valid_json, extract/dnd/combat-turns/shape, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness | generic/valid_json, extract/dnd/combat-turns/shape, normalize/dnd/combat-turns/invariants, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness |
|
|
| Item occurrences | generic/valid_json, extract/dnd/item-occurrences/shape, extract/dnd/item-occurrences/registry, extract/dnd/item-occurrences/source_refs, generic/valid_json_schema, extract/dnd/item-occurrences/source_relatedness | generic/valid_json, extract/dnd/item-occurrences/shape, extract/dnd/item-occurrences/registry, normalize/dnd/item-occurrences/invariants, extract/dnd/item-occurrences/source_refs, generic/valid_json_schema, extract/dnd/item-occurrences/source_relatedness |
|
|
| Item registry | generic/valid_json, extract/dnd/item-registry/shape, extract/dnd/item-registry/source_refs, generic/valid_json_schema, extract/dnd/item-registry/source_relatedness | generic/valid_json, extract/dnd/item-registry/shape, normalize/dnd/item-registry/identity, extract/dnd/item-registry/source_refs, generic/valid_json_schema, extract/dnd/item-registry/source_relatedness |
|
|
| NPC occurrences | generic/valid_json, extract/dnd/npc-occurrences/shape, extract/dnd/npc-occurrences/registry, extract/dnd/npc-occurrences/source_refs, generic/valid_json_schema, extract/dnd/npc-occurrences/source_relatedness | generic/valid_json, extract/dnd/npc-occurrences/shape, extract/dnd/npc-occurrences/registry, normalize/dnd/npc-occurrences/invariants, extract/dnd/npc-occurrences/source_refs, generic/valid_json_schema, extract/dnd/npc-occurrences/source_relatedness |
|
|
| Scene descriptions | generic/valid_json, extract/dnd/scene-descriptions/shape, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness | generic/valid_json, extract/dnd/scene-descriptions/shape, normalize/dnd/scene-descriptions/invariants, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness |
|
|
| Enemy events | generic/valid_json, extract/dnd/enemy-events/shape, extract/dnd/enemy-events/engagements, extract/dnd/enemy-events/source_refs, generic/valid_json_schema, extract/dnd/enemy-events/source_relatedness | generic/valid_json, extract/dnd/enemy-events/shape, normalize/dnd/enemy-events/invariants, extract/dnd/enemy-events/source_refs, generic/valid_json_schema, extract/dnd/enemy-events/source_relatedness |
|
|
| Location registry | generic/valid_json, extract/dnd/location-registry/shape, extract/dnd/location-registry/source_refs, generic/valid_json_schema, extract/dnd/location-registry/source_relatedness | generic/valid_json, extract/dnd/location-registry/shape, normalize/dnd/location-registry/identity, extract/dnd/location-registry/source_refs, generic/valid_json_schema, extract/dnd/location-registry/source_relatedness |
|
|
| Location occurrences | generic/valid_json, extract/dnd/location-occurrences/shape, extract/dnd/location-occurrences/registry, extract/dnd/location-occurrences/source_refs, generic/valid_json_schema, extract/dnd/location-occurrences/source_relatedness | generic/valid_json, extract/dnd/location-occurrences/shape, extract/dnd/location-occurrences/registry, normalize/dnd/location-occurrences/invariants, extract/dnd/location-occurrences/source_refs, generic/valid_json_schema, extract/dnd/location-occurrences/source_relatedness |
|
|
|
|
Chains are only registered for the D&D extract and normalize modules shown
|
|
above; select an explicit override when a different compatible chain is
|
|
required.
|
|
|
|
## Validation
|
|
|
|
Validate a file and one pipeline before running it:
|
|
|
|
~~~sh
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-minimal.config.yml \
|
|
--pipeline dnd-session
|
|
~~~
|
|
|
|
Configuration validation rejects invalid YAML, unsupported fields, invalid
|
|
defaults or explicit CLI overrides, incompatible module keys, unknown options,
|
|
invalid reference bindings, missing required reference slots, invalid validator
|
|
overrides, and incompatible generated artifact handoffs. Use
|
|
[pipelines list](cli.md#pipelines-list) to inspect configured IDs.
|
|
|
|
## Inspection
|
|
|
|
Use [config effective](cli.md#config-effective) to write the selected composed
|
|
configuration as deterministic, redacted YAML. Use
|
|
[config explain](cli.md#config-explain) with a canonical JSON Pointer to inspect
|
|
the effective or historical provenance for one path, including a value displaced
|
|
by a selected overlay. Neither command resolves a pipeline, prepares modules,
|
|
materializes references, or requires provider credentials.
|