Align configuration documentation with native loading

This commit is contained in:
2026-11-17 22:42:53 +00:00
parent ff4bdaa483
commit 413b0d7134
16 changed files with 188 additions and 319 deletions

View File

@@ -12,10 +12,20 @@ Narratio resolves three YAML documents:
### `pipeline.yml`
When `--config` is omitted, search order is:
An explicit `--config` selects exactly that file, including a custom filename.
Otherwise ConfigKit searches these locations in order:
1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml`
1. The working directory, for `narratio.yml` then `narratio.yaml`.
2. The Go user-config directory's `narratio/` subdirectory, for those named
files followed by `config.yml`, `config.yaml`, `main.yml`, and `main.yaml`.
3. The Go user-config directory itself, for the named files.
4. `/usr/local/etc/narratio/` and then `/etc/narratio/`, each for named files
followed by the generic filenames above.
5. `/etc/`, for the named files.
On Linux, the user-config directory normally follows `XDG_CONFIG_HOME`, or
`~/.config` when it is unset. A selected invalid file fails loading; discovery
does not continue to another candidate.
### `campaign.yml`
@@ -32,15 +42,12 @@ When `--session` is omitted, local search order is:
1. `/usr/local/etc/narratio/session.yml`
2. `/etc/narratio/session.yml`
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
using configured object storage.
The downloaded remote session file is command-scoped: Narratio removes it after
the command finishes and records only the remote object provenance alongside
the durable copied session input.
If no local session exists and a `session_id` is known, Narratio loads the
session from configured object storage. It selects the prepared session in the
current committed run when available; if no current pointer exists or an older
commit has no prepared session, it falls back to the session-init `session.yml`
object. Downloaded session bytes are contextless snapshots, with remote object
provenance retained independently of locally materialized checksums.
### Read-only effective pipeline inspection
@@ -72,9 +79,10 @@ several files for one family member. The [CLI reference](cli.md#config-validate-
owns command syntax and output conventions.
`config diff` compares normalized field values rather than YAML text or source
ownership. It emits sorted `added`, `removed`, and `changed` records, uses
compact deterministic JSON values, treats lists atomically, and reports `no
differences` when the complete effective configurations are equal. Concrete
ownership. It emits native ordered `added`, `removed`, and `changed` records with
JSON Pointer paths and compact JSON values. Lists are compared atomically, and
changed regions may contain whole values. It reports `no differences` when the
complete effective configurations are equal. Concrete
family members and generated publish rules participate after expansion; moving
an equal value between eligible root/import sources does not create a
difference.
@@ -105,23 +113,14 @@ remote state with an unsafe legacy identity must be migrated before use.
- ConfigKit performs strict typed YAML decoding and accepts exactly one document;
unknown fields, duplicate keys, and trailing documents fail load.
- A pipeline file may explicitly import additive YAML fragments through the
root-only `configkit.imports` list. Imported files contribute fields to one
logical pipeline document; they do not override fields supplied by the root
or another import.
- An omitted overlay, an empty overlay option, and the exact name `default`
select the base configuration. Any other name selects
`overlays/<name>.yml` relative to the pipeline root. Move previously selected
`profiles/<name>.yml` files there; the old directory is not searched, and
`--profile` is no longer accepted for Narratio pipeline selection. Explicit
imports from a directory named `profiles` remain valid.
- Campaign, session, and canonical-party documents are strict single-source
ConfigKit schemas. They support sibling inheritance but do not enable imports,
selected overlays, or `configkit` directives.
- Local pipeline, campaign, session, and party documents may declare native
imports and sibling inheritance. Only the pipeline selects an overlay.
- An omitted overlay, empty overlay option, and exact `default` select the base;
another name selects `overlays/<name>.yml` relative to the pipeline root.
- Configured timeout and retry-delay durations must be positive. An omitted
artifact timeout continues to inherit its configured Scriptorium timeout.
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
- Session and party inputs inspected before typed loading are limited to 8 MiB.
- ConfigKit applies its acquisition limits to configuration sources and imports.
- Pipeline defaults are applied before validation.
- Defaults apply only to absent fields. Explicit empty strings, zero values,
`false`, and `null` retain ConfigKit's typed meaning and are then checked by
@@ -143,82 +142,74 @@ remote state with an unsafe legacy identity must be migrated before use.
### Pipeline imports and overlays
Large pipeline configurations may be split into explicitly named fragments and
may use fixed-path selected overlays:
Local pipeline, campaign, session, and party files can use source-root
`config.imports` and `config.sequences`. Every acquired document has its own
source-root controls, including imported fragments. Nested `config` keys are
ordinary data. `extends` is reserved for native sibling inheritance; parents
remain ordinary entries subject to Narratio's effective-domain validation.
```yaml
config:
imports:
- path: config/storage.yml
- path: config/integrations.yaml
campaigns:
root: /usr/local/share/narratio/campaigns
- path: artifacts/session-recap.yml
destination: [scriptorium, artifacts, session_recap]
required: true
- dir: artifacts/families
pattern: "*.yml"
```
The maintained [production/testing bundle](../examples/production-testing/pipeline.yml)
is a complete copyable example of this structure, including canonical-party
artifact families.
Imports support `path`, `dir`, and `glob` selectors, applicable filters and
relative destinations. Paths are relative to the declaring source. Missing or
empty selections are optional by default; use `required: true` when a selection
is essential. Selected invalid input fails. Destinations place content under
literal mapping keys, independent of filenames; nested imports can declare
relative destinations. Strict peer contributions must satisfy ConfigKit's
composition rules, and Narratio validates the resulting effective entries.
Imports are resolved relative to the directory containing the root pipeline
file and are loaded in declaration order. Narratio does not scan directories or
infer fragments. Each import must be a confined regular `.yml` or `.yaml` file:
absolute paths, traversal, symlinks, directories, duplicate files, and an
import of the root pipeline itself are rejected. Imported files cannot declare
another `configkit` control section.
A selected pipeline overlay is applied after base composition. It may declare
root `config.sequences` but cannot import other files. Ordinary overlay lists
replace the lower list; declared collections can use native `replace`, `append`,
or `upsert`. Upsert replaces a whole matching record; an empty append or upsert
does not clear the lower list. Mapping and identified-list inheritance follow
native sibling rules. See the pinned [ConfigKit configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
for full selection, composition, collection, and reserved-key semantics.
The [maintained bundle](../examples/production-testing/pipeline.yml) illustrates
these controls, family expansion, and a split roster.
Base merging is additive. A map may be extended by multiple files when every
leaf is distinct, but a scalar, list, or map/list/scalar kind cannot be claimed
more than once, even when the repeated values are identical. Conflict errors
name the full field path and every source that claimed it. These rules apply at
every map path, including artifact and artifact-family entries, so distinct
fields of one entry can compose across imports. Narratio validates the resulting
effective entry using its ordinary domain rules.
Relative host-resource paths, including integration configuration files,
workspace/campaign/spool/cache roots, campaign template and stable inputs, and
session stable inputs and local audio references, resolve from the effective
value's defining source. Inherited values follow their origin. When no distinct
usable source context exists, the main source is the fallback; if it too is
contextless, the invocation directory is used. Explicit CLI-relative file paths
use the invocation directory. Artifact outputs, publish destinations, logical
IDs, and binary names retain their domain meanings.
After base sources are composed and the selected overlay is applied, ConfigKit
resolves native sibling inheritance before typed decoding. Any mapping may use
`extends: <sibling-key>` to inherit from an exact key in the same enclosing map;
lookup does not search parent maps or unrelated scopes. The child keeps its own
key and overlays its explicit fields on the resolved parent. Mappings combine
recursively, while explicit scalar, list, false, zero, empty, and null values
remain the child's values. Parent entries remain ordinary configuration entries
and must satisfy Narratio's normal effective-domain validation; inheritance does
not create templates exempt from domain rules. Inherited explicit input counts
as present for absence-based defaults.
Use [`config sources`](./cli.md#config-validate-config-show-config-sources-config-artifacts-and-config-diff)
to inspect native effective contributors and inheritance origins; composite
artifact definitions may report multiple source paths.
The effective digest excludes source layout, overlay identity, and secret values;
credential reference names and resolved resource paths participate. Prepared
campaign, session, and party documents contain effective standalone values rather
than authored import directives. Session fields omitted by the author remain
eligible to inherit the receiving campaign's values.
The key `extends` is reserved structural input wherever ConfigKit is used,
including pipeline roots, imported files, selected overlay values, dynamic
artifact IDs, prompt-variable mappings, campaigns, sessions, and canonical-party
inputs. Rename any existing ordinary-data key named `extends`;
there is no opt-in or automatic conversion. See the pinned [ConfigKit v0.5.0
configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#sibling-inheritance)
for exact lookup, composition, presence, and provenance semantics.
### Upgrade from earlier configuration
Overlay names are case-sensitive and use ConfigKit's safe fixed-path naming
rules. Only the selected overlay is opened; a malformed unselected overlay does
not affect a base or different-overlay load. Maps merge recursively, overlay
scalars replace base scalars, and overlay lists replace base lists completely.
Explicit `false`, zero, empty strings, empty containers, and null values remain
explicit inputs and are not refilled by defaults. Overlay files cannot inherit
from or stack on other overlay files, import files, or declare another ConfigKit
control section. Values in the selected overlay may use native sibling
`extends` directives; those relationships resolve after overlay application.
- Rename source-root `configkit:` controls to `config:`. Nested `config` data
and the Go `configkit` package and secret tags have different meanings.
- Rename or move pipeline files to the discovery names above, or retain a custom
filename such as `pipeline.yml` and pass it with `--config`.
- Move conventional secrets to `/usr/local/etc/narratio/secrets`, supply them
through the environment, or pass the previous directory with `--secrets-dir`.
- Review relative resource references moved into fragments: their defining
source now supplies the base. Mark necessary imports `required: true`.
- Review lists and use `config.sequences` where append or whole-record upsert is
intended. Verify the effective result with the inspection commands.
- Prepared documents contain effective values; retain authored roots and their
companions separately for editing. Separately installed downstream tools own
their own configuration migrations.
During ConfigKit loading, Narratio applies fresh typed defaults, recorded
entry-level defaults and normalization, read-only file-local validation, and
ordinary path resolution before computing a deterministic effective
configuration digest. The digest represents the normalized, secret-free
runtime pipeline mapping; it excludes ConfigKit directives, source
provenance, overlay identity, and raw environment secret values. Equivalent
effective mappings therefore have the same digest regardless of how fields are
split among the root and imports.
An imported field has the same meaning it would have in a monolithic root
pipeline. In particular, ordinary relative pipeline paths continue to resolve
from the root pipeline directory, not from the importing fragment's directory.
These control-key changes require a major version for the eventual Narratio
release. This migration does not itself publish a release.
## Minimal Working Configuration
@@ -263,8 +254,8 @@ inputs:
stage range, or eligible for resume. Omit the field or set it to `null` when
that deployment should not require the credential.
- Every command that loads a pipeline accepts `--secrets-dir <path>`. When the
flag is omitted, Narratio uses a present `secrets/` directory beside the
resolved root `pipeline.yml`; an absent conventional directory is ignored.
flag is omitted, Narratio uses `/usr/local/etc/narratio/secrets` when present;
an absent conventional directory is ignored.
Explicit relative paths are resolved from the caller's working directory,
and a missing or empty explicit selection is an error.
- Each immediate regular file whose name is a valid environment-variable name
@@ -320,7 +311,7 @@ Rules:
| Field | Type | Required | Default / Rule |
| --- | --- | --- | --- |
| `configkit.imports[].path` | string | No | explicit additive pipeline fragment relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
| `config.imports` | list | No | native source-root import selections; see [imports and overlays](#pipeline-imports-and-overlays) |
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false`; selects the published run's workspace target for final cleanup and requires effective `pipeline.publish.enabled=true` and `pipeline.publish.upload_run=true` |
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
@@ -385,10 +376,10 @@ Rules:
| `pipeline.notarius.binary` | string | No | `notarius` |
| `pipeline.notarius.credentials[]` | list[secret reference] | No | empty; exact environment-variable names forwarded to Notarius |
| `pipeline.notarius.config_overlay` | string | No | unset; exact Notarius ConfigKit overlay selector |
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from their defining source |
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from their defining source |
| `pipeline.notarius.resume` | bool | No | `false`; request compatible Notarius checkpoint reuse during ordinary executed extraction |
| `pipeline.notarius.chunk_cache` | string | No | empty; when set, exactly `auto`, `bypass`, or `refresh` |
| `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries |
@@ -697,7 +688,7 @@ integration.
| `inputs.speakers_file` | string | Yes | stable input default |
| `inputs.autocorrect_file` | string | Yes | stable input default |
| `inputs.glossary_file` | string | Yes | stable input default |
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from `campaign.yml` |
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from their defining source |
| `inputs.spell_catalog_file` | string | No | optional spell-catalog overlay default; required when a Notarius reference selects `narratio.input.spell_catalog` |
### Session