Align configuration documentation with native loading
This commit is contained in:
189
docs/config.md
189
docs/config.md
@@ -12,10 +12,20 @@ Narratio resolves three YAML documents:
|
||||
|
||||
### `pipeline.yml`
|
||||
|
||||
When `--config` is omitted, search order is:
|
||||
An explicit `--config` selects exactly that file, including a custom filename.
|
||||
Otherwise ConfigKit searches these locations in order:
|
||||
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
1. The working directory, for `narratio.yml` then `narratio.yaml`.
|
||||
2. The Go user-config directory's `narratio/` subdirectory, for those named
|
||||
files followed by `config.yml`, `config.yaml`, `main.yml`, and `main.yaml`.
|
||||
3. The Go user-config directory itself, for the named files.
|
||||
4. `/usr/local/etc/narratio/` and then `/etc/narratio/`, each for named files
|
||||
followed by the generic filenames above.
|
||||
5. `/etc/`, for the named files.
|
||||
|
||||
On Linux, the user-config directory normally follows `XDG_CONFIG_HOME`, or
|
||||
`~/.config` when it is unset. A selected invalid file fails loading; discovery
|
||||
does not continue to another candidate.
|
||||
|
||||
### `campaign.yml`
|
||||
|
||||
@@ -32,15 +42,12 @@ When `--session` is omitted, local search order is:
|
||||
1. `/usr/local/etc/narratio/session.yml`
|
||||
2. `/etc/narratio/session.yml`
|
||||
|
||||
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
|
||||
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||
|
||||
using configured object storage.
|
||||
|
||||
The downloaded remote session file is command-scoped: Narratio removes it after
|
||||
the command finishes and records only the remote object provenance alongside
|
||||
the durable copied session input.
|
||||
If no local session exists and a `session_id` is known, Narratio loads the
|
||||
session from configured object storage. It selects the prepared session in the
|
||||
current committed run when available; if no current pointer exists or an older
|
||||
commit has no prepared session, it falls back to the session-init `session.yml`
|
||||
object. Downloaded session bytes are contextless snapshots, with remote object
|
||||
provenance retained independently of locally materialized checksums.
|
||||
|
||||
### Read-only effective pipeline inspection
|
||||
|
||||
@@ -72,9 +79,10 @@ several files for one family member. The [CLI reference](cli.md#config-validate-
|
||||
owns command syntax and output conventions.
|
||||
|
||||
`config diff` compares normalized field values rather than YAML text or source
|
||||
ownership. It emits sorted `added`, `removed`, and `changed` records, uses
|
||||
compact deterministic JSON values, treats lists atomically, and reports `no
|
||||
differences` when the complete effective configurations are equal. Concrete
|
||||
ownership. It emits native ordered `added`, `removed`, and `changed` records with
|
||||
JSON Pointer paths and compact JSON values. Lists are compared atomically, and
|
||||
changed regions may contain whole values. It reports `no differences` when the
|
||||
complete effective configurations are equal. Concrete
|
||||
family members and generated publish rules participate after expansion; moving
|
||||
an equal value between eligible root/import sources does not create a
|
||||
difference.
|
||||
@@ -105,23 +113,14 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
|
||||
- ConfigKit performs strict typed YAML decoding and accepts exactly one document;
|
||||
unknown fields, duplicate keys, and trailing documents fail load.
|
||||
- A pipeline file may explicitly import additive YAML fragments through the
|
||||
root-only `configkit.imports` list. Imported files contribute fields to one
|
||||
logical pipeline document; they do not override fields supplied by the root
|
||||
or another import.
|
||||
- An omitted overlay, an empty overlay option, and the exact name `default`
|
||||
select the base configuration. Any other name selects
|
||||
`overlays/<name>.yml` relative to the pipeline root. Move previously selected
|
||||
`profiles/<name>.yml` files there; the old directory is not searched, and
|
||||
`--profile` is no longer accepted for Narratio pipeline selection. Explicit
|
||||
imports from a directory named `profiles` remain valid.
|
||||
- Campaign, session, and canonical-party documents are strict single-source
|
||||
ConfigKit schemas. They support sibling inheritance but do not enable imports,
|
||||
selected overlays, or `configkit` directives.
|
||||
- Local pipeline, campaign, session, and party documents may declare native
|
||||
imports and sibling inheritance. Only the pipeline selects an overlay.
|
||||
- An omitted overlay, empty overlay option, and exact `default` select the base;
|
||||
another name selects `overlays/<name>.yml` relative to the pipeline root.
|
||||
- Configured timeout and retry-delay durations must be positive. An omitted
|
||||
artifact timeout continues to inherit its configured Scriptorium timeout.
|
||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||
- Session and party inputs inspected before typed loading are limited to 8 MiB.
|
||||
- ConfigKit applies its acquisition limits to configuration sources and imports.
|
||||
- Pipeline defaults are applied before validation.
|
||||
- Defaults apply only to absent fields. Explicit empty strings, zero values,
|
||||
`false`, and `null` retain ConfigKit's typed meaning and are then checked by
|
||||
@@ -143,82 +142,74 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
|
||||
### Pipeline imports and overlays
|
||||
|
||||
Large pipeline configurations may be split into explicitly named fragments and
|
||||
may use fixed-path selected overlays:
|
||||
Local pipeline, campaign, session, and party files can use source-root
|
||||
`config.imports` and `config.sequences`. Every acquired document has its own
|
||||
source-root controls, including imported fragments. Nested `config` keys are
|
||||
ordinary data. `extends` is reserved for native sibling inheritance; parents
|
||||
remain ordinary entries subject to Narratio's effective-domain validation.
|
||||
|
||||
```yaml
|
||||
config:
|
||||
imports:
|
||||
- path: config/storage.yml
|
||||
- path: config/integrations.yaml
|
||||
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
- path: artifacts/session-recap.yml
|
||||
destination: [scriptorium, artifacts, session_recap]
|
||||
required: true
|
||||
- dir: artifacts/families
|
||||
pattern: "*.yml"
|
||||
```
|
||||
|
||||
The maintained [production/testing bundle](../examples/production-testing/pipeline.yml)
|
||||
is a complete copyable example of this structure, including canonical-party
|
||||
artifact families.
|
||||
Imports support `path`, `dir`, and `glob` selectors, applicable filters and
|
||||
relative destinations. Paths are relative to the declaring source. Missing or
|
||||
empty selections are optional by default; use `required: true` when a selection
|
||||
is essential. Selected invalid input fails. Destinations place content under
|
||||
literal mapping keys, independent of filenames; nested imports can declare
|
||||
relative destinations. Strict peer contributions must satisfy ConfigKit's
|
||||
composition rules, and Narratio validates the resulting effective entries.
|
||||
|
||||
Imports are resolved relative to the directory containing the root pipeline
|
||||
file and are loaded in declaration order. Narratio does not scan directories or
|
||||
infer fragments. Each import must be a confined regular `.yml` or `.yaml` file:
|
||||
absolute paths, traversal, symlinks, directories, duplicate files, and an
|
||||
import of the root pipeline itself are rejected. Imported files cannot declare
|
||||
another `configkit` control section.
|
||||
A selected pipeline overlay is applied after base composition. It may declare
|
||||
root `config.sequences` but cannot import other files. Ordinary overlay lists
|
||||
replace the lower list; declared collections can use native `replace`, `append`,
|
||||
or `upsert`. Upsert replaces a whole matching record; an empty append or upsert
|
||||
does not clear the lower list. Mapping and identified-list inheritance follow
|
||||
native sibling rules. See the pinned [ConfigKit configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
|
||||
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
|
||||
for full selection, composition, collection, and reserved-key semantics.
|
||||
The [maintained bundle](../examples/production-testing/pipeline.yml) illustrates
|
||||
these controls, family expansion, and a split roster.
|
||||
|
||||
Base merging is additive. A map may be extended by multiple files when every
|
||||
leaf is distinct, but a scalar, list, or map/list/scalar kind cannot be claimed
|
||||
more than once, even when the repeated values are identical. Conflict errors
|
||||
name the full field path and every source that claimed it. These rules apply at
|
||||
every map path, including artifact and artifact-family entries, so distinct
|
||||
fields of one entry can compose across imports. Narratio validates the resulting
|
||||
effective entry using its ordinary domain rules.
|
||||
Relative host-resource paths, including integration configuration files,
|
||||
workspace/campaign/spool/cache roots, campaign template and stable inputs, and
|
||||
session stable inputs and local audio references, resolve from the effective
|
||||
value's defining source. Inherited values follow their origin. When no distinct
|
||||
usable source context exists, the main source is the fallback; if it too is
|
||||
contextless, the invocation directory is used. Explicit CLI-relative file paths
|
||||
use the invocation directory. Artifact outputs, publish destinations, logical
|
||||
IDs, and binary names retain their domain meanings.
|
||||
|
||||
After base sources are composed and the selected overlay is applied, ConfigKit
|
||||
resolves native sibling inheritance before typed decoding. Any mapping may use
|
||||
`extends: <sibling-key>` to inherit from an exact key in the same enclosing map;
|
||||
lookup does not search parent maps or unrelated scopes. The child keeps its own
|
||||
key and overlays its explicit fields on the resolved parent. Mappings combine
|
||||
recursively, while explicit scalar, list, false, zero, empty, and null values
|
||||
remain the child's values. Parent entries remain ordinary configuration entries
|
||||
and must satisfy Narratio's normal effective-domain validation; inheritance does
|
||||
not create templates exempt from domain rules. Inherited explicit input counts
|
||||
as present for absence-based defaults.
|
||||
Use [`config sources`](./cli.md#config-validate-config-show-config-sources-config-artifacts-and-config-diff)
|
||||
to inspect native effective contributors and inheritance origins; composite
|
||||
artifact definitions may report multiple source paths.
|
||||
The effective digest excludes source layout, overlay identity, and secret values;
|
||||
credential reference names and resolved resource paths participate. Prepared
|
||||
campaign, session, and party documents contain effective standalone values rather
|
||||
than authored import directives. Session fields omitted by the author remain
|
||||
eligible to inherit the receiving campaign's values.
|
||||
|
||||
The key `extends` is reserved structural input wherever ConfigKit is used,
|
||||
including pipeline roots, imported files, selected overlay values, dynamic
|
||||
artifact IDs, prompt-variable mappings, campaigns, sessions, and canonical-party
|
||||
inputs. Rename any existing ordinary-data key named `extends`;
|
||||
there is no opt-in or automatic conversion. See the pinned [ConfigKit v0.5.0
|
||||
configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#sibling-inheritance)
|
||||
for exact lookup, composition, presence, and provenance semantics.
|
||||
### Upgrade from earlier configuration
|
||||
|
||||
Overlay names are case-sensitive and use ConfigKit's safe fixed-path naming
|
||||
rules. Only the selected overlay is opened; a malformed unselected overlay does
|
||||
not affect a base or different-overlay load. Maps merge recursively, overlay
|
||||
scalars replace base scalars, and overlay lists replace base lists completely.
|
||||
Explicit `false`, zero, empty strings, empty containers, and null values remain
|
||||
explicit inputs and are not refilled by defaults. Overlay files cannot inherit
|
||||
from or stack on other overlay files, import files, or declare another ConfigKit
|
||||
control section. Values in the selected overlay may use native sibling
|
||||
`extends` directives; those relationships resolve after overlay application.
|
||||
- Rename source-root `configkit:` controls to `config:`. Nested `config` data
|
||||
and the Go `configkit` package and secret tags have different meanings.
|
||||
- Rename or move pipeline files to the discovery names above, or retain a custom
|
||||
filename such as `pipeline.yml` and pass it with `--config`.
|
||||
- Move conventional secrets to `/usr/local/etc/narratio/secrets`, supply them
|
||||
through the environment, or pass the previous directory with `--secrets-dir`.
|
||||
- Review relative resource references moved into fragments: their defining
|
||||
source now supplies the base. Mark necessary imports `required: true`.
|
||||
- Review lists and use `config.sequences` where append or whole-record upsert is
|
||||
intended. Verify the effective result with the inspection commands.
|
||||
- Prepared documents contain effective values; retain authored roots and their
|
||||
companions separately for editing. Separately installed downstream tools own
|
||||
their own configuration migrations.
|
||||
|
||||
During ConfigKit loading, Narratio applies fresh typed defaults, recorded
|
||||
entry-level defaults and normalization, read-only file-local validation, and
|
||||
ordinary path resolution before computing a deterministic effective
|
||||
configuration digest. The digest represents the normalized, secret-free
|
||||
runtime pipeline mapping; it excludes ConfigKit directives, source
|
||||
provenance, overlay identity, and raw environment secret values. Equivalent
|
||||
effective mappings therefore have the same digest regardless of how fields are
|
||||
split among the root and imports.
|
||||
|
||||
An imported field has the same meaning it would have in a monolithic root
|
||||
pipeline. In particular, ordinary relative pipeline paths continue to resolve
|
||||
from the root pipeline directory, not from the importing fragment's directory.
|
||||
These control-key changes require a major version for the eventual Narratio
|
||||
release. This migration does not itself publish a release.
|
||||
|
||||
## Minimal Working Configuration
|
||||
|
||||
@@ -263,8 +254,8 @@ inputs:
|
||||
stage range, or eligible for resume. Omit the field or set it to `null` when
|
||||
that deployment should not require the credential.
|
||||
- Every command that loads a pipeline accepts `--secrets-dir <path>`. When the
|
||||
flag is omitted, Narratio uses a present `secrets/` directory beside the
|
||||
resolved root `pipeline.yml`; an absent conventional directory is ignored.
|
||||
flag is omitted, Narratio uses `/usr/local/etc/narratio/secrets` when present;
|
||||
an absent conventional directory is ignored.
|
||||
Explicit relative paths are resolved from the caller's working directory,
|
||||
and a missing or empty explicit selection is an error.
|
||||
- Each immediate regular file whose name is a valid environment-variable name
|
||||
@@ -320,7 +311,7 @@ Rules:
|
||||
|
||||
| Field | Type | Required | Default / Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `configkit.imports[].path` | string | No | explicit additive pipeline fragment relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
|
||||
| `config.imports` | list | No | native source-root import selections; see [imports and overlays](#pipeline-imports-and-overlays) |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false`; selects the published run's workspace target for final cleanup and requires effective `pipeline.publish.enabled=true` and `pipeline.publish.upload_run=true` |
|
||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||
@@ -385,10 +376,10 @@ Rules:
|
||||
| `pipeline.notarius.binary` | string | No | `notarius` |
|
||||
| `pipeline.notarius.credentials[]` | list[secret reference] | No | empty; exact environment-variable names forwarded to Notarius |
|
||||
| `pipeline.notarius.config_overlay` | string | No | unset; exact Notarius ConfigKit overlay selector |
|
||||
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
|
||||
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from their defining source |
|
||||
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
||||
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
||||
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
||||
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from their defining source |
|
||||
| `pipeline.notarius.resume` | bool | No | `false`; request compatible Notarius checkpoint reuse during ordinary executed extraction |
|
||||
| `pipeline.notarius.chunk_cache` | string | No | empty; when set, exactly `auto`, `bypass`, or `refresh` |
|
||||
| `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries |
|
||||
@@ -697,7 +688,7 @@ integration.
|
||||
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from `campaign.yml` |
|
||||
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from their defining source |
|
||||
| `inputs.spell_catalog_file` | string | No | optional spell-catalog overlay default; required when a Notarius reference selects `narratio.input.spell_catalog` |
|
||||
|
||||
### Session
|
||||
|
||||
Reference in New Issue
Block a user