Align configuration documentation with native loading
This commit is contained in:
@@ -59,8 +59,8 @@ Rules:
|
||||
base configuration, while an unknown named overlay fails. The same selection
|
||||
applies to all common-flag commands, including `regenerate-artifacts`.
|
||||
- `--secrets-dir` selects ConfigKit credential-file bootstrap explicitly. A
|
||||
relative value uses caller CWD. Without the flag, a present `secrets/`
|
||||
directory beside the resolved root pipeline is used; an absent one is skipped.
|
||||
relative value uses caller CWD. Without the flag, Narratio uses the optional
|
||||
`/usr/local/etc/narratio/secrets` directory.
|
||||
- `clean --all` cannot be combined with campaign/session selectors.
|
||||
- notification delivery is currently limited to the configured `noop` mode; see
|
||||
the [configuration reference](./config.md#notifications).
|
||||
|
||||
189
docs/config.md
189
docs/config.md
@@ -12,10 +12,20 @@ Narratio resolves three YAML documents:
|
||||
|
||||
### `pipeline.yml`
|
||||
|
||||
When `--config` is omitted, search order is:
|
||||
An explicit `--config` selects exactly that file, including a custom filename.
|
||||
Otherwise ConfigKit searches these locations in order:
|
||||
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
1. The working directory, for `narratio.yml` then `narratio.yaml`.
|
||||
2. The Go user-config directory's `narratio/` subdirectory, for those named
|
||||
files followed by `config.yml`, `config.yaml`, `main.yml`, and `main.yaml`.
|
||||
3. The Go user-config directory itself, for the named files.
|
||||
4. `/usr/local/etc/narratio/` and then `/etc/narratio/`, each for named files
|
||||
followed by the generic filenames above.
|
||||
5. `/etc/`, for the named files.
|
||||
|
||||
On Linux, the user-config directory normally follows `XDG_CONFIG_HOME`, or
|
||||
`~/.config` when it is unset. A selected invalid file fails loading; discovery
|
||||
does not continue to another candidate.
|
||||
|
||||
### `campaign.yml`
|
||||
|
||||
@@ -32,15 +42,12 @@ When `--session` is omitted, local search order is:
|
||||
1. `/usr/local/etc/narratio/session.yml`
|
||||
2. `/etc/narratio/session.yml`
|
||||
|
||||
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
|
||||
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||
|
||||
using configured object storage.
|
||||
|
||||
The downloaded remote session file is command-scoped: Narratio removes it after
|
||||
the command finishes and records only the remote object provenance alongside
|
||||
the durable copied session input.
|
||||
If no local session exists and a `session_id` is known, Narratio loads the
|
||||
session from configured object storage. It selects the prepared session in the
|
||||
current committed run when available; if no current pointer exists or an older
|
||||
commit has no prepared session, it falls back to the session-init `session.yml`
|
||||
object. Downloaded session bytes are contextless snapshots, with remote object
|
||||
provenance retained independently of locally materialized checksums.
|
||||
|
||||
### Read-only effective pipeline inspection
|
||||
|
||||
@@ -72,9 +79,10 @@ several files for one family member. The [CLI reference](cli.md#config-validate-
|
||||
owns command syntax and output conventions.
|
||||
|
||||
`config diff` compares normalized field values rather than YAML text or source
|
||||
ownership. It emits sorted `added`, `removed`, and `changed` records, uses
|
||||
compact deterministic JSON values, treats lists atomically, and reports `no
|
||||
differences` when the complete effective configurations are equal. Concrete
|
||||
ownership. It emits native ordered `added`, `removed`, and `changed` records with
|
||||
JSON Pointer paths and compact JSON values. Lists are compared atomically, and
|
||||
changed regions may contain whole values. It reports `no differences` when the
|
||||
complete effective configurations are equal. Concrete
|
||||
family members and generated publish rules participate after expansion; moving
|
||||
an equal value between eligible root/import sources does not create a
|
||||
difference.
|
||||
@@ -105,23 +113,14 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
|
||||
- ConfigKit performs strict typed YAML decoding and accepts exactly one document;
|
||||
unknown fields, duplicate keys, and trailing documents fail load.
|
||||
- A pipeline file may explicitly import additive YAML fragments through the
|
||||
root-only `configkit.imports` list. Imported files contribute fields to one
|
||||
logical pipeline document; they do not override fields supplied by the root
|
||||
or another import.
|
||||
- An omitted overlay, an empty overlay option, and the exact name `default`
|
||||
select the base configuration. Any other name selects
|
||||
`overlays/<name>.yml` relative to the pipeline root. Move previously selected
|
||||
`profiles/<name>.yml` files there; the old directory is not searched, and
|
||||
`--profile` is no longer accepted for Narratio pipeline selection. Explicit
|
||||
imports from a directory named `profiles` remain valid.
|
||||
- Campaign, session, and canonical-party documents are strict single-source
|
||||
ConfigKit schemas. They support sibling inheritance but do not enable imports,
|
||||
selected overlays, or `configkit` directives.
|
||||
- Local pipeline, campaign, session, and party documents may declare native
|
||||
imports and sibling inheritance. Only the pipeline selects an overlay.
|
||||
- An omitted overlay, empty overlay option, and exact `default` select the base;
|
||||
another name selects `overlays/<name>.yml` relative to the pipeline root.
|
||||
- Configured timeout and retry-delay durations must be positive. An omitted
|
||||
artifact timeout continues to inherit its configured Scriptorium timeout.
|
||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||
- Session and party inputs inspected before typed loading are limited to 8 MiB.
|
||||
- ConfigKit applies its acquisition limits to configuration sources and imports.
|
||||
- Pipeline defaults are applied before validation.
|
||||
- Defaults apply only to absent fields. Explicit empty strings, zero values,
|
||||
`false`, and `null` retain ConfigKit's typed meaning and are then checked by
|
||||
@@ -143,82 +142,74 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
|
||||
### Pipeline imports and overlays
|
||||
|
||||
Large pipeline configurations may be split into explicitly named fragments and
|
||||
may use fixed-path selected overlays:
|
||||
Local pipeline, campaign, session, and party files can use source-root
|
||||
`config.imports` and `config.sequences`. Every acquired document has its own
|
||||
source-root controls, including imported fragments. Nested `config` keys are
|
||||
ordinary data. `extends` is reserved for native sibling inheritance; parents
|
||||
remain ordinary entries subject to Narratio's effective-domain validation.
|
||||
|
||||
```yaml
|
||||
config:
|
||||
imports:
|
||||
- path: config/storage.yml
|
||||
- path: config/integrations.yaml
|
||||
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
- path: artifacts/session-recap.yml
|
||||
destination: [scriptorium, artifacts, session_recap]
|
||||
required: true
|
||||
- dir: artifacts/families
|
||||
pattern: "*.yml"
|
||||
```
|
||||
|
||||
The maintained [production/testing bundle](../examples/production-testing/pipeline.yml)
|
||||
is a complete copyable example of this structure, including canonical-party
|
||||
artifact families.
|
||||
Imports support `path`, `dir`, and `glob` selectors, applicable filters and
|
||||
relative destinations. Paths are relative to the declaring source. Missing or
|
||||
empty selections are optional by default; use `required: true` when a selection
|
||||
is essential. Selected invalid input fails. Destinations place content under
|
||||
literal mapping keys, independent of filenames; nested imports can declare
|
||||
relative destinations. Strict peer contributions must satisfy ConfigKit's
|
||||
composition rules, and Narratio validates the resulting effective entries.
|
||||
|
||||
Imports are resolved relative to the directory containing the root pipeline
|
||||
file and are loaded in declaration order. Narratio does not scan directories or
|
||||
infer fragments. Each import must be a confined regular `.yml` or `.yaml` file:
|
||||
absolute paths, traversal, symlinks, directories, duplicate files, and an
|
||||
import of the root pipeline itself are rejected. Imported files cannot declare
|
||||
another `configkit` control section.
|
||||
A selected pipeline overlay is applied after base composition. It may declare
|
||||
root `config.sequences` but cannot import other files. Ordinary overlay lists
|
||||
replace the lower list; declared collections can use native `replace`, `append`,
|
||||
or `upsert`. Upsert replaces a whole matching record; an empty append or upsert
|
||||
does not clear the lower list. Mapping and identified-list inheritance follow
|
||||
native sibling rules. See the pinned [ConfigKit configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
|
||||
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
|
||||
for full selection, composition, collection, and reserved-key semantics.
|
||||
The [maintained bundle](../examples/production-testing/pipeline.yml) illustrates
|
||||
these controls, family expansion, and a split roster.
|
||||
|
||||
Base merging is additive. A map may be extended by multiple files when every
|
||||
leaf is distinct, but a scalar, list, or map/list/scalar kind cannot be claimed
|
||||
more than once, even when the repeated values are identical. Conflict errors
|
||||
name the full field path and every source that claimed it. These rules apply at
|
||||
every map path, including artifact and artifact-family entries, so distinct
|
||||
fields of one entry can compose across imports. Narratio validates the resulting
|
||||
effective entry using its ordinary domain rules.
|
||||
Relative host-resource paths, including integration configuration files,
|
||||
workspace/campaign/spool/cache roots, campaign template and stable inputs, and
|
||||
session stable inputs and local audio references, resolve from the effective
|
||||
value's defining source. Inherited values follow their origin. When no distinct
|
||||
usable source context exists, the main source is the fallback; if it too is
|
||||
contextless, the invocation directory is used. Explicit CLI-relative file paths
|
||||
use the invocation directory. Artifact outputs, publish destinations, logical
|
||||
IDs, and binary names retain their domain meanings.
|
||||
|
||||
After base sources are composed and the selected overlay is applied, ConfigKit
|
||||
resolves native sibling inheritance before typed decoding. Any mapping may use
|
||||
`extends: <sibling-key>` to inherit from an exact key in the same enclosing map;
|
||||
lookup does not search parent maps or unrelated scopes. The child keeps its own
|
||||
key and overlays its explicit fields on the resolved parent. Mappings combine
|
||||
recursively, while explicit scalar, list, false, zero, empty, and null values
|
||||
remain the child's values. Parent entries remain ordinary configuration entries
|
||||
and must satisfy Narratio's normal effective-domain validation; inheritance does
|
||||
not create templates exempt from domain rules. Inherited explicit input counts
|
||||
as present for absence-based defaults.
|
||||
Use [`config sources`](./cli.md#config-validate-config-show-config-sources-config-artifacts-and-config-diff)
|
||||
to inspect native effective contributors and inheritance origins; composite
|
||||
artifact definitions may report multiple source paths.
|
||||
The effective digest excludes source layout, overlay identity, and secret values;
|
||||
credential reference names and resolved resource paths participate. Prepared
|
||||
campaign, session, and party documents contain effective standalone values rather
|
||||
than authored import directives. Session fields omitted by the author remain
|
||||
eligible to inherit the receiving campaign's values.
|
||||
|
||||
The key `extends` is reserved structural input wherever ConfigKit is used,
|
||||
including pipeline roots, imported files, selected overlay values, dynamic
|
||||
artifact IDs, prompt-variable mappings, campaigns, sessions, and canonical-party
|
||||
inputs. Rename any existing ordinary-data key named `extends`;
|
||||
there is no opt-in or automatic conversion. See the pinned [ConfigKit v0.5.0
|
||||
configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#sibling-inheritance)
|
||||
for exact lookup, composition, presence, and provenance semantics.
|
||||
### Upgrade from earlier configuration
|
||||
|
||||
Overlay names are case-sensitive and use ConfigKit's safe fixed-path naming
|
||||
rules. Only the selected overlay is opened; a malformed unselected overlay does
|
||||
not affect a base or different-overlay load. Maps merge recursively, overlay
|
||||
scalars replace base scalars, and overlay lists replace base lists completely.
|
||||
Explicit `false`, zero, empty strings, empty containers, and null values remain
|
||||
explicit inputs and are not refilled by defaults. Overlay files cannot inherit
|
||||
from or stack on other overlay files, import files, or declare another ConfigKit
|
||||
control section. Values in the selected overlay may use native sibling
|
||||
`extends` directives; those relationships resolve after overlay application.
|
||||
- Rename source-root `configkit:` controls to `config:`. Nested `config` data
|
||||
and the Go `configkit` package and secret tags have different meanings.
|
||||
- Rename or move pipeline files to the discovery names above, or retain a custom
|
||||
filename such as `pipeline.yml` and pass it with `--config`.
|
||||
- Move conventional secrets to `/usr/local/etc/narratio/secrets`, supply them
|
||||
through the environment, or pass the previous directory with `--secrets-dir`.
|
||||
- Review relative resource references moved into fragments: their defining
|
||||
source now supplies the base. Mark necessary imports `required: true`.
|
||||
- Review lists and use `config.sequences` where append or whole-record upsert is
|
||||
intended. Verify the effective result with the inspection commands.
|
||||
- Prepared documents contain effective values; retain authored roots and their
|
||||
companions separately for editing. Separately installed downstream tools own
|
||||
their own configuration migrations.
|
||||
|
||||
During ConfigKit loading, Narratio applies fresh typed defaults, recorded
|
||||
entry-level defaults and normalization, read-only file-local validation, and
|
||||
ordinary path resolution before computing a deterministic effective
|
||||
configuration digest. The digest represents the normalized, secret-free
|
||||
runtime pipeline mapping; it excludes ConfigKit directives, source
|
||||
provenance, overlay identity, and raw environment secret values. Equivalent
|
||||
effective mappings therefore have the same digest regardless of how fields are
|
||||
split among the root and imports.
|
||||
|
||||
An imported field has the same meaning it would have in a monolithic root
|
||||
pipeline. In particular, ordinary relative pipeline paths continue to resolve
|
||||
from the root pipeline directory, not from the importing fragment's directory.
|
||||
These control-key changes require a major version for the eventual Narratio
|
||||
release. This migration does not itself publish a release.
|
||||
|
||||
## Minimal Working Configuration
|
||||
|
||||
@@ -263,8 +254,8 @@ inputs:
|
||||
stage range, or eligible for resume. Omit the field or set it to `null` when
|
||||
that deployment should not require the credential.
|
||||
- Every command that loads a pipeline accepts `--secrets-dir <path>`. When the
|
||||
flag is omitted, Narratio uses a present `secrets/` directory beside the
|
||||
resolved root `pipeline.yml`; an absent conventional directory is ignored.
|
||||
flag is omitted, Narratio uses `/usr/local/etc/narratio/secrets` when present;
|
||||
an absent conventional directory is ignored.
|
||||
Explicit relative paths are resolved from the caller's working directory,
|
||||
and a missing or empty explicit selection is an error.
|
||||
- Each immediate regular file whose name is a valid environment-variable name
|
||||
@@ -320,7 +311,7 @@ Rules:
|
||||
|
||||
| Field | Type | Required | Default / Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `configkit.imports[].path` | string | No | explicit additive pipeline fragment relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
|
||||
| `config.imports` | list | No | native source-root import selections; see [imports and overlays](#pipeline-imports-and-overlays) |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false`; selects the published run's workspace target for final cleanup and requires effective `pipeline.publish.enabled=true` and `pipeline.publish.upload_run=true` |
|
||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||
@@ -385,10 +376,10 @@ Rules:
|
||||
| `pipeline.notarius.binary` | string | No | `notarius` |
|
||||
| `pipeline.notarius.credentials[]` | list[secret reference] | No | empty; exact environment-variable names forwarded to Notarius |
|
||||
| `pipeline.notarius.config_overlay` | string | No | unset; exact Notarius ConfigKit overlay selector |
|
||||
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
|
||||
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from their defining source |
|
||||
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
||||
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
||||
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
||||
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from their defining source |
|
||||
| `pipeline.notarius.resume` | bool | No | `false`; request compatible Notarius checkpoint reuse during ordinary executed extraction |
|
||||
| `pipeline.notarius.chunk_cache` | string | No | empty; when set, exactly `auto`, `bypass`, or `refresh` |
|
||||
| `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries |
|
||||
@@ -697,7 +688,7 @@ integration.
|
||||
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from `campaign.yml` |
|
||||
| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from their defining source |
|
||||
| `inputs.spell_catalog_file` | string | No | optional spell-catalog overlay default; required when a Notarius reference selects `narratio.input.spell_catalog` |
|
||||
|
||||
### Session
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Canonical Party Input
|
||||
|
||||
`party.yml` is a campaign-owned roster input. It must be a single, versioned
|
||||
`narratio.party.v1` document when Narratio resolves a pipeline, campaign, and
|
||||
session together.
|
||||
`party.yml` is a campaign-owned roster input. Its main document must declare
|
||||
`narratio.party.v1` when Narratio resolves the roster. Local party files may
|
||||
import fragments; fragments need no version header.
|
||||
|
||||
```yaml
|
||||
schema_version: narratio.party.v1
|
||||
@@ -48,8 +48,8 @@ The configuration rules and source-relative path behavior are defined in the
|
||||
|
||||
## Derived players document
|
||||
|
||||
During `prepare`, Narratio copies the canonical party source bytes unchanged
|
||||
to `inputs/party.yml` and writes this deterministic players-only projection to
|
||||
During `prepare`, Narratio renders the effective roster as standalone canonical
|
||||
`inputs/party.yml` and writes this deterministic players-only projection to
|
||||
`inputs/players.yml`:
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -37,6 +37,8 @@ Restore planner action kinds:
|
||||
|
||||
Planner behavior:
|
||||
|
||||
- prepared campaign, session, and party inputs are restored from their declared
|
||||
immutable run objects to their canonical local input paths;
|
||||
- a new-protocol restore uses only the selected commit's declared artifact set;
|
||||
each action carries that artifact's immutable key, checksum, size, and
|
||||
generation;
|
||||
|
||||
@@ -1,210 +1,81 @@
|
||||
# Configuration Internals
|
||||
|
||||
User-visible fields, defaults, and selection behavior belong in the
|
||||
[Configuration Reference](../config.md). This document describes the internal
|
||||
pipeline-loading boundary implemented by `internal/config`.
|
||||
User-visible fields, defaults, discovery, and selection belong in the
|
||||
[Configuration Reference](../config.md). This document describes the
|
||||
`internal/config` loading and resolution boundary.
|
||||
|
||||
## Pipeline Loading
|
||||
|
||||
`LoadPipeline` assembles and validates a pipeline in this order:
|
||||
Commands bootstrap secrets once before loading, then select an explicit file or
|
||||
native required discovery. ConfigKit loads typed defaults, source-root imports,
|
||||
and the selected overlay, resolves inheritance, and strictly decodes the
|
||||
result. Narratio applies recorded defaults, normalization, resource resolution,
|
||||
and effective-domain validation. The selected root, source contexts, metadata,
|
||||
and authoring result are retained for provenance. Family expansion and checks
|
||||
requiring the campaign occur after campaign and party resolution.
|
||||
|
||||
1. Give ConfigKit a fresh typed `PipelineConfig` default value, the root file,
|
||||
enabled imports, and the caller's overlay selection. Omitted, empty, and
|
||||
exact `default` selections use the base; named overlays use the fixed
|
||||
`overlays/<name>.yml` convention.
|
||||
2. ConfigKit parses the root and declared imports, additively merges base
|
||||
sources, applies only the selected overlay, resolves native sibling
|
||||
inheritance, and strictly decodes the result.
|
||||
3. The definition resolver applies recorded entry-level and inherited defaults,
|
||||
runs deterministic domain normalization, records derived publish destinations,
|
||||
and resolves ordinary paths relative to the root pipeline file.
|
||||
4. The definition validator performs read-only checks that require only the
|
||||
pipeline file. Checks requiring canonical-party family expansion remain at
|
||||
the existing explicit pipeline/campaign resolution boundary.
|
||||
5. Narratio retains public ConfigKit metadata for source ownership, attaches
|
||||
runtime-only state, and computes the normalized effective digest. Artifact
|
||||
and family validation applies to the composed effective entries.
|
||||
|
||||
This ordering preserves monolithic configuration behavior. Moving a field to
|
||||
an imported fragment changes its source ownership, not its path base, default,
|
||||
or schema semantics.
|
||||
Resource resolution consults native entry origins and source contexts before
|
||||
normalization. An effective source supplies the base when usable; otherwise the
|
||||
main source, then the captured invocation directory, is the fallback. Explicit
|
||||
CLI file references use the invocation directory. Derived values record their
|
||||
dependencies. See the [configuration reference](../config.md#pipeline-imports-and-overlays)
|
||||
for the resource classifications and native authoring contract.
|
||||
|
||||
## Loaded Context Resolution
|
||||
|
||||
`LoadedPipelineCampaign` carries one already composed pipeline and its selected
|
||||
campaign into session resolution. `LoadSessionWithPipelineCampaignOptions`
|
||||
loads a local session against that context, while
|
||||
`ResolveLoadedPipelineCampaign` also accepts an already loaded remote session
|
||||
or no session while a caller retrieves one. Compatibility loaders route through
|
||||
these functions after their initial pipeline and campaign reads.
|
||||
`LoadedPipelineCampaign` carries the selected pipeline and campaign into local
|
||||
or remote session resolution. Local campaign, session, and party loads enable
|
||||
imports through their own ConfigKit definitions; only the pipeline selects an
|
||||
overlay. Native optional search distinguishes an absent local session from an
|
||||
invalid selected file. Remote session bytes have no incidental download-directory
|
||||
context, and retain object provenance. Effective session strings are checked for
|
||||
unresolved template tokens, and campaign/session identity and stable-input rules
|
||||
remain Narratio-owned.
|
||||
|
||||
Application commands own pipeline and campaign discovery, campaign-file versus
|
||||
registry selection, and the corresponding mutual-exclusion rules. Once they
|
||||
have a `LoadedPipelineCampaign`, local session discovery and remote-session
|
||||
download retain that exact pipeline object and its private provenance. Removing
|
||||
a temporary downloaded session file therefore cannot invalidate the resolved
|
||||
pipeline or campaign context.
|
||||
Campaign and session loaders retain detached document values after their own
|
||||
resolution. The session document is captured before campaign defaults are copied
|
||||
into runtime session state. Configuration-owned renderers produce standalone
|
||||
campaign and session YAML, preserving omitted session overrides. Prepare uses
|
||||
those renderers and canonical party rendering for materialization and resume
|
||||
comparison.
|
||||
|
||||
Campaign and session files use independent single-source ConfigKit definitions
|
||||
with imports and selected overlays disabled; native sibling inheritance remains
|
||||
available within each document. Canonical-party input follows the same
|
||||
single-source rule. Campaign files normally load directly from their source;
|
||||
session files are first read through the 8 MiB bounded-read boundary because the
|
||||
application must reject unresolved template placeholders before passing an exact
|
||||
byte snapshot to ConfigKit. Local and remotely downloaded session bytes then use
|
||||
the same typed loader and retain the existing post-load identity expectations,
|
||||
stable-input precedence, and source-relative path semantics.
|
||||
## Inspection And Provenance
|
||||
|
||||
The application also has a separate read-only inspection resolver for `config
|
||||
validate`, `config show`, `config sources`, and `config artifacts`. It uses the
|
||||
same production root, overlay, and campaign selection functions, but never routes
|
||||
through session discovery,
|
||||
remote-session download, adapter composition, workspace
|
||||
initialization, manifest access, or cleanup. A pipeline with retained artifact
|
||||
family declarations must resolve its selected campaign before ordinary pipeline
|
||||
validation, which expands its canonical-party members and generated publish
|
||||
rules. A pipeline without those declarations may be validated by itself.
|
||||
Secret-directory bootstrap remains invocation-scoped and happens before the
|
||||
ordinary ConfigKit load because explicit typed references must resolve even for
|
||||
inspection.
|
||||
Read-only inspection shares production pipeline and campaign selection without
|
||||
session discovery, workspace initialization, adapters, or manifest access. After
|
||||
applicable campaign and family resolution, a safe typed final-domain DTO contains
|
||||
concrete artifacts and generated publish rules. It stores credential reference
|
||||
names rather than typed secrets. A defaults-backed native result freezes this
|
||||
DTO without resolving credentials again; `WriteEffective` supplies show and
|
||||
prepared pipeline YAML, and deterministic JSON of the same DTO supplies the
|
||||
versioned semantic digest. Resolved resources participate in that digest, while
|
||||
source layout and secret values do not.
|
||||
|
||||
`MarshalEffectivePipeline` is the configuration-owned projection for `config
|
||||
show`. It serializes one explicit typed safe projection with the existing YAML
|
||||
encoder. The projection has no slots for resolution-only artifact family
|
||||
declarations or private provenance. The result has one trailing newline;
|
||||
commands do not marshal runtime objects directly.
|
||||
|
||||
`EffectivePipelineSources` and `EffectiveCampaignSources` provide the separate
|
||||
safe provenance projection for `config sources`. Pipeline ownership begins with
|
||||
the complete logical field paths projected from the schema and classifies
|
||||
each contributor as root, import, overlay, or centralized default. It reads
|
||||
ConfigKit's effective origins and inheritance locations when building each
|
||||
projection, so inherited values retain their contributors and controlling
|
||||
directives without a second cached leaf-ownership table. The
|
||||
projection replaces generated concrete member paths with paired family and
|
||||
canonical-party records, and does the same for generated publish rules.
|
||||
Campaign records identify campaign-owned fields and party inputs; canonical
|
||||
derived players point to the party source. The application command only joins these sorted
|
||||
records with selection metadata and never reparses configuration files.
|
||||
|
||||
`EffectiveArtifactCatalog` projects the same finalized concrete artifact map for
|
||||
`config artifacts`. It derives source identity from each map key, carries sorted
|
||||
effective dependencies, aggregates ConfigKit definition contributors, and adds
|
||||
family/character/party provenance for generated members. Family source evidence
|
||||
can name multiple contributing files. It does not inspect files named after
|
||||
artifacts or infer another runtime graph.
|
||||
|
||||
`config diff` uses a thin paired loader that performs two independent ConfigKit
|
||||
loads. When campaign
|
||||
resolution is needed, the command loads one selected campaign and party and
|
||||
expands both effective pipelines from that same party value. The configuration
|
||||
owner projects each normalized effective mapping directly from that same typed
|
||||
projection into sorted logical paths using JSON number preservation;
|
||||
mapping leaves are compared individually while sequence values remain atomic.
|
||||
Values are compact deterministic JSON representations for command output, not
|
||||
raw YAML fragments, ownership records, or secret material. A differing digest
|
||||
with no projected difference is treated as an internal consistency error.
|
||||
|
||||
Campaign context construction reads the campaign-owned party source through
|
||||
`ParseParty`. The required canonical party retains its raw bytes and normalized
|
||||
roster in runtime-only `ResolvedParty` provenance. Canonical resolution creates
|
||||
a deterministic `derived_from_party` players projection. The party is the only
|
||||
configured roster authority; player and session-party fields are unsupported.
|
||||
Authoring results separately supply native origins, source histories, and empty
|
||||
optional-import evidence to source and artifact inspection. Family and party
|
||||
context identifies generated entries without reparsing sources. Diff fully
|
||||
resolves each side against the same selected campaign and party, then uses
|
||||
ConfigKit `Diff` on the safe snapshots. The command presents native JSON Pointer
|
||||
paths and safe compact values.
|
||||
|
||||
## Canonical Party Domain
|
||||
|
||||
`ParseParty` is the package-owned boundary for parsing a party source. Its
|
||||
single-source ConfigKit definition disables imports and overlays, requires
|
||||
`schema_version` through `VersionRule`, and strictly decodes the
|
||||
character-keyed typed schema before applying Narratio's party-domain checks.
|
||||
Successful metadata coordinates recover authored character-map order without a
|
||||
second YAML parse; class and alias slice order comes directly from typed decode.
|
||||
The canonical value retains a separate exact byte copy of its source so
|
||||
consumers can materialize the authored party document without reserializing it.
|
||||
Its `PlayersYAML` method deterministically derives the ID-sorted, versioned
|
||||
players-only projection.
|
||||
|
||||
An unversioned source is rejected before resolution, family expansion, or
|
||||
prepare-stage materialization.
|
||||
File-backed party loading enables imports; `ParseParty` accepts contextless bytes.
|
||||
The shared definition checks the main document's version and Narratio validates
|
||||
the effective character roster. Native acquisition order and source coordinates
|
||||
establish roster order, favoring each child's own declaration. A deterministic
|
||||
renderer writes standalone canonical party YAML from that roster; the players
|
||||
projection and family expansion use the same resolved roster. The external format
|
||||
is defined in the [party contract](../integrations/party.md).
|
||||
|
||||
## Validation Ownership
|
||||
|
||||
Configuration validation remains within `internal/config`, with the public
|
||||
entry points and pipeline-domain sequencing separated from the private owners
|
||||
for core pipeline/session rules, external adapters, publish/storage, Notarius,
|
||||
Scriptorium, and shared scalar/path checks. These are package-local ownership
|
||||
boundaries rather than extension points or a general validation framework.
|
||||
|
||||
ConfigKit supplies composition, inheritance, presence, typed-secret, and
|
||||
provenance mechanics; Narratio supplies the domain rules applied to the
|
||||
effective decoded entries. There is no separate authoring-history acceptance
|
||||
gate: contributors and declaration history remain inspection evidence, not an
|
||||
additional validity condition.
|
||||
The [configuration reference](../config.md#pipeline-imports-and-overlays)
|
||||
documents sibling syntax, reserved-key migration, and the operator-visible
|
||||
inspection behavior.
|
||||
|
||||
Validation is read-only, fails fast, and does not define which of several
|
||||
independent invalid fields is reported first. Map-backed declarations are
|
||||
nevertheless traversed deterministically. Reusable domain normalization is an
|
||||
explicit preceding operation for both loaded and programmatically constructed
|
||||
pipelines. Notarius normalization commits canonical scalar, reference, and output
|
||||
values only after the complete normalization succeeds; its validator then checks
|
||||
that candidate without changing it. Scriptorium validates artifact declarations
|
||||
and outputs, dependencies and inputs, variables, referenced outputs, and the final
|
||||
dependency graph in deterministic artifact order.
|
||||
|
||||
Diagnostics identify the relevant semantic field. Merge conflicts also
|
||||
retain every claiming source needed to correct an additive ownership conflict.
|
||||
Complete prose, punctuation, formatting, and private validation choreography are
|
||||
not compatibility contracts.
|
||||
|
||||
## Diagnostics And Runtime Metadata
|
||||
|
||||
Syntax, duplicate-key, merge, overlay, and schema failures include the
|
||||
relevant source name and full field path. Additive conflicts report every
|
||||
claiming source so operators can repair the split without repeatedly
|
||||
rediscovering additional conflicts.
|
||||
|
||||
The loaded pipeline retains private runtime metadata for the absolute root
|
||||
path, ordered imports, selected overlay name, source, and path, original
|
||||
ConfigKit metadata and source snapshots, and effective digest. Leaf source
|
||||
records are projected from that retained native metadata when requested. This
|
||||
metadata does not participate in YAML decoding or alter the public
|
||||
configuration model.
|
||||
|
||||
The effective digest is SHA-256 over a versioned deterministic JSON encoding of
|
||||
the same typed safe projection used by show and diff. Runtime Notarius paths remain absolute for
|
||||
execution, but the digest substitutes their normalized logical values captured
|
||||
before root-relative resolution, so relocating an equivalent configuration
|
||||
bundle does not change provenance. Because ConfigKit and resolution metadata
|
||||
are private, the digest excludes source layout, overlay name, and ownership.
|
||||
Configuration stores credentials as ConfigKit typed references. A successful
|
||||
load resolves their values inside the opaque typed boundary, while this safe
|
||||
projection serializes only `EnvName()` identities. Credential-value changes are
|
||||
therefore never hashed; reference-name changes remain ordinary provenance.
|
||||
The optional Notarius credential follows the same path even for disabled or
|
||||
resumable extraction; normalization copies the opaque reference without reading
|
||||
or resolving it again. Direct YAML serialization has a configuration-owned safe
|
||||
projection so only the reference name can leave this boundary.
|
||||
`recomputePipelineEffectiveDigest` is the single package-owned refresh point
|
||||
for later runtime expansion.
|
||||
|
||||
## Test Surfaces
|
||||
|
||||
`pipeline_configkit_test.go` exercises real imports and fixed-path overlays,
|
||||
base selection, unselected-overlay isolation, source attribution, cross-file
|
||||
inheritance, and effective composition. Existing focused configuration
|
||||
tests protect defaults, explicit-value semantics, strict decoding, validation,
|
||||
metadata, and root-relative path behavior. Application configuration-loader tests protect the
|
||||
single-read boundary by changing the pipeline file after its initial load and
|
||||
confirming local session resolution retains the original pipeline. Other
|
||||
configuration tests continue to protect defaults and validation after assembly.
|
||||
`party_test.go` protects the versioned typed party schema, domain invariants,
|
||||
authored character order, retained bytes, and deterministic players projection
|
||||
without involving campaign or runtime wiring.
|
||||
`party_resolution_test.go` protects campaign-owned party loading, canonical
|
||||
input restrictions, source provenance, and virtual players input selection.
|
||||
ConfigKit owns composition, inheritance, presence, typed-secret resolution, and
|
||||
provenance. Narratio owns validation and normalization of effective domain
|
||||
values. Source histories are inspection evidence, not an additional acceptance
|
||||
gate. Focused configuration and application tests exercise imports, resource
|
||||
contexts, snapshots, secret-safe inspection, party resolution, and materialized
|
||||
inputs through the production boundaries.
|
||||
|
||||
## Artifact Family Resolution
|
||||
|
||||
|
||||
@@ -107,7 +107,7 @@ so any earlier replacement can stale a prior cleanup result.
|
||||
## Focused Documentation
|
||||
|
||||
- [Configuration Internals](configuration.md): pipeline imports and overlays,
|
||||
field ownership, decoding, and root-relative path semantics.
|
||||
field ownership, decoding, and source-relative resource semantics.
|
||||
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
|
||||
failure behavior, and test surfaces.
|
||||
- [Artifact Internals](artifacts.md): source identities, runtime catalog,
|
||||
|
||||
@@ -39,8 +39,8 @@ Materialize canonical current-session inputs before processing stages.
|
||||
- materializes a configured spell catalog with checksum and provenance, or
|
||||
safely removes an obsolete canonical spell catalog and its manifest record
|
||||
when the effective input is omitted.
|
||||
- copies the validated canonical party bytes unchanged and deterministically
|
||||
generates the prepared players projection.
|
||||
- renders standalone effective campaign, session, and canonical party documents,
|
||||
and deterministically generates the prepared players projection.
|
||||
- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements.
|
||||
- clears managed `previous/` state on every invocation, then, when requirements exist:
|
||||
- resolves the pointer-selected previous source through the shared resolver;
|
||||
@@ -71,15 +71,15 @@ previous-artifact requirement set. A change reruns prepare and
|
||||
stales its fixed descendants. Existing successful records without this
|
||||
evidence rerun once when selected.
|
||||
|
||||
Workspace, spool, and cache placement and absolute source relocation are not
|
||||
semantic when logical selection, canonical names, and bytes are equivalent.
|
||||
Workspace, spool, and cache placement are not semantic when logical selection,
|
||||
canonical names, and effective bytes are equivalent.
|
||||
The fingerprint deliberately does not read or rehash large audio. Prepared
|
||||
input checksums remain the content provenance. Before reusing success, prepare
|
||||
validates every durable prepared copy and compares current stable-input bytes,
|
||||
canonical party and derived-player bytes, local audio membership/checksums, or
|
||||
S3 key/size/entity-tag identity with that provenance. Source relocation with
|
||||
equivalent names and bytes remains reusable; changed or unavailable evidence
|
||||
causes a normal prepare rerun.
|
||||
effective campaign/session, canonical party and derived-player bytes, local audio
|
||||
membership/checksums, or S3 key/size/entity-tag identity with that provenance.
|
||||
Source relocation with equivalent effective documents, names, and bytes remains
|
||||
reusable; changed or unavailable evidence causes a normal prepare rerun.
|
||||
|
||||
## Related Contracts And Tests
|
||||
|
||||
|
||||
@@ -15,8 +15,8 @@ Upload run/session outputs to object storage and atomically advance remote curre
|
||||
|
||||
## Outputs
|
||||
|
||||
- uploaded session manifest, declared run-local diagnostics, and selected
|
||||
publish outputs;
|
||||
- uploaded session manifest, prepared campaign/session/party inputs, declared
|
||||
run-local diagnostics, and selected publish outputs;
|
||||
- uploaded durable previous-session cache files when present;
|
||||
- immutable run-scoped commit manifest; and
|
||||
- current commit pointer, written last.
|
||||
@@ -34,7 +34,9 @@ and optional rules, destination resolution, and collision inputs live in
|
||||
remote verification live in `publish_transfer.go`; committed-manifest,
|
||||
immutable-commit, and current-pointer encoding and writes live in
|
||||
`publish_commit.go`. `publish_output_plan.go` owns the deterministic, I/O-free
|
||||
five-outcome model and its secret-free metadata projection.
|
||||
five-outcome model and its secret-free metadata projection. Prepared input
|
||||
objects are committed under the immutable run; remote session loading selects
|
||||
the committed standalone session through the current pointer.
|
||||
|
||||
- when publishing or run upload is disabled, completes successfully with no
|
||||
outputs and records explanatory metadata. This is not an explicit self-skip:
|
||||
|
||||
@@ -626,6 +626,8 @@ Publish commit model:
|
||||
|
||||
- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
|
||||
audio and the run-local Notarius staging bundle;
|
||||
- uploads prepared campaign, session, and party documents under the run's
|
||||
`prepare/inputs/` prefix, when recorded by the manifest;
|
||||
- uploads configured published outputs and `previous/**` cache files into the
|
||||
same immutable run scope, including only explicitly configured extraction
|
||||
lanes;
|
||||
@@ -714,8 +716,8 @@ plan, so it is not a network-free operation.
|
||||
|
||||
Default restore scope:
|
||||
|
||||
- the committed session manifest and the committed transcript/artifact objects
|
||||
declared by the selected remote commit
|
||||
- the committed session manifest, prepared campaign/session/party documents,
|
||||
and transcript/artifact objects declared by the selected remote commit
|
||||
- `previous/**` when needed by configured previous-session artifact inputs;
|
||||
each is selected from its exact committed `published_output` declaration
|
||||
|
||||
@@ -821,7 +823,7 @@ ConfigKit.
|
||||
Every pipeline-loading command bootstraps credentials once, before the first
|
||||
configuration load and before any adapter is constructed. `--secrets-dir`
|
||||
selects an explicit directory (relative paths use caller CWD). Otherwise a
|
||||
present `secrets/` beside the resolved root pipeline is used, while an absent
|
||||
present `/usr/local/etc/narratio/secrets` directory is used, while an absent
|
||||
conventional directory is skipped. Explicit missing/empty selections and
|
||||
invalid existing directories fail the command. ConfigKit follows a symlink used
|
||||
as the selected directory, but immediate entries must be regular non-symlink
|
||||
|
||||
@@ -183,7 +183,7 @@ fail-fast validation of the effective configuration.
|
||||
values remain present and are evaluated by Narratio's domain rules.
|
||||
- Validation rejects invalid configuration before stage execution where
|
||||
practical.
|
||||
- Root-owned imports and one selected overlay resolve deterministically through
|
||||
- Local document imports and one selected pipeline overlay resolve through
|
||||
the configuration owner; commands do not implement their own merge rules.
|
||||
- Native ConfigKit metadata and supported resolution callbacks provide presence,
|
||||
inheritance, and provenance information without a parallel loading or
|
||||
|
||||
@@ -365,7 +365,7 @@ Load examples through production definitions. Check dependent logical IDs,
|
||||
concrete family members, testing differences, and standalone prepared inputs.
|
||||
Keep examples synthetic and credential-free.
|
||||
|
||||
## Stage 11: Align canonical documentation and project-specific policy
|
||||
## Stage 11: Align canonical documentation and project-specific policy [x]
|
||||
|
||||
Read documentation/OSS/release policies, affected references, and the approved
|
||||
roadmap. Finish current-behavior documentation for the assembled implementation.
|
||||
|
||||
@@ -90,12 +90,12 @@ Symptom:
|
||||
|
||||
Likely cause:
|
||||
|
||||
- the campaign `party.yml` is an unversioned legacy document, or names a
|
||||
- the campaign party main document is unversioned, or names a
|
||||
schema other than `narratio.party.v1`.
|
||||
|
||||
Safe fix:
|
||||
|
||||
- replace the file with one exact, single-document
|
||||
- replace the main file with a versioned
|
||||
[`narratio.party.v1` party](./integrations/party.md); do not add a version
|
||||
marker to an otherwise incompatible legacy structure. The versioned party is
|
||||
the only configured roster authority.
|
||||
@@ -118,7 +118,8 @@ narratio config sources --config /path/pipeline.yml --overlay testing
|
||||
Add `--campaign` or `--campaign-file` when the pipeline has party-driven
|
||||
artifact families. The output identifies each effective logical field's root,
|
||||
import, overlay, default, campaign, party, or family source without printing
|
||||
the field value or credential contents.
|
||||
the field value or credential contents. Empty optional import selections are
|
||||
reported separately; selected invalid files still fail.
|
||||
|
||||
Safe fix:
|
||||
|
||||
@@ -958,7 +959,7 @@ Symptom:
|
||||
Likely causes:
|
||||
|
||||
- missing, unreadable, or invalid explicit `--secrets-dir`;
|
||||
- invalid entries in the conventional `secrets/` directory beside the root pipeline;
|
||||
- invalid entries in `/usr/local/etc/narratio/secrets`;
|
||||
- missing credential environment variables;
|
||||
- invalid S3 endpoint/bucket settings.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user