configkit/docs/roadmap/overlays.md

# Single-file named overlays

Status: strict composition, direct catalog structure, and file-level version
checking are settled. Remaining decisions are listed below. Not implemented.
Implementation baseline: `ce6005c`.

## Purpose

Support a single `overlays.yml` containing named overlay definitions alongside
the existing `overlays/<name>.yml` convention. Small applications can keep their
profiles together, while applications with larger configurations can split them
into individual files or contribute through both forms.

Applications continue to select one overlay name. ConfigKit selects the matching
contributions, strictly composes them, and applies the resulting overlay once.
This follows [ADR 0001](../adr/0001-separate-composition-and-overlay.md): peer
composition and intentional precedence remain separate operations.

This roadmap extends file-based selected overlays. The selected contributions
compose as peers at the overlay stage; overlay definitions do not acquire
underlay or main-input precedence tiers. A configuration-wide catalog populated
through inline `config.overlays`, ordinary imports, or application underlays is
outside this feature's scope. No new application or source-control key is reserved.

## Current baseline

The [selected-overlay contract](../config.md#selected-overlays) currently loads
one YAML mapping from `overlays/<name>.yml`. Overlay names are not paths. Empty
selection and the exact lowercase name `default` select no overlay. A selected
missing file fails; there is no extension fallback or overlay stack.

Ordinary loads and explicit layered overlay contexts locate that file beneath
the configured root. Layered main-derived and fallback contexts use their
resolved base for lookup, with a separate containment root. The implemented
[main-relative context decision](../adr/0022-main-relative-context.md) supplies
the context needed by this feature; it does not implement aggregate overlays.

The existing loader in `internal/engine/selected_overlay.go` acquires and parses
the file, checks overlay directives and source-version policy, and applies its
payload with the effective collection policy. Overlays apply after base
composition and main-version checking, before inheritance, defaults, decoding,
CLI overrides, application resolution, and final validation. Source records,
locations, resource contexts, history, repetition checks, limits, and handle
cleanup already cover the individual-file form.

## Targeted end state

### File layout and selection

The catalog maps overlay names directly to overlay bodies, without an enclosing
`overlays` key. It may also carry file-level version metadata under the application's
configured version path. The following example omits optional version metadata:

```yaml
# overlays.yml — planned format
production:
  service:
    workers: 8
development:
  service:
    workers: 1
    log_level: debug
```

Selecting `production` takes only that entry as the catalog's contribution.
The same selection may also acquire this existing-format file:

```yaml
# overlays/production.yml
service:
  log_level: warning
```

Strict composition produces this one overlay payload:

```yaml
service:
  workers: 8
  log_level: warning
```

If the individual file instead also defines `service.workers: 12`, ordinary
strict-composition conflict rules apply. File order grants no precedence.
The other catalog entries do not contribute to the selected configuration.
OD-1 settles the direct catalog structure. Version checks are file-scoped as
described below; metadata-name collisions, collection-control placement, and
the validation boundary remain in OD-2 and OD-3.

Use the same lookup directory for the catalog and the existing `overlays/`
directory. Preserve ordinary versus layered lookup semantics, explicit context
precedence, main-relative and fallback behavior, namespace identity, symlink
handling, and containment. The new filename variants and activation behavior
remain in OD-4 and OD-5.

No selected overlay means neither form is read. With a selection, either form
may supply the named overlay. An absent catalog entry is not an error when the
individual file supplies the selection, and a missing individual file is not an
error when the catalog supplies it. If neither contributes, report the existing
missing-overlay condition. A selected empty mapping is a present contribution,
not a missing overlay. Acquisition or parsing failures are not a reason to
silently prefer the other form.

### File-level version checks

**Settled:** a version declaration checks whether a physical configuration file
declares an acceptable specification version. It does not independently version
each overlay body. Use the existing application-configured `Definition.Version`
path and source-version mode; do not hard-code the spelling `version`.

For an application using the root `version` key, a catalog can contain:

```yaml
version: 2
production:
  service:
    workers: 8
development:
  service:
    workers: 1
```

Apply the configured source-version policy once to each acquired physical file,
before its contribution enters strict composition. The catalog check is independent
of which name is selected and also applies when the acquired catalog lacks that
name. No selected overlay still means neither form is read or checked. In enabled
modes, evaluate the catalog declaration at its physical document root, not inside
the selected body. Preserve the existing distinction between check-when-present
and required modes and the application's authority to accept the declared value.

An acquired individual overlay file has its own declaration and existing version
processing. When both files contribute, each is checked independently; one file
cannot satisfy another's required declaration. Do not check or require separate
version declarations in the catalog's named bodies. Keep catalog version metadata
out of the applied overlay, and keep its evidence attached to the actual catalog
source and authored location. Body values remain subject to normal protection of
the application's version path when applying the overlay.

This decision concerns stale-file detection. Collection controls describe overlay
operations and need not share the version declaration's scope. OD-2 separates the
remaining control-placement and catalog-metadata recognition decisions.

### Strict composition, then one overlay application

**Settled:** contributions for the selected name from both forms are peers.
Strictly compose them before applying their combined result to the lower
configuration. Do not apply them sequentially as precedence layers, suppress
one form, or reject a shared name merely because both forms contribute.

Reuse existing composition and collection-policy rules. Effective lower-tier
identity and applicable selected-overlay declarations must be available when
composing the peers. Disjoint identified records can compose; conflicting
identities among peers fail. The combined collection then uses its applicable
`replace`, `append`, or `upsert` operation against the lower value once.
Unidentified sequences retain existing strict-composition and overlay behavior.

Resolve collection declarations across the contributions using the existing
same-tier rules; do not independently apply each contribution's mode to the
lower configuration. Keep the existing rules for version-path protection and
forbidding overlay imports. This feature adds neither overlay inheritance nor
selection of several overlay names.

### Processing and source evidence

Acquire selected contributions within the existing load, after base processing
reaches the overlay phase. Preserve subsequent lifecycle order and normal
failure behavior. Reuse parser and composition representations rather than
decoding the catalog into application structs and reconstructing a document.

The selected catalog body's application paths are relative to that body, while
its locations still identify the real catalog file and authored line/column.
Diagnostics must distinguish application coordinates from the catalog wrapper;
never fabricate a physical file such as `overlays.yml/production`. Relative
resource contexts follow the physical declaring file, so resources beside
`overlays.yml` and beside `overlays/production.yml` have different bases.
Preserve both contributors when mappings combine, and normal field origins and
history when the completed overlay is applied.

Keep real acquisition accounting: parse the acquired catalog under existing
YAML limits and charge its complete bytes and parsed nodes, including unselected
bodies. Each acquired physical file contributes a source record; unused names
are not separately acquired sources. Preserve repetition/containment checks,
detached inspection, and cleanup on every exit. Do not add discovery-attempt
history or a new inspection API solely for this feature.

Use deterministic acquisition order for source IDs and sequence assembly. The
recommended order is catalog contribution, then individual-file contribution;
this orders evidence and composed records, not precedence. Reuse existing
diagnostic classifications where they fit. Record file-level version evidence
once per acquired source; collection-control scope remains open in OD-2.

## Work from the baseline

1. **Settle the remaining contract choices.** Resolve the decisions below,
   including activation compatibility, and prepare examples for each source
   form, combined contributions, source-version modes, and collection controls.
2. **Separate overlay acquisition from application.** Adapt the selected-overlay
   loader to acquire optional contributions and distinguish missing selection
   from acquisition failure. Retain existing context selection and access rules.
3. **Check files and select catalog bodies without losing source evidence.** Parse
   and apply file-level source-version policy to each acquired file. Separate
   catalog metadata from named definitions, select the requested body, and retain
   actual locations. Apply the chosen collection-control scope and account for
   the complete acquired file.
4. **Compose peers and apply once.** Reuse strict composition with the lower
   collection policy and selected declarations, then feed the composed result
   into the existing overlay operation. Preserve contributor origins, history,
   sensitivity, version protection, and resource contexts.
5. **Verify and document.** Add public integration coverage and maintained
   consumer examples; update selected-overlay contracts, public declaration
   comments as needed, and internal ownership documentation when implemented.

This is a dependency order for a later implementation plan, not a prescribed
commit layout. Do not change existing ADR decisions merely to document a new
storage form. Review any durable ownership change separately if one emerges.

## Validation and completion

Cover the public loading boundary with:

- Catalog only, individual file only, both, neither, a catalog without the
  selected name, selected empty mappings, and inactive/default selections.
- Complementary mappings, conflicting fields, identified-sequence composition,
  duplicate identities, mode declarations, and one final collection operation.
- Ordinary and layered loading, discovered/explicit/non-file/absent mains,
  explicit overlay contexts, main-derived contexts, fallback contexts, wider
  roots, host and native filesystems, and existing link-handling capabilities.
- Invalid YAML, catalog shape, selected/unselected body validation, source-version
  modes, protected version paths, and forbidden directives under the settled rules.
- One file-level version evaluation per acquired catalog, independent of the
  selected name, including a catalog without that name. Cover both source forms
  under required and check-when-present modes, configured metadata paths and name
  collisions, metadata exclusion from payloads, and actual declaration locations.
- Actual source paths and locations, origins and contributor history, sensitive
  values, distinct resource bases, source IDs, and detached snapshots.
- Missing versus unreadable sources, repetition, complete catalog accounting,
  limits, and cleanup on failure. No duplicate load lifecycle callbacks.
- Existing individual-file workflows and the activation compatibility cases
  settled in OD-5.

Use focused tests and normal repository checks during implementation. No new
benchmark campaign or release gate is required. Completion means applications
can select one named overlay from either or both forms with the same composition,
precedence, and inspection model.

## Settled Decisions

### OD-1: Catalog structure — settled

Use the direct name-to-body mapping illustrated above, with no enclosing
`overlays` mapping. The filename identifies the document's role; each named body
remains recognizable as an individual overlay file's contents. The top-level
mapping is a selection container and is not passed to application decoding.
Overlay names retain existing spelling and selection rules, including empty
selection and exact lowercase `default` meaning no overlay.

This settles the storage structure within the narrower file-based feature.
File-level version metadata coexists with the named entries. Its recognition and
name-collision rules, and collection-control placement, remain subject to OD-2.

## Open Decisions

### OD-2: File-level versions and overlay controls — partially settled

**Settled:** source-version checking operates once per acquired physical file,
using the existing application rule and mode. The catalog's declaration belongs
at its configured file-level path, not inside every named body. Collection-control
placement is a separate decision. The targeted end state above records the
version behavior; the remaining questions follow.

#### Collection-control placement

**Recommendation:** interpret `config.sequences` inside the selected body, using
application-relative destinations such as `/prompts`. Process each selected
contribution's controls before strict composition under existing same-tier rules.
This keeps an overlay's collection operations beside its payload and preserves
their meaning when moving that payload between forms. The file-level version
declaration stays separate. Imports remain forbidden under existing overlay rules;
nested application data does not activate controls.

**Alternative:** place shared collection controls at catalog level and apply them
to every selected body. This reduces repetition where profiles use identical
operations, but needs metadata-name rules, makes operations less local to their
payloads, and differs from the structure of individual overlay files.

#### Version metadata recognition and name collisions

**Recommendation:** when the application configures a version path, give that
path a defined metadata role in the catalog and reserve its enclosing top-level
entry from overlay-name selection. For a root `version` path this reserves only
the catalog name `version`; for a nested path it makes the enclosing entry a
metadata container. This avoids guessing whether a value is a header or a profile.
Do not reserve a hard-coded version name when no version rule is configured.

Specify how recognition works in main-only mode separately from whether a checker
runs, keeping metadata out of overlay payloads. Retain support for existing version
path forms; concrete examples must cover nested paths and retained empty ancestors
after header consumption. These rules are still open, not implicit additions to
the settled file-level check.

**Alternative:** reserve only the exact version path and allow other fields in
its enclosing entry to form an overlay body. This permits more names and avoids
reserving a whole metadata container, but gives one mapping both metadata and
profile roles. Selection, header removal, and empty-body behavior become harder
to explain and need explicit rules.

### OD-3: Validation of unselected bodies

**Recommendation:** require a valid YAML mapping catalog and look up the selected
name. Preserve parser-level validation of the whole file, including duplicate
mapping-key rules and limits. Require the selected body to satisfy normal overlay
mapping and directive rules, but do not run those semantic checks or application
callbacks for unselected bodies. File-level version policy still applies to the
acquired catalog independently of selection. Final application validation assesses
the resulting effective configuration, not each profile in isolation.

This keeps a load focused on the requested configuration while acknowledging
that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog
contains no definitions; selected null/scalar/sequence bodies follow the existing
mapping-root rules rather than becoming shorthand for absence.

**Alternative:** validate every body's shape, name, and directives when
reading the catalog. This catches unused-profile defects earlier, but makes one
profile's load depend on unrelated definitions and expands semantic validation.
A whole-catalog lint facility could serve that purpose separately if requested.

### OD-4: Filename variants

**Recommendation:** initially recognize exactly `overlays.yml` and the existing
`overlays/<name>.yml`. This matches current overlay spelling without adding a
separate extension-precedence decision. Both recognized forms compose as peers.

**Alternative:** also recognize `.yaml`. This accommodates common YAML filenames,
but requires deciding whether multiple extensions are additional peers or fallback
alternatives. If adopted, prefer strict peer composition over hidden precedence;
document stable acquisition order and collisions consistently for both forms.

### OD-5: Activation and compatibility

**Recommendation:** use the existing selected overlay name to activate lookup of
both forms, without another option in the common workflow. The storage layout
should not require consumers to implement different selection APIs. Document
that an acquired catalog participates in parsing even if its selected name is
absent, and that conflicting selected contributions fail strict composition.

Before committing to release compatibility, assess existing deployments that
have an `overlays.yml` previously ignored by ConfigKit: automatic participation
can expose invalid content or conflicts alongside an otherwise successful
individual-file load. The [release policy](../policy/release.md#versioning-and-compatibility)
preserves successful valid configurations; merely accepting additional layouts
does not resolve this interaction. No release version is selected here.

**Alternative:** add explicit catalog enablement, preserving existing directory-only
lookup until selected by the application. This provides a clear compatibility
boundary and lets consumers adopt the feature deliberately, but introduces an
extra setting for the common layout. Once enabled, contributions from both forms
must still compose strictly; this alternative does not reopen the settled
coexistence decision.