configkit/docs/roadmap/overlays.md

# Single-file named overlays

Status: proposal; strict composition between the two forms is accepted. Remaining
decisions are listed below. Not implemented. Implementation baseline: `ce6005c`.

## Purpose

Support a single `overlays.yml` containing named overlay definitions alongside
the existing `overlays/<name>.yml` convention. Small applications can keep their
profiles together, while applications with larger configurations can split them
into individual files or contribute through both forms.

Applications continue to select one overlay name. ConfigKit selects the matching
contributions, strictly composes them, and applies the resulting overlay once.
This follows [ADR 0001](../adr/0001-separate-composition-and-overlay.md): peer
composition and intentional precedence remain separate operations.

## Current baseline

The [selected-overlay contract](../config.md#selected-overlays) currently loads
one YAML mapping from `overlays/<name>.yml`. Overlay names are not paths. Empty
selection and the exact lowercase name `default` select no overlay. A selected
missing file fails; there is no extension fallback or overlay stack.

Ordinary loads and explicit layered overlay contexts locate that file beneath
the configured root. Layered main-derived and fallback contexts use their
resolved base for lookup, with a separate containment root. The implemented
[main-relative context decision](../adr/0022-main-relative-context.md) supplies
the context needed by this feature; it does not implement aggregate overlays.

The existing loader in `internal/engine/selected_overlay.go` acquires and parses
the file, checks overlay directives and source-version policy, and applies its
payload with the effective collection policy. Overlays apply after base
composition and main-version checking, before inheritance, defaults, decoding,
CLI overrides, application resolution, and final validation. Source records,
locations, resource contexts, history, repetition checks, limits, and handle
cleanup already cover the individual-file form.

## Targeted end state

### File layout and selection

The proposed catalog is a mapping from overlay names directly to overlay bodies:

```yaml
# overlays.yml — proposed format
production:
  service:
    workers: 8
development:
  service:
    workers: 1
    log_level: debug
```

Selecting `production` takes only that entry as the catalog's contribution.
The same selection may also acquire this existing-format file:

```yaml
# overlays/production.yml
service:
  log_level: warning
```

Strict composition produces this one overlay payload:

```yaml
service:
  workers: 8
  log_level: warning
```

If the individual file instead also defines `service.workers: 12`, ordinary
strict-composition conflict rules apply. File order grants no precedence.
The other catalog entries do not contribute to the selected configuration.
The exact catalog structure and its validation boundary remain in OD-1 through
OD-3; this example illustrates the recommended structure.

Use the same lookup directory for the catalog and the existing `overlays/`
directory. Preserve ordinary versus layered lookup semantics, explicit context
precedence, main-relative and fallback behavior, namespace identity, symlink
handling, and containment. The new filename variants and activation behavior
remain in OD-4 and OD-5.

No selected overlay means neither form is read. With a selection, either form
may supply the named overlay. An absent catalog entry is not an error when the
individual file supplies the selection, and a missing individual file is not an
error when the catalog supplies it. If neither contributes, report the existing
missing-overlay condition. A selected empty mapping is a present contribution,
not a missing overlay. Acquisition or parsing failures are not a reason to
silently prefer the other form.

### Strict composition, then one overlay application

**Settled:** contributions for the selected name from both forms are peers.
Strictly compose them before applying their combined result to the lower
configuration. Do not apply them sequentially as precedence layers, suppress
one form, or reject a shared name merely because both forms contribute.

Reuse existing composition and collection-policy rules. Effective lower-tier
identity and applicable selected-overlay declarations must be available when
composing the peers. Disjoint identified records can compose; conflicting
identities among peers fail. The combined collection then uses its applicable
`replace`, `append`, or `upsert` operation against the lower value once.
Unidentified sequences retain existing strict-composition and overlay behavior.

Resolve collection declarations across the contributions using the existing
same-tier rules; do not independently apply each contribution's mode to the
lower configuration. Keep the existing rules for version-path protection and
forbidding overlay imports. This feature adds neither overlay inheritance nor
selection of several overlay names.

### Processing and source evidence

Acquire selected contributions within the existing load, after base processing
reaches the overlay phase. Preserve subsequent lifecycle order and normal
failure behavior. Reuse parser and composition representations rather than
decoding the catalog into application structs and reconstructing a document.

The selected catalog body's application paths are relative to that body, while
its locations still identify the real catalog file and authored line/column.
Diagnostics must distinguish application coordinates from the catalog wrapper;
never fabricate a physical file such as `overlays.yml/production`. Relative
resource contexts follow the physical declaring file, so resources beside
`overlays.yml` and beside `overlays/production.yml` have different bases.
Preserve both contributors when mappings combine, and normal field origins and
history when the completed overlay is applied.

Keep real acquisition accounting: parse the acquired catalog under existing
YAML limits and charge its complete bytes and parsed nodes, including unselected
bodies. Each acquired physical file contributes a source record; unused names
are not separately acquired sources. Preserve repetition/containment checks,
detached inspection, and cleanup on every exit. Do not add discovery-attempt
history or a new inspection API solely for this feature.

Use deterministic acquisition order for source IDs and sequence assembly. The
recommended order is catalog contribution, then individual-file contribution;
this orders evidence and composed records, not precedence. Reuse existing
diagnostic classifications where they fit. Source-version/control scope is an
open decision in OD-2 and must be reflected consistently in inspection.

## Work from the baseline

1. **Settle the remaining contract choices.** Resolve the decisions below,
   including activation compatibility, and prepare examples for each source
   form, combined contributions, source-version modes, and collection controls.
2. **Separate overlay acquisition from application.** Adapt the selected-overlay
   loader to acquire optional contributions and distinguish missing selection
   from acquisition failure. Retain existing context selection and access rules.
3. **Select catalog bodies without losing source evidence.** Parse the physical
   catalog, select the requested body, and retain its actual locations. Apply
   the chosen version/directive scope and account for the complete acquired file.
4. **Compose peers and apply once.** Reuse strict composition with the lower
   collection policy and selected declarations, then feed the composed result
   into the existing overlay operation. Preserve contributor origins, history,
   sensitivity, version protection, and resource contexts.
5. **Verify and document.** Add public integration coverage and maintained
   consumer examples; update selected-overlay contracts, public declaration
   comments as needed, and internal ownership documentation when implemented.

This is a dependency order for a later implementation plan, not a prescribed
commit layout. Do not change existing ADR decisions merely to document a new
storage form. Review any durable ownership change separately if one emerges.

## Validation and completion

Cover the public loading boundary with:

- Catalog only, individual file only, both, neither, a catalog without the
  selected name, selected empty mappings, and inactive/default selections.
- Complementary mappings, conflicting fields, identified-sequence composition,
  duplicate identities, mode declarations, and one final collection operation.
- Ordinary and layered loading, discovered/explicit/non-file/absent mains,
  explicit overlay contexts, main-derived contexts, fallback contexts, wider
  roots, host and native filesystems, and existing link-handling capabilities.
- Invalid YAML, catalog shape, selected/unselected body validation, source-version
  modes, protected version paths, and forbidden directives under the settled rules.
- Actual source paths and locations, origins and contributor history, sensitive
  values, distinct resource bases, source IDs, and detached snapshots.
- Missing versus unreadable sources, repetition, complete catalog accounting,
  limits, and cleanup on failure. No duplicate load lifecycle callbacks.
- Existing individual-file workflows and the activation compatibility cases
  settled in OD-5.

Use focused tests and normal repository checks during implementation. No new
benchmark campaign or release gate is required. Completion means applications
can select one named overlay from either or both forms with the same composition,
precedence, and inspection model.

## Open Decisions

### OD-1: Catalog structure

**Recommendation:** use the direct name-to-body mapping illustrated above.
It adds only one enclosing mapping and keeps each body recognizable as an
ordinary overlay. The catalog wrapper is a selection container, not an application
document; overlay names retain existing spelling and selection rules. The special
`default` selection continues to mean no overlay.

**Alternative:** wrap entries under an `overlays` key, leaving space for catalog
metadata. This gives file-level headers an explicit home, but adds boilerplate
and another reserved structure before a need for catalog metadata is established.

### OD-2: Source-version declarations and document controls

**Recommendation:** treat each selected body as the overlay's logical document
root for existing source-version and overlay-control processing. A version header
or `config.sequences` declaration appears inside that body, just as it appears
at the root of an individual overlay file. Process each participating contribution
before strict composition, including its own required version check when enabled.
Preserve the physical file's authored locations and report version evidence for
the selected logical document. Imports remain forbidden under existing overlay
rules; nested application data is not reinterpreted as root controls.

This permits moving a body between forms without rewriting its payload or
changing its policy meaning. It requires explicitly distinguishing the catalog's
physical document from the selected logical overlay in version processing and
inspection; it must not silently change source-version semantics for other inputs.

**Alternative:** declare the version and shared controls once at catalog level,
with entries inheriting them. This reduces repeated headers but needs the wrapper
from OD-1 or another metadata convention, plus rules for shared versus per-entry
controls. It also makes a standalone overlay body differ from its catalog form.

### OD-3: Validation of unselected bodies

**Recommendation:** require a valid YAML mapping catalog and look up the selected
name. Preserve parser-level validation of the whole file, including duplicate
mapping-key rules and limits. Require the selected body to satisfy normal overlay
mapping, directive, and version rules, but do not run those semantic checks or
application callbacks for unselected bodies. Final application validation assesses
the resulting effective configuration, not each profile in isolation.

This keeps a load focused on the requested configuration while acknowledging
that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog
contains no definitions; selected null/scalar/sequence bodies follow the existing
mapping-root rules rather than becoming shorthand for absence.

**Alternative:** validate every body's shape, name, directives, and version when
reading the catalog. This catches unused-profile defects earlier, but makes one
profile's load depend on unrelated definitions and expands checker invocation.
A whole-catalog lint facility could serve that purpose separately if requested.

### OD-4: Filename variants

**Recommendation:** initially recognize exactly `overlays.yml` and the existing
`overlays/<name>.yml`. This matches current overlay spelling without adding a
separate extension-precedence decision. Both recognized forms compose as peers.

**Alternative:** also recognize `.yaml`. This accommodates common YAML filenames,
but requires deciding whether multiple extensions are additional peers or fallback
alternatives. If adopted, prefer strict peer composition over hidden precedence;
document stable acquisition order and collisions consistently for both forms.

### OD-5: Activation and compatibility

**Recommendation:** use the existing selected overlay name to activate lookup of
both forms, without another option in the common workflow. The storage layout
should not require consumers to implement different selection APIs. Document
that an acquired catalog participates in parsing even if its selected name is
absent, and that conflicting selected contributions fail strict composition.

Before committing to release compatibility, assess existing deployments that
have an `overlays.yml` previously ignored by ConfigKit: automatic participation
can expose invalid content or conflicts alongside an otherwise successful
individual-file load. The [release policy](../policy/release.md#versioning-and-compatibility)
preserves successful valid configurations; merely accepting additional layouts
does not resolve this interaction. No release version is selected here.

**Alternative:** add explicit catalog enablement, preserving existing directory-only
lookup until selected by the application. This provides a clear compatibility
boundary and lets consumers adopt the feature deliberately, but introduces an
extra setting for the common layout. Once enabled, contributions from both forms
must still compose strictly; this alternative does not reopen the settled
coexistence decision.