configkit/docs/roadmap/overlays.md

Single-file named overlays

Status: proposal; strict composition between the two forms is accepted. Remaining decisions are listed below. Not implemented. Implementation baseline: ce6005c.

Purpose

Support a single overlays.yml containing named overlay definitions alongside the existing overlays/<name>.yml convention. Small applications can keep their profiles together, while applications with larger configurations can split them into individual files or contribute through both forms.

Applications continue to select one overlay name. ConfigKit selects the matching contributions, strictly composes them, and applies the resulting overlay once. This follows ADR 0001: peer composition and intentional precedence remain separate operations.

Current baseline

The selected-overlay contract currently loads one YAML mapping from overlays/<name>.yml. Overlay names are not paths. Empty selection and the exact lowercase name default select no overlay. A selected missing file fails; there is no extension fallback or overlay stack.

Ordinary loads and explicit layered overlay contexts locate that file beneath the configured root. Layered main-derived and fallback contexts use their resolved base for lookup, with a separate containment root. The implemented main-relative context decision supplies the context needed by this feature; it does not implement aggregate overlays.

The existing loader in internal/engine/selected_overlay.go acquires and parses the file, checks overlay directives and source-version policy, and applies its payload with the effective collection policy. Overlays apply after base composition and main-version checking, before inheritance, defaults, decoding, CLI overrides, application resolution, and final validation. Source records, locations, resource contexts, history, repetition checks, limits, and handle cleanup already cover the individual-file form.

Targeted end state

File layout and selection

The proposed catalog is a mapping from overlay names directly to overlay bodies:

# overlays.yml — proposed format
production:
  service:
    workers: 8
development:
  service:
    workers: 1
    log_level: debug

Selecting production takes only that entry as the catalog's contribution. The same selection may also acquire this existing-format file:

# overlays/production.yml
service:
  log_level: warning

Strict composition produces this one overlay payload:

service:
  workers: 8
  log_level: warning

If the individual file instead also defines service.workers: 12, ordinary strict-composition conflict rules apply. File order grants no precedence. The other catalog entries do not contribute to the selected configuration. The exact catalog structure and its validation boundary remain in OD-1 through OD-3; this example illustrates the recommended structure.

Use the same lookup directory for the catalog and the existing overlays/ directory. Preserve ordinary versus layered lookup semantics, explicit context precedence, main-relative and fallback behavior, namespace identity, symlink handling, and containment. The new filename variants and activation behavior remain in OD-4 and OD-5.

No selected overlay means neither form is read. With a selection, either form may supply the named overlay. An absent catalog entry is not an error when the individual file supplies the selection, and a missing individual file is not an error when the catalog supplies it. If neither contributes, report the existing missing-overlay condition. A selected empty mapping is a present contribution, not a missing overlay. Acquisition or parsing failures are not a reason to silently prefer the other form.

Strict composition, then one overlay application

Settled: contributions for the selected name from both forms are peers. Strictly compose them before applying their combined result to the lower configuration. Do not apply them sequentially as precedence layers, suppress one form, or reject a shared name merely because both forms contribute.

Reuse existing composition and collection-policy rules. Effective lower-tier identity and applicable selected-overlay declarations must be available when composing the peers. Disjoint identified records can compose; conflicting identities among peers fail. The combined collection then uses its applicable replace, append, or upsert operation against the lower value once. Unidentified sequences retain existing strict-composition and overlay behavior.

Resolve collection declarations across the contributions using the existing same-tier rules; do not independently apply each contribution's mode to the lower configuration. Keep the existing rules for version-path protection and forbidding overlay imports. This feature adds neither overlay inheritance nor selection of several overlay names.

Processing and source evidence

Acquire selected contributions within the existing load, after base processing reaches the overlay phase. Preserve subsequent lifecycle order and normal failure behavior. Reuse parser and composition representations rather than decoding the catalog into application structs and reconstructing a document.

The selected catalog body's application paths are relative to that body, while its locations still identify the real catalog file and authored line/column. Diagnostics must distinguish application coordinates from the catalog wrapper; never fabricate a physical file such as overlays.yml/production. Relative resource contexts follow the physical declaring file, so resources beside overlays.yml and beside overlays/production.yml have different bases. Preserve both contributors when mappings combine, and normal field origins and history when the completed overlay is applied.

Keep real acquisition accounting: parse the acquired catalog under existing YAML limits and charge its complete bytes and parsed nodes, including unselected bodies. Each acquired physical file contributes a source record; unused names are not separately acquired sources. Preserve repetition/containment checks, detached inspection, and cleanup on every exit. Do not add discovery-attempt history or a new inspection API solely for this feature.

Use deterministic acquisition order for source IDs and sequence assembly. The recommended order is catalog contribution, then individual-file contribution; this orders evidence and composed records, not precedence. Reuse existing diagnostic classifications where they fit. Source-version/control scope is an open decision in OD-2 and must be reflected consistently in inspection.

Work from the baseline

1. Settle the remaining contract choices. Resolve the decisions below, including activation compatibility, and prepare examples for each source form, combined contributions, source-version modes, and collection controls.
2. Separate overlay acquisition from application. Adapt the selected-overlay loader to acquire optional contributions and distinguish missing selection from acquisition failure. Retain existing context selection and access rules.
3. Select catalog bodies without losing source evidence. Parse the physical catalog, select the requested body, and retain its actual locations. Apply the chosen version/directive scope and account for the complete acquired file.
4. Compose peers and apply once. Reuse strict composition with the lower collection policy and selected declarations, then feed the composed result into the existing overlay operation. Preserve contributor origins, history, sensitivity, version protection, and resource contexts.
5. Verify and document. Add public integration coverage and maintained consumer examples; update selected-overlay contracts, public declaration comments as needed, and internal ownership documentation when implemented.

This is a dependency order for a later implementation plan, not a prescribed commit layout. Do not change existing ADR decisions merely to document a new storage form. Review any durable ownership change separately if one emerges.

Validation and completion

Cover the public loading boundary with:

• Catalog only, individual file only, both, neither, a catalog without the selected name, selected empty mappings, and inactive/default selections.
• Complementary mappings, conflicting fields, identified-sequence composition, duplicate identities, mode declarations, and one final collection operation.
• Ordinary and layered loading, discovered/explicit/non-file/absent mains, explicit overlay contexts, main-derived contexts, fallback contexts, wider roots, host and native filesystems, and existing link-handling capabilities.
• Invalid YAML, catalog shape, selected/unselected body validation, source-version modes, protected version paths, and forbidden directives under the settled rules.
• Actual source paths and locations, origins and contributor history, sensitive values, distinct resource bases, source IDs, and detached snapshots.
• Missing versus unreadable sources, repetition, complete catalog accounting, limits, and cleanup on failure. No duplicate load lifecycle callbacks.
• Existing individual-file workflows and the activation compatibility cases settled in OD-5.

Use focused tests and normal repository checks during implementation. No new benchmark campaign or release gate is required. Completion means applications can select one named overlay from either or both forms with the same composition, precedence, and inspection model.

Open Decisions

OD-1: Catalog structure

Recommendation: use the direct name-to-body mapping illustrated above. It adds only one enclosing mapping and keeps each body recognizable as an ordinary overlay. The catalog wrapper is a selection container, not an application document; overlay names retain existing spelling and selection rules. The special default selection continues to mean no overlay.

Alternative: wrap entries under an overlays key, leaving space for catalog metadata. This gives file-level headers an explicit home, but adds boilerplate and another reserved structure before a need for catalog metadata is established.

OD-2: Source-version declarations and document controls

Recommendation: treat each selected body as the overlay's logical document root for existing source-version and overlay-control processing. A version header or config.sequences declaration appears inside that body, just as it appears at the root of an individual overlay file. Process each participating contribution before strict composition, including its own required version check when enabled. Preserve the physical file's authored locations and report version evidence for the selected logical document. Imports remain forbidden under existing overlay rules; nested application data is not reinterpreted as root controls.

This permits moving a body between forms without rewriting its payload or changing its policy meaning. It requires explicitly distinguishing the catalog's physical document from the selected logical overlay in version processing and inspection; it must not silently change source-version semantics for other inputs.

Alternative: declare the version and shared controls once at catalog level, with entries inheriting them. This reduces repeated headers but needs the wrapper from OD-1 or another metadata convention, plus rules for shared versus per-entry controls. It also makes a standalone overlay body differ from its catalog form.

OD-3: Validation of unselected bodies

Recommendation: require a valid YAML mapping catalog and look up the selected name. Preserve parser-level validation of the whole file, including duplicate mapping-key rules and limits. Require the selected body to satisfy normal overlay mapping, directive, and version rules, but do not run those semantic checks or application callbacks for unselected bodies. Final application validation assesses the resulting effective configuration, not each profile in isolation.

This keeps a load focused on the requested configuration while acknowledging that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog contains no definitions; selected null/scalar/sequence bodies follow the existing mapping-root rules rather than becoming shorthand for absence.

Alternative: validate every body's shape, name, directives, and version when reading the catalog. This catches unused-profile defects earlier, but makes one profile's load depend on unrelated definitions and expands checker invocation. A whole-catalog lint facility could serve that purpose separately if requested.

OD-4: Filename variants

Recommendation: initially recognize exactly overlays.yml and the existing overlays/<name>.yml. This matches current overlay spelling without adding a separate extension-precedence decision. Both recognized forms compose as peers.

Alternative: also recognize .yaml. This accommodates common YAML filenames, but requires deciding whether multiple extensions are additional peers or fallback alternatives. If adopted, prefer strict peer composition over hidden precedence; document stable acquisition order and collisions consistently for both forms.

OD-5: Activation and compatibility

Recommendation: use the existing selected overlay name to activate lookup of both forms, without another option in the common workflow. The storage layout should not require consumers to implement different selection APIs. Document that an acquired catalog participates in parsing even if its selected name is absent, and that conflicting selected contributions fail strict composition.

Before committing to release compatibility, assess existing deployments that have an overlays.yml previously ignored by ConfigKit: automatic participation can expose invalid content or conflicts alongside an otherwise successful individual-file load. The release policy preserves successful valid configurations; merely accepting additional layouts does not resolve this interaction. No release version is selected here.

Alternative: add explicit catalog enablement, preserving existing directory-only lookup until selected by the application. This provides a clear compatibility boundary and lets consumers adopt the feature deliberately, but introduces an extra setting for the common layout. Once enabled, contributions from both forms must still compose strictly; this alternative does not reopen the settled coexistence decision.