configkit/docs/roadmap/main-relative-inputs.md

# Main-relative underlays and overlays

Status: accepted direction; API and filesystem-policy decisions remain open.
Not implemented. Planning baseline: `191015e`.

## Purpose

Allow an application to select its main configuration through ConfigKit's
existing discovery descriptions, then load explicitly declared underlay files
and a selected overlay relative to that main file. Applications should not need
to repeat discovery themselves or run a preliminary configuration load to learn
the selected directory.

The application continues to choose its search conventions, underlay tiers, and
overlay name. ConfigKit resolves their filesystem relationship within the same
load. Independently located inputs remain useful and can be combined with
main-relative inputs.

## Current baseline

The [discovery contract](../config.md#configuration-discovery) resolves a
`SearchConfig` source inside loading. It selects one host file or absent input;
errors processing the selected file do not resume the search. The public
constructor returns a source description, not a selected filename.

Ordinary `Load` already derives an overlay context from a selected file. Without
an explicit root override, selecting `/etc/example/config.yml` permits overlay
`production` to load from `/etc/example/overlays/production.yml`. Optional search
exhaustion supplies no inferred root; ordinary loading then needs an explicit
context for an active overlay.

The [layered API](../config.md#layered-loading) differs:

- An active overlay requires `LayeredOptions.OverlayFilesystem` independently of
  the main source. Grouped preflight rejects its absence before discovery runs.
- `Underlay.Inputs` contains independently specified sources and contexts. A
  file source can establish its own context, but no declaration binds an
  underlay filename to the main file that discovery will select.
- Main acquisition already precedes underlay acquisition. Underlay groups are
  subsequently composed from lowest to highest priority, beneath the main tier.
- Ordinary relative imports are peers of their declaring input. They do not
  express lower-priority underlay tiers.

The extension can therefore reuse the selected main's acquired filesystem
information and the existing grouped lifecycle. It needs a way to express the
relationship publicly and bind it before dependent acquisition.

## Targeted end state

### Application declarations

An application can declare, before loading:

1. A main `Input` using `SearchConfig`, with any supported combination of search
   presets and custom candidates.
2. One or more underlay files whose paths are relative to the selected main
   file's directory, grouped and ordered through the existing underlay API.
3. A selected overlay whose context is derived from the selected main file.

The relationship is explicit and declarative. It does not require a new
application callback, a second load, or a public intermediate configuration.
Exact API spelling and context selection are subject to OD-1. Missing-main
behavior, filesystem policy, and eligible main sources are subject to OD-2
through OD-4; the examples below assume a successfully selected host file.

Existing input destinations, placement, import opt-in, and load-wide policies
remain applicable to underlay inputs. The new relationship determines where an
input is acquired; it does not change its precedence or composition rules.
An underlay's own relative imports continue to resolve from that underlay's
declaring directory.

### Examples

These describe intended behavior, not implemented Go syntax. Assume there is no
explicit configuration-root override and that the selected main file is:

```text
/etc/example/config.yml
```

| Application declaration | Selected input |
| --- | --- |
| Main-relative underlay `defaults/base.yml` | `/etc/example/defaults/base.yml` |
| Main-relative underlay `defaults/site.yml` | `/etc/example/defaults/site.yml` |
| Main-relative overlay context, overlay name `production` | `/etc/example/overlays/production.yml` |

If the underlays are placed in separate groups in that order, effective priority
is typed defaults, base underlay, site underlay, main, overlay, then CLI.
Putting both underlay inputs in one group instead composes them strictly as
peers. A declaration of `extras.yml` inside `defaults/base.yml` resolves beside
that file, at `/etc/example/defaults/extras.yml`.

If discovery instead selects a file in the user's configuration directory, the
same relative declarations follow that file's directory. An embedded or otherwise
independent underlay may coexist with these inputs and retains its own namespace
and context. No directory is searched for underlays merely because it exists.

### Loading and inspection

Resolve the main-relative descriptions using the selected main's acquired
filesystem information, before acquiring the inputs that depend on it. Keep
discovery within the existing single loading lifecycle and preserve main-first
acquisition, underlay tier order, version processing, selected-overlay timing,
and the existing defaults, resolution, secret, and validation phases.

Static declaration errors can still be checked before source acquisition;
checks that need the selected directory must run after it is available. The
implementation plan should identify this division without imposing a new global
ordering guarantee for unrelated diagnostics or filesystem checks.

After binding, use ordinary source acquisition, parsing, import handling, and
overlay loading. A missing required input, unreadable selected file, or invalid
configuration follows the applicable existing failure rules; it does not cause
main discovery to try another candidate. This feature does not itself introduce
optional-file syntax, underlay globbing, additional overlay filenames, or a new
overlay stack.

Acquired files retain their actual source identity, path, source-local locations,
layer, underlay index, and passive resource context. Do not create an extra
source or precedence tier for the relative declaration itself. Preserve existing
namespace identity, repetition scope, shared acquisition budgets, detached
inspection, and resource cleanup. Use ordinary diagnostic classifications where
they fit; decide whether any new classification is needed during API design.

### Compatibility and scope

Preserve successful existing ordinary and layered loading behavior. Main-relative
selection should be additive and opt-in, with independently supplied sources and
contexts continuing to work. Preserve existing named-overlay lookup semantics;
OD-3 determines the root used by the new context selection.

The work does not add general resource-path resolution, changes to process working
directory, remote acquisition, or a generic source-provider callback framework.
These are scope boundaries for this feature, not new architectural exclusions.
There is no new filesystem race-protection or sandbox guarantee.

The design follows [ADR 0019](../adr/0019-explicit-configuration-discovery.md):
applications select conventions while ConfigKit performs selection and loading.
Review whether the finalized context-ownership decision needs an additional ADR;
do not rewrite an accepted ADR's decision content. No release version or new
release gate is selected by this roadmap.

## Work from the baseline

1. **Settle the public declaration and filesystem policies.** Resolve the open
   decisions below, document combinations with explicit contexts, and prepare
   representative API examples. Keep public types owned by the root package.
2. **Represent the relationship through the facade and request preparation.**
   Update the relevant declarations and conversions in `layered.go`,
   `lifecycle.go`, or `source.go` as the selected API requires. Carry the intent
   through the private request without prematurely interpreting paths against
   the working directory. Preserve reusable-description and input-ownership
   conventions.
3. **Bind dependent inputs during acquisition.** Extend the request preparation
   and acquisition boundary in `internal/engine/load_request.go`,
   `load_input.go`, and `acquisition.go`. Reuse discovery's selected concrete
   source and filesystem-context resolution. Establish which checks remain in
   preflight and which require resolved main information.
4. **Integrate underlays and overlays.** Resolve relative underlay files into
   ordinary acquisitions with their own import bases; derive the selected
   overlay context and reuse its loader. Preserve independent input contexts,
   namespace handling, cleanup on failure, and source/error evidence.
5. **Verify observable behavior and update documentation.** Add focused public
   coverage and maintained consumer examples, then update the discovery,
   layered-loading, and selected-overlay contracts, declaration comments, and
   relevant internal ownership documentation. Explain the difference between a
   file's directory and the configured root when they differ.

This is dependency order for a later implementation plan, not a required commit
layout or a requirement to expose intermediate stages to consumers. Reuse the
existing lifecycle; a new discovery pass or a second configuration load is not
needed.

## Validation and completion

Exercise the behavior through the public API, supported by focused engine tests:

- Main discovery selects an early or later candidate; the dependent files follow
  the actual selection, including when it is outside the working directory.
- Relative and independent underlays coexist; multiple groups retain precedence,
  and peers in one group retain strict-composition conflicts.
- Underlay imports use their own file's directory. Placement and source-version
  policies remain effective after binding.
- Active and inactive overlays, explicit contexts, optional search exhaustion,
  and required-search failures follow the settled policies.
- Symlinked main files, explicit root overrides, parent-relative paths, and
  namespace boundaries follow OD-3 and OD-4 without broadening existing access
  guarantees.
- Selected-file failures and dependent acquisition failures are terminal; no
  repeated full load or duplicate defaults/resolution/validation callbacks are
  introduced.
- Source records, underlay indexes, origins, passive contexts, and failed-load
  evidence identify the actual acquired inputs. Existing handle cleanup and
  budget accounting still cover dependent acquisitions.
- Existing ordinary discovery and overlay loading, independent layered inputs,
  and reuse of descriptions across loads retain their behavior.

Run focused checks during implementation and the repository's applicable normal
checks before completion. No benchmark campaign or additional release gate is
required for this feature. Completion means applications can demonstrate the
discovery-plus-relative-input workflow in one load, with documented public
declarations and verified failure behavior.

## Open Decisions

### OD-1: Public declaration shape and explicit-context combinations

**Recommendation:** add declarative, explicit choices for a main-relative
underlay file and a main-derived overlay context. A dedicated underlay source
constructor and a small overlay context-selection option are plausible shapes;
names remain to be chosen. Keep the existing independent-source/context path as
the default. For overlays, an explicitly supplied `OverlayFilesystem` should
take precedence over a request to derive context, allowing application overrides
without changing shared setup. Specify analogous precedence for any permitted
underlay context overrides. This keeps common use short without making nil
contexts change meaning or introducing callback lifetime concerns.

**Alternative:** add a general deferred filesystem-context descriptor shared by
inputs and overlays. This could give context selection one uniform representation
and accommodate later relationships, but risks expanding a small feature into
a general context API. It is attractive if concrete API sketches show less
duplication than separate declarations. Rejecting simultaneous explicit and
derived overlay choices is also viable: it catches ambiguous setup, but requires
callers to clear the derived choice when supplying an override.

### OD-2: No main file is selected

**Recommendation:** when an active dependent declaration needs a main-file base
and none exists, fail with a clear diagnostic. Do not use an attempted candidate
directory or the working directory as an implicit substitute. This makes an
unresolvable relationship explicit without making discovery globally required.
Applications can require the main search or use independent inputs and an
explicit overlay context when absent-main operation is intended. Inactive
overlays do not require a derived context.

**Alternative:** treat main-relative inputs as conditional on main presence and
skip them after optional exhaustion. This supports defaults-only startup and is
consistent with a permissive search, but could silently omit an explicitly
selected overlay or expected underlay. A selectable skip policy could support
that use case, at the cost of another option; adopting it would require defining
its effect on tier evidence and distinguishing absent main from a missing file
beside an existing main.

### OD-3: Relative base, configuration root, and symlinks

**Recommendation:** anchor underlay filenames to the canonical selected main
file's directory, consistent with existing file/import behavior. Derive overlay
context from the main's resolved context, so overlay lookup remains relative to
its configuration root. When an explicit main root is wider than the main
directory, underlay filenames still use the directory as their base and overlays
use that root. Inherit the main's root for dependent underlay acquisition and
imports unless an explicitly supported context override selects another root.
Each acquired underlay supplies its own import base. Apply existing canonical
containment checks; allow parent-relative traversal when it stays within the
selected root rather than adding a blanket prohibition on `..`.

This distinguishes lookup from containment, reuses existing machinery, and
supports sibling directories under an explicitly wider root. It also means a
symlinked main selects resources beside its target, which must be documented.

**Alternative:** always create a fresh root at the main file's directory for
both kinds of input. This makes every main-relative lookup visually local,
including with a wider main root, but discards intentional root configuration
and prevents some sibling-directory layouts. Using the main file's lexical
symlink directory instead is another viable convention for deployment bundles;
it would differ from current canonical import behavior and require separate
handling for lookup and containment.

### OD-4: Eligible main sources and namespaces

**Recommendation:** support discovered and explicit host files, and native
`Filesystem.File` mains, using each namespace's existing path rules and resource
ownership. Discovery currently selects host files; a native file should not
require discovery merely to provide the same directory relationship. Bind paths
within the selected namespace and reuse its identity. For this initial feature,
byte, reader, mapping, and absent mains do not supply a main-file directory;
independently specified contexts remain available for those workflows.

**Alternative:** derive from any resolved main filesystem context, including an
explicit `BaseDir` for non-file inputs. This is more uniform for context-based
loading and avoids an artificial file requirement when a base is already known,
but changes the concept from relative-to-main-file to relative-to-main-context.
It needs explicit rules for absent mains and root-only contexts, where root and
base are not interchangeable. Limiting the first implementation to host files
is also viable and smaller, but would leave native-filesystem consumers with
the same coordination gap.