configkit/docs/roadmap/main-relative-inputs.md

Main-relative underlays and overlays

Status: accepted direction; API and filesystem-policy decisions remain open. Not implemented. Planning baseline: 191015e.

Purpose

Allow an application to select its main configuration through ConfigKit's existing discovery descriptions, then load explicitly declared underlay files and a selected overlay relative to that main file. Applications should not need to repeat discovery themselves or run a preliminary configuration load to learn the selected directory.

The application continues to choose its search conventions, underlay tiers, and overlay name. ConfigKit resolves their filesystem relationship within the same load. Independently located inputs remain useful and can be combined with main-relative inputs.

Current baseline

The discovery contract resolves a SearchConfig source inside loading. It selects one host file or absent input; errors processing the selected file do not resume the search. The public constructor returns a source description, not a selected filename.

Ordinary Load already derives an overlay context from a selected file. Without an explicit root override, selecting /etc/example/config.yml permits overlay production to load from /etc/example/overlays/production.yml. Optional search exhaustion supplies no inferred root; ordinary loading then needs an explicit context for an active overlay.

The layered API differs:

• An active overlay requires LayeredOptions.OverlayFilesystem independently of the main source. Grouped preflight rejects its absence before discovery runs.
• Underlay.Inputs contains independently specified sources and contexts. A file source can establish its own context, but no declaration binds an underlay filename to the main file that discovery will select.
• Main acquisition already precedes underlay acquisition. Underlay groups are subsequently composed from lowest to highest priority, beneath the main tier.
• Ordinary relative imports are peers of their declaring input. They do not express lower-priority underlay tiers.

The extension can therefore reuse the selected main's acquired filesystem information and the existing grouped lifecycle. It needs a way to express the relationship publicly and bind it before dependent acquisition.

Targeted end state

Application declarations

An application can declare, before loading:

1. A main Input using SearchConfig, with any supported combination of search presets and custom candidates.
2. One or more underlay files whose paths are relative to the selected main file's directory, grouped and ordered through the existing underlay API.
3. A selected overlay whose context is derived from the selected main file.

The relationship is explicit and declarative. It does not require a new application callback, a second load, or a public intermediate configuration. Exact API spelling and context selection are subject to OD-1. Missing-main behavior, filesystem policy, and eligible main sources are subject to OD-2 through OD-4; the examples below assume a successfully selected host file.

Existing input destinations, placement, import opt-in, and load-wide policies remain applicable to underlay inputs. The new relationship determines where an input is acquired; it does not change its precedence or composition rules. An underlay's own relative imports continue to resolve from that underlay's declaring directory.

Examples

These describe intended behavior, not implemented Go syntax. Assume there is no explicit configuration-root override and that the selected main file is:

/etc/example/config.yml

Application declaration	Selected input
Main-relative underlay defaults/base.yml	/etc/example/defaults/base.yml
Main-relative underlay defaults/site.yml	/etc/example/defaults/site.yml
Main-relative overlay context, overlay name production	/etc/example/overlays/production.yml

If the underlays are placed in separate groups in that order, effective priority is typed defaults, base underlay, site underlay, main, overlay, then CLI. Putting both underlay inputs in one group instead composes them strictly as peers. A declaration of extras.yml inside defaults/base.yml resolves beside that file, at /etc/example/defaults/extras.yml.

If discovery instead selects a file in the user's configuration directory, the same relative declarations follow that file's directory. An embedded or otherwise independent underlay may coexist with these inputs and retains its own namespace and context. No directory is searched for underlays merely because it exists.

Loading and inspection

Resolve the main-relative descriptions using the selected main's acquired filesystem information, before acquiring the inputs that depend on it. Keep discovery within the existing single loading lifecycle and preserve main-first acquisition, underlay tier order, version processing, selected-overlay timing, and the existing defaults, resolution, secret, and validation phases.

Static declaration errors can still be checked before source acquisition; checks that need the selected directory must run after it is available. The implementation plan should identify this division without imposing a new global ordering guarantee for unrelated diagnostics or filesystem checks.

After binding, use ordinary source acquisition, parsing, import handling, and overlay loading. A missing required input, unreadable selected file, or invalid configuration follows the applicable existing failure rules; it does not cause main discovery to try another candidate. This feature does not itself introduce optional-file syntax, underlay globbing, additional overlay filenames, or a new overlay stack.

Acquired files retain their actual source identity, path, source-local locations, layer, underlay index, and passive resource context. Do not create an extra source or precedence tier for the relative declaration itself. Preserve existing namespace identity, repetition scope, shared acquisition budgets, detached inspection, and resource cleanup. Use ordinary diagnostic classifications where they fit; decide whether any new classification is needed during API design.

Compatibility and scope

Preserve successful existing ordinary and layered loading behavior. Main-relative selection should be additive and opt-in, with independently supplied sources and contexts continuing to work. Preserve existing named-overlay lookup semantics; OD-3 determines the root used by the new context selection.

The work does not add general resource-path resolution, changes to process working directory, remote acquisition, or a generic source-provider callback framework. These are scope boundaries for this feature, not new architectural exclusions. There is no new filesystem race-protection or sandbox guarantee.

The design follows ADR 0019: applications select conventions while ConfigKit performs selection and loading. Review whether the finalized context-ownership decision needs an additional ADR; do not rewrite an accepted ADR's decision content. No release version or new release gate is selected by this roadmap.

Work from the baseline

1. Settle the public declaration and filesystem policies. Resolve the open decisions below, document combinations with explicit contexts, and prepare representative API examples. Keep public types owned by the root package.
2. Represent the relationship through the facade and request preparation. Update the relevant declarations and conversions in layered.go, lifecycle.go, or source.go as the selected API requires. Carry the intent through the private request without prematurely interpreting paths against the working directory. Preserve reusable-description and input-ownership conventions.
3. Bind dependent inputs during acquisition. Extend the request preparation and acquisition boundary in internal/engine/load_request.go, load_input.go, and acquisition.go. Reuse discovery's selected concrete source and filesystem-context resolution. Establish which checks remain in preflight and which require resolved main information.
4. Integrate underlays and overlays. Resolve relative underlay files into ordinary acquisitions with their own import bases; derive the selected overlay context and reuse its loader. Preserve independent input contexts, namespace handling, cleanup on failure, and source/error evidence.
5. Verify observable behavior and update documentation. Add focused public coverage and maintained consumer examples, then update the discovery, layered-loading, and selected-overlay contracts, declaration comments, and relevant internal ownership documentation. Explain the difference between a file's directory and the configured root when they differ.

This is dependency order for a later implementation plan, not a required commit layout or a requirement to expose intermediate stages to consumers. Reuse the existing lifecycle; a new discovery pass or a second configuration load is not needed.

Validation and completion

Exercise the behavior through the public API, supported by focused engine tests:

• Main discovery selects an early or later candidate; the dependent files follow the actual selection, including when it is outside the working directory.
• Relative and independent underlays coexist; multiple groups retain precedence, and peers in one group retain strict-composition conflicts.
• Underlay imports use their own file's directory. Placement and source-version policies remain effective after binding.
• Active and inactive overlays, explicit contexts, optional search exhaustion, and required-search failures follow the settled policies.
• Symlinked main files, explicit root overrides, parent-relative paths, and namespace boundaries follow OD-3 and OD-4 without broadening existing access guarantees.
• Selected-file failures and dependent acquisition failures are terminal; no repeated full load or duplicate defaults/resolution/validation callbacks are introduced.
• Source records, underlay indexes, origins, passive contexts, and failed-load evidence identify the actual acquired inputs. Existing handle cleanup and budget accounting still cover dependent acquisitions.
• Existing ordinary discovery and overlay loading, independent layered inputs, and reuse of descriptions across loads retain their behavior.

Run focused checks during implementation and the repository's applicable normal checks before completion. No benchmark campaign or additional release gate is required for this feature. Completion means applications can demonstrate the discovery-plus-relative-input workflow in one load, with documented public declarations and verified failure behavior.

Open Decisions

OD-1: Public declaration shape and explicit-context combinations

Recommendation: add declarative, explicit choices for a main-relative underlay file and a main-derived overlay context. A dedicated underlay source constructor and a small overlay context-selection option are plausible shapes; names remain to be chosen. Keep the existing independent-source/context path as the default. For overlays, an explicitly supplied OverlayFilesystem should take precedence over a request to derive context, allowing application overrides without changing shared setup. Specify analogous precedence for any permitted underlay context overrides. This keeps common use short without making nil contexts change meaning or introducing callback lifetime concerns.

Alternative: add a general deferred filesystem-context descriptor shared by inputs and overlays. This could give context selection one uniform representation and accommodate later relationships, but risks expanding a small feature into a general context API. It is attractive if concrete API sketches show less duplication than separate declarations. Rejecting simultaneous explicit and derived overlay choices is also viable: it catches ambiguous setup, but requires callers to clear the derived choice when supplying an override.

OD-2: No main file is selected

Recommendation: when an active dependent declaration needs a main-file base and none exists, fail with a clear diagnostic. Do not use an attempted candidate directory or the working directory as an implicit substitute. This makes an unresolvable relationship explicit without making discovery globally required. Applications can require the main search or use independent inputs and an explicit overlay context when absent-main operation is intended. Inactive overlays do not require a derived context.

Alternative: treat main-relative inputs as conditional on main presence and skip them after optional exhaustion. This supports defaults-only startup and is consistent with a permissive search, but could silently omit an explicitly selected overlay or expected underlay. A selectable skip policy could support that use case, at the cost of another option; adopting it would require defining its effect on tier evidence and distinguishing absent main from a missing file beside an existing main.

OD-3: Relative base, configuration root, and symlinks

Recommendation: anchor underlay filenames to the canonical selected main file's directory, consistent with existing file/import behavior. Derive overlay context from the main's resolved context, so overlay lookup remains relative to its configuration root. When an explicit main root is wider than the main directory, underlay filenames still use the directory as their base and overlays use that root. Inherit the main's root for dependent underlay acquisition and imports unless an explicitly supported context override selects another root. Each acquired underlay supplies its own import base. Apply existing canonical containment checks; allow parent-relative traversal when it stays within the selected root rather than adding a blanket prohibition on ...

This distinguishes lookup from containment, reuses existing machinery, and supports sibling directories under an explicitly wider root. It also means a symlinked main selects resources beside its target, which must be documented.

Alternative: always create a fresh root at the main file's directory for both kinds of input. This makes every main-relative lookup visually local, including with a wider main root, but discards intentional root configuration and prevents some sibling-directory layouts. Using the main file's lexical symlink directory instead is another viable convention for deployment bundles; it would differ from current canonical import behavior and require separate handling for lookup and containment.

OD-4: Eligible main sources and namespaces

Recommendation: support discovered and explicit host files, and native Filesystem.File mains, using each namespace's existing path rules and resource ownership. Discovery currently selects host files; a native file should not require discovery merely to provide the same directory relationship. Bind paths within the selected namespace and reuse its identity. For this initial feature, byte, reader, mapping, and absent mains do not supply a main-file directory; independently specified contexts remain available for those workflows.

Alternative: derive from any resolved main filesystem context, including an explicit BaseDir for non-file inputs. This is more uniform for context-based loading and avoids an artificial file requirement when a base is already known, but changes the concept from relative-to-main-file to relative-to-main-context. It needs explicit rules for absent mains and root-only contexts, where root and base are not interchangeable. Limiting the first implementation to host files is also viable and smaller, but would leave native-filesystem consumers with the same coordination gap.