configkit/docs/roadmap/main-relative-inputs.md

# Main-relative companion imports and overlays

Status: accepted direction, public API structure, and absent-main behavior;
containment and main-source eligibility remain open. Not implemented. Baseline:
`191015e`.

## Purpose

Make the common workflow straightforward: discover the main configuration file,
load optional companion fragments beside it, and apply a selected native overlay
in one configuration load. Applications choose their conventions; operators can
drop resource definitions into the declared locations without adding import
boilerplate to the main file or creating a manifest on disk.

The central addition is deferred filesystem-context binding for application-owned
underlay inputs, including `Mapping` and `Bytes` sources containing native import
declarations. Bind their imports to the selected main directory during loading.
Reuse existing optional-file, directory, recursion, pattern, destination, and
composition behavior. A new file constructor alone would not cover this workflow,
and a second fragment-discovery implementation is unnecessary.

## Current baseline

The [discovery contract](../config.md#configuration-discovery) resolves
`SearchConfig` inside loading to one host file or absent input. Selected-file
failures do not resume discovery. The constructor returns a source description,
not a selected filename.

Ordinary `Load` already infers overlay context from a file source. Without a
root override, `/srv/campaign/audita.yml` supplies the location
`/srv/campaign/overlays/production.yml` for overlay `production`. Existing overlay
lookup uses the configuration root, which can differ from the main directory.
Optional discovery exhaustion provides no inferred context.

The [layered API](../config.md#layered-loading) has the coordination gap:

- An active overlay requires `LayeredOptions.OverlayFilesystem`. Preflight
  rejects its absence before discovery runs.
- Underlay sources and contexts are supplied independently. Application-owned
  mappings or bytes can already contain import declarations, but cannot obtain
  their import context from the main file that discovery will select.
- Native imports already support optional named files and recursive directory
  searches with filename filtering and placement. They compose as peers of
  their declaring input, so putting declarations in an underlay gives them
  underlay priority. Putting the same declarations in the main file does not.
- Main acquisition already precedes underlay acquisition. Existing composition
  subsequently applies underlay groups from lowest to highest beneath the main.

This feature extends context selection and binding within that lifecycle.

## Audita workflow

Audita supplies the companion import declarations in application-owned mappings
or bytes. Their filesystem bases follow the selected main file, whether selected
explicitly or through ConfigKit's built-in searches. For a main file at
`/srv/campaign/audita.yml`, running from `/tmp` must not relocate these companions:

| Resource | Application-declared locations | Role |
| --- | --- | --- |
| Glossaries | Optional `glossary.yml` and recursive `glossary/` | Underlay contributions mounted at `glossary_collections` using the appropriate existing destination and placement controls |
| Prompt definitions | Optional `prompts.yml` and recursive `prompts/` | Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts |
| Selected native overlay | `overlays/<name>.yml` | The one named overlay applied after main composition |

Both resource-directory searches select `*.yml` and `*.yaml`. The file and
its corresponding directory contribute together if both exist; neither suppresses
the other. Each pair composes strictly as peers within its tier. Missing optional
resources contribute nothing; unreadable selected files, invalid definitions,
and composition conflicts remain errors.

Prompt overrides rely on existing tier and identity rules. The operator prompt
tier sits above embedded prompt tiers, allowing whole-record upsert by registered
prompt ID under the applicable collection policy. Two operator fragments defining
the same ID in that tier conflict; filename ordering does not choose a winner.
The consuming application selects each fragment's schema and placement.

This does not require combining Audita's and PromptKit's schemas. If PromptKit
uses a separate engine, its catalog load establishes the prompt tiers and can
use the selected main's recorded context after Audita's application load. That
is a separate domain load, not a preliminary load to repeat main discovery.
This feature's acceptance does not require changing another repository.

The overlay directory is not imported wholesale as an underlay: its files are
alternative named configurations. No selection means no overlay. Selecting a
name retains the existing missing-selected-overlay error. An aggregate
`overlays.yml` would need its own named-definition and conflict rules and is
outside this feature; it is not a second spelling for current overlay lookup.

## Targeted end state

### Companion underlays

The application declares an underlay `Input` containing native imports and
selects main-derived filesystem binding for that input. `Mapping` and `Bytes`
are first-class supported sources for this use. OD-1 settles the public API
structure below; no new operator-facing YAML syntax is needed.

Once the main is selected, ConfigKit supplies the underlay import base from
that file's directory and binds the namespace and containment policy. Existing
import selection then acquires the companion files. Every imported file uses
its own directory for its further relative imports. For example, an import of
`extra.yml` from `glossary/terms.yml` resolves to `glossary/extra.yml`.

Applications choose underlay groups and their order. A group composes strictly;
separate groups establish precedence. Existing destinations, sequence placement,
source-version policies, and load-wide import defaults still apply. Binding a
context does not itself enable imports or change collection operations.

The relationship is explicit on companion inputs. Independently located and
embedded inputs keep their own contexts and can coexist in the same load.
Providing a concrete context selects that complete context over derived
binding, without field-by-field fallback or recovery from an invalid explicit
context. OD-1 records the representation and precedence rules.

There is no requirement to add a dedicated main-relative file constructor.
Optional named-file acquisition is already expressible through the underlay's
native imports. Ordinary `File` paths do not silently gain new relative semantics.

### Layered overlays

For an active layered overlay, use an explicit `OverlayFilesystem` when supplied.
Otherwise derive the overlay context automatically from the available main-file
information; the common workflow requires no `OverlayFromMain` flag. An invalid
explicit context fails rather than falling back. Inactive overlays need no
context. Missing-main and fallback cases are addressed below and in OD-2.

For this newly derived layered context, the lookup location is the main file's
directory, independently of a wider containment root. If the main is
`/srv/campaign/audita.yml` and its permitted root is `/srv`, the selected file
is `/srv/campaign/overlays/production.yml`, not `/srv/overlays/production.yml`.
Widening containment must not relocate the automatically selected companions.

Keep existing explicit-overlay-context lookup behavior and ordinary `Load`
behavior. The implementation must represent the derived lookup base separately
from its containment root where necessary, rather than simply copying today's
root-relative overlay context. Native overlay names, filenames, selection,
precedence, and validation retain their existing semantics.

### Absent main and optional companions

Optional discovery may find no main file while defaults and CLI settings still
permit a successful application load. When no usable base is available, skip
optional companion import expansion while retaining the declaring input's
ordinary payload and validation. Required imports still fail. Applications can
require a base or supply an explicit fallback context on each derived binding.
OD-2 records the settled precedence, skip scope, and fallback behavior.

Do not infer a directory from unsuccessful candidates or silently use the working
directory. An application may explicitly choose the working directory as its
fallback. An active selected overlay still needs a usable context and the selected
file; optional companion imports do not make that overlay optional. Independent
underlays and explicitly located overlays retain their absent-main behavior.
No fallback applies to a failure reading or processing an actually selected main.

### Lifecycle, evidence, and compatibility

Bind dependent contexts after the main information is available and before
traversing their imports or loading the overlay. Preserve main-first acquisition,
underlay precedence, version processing, selected-overlay timing, and the existing
defaults, CLI, resolution, secret, and validation phases. No application context
callback, second discovery API, or preliminary full load is required.

Check static declarations during preflight where possible; defer checks requiring
the selected directory. Do not introduce a global ordering guarantee for unrelated
diagnostics or filesystem checks. Selected-file and companion acquisition errors
remain terminal rather than restarting discovery.

Application-supplied mappings or bytes remain ordinary input sources with their
own labels, evidence, and existing validation obligations. Binding a context does
not manufacture a main document, exempt these inputs from source-version policy,
or give them a fabricated physical file path. Imported files retain their actual
source identity, locations, precedence layer, underlay index, and passive resource
context. Preserve namespace identity, repetition scope, shared budgets, detached
inspection, and acquisition-handle cleanup.

Successful existing ordinary and explicitly configured layered loads should retain
their behavior. Automatically deriving a layered overlay context intentionally
relaxes the former missing-context error. Companion directory conventions are
selected by applications; ConfigKit does not universally scan `glossary/` or
`prompts/` without those declarations.

This work adds no general resource-path resolver, remote acquisition, mutation of
the process working directory, filesystem race-protection guarantee, or general
source-provider callback framework. These are scope boundaries, not new permanent
architectural exclusions. The design follows
[ADR 0019](../adr/0019-explicit-configuration-discovery.md). Review ADR and policy
follow-up for deferred context ownership without rewriting accepted decisions.
No release version or additional release gate is selected here.

## Work from the baseline

1. **Finalize context declarations and absence policy.** Resolve the remaining
   choices below and prepare Go examples for Audita's mapping/bytes workflow,
   explicit context overrides, and independently located embedded inputs.
2. **Carry binding intent through the public facade and load request.** Update
   the relevant input/options declarations and conversions in `layered.go` and
   `lifecycle.go`, retaining root-owned public types and normal ownership rules.
   Distinguish a deferred relationship from a concrete filesystem context.
3. **Bind underlay import contexts after main selection.** Extend preparation
   and acquisition in `internal/engine/load_request.go`, `load_input.go`, and
   `acquisition.go` as needed. Reuse the acquired main's directory and namespace,
   apply the chosen missing-base policy, and delegate selection to existing import
   machinery. Preserve independently supplied contexts.
4. **Derive overlay context and separate lookup from containment.** Remove the
   unconditional explicit-context requirement for the new layered default.
   Preserve explicit and ordinary lookup while enabling a main-directory lookup
   base inside a potentially wider root. Reuse selected-overlay loading and its
   failure and resource-ownership behavior.
5. **Verify the complete workflow and document it.** Add maintained consumer
   examples, public integration coverage, and focused engine tests. Update the
   discovery, layered-loading, import-context, and selected-overlay contracts,
   declaration comments, and internal ownership documentation when implemented.

This is dependency order for a later implementation plan, not a required commit
layout or a requirement to expose intermediate stages to consumers.

## Validation and completion

Validate behavior through the public loading boundary, including:

- Explicit and discovered main files, early and later search candidates, and an
  unrelated invocation directory. Companions follow the same selected main.
- Mapping and bytes underlays containing optional named-file and recursive
  directory imports with `*.yml`/`*.yaml` filtering and mounted destinations.
- File only, directory only, both, neither, empty directories, and invalid or
  unreadable selected fragments. Both forms compose as peers when present.
- Operator prompt overrides above embedded catalogs using registered identity;
  same-tier duplicate IDs remain conflicts. Main and overlay precedence survive.
- Mixed embedded and companion inputs, multiple underlay tiers, transitive
  imports, placement, source-version checks, and load-wide import defaults.
- Automatic derived overlays and explicit overrides, active and inactive names,
  broader containment roots, and missing selected overlays.
- Optional discovery exhaustion, required discovery, fallback and skip policies,
  mixed payload/required-import cases from OD-2, and terminal selected-main errors.
- Symlinks, eligible namespaces, parent-relative paths within the selected scope,
  context validity, and namespace identity under OD-3 and OD-4.
- Normal source evidence, diagnostics, passive resource contexts, cleanup, and
  budget accounting; no duplicate lifecycle callbacks or preliminary full load.
- Existing ordinary and explicit layered workflows and description reuse.

Run focused checks during implementation and applicable normal repository checks
before completion. No benchmark campaign is required. Completion means the
application can discover its main, bind companion imports and the selected
overlay, and receive normal resolved configuration and inspection in one load,
without an application directory walker, parser, or extra discovery pass.

## Settled Decisions

### OD-1: Public representation of deferred context binding — settled

Add a small, optional main-relative binding description to `Input`, retaining
the concrete `Input.Filesystem` field. A helper supplies the common default
description; explicit settings group the absent-main controls settled in OD-2.
Companion `Mapping` and `Bytes` inputs use native imports for fragment selection.
Preserve the input's source, placement, import enablement, and tier.

The proposed shape is an `Input.MainRelative` field of type
`*MainRelativeContext`, with a `MainRelative()` convenience helper. These names
illustrate the settled structure; final spelling can be finished during API
design. For example, this unimplemented API sketch assumes
`companionDeclarations` contains ordinary native import declarations:

```go
configkit.Input{
    Source:       configkit.Mapping("companions", companionDeclarations),
    Imports:      true,
    MainRelative: configkit.MainRelative(),
}
```

A nil binding preserves existing independent context behavior. A non-nil binding
requests context derivation during loading. If `Input.Filesystem` is also supplied,
that complete concrete context wins. Do not merge fields or fall back to derivation
when the explicit context is invalid. This permits overriding reusable companion
setup without clearing its binding declaration.

Layered overlays derive automatically when `OverlayFilesystem` is absent; no
additional opt-in flag is needed. An explicit overlay context wins as a whole.
Support an explicit overlay fallback under OD-2, reusing the small binding
settings where appropriate without requiring a general context-selection API.

Keep the binding on `Input`: `Source` still describes the supplied content and
`FilesystemContext` still describes a concrete location. This does not reinterpret
a file source's own path. Grouping related settings avoids scattering them across
a boolean and sibling fields, while keeping the addition smaller than a general
context selector. No source wrapper, deferred fields on `FilesystemContext`,
callback, or second file-selection API is needed for this design.

### OD-2: Scope and representation of absent-main handling — settled

Put absence behavior on each derived binding. When no usable base exists, the
default skips optional companion import expansion, retaining the declaring
input's ordinary configuration values. Required imports produce an error;
requiredness includes applicable application import defaults. Continue normal
input and directive validation, so absence does not excuse invalid declarations.
This preserves independent payload and existing required/optional distinctions
without treating the entire input as conditional on main presence.

Support a concrete fallback context and a require-base option within the binding
description. Illustrative fields are `Fallback *FilesystemContext` and
`RequireBase bool`; final spelling remains an API design detail. Select the
context in this order:

1. A supplied `Input.Filesystem` is the complete override.
2. Otherwise, use a usable main-derived context.
3. Otherwise, use the explicitly supplied fallback context, if any.
4. Without any usable context, skip optional imports unless the binding requires
   a base. A require-base binding or required import produces an error.

A fallback can satisfy require-base: the option requires a usable context, not
an actual main file. Consumers requiring a main file can make discovery required.
An invalid selected context fails; it does not activate a lower-priority choice.
Fallback applies only when the main cannot provide a base, never after a failure
reading or processing a selected main, or after missing or invalid companion
files beside an existing main. Once bound, normal import selection and failure
rules apply. Do not guess from unsuccessful discovery candidates or implicitly
use the working directory; applications may explicitly choose it as a fallback.

A fallback uses `FilesystemContext.BaseDir` as its lookup directory, with `Root`
and `FS` retaining their containment and namespace roles. This also applies to
the new overlay fallback: look for `overlays/<name>.yml` beneath its base, while
preserving existing explicit `OverlayFilesystem` lookup behavior. An explicit
overlay context wins, followed by main derivation, then the declared overlay
fallback. When no overlay is selected, no overlay context is needed. An active
selection requires a usable context and the named file; never silently skip it.
Companion absence settings do not make a selected overlay optional.

## Open Decisions

### OD-3: Containment inheritance and symlink convention

**Settled:** derived companion imports and derived layered overlays use the main
file's directory for lookup, even with a wider configured containment root.
Explicit overlay contexts retain their current lookup semantics.

**Recommendation:** use the selected main's canonical directory, and inherit its
resolved namespace and containment root for derived bindings. Existing import
rules then provide file-local bases for transitive imports. Permit parent-relative
paths when existing containment rules allow them. This reuses current canonical
file behavior and preserves intentional wider roots without relocating companions.
A concrete underlay context override can select a different scope under its
normal rules. Document the symlink-target-directory convention explicitly.

**Alternative:** create a fresh containment root at the selected main directory.
This keeps companions locally scoped but discards intentional wider roots and
prevents some sibling-directory layouts. Anchoring to the lexical symlink directory
is also viable for deployment bundles; it differs from current file/import
behavior and would need distinct lookup and containment handling.

### OD-4: Main-source eligibility beyond discovered host files

**Settled:** both explicitly selected and discovered host mains support the
workflow. Non-file companion inputs, especially mappings and bytes, are in scope;
this decision concerns the main input that supplies their base.

**Recommendation:** also support native `Filesystem.File` mains using their
namespace's path and ownership rules. For an initial file-based scope, non-file
mains use independently supplied or fallback contexts rather than implicitly
providing a main-file directory. This covers the requested workflow and avoids
making discovery a prerequisite for equivalent file inputs.

**Alternative:** derive from any usable main context, including `BaseDir` on a
byte, reader, or mapping main. This is more uniform with context-based loading
and avoids excluding a known base, but requires explaining main-relative as
main-context-relative and deciding how root-only or absent-main contexts behave.
A host-only initial scope is smaller but leaves native-file consumers with the
same coordination gap.