Plan main-relative companion inputs

docs/roadmap/audita-request.md

@@ -0,0 +1,81 @@
+# Audita request: main-relative companion resources
+
+Consumer requirements against ConfigKit v1.1.0. These complement the proposed
+[main-relative underlays and overlays](main-relative-inputs.md); that roadmap
+owns upstream API design and implementation planning.
+
+## What Audita is doing
+
+Audita is a transcript-processing CLI adopting ConfigKit, PromptKit, and
+CommandKit's native loading and inspection workflows. ConfigKit should own
+discovery, imports, composition, overlays, and configuration resolution. Audita
+should receive the resolved configuration and then apply transcript-specific
+business logic.
+
+The main application configuration is either an explicit file or the result of
+native `SearchConfig` discovery. Companion resources should follow the directory
+containing the selected main file, regardless of the process working directory:
+
+| Resource | Intended location and use |
+| --- | --- |
+| Glossaries | Optional `glossary.yml` and recursive `glossary/` imports, mounted at `glossary_collections` in an application underlay. |
+| PromptKit definitions | Optional `prompts.yml` and recursive `prompts/` imports, in an operator tier above embedded Audita prompts and PromptKit's bundled catalog. |
+| Selected named overlay | Beside the main file, using native `overlays/<name>.yml` lookup. |
+
+Both recursive imports select `*.yml` and `*.yaml`. Audita supplies their import
+declarations, so operators need not repeat discovery boilerplate in every main
+file or create a manifest just to use a resource directory. Each pair composes
+as peers within its tier; native precedence and diagnostics remain authoritative.
+Embedded assets retain their separate filesystem contexts.
+
+For example, selecting `/srv/campaign/audita.yml` while running from `/tmp`
+should use `/srv/campaign/glossary/`, `/srv/campaign/prompts/`, and
+`/srv/campaign/overlays/<name>.yml`.
+
+An aggregate `overlays.yml` was also suggested for symmetry. It is not supported
+by today's named-overlay contract and is a separate possible enhancement, not
+necessary to solve the directory-binding requirement described here.
+
+## Missing API support
+
+In the current [layered API](../config.md#layered-loading), `SearchConfig`
+selects the main during loading, while underlay filesystem contexts and
+`OverlayFilesystem` must be supplied beforehand. Audita cannot bind these to
+the selected directory without duplicating discovery or performing a preliminary
+load. Imports in the main file would be peers, not the required lower-priority
+underlay.
+
+1. **Bind an underlay import context to the selected main directory.** Audita
+   needs this for an application-supplied mapping or bytes containing native
+   import declarations, including optional file and recursive-directory imports
+   and their destinations. A constructor for a required main-relative file alone
+   would not cover this workflow. Resolve the relationship within one layered
+   load, then let existing import machinery acquire and compose the resources.
+2. **Derive the selected overlay context during that same load.** Support an
+   explicit request for main-relative lookup with either an explicit or discovered
+   main file. Preserve native overlay naming, precedence, and errors. Audita's
+   intended lookup base is the main file's directory; distinguish that choice
+   from a potentially broader filesystem containment root.
+3. **Make absent-main behavior explicit.** Audita permits optional discovery to
+   find no main file and still run with sufficient defaults and CLI settings.
+   The API should allow an explicit policy for dependent resources in that case,
+   without manufacturing a main source or guessing from unsuccessful candidates.
+   A caller-supplied fallback directory is one useful option; whether Audita
+   chooses the working directory or skips automatic companion imports remains
+   an application decision. Failures reading a selected main must remain terminal.
+
+PromptKit's separate engine can be constructed after application loading and use
+the selected main's recorded source context. Existing context inspection may
+already suffice there; this request does not require combining the two schemas
+or adding a second discovery API.
+
+## Completion from Audita's perspective
+
+One native application load can discover the main, bind its companion imports
+and selected overlay, and return normal source evidence and errors. Changing
+only the invocation directory does not relocate companions of the same selected
+main. Processing and inspection share this behavior, with no Audita directory
+walker, configuration parser, discovery pass, or preliminary full load.
+
+The API shape is ConfigKit's choice. The important addition is native context
+binding for both declared underlay imports and selected overlays.

docs/roadmap/implementation.md

@@ -0,0 +1,239 @@
+# Implement main-relative companion inputs
+
+The [feature roadmap](main-relative-inputs.md) owns the target behavior and the
+four settled decisions. The [Audita request](audita-request.md) supplies the
+consumer workflow. Implement in order as one sprint. Intermediate stages may
+leave code or tests incomplete; run focused checks when their dependencies are
+in place, and require a coherent, validated tree at the end. Stage boundaries
+are handoffs for Luna, not required commit boundaries.
+
+Start with the [development guide](../development.md) and its applicable
+agent, architecture, testing, and documentation policies. Keep the public
+facade root-owned. Keep ordinary loading, file source semantics, and existing
+explicit layered overlays compatible. No dependency change, new YAML syntax,
+second discovery pass, application callback for context selection, or new
+release gate is part of this work.
+
+## Stage 1 — Define the Go API and preflight rules
+
+In root and engine `layered.go`, add matching descriptions and carry them into
+`loadInput` and `loadRequest` in the engine. Use these public names:
+
+```go
+type MainRelativeContext struct {
+    Fallback    *FilesystemContext
+    RequireBase bool
+}
+
+func MainRelative() *MainRelativeContext
+```
+
+`MainRelative()` returns a fresh pointer to a zero-valued description. Add
+`Input.MainRelative *MainRelativeContext` and
+`LayeredOptions[T].OverlayFallback *FilesystemContext`. The latter only needs
+a concrete context: a selected overlay already requires a base. Use keyed
+literals in new examples. Copy the description and its concrete fallback into
+the private request using the existing filesystem-context conversion in
+`lifecycle.go`; preserve the caller's native namespace and reader ownership
+rules. Store resolved state only in the current load.
+
+In layered preflight, apply the concrete-context override before checking
+binding scope. Reject an active main binding and active bindings on `File`,
+`Filesystem.File`, and `SearchConfig` underlays with `CodeOptionsInvalid`.
+Keep ordinary source validation, including zero-source rejection. A valid
+`NoSource` underlay remains absent; binding does not make it contribute content
+or evidence, and `RequireBase` still applies to an active binding. Suppress the
+synthetic-source missing-context preflight error only for a valid active
+underlay binding. Binding does not enable imports.
+
+Check the static shape of a fallback that can participate, using the existing
+non-file context rules (both base and root, host absolute paths or native
+namespace paths). Defer existence, directory, access, and resolved containment
+checks until selection. Ignore the shadowed binding entirely when
+`Input.Filesystem` is supplied; likewise ignore the overlay fallback when the
+overlay is inactive or an explicit `OverlayFilesystem` wins. Keep overlay
+name and explicit-context preflight checks. Move the missing-context error for
+active layered overlays to Stage 5, where selection can see the main. Do not
+change preflight for ordinary `Load`.
+
+Cover active binding scope, explicit overrides (including invalid explicit
+contexts), and fallback shape through public option tests. Reuse existing
+option diagnostic codes; avoid promising order among unrelated errors.
+
+## Stage 2 — Bind companion inputs with a usable context
+
+In `assembleLayeredInput`, pass the main's `assembledInput.resolvedFilesystem`
+to `acquireUnderlayGroupScopes`. Reuse the acquisition handoff in
+`acquisition.go` and resolution helpers in `filesystem_context.go`; do not
+select or open the main again. Verify the handoff covers a file or selected
+search without imports, a native file, explicit non-file contexts, and
+`NoSource` or exhausted optional search with explicit context. Preserve the
+absent host root-only rule and the native context requirements. A main
+acquisition/context error is terminal, never an absent-base signal.
+
+For each underlay, use its explicit filesystem first; otherwise, for active
+bindings, use the resolved main context, then its fallback. For main-derived
+contexts copy the resolved base and root and keep the native namespace object
+intact. Clear `mainPath` in the copy: the companion is still a synthetic
+source, not a second copy of the main. Use the receiving tier's existing host
+identity token and visited state; the main's host location and scope carry
+over, not its tier bookkeeping. A selected fallback uses the same
+resolution/validation rules as a concrete non-file context; pass it through
+the normal acquisition path where possible. Validate selected contexts even
+when imports are inactive as required by the existing context contract;
+do not probe the fallback when main derivation wins.
+
+Carry a pre-resolved derived context through the private acquisition request
+to `acquireImportedInput` and the standalone/context completion path. Use
+existing import access, traversal and closure; keep root/base checks, parse
+and version processing, budget accounting, and source evidence. Bound
+synthetic sources can expose the same passive context as explicitly located
+synthetic sources but keep their original kind, label and empty physical path.
+The imported files use their own paths and directory bases.
+
+Make the existing native namespace anchor selection in
+`acquireUnderlayGroupScopes` use the **selected** namespace, including
+derived or fallback contexts. Determine anchors from selected descriptors
+without opening directories or validating later inputs out of turn. Keep the
+first input position anchor for each namespace and the existing host and
+synthetic anchoring and tier boundaries. Preserve independent sources and
+repetition semantics, including two different native namespaces with the
+same label and paths.
+
+Cover successful bindings for host and native contexts, an unrelated working
+directory, canonical host symlink target directory, a wider root that permits
+contained parent-relative imports, and reuse of one description across loads.
+Test selected and unused fallbacks. Defer the no-base path to Stages 3–4.
+
+## Stage 3 — Share import declaration preparation
+
+In `import_acquisition.go`, extract the context-independent first pass of
+`walkDeclarations` so both normal traversal and the contextless path can
+use it. Prepare from the already parsed declaration specs, apply
+`resolveImportPolicy` with the load-wide defaults, and retain existing
+placement checks and diagnostic sites. For an input placed as a sequence
+item, prepare against its local mapping scope and keep the existing diagnostic
+path projection. Use the already processed root directives from
+`processAcquiredInputVersion`; do not reparse YAML, repeat version checks,
+or infer a filesystem path from a source label.
+
+Keep selection and admission in the existing normal walk. This stage changes
+no bound import behavior. Reuse import specification, defaults, placement,
+selection, and diagnostic tests for verification.
+
+## Stage 4 — Finish companion handling when no base exists
+
+Handle the no-base outcome only for an active underlay binding with no
+override, usable main context, or fallback. `RequireBase` fails with
+`CodeOptionsInvalid` regardless of import enablement or declaration count.
+Otherwise retain normal source parsing, version handling, disabled-import
+validation, and contribution assembly. When imports are enabled, use Stage 3
+to validate their declarations even if none will be expanded.
+
+For required declarations use `CodeImportMissing` at the first named target
+site for `path`, or the selector field site for `dir` and `glob`. Say that no
+base is available; do not report a file search that did not happen. A required named-path declaration with nonempty
+targets fails, as do required directory or glob selections. Empty
+`path: []` has no targets even if required. Explicit per-declaration
+requiredness continues to override the application default. Skip optional
+expansion without calling a filesystem selector.
+
+Assemble the declaring payload once, with its original source record,
+placement, sequence declarations, and version evidence. Share the existing
+root contribution assembly and accounting where needed for imports-enabled
+payloads, including the pending local scope for sequence-item placement.
+Do not turn imports off to bypass validation or introduce a dummy access
+object, dummy filesystem, acquired child, resource context, or
+`EmptyImportSelections` entry. Charge the declaring payload as usual; charge
+no traversal or child work for skipped imports. Normal searches with a
+selected context still produce their normal empty-selection evidence.
+
+Test absent optional discovery and explicit absence, mixed payload with
+optional and required declarations, application defaults and explicit
+overrides, empty paths under both requiredness values, require-base with and
+without enabled imports, malformed declarations and invalid placement,
+source version policy, mounted payloads, and absence of synthetic search
+evidence. Required discovery and failed selected main processing must not
+activate this path.
+
+## Stage 5 — Derive layered overlay context and lookup base
+
+In the existing overlay branch of `assembleLayeredInput`, select in order:
+explicit `OverlayFilesystem`, usable main context, `OverlayFallback`.
+Skip selection entirely for an inactive name. For an active name without a
+base, return `CodeOptionsInvalid`; if context validation fails, propagate
+its normal diagnostics without trying another context. Keep overlay selection
+after source assembly and main version handling, and preserve existing
+failures for missing files, invalid overlays, and repeated sources.
+
+In `selected_overlay.go`, pass a lookup directory separately to the existing
+selected overlay loader (for example on `selectedOverlayRequest`). Ordinary
+`Load` and explicit layered contexts use the resolved root as before.
+Derived and fallback contexts use the resolved base. Open access at the
+containment root; join the lookup directory with `overlays/<name>.yml`
+using the selected namespace's path operations, then use normal
+canonicalization and containment. Keep the namespace, source identities,
+version processing, limits, source contexts, and handle ownership on the
+existing path. Do not use the lookup directory as a replacement root or
+construct a new namespace.
+
+Cover canonical host symlink targets, wider roots with distinct overlay
+files at base and root, native filesystems, context on a non-file or absent
+main, inactive overlay with an invalid fallback, explicit override with no
+fallback on error, base-relative fallback, and missing selected overlays
+without recovery. Retain ordinary and explicit root-relative regression
+coverage, including existing namespace and repeated-source cases.
+
+## Stage 6 — Exercise the combined workflow
+
+Add a maintained executable example in `examples/consumers` for a single
+layered load using `SearchConfig`, a mapping companion with optional named
+files and recursive directory imports, an independently located embedded
+tier, and one selected overlay. Use actual fixture files and the existing
+`*.yml` and `*.yaml` pattern list; show destination placement, transitive
+imports, and strict peers. Demonstrate precedence from embedded catalog
+through operator underlay, main, and overlay. Include a registered identity
+case for whole-record upsert across tiers and a same-tier duplicate failure.
+No change to Audita or PromptKit is needed.
+
+## Stage 7 — Close remaining integration and evidence gaps
+
+Fill remaining public integration gaps without repeating every engine test:
+first and later search candidates, explicit file and mapping/bytes/reader
+sources, only the file, only the directory, both, neither and an empty
+directory, invalid or inaccessible selected fragments (use a deterministic
+filesystem error where permissions are unreliable), and a selected main
+failure that cannot resume search or use the fallback. Use focused fixtures
+to check layer/index, physical paths, passive resource contexts, normal
+source-version evidence, shared acquisition budgets and cleanup on failure.
+Check that callbacks execute only in the existing lifecycle, and that a
+reused description binds anew. Preserve existing ownership and detached
+inspection checks. Do not require new global diagnostic ordering or test
+incidental call counts.
+
+## Stage 8 — Document and verify the completed sprint
+
+Update the public declaration comments and the discovery, layered loading,
+imports, resource context and selected overlay sections of
+[the configuration contract](../config.md). Document lookup versus
+containment, canonical host symlink behavior, override and fallback
+precedence, and absent-base behavior. Link the maintained examples from
+consumer guidance. Update the [internal overview](../internal/overview.md)
+and [import internals](../internal/imports.md) for the actual handoff,
+contextless path, and ownership.
+
+Record the deferred context decision in the next numbered ADR, extending
+[ADR 0019](../adr/0019-explicit-configuration-discovery.md) without rewriting
+it. Clarify in the architecture policy that the layered loader binds
+application-selected dependent locations after main selection; applications
+still own search conventions, and an explicit overlay selection is required.
+Preserve existing scope boundaries. Use the documentation policy's canonical
+owners rather than restating the full contract in every document.
+
+Run the focused engine, facade, and consumer tests added or affected above,
+then the everyday `./scripts/check.sh`. Run the race suite for the
+load-local ownership and description reuse changes. Check changed Markdown
+links, public facade boundaries and examples. Use the declared Go toolchain
+and existing dependencies. No release or benchmark is part of completion.
+Report validation and any remaining limitations; commit the completed sprint
+under the agent policy.

docs/roadmap/main-relative-inputs.md

@@ -1,7 +1,7 @@
 # Main-relative companion imports and overlays
 
-Status: all four design decisions settled; not implemented. Final API spelling
+Status: design settled; not implemented. The [implementation plan](implementation.md)
-and implementation planning remain. Baseline: `191015e`.
+owns the concrete API spelling, execution order, and verification. Baseline: `191015e`.
 
 ## Purpose
 
@@ -205,74 +205,6 @@ architectural exclusions. The design follows
 follow-up for deferred context ownership without rewriting accepted decisions.
 No release version or additional release gate is selected here.
 
-## Work from the baseline
-
-1. **Finalize API spelling and examples.** Translate the settled decisions below
-   into concrete declarations and prepare Go examples for Audita's mapping/bytes
-   workflow, explicit overrides, non-file main contexts, and independently located
-   embedded inputs.
-2. **Carry binding intent through the public facade and load request.** Update
-   the relevant input/options declarations and conversions in `layered.go` and
-   `lifecycle.go`, retaining root-owned public types and normal ownership rules.
-   Distinguish a deferred relationship from a concrete filesystem context.
-3. **Bind underlay import contexts after main context resolution.** Extend
-   preparation and acquisition in `internal/engine/load_request.go`, `load_input.go`, and
-   `acquisition.go` as needed. Reuse the main's resolved base, root, and namespace,
-   apply the settled missing-base policy, and delegate selection to existing import
-   machinery. Preserve independently supplied contexts.
-4. **Derive overlay context and separate lookup from containment.** Remove the
-   unconditional explicit-context requirement for the new layered default.
-   Preserve explicit and ordinary lookup while using the main context's resolved
-   base inside a potentially wider root. Reuse selected-overlay loading and its
-   failure and resource-ownership behavior.
-5. **Verify the complete workflow and document it.** Add maintained consumer
-   examples, public integration coverage, and focused engine tests. Update the
-   discovery, layered-loading, import-context, and selected-overlay contracts,
-   declaration comments, and internal ownership documentation when implemented.
-
-This is dependency order for a later implementation plan, not a required commit
-layout or a requirement to expose intermediate stages to consumers.
-
-## Validation and completion
-
-Validate behavior through the public loading boundary, including:
-
-- Explicit and discovered main files, early and later search candidates, and an
-  unrelated invocation directory. Companions follow the same selected main.
-- Native file mains, bytes/readers/mappings with explicit host or native contexts,
-  and absent mains with explicit contexts. Cover absent host root-only contexts,
-  missing and invalid contexts, and context reuse without enabling main imports.
-- Mapping, bytes, and reader underlays containing optional named-file and recursive
-  directory imports with `*.yml`/`*.yaml` filtering and mounted destinations.
-- Receiving-input scope: active bindings on main inputs and file-backed underlays
-  fail with option errors; concrete context overrides retain their precedence.
-- File only, directory only, both, neither, empty directories, and invalid or
-  unreadable selected fragments. Both forms compose as peers when present.
-- Operator prompt overrides above embedded catalogs using registered identity;
-  same-tier duplicate IDs remain conflicts. Main and overlay precedence survive.
-- Mixed embedded and companion inputs, multiple underlay tiers, transitive
-  imports, placement, source-version checks, and load-wide import defaults.
-- Automatic derived overlays and explicit overrides, active and inactive names,
-  broader containment roots, and missing selected overlays.
-- Optional discovery exhaustion, required discovery, fallback and skip policies,
-  mixed payload/required-import cases from OD-2, and terminal selected-main errors.
-- Unused fallback paths need not exist or be accessible; selected fallback paths
-  receive normal validation. Inactive overlays ignore fallback settings.
-- Empty `path: []` with either requiredness value when no base exists, including
-  the independent effect of `RequireBase`, and skipped imports without fabricated
-  completed-search evidence.
-- Symlinks, eligible namespaces, parent-relative paths within the selected scope,
-  context validity, and namespace identity under OD-3 and OD-4.
-- Normal source evidence, diagnostics, passive resource contexts, cleanup, and
-  budget accounting; no duplicate lifecycle callbacks or preliminary full load.
-- Existing ordinary and explicit layered workflows and description reuse.
-
-Run focused checks during implementation and applicable normal repository checks
-before completion. No benchmark campaign is required. Completion means the
-application can discover its main, bind companion imports and the selected
-overlay, and receive normal resolved configuration and inspection in one load,
-without an application directory walker, parser, or extra discovery pass.
-
 ## Settled Decisions
 
 ### OD-1: Public representation of deferred context binding — settled