Files
configkit/docs/roadmap/implementation.md

387 lines
22 KiB
Markdown

# Configured secrets bootstrap implementation plan
## Scope and execution
Implement the [secrets-directory roadmap](secrets-directory.md) and accepted
[ADR 0023](../adr/0023-explicit-configured-secrets-bootstrap.md). The starting
revision is `fb0560d`. Read [development](../development.md), the
[agent policy](../policy/agents.md), and the applicable architecture, testing,
and documentation policies before implementing a stage.
The stages run sequentially in one sprint. Intermediate stages may leave callers
temporarily uncompilable while an internal handoff changes. Finish the shared
integration before final validation. Keep tests with their behavior's owning
stage and reuse existing fixtures and failure seams. Each stage is intended to
fit one Luna implementation prompt.
The completed feature provides explicit bootstrap before ordinary configuration
loading. Discovery and the captured main document are shared between those two
calls. Ordinary secret resolution continues to read the environment at its
existing lifecycle point. Filesystem access and bootstrap finish synchronously.
## Decisions for implementation
### Public API
Keep the existing function and its exact function type. Add these root-owned
public declarations, following the existing facade-to-engine conversion pattern.
```go
func LoadSecretsIntoEnvironment(directory string) (*SecretsReport, error)
type SecretsOptions struct {
Overwrite bool
}
func LoadSecretsIntoEnvironmentWithOptions(
directory string, options SecretsOptions,
) (*SecretsReport, error)
type SecretsFromConfigOptions struct {
Filesystem *FilesystemContext
Overwrite *bool
}
func LoadSecretsIntoEnvironmentFromConfig(
main Source, supplied ...SecretsFromConfigOptions,
) (*SecretsReport, Source, error)
```
The companion accepts zero or one options value, matching the load entry points.
Multiple values produce `LoadError` with `CodeOptionsInvalid`, an empty non-nil
report, and zero `Source` before acquisition or writes. Snapshot the supplied
context and Boolean value at call entry using existing conversion conventions.
Nil overwrite uses YAML, then false. A supplied false or true wins over YAML.
The companion accepts host `File` and `SearchConfig` sources and `NoSource`.
A prepared host source can also be supplied to an explicit companion call using
its captured main. Reject other source kinds and native filesystem contexts as
`LoadError` option failures before consuming them. Preserve constructor and search
validation diagnostics through the existing helpers. NoSource and optional search
exhaustion succeed with an empty report and `NoSource()`. Explicit context on an
absent input follows existing absent-host-context validation; subsequent absent
loading uses the ordinary caller-supplied context rules.
Every failure returns a non-nil report and zero `Source`. Discovery, acquisition,
parse, control, and preparation-context failures use `LoadError`. Directory
resolution/inspection, credential reads, and environment writes use
`SecretsError`. Keep both error wrappers and their existing safe rendering and
detached records. Successful preparation with no directory returns the prepared
main and an empty report.
### Controls and source roles
Recognize root `config.secrets_dir` as a nonempty string containing a relative
host path and root `config.secrets_overwrite` as a Boolean. Accept ordinary YAML
Boolean spellings through the parser's existing type handling. Nulls, wrong
types, empty directory strings, absolute paths, and host volume-qualified paths
are invalid. Treat path text literally using host path operations. Parent-relative
paths are usable when their canonical targets stay inside the selected root.
Only the main input's document root owns these controls. Imports, underlays,
and selected overlays containing them produce a source-local control diagnostic.
Nested application `config` mappings and root application data introduced by
placement, defaults, CLI, or resolution retain their existing meaning under
[ADR 0020](../adr/0020-document-root-controls.md).
Normal loading checks and consumes the main controls, with imports enabled or
disabled. Filesystem checks for the declared secrets directory happen in the
companion helper. Both ordinary loading entry points perform their usual secret
lookup. Preparation validates the root control mapping and the bootstrap fields,
while other control grammars and import enablement are checked by the later load.
Retain the original main tree so imports and collection declarations are still
available to that load. A valid overwrite declaration without a directory is a
no-op. Invalid bootstrap fields fail even if a Go overwrite choice is supplied.
Use existing diagnostic codes. Use `CodeSecretsInvalidPath` for invalid directory
syntax, `CodeSecretsInvalidValue` for a non-Boolean overwrite setting, and the
existing directive-context codes for misplaced controls (`CodeImportInvalid` or
`CodeOverlayDirective`). Root mapping errors keep their current classification.
Bootstrap-field diagnostics carry `/config/secrets_dir` or
`/config/secrets_overwrite` and source-local locations, inside a `LoadError`.
Avoid raw input, parser errors, or credential contents in messages.
### Captured-source ownership and context
Represent preparation privately as a captured host-file payload attached to an
engine `sourceFile`. Keep public `Source` opaque and report `SourceKindFile`.
Capture the original parsed document, original byte count, canonical main path,
canonical base, and canonical containment root. Retain the unconsumed controls.
Keep captured data read-only and give each load an owned document clone with
fresh source IDs and accounting. Namespace tokens and acquisition state belong
to each load. Retain passive path facts with the source.
Normal acquisition consumes the captured main without discovery, file reading,
or bootstrap. Apply the usual byte, document, placement, and aggregate accounting
to the captured input in every load. A preparation-time count never substitutes
for load-wide admission. Imports and resources continue to be acquired when their
normal operations run. Main deletion or modification does not change captured
contents; later filesystem operations may fail under their existing contracts.
Use the captured context when later loading supplies none. Resolve an explicit
later host context against the captured canonical main path. It is compatible
when its canonical root, base, and main path match the captured facts. Compare
path facts using the existing host path rules, allowing equivalent spellings
and root symlink aliases. Native contexts, a supplied file BaseDir, and genuinely
different roots fail with `CodeOptionsInvalid` in `LoadError`. Avoid re-resolving
the original main filename to compare contexts, since it may now point elsewhere.
Use the same check for ordinary options and layered main input contexts before
import or overlay handling can substitute a context.
### Directory bootstrap semantics
Reuse `loadSecretsDir` and its environment/access seams for all three public
helpers. Preservation mode retains both existing-variable checks, including
empty values and a variable created between preflight and application.
Overwrite mode reads existing names' eligible files and applies them regardless
of environment presence. Unrelated environment variables retain their values.
Use the existing loaded/skipped/failed/not-applied statuses. `ExistingEmpty`
continues to describe skipped existing values.
Configured directory resolution uses the captured base and root. Canonicalize
the selected directory with the existing filesystem helpers and check containment
of that canonical target before reads or writes. A directory symlink can resolve
to a contained target, following the existing bootstrap directory convention.
Return `SecretsError` with `CodeSecretsInvalidPath` for an outside-root target;
use existing filesystem classifications for missing or unreadable targets.
The direct-directory helpers keep their existing path resolution behavior.
Bootstrap includes all eligible directory entries, independently of the later
typed schema. Reuse dot-entry exclusion, regular-file checks, limits, sorted
order, newline handling, empty-file handling, and NUL rejection. Complete
metadata checks and required reads before writes. A write failure retains prior
writes and reports remaining entries as not applied. Later load failures leave
bootstrap effects in place. Reports and prepared sources retain no credential
bytes from secret files. Applications own startup ordering and concurrent
process-environment mutation under the existing bootstrap contract.
## Stage 1 — Extend shared directory bootstrap with overwrite
**Completed.**
Work in `internal/engine/secrets.go` and `internal/engine/secrets_test.go`.
1. Add a small engine options value for overwrite and keep `LoadSecretsDir`
delegating with preservation enabled. Add its options-bearing counterpart.
Extend the shared private worker and existing test seams accordingly.
2. Make overwrite affect only staging and applying values. Preserve sorted
enumeration, all metadata validation before reads, bounded reads before
writes, skipped-file behavior in preservation mode, and failure reporting.
3. Add focused behavioral cases for overwriting nonempty and empty existing
variables, overwriting with empty file content, and leaving names without
files alone. Extend the existing staged-presence-race and partial-apply-failure
cases to cover overwrite. Reuse current read-limit, dot-file, NUL, and safe
error coverage. Process-mutating tests restore state and run serially.
Stage completion means both policies work through one bootstrap worker and
existing preservation behavior remains covered. Run the relevant existing and
new `LoadSecretsDir` tests once callers compile.
## Stage 2 — Add main-scoped bootstrap control parsing
**Completed.**
Work primarily in `internal/engine/directives.go`, `acquisition.go`, and the
directive/version parsing handoffs in `source_version.go`.
1. Introduce a small private bootstrap-settings value with declaration presence,
directory text, overwrite value, and source-local diagnostic locations. Share
its field validation between preparation and normal control extraction.
2. Give extraction enough source-role information to allow these controls only
on the actual main document. `rootDirectiveContextSource` and
`rootDirectiveContextLayered` alone cannot identify that role because those
contexts also occur on contributions. Carry the role through the relevant
acquisition/version and import traversal call sites.
3. Keep bootstrap-only extraction usable before normal imports/sequence grammar
validation. The companion can prepare a main that declares imports without
enabling or acquiring them. Normal extraction checks the complete control
block and removes it before placement and application decoding.
4. Extend directive-scope and facade-control tests with main declarations,
disabled imports, wrong types, explicit false, misplaced controls, and nested
ordinary data. Verify source-version control-overlap checks recognize the
new fields and that version acceptance itself stays in the load.
Stage completion means the normal load accepts and consumes valid main settings
while leaving environment state untouched. Other source-root controls retain
their existing grammar, enablement, and error precedence.
## Stage 3 — Share host-main selection and capture
**Completed.**
Work in `internal/engine/source.go`, `parse.go`, `discovery.go`, and the initial
selection/read portion of `acquisition.go`. A focused `prepared_source.go` may
hold the new private payload and preparation helpers.
1. Factor the existing selection and bounded main parsing handoff for reuse by
normal acquisition and preparation. Reuse `resolveSearchInput`, search option
checks, `readSource`, and YAML adapter diagnostics. Keep candidate ordering,
optional exhaustion, selected-file failure, and regular-file semantics.
2. Add the captured host payload specified above. Preserve original source line
locations and all controls. Construct independent documents for subsequent
loads with `document.Clone` and `setDocumentSourceID`. Count captured bytes
under the current read limits before returning a document to acquisition.
3. For capture, resolve and validate the host base/root through existing
`filesystemOptionDiagnostics`, `resolveFilesystemContext`, and directory
validation. Store path facts without access handles or a preparation token.
4. Add internal cases proving captured data stays fixed after the original file
changes, becomes absent, or another search candidate appears. Retain ordinary
File/SearchConfig reread behavior. Cover parser/read failures and limits with
existing narrow test fixtures. Keep sources and payloads private until the
public wrapper stage.
Stage completion means the acquisition front end can consume an ordinary host
file or its captured payload through the same parsed-source handoff.
## Stage 4 — Integrate captured contexts with ordinary loading
**Completed.**
Work in `options.go`, `filesystem_context.go`, `filesystem.go`, `acquisition.go`,
`load_input.go`, `selected_overlay.go`, and `source_context.go` as needed.
1. Make context resolution select captured path facts and enforce the explicit
context equivalence rule at the shared acquisition boundary. Ensure both
imported and standalone branches receive the selected resolved context.
2. Carry the captured base/root into standalone main acquisition, ordinary
overlay resolution, and passive source-context completion. Ordinary overlays
keep their established root-relative lookup convention.
3. Preserve file-kind records, canonical paths, source-local diagnostics, and
placement projection. Reuse normal source registration and accounting.
4. Add focused internal cases for omitted/equivalent/incompatible later contexts,
explicit wider roots, a symlinked selected main, main removal, destinations,
and ordinary overlay/resource-context behavior. Compare relevant records and
values with a normal load of the same unmodified file.
Stage completion means an ordinary load uses captured contents and context
through all its normal paths. Real filesystem failures are handled at the
existing access boundaries without retaining an open handle in the source.
## Stage 5 — Integrate imports and layered main-relative consumers
Work in `import_acquisition.go`, `load_request.go`, and layered/context owners
identified by the shared acquisition handoff.
1. Feed the captured main document and resolved context into imported acquisition.
Charge its raw bytes and nodes through existing accounting exactly once per
load. Keep placement wrapper accounting and per-tier repetition tracking.
2. Treat the captured main as the same canonical host file for repeated-file and
cycle checks. A fresh load gets its own host namespace and source IDs.
3. Ensure layered acquisition propagates the captured main context to declared
MainRelative underlays and selected overlays, with existing explicit context
and fallback precedence. Preserve main-first acquisition and tier precedence.
4. Verify live imports after main capture, repeat/cycle detection involving the
captured main, source/version evidence with imports, an explicitly wider root
permitting contained parent-relative imports, and independent repeated loads.
Include one integrated layered case with a main-relative underlay and overlay.
Use existing import/placement/version suites for unchanged semantics.
Stage completion means prepared main sources behave as host main inputs across
ordinary and layered composition, metadata, and main-relative bindings.
## Stage 6 — Implement configured bootstrap orchestration
Add a focused engine companion in `secrets_from_config.go`, using the capture
and bootstrap helpers from earlier stages.
1. Validate and snapshot options, resolve the main, capture its document and
context, and extract bootstrap settings. Return the no-directory and absent
cases under the public return contract. A valid Go overwrite override does
not bypass validation of malformed YAML bootstrap fields.
2. Join the declared directory to the captured base. Add an optional internal
containment-root constraint to shared directory preparation so canonical
resolution and containment validation occur before the worker uses that
canonical directory. Direct-directory calls supply no such constraint.
3. Select overwrite using Go, YAML, then false, and invoke the shared worker.
Return the prepared source only after successful bootstrap. All branches
return the non-nil report and appropriate existing error family.
4. Test no declaration, overwrite without a directory, optional/required missing
main, unsupported source/context, malformed controls, missing or outside-root
directory, and directory symlink containment. Assert empty/partial reports and
zero prepared sources on errors. Use the existing environment seam for
deterministic apply failures and verify safe error/report output.
Stage completion means configured bootstrap can finish independently of schema,
imports, version hooks, resolution, and validation, returning a reusable main
only when its own operation succeeds.
## Stage 7 — Expose root-owned APIs and convert both error families
Work in root `secrets.go`, `source.go`, and existing facade conversion owners.
1. Add the two options types and new functions with the signatures above. Keep
the existing function type intact and route its default behavior through the
shared bootstrap implementation.
2. Convert the companion's report, prepared engine Source, and errors to the
existing public wrappers. Preserve `LoadError` source records as well as
diagnostics. Preserve `SecretsError` without exposing underlying filesystem
or environment errors. Add only the small conversion logic this boundary
needs, reusing existing converters.
3. Document signatures, options precedence, reports, error types, and source
ownership on their declarations. Explain captured reuse versus File/SearchConfig
reuse in Source's documentation. Keep implementation types private.
4. Add facade tests for both new functions and assignment of the old function to
`func(string) (*SecretsReport, error)`. Exercise YAML true with explicit Go
false, YAML false with Go true, omitted choices, and multiple options. Verify
public `errors.As` behavior for preparation and bootstrap failures.
Stage completion means applications can use the complete two-call startup flow
through ConfigKit's public package with the documented ownership and errors.
## Stage 8 — Verify the complete startup handoff
Extend public integration coverage and add one maintained consumer startup
example with ordinary and layered use explained from the same helper.
1. Start in a different working directory, discover a main with relative secrets,
bootstrap all eligible entries including an SDK-only name, and load the
prepared source. Verify configured overwrite and existing empty-variable
behavior with disposable environment names.
2. Change the main, discovery candidates, an import, and an environment value
after preparation. Verify captured main values, live import values, fresh
environment lookup, and independent metadata/results across subsequent loads.
Include selected-main provenance and source resource context assertions.
3. Exercise the layered handoff with explicit root preparation and companion
inputs. Confirm compatible later context succeeds and incompatible context
fails safely after the already-completed bootstrap.
4. Show a later version/decoding or validation failure preserves the bootstrap
report and environment effects. Separately verify ordinary Load/LoadLayered
on a main containing the controls perform no environment writes, even with
YAML overwrite true. Existing secret-resolution tests continue to own
credential lookup and redaction behavior.
5. Verify captured sources retain no handles through successful and failed
preparation. Use existing resource-lifetime techniques and narrow seams,
avoiding a new filesystem framework. Exercise independent captured-source
loads under the race detector without concurrent process mutation.
Stage completion means the feature works through normal entry points and its
integration risks are covered without duplicating every internal test at the
facade boundary.
## Stage 9 — Update canonical documentation and validate the sprint
1. Update `docs/config.md` with the two main controls, preparation and bootstrap
boundaries, lookup timing, overwrite precedence, reports/errors, host scope,
and prepared-source reuse/context behavior. Keep source-local controls
distinct from application paths and preserve existing loading semantics.
2. Update `docs/consumers/README.md` and its test map to link the new maintained
example. Update `docs/development.md` only where navigation/test ownership
changes. Explain internal capture and acquisition handoffs in
`docs/internal/overview.md` and relevant import documentation.
3. Update the implemented-behavior paragraphs in architecture policy and retire
its roadmap implementation-status link. Keep ADR 0023's accepted decision
text intact. Preserve the existing bootstrap limits and concurrency/rollback
contract while documenting explicit overwrite as the selected extension.
4. Format changed Go files and run the focused tests for changed owners. Run
`./scripts/check.sh` and `go test -race ./...` on the completed implementation.
Verify edited Markdown links and headings, exported declaration comments,
and the public example. Reuse successful checks for unchanged content.
5. Review the final diff for captured-state aliasing, handles, extra parsing or
discovery, secret disclosure, omitted accounting, and accidental changes to
current configuration processing or helper signatures. Resolve failures and
finish the documented target before marking the sprint complete.
The final result is the implemented feature with current documentation and
appropriate behavioral coverage. Release preparation follows its separate
procedure when requested.